A method and apparatus for mitigating high latency of a neural network model on a hardware device

Abstract

The application discloses a high-delay relieving method and device of a neural network model on a hardware device. The method comprises the following steps: initializing basic hyperparameters and related environments of a deep neural network model, constructing a pruning computation graph of the deep neural network model, determining network layers suitable for pruning in the deep neural network model through the pruning computation graph, and merging sibling nodes with common parent nodes; resetting an exploration environment of the neural network model, setting a predetermined exploration number, obtaining observation values corresponding to states in each exploration environment through an agent, and determining the number of prunings of each network layer suitable for pruning in each exploration; pruning the deep neural network model according to the number of prunings of each network layer in each exploration, and determining the deep neural network model after pruning with the predetermined exploration number; evaluating the accuracy of the deep neural network model with the predetermined exploration number by using a real data set collected, and determining an optimal deep neural network model and the pruning parameters of each layer thereof.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and more particularly, to a high-latency mitigation method and device for a neural network model on a hardware device. BACKGROUND

[0002] In recent years, deep neural networks (DNN) play a vital role in autonomous driving, content generation, information recommendation and other artificial intelligence applications. As the problem size and dataset size increase, the model also becomes larger and larger, from the initial 6000 million parameter quantity of AlexNet to the current 1750 billion parameter quantity of GPT-3. Such a huge amount of parameters brings problems in terms of latency, storage, computing power and energy consumption, making it extremely difficult to deploy deep neural network models on devices with limited resources.

[0003] In order to enable deep neural network models to be deployed on resource-constrained devices, relevant researchers have gradually begun to design neural network models with lightweight structures, such as the MobileNet series, the ShuffleNet series and the Xception series. The common feature of these lightweight neural network models is to split the ordinary convolution, so that the parameter quantity and the calculation quantity of the neural network model are greatly reduced on the basis of keeping the precision unchanged. However, manually designing lightweight neural network structures requires a lot of professional practical experience and time cost, and is easy to fall into a local optimal solution. In view of these problems, neural network architecture search (NAS) has emerged, which can automatically find the optimal neural network structure for a specific task on a specific dataset according to the set parameter quantity and calculation quantity, avoiding the tediousness of manually designing neural network structures. The emergence and development of these technologies have made it possible to deploy deep neural network models on resource-constrained devices.

[0004] But no matter whether the lightweight network MobileNetv1 designed manually or MobileNetv3 searched through the neural network architecture search algorithm, the network parameters still have redundancy, that is, there is a smaller parameter network structure, and within a certain number of training times, the performance can reach similar or even better performance than the original network model. Pruning is a common method in the field of deep neural network model compression and acceleration, which can remove redundant parameters in the neural network model without losing model accuracy as much as possible, so as to reduce the parameter quantity and calculation quantity of the model. The current pruning method is divided into structured pruning and unstructured pruning, the structural granularity of structured pruning operation is larger, such as the whole convolutional layer can be pruned, but the dependence between convolutional layers may be considered in the pruning process; the granularity of unstructured pruning operation is finer, which can prune the kernel parameters in the convolutional layer, such that the pruning method is not restricted by any condition and has less damage to the model when pruning the neural network model, but requires hardware to support sparse matrix acceleration operation.

[0005] At present, the most practical application is structured pruning, and the neural network model after pruning almost does not depend on hardware design to achieve acceleration effect, but the acceleration effect on a specific hardware device may not reach the theoretical value, and even the acceleration effect is much lower than the theoretical value. SUMMARY

[0006] In view of the defects of the prior art, the application provides a high latency relief method and device of a neural network model on a hardware device.

[0007] According to one aspect of the application, a high latency relief method of a neural network model on a hardware device is provided, comprising:

[0008] Initializing the basic hyperparameters and related environment of the deep neural network model, constructing a pruning computation graph of the deep neural network model, and determining the network layer in the deep neural network model suitable for pruning through the pruning computation graph;

[0009] Resetting the exploration environment of the neural network model, setting a predetermined number of explorations, and obtaining the observation value corresponding to the state in each exploration environment through an agent, and determining the pruning number of each network layer suitable for pruning in each exploration;

[0010] According to the pruning number of each network layer in each exploration, the deep neural network model is pruned to determine the deep neural network model after pruning with a predetermined number of explorations;

[0011] Using the collected real data set to evaluate the accuracy of the deep neural network model with a predetermined number of explorations, and determining the optimal deep neural network model and the pruning parameters of each layer thereof.

[0012] Optionally, further comprising: updating the related parameters of the agent including the neural network of the agent itself and the experience replay pool according to the performance indicators of the optimal deep neural network model and the pruning parameters of each layer thereof after each exploration.

[0013] Optionally, a pruning computation graph of the deep neural network model is constructed, and the network layer suitable for pruning in the deep neural network model is determined through the pruning computation graph, including:

[0014] The prunable convolutional layer and the fully connected layer analyzed in the process of constructing the pruning computation graph, and the previous layer and the next layer of the current network layer are determined according to the dependency relationship between the layers in the pruning computation graph.

[0015] According to the relationship between the pruning computation graph and the layer nodes, the sibling nodes with a common parent node can be merged to determine the network layer suitable for pruning.

[0016] Optionally, the accuracy of the deep neural network model with the predetermined number of explorations is evaluated using the collected real data set, and the optimal deep neural network model and the pruning parameters of each layer thereof are determined, including:

[0017] Using the real data set, the related accuracy indicators are obtained according to the task type of the deep neural network model.

[0018] The inference time of the deep neural network model on a specific inference device is obtained.

[0019] The accuracy indicators and the inference time are fused to calculate the reward of the agent corresponding to the deep neural network model with the predetermined number of explorations, and the optimal deep neural network model and the pruning parameters of each layer thereof are determined.

[0020] Optionally, the formula for fusing the accuracy indicators and the inference time is as follows:

[0021]

[0022] Wherein, ind represents the accuracy indicator of the pruning model on the real data set; time pruning represents the actual inference time of the pruning model on the specific hardware; time dense represents the inference time of the original neural network model on the specific hardware; and a is an auxiliary coefficient, when a = 1, the accuracy indicator of the pruning model is taken as the reward of the agent; when a = 0, the inference time of the pruning model is taken as the reward of the agent; when a ∈ (0, 1), the accuracy indicator and the inference time of the pruning model are taken as the reward of the agent at the same time.

[0023] According to another aspect of the present application, a high latency mitigation device of a neural network model on a hardware device is provided, including:

[0024] The first determining module is configured to initialize basic hyperparameters and related environments of the deep neural network model, construct a pruning computation graph of the deep neural network model, and determine network layers suitable for pruning in the deep neural network model through the pruning computation graph.

[0025] The second determining module is configured to reset an exploration environment of the neural network model, set a predetermined number of explorations, and determine the number of pruned network layers in each exploration through an agent to obtain observation values corresponding to states in each exploration environment.

[0026] The third determining module is configured to prune the deep neural network model according to the number of pruned network layers in each exploration, and determine a deep neural network model after pruning for the predetermined number of explorations.

[0027] The fourth determining module is configured to evaluate the accuracy of the deep neural network model after pruning for the predetermined number of explorations using a real data set, and determine an optimal deep neural network model and pruning parameters of each layer thereof.

[0028] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program for executing the method according to any one of the above aspects of the present application.

[0029] According to another aspect of the present application, an electronic device is provided, which comprises a processor, a memory for storing executable instructions of the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of the above aspects of the present application.

[0030] Therefore, the method for relieving high latency of a neural network model on a hardware device can be applied in the fields of computer vision and natural language processing, and can compress weights of a neural network model containing convolutional layers or fully connected layers to obtain a faster inference speed on a specific hardware device with a slight loss of accuracy, thereby effectively relieving the problem of high latency of a neural network model on a specific hardware device such as an edge device. BRIEF DESCRIPTION OF DRAWINGS

[0031] The exemplary embodiments of the present application can be more completely understood in reference to the following drawings:

[0032] Figure 1 is a flowchart of the method for relieving high latency of a neural network model on a hardware device according to an exemplary embodiment of the present application;

[0033] Figure 2 is a schematic diagram of network layer merging in the process of constructing a pruning computation graph according to an exemplary embodiment of the present application;

[0034] Figure 3 FIG. 1 is a pruning diagram of a pruning inter-layer parameter-containing operator layer provided by an example embodiment of the present application;

[0035] Figure 4 FIG. 2 is a reward function simulation diagram provided by an example embodiment of the present application;

[0036] Figure 5 FIG. 3 is a structure diagram of a high-latency relieving device of a neural network model on a hardware device provided by an example embodiment of the present application;

[0037] Figure 6 FIG. 4 is a structure of an electronic device provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0038] Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and are not all the embodiments of the present application. It should be understood that the present application is not limited to the described example embodiments.

[0039] It should be noted that: unless otherwise specified, the relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0040] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present application are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor do they represent the inevitable logical sequence between them.

[0041] It should also be understood that in the embodiments of the present application, "a plurality of" can mean two or more, and "at least one" can mean one, two or more.

[0042] It should also be understood that for any component, data or structure mentioned in the embodiments of the present application, unless specifically limited or given the opposite implication by the context, it can be understood as one or more in general.

[0043] In addition, the term "and / or" in the present application is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the existence of A alone, the existence of A and B together, and the existence of B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0044] It should also be understood that the description of each embodiment of the present application emphasizes the differences between each embodiment, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0045] It should be understood that the dimensions of the various elements shown in the figures are chosen for convenience only, and do not bear any relationship to actual proportions.

[0046] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the application or its application or uses.

[0047] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered part of the specification.

[0048] It is to be noted that like reference numerals and letters refer to like items in the figures, and once an item is defined in one figure, it need not be discussed further in subsequent figures.

[0049] Embodiments of the application can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with terminal devices, computer systems, servers, etc. include, but are not limited to: personal computers, servers, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputers, mainframe computers, and including any system of any of the above, distributed cloud computing environments, etc.

[0050] Terminal devices, computer systems, servers, etc. can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer systems / servers can be practiced in distributed cloud computing environments with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media including memory storage devices.

[0051] Exemplary method

[0052] Figure 1 is a flowchart of a high latency mitigation method of a neural network model on a hardware device provided by an exemplary embodiment of the application. The present embodiment can be applied to electronic devices such as terminal devices, computer systems, servers, etc. Figure 1 As shown in the figure, the high latency mitigation method 100 of the neural network model on the hardware device includes the following steps:

[0053] Step 101, initialize the basic hyperparameters of the deep neural network model and the related environment, construct a pruning computation graph of the deep neural network model, and determine the network layers suitable for pruning in the deep neural network model through the pruning computation graph;

[0054] Step 102, reset the exploration environment of the neural network model, set the predetermined number of explorations, and obtain the observation value corresponding to the state in each exploration environment through the agent, and determine the pruning number of each network layer suitable for pruning in each exploration;

[0055] Step 103, according to the pruning number of each network layer of each exploration, the deep neural network model is pruned, and the deep neural network model after pruning is determined.

[0056] Step 104, using the collected real data set to evaluate the accuracy of the deep neural network model of the predetermined number of explorations, and determining the optimal deep neural network model and the pruning parameters of each layer thereof.

[0057] Optionally, it also includes: updating the related parameters of the agent after each exploration according to the performance indicators of the optimal deep neural network model and the pruning parameters of each layer thereof, including the neural network of the agent itself and the experience replay pool.

[0058] Optionally, the pruning computation graph of the deep neural network model is constructed, and the network layers suitable for pruning in the deep neural network model are determined through the pruning computation graph, including:

[0059] According to the analysis of the prunable convolutional layer and the fully connected layer in the process of constructing the pruning computation graph, and according to the dependency relationship between the layers in the pruning computation graph, the previous layer and the next layer of the current network layer are determined.

[0060] According to the relationship between the pruning computation graph and the layer nodes, the sibling nodes with common parent nodes can be merged to determine the network layers suitable for pruning.

[0061] Optionally, the accuracy of the deep neural network model of the predetermined number of explorations is evaluated using the collected real data set, and the optimal deep neural network model and the pruning parameters of each layer thereof are determined, including:

[0062] Using the real data set, the related accuracy indicators are obtained according to the task type of the deep neural network model;

[0063] Obtaining the inference time of the deep neural network model on a specific inference device;

[0064] The accuracy indicators and inference time are fused to calculate the reward of the agent corresponding to the deep neural network model of the predetermined number of explorations, and the optimal deep neural network model and the pruning parameters of each layer thereof are determined.

[0065] Optionally, the formula of fusing the precision indicator and the inference time is as follows:

[0066]

[0067] Wherein, ind represents the precision indicator of the pruned model on the real data set; time pruning represents the actual inference time of the pruned model on the specific hardware; time dense represents the inference time of the original neural network model on the specific hardware; alpha is an auxiliary coefficient, when alpha = 1, the precision indicator of the pruned model is taken as the reward of the agent; when alpha = 0, the inference time of the pruned model is taken as the reward of the agent; when alpha is in (0, 1), the precision indicator and the inference time of the pruned model are taken as the reward of the agent at the same time.

[0068] Specifically, the application relieves the high delay of the neural network model on the hardware device through the deep neural network model pruning method of software and hardware cooperation, and the specific steps are as follows:

[0069] Step S1: initialize basic hyperparameters and related environment, such as the maximum number of iterations of reinforcement learning, the number of preheating, the size of the experience replay pool, the batch data size, and the inference time of the original neural network model and other hyperparameters;

[0070] Step S2: build a pruning computation graph of the deep neural network model, which can automatically analyze the network layers suitable for pruning in the model through the pruning computation graph, mainly perform pruning operation on the convolutional layer and the fully connected layer in the neural network model, and combine the sibling nodes with common parent nodes;

[0071] Step S3: reset the exploration environment, obtain the initial observation value, and obtain the action that should be taken according to the observation value;

[0072] Step S4: explore the layers that can be pruned in the neural network model according to the initial observation value, to obtain the best pruning number of each layer;

[0073] Step S5: prune the deep neural network model according to the pruning number of each layer;

[0074] Step S6: use the real data set to evaluate the precision of the model, and obtain the inference time of the model on the specific hardware;

[0075] Step S7: save the optimal pruned model and the pruning parameters of each layer thereof;

[0076] Step S8: update the related parameters of the agent, including the neural network of the agent itself and the experience replay pool.

[0077] The soft and hardware cooperative deep neural network model pruning method has the characteristics that the step S1 specifically comprises:

[0078] Step S101: The inference time of the original neural network model is not randomly initialized, but the real inference time of the model on a specific hardware (including a simulator);

[0079] Step S102: The initialization of the hardware environment can be a docker environment containing a hardware simulator, or a real hardware environment. The docker environment containing the hardware simulator automatically loads the corresponding image package and builds the corresponding container during initialization, and the real hardware environment needs to prepare the related model deployment environment during initialization.

[0080] The soft and hardware cooperative deep neural network model pruning method has the characteristics that the step S2 specifically comprises:

[0081] Step S201: During the construction of the pruning computation graph, the convolutional layers and fully connected layers that can be pruned are automatically analyzed, and according to the dependency relationship between the layers in the pruning computation graph, the previous layer (parent node) and the next layer (child node) of the current network layer (current node) can be determined;

[0082] Step S202: According to the pruning computation graph and the relationship between the nodes, the sibling nodes with the same parent node can be merged to solve the inter-layer dependency problem in the pruning process.

[0083] The soft and hardware cooperative deep neural network model pruning method has the characteristics that the step S3 specifically comprises:

[0084] Step S301: The exploration method adopts the DQN (Deep Q-Network) algorithm in reinforcement learning, that is, the action of the agent is a discrete value, specifically the specific pruning number of the network layer that can be pruned;

[0085] Step S302: The observation value of the agent is composed of a group of vectors, which includes operator information such as type information, calculation amount, parameter amount and channel number of the network layer, and also includes hardware end information such as the number of calculations of the network layer on the hardware, the number of DMA (Direct Memory Access) read data bytes, and the number of DMA write data bytes.

[0086] The soft and hardware cooperative deep neural network model pruning method has the characteristics that the step S4 specifically comprises:

[0087] Step S401: The number of pruned layers is determined by the agent according to the observation value. According to the hardware instruction characteristics, the hardware can perform coarse-grained calculation in groups of 32 channels in the input channel direction. Therefore, the number of pruned layers is a multiple of 32.

[0088] Step S402: Select the pruning threshold according to the obtained number of pruned channels. The input channel total value less than the pruning threshold needs to be pruned, and the index of the unpruned channel is recorded.

[0089] The above-mentioned soft and hardware cooperative deep neural network model pruning method, characterized in that, the step S5 specifically comprises:

[0090] Step S501: After all the prunable network layers in the neural network model are pruned, the neural network model is hard pruned according to the number of pruned layers and the index of unpruned channels. The number of channels in the pruned model changes, and the computational complexity and parameter quantity of the model are substantially reduced.

[0091] Step S502: During the pruning process, other parameter-containing operator layers between the pruned layers are also pruned according to the context pruning information, such as BatchNorm layers and PReLU layers.

[0092] Step S503: The number of pruned layers determined by the agent may exceed the number of channels of the pruned layer. For this case, maximum pruning is needed when hard pruning the model, i.e. only 32 channels are left after pruning the operator layer, and the number of channels and filters of the corresponding network layer is adjusted according to the context pruning information of the current layer.

[0093] The above-mentioned soft and hardware cooperative deep neural network model pruning method, characterized in that, the step S6 specifically comprises:

[0094] Step S601: The pruned model is directly evaluated using a real data set, and relevant evaluation indicators are obtained according to the task type.

[0095] Step S602: The pruned model needs to be inferred on a specific hardware (including a docker environment containing a simulator) to obtain the inference time of the model on the specific device.

[0096] Step S603: The precision indicator and inference time of the model are fused as the reward of the agent in this round, and the specific calculation formula is:

[0097]

[0098] Where ind represents the accuracy indicator of the pruned model on the real data set; time pruningdenotes the actual inference time of the pruned model on the specific hardware; time dense denotes the inference time of the original neural network model on the specific hardware; alpha is an auxiliary coefficient, when alpha = 1, the accuracy indicator of the pruned model is taken as the reward of the agent; when alpha = 0, the inference time of the pruned model is taken as the reward of the agent; when alpha is in (0, 1), the accuracy indicator and the inference time of the pruned model are taken as the reward of the agent.

[0099] The soft and hardware cooperative deep neural network model pruning method described above, characterized in that the step S7 specifically comprises:

[0100] Step S701: Take the reward of the agent as the indicator for selecting the model, the higher the reward, the better the theoretical performance of the model;

[0101] Step S702: When saving the optimal pruned model, the pruning parameters of each layer of the neural network model are also saved to facilitate the loading and training of the subsequent model.

[0102] The soft and hardware cooperative deep neural network model pruning method described above, characterized in that the step S8 specifically comprises:

[0103] Step S801: When updating the experience replay pool of the agent, an action taken by the agent in each round needs to be assigned a reward, and by default, the reward is the final reward of the round, but if the action is greater than the input channel number of the current pruning layer, the reward of the action is set to -1.0.

[0104] Step S802: According to the information returned by the hardware end, the observation value vector of the agent is updated, and the observation value vector, the reward and the action are updated into the experience replay pool.

[0105] The specific examples realized by the method provided by the present application are as follows:

[0106] Step S1: Initialize basic hyperparameters and related environments, such as the maximum number of iterations of reinforcement learning, the number of preheating, the size of the experience replay pool, the batch data size, and the inference time of the original neural network model, etc. hyperparameters. Specifically:

[0107] The maximum number of iterations of reinforcement learning max_episodes is set to 1000, the prediction number warm_up is 150, the experience replay pool size buffer_size is 6400, the batch data size batch_size is 256, the hidden layer of the agent model hidden is 256, the inference time of the original neural network model time_dense is 3.677 (the inference time of ResNet50-CIFAR10 on specific hardware, in milliseconds), the maximum input channel number in ResNet50 is 2048, and therefore the action range of the agent action_range is [32, 64, 96, …, 2016].

[0108] Taking the initialization of a docker environment containing a hardware simulator as an example, first, an image package of a corresponding hardware tool chain is loaded, and parameters such as a container name container_name, a mapping directory of a local host localhost_dir and a container container_dir, and an environment variable PYTHONPATH are set, and a container is created according to these parameters.

[0109] Step S2: The Trace function provided in the deep learning framework PyTorch can be used to track the data flow, and a pruning computation graph can be constructed in this way. During the construction of the pruning computation graph, all convolutional layers and fully connected layers are extracted as candidate pruning layers, and the convolutional layers and fully connected layers that can be pruned are automatically analyzed according to the constraint that the input channel number is a multiple of 32.

[0110] According to the tracking of the data flow, the dependency relationship between layers in the pruning computation graph can be determined, and according to the pruning computation graph and the relationship between nodes, sibling nodes with a common parent node can be merged to solve the inter-layer dependency problem in the pruning process. A merging schematic diagram is shown in Figure 2 , and a specific example is:

[0111] The weight shape of Node 1 convolutional layer is (512, 256, 1, 1), and the weight shape of Node 2 convolutional layer is (128, 256, 1, 1), and they have a common parent node ReLU layer, so Node 1 and Node 2 can be merged, and the weight shape after merging is (640, 256, 1, 1). It should be noted that the merged network layer is only used for pruning in the subsequent steps to make Node 1 and Node 2 have the same pruning parameters, and the actual neural network model is not merged.

[0112] Step S3: The exploration method uses the DQN algorithm in reinforcement learning, i.e., the action of the agent is a discrete value, specifically the specific pruning number of the prunable network layer. Taking ResNet50 as an example, the action range of the agent is [32, 64, 96, …, 2016].

[0113] The observation value of the agent is composed of a group of vectors, which include operator information such as type information, calculation amount, parameter amount, and channel number of the network layer, and hardware end information such as the number of calculations of the network layer on the hardware, the number of read data bytes from the DMA, the number of write data bytes to the DMA, and the like.

[0114] The specific composition of the observation value of the agent is [op_type, filter, channel, height, width, kernel, stride, macs, params, reduced, cal_amount, rdma, wdma, bandwidth, action] 15-dimensional information, wherein op_type is the operator type, 0 represents convolution, and 1 represents matrix multiplication (full connection layer), filter represents the filter number of the network layer, channel represents the input channel number of the network layer, height represents the height of the output feature of the network layer, width represents the width of the output feature of the network layer, kernel represents the kernel size of the convolution (0 for the full connection layer), stride represents the stride of the convolution (0 for the full connection layer), macs represents the calculation amount of the network layer, params represents the parameter amount of the network layer, reduced represents the calculation amount of the network layer that has been pruned, cal_amount represents the number of calculations of the network layer on the hardware, rdma represents the number of read data bytes of the network layer from the DMA, wdma represents the number of write data bytes of the network layer to the DMA, bandwidth represents the on-chip cache bandwidth, and action represents the decision of the current agent for the network layer.

[0115] Step S4: The pruning number of each layer is decided by the agent according to the observation value. According to the hardware instruction characteristics, the hardware can perform coarse-grained calculation in groups of 32 channels in the input channel direction, so the pruning number of each layer is a multiple of 32. Taking a convolution with a weight shape of (256, 128, 3, 3) as an example, the input channel number is 128, so the effective strategies for the agent to explore the network layer are [32, 64, 96] three kinds, and the ineffective strategies are [128, 160, …, 2016].

[0116] According to the obtained number of pruned channels, a threshold value for pruning is selected, which is illustrated in the example in step S4. Assuming that the decision of the agent is 64, the absolute values of the weights of the convolutional layer are summed to obtain a vector value with a length of 128, and the element value [i] in the vector represents the value of channel i. The vector value is sorted in ascending order, and the threshold threshold is calculated according to the pruning number 64 decided by the agent. The channels with values less than the threshold need to be pruned, and the indexes of the channels that are not pruned are recorded. Assuming that the indexes of the channels that are not pruned are [1, 2, 5, 13, 20, …] and there are 64 channels in total.

[0117] Step S5: When all the prunable network layers in the neural network model are pruned, the neural network model is pruned according to the pruning number of each layer and the indexes of the channels that are not pruned, that is, the number of channels of the pruned model changes, and the calculation amount and parameter amount of the model are substantially reduced.

[0118] In the pruning process, other parameter-containing operator layers between the pruned layers also need to be pruned according to the context pruning information, such as the BatchNorm layer and the PReLU layer. As shown in Figure 3

[0119] Assuming that the node Node 3 convolutional layer is pruned, the strategy adopted by the agent is 32, that is, 32 channels are pruned from the input channels, and the remaining 96 channels. The last prunable convolutional layer of the node Node 3 convolutional layer is Node 1, so the filters of the Node 1 convolutional layer need to be pruned, and the number of pruned filters is 96. Between Node 1 and Node 3, there is a BatchNorm layer containing training parameters, so the parameters of the network layer need to be pruned, and the pruned parameters are 96.

[0120] The number of pruned channels decided by the agent may exceed the number of channels of the pruned layer. For this case, maximum pruning is needed when the model is pruned, that is, only 32 channels are left after the operator layer is pruned, and the number of channels and the number of filters of the corresponding network layer are adjusted according to the context pruning information of the current layer.

[0121] Step S6: The pruned model is directly evaluated using a real data set, and relevant evaluation indicators are obtained according to the task type. For a classification task, Acc@Top1 is used as an evaluation indicator, and for a detection task, mAP@0.5 is used as an evaluation indicator. The pruned model needs to be inferred on a specific hardware (including a docker environment containing a simulator) to obtain the inference time of the model on the specific device.

[0122] The precision indicator and the inference time of the model are fused as the reward of the agent in this round, and the specific calculation formula is: ​

[0123]

[0124] wherein, ind represents the accuracy indicator of the pruned model on the real dataset; time pruning represents the actual inference time of the pruned model on the specific hardware; time dense represents the inference time of the original neural network model on the specific hardware; alpha is an auxiliary coefficient, when alpha = 1, the accuracy indicator of the pruned model is taken as the reward of the agent; when alpha = 0, the inference time of the pruned model is taken as the reward of the agent; when alpha is in (0, 1), the accuracy indicator and the inference time of the pruned model are taken as the reward of the agent at the same time. The default value of alpha is 0.5.

[0125] The simulation effect of the reward function is as shown in Figure 4 The x-axis (acc) represents the accuracy of the pruned ResNet50-CIFAR10, the y-axis (ratio) represents the speedup ratio of the pruned model on the hardware, and the calculation formula is

[0126]

[0127] The z-axis (reward) represents the reward value. The function satisfies that the higher the model accuracy, the greater the speedup ratio, the better the performance of the model, and the higher the corresponding reward.

[0128] Step S7: The reward of the agent is taken as the index for selecting the model, the higher the reward, the better the theoretical performance of the model, and the pruned model and the pruning parameters of each layer of the neural network model are saved when the optimal pruned model is saved, so as to facilitate the loading and training of the subsequent model.

[0129] Step S8: When updating the experience replay pool of the agent, an action taken by the agent in each round needs to be assigned a reward, and by default, the reward is the final reward of the round, but if the action is greater than the input channel number of the current pruning layer, the reward of the action is set to -1.0. At the same time, according to the information returned by the hardware end, the observation value vector of the agent is updated, and the observation value vector, the reward and the action are updated into the experience replay pool.

[0130] The application proposes a high delay relief method of a network model on a hardware device, which has the following beneficial effects compared with the existing technology:

[0131] The application proposes to use the DQN algorithm to output the action of the agent as a discrete value, specifically, the value of the action is related to the instruction of convolution and matrix multiplication calculation of the specific hardware, so as to ensure that the pruned model can fully utilize the related instructions on the specific hardware to achieve the optimal acceleration effect.

[0132] The observation tensor of the intelligent agent provided by the application comprises not only the operator type parameters such as the pruning layer type, the calculation amount and the parameter amount of the current pruning layer, but also the hardware end information such as the calculation number of the network layer on the hardware, the number of reading data bytes from the DMA and the number of writing data bytes to the DMA, so that the intelligent agent makes a reasonable decision under the premise of referring to the software and hardware related information.

[0133] The expected reward calculation formula of the intelligent agent provided by the application can consider both the model accuracy after pruning and the actual inference time of the hardware end, so that the accuracy and inference time of the model after retraining after pruning reach the theoretical optimum.

[0134] The pruning computation graph provided by the application can automatically analyze the convolutional layers and fully connected layers that can be pruned, and merge the layers with common parent nodes, so as to effectively solve the problem that part of the network layers cannot be pruned due to the dependency relationship between layers in the pruning process.

[0135] Therefore, the high delay relief method of the network model on the hardware device provided by the application can be applied in the field of computer vision and natural language processing, and can perform weight compression on the neural network model containing convolutional layers or fully connected layers, so that the accuracy is slightly lost, and the inference speed on specific hardware devices is faster, effectively relieving the high delay problem of the neural network model on the edge device and other specific hardware devices.

[0136] Exemplary apparatus

[0137] Figure 5 is a structural schematic diagram of the high delay relief device of the neural network model on the hardware device provided by an exemplary embodiment of the application. Figure 5 As shown in the figure,

[0138] The first determination module 510 is configured to initialize the basic hyperparameters and related environment of the deep neural network model, construct a pruning computation graph of the deep neural network model, and determine the network layers suitable for pruning in the deep neural network model through the pruning computation graph.

[0139] The second determination module 520 is configured to reset the exploration environment of the neural network model, set a predetermined number of explorations, and obtain the observation value corresponding to the state in each exploration environment through the intelligent agent, and determine the pruning number of each network layer suitable for pruning in each exploration.

[0140] The third determination module 530 is configured to prune the deep neural network model according to the pruning number of each network layer in each exploration, and determine the deep neural network model after pruning for the predetermined number of explorations.

[0141] The fourth determination module 540 is configured to evaluate the accuracy of the deep neural network model with the predetermined number of explorations using the collected real data set, and determine the optimal deep neural network model and the pruning parameters of each layer thereof.

[0142] Optionally, the apparatus 500 further comprises an updating module configured to update the related parameters of the agent, including the neural network of the agent itself and the experience replay pool, according to the performance indicators of the optimal deep neural network model and the pruning parameters of each layer thereof after each exploration.

[0143] Optionally, the first determination module 510 comprises:

[0144] The first determination sub-module is configured to determine the previous layer and the next layer of the current network layer according to the prunable convolutional layers and fully connected layers analyzed in the process of constructing the pruning computation graph, and according to the dependency relationship between the layers in the pruning computation graph.

[0145] The second determination sub-module is configured to determine the network layer suitable for pruning according to the pruning computation graph and the relationship between the layer nodes, and to merge the sibling nodes with a common parent node.

[0146] Optionally, the fourth determination module 540 comprises:

[0147] The first acquisition sub-module is configured to acquire the related accuracy indicators according to the task type of the deep neural network model using the real data set.

[0148] The second acquisition sub-module is configured to acquire the inference time of the deep neural network model on a specific inference device.

[0149] The fusion module is configured to fuse the accuracy indicators and the inference time, calculate the reward of the agent corresponding to the deep neural network model with the predetermined number of explorations, and determine the optimal deep neural network model and the pruning parameters of each layer thereof.

[0150] Optionally, the formula for fusing the accuracy indicators and the inference time is as follows:

[0151]

[0152] wherein ind represents the accuracy indicator of the pruned model on the real data set; time pruning represents the actual inference time of the pruned model on the specific hardware; time dense represents the inference time of the original neural network model on the specific hardware; and a is an auxiliary coefficient, when a = 1, the accuracy indicator of the pruned model is taken as the reward of the agent; when a = 0, the inference time of the pruned model is taken as the reward of the agent; and when a ∈ (0, 1), the accuracy indicator and the inference time of the pruned model are taken as the reward of the agent at the same time.

[0153] Exemplary electronic device

[0154] Figure 6 is a configuration of an electronic device according to an exemplary embodiment of the present application. As shown in Figure 6 the electronic device 60 includes one or more processors 61 and a memory 62.

[0155] The processor 61 can be a central processing unit (CPU) or other form of processing unit that has data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0156] The memory 62 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 61 can run the program instructions to implement the methods of the software programs of the various embodiments of the present application described above and / or other desired functions. In one example, the electronic device can further include an input device 63 and an output device 64, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0157] In addition, the input device 63 can include, for example, a keyboard, a mouse, and / or the like.

[0158] The output device 64 can output various information to the outside. The output device 64 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0159] Of course, in order to simplify, Figure 6 only some of the components related to the present application among the electronic device are shown, and components such as a bus, an input / output interface, and / or the like are omitted. In addition to this, the electronic device can include any other appropriate components according to a specific application.

[0160] Exemplary computer program product and computer readable storage medium

[0161] In addition to the above-described methods and devices, embodiments of the present application can be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the methods according to various embodiments of the present application described in the above "Exemplary Methods" section of the specification.

[0162] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Python, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0163] Furthermore, embodiments of the present application can also be a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform steps described in the above "Exemplary Method" section of the present specification.

[0164] The computer readable storage medium can be a combination of one or more computer readable media. The computer readable media can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or apparatus or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] The above describes the basic principles of the present application in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the above specific details are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to the above specific details.

[0166] Each embodiment in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between each embodiment can be understood by mutual reference. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be understood by referring to the part of the method embodiment.

[0167] The block diagrams of the devices, systems, apparatuses, systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, configurations be as shown in the block diagrams. As will be recognized by one of ordinary skill in the art, the devices, systems, apparatuses, systems can be connected, arranged, configured in any manner. Words such as "including," "containing," "comprising," and the like are to be construed in an inclusive fashion, indicating open-ended duration, and are intended to be equivalent to "including, but not limited to." As used herein, the terms "or" and "and" shall each be construed as the term "and / or" unless expressly indicated otherwise. As used herein, the term "such as" shall be construed as the phrase "such as but not limited to."

[0168] The methods and systems of the present application can be implemented in a number of ways. For example, the methods and systems of the present application can be implemented via software, hardware, firmware, or any combination of software, hardware, and firmware. The above described order of steps for the methods is merely illustrative, and the steps of the methods of the present application are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present application can also be implemented as a program recorded on a recording medium, which includes machine readable instructions for implementing the methods according to the present application. Thus, the present application also covers a recording medium storing a program for executing the methods according to the present application.

[0169] It is also important to note that the systems, apparatuses, and methods of the present application can be embodied in a computer-readable medium which can be a built-in medium or an external medium which stores program codes including machine readable instructions for implementing the methods according to the present application. Accordingly, the present application also covers a recording medium storing a program for executing the methods according to the present application.

[0170] The above description has been presented for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will readily appreciate that some modifications, changes, additions, and omissions can be made without departing from the scope of the application.

Claims

1. A method for high latency mitigation of a neural network model on a hardware device, the method comprising: The method comprises the following steps: initializing basic hyperparameters of a deep neural network model and a related environment, constructing a pruning computation graph of the deep neural network model, and determining network layers in the deep neural network model that are suitable for pruning through the pruning computation graph; resetting an exploration environment of the neural network model, setting a predetermined number of explorations, and obtaining an observation value corresponding to a state in the exploration environment each time through an agent, and determining a pruning number of each network layer that is suitable for pruning each time, wherein the observation value is composed of a group of vectors, which include type information, computation amount, parameter amount and channel number of the network layer, and also include a number of calculations on hardware, a number of DMA read data bytes and a number of DMA write data bytes; pruning the deep neural network model according to the pruning number of each network layer each time, and determining a deep neural network model after pruning for a predetermined number of explorations; evaluating the accuracy of the deep neural network model for a predetermined number of explorations using a real data set collected, and determining an optimal deep neural network model and pruning parameters of each layer thereof; evaluating the accuracy of the deep neural network model for a predetermined number of explorations using a real data set collected, and determining an optimal deep neural network model and pruning parameters of each layer thereof, comprising: obtaining relevant accuracy indicators according to the task type of the deep neural network model using the real data set; obtaining inference time of the deep neural network model on a specific inference device; fusing the accuracy indicators and the inference time, calculating the reward of the agent corresponding to the deep neural network model for a predetermined number of explorations, and determining the optimal deep neural network model and the pruning parameters of each layer thereof; The formula for fusing the accuracy indicators and the inference time is as follows: wherein ind represents the accuracy indicator of the pruned model on the real dataset; time pruning represents the actual inference time of the pruned model on the specific hardware; time dense represents the inference time of the original neural network model on the specific hardware; and a is an auxiliary coefficient, when a = 1, the accuracy indicator of the pruned model is taken as the reward of the agent; when a = 0, the inference time of the pruned model is taken as the reward of the agent; when a e (0, 1), the accuracy indicator and the inference time of the pruned model are taken as the reward of the agent.

2. The method of claim 1, wherein, Further comprising: updating related parameters of the agent including a neural network of the agent itself and an experience replay pool according to the performance indicators of the optimal deep neural network model and the pruning parameters of each layer thereof after each exploration.

3. The method of claim 1, wherein, constructing a pruning computation graph of the deep neural network model, and determining network layers in the deep neural network model that are suitable for pruning through the pruning computation graph, comprising: analyzing the prunable convolutional layers and fully connected layers in the process of constructing the pruning computation graph, and determining the previous layer and the next layer of the current network layer according to the dependency relationship between layers in the pruning computation graph; According to the pruning computation graph and the relationship between layer nodes, sibling nodes with a common parent node can be merged to determine network layers that are suitable for pruning.

4. A high latency mitigation apparatus of a neural network model on a hardware device, the apparatus comprising: The method comprises the following steps: a first determining module for initializing basic hyperparameters of a deep neural network model and a related environment, constructing a pruning computation graph of the deep neural network model, and determining network layers in the deep neural network model that are suitable for pruning through the pruning computation graph; The second determining module is configured to reset an exploration environment of the neural network model, set a predetermined number of explorations, and obtain an observation value corresponding to a state in the exploration environment each time through an agent, and determine a pruning number of each network layer suitable for pruning each time, wherein the observation value is composed of a group of vectors, which include type information, computation, parameter quantity, and channel number of a network layer, and also include a number of calculations on hardware, a number of reading data bytes from a DMA, and a number of writing data bytes to the DMA. The third determining module is configured to prune the deep neural network model according to the pruning number of each network layer each time, and determine a deep neural network model after pruning with the predetermined number of explorations. The fourth determining module is configured to evaluate the accuracy of the deep neural network model with the predetermined number of explorations by using a real data set collected, and determine an optimal deep neural network model and a pruning parameter of each layer thereof. The method for evaluating the accuracy of the deep neural network model with the predetermined number of explorations by using the real data set collected, and determining the optimal deep neural network model and the pruning parameter of each layer thereof, comprises the following steps. The accuracy index is obtained according to the task type of the deep neural network model by using the real data set. The inference time of the deep neural network model on a specific inference device is obtained. The accuracy index and the inference time are fused to calculate the reward of the agent corresponding to the deep neural network model with the predetermined number of explorations, and the optimal deep neural network model and the pruning parameter of each layer thereof are determined. The formula for fusing the accuracy index and the inference time is as follows: wherein ind represents the accuracy indicator of the pruned model on the real dataset; time pruning represents the actual inference time of the pruned model on the specific hardware; time dense represents the inference time of the original neural network model on the specific hardware; and a is an auxiliary coefficient, when a = 1, the accuracy indicator of the pruned model is taken as the reward of the agent; when a = 0, the inference time of the pruned model is taken as the reward of the agent; when a e (0, 1), the accuracy indicator and the inference time of the pruned model are taken as the reward of the agent.

5. The apparatus of claim 4, wherein, The method further comprises the following steps: The updating module is configured to update the related parameters of the agent, including a neural network of the agent itself and an experience replay pool, according to the performance index of the optimal deep neural network model and the pruning parameter of each layer thereof after each exploration.

6. The apparatus of claim 4, wherein, The first determining module comprises the following steps: The first determining submodule is configured to determine a previous layer and a next layer of a current network layer according to the prunable convolutional layer and the fully connected layer analyzed in the process of constructing the pruning computation graph, and according to the dependency relationship between layers in the pruning computation graph. The second determining submodule is configured to determine a network layer suitable for pruning by merging sibling nodes with a common parent node according to the pruning computation graph and the relationship between layer nodes.

7. A computer readable storage medium characterized in that, The storage medium stores a computer program, which is configured to execute the method of any one of claims 1-3.

8. An electronic device, comprising: The electronic device comprises: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-3.

Images