Method, device, apparatus and medium for compressing neural network

Abstract

According to an embodiment of the present disclosure, a method, apparatus, device and storage medium for compressing a neural network are provided. The method includes: determining a plurality of auxiliary parameters by training the neural network using training data, wherein the plurality of auxiliary parameters correspond to a plurality of output channels included in the convolutional layer of the neural network; determining a plurality of cropping parameters corresponding to the plurality of output channels based on the plurality of auxiliary parameters and the number of iterations of the training, wherein the cropping parameters indicate whether the corresponding output channels will be cropped; and cropping at least one of the plurality of output channels based on the plurality of cropping parameters. Based on the above method, the present disclosure can effectively crop the output channels in the neural network, thereby compressing the neural network.

Description

Technical Field

[0001] The various implementations of the present disclosure relate to the field of artificial intelligence, and more specifically, to methods, devices, equipment, and media for compressing neural networks. Background Art

[0002] In recent years, with the development of artificial intelligence technology, neural networks have begun to be widely used in many technical fields such as image processing and speech recognition, and have played an important role.

[0003] In order to perform more complex tasks, neural networks contain more and more layers, and the scale of their network parameters and calculations are also getting larger. This requires a lot of computing resources when training and using neural networks, and such complex neural networks are difficult to deploy on devices with limited computing resources and memory (such as mobile devices and embedded systems).

[0004] Therefore, how to compress the size of the neural network model and reduce the amount of calculation of the neural network model while ensuring its accuracy has become a hot topic. Summary of the invention

[0005] An embodiment of the present disclosure provides a scheme for compressing a neural network.

[0006] In a first aspect of the present disclosure, a method for compressing a neural network is provided. The method includes: determining a plurality of auxiliary parameters by training the neural network using training data, the plurality of auxiliary parameters corresponding to a plurality of output channels included in a convolutional layer of the neural network; determining a plurality of cropping parameters corresponding to the plurality of output channels based on the plurality of auxiliary parameters and the number of iterations of training, the cropping parameters indicating whether the corresponding output channels are to be cropped; and cropping at least one output channel of the plurality of output channels based on the plurality of cropping parameters.

[0007] In a second aspect of the present disclosure, a device for compressing a neural network is provided. The device includes: an auxiliary parameter determination module configured to determine a plurality of auxiliary parameters by training the neural network using training data, the plurality of auxiliary parameters corresponding to a plurality of output channels included in a convolutional layer of the neural network; a cropping parameter determination module configured to determine a plurality of cropping parameters corresponding to a plurality of output channels based on the plurality of auxiliary parameters and the number of iterations of training, the cropping parameters indicating whether the corresponding output channels are to be cropped; and a cropping module configured to crop at least one of the plurality of output channels based on the plurality of cropping parameters.

[0008] In a third aspect of the present disclosure, an electronic device is provided, comprising: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to the first aspect of the present disclosure.

[0009] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect of the present disclosure.

[0010] According to various embodiments of the present disclosure, the output channels in the convolutional layer of the neural network can be effectively pruned, thereby compressing the size of the neural network and reducing the requirements of the neural network for the deployed computing device, thereby improving the scalability of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure can be implemented;

[0013] Figure 2 A flowchart illustrating a process of compressing a network according to various embodiments of the present disclosure;

[0014] Figure 3 A flowchart illustrating an example process of determining auxiliary parameters according to various embodiments of the present disclosure;

[0015] FIG. 4A to FIG. 4C A schematic diagram showing the variation of the cutting parameters with the annealing temperature;

[0016] Figure 5 A schematic structural block diagram showing an apparatus for compressing a neural network according to some embodiments of the present disclosure; and

[0017] Figure 6 A block diagram of a computing device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] As used herein, a "neural network" is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. The layers in a neural network are connected in sequence so that the output of the previous layer is provided as the input of the next layer, wherein the input layer receives the input of the neural network model, and the output of the output layer serves as the final output of the neural network model. Each layer of the neural network model includes one or more nodes (also referred to as processing nodes or neurons), each of which processes the input from the previous layer. In this article, the terms "neural network", "model", "network" and "neural network model" are used interchangeably.

[0021] As discussed above, in recent years, neural networks (e.g., convolutional neural networks) have become deeper and wider, which results in the consumption of large computing resources for both training and prediction of these neural networks. This makes it difficult to deploy these neural networks to devices with limited computing resources, such as robots, autonomous vehicles, and mobile terminals.

[0022] Some existing solutions propose to reduce the size of the neural network and reduce the amount of computation by pruning the output channels of the convolutional layer in the neural network. Such solutions are also called "channel pruning". For example, some traditional solutions filter out unimportant channels by formulating screening rules and pruning these channels. However, such pruning is often coarse-grained and the pruning effect is poor.

[0023] According to the implementation of the present disclosure, a scheme for compressing a neural network is proposed. In the scheme, first, a plurality of auxiliary parameters are determined by training the neural network using training data, wherein the plurality of auxiliary parameters correspond to a plurality of output channels included in a convolutional layer of the neural network. Subsequently, a plurality of cropping parameters corresponding to the plurality of output channels are determined based on the plurality of auxiliary parameters and the number of iterations of the training, wherein the cropping parameters indicate whether the corresponding output channels are to be cropped. Furthermore, at least one of the plurality of output channels is cropped based on the plurality of cropping parameters.

[0024] By collaboratively training the weight parameters of the neural network and the auxiliary parameters corresponding to the output channels, a clipping parameter indicating whether the output channel will be clipped can be determined, and the clipping parameter can be converged to two different values ​​(e.g., 0 and 1) as the number of iterations increases. In this way, one or more output channels in the neural network can be clipped in a fine-grained manner while ensuring the accuracy of the neural network, thereby reducing the size of the neural network and the demand for computing resources.

[0025] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.

[0026] See first Figure 1 , which schematically shows a schematic diagram of an environment 100 in which an exemplary implementation according to the present disclosure can be implemented. Figure 1 As shown, environment 100 includes computing device 135. In some implementations, computing device 135 may be a computing device with sufficient computing resources.

[0027] The computing device 135 may receive the neural network 120 to be compressed. In some implementations, the neural network 120 is a convolutional neural network (CNN) and includes one or more convolutional layers, such as Figure 1 The convolutional layers 125-1 to 125-N (individually or collectively referred to as convolutional layers 125) shown in FIG. 1 include a plurality of output channels 130-1 to 130-M (individually or collectively referred to as output channels 130). These output channels will be provided to the next convolutional layer 125 as input.

[0028] like Figure 1 As shown, the computing device 135 receives the training data 110 and uses the training data 110 to train the neural network 120 to be compressed. In some implementations, examples of the training data 110 may include, but are not limited to, image training data, text training data, and voice training data, etc. Accordingly, the neural network 120 may be a neural network for image processing, a neural network for text processing, or a neural network for voice processing.

[0029] During the training process, the computing device 135 can set corresponding auxiliary parameters for the output channels, and obtain weight parameters and auxiliary parameters of the neural network 120 through collaborative training. Subsequently, the computing device 135 can determine the clipping parameters corresponding to the output channels 130 according to the auxiliary parameters and the number of training iterations, and clip one or more output channels 130 in the neural network 120 based on the clipping parameters.

[0030] exist Figure 1 In the example of FIG. 1 , the output channels 130-2 and 130-3 in the neural network 120 to be compressed are pruned (such as the output channels 150-2 and 150-3 indicated by the dotted lines) to obtain the compressed neural network 140. In the compressed neural network 140, it still includes a plurality of convolutional layers 145-1 to 145-N. The difference is that the output channels 150-2 and 150-3 in the convolutional layer 145-2 have been pruned, thereby reducing the amount of computation of the neural network.

[0031] It should be understood that Figure 1 The structure of the neural network shown and the number of convolutional layers and output channels therein are all illustrative and not restrictive. In different applications, the neural network can be designed to have other appropriate architectures and / or appropriate numbers of convolutional layers and output channels as needed.

[0032] The following will refer to Figure 2 The specific process of compressing the neural network is described in more detail with reference to FIG4 . Figure 2 FIG. 2 is a flowchart showing a process 200 for compressing a neural network according to some embodiments of the present disclosure. The process 200 may be performed by Figure 1 For the convenience of discussion, the following will be combined with Figure 1 The process 200 is described below.

[0033] like Figure 2 As shown, at box 202 , the computing device 135 determines a plurality of auxiliary parameters by training a neural network using the training data 110 , wherein the plurality of auxiliary parameters correspond to a plurality of output channels 130 included in the convolutional layer 125 of the neural network 120 .

[0034] The following will be combined Figure 3 To describe the specific process of block 202, Figure 3 A flowchart of an example process for determining auxiliary parameters according to some embodiments of the present disclosure is shown.

[0035] like Figure 3 As shown, at block 302 , computing device 135 may determine at least one objective function for neural network 120 based on training data 110 .

[0036] In some implementations, for a neural network with multiple convolutional layers, the calculation process of the lth convolutional layer can be expressed as formula (1):

[0037] O l =F l (W l , O l-1 ), (1) where: Represents the convolution kernel of the lth layer, whose input channel is c l-1 , and the output channel is c l ; F l (·) represents the convolution operation, O l Represents the output feature tensor of the lth layer.

[0038] Some traditional solutions directly configure a corresponding flag (e.g., 0 and 1) for each output channel to indicate whether it will be pruned. Accordingly, the pruned output feature tensor can be expressed as formula (2):

[0039]

[0040] in, Represents the vector composed of flag bits, represents the tensor product, O′ l Indicates the use of I l The output feature tensor after cropping.

[0041] Because I l The value of the flag is discontinuous (0 and 1), so it cannot be trained together with the weight parameters of the neural network. In order to overcome the above problems, in the implementation of the present disclosure, an auxiliary parameter corresponding to the output channel 130 is set And use it to determine the corresponding cutting parameters Specifically, the computing device 135 may use the cropping parameters It is expressed as formula (3):

[0042]

[0043] Among them, that is is converted to The sigmoid function of , where T represents the temperature variable, which changes with the number of training iterations. For example, T can be expressed as formula (4):

[0044] T=T0 / σ(n) (4)

[0045] Where T0 represents the initial temperature variable (for example, it can be set to 1 or other values), n represents the number of training iterations, and σ(·) represents the temperature annealing function, which increases with the increase of the number of iterations.

[0046] As the number of iterations increases, T will gradually decrease and approach 0. At this time, will approach 0 or 1. Based on this approach, the implementation of the present disclosure can be used to set the discontinuous clipping parameters The search is converted to continuous auxiliary parameters 's search.

[0047] In some implementations, to jointly train the weight parameters and auxiliary parameters of the neural network 120 , the computing device 135 may determine one or more objective functions.

[0048] In some implementations, the objective function may indicate the prediction accuracy of the neural network for the training samples. Specifically, the computing device 135 may weight the output of the convolution layer based on a plurality of training auxiliary parameters corresponding to the plurality of output channels 130. For example, the weight corresponding to the output channel 130 may be determined according to formula (3): Subsequently, the computing device 135 may determine the loss function of the neural network 120 based on the weighted output and use it as the first objective function. For example, the first objective function may be expressed as formula (5):

[0049]

[0050] in, represents the weight parameter of the neural network 120, α represents the auxiliary parameter of the neural network 120, represents the loss function (ie, the first objective function) corresponding to the validation data set in the training data 110. By setting the first objective function, the computing device 135 can ensure the accuracy of the pruned neural network.

[0051] In some implementations, the computing device 135 may also consider sparsity constraints. Specifically, the computing device 135 may determine the degree of model compression based on a plurality of training auxiliary parameters corresponding to the plurality of output channels 130. Exemplarily, the degree of model compression may indicate the floating point calculation number of the neural network 120, which may be represented, for example, by formula (6):

[0052]

[0053] Among them, k l represents the convolution kernel size of the lth layer, h l and w l Indicates the spatial size of the output feature map.

[0054] Subsequently, the computing device 135 may also determine a second objective function based on the model compression degree and the target compression degree. Exemplarily, the second objective function may be expressed as formula (7):

[0055]

[0056] Where F represents the target floating-point calculation number, that is, the target compression degree, and ∈ represents a constant less than 1.

[0057] By introducing the second objective function, the computing device 135 can prevent the neural network 120 from being overly compressed, thereby affecting the robustness of the neural network 120 .

[0058] In some implementations, the computing device 135 may also consider symmetry constraints on the residual block. In some implementations, for a neural network 120 including a residual block, the computing device may determine a first number of input channels of the residual block based on a plurality of training auxiliary parameters corresponding to a plurality of output channels. For example, the first number may be expressed as:

[0059]

[0060] Subsequently, the computing device 135 may also determine the second number of output channels of the residual block based on the plurality of training auxiliary parameters. For example, the second number may be expressed as:

[0061]

[0062] Further, the computing device 135 may determine a third objective function based on the difference between the first number and the second number. For example, the third objective function may be expressed as formula (10):

[0063]

[0064] Where (l, l') represents the l input channels and has c l′ The residual block has 100 output channels.

[0065] By introducing the third objective function, for a neural network with a residual block, the computing device 135 can ensure the symmetry of the residual block (i.e., having the same number of input channels and output channels), thereby avoiding increasing the difficulty of gradient propagation across multiple layers.

[0066] At block 304 , computing device 135 may determine an overall objective function for training neural network 120 by combining at least a portion of the at least one objective function.

[0067] In some implementations, by setting the auxiliary parameters corresponding to the output channel 130 The overall objective function of the neural network 120 can be set to, for example, formula (11):

[0068]

[0069] in, represents the weight parameter of the neural network 120, α represents the auxiliary parameter of the neural network 120, represents the loss function (i.e., the first objective function) corresponding to the validation data set in the training data 110, λ represents the weight coefficient, represents a sparsity constraint, which can be any one of the second objective function and the third objective function, or a combination thereof.

[0070] At block 306 , the computing device 135 may determine a plurality of auxiliary parameters of the neural network by locally minimizing the overall objective function.

[0071] In some implementations, computing device 135 may jointly learn the weight parameters of neural network 120 and auxiliary parameter α, the joint optimization can be regarded as a two-layer optimization problem, where the auxiliary parameter α is the upper layer variable and the weight parameter is the lower layer variable. Weight parameter For example, it can be determined by formula (12):

[0072]

[0073] in represents the loss function on the training data set in the training data 110. The computing device 135 can collaboratively learn the weight parameters and auxiliary parameters α to minimize the total objective function (11) locally. The specific process of the auxiliary parameter α can be found in “Darts: Differentiable architecture search” published by Hanxiao Liu, Karen Simonyan, and Yiming Yang in 2019. This disclosure will not be described in detail here.

[0074] Continue to refer Figure 2 In block 204 , the computing device 135 determines a plurality of cropping parameters corresponding to the plurality of output channels based on the plurality of auxiliary parameters and the number of iterations of the training, the cropping parameters indicating whether the corresponding output channels are to be cropped.

[0075] Specifically, according to the discussion above in combination with formulas (3) and (4), the computing device 135 can first determine a first intermediate parameter (ie, the temperature variable T discussed above) based on the number of iterations, where the first intermediate parameter indicates the heat of the training.

[0076] Subsequently, the computing device 135 may determine the second intermediate parameter (i.e., ), wherein the intermediate parameter is proportional to the auxiliary parameter and inversely proportional to the first intermediate parameter.

[0077] Further, the computing device 135 may determine the cropping parameters corresponding to the auxiliary parameters based on the S-type function of the intermediate parameters. After that, the corresponding clipping parameters can be determined according to formula (3)

[0078] FIG. 4A to FIG. 4C A schematic diagram showing how the trimming parameters vary with the annealing temperature in some implementations of the present disclosure is shown. Figure 4A The figure shows how the clipping parameter changes with the auxiliary parameter when the temperature variable T=1 (eg, the first iteration). It can be seen that the clipping parameter is a value between 0 and 1, and the distribution is relatively dispersed.

[0079] Figure 4B The figure shows how the trimming parameters change with the auxiliary parameters when the temperature variable T=0.02. It can be seen that compared with when T=1, the distribution of the trimming parameters has become relatively concentrated and gradually approaches 0 or 1.

[0080] Figure 4C The figure shows how the clipping parameters change with the auxiliary parameters when the temperature variable T=0.002. It can be seen that the clipping parameters have been basically binarized to 0 or 1 compared to when T=0.02.

[0081] pass FIG. 4A to FIG. 4C As can be seen from the schematic diagram, by introducing auxiliary parameters and temperature variables, the embodiments of the present disclosure can overcome the problem that the space of clipping parameter values ​​is discrete and difficult to search.

[0082] At block 206, computing device 135 crops at least one of the plurality of output channels based on the plurality of cropping parameters. Specifically, if a cropping parameter of the plurality of cropping parameters is less than a predetermined threshold, computing device 135 may crop the output channel corresponding to the cropping parameter.

[0083] For example, Figure 4C As an example, where the clipping parameters and is close to 0 and is determined to be less than a predetermined threshold. Figure 1 As shown, computing device 135 may prune neural network 120 with pruning parameters and The corresponding output channels are 130-2 and 130-3.

[0084] Through the method described above, the embodiments of the present disclosure can automatically prune the output channels contained in the convolutional layer of the neural network in a fine-grained manner while ensuring the accuracy of the neural network, thereby obtaining a neural network with lower computational complexity.

[0085] In some implementations, computing device 135 may also deploy pruned neural network 140 to a target computing device, where the number of computing resources of the target computing device is less than a threshold number. Examples of target computing devices include, but are not limited to, devices with relatively limited computing resources, such as robots, autonomous vehicles, and mobile terminals.

[0086] In this way, the embodiments of the present disclosure can reduce the demand for computing resources of the neural network, improve the universality of the neural network, and thus can be applied to more devices.

[0087] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an apparatus 500 for compressing a neural network according to some embodiments of the present disclosure is shown.

[0088] like Figure 5 As shown, the device 500 may include an auxiliary parameter determination module 510, which is configured to determine multiple auxiliary parameters by training a neural network using training data, and the multiple auxiliary parameters correspond to multiple output channels included in the convolution layer of the neural network. In addition, the device 500 also includes a cropping parameter determination module 520, which is configured to determine multiple cropping parameters corresponding to multiple output channels based on multiple auxiliary parameters and the number of iterations of training, and the cropping parameters indicate whether the corresponding output channels will be cropped. The device 500 also includes a cropping module 530, which is configured to crop at least one output channel of the multiple output channels based on the multiple cropping parameters.

[0089] In some implementations, the auxiliary parameter determination module 510 includes: an objective function determination module, configured to determine at least one objective function of the neural network based on training data; an objective function combination module, configured to determine an overall objective function for training the neural network by combining at least a portion of the at least one objective function; and an objective function solving module, configured to determine multiple auxiliary parameters of the neural network by locally minimizing the overall objective function.

[0090] In some implementations, the objective function determination module includes: a weighting module, configured to weight the output of the convolution layer based on multiple training auxiliary parameters corresponding to multiple output channels; and a first objective function determination module, configured to determine the loss function of the neural network as the first objective function based on the weighted output.

[0091] In some implementations, the objective function determination module includes: a model compression degree determination module, configured to determine the model compression degree based on multiple training auxiliary parameters corresponding to multiple output channels; and a second objective function determination module, configured to determine the second objective function based on the difference between the model compression degree and the target compression degree.

[0092] In some implementations, the degree of model compression indicates the number of floating point calculations of the neural network.

[0093] In some implementations, the neural network includes a residual block, and the objective function determination module includes: a first number determination module, configured to determine a first number of input channels of the residual block based on a plurality of training auxiliary parameters corresponding to a plurality of output channels; a second number determination module, configured to determine a second number of output channels of the residual block based on the plurality of training auxiliary parameters; and a third objective function determination module, configured to determine a third objective function based on a difference between the first number and the second number.

[0094] In some implementations, the trimming parameter determination module 520 includes: a first intermediate parameter determination module, configured to determine a first intermediate parameter based on the number of iterations, the first intermediate parameter indicating the heat of training; a second intermediate parameter determination module, configured to determine a second intermediate parameter based on an auxiliary parameter and the first intermediate parameter, the intermediate parameter being proportional to the auxiliary parameter and inversely proportional to the first intermediate parameter; and an S-type function calculation module, configured to determine the trimming parameter corresponding to the auxiliary parameter based on the S-type function of the intermediate parameter.

[0095] In some implementations, the cropping module 530 includes: a channel cropping module configured to crop an output channel corresponding to a cropping parameter among the plurality of cropping parameters if the cropping parameter is less than a predetermined threshold.

[0096] In some implementations, the apparatus 500 further includes: a deployment module configured to deploy the pruned neural network on a target computing device, wherein the number of computing resources of the target computing device is less than a threshold number.

[0097] In some implementations, the training data is image training data, and the neural network is a neural network for image processing.

[0098] The units included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0099] Figure 6 1 shows a block diagram of a computing device / server 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The computing device / server 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.

[0100] like Figure 6 As shown, computing device / server 600 is in the form of a general-purpose computing device. Components of computing device / server 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device / server 600.

[0101] The computing device / server 600 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device / server 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device / server 600.

[0102] The computing device / server 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0103] The communication unit 640 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device / server 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device / server 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0104] Input device 650 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 may be one or more output devices, such as a display, speaker, printer, etc. Computing device / server 600 may also communicate with one or more external devices (not shown) as needed, such as storage devices, display devices, etc., with one or more devices that enable a user to interact with computing device / server 600, or with any device that enables computing device / server 600 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.) through communication unit 640. Such communication may be performed via an input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the one or more computer instructions are executed by a processor to implement the method described above.

[0106] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0110] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the implementations disclosed herein.

Claims

1. A method for compressing a neural network, comprising: Determining a plurality of auxiliary parameters by training the neural network using training data, wherein the plurality of auxiliary parameters correspond to a plurality of output channels included in a convolutional layer of the neural network; The training data is image training data, and the neural network is a neural network for image processing; Based on the plurality of auxiliary parameters and the number of iterations of the training, determining a plurality of cropping parameters corresponding to the plurality of output channels, the cropping parameters indicating whether the corresponding output channels are to be cropped; as well as Based on the plurality of cropping parameters, cropping at least one output channel of the plurality of output channels; Determining the plurality of auxiliary parameters comprises: Determining at least one objective function of the neural network based on the training data; Determining an overall objective function for training the neural network by combining at least a portion of the at least one objective function; and Determining the plurality of auxiliary parameters of the neural network by locally minimizing the overall objective function; The determining of a plurality of cropping parameters corresponding to the plurality of output channels comprises: Based on the number of iterations, determining a first intermediate parameter, the first intermediate parameter indicating the heat of the training; Based on the auxiliary parameter and the first intermediate parameter, determining a second intermediate parameter, the second intermediate parameter being proportional to the auxiliary parameter and inversely proportional to the first intermediate parameter; and Based on the S-type function of the second intermediate parameter, a clipping parameter corresponding to the auxiliary parameter is determined.

2. The method of claim 1, wherein determining at least one objective function of the neural network comprises: weighting the output of the convolutional layer based on a plurality of training auxiliary parameters corresponding to the plurality of output channels; as well as Based on the weighted output, a loss function of the neural network is determined as a first objective function.

3. The method of claim 1 , wherein determining at least one objective function of the neural network comprises: Determining a degree of model compression based on a plurality of training auxiliary parameters corresponding to the plurality of output channels; as well as Based on the model compression degree and the target compression degree, a second objective function is determined.

4. The method of claim 3, wherein the degree of model compression indicates the number of floating point calculations of the neural network.

5. The method of claim 1 , wherein the neural network comprises a residual block, and determining at least one objective function of the neural network comprises: Determining a first number of input channels of the residual block based on a plurality of training auxiliary parameters corresponding to the plurality of output channels; Determining a second number of output channels of the residual block based on the multiple training auxiliary parameters; as well as Based on the first number and the second number, a third objective function is determined.

6. The method of claim 1 , wherein clipping at least one of the plurality of output channels comprises: If a cropping parameter among the plurality of cropping parameters is less than a predetermined threshold, an output channel corresponding to the cropping parameter is cropped.

7. The method according to claim 1, further comprising: The pruned neural network is deployed on a target computing device, wherein the number of computing resources of the target computing device is less than a threshold number.

8. A device for compressing a neural network, comprising: an auxiliary parameter determination module configured to determine a plurality of auxiliary parameters by training the neural network using training data, wherein the plurality of auxiliary parameters correspond to a plurality of output channels included in the convolutional layer of the neural network; the training data is image training data, and the neural network is a neural network for image processing; A cropping parameter determination module is configured to determine a plurality of cropping parameters corresponding to the plurality of output channels based on the plurality of auxiliary parameters and the number of iterations of the training, the cropping parameters indicating whether the corresponding output channels are to be cropped; as well as a cropping module configured to crop at least one output channel of the plurality of output channels based on the plurality of cropping parameters; The auxiliary parameter determination module comprises: an objective function determination module, configured to determine at least one objective function of the neural network based on the training data; an objective function combining module configured to determine an overall objective function for training the neural network by combining at least a portion of the at least one objective function; and An objective function solving module, configured to determine the plurality of auxiliary parameters of the neural network by locally minimizing the total objective function; The cropping parameter determination module comprises: A first intermediate parameter determination module is configured to determine a first intermediate parameter based on the number of iterations, wherein the first intermediate parameter indicates the heat of the training; A second intermediate parameter determination module is configured to determine a second intermediate parameter based on an auxiliary parameter and the first intermediate parameter, wherein the second intermediate parameter is proportional to the auxiliary parameter and inversely proportional to the first intermediate parameter; and The S-type function calculation module is configured to determine the clipping parameters corresponding to the auxiliary parameters based on the S-type function of the second intermediate parameter.

9. The apparatus according to claim 8, wherein the objective function determination module comprises: A weighting module, configured to weight the output of the convolutional layer based on a plurality of training auxiliary parameters corresponding to the plurality of output channels; as well as The first objective function determination module is configured to determine the loss function of the neural network as the first objective function based on the weighted output.

10. The device according to claim 8, wherein the target function determination module comprises: a model compression degree determination module, configured to determine the model compression degree based on a plurality of training auxiliary parameters corresponding to the plurality of output channels; as well as The second objective function determination module is configured to determine the second objective function based on the model compression degree and the target compression degree.

11. The apparatus of claim 10, wherein the degree of model compression indicates a floating point calculation number of the neural network.

12. The apparatus of claim 8, wherein the neural network comprises a residual block, and the objective function determination module comprises: A first number determination module is configured to determine a first number of input channels of the residual block based on a plurality of training auxiliary parameters corresponding to the plurality of output channels; A second number determination module is configured to determine a second number of output channels of the residual block based on the multiple training auxiliary parameters; as well as The third objective function determination module is configured to determine a third objective function based on the first number and the second number.

13. The apparatus according to claim 8, wherein the cropping module comprises: The channel clipping module is configured to clip the output channel corresponding to the clipping parameter if the clipping parameter among the plurality of clipping parameters is less than a predetermined threshold.

14. The apparatus according to claim 8, further comprising: The deployment module is configured to deploy the pruned neural network on a target computing device, wherein the number of computing resources of the target computing device is less than a threshold number.

15. An electronic device, comprising: Memory and processor; The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

16. A computer-readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of claims 1 to 7.

Images