Pruning processing method and device and electronic equipment

Abstract

The invention discloses a pruning processing method and device and electronic equipment. The method comprises the steps that input channels and output channels of all convolutional layers and / or full connection layers of a to-be-pruned target model are closed; on the basis of the connection relation of each convolutional layer and / or full connection layer, a network fragment set is obtained, and each network fragment in the network fragment set comprises an input channel and an output channel, an independent input channel or an independent output channel which have a causal relation in the target model; and based on the importance of each network fragment in the network fragment set, opening channels in a predetermined number of network fragments in the target model so as to carry out pruning processing.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology, and more specifically, to a pruning processing method and device, and an electronic device. Background Art

[0002] Deep learning neural networks are the foundation of artificial intelligence, such as object detection, speech recognition, and natural speech processing. Modern neural networks have a huge number of parameters that need to be trained. Studies have shown that these parameters are highly redundant and some parameters can be pruned without affecting the accuracy, so that neural networks can be trained and deployed on mobile devices with limited resources, which has attracted a large number of researchers to conduct research in this area. However, unstructured pruning, due to poor data locality and actual computing problems, makes it difficult to optimize execution speed and memory usage. Therefore, in order to execute artificial intelligence on resource-limited platforms, the research of structured pruning algorithms has been stimulated, such as the channel pruning algorithm, which prunes the input and output channel dimensions to generate a smaller model, thereby achieving an increase in running speed and a reduction in running memory.

[0003] However, current pruning algorithms, whether unstructured or structured, are fully coupled to the network structure of the neural network. If they need to be applied to neural networks with other network structures, even if they are all convolutional neural networks (CNN), the pruning algorithms need to be readjusted to run, and cannot be quickly and conveniently applied to neural networks with other network structures. Summary of the invention

[0004] The present disclosure provides a pruning processing method and device, and an electronic device to at least solve the problem of poor versatility of the pruning algorithm in the above-mentioned related technology.

[0005] According to a first aspect of an exemplary embodiment of the present disclosure, a pruning method is provided, comprising: closing input channels and output channels of all convolutional layers and fully connected layers of a target model to be pruned, wherein the target model is used for image processing, text processing or speech processing; based on the connection relationship of each convolutional layer and / or fully connected layer, obtaining a set of network segments, wherein each network segment in the set of network segments includes input channels and output channels with a causal relationship, a separate input channel or a separate output channel in the target model; based on the importance of each network segment in the set of network segments, opening channels in a predetermined number of network segments in the target model for pruning.

[0006] Optionally, before closing the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned, it also includes: performing the following first processing on each convolutional layer and fully connected layer: inputting initial data into the target model to obtain a second input of any next layer of the current convolutional layer or the current fully connected layer, wherein the output of the current convolutional layer or the current fully connected layer in the target model is pre-set to a random number, and any next layer is a convolutional layer or a fully connected layer; in response to the second input being inconsistent with the first input, determining that the output channel of the current convolutional layer or the current fully connected layer has a causal relationship with the input channel of any next layer, and the first input is the input of any next layer when the output of the current convolutional layer or the current fully connected layer in the target model is not set to a random number.

[0007] Optionally, based on the connection relationship of each convolutional layer and / or fully connected layer, a set of network segments is obtained, including: performing the following second processing on each convolutional layer and fully connected layer to obtain the set of network segments: in response to the output channel of the current convolutional layer or the current fully connected layer not being opened, using the output channel of the current convolutional layer or the current fully connected layer as the output channel in the current network segment; performing the following third processing on all next layers corresponding to the current convolutional layer or the current fully connected layer in sequence: in response to the input channel of the current next layer not being opened, using the input channel of the current next layer as the input channel in the current network segment, and adding the current network segment to the set of network segments.

[0008] Optionally, in response to the input channel of the current next layer being opened, the current next layer is skipped; the fourth processing is performed on each predetermined next layer: in response to the input channel of the predetermined next layer not being opened, the input channel of the predetermined next layer is taken as a network segment separately, and added to the network segment set; wherein the predetermined next layer is the next layer waiting for the third processing after the current next layer among all the next layers corresponding to the current convolutional layer or the current fully connected layer.

[0009] Optionally, in response to all input channels of next layers being opened, all next layers are skipped and the current network segment is added to the network segment set, wherein the current network segment only contains output channels of the current convolutional layer or the current fully connected layer.

[0010] Optionally, after opening channels in a predetermined number of network segments in the target model based on the importance of each network segment in the network segment set, the method further includes: in response to the proportion of opened network segments in the target model not reaching a predetermined proportion, continuing to obtain the next network segment set based on the relationship between each convolutional layer and the fully connected layer, and opening channels in a predetermined number of network segments in the target model based on the importance of each network segment in the next network segment set, until the proportion of opened network segments in the target model reaches a predetermined proportion.

[0011] Optionally, before performing the following first processing on each convolutional layer and fully connected layer, it also includes: inputting the initial data into the target model, obtaining the input of all convolutional layers and fully connected layers of the target model, and recording them as the first input of all convolutional layers and fully connected layers.

[0012] Optionally, before inputting the initial data into the target model to obtain the inputs of all convolutional layers and fully connected layers of the target model, the method further includes: setting the weights of all convolutional layers and fully connected layers of the target model to zero.

[0013] According to a second aspect of an exemplary embodiment of the present disclosure, a pruning processing device is provided, comprising: a channel closing unit, configured to close input channels and output channels of all convolutional layers and fully connected layers of a target model to be pruned, wherein the target model is used for image processing, text processing or speech processing; a set acquisition unit, configured to acquire a set of network fragments based on the connection relationship of each convolutional layer and / or fully connected layer, wherein each network fragment in the set of network fragments includes input channels and output channels with a causal relationship, a separate input channel or a separate output channel in the target model; and a channel opening unit, configured to open channels in a predetermined number of network fragments in the target model based on the importance of each network fragment in the set of network fragments for pruning processing.

[0014] Optionally, the channel closing unit is also configured to perform the following first processing on each convolutional layer and fully connected layer before closing the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned: input the initial data into the target model to obtain a second input of any next layer of the current convolutional layer or the current fully connected layer, wherein the output of the current convolutional layer or the current fully connected layer in the target model is pre-set to a random number, and any next layer is a convolutional layer or a fully connected layer; in response to the second input being inconsistent with the first input, determining that the output channel of the current convolutional layer or the current fully connected layer has a causal relationship with the input channel of any next layer, and the first input is the input of any next layer when the output of the current convolutional layer or the current fully connected layer in the target model is not set to a random number.

[0015] Optionally, the set acquisition unit is further configured to perform the following second processing on each convolutional layer and fully connected layer to obtain a set of network fragments: in response to the output channel of the current convolutional layer or the current fully connected layer not being opened, using the output channel of the current convolutional layer or the current fully connected layer as the output channel in the current network fragment; performing the following third processing on all next layers corresponding to the current convolutional layer or the current fully connected layer in sequence: in response to the input channel of the current next layer not being opened, using the input channel of the current next layer as the input channel in the current network fragment, and adding the current network fragment to the set of network fragments.

[0016] Optionally, the set acquisition unit is further configured to, in response to the input channel of the current next layer being opened, skip the current next layer; perform the fourth processing on each predetermined next layer: in response to the input channel of the predetermined next layer not being opened, treat the input channel of the predetermined next layer as a separate network segment, and add it to the network segment set; wherein the predetermined next layer is the next layer waiting for the third processing after the current next layer among all the next layers corresponding to the current convolutional layer or the current fully connected layer.

[0017] Optionally, the set acquisition unit is further configured to skip all next layers in response to all input channels of the next layers being opened, and add the current network segment to the network segment set, wherein the current network segment only contains the output channels of the current convolutional layer or the current fully connected layer.

[0018] Optionally, the channel opening unit is further configured to, after opening the channels in a predetermined number of network segments in the target model based on the importance of each network segment in the network segment set, continue to obtain the next network segment set based on the relationship between each convolutional layer and the fully connected layer in response to the proportion of the opened network segments in the target model not reaching a predetermined proportion, and open the channels in a predetermined number of network segments in the target model based on the importance of each network segment in the next network segment set, until the proportion of the opened network segments in the target model reaches a predetermined proportion.

[0019] Optionally, the channel closing unit is also configured to input the initial data into the target model before performing the following first processing on each convolutional layer and fully connected layer, obtain the input of all convolutional layers and fully connected layers of the target model, and record it as the first input of all convolutional layers and fully connected layers.

[0020] Optionally, the channel closing unit is further configured to set the weights of all convolutional layers and fully connected layers of the target model to zero before inputting the initial data into the target model to obtain the inputs of all convolutional layers and fully connected layers of the target model.

[0021] According to a third aspect of an exemplary embodiment of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the pruning processing method described in the present disclosure is implemented.

[0022] According to a fourth aspect of an exemplary embodiment of the present disclosure, there is provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the pruning processing method described in the present disclosure.

[0023] Utilizing the technical solution disclosed in the present invention, a pruning processing method, device, and electronic device are provided. First, the input channels and output channels corresponding to all convolutional layers and fully connected layers of the target model to be pruned are closed. Then, based on the connection relationship of each convolutional layer and / or fully connected layer, a network segment set is obtained. Each network segment in the network segment set includes input channels and output channels with causal relationships, separate input channels, or separate output channels in the target model. Then, based on the importance of the network segment, channels in a predetermined number of network segments in the target model are opened to achieve the purpose of pruning processing, so that the pruning processing method disclosed in the present invention is no longer coupled with the network structure of the target model. Therefore, it can be quickly and conveniently applied to neural networks with different network structures, that is, applied to different models, thereby improving the versatility of the pruning algorithm.

[0024] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects and features of the exemplary embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate the embodiments, in which:

[0026] Figure 1 It is a flowchart of a pruning algorithm in the related art;

[0027] Figure 2 is a flowchart of a pruning method according to an exemplary embodiment of the present disclosure;

[0028] Figure 3 is a flow chart of a pruning processing method according to an exemplary embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram of the influence of the relationship between conv input and output according to an exemplary embodiment of the present disclosure;

[0030] Figure 5 is a flowchart of determining a causal relationship according to an exemplary embodiment of the present disclosure;

[0031] Figure 6 is a basic structural diagram of a deep residual network according to an exemplary embodiment of the present disclosure;

[0032] Figure 7 is a schematic diagram of the channel relationship of a partial convolutional layer according to an exemplary embodiment of the present disclosure;

[0033] Figure 8 is a schematic diagram of an initialization process according to an exemplary embodiment of the present disclosure;

[0034] Fig. 9 is a schematic diagram of a network segment according to an exemplary embodiment of the present disclosure;

[0035] Fig.10 is a flowchart of a pruning method according to an exemplary embodiment of the present disclosure;

[0036] Fig.11 is a schematic diagram of a process of identifying candidate network segments according to an exemplary embodiment of the present disclosure;

[0037] Fig.12 is a schematic diagram of a model structure relationship according to an exemplary embodiment of the present disclosure;

[0038] Fig.13 is a schematic diagram of the channel relationship of the convolutional layer according to an exemplary embodiment of the present disclosure;

[0039] Fig.14 is a schematic diagram of the returned TopK network fragments according to an exemplary embodiment of the present disclosure;

[0040] Fig.15 is a block diagram of a pruning processing apparatus according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0042] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0043] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0045] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0046] like Figure 1 As shown, with the current pruning algorithm, if the user wants to import a neural network with a new network structure, the tasks in the dotted box must be completed before the pruning algorithm can run. That is, if you want to import a neural network with a new network structure, you must create the corresponding data structure. It can be seen that the current pruning algorithm is completely coupled with the network structure of the neural network, resulting in weak versatility.

[0047] The present invention generalizes the pruning algorithm by scanning the network structure of the neural network and performing corresponding processing. For neural networks with different network structures (i.e., different models), the pruning algorithm can be used without any modification, and can prune all types of CNN networks, while basically maintaining the accuracy of the original CNN network. Figure 2 As shown, for neural networks with different network structures, the pruning processing method disclosed in the present invention can be used without any modification, and any CNN network (i.e. Figure 2 medium model).

[0048] According to one aspect of an exemplary embodiment of the present disclosure, a pruning method is provided. Figure 3 is a flowchart of a pruning processing method according to an exemplary embodiment of the present disclosure. Figure 3 , the pruning processing method comprises the following steps:

[0049] In step S301, the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned are closed, wherein the target model is used for image processing, text processing or speech processing.

[0050] Compared with the current pruning algorithm that deletes some weight branches on the basis of the original model to achieve the purpose of optimizing the model, the pruning processing method disclosed in the present invention is an addition operation, that is, the input channels and output channels of all convolutional layers and fully connected layers in the model are closed, and then some network segments are continuously selected and opened until the pruning ratio target is reached. Therefore, when the present invention performs pruning processing, it is necessary to first close the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned.

[0051] As an example, the above-mentioned image processing can be image recognition, image segmentation, image generation, predicted image category, etc., which is not limited to the present disclosure; the above-mentioned text processing can be predicted text category (such as politics, economy, sports, entertainment, etc.), predicted text emotion (such as positive, negative or neutral emotional tendency), text generation, etc., which is not limited to the present disclosure; the above-mentioned speech processing can be speech recognition, speech synthesis, predicted speech emotion, speech enhancement, etc., which is not limited to the present disclosure.

[0052] In an embodiment of the present disclosure, before closing the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned, each convolutional layer and fully connected layer may be subjected to the following first processing: inputting the initial data into the target model to obtain a second input of any next layer of the current convolutional layer or the current fully connected layer, wherein the output of the current convolutional layer or the current fully connected layer in the target model is pre-set to a random number, and any next layer is a convolutional layer or a fully connected layer; in response to the second input being inconsistent with the first input, determining that the output channel of the current convolutional layer or the current fully connected layer has a causal relationship with the input channel of any next layer, and the first input is the input of any next layer when the output of the current convolutional layer or the current fully connected layer in the target model is not set to a random number. Through this embodiment, the output of the current convolutional layer or the current fully connected layer is pre-set to a random number, and then comparing whether the input of the next layer before and after the setting is consistent, so that it is convenient and fast to determine whether the output of the current convolutional layer or the current fully connected layer has a causal relationship with the next layer.

[0053] Specifically, the typical method in the structured pruning algorithm is to prune on the channel dimension. The input and output channels of each layer in the convolutional layer (conv) and fully connected layer (FC) of the general model are independent, but the input and output channels between different layers may be related. For example, some input channels may become inactive due to the pruning of the output channels of some previous layers (conv or FC), such as Figure 4 A schematic diagram of the relationship between different conv input and output is given, from Figure 4 It can be seen that W (l)The right column is pruned to 0, resulting in X (l) The bottom layer is 0, which affects W (l+1) Therefore, obtaining the relationship between the input and output of different layers (conv and FC) can easily trim all inactive channels, which is particularly important for CNN trimming.

[0054] As an example, Figure 5 is a flowchart of determining a causal relationship according to an exemplary embodiment of the present disclosure, such as Figure 5 As shown, this process can be used to identify the relationship between the convolutional layer (Conv) and / or the fully connected layer (FC) in the target model, so as to provide data support for the subsequent acquisition of network fragments. The process mainly includes the following steps: 1. Loop on the output channels of all convolutional layers and fully connected layers, that is, loop once for all output channels of each layer in all convolutional layers and fully connected layers; 2. Set the output of the output channel of the current convolutional layer or the current fully connected layer (denoted as U(L1, C1)) to a random number through the hook function, where the hook function of each layer is registered in advance; 3. Use any initial data for forward reasoning operation, that is, input the initial data into the target model, and obtain the second input of any next layer of the current convolutional layer or the current fully connected layer, where U(L1, C1) is a random number; 4. Detect all subsequent layers of the current convolutional layer or the current fully connected layer (that is, all subsequent layers) through the hook function 5. If the judgment result shows that a certain input channel (denoted as V(L2, C2)) is inconsistent with the corresponding input saved previously, it means that this input channel V(L2, C2) will be affected by the output channel U(L1, C1). Therefore, the input channel V(L2, C2) and the output channel U(L1, C1) have a production and consumption relationship, that is, a causal relationship, which can be recorded as U(L1, C1)->V(L2, C2). After the output channel cycle of all convolutional layers and fully connected layers is completed, the connection relationship between the input channels and output channels of all convolutional layers and fully connected layers of the target model can be obtained.

[0055] It should be noted that a simple model can also manually analyze the connection relationship between the input channels and output channels of all convolutional layers and fully connected layers. Figure 6 Taking the basic structure shown in the figure as an example, the connection relationship between the input channels and output channels of all convolutional layers and fully connected layers is briefly explained. Assume that Figure 6The basic structure shown is a basic structure of a deep residual network (resnet). For this basic structure, it can be found that some convolutional layers are connected, and the output of some convolutional layers will affect the input of other convolutional layers. Figure 7 As shown in the figure, the channels of some connected convolutional layers are marked, among which non-critical layers such as BN and RELU are integrated into the dotted lines.

[0056] In the embodiment of the present disclosure, before performing the following first processing on each convolutional layer and fully connected layer, the initial data can also be input into the target model to obtain the input of all convolutional layers and fully connected layers of the target model, and record it as the first input of all convolutional layers and fully connected layers. Through this embodiment, the input of all convolutional layers and fully connected layers of the target model is obtained in advance when the random number is not set, which facilitates calling the required input when determining the causal relationship between the output channel and the input channel.

[0057] As an example, retain all the original structures of the target model, that is, do not set the output of the convolutional layer or the fully connected layer in the target model to a random number. Figure 5 The initial data used in is input into the target model to obtain the input of all convolutional layers and fully connected layers of the target model, and is recorded as the first input of all convolutional layers and fully connected layers so that Figure 5 The required input is called when determining the causal relationship between output channels and input channels.

[0058] In an embodiment of the present disclosure, before inputting the initial data into the target model and obtaining the input of all convolutional layers and fully connected layers of the target model, the weights of all convolutional layers and fully connected layers of the target model may be set to zero. Through this embodiment, the weights of all convolutional layers and fully connected layers are set to zero during the pruning process, which can reduce the amount of calculation and improve the pruning speed.

[0059] As an example, the weights of all convolutional layers and fully connected layers can be set to 0 by setting the U / V array, which is not limited in the present disclosure. This can facilitate model reasoning, improve the model reasoning speed, and further improve the pruning speed.

[0060] As an example, Figure 8 is a schematic diagram of the initialization process according to an exemplary embodiment of the present disclosure, such as Figure 8As shown in the figure, this process can be used to obtain all the nodes to be pruned in the target model, that is, all the convolutional layers and fully connected layers in the target model, and record the current input data of each node to be pruned. The process mainly includes the following steps: 1. Scan all the convolutional layers (Conv) and fully connected layers (FC) of the target model, and register a hook function for each layer; 2. Set the weights of all convolutional layers and fully connected layers to 0 by setting the U / V array; 3. Use any initial data for inference calculation, that is, input the initial data into the target model, obtain the input of all convolutional layers and fully connected layers of the target model, and record it as the first input of all convolutional layers and fully connected layers; 4. In the hook function, record all the information of the convolutional layers and fully connected layers of interest, including input and output dimensions, number of groups (group count), model structure, etc., and initialize some data structures based on this information. These data structures include U / V arrays that record the enable status (i.e., open or close status) of the input channels and output channels, the number of Convs, the number of FCs, etc.; 5. Record the input data of each layer in the convolutional layer and the fully connected layer through the hook function, number these input data according to the execution order of the layers and save them in the corresponding data structure.

[0061] In step S302, based on the connection relationship of each convolutional layer and / or fully connected layer, a set of network segments is obtained, wherein each network segment in the set of network segments includes an input channel and an output channel with a causal relationship in the target model, a separate input channel, or a separate output channel.

[0062] As an example, the present disclosure obtains the connection relationship of each convolutional layer and / or fully connected layer, as shown above Figure 7 The relationship between the channels of the convolutional layer shown in Fig. 9 The network segment shown is used by the pruning algorithm. It should be noted that Fig. 9 Network fragments containing separate input channels or separate output channels are not shown, and the network fragments are looped in units of output channels when obtaining them, so each network fragment will only contain one output channel or no output channel, but can contain at least zero input channels.

[0063] In an embodiment of the present disclosure, based on the connection relationship of each convolutional layer and / or fully connected layer, obtaining a network segment set may include: performing the following second processing on each convolutional layer and fully connected layer to obtain a network segment set: in response to the output channel of the current convolutional layer or the current fully connected layer not being opened, using the output channel of the current convolutional layer or the current fully connected layer as the output channel in the current network segment; performing the following third processing on all next layers corresponding to the current convolutional layer or the current fully connected layer in sequence: in response to the input channel of the current next layer not being opened, using the input channel of the current next layer as the input channel in the current network segment, and adding the current network segment to the network segment set. Through this embodiment, according to the opening of the output channel of each convolutional layer or the fully connected layer and the input channel of the next layer corresponding to the output channel, the network segment that can be added to the network segment set is determined, thereby avoiding adding the already opened channel to the network segment set and repeatedly opening the already opened channel.

[0064] As an example, the output channels of all convolutional layers and fully connected layers can be iterated in a loop, that is, each convolutional layer and fully connected layer is processed as follows: determine whether the output channel of the current layer is opened, if the output channel is not opened, use the output channel as the output channel of the current network segment, then find all the next layers corresponding to the current layer, and iterate the input channels of each next layer in turn, that is, all the next layers are processed as follows: determine whether the input channel of the current next layer is opened, if the input channel is not opened, use the input channel of the current next layer as the input channel of the current network segment, so that the current network segment is obtained, and the current network segment is added to the network segment set to obtain the network segment set.

[0065] It should be noted that, assuming that the current layer corresponds to two next layers, and the input channels of each next layer are not opened, three network fragments can be obtained, namely: the first network fragment, including the output channel of the current layer and two input channels of the next layer; the second network fragment, including the output channel of the current layer and one input channel of the next layer; the third network fragment, including the output channel of the current layer and another input channel of the next layer, which is not limited in the present disclosure.

[0066] As an example, the enable status of the output channel recorded in the U array of each layer can be used to determine whether the corresponding output channel has been turned on, and the enable status of the input channel recorded in the V array of each layer can be used to determine whether the corresponding input channel has been turned on.

[0067] It should be noted that the pruning algorithm of the present invention selects a predetermined number of network segments according to significance and opens channels in the predetermined number of network segments to perform pruning processing. If the pruning ratio of the target model with the predetermined number of network segments opened still does not meet the user's requirements, the predetermined number of network segments will continue to be selected and the channels in the re-selected predetermined number of network segments will continue to be opened until the pruning ratio specified by the user is reached. Therefore, during the pruning process, some channels of the target model may be opened. When selecting a predetermined number of network segments, it is necessary to determine whether the corresponding channels are opened through the U array or the V array.

[0068] In an embodiment of the present disclosure, the third processing may further include: in response to the input channel of the current next layer being opened, skipping the current next layer; performing a fourth processing on each predetermined next layer: in response to the input channel of the predetermined next layer not being opened, treating the input channel of the predetermined next layer as a single network segment and adding it to the network segment set; wherein the predetermined next layer is the next layer waiting for the third processing after the current next layer among all the next layers corresponding to the current convolutional layer or the current fully connected layer. Through this embodiment, if the input channel of a next layer corresponding to the output channel of the current convolutional layer or the current fully connected layer is opened, then when the input channel of the next layer waiting for the third processing after the next layer is not opened, it can be added to the network segment set as a single network segment to avoid missing the input channel that is not opened.

[0069] As an example, when the output channels of all convolutional layers and fully connected layers are iterated in a loop, each convolutional layer and fully connected layer is processed as follows: determine whether the output channel of the current layer is opened. If the output channel is not opened, use the output channel as the output channel of the current network segment. Then find all the next layers corresponding to the current layer, and iterate the input channels of each next layer in turn. That is, all the next layers are processed as follows: determine whether the input channel of the current next layer is opened. If the input channel is opened, skip the current next layer. For the next layer waiting for the above processing after the current next layer, it can be determined whether the input channels of these next layers are opened, and the input channels that are not opened are taken as a network segment separately and added to the network segment set. In this way, the network segment set is further enriched to avoid missing the input channels that are not opened.

[0070] In an embodiment of the present disclosure, the third processing may further include: in response to all input channels of the next layer being opened, skipping all next layers, and adding the current network segment to the network segment set, wherein the current network segment only contains the output channel of the current convolutional layer or the current fully connected layer. Through this embodiment, when all input channels of the next layer corresponding to the output channel of the current convolutional layer or the current fully connected layer are opened, the output channel of the current convolutional layer or the current fully connected layer is taken as the current network segment and added to the network segment set to avoid missing the output channel that is not opened.

[0071] As an example, when looping and iterating the output channels of each layer in all convolutional layers and fully connected layers, that is, performing the following processing on each convolutional layer and fully connected layer: determine whether the output channel of the current layer is opened. If the output channel is not opened, use the output channel as the output channel of the current network segment, then find all the next layers corresponding to the current layer, loop and iterate the input channels of each next layer in turn. If the input channels of all the next layers are opened, skip all the next layers, and add the current network segment that only contains the output channel of the current layer to the network segment set to avoid missing the output channel that is not opened.

[0072] In step S303, based on the importance of each network segment in the network segment set, channels in a predetermined number of network segments in the target model are opened for pruning.

[0073] As an example, the pruning algorithm of the present invention selects a predetermined number of network segments according to significance and opens channels in the predetermined number of network segments to perform pruning processing. If the pruning ratio of the target model after opening the predetermined number of network segments still does not meet the user's requirements, the predetermined number of network segments will continue to be selected and the channels in the re-selected predetermined number of network segments will continue to be opened until the pruning ratio specified by the user is reached.

[0074] As an example, the above importance can be determined by the weights of the layers corresponding to the input channels and / or output channels in the network segment. For example, the importance (v) can be obtained as follows:

[0075] v1 = np.sum(np.abs(weight),axis = (2,3)) / / Add the absolute values ​​of all values ​​in the [2,3] dimension of weight

[0076] v = v1 / np.sqrt (np.sum (v1 * v1)) / / v1 divided by the square root of v1

[0077] As an example, the above importance can also be replaced by a density value. The density value of a network segment can be obtained by the importance and the corresponding computation amount. The computation amount (FLOPS) of the convolutional layer and the fully connected layer can be obtained by the following methods respectively:

[0078] Convolutional layer FLOPS = 2HW (C in K 2 +1)C out Among them, C in is the number of channels of the input feature map, K is the size of the square convolution kernel, H, W, C out is the height, width and number of channels of the output feature map;

[0079] Fully connected layer FLOPS = (2I-1)O, where I represents the input dimension and O represents the output dimension.

[0080] It should be noted that importance may also be replaced by other parameters, which is not limited in the present disclosure.

[0081] In an embodiment of the present disclosure, after opening channels in a predetermined number of network segments in the target model based on the importance of each network segment in the network segment set, in response to the ratio of the network segments opened in the target model not reaching the predetermined ratio, the next network segment set can be obtained based on the relationship between each convolutional layer and the fully connected layer, and the channels in a predetermined number of network segments in the target model can be opened based on the importance of each network segment in the next network segment set, until the ratio of the network segments opened in the target model reaches the predetermined ratio. Through this embodiment, a predetermined number of network segments are opened step by step, avoiding opening too many network segments at one time and occupying too many resources, and re-acquiring the corresponding network segment set each time further avoids the problem that the network segments selected are not optimal when a predetermined ratio of network segments are opened at one time.

[0082] As an example, after a predetermined number of network segments are selected based on the importance of each network segment in the network segment set, the channels in the predetermined number of network segments in the target model can be opened, and it can be determined whether the proportion of the opened network segments in the target model reaches the predetermined proportion specified by the user. If the proportion of the opened network segments in the target model does not reach the predetermined proportion, the next network segment set can be obtained based on the relationship between each convolutional layer and the fully connected layer, and based on the importance of each network segment in the next network segment set, a predetermined number of network segments can continue to be selected, and the channels in the reselected predetermined number of network segments in the target model can be opened until the proportion of the opened network segments in the target model reaches the predetermined proportion.

[0083] In order to facilitate the understanding of the present disclosure, Fig.10 Provide a description of the system.

[0084] Fig.10 is a flowchart of a pruning method according to an exemplary embodiment of the present disclosure, such as Fig.10 As shown in the figure, it shows the automatic acquisition of the relationship between the channels of Conv and FC in the CNN model and the network fragments, which mainly includes the following steps: 1. The CNN model to be pruned (i.e., the target model) is used as input and input into the pruning algorithm; 2. All convolutional layers and fully connected layers in CNN are scanned (Retrieve all the Conv&FC modules in CNN), that is, all convolutional layers and fully connected layers in the CNN model are obtained through the hook function and saved in a container; 3. Information such as the input of each layer in all convolutional layers and fully connected layers (Record input for each Conv&FC layer), such as obtaining the input of all convolutional layers and fully connected layers in the CNN model with a weight of 0 through the hook function, and saving the obtained input in a container; 4. Identify the connection relationship among all convolutional layers and / or fully connected layers (IdentifyRelationships amongst Conv&FC layers), that is, identify and save the causal relationship between all convolutional layers and fully connected layers in this CNN model; 5. Identify candidate network fragments (Identify Candidate Segments), that is, obtaining a predetermined number of network segments, that is, identifying and obtaining the most meaningful network segments that can be opened currently, such as selecting a predetermined number of network segments according to importance, so that the pruning algorithm can open the corresponding network segments.

[0085] It should be noted that Fig.10 For the specific process of steps 2 and 3, please refer to Figure 8 The process shown, Fig.10 The specific process of step 4 can be referred to Figure 5 The process shown is not discussed here, but for Fig.10 The specific process of step 5 can be referred to Fig.11 The process shown.

[0086] Fig.11 is a flow chart of identifying candidate network segments according to an exemplary embodiment of the present disclosure, such as Fig.11 As shown in the figure, the purpose is to obtain a predetermined number of network segments for use by the pruning algorithm based on the connection relationship between the input channels and output channels of all convolutional layers and fully connected layers obtained previously. Fig.11 There is another loop outside the process shown in the figure, which is based on the output channels (U channel) of all convolutional layers and fully connected layers. For simplicity, this loop is Fig.11 Not reflected in.

[0087] like Fig.11 As shown, the process of identifying candidate network segments mainly includes the following steps: 1. Get the Uchannel of the current loop iteration; 2. Determine whether this U channel has been enabled, that is, whether it has been opened, according to the U array; 3. If the judgment result shows that the U channel is not enabled, then add this Uchannel to the current network segment; 4. Perform a loop operation on all input channels (Vchannel) corresponding to this U channel; 5. For the VChannel of the current loop iteration, determine whether this V Channel has been enabled, that is, whether it has been opened, according to the V array; 6. If the judgment result shows that the V channel has not been enabled, then add this V Channel to the current network segment; 7. The current network segment forms a candidate network segment; 8. Record the information of this candidate network segment, such as recording it in the corresponding container, that is, adding it to the network segment set; 9. If this V channel has been enabled, that is, it has been opened, then clear the current network segment; 10. Start to obtain a new candidate network segment, for any V channel of all V channels behind this V channel, if it is determined that a certain V If the channel is not enabled, the corresponding Vchannel will be used as a candidate network fragment, that is, these candidate network fragments will only have V channels and no Uchannel; 11. After all convolutional layers and fully connected layers have completed the U channel cycle, return the topK network fragments with the highest density value and lowest density value for the pruning algorithm. It should be noted that the above density value can be obtained by dividing the importance of the channel by the corresponding computational effort (FLOPS).

[0088] In order to illustrate the feasibility of the present disclosure, the pruning processing method of the present disclosure is verified as follows: Figure 12 to Figure 14 As shown, the data and network fragments obtained by the present disclosure on the relationship between channels of the convolutional layer are shown. The general model structure relationship is used when the pruning algorithm selects network fragments, and the channel direct relationship is used in the process of initializing the pruning algorithm, where: Fig.12 A structural relationship diagram of a model is shown. Based on the structural relationship of the model, the relationship between the convolutional layers can be obtained. Fig.13 It shows some channel relationships of some convolutional layers. Fig.13 The brackets (n, m) before the equal sign indicate Fig.12 The channel m of the n-1th convolutional layer in the equation is represented by the brackets (x, y) after the equal sign. Fig.12The channel y of the x-th convolutional layer in, where Fig.13 The first box shows Fig.12 The connection relationship of some channels of the 0th convolutional layer in FIG. 1 , where since the 0th convolutional layer has 64 channels, the present disclosure only shows the channel relationship corresponding to channels 0 to 6. For example, the first row in the box indicates that channel 0 of the 0th convolutional layer is connected to channel 0 of the 1st, 3rd, 5th and 7th convolutional layers; Fig.13 The second box shows Fig.12 The connection relationship of some channels of the fourth convolutional layer is similar to that of the first box, which will not be discussed here; Fig.13 The third box shows Fig.12 The direct connection relationship of some channels of the 0th convolutional layer. For example, the first row in the box indicates that channel 0 of the 0th convolutional layer is directly connected to channel 0 of the 1st convolutional layer, that is, only through one Relu; Fig.13 The fourth box shows Fig.12 The direct connection relationship of some channels of the fourth convolutional layer in is similar to that of the third box, which will not be discussed here. Fig.12 After the channel relationship of all convolutional layers is obtained, the channel relationship executes the scheme disclosed in the present invention, and the TopK network fragments with decreasing importance can be obtained. Fig.14 Some of the top 10 network fragments returned are shown for use by the pruning algorithm. Fig.12 The relationship between the channels of the convolutional layer shown in the model structure can be used to input any CNN model without modifying any code, thereby completing the pruning operation.

[0089] According to another aspect of the exemplary embodiment of the present disclosure, a pruning processing device is provided. Fig.15 is a block diagram of a pruning processing device according to an exemplary embodiment of the present disclosure. Fig.15 , the pruning processing device comprises:

[0090] The channel closing unit 150 is configured to close the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned, wherein the target model is used for image processing, text processing or speech processing; the set acquisition unit 152 is configured to acquire a set of network fragments based on the connection relationship of each convolutional layer and / or fully connected layer, wherein each network fragment in the network fragment set includes input channels and output channels with causal relationships, separate input channels or separate output channels in the target model; the channel opening unit 154 is configured to open channels in a predetermined number of network fragments in the target model based on the importance of each network fragment in the network fragment set for pruning.

[0091] In an embodiment of the present disclosure, the channel closing unit 150 is also configured to perform the following first processing on each convolutional layer and fully connected layer before closing the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned: input the initial data into the target model to obtain a second input of any next layer of the current convolutional layer or the current fully connected layer, wherein the output of the current convolutional layer or the current fully connected layer in the target model is pre-set to a random number, and any next layer is a convolutional layer or a fully connected layer; in response to the second input being inconsistent with the first input, determining that the output channel of the current convolutional layer or the current fully connected layer has a causal relationship with the input channel of any next layer, and the first input is the input of any next layer when the output of the current convolutional layer or the current fully connected layer in the target model is not set to a random number.

[0092] In an embodiment of the present disclosure, the set acquisition unit 152 is further configured to perform the following second processing on each convolutional layer and the fully connected layer to obtain a network segment set: in response to the output channel of the current convolutional layer or the current fully connected layer not being opened, the output channel of the current convolutional layer or the current fully connected layer is used as the output channel in the current network segment; the following third processing is performed on all next layers corresponding to the current convolutional layer or the current fully connected layer in sequence: in response to the input channel of the current next layer not being opened, the input channel of the current next layer is used as the input channel in the current network segment, and the current network segment is added to the network segment set.

[0093] In an embodiment of the present disclosure, the set acquisition unit 152 is further configured to, in response to the input channel of the current next layer being opened, skip the current next layer; perform the fourth processing on each predetermined next layer: in response to the input channel of the predetermined next layer not being opened, treat the input channel of the predetermined next layer as a network segment separately, and add it to the network segment set; wherein the predetermined next layer is the next layer waiting for the third processing after the current next layer among all the next layers corresponding to the current convolutional layer or the current fully connected layer.

[0094] In an embodiment of the present disclosure, the set acquisition unit 152 is also configured to skip all next layers in response to the input channels of all next layers being opened, and add the current network segment to the network segment set, wherein the current network segment only contains the output channels of the current convolutional layer or the current fully connected layer.

[0095] In an embodiment of the present disclosure, the channel opening unit 154 is also configured to, after opening the channels in a predetermined number of network segments in the target model based on the importance of each network segment in the network segment set, continue to obtain the next network segment set based on the relationship between each convolutional layer and the fully connected layer in response to the proportion of the opened network segments in the target model not reaching the predetermined proportion, and open the channels in a predetermined number of network segments in the target model based on the importance of each network segment in the next network segment set, until the proportion of the opened network segments in the target model reaches the predetermined proportion.

[0096] In an embodiment of the present disclosure, the channel closing unit 150 is also configured to input the initial data into the target model before performing the following first processing on each convolutional layer and fully connected layer, obtain the input of all convolutional layers and fully connected layers of the target model, and record it as the first input of all convolutional layers and fully connected layers.

[0097] In an embodiment of the present disclosure, the channel closing unit 150 is also configured to set the weights of all convolutional layers and fully connected layers of the target model to zero before inputting initial data into the target model to obtain inputs of all convolutional layers and fully connected layers of the target model.

[0098] It should be understood that each unit / module in the storage method and storage device according to the exemplary embodiments of the present disclosure can be implemented as a hardware component and / or a software component. Those skilled in the art can implement each unit / module, for example, using a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), according to the processing performed by each defined unit / module.

[0099] According to another aspect of the exemplary embodiments of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the pruning processing method described in the present disclosure is implemented.

[0100] Specifically, the quantization method for a deep learning model according to an exemplary embodiment of the present disclosure may be written as a computer program, a code segment, an instruction, or any combination thereof, and recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. The computer-readable storage medium is any data storage device that can store data read out by a computer system. Examples of computer-readable storage media include: read-only memory, random access memory, read-only optical disk, magnetic tape, floppy disk, optical data storage device, and carrier wave (such as data transmission over the Internet via a wired or wireless transmission path).

[0101] According to another aspect of the exemplary embodiments of the present disclosure, an electronic device is provided, wherein the electronic device includes: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the pruning processing method described in the present disclosure.

[0102] Specifically, the electronic device can be broadly a tablet computer, a smart phone, a smart watch, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the electronic device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the electronic device can be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device can be used to connect and communicate with external devices via a network.

Claims

1. A pruning method, characterized in that: include: Closing input channels and output channels of all convolutional layers and fully connected layers of a target model to be pruned, wherein the target model is used for image processing, text processing, or speech processing; Based on the connection relationship of each convolutional layer and / or fully connected layer, a set of network segments is obtained, wherein each network segment in the set of network segments includes the input channel and the output channel having a causal relationship in the target model, the input channel alone, or the output channel alone; Based on the importance of each network segment in the network segment set, channels in a predetermined number of network segments in the target model are opened to perform pruning processing.

2. The pruning method according to claim 1, characterized in that: Before closing the input channels and output channels of all convolutional layers and fully connected layers of the target model to be pruned, it also includes: The first processing is performed on each convolutional layer and fully connected layer as follows: Inputting initial data into the target model to obtain a second input of any next layer of the current convolutional layer or the current fully connected layer, wherein the output of the current convolutional layer or the current fully connected layer in the target model is pre-set to a random number, and any next layer is a convolutional layer or a fully connected layer; In response to the second input being inconsistent with the first input, it is determined that an output channel of the current convolutional layer or the current fully connected layer has a causal relationship with an input channel of any next layer, and the first input is the input of any next layer when the output of the current convolutional layer or the current fully connected layer in the target model is not set to a random number.

3. The pruning method according to claim 1, characterized in that: The obtaining of a network segment set based on the connection relationship of each convolutional layer and / or fully connected layer includes: Perform the following second processing on each convolutional layer and fully connected layer to obtain the network fragment set: In response to an output channel of a current convolutional layer or a current fully connected layer not being opened, using the output channel of the current convolutional layer or the current fully connected layer as an output channel in a current network segment; The following third processing is performed in sequence on all next layers corresponding to the current convolutional layer or the current fully connected layer: in response to the input channel of the current next layer not being opened, the input channel of the current next layer is used as the input channel in the current network segment, and the current network segment is added to the network segment set.

4. The pruning method according to claim 3, characterized in that: Also includes: In response to the input channel of the current next layer being opened, skipping the current next layer; Performing a fourth process on each predetermined next layer: in response to the input channel of the predetermined next layer not being opened, treating the input channel of the predetermined next layer as a network segment separately, and adding the network segment set; The predetermined next layer is a next layer waiting for the third processing to be performed after the current next layer among all next layers corresponding to the current convolutional layer or the current fully connected layer.

5. The pruning method according to claim 3, characterized in that: Also includes: In response to all input channels of next layers being opened, all next layers are skipped, and the current network segment is added to the network segment set, wherein the current network segment only includes the output channels of the current convolutional layer or the current fully connected layer.

6. The pruning method according to claim 1, characterized in that: After opening channels in a predetermined number of network segments in the target model based on the importance of each network segment in the network segment set, the method further includes: In response to the proportion of opened network segments in the target model not reaching a predetermined proportion, continue to obtain the next set of network segments based on the relationship between each convolutional layer and the fully connected layer, and based on the importance of each network segment in the next set of network segments, open channels in the predetermined number of network segments in the target model until the proportion of opened network segments in the target model reaches the predetermined proportion.

7. The pruning method according to claim 2, characterized in that: Before performing the following first processing on each convolutional layer and fully connected layer, it also includes: The initial data is input into the target model to obtain the input of all convolutional layers and fully connected layers of the target model, and is recorded as the first input of all convolutional layers and fully connected layers.

8. The pruning method according to claim 7, characterized in that: Before inputting the initial data into the target model to obtain inputs of all convolutional layers and fully connected layers of the target model, the method further includes: Set the weights of all convolutional and fully connected layers of the target model to zero.

9. A pruning device, characterized in that: include: a channel closing unit, configured to close input channels and output channels of all convolutional layers and fully connected layers of a target model to be pruned, wherein the target model is used for image processing, text processing, or speech processing; A set acquisition unit is configured to acquire a network segment set based on a connection relationship of each convolutional layer and / or fully connected layer, wherein each network segment in the network segment set includes the input channel and the output channel having a causal relationship in the target model, the input channel alone, or the output channel alone; The channel opening unit is configured to open channels in a predetermined number of network segments in the target model based on the importance of each network segment in the network segment set so as to perform pruning processing.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the pruning method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the pruning processing method according to any one of claims 1 to 8.

Images