Cleaning optimization method for deep convolutional neural network after structured pruning
By constructing the DCNN network-wide connection topology diagram and cleaning, the redundant structure of discontinuous signal flow remaining after pruning is solved, the problem of redundant structure of DCNN after pruning is achieved, further compression and acceleration of the network are achieved, and suitable for edges Equipment deployment.
Patent Information
- Application Number
- CN202411269400.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-29
AI Technical Summary
The existing deep convolutional neural network (DCNN) structured pruning method retains a large number of redundant structures of discontinuous signal flow after pruning, affecting the compression and acceleration effects of the network.
By constructing the DCNN network-wide connection topology diagram, forward and backward cleaning are performed, the redundant structure of discontinuous characteristic signal flow that does not contribute to the network output is cleared, and a more streamlined model is generated.
Without increasing network accuracy loss, further compress and accelerate the network, suitable for resource-constrained edge device deployment.
Smart Images

Figure CN119047532A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a cleaning and optimization method for a deep convolutional neural network after structured pruning. Background Art
[0002] In recent years, deep convolutional neural networks (DCNNs) have achieved great success in many aspects of computer vision. In order to further improve network performance, the number of convolutional layers of DCNNs is increasing, while at the same time the network size is also expanding, requiring more storage and computing resources, which limits the deployment of DCNNs on resource-constrained edge devices. To address this problem, DCNN structure compression and optimization techniques have gradually attracted attention. This type of technology aims to make the performance of network models within a controllable range of accuracy loss, massively compress the network size, reduce the model's floating-point operations and memory resources, thereby greatly expanding the scope of application of DCNNs.
[0003] Network pruning is one of the main technologies for DCNN structure compression optimization. Pruning compresses the network by discarding useless parameters in the network, thereby reducing the amount of computation and memory usage and accelerating the model's reasoning process. DCNN network pruning is generally divided into two categories: unstructured pruning and structured pruning.
[0004] The typical method of DCNN unstructured pruning is weight pruning. Weight pruning can theoretically achieve a very high DCNN structure compression rate. However, in actual implementation, since the distribution of the pruned weights is relatively random during the weight pruning process, in order to obtain a high actual structure compression rate for the pruned DCNN model, the original efficient DCNN convolution calculation structure needs to be changed, so it cannot be directly implemented using the existing mature computing architecture. Therefore, this type of technology is subject to relatively large restrictions in actual implementation.
[0005] The structured pruning method of DCNN mainly includes filter pruning and convolution kernel pruning. Each convolution layer of DCNN is composed of a group of filters, and the number of filters is generally equal to the number of output channels (output feature maps) in the convolution layer. A filter is composed of convolution kernels equal to the number of input channels (input feature maps) of the convolution layer in which it is located. The function of a filter is to convolve each input channel feature map in the convolution layer with the corresponding convolution kernel in the filter to obtain a new feature map, and then superimpose these feature maps together, and through a nonlinear activation function, form an output channel feature map corresponding to the filter in the convolution layer. In filter pruning, the minimum pruning unit is the filter. Pruning a filter is equivalent to removing an output channel. In convolution kernel pruning, the minimum pruning unit is the convolution kernel. Pruning a convolution kernel is equivalent to removing the mapping information of the part of the input channel transmitted through the convolution kernel in the output channel of the filter where the convolution kernel is located. Therefore, filter pruning is a large-scale structural pruning, while convolution kernel pruning is a small-scale structural pruning.
[0006] DCNN structured pruning has received more extensive attention due to its high efficiency, clear physical meaning, and the pruned network is easier to implement using existing mature computing architectures.
[0007] However, many existing DCNN structured pruning methods for filter pruning and convolution kernel pruning generally only process each convolution layer of DCNN in isolation, without considering the structural correlation between adjacent convolution layers or even all convolution layers of the entire DCNN and the continuity of the characteristic signal flow. After pruning, a large number of redundant structures that appear as discontinuous signal flows will remain. Summary of the invention
[0008] In view of this, the present invention provides a cleaning and optimization method for a deep convolutional neural network after structured pruning, so as to solve the problem that a large number of redundant structures manifested as discontinuous signal flows remain after pruning.
[0009] In a first aspect, the present invention provides a method for cleaning and optimizing a deep convolutional neural network after structured pruning, the method comprising:
[0010] Obtaining a DCNN model, and performing structured pruning marking on a redundant structure of a DCNN network of the model;
[0011] Based on the connection relationship between each node in the DCNN network after pruning and marking, the corresponding DCNN full network connection topology graph is constructed;
[0012] According to the supporting node connection topology subgraphs corresponding to each node in the DCNN full network connection topology graph, forward cleaning and backward cleaning are performed on the DCNN full network connection topology graph respectively;
[0013] A simplified DCNN model is generated based on the cleaned DCNN full-network connection topology graph.
[0014] This method constructs a full-network connection topology graph by the connection relationship between each node in the DCNN network corresponding to the DCNN model marked by structured pruning, and then performs forward cleaning and backward cleaning according to the supporting node connection topology subgraph corresponding to each node in the full-network connection topology graph, so as to clean up the redundant structure of the non-continuous feature signal flow that does not contribute to the final output of the network, thereby generating a more streamlined DCNN model, and achieving further compression and acceleration of the network without increasing the loss of network accuracy.
[0015] In an optional implementation, the connection relationship between each node in the DCNN network after pruning marking is constructed to form a corresponding DCNN full network connection topology graph, including:
[0016] Determine all nodes in the DCNN network and generate a node set;
[0017] According to the directed edges between the nodes, an edge set of the DCNN network is constructed, and each directed edge in the edge set is associated with a corresponding attribute matrix;
[0018] Based on the directed edges and connection modes between each node and the corresponding predecessor node and successor node, a supporting node connection topology subgraph corresponding to each node is constructed;
[0019] Based on the node set, edge set and supporting node connection topology subgraphs corresponding to each node, a DCNN full-network connection topology graph is constructed.
[0020] In this implementation, by determining all the nodes in the network and the directed edges between the nodes, node sets and edge sets are generated respectively, and all directed edges are associated with the corresponding attribute matrices. At the same time, according to the specific connection conditions between the predecessor nodes and the successor nodes of each node, a supporting node connection topology subgraph corresponding to each node is generated, and then the whole network connection topology graph corresponding to the entire DCNN network is obtained, which can ensure that the whole network connection topology graph can effectively reflect the specific connection conditions between each node in the entire network structure.
[0021] In an optional implementation, the supporting node connection topology subgraph includes: a predecessor configuration, a successor configuration, a predecessor node set, a successor node set, a predecessor edge set and a corresponding predecessor edge update flag set, a successor edge set and a corresponding successor edge update flag set, and each update flag in the update flag set is used to indicate whether the attribute matrix of the corresponding edge can be updated;
[0022] According to the supporting node connection topology subgraph corresponding to each node in the DCNN full network connection topology graph, forward cleaning of the DCNN full network connection topology graph comprises:
[0023] Setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value;
[0024] Constructing a forward queue to be cleaned according to the successor node of the input node of the DCNN network, and setting the update flag of the successor edge of the input node to a second preset value;
[0025] According to the values of each update flag in the node's leading edge update flag set, determine in turn whether the node in the forward to-be-cleaned queue is a cleanable node;
[0026] If yes, based on the predecessor configuration and successor configuration of the cleanable node and the attribute matrices of each predecessor edge and successor edge, the attribute matrix of each successor edge of the cleanable node is updated;
[0027] The update flags of all the successor edges of the cleanable node are set to a second preset value, the forward queue to be cleaned is updated, and the step of sequentially determining whether the nodes in the forward queue to be cleaned are cleanable nodes is returned to be executed until the determined node is an output node of the DCNN network.
[0028] This implementation method starts from the input node of the entire DCNN network, builds a forward cleaning queue, and updates the attribute matrix of the successor edge in turn according to the specific information in the supporting node topological connection subgraph of each node. At the same time, during the update process, by updating the update flag and cleaning queue corresponding to each edge, it is ensured that the entire forward cleaning process can be carried out in an orderly manner, thereby ensuring the stability of the cleaning process.
[0029] In an optional implementation, the back-cleaning of the DCNN full network connection topology graph according to the supporting node connection topology subgraph corresponding to each node in the DCNN full network connection topology graph includes:
[0030] Setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value;
[0031] Construct a backward queue to be cleaned according to the leading node of the output node of the DCNN network, and set the update flag of the leading edge of the output node to a second preset value;
[0032] According to the value of each update flag in the node's successor edge update flag set, determine in turn whether the node in the backward to-be-cleaned queue is a cleanable node;
[0033] If yes, based on the predecessor configuration and successor configuration of the cleanable node and the attribute matrices of each predecessor edge and successor edge, the attribute matrix of each predecessor edge of the cleanable node is updated;
[0034] The update flags of all leading edges of the cleanable node are set to a second preset value, the backward queue to be cleaned is updated, and the steps of sequentially determining whether the nodes in the backward queue to be cleaned are cleanable nodes are returned to be executed until the determined node is an input node of the DCNN network.
[0035] This implementation method starts from the output node of the entire DCNN network, builds a backward cleaning queue, and updates the attribute matrix of the leading edge in turn according to the specific information in the supporting node topological connection subgraph of each node. At the same time, during the update process, by updating the update flags and cleaning queues corresponding to each edge, it is ensured that the entire forward cleaning process can be carried out in an orderly manner, thereby ensuring the stability of the cleaning process.
[0036] In an optional implementation, the updating of the attribute matrix of each successor edge of the cleanable node based on the predecessor configuration, the successor configuration and the attribute matrix of each predecessor edge and the successor edge of the cleanable node comprises:
[0037] According to the predecessor configuration and the successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge, respectively construct a predecessor stacking matrix and a successor stacking matrix of the cleanable node;
[0038] Based on the leader stacking matrix and a preset operation formula, a channel validity leader indication matrix is established;
[0039] The successor stacking matrix is cleaned based on the channel validity predecessor indication matrix to obtain an updated attribute matrix of each successor edge of the cleanable node.
[0040] In this embodiment, the predecessor configuration and successor configuration of the cleansable node and each predecessor edge and successor edge are combined to determine the predecessor stacking matrix and successor stacking matrix of the cleansable node, thereby obtaining the validity predecessor indication matrix based on the predecessor stacking matrix to clean the successor stacking matrix, and finally obtaining the updated attribute matrix of each successor edge of the cleansable node, which can efficiently and accurately complete the update of the attribute matrix of the successor edge of the cleansable node.
[0041] In an optional implementation, the updating of the attribute matrix of each leading edge of the cleanable node based on the leading configuration, the successor configuration and the attribute matrix of each leading edge and the successor edge of the cleanable node comprises:
[0042] According to the predecessor configuration and the successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge, respectively construct a predecessor stacking matrix and a successor stacking matrix of the cleanable node;
[0043] Based on the successor stacking matrix and the preset operation formula, a channel validity successor indication matrix is established;
[0044] The predecessor stacking matrix is cleaned based on the channel validity successor indication matrix to obtain an updated attribute matrix of each predecessor edge of the cleanable node.
[0045] In this embodiment, the predecessor configuration and successor configuration of the cleansable node and each predecessor edge and successor edge are combined to determine the predecessor stacking matrix and successor stacking matrix of the cleansable node, thereby obtaining a validity successor indication matrix based on the successor stacking matrix to clean the predecessor stacking matrix, and finally obtain an updated attribute matrix of each predecessor edge of the cleansing node, which can efficiently and accurately complete the update of the attribute matrix of the predecessor edge of the cleansable node.
[0046] In an optional implementation, the step of sequentially judging whether the nodes in the forward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the leading edge update flag set of the node includes:
[0047] Determine in sequence whether all update flags in the leading edge update flag set of each node in the forward queue to be cleaned are second preset values;
[0048] If yes, the corresponding node is determined as a cleanable node;
[0049] Alternatively, judging in sequence whether the nodes in the backward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the successor edge update flag set of the node includes:
[0050] Determine in sequence whether all update flags in the successor edge update flag set of each node in the backward to-be-cleaned queue are second preset values;
[0051] If yes, the corresponding node is determined as a cleanable node.
[0052] This embodiment determines whether a node is a cleansable node by determining whether all of the leading edge update flag sets or the successor edge update flag sets corresponding to each node in the to-be-cleaned list are second preset values, thereby ensuring that the entire forward cleaning or backward cleaning process is carried out accurately and efficiently.
[0053] In an optional implementation, generating a simplified DCNN model based on the cleaned DCNN full network connection topology graph includes:
[0054] The attribute matrix corresponding to each directed edge in the cleaned DCNN full network connection topology graph is simplified to obtain a simplified channel topology matrix;
[0055] According to the simplified channel topology matrix, retaining valid channels in each node;
[0056] According to the simplified channel topology matrix, a simplified convolutional layer is established and parameters are initialized;
[0057] Fill in the effective convolution kernel parameters of the streamlined convolution layer, and build a streamlined DCNN model based on the effective channels in each node.
[0058] In this embodiment, a streamlined DCNN model is established by retaining the valid channels within the nodes through the cleaned DCNN full-network connection topology diagram, establishing a streamlined convolution layer and filling in the valid convolution kernel parameters, which can achieve further compression and acceleration of the network without increasing the loss of network accuracy.
[0059] In a second aspect, the present invention provides a cleaning and optimization system for a deep convolutional neural network after structured pruning, the system comprising:
[0060] A pruning and marking module is used to obtain a DCNN model and perform structured pruning and marking on the redundant structure of the DCNN network of the model;
[0061] A topology map generation module is used to construct a corresponding DCNN full network connection topology map based on the connection relationship between each node in the DCNN network after pruning and marking;
[0062] A topology cleaning module, for performing forward cleaning and backward cleaning on the DCNN full network connection topology map according to the supporting node connection topology subgraphs corresponding to each node in the DCNN full network connection topology map;
[0063] The simplified model generation module is used to generate a simplified DCNN model based on the cleaned DCNN full network connection topology graph.
[0064] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the cleaning and optimization method of the structured pruned deep convolutional neural network of the first aspect or any corresponding embodiment thereof.
[0065] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for cleaning and optimizing a deep convolutional neural network after structured pruning according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0067] Figure 1 It is a flow chart of a cleaning and optimization method of a deep convolutional neural network after structured pruning according to an embodiment of the present invention;
[0068] Figure 2 Schematic diagram of a typical structure of a DCNN convolutional layer provided in an embodiment of the present invention;
[0069] Figure 3 It is a schematic diagram of the relationship between the convolution kernel (kernel) in the typical convolution layer of the DCNN provided by an embodiment of the present invention, the convolution kernel set (group) associated with the input channel, and the filter (filter);
[0070] Figure 4 This is an example diagram of an interruption of a feature signal flow after DCNN pruning provided by an embodiment of the present invention;
[0071] Figure 5 It is a flow chart for constructing a DCNN network full network connection topology graph (CTG) provided by an embodiment of the present invention;
[0072] Figure 6 It is an example diagram of one of the convolutional layers, the retained convolution kernel labels, the channel topology matrix, and the connection topology diagram after DCNN pruning provided by an embodiment of the present invention;
[0073] Figure 7a to Figure 7b It is an example diagram of multiple convolutional layers, channel topology matrix and connection topology diagram sequentially connected after DCNN pruning provided by an embodiment of the present invention;
[0074] Figures 8a to 8d It is an example diagram of different structural forms of DCNN network channel topology subgraphs provided by an embodiment of the present invention;
[0075] Fig. 9 is a forward cleaning flow chart of a pruned DCNN network provided by an embodiment of the present invention;
[0076] Fig.10 It is a forward cleaning flow chart of a support node connection topology subgraph of a pruned DCNN network provided by an embodiment of the present invention;
[0077] Fig.11is a backward cleaning flow chart of a pruned DCNN network provided by an embodiment of the present invention;
[0078] Fig.12 It is a backward cleaning flow chart of a support node connection topology subgraph of a pruned DCNN network provided by an embodiment of the present invention;
[0079] Attached Fig.13 An intuitive mapping flow chart for generating a simplified DCNN model using a cleaned channel topology matrix provided by an embodiment of the present invention;
[0080] Fig.14 It is a structural block diagram of a cleaning optimization system of a deep convolutional neural network after structured pruning according to an embodiment of the present invention;
[0081] Fig.15 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0082] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0083] In order to more clearly illustrate the concept of the present invention, some concepts of the deep convolutional neural network (DCNN) involved are first explained.
[0084] A DCNN consists of multiple convolutional layers. Figure 2 A typical structural diagram of the DCNN convolutional layer is given.
[0085] like Figure 2 As shown in the figure, the input channel of the lth layer is the output channel of the l-1th layer; the output channel of the lth layer is the input channel of the l+1th layer. Assuming that the lth convolutional layer of DCNN has M input channels and N output channels, the input and output of the lth convolutional layer are mapped by N filters.
[0086] A filter contains M convolution kernels (kernels, convolutional kernels), and its function is to map the information of M input channels to a specific output channel through convolution, superposition, nonlinear activation and pooling operations through the corresponding convolution kernels.
[0087] The convolution kernel is the basic processing structure in DCNN. A convolution kernel maps the information of a specific input channel through convolution and sends it to the output channel where the filter containing the convolution kernel is located. In a convolution layer consisting of M input channels and N output channels, there are a total of M*N convolution kernels.
[0088] The convolution kernel set (group) associated with the input channel is the set of convolution kernels divided according to the input channel in the convolution layer. In a convolution layer consisting of M input channels and N output channels, there are M groups, each containing N convolution kernels. For example, the i-th group is a set of M convolution kernels of equal position associated with the i-th input channel in all N filters.
[0089] In summary, a layer is associated with multiple input channels and multiple output channels, an input channel is associated with a group, an output channel is associated with a filter, and a kernel establishes a connection between a specific input channel and a specific output channel. The relationship between the filter, convolution kernel, and the convolution kernel set associated with the input channel in the convolution layer (layer) is as follows: Figure 3 shown.
[0090] In many existing DCNN structured pruning methods based on filter pruning or convolution kernel pruning, redundant discontinuous feature signal flows often appear after pruning, resulting in interruption of the network feature signal flow after pruning, because the structural correlation between convolutional layers and the continuity of feature signal flows are not fully considered. Figure 4 As shown in the example, the square blocks represent the input feature maps (Input Feature Map) on the input channel and the output feature maps (Output Feature Map) on the output channel; the middle dots represent the convolution kernels (kernels), the black solid dots represent the kernels retained after pruning, and the white hollow dots represent the kernels cut off after pruning.
[0091] In the first case, Figure 4In the example, since all kernels in the first row have been pruned, it means that the filter composed of these kernels has been pruned, and the output channel corresponding to the filter will not be able to produce valid output, and the signal flow will be interrupted on the output channel. In this way, all kernels in the group related to this channel in the next convolution layer will be invalid and need to be removed. The pruning of the kernels in the next layer will generate redundant structures in the subsequent layers, and so on.
[0092] In the second case, Figure 4 In the example, since all kernels in the second column have been cut off, it means that the group composed of these kernels has been cut off. The information of the input channel corresponding to the group will not be received by the filters in this convolution layer, and the signal flow will be interrupted on the input channel. In this way, the filter related to the channel in the previous convolution layer will be invalid and need to be removed. The removal of the filter in the previous layer will generate redundant structures in the previous layer, and so on.
[0093] To this end, an embodiment of the present invention provides a method for cleaning and optimizing the redundant structure of a structured pruned DCNN network, which is precisely to eliminate the redundant structure of these interrupted non-continuous channels in the pruned DCNN network, and then remove the entire redundant structure affected by these non-continuous channels in the forward and backward propagation of the pruned DCNN network, so as to achieve further compression and acceleration of the network without increasing additional accuracy loss.
[0094] According to an embodiment of the present invention, an embodiment of a method for cleaning and optimizing a redundant structure of a DCNN network after structured pruning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.
[0095] Figure 1 is a flow chart of a cleaning and optimization method for a deep convolutional neural network after structured pruning according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:
[0096] Step S1: Obtain a DCNN model and perform structured pruning marking on the redundant structure of the DCNN network of the model.
[0097] Users can select or design a suitable DCNN neural network model based on their specific tasks, and fully pre-train the selected DCNN using public datasets such as ImageNet or user-proprietary datasets to ensure that the model performance reaches the required accuracy.
[0098] At present, academia and industry have proposed a variety of DCNN model types for different tasks, such as single-branch VGG, densely connected DenseNet, multi-branch ResNet, GoogleNet, Unet, etc., and there are nearly 100 variants of each type of DCNN. There are many DCNN network model construction and training frameworks, such as Pytorch, Tensorflow, etc., which will not be elaborated here.
[0099] The existing DCNN structured pruning methods are roughly divided into two types in the implementation process: one-time pruning or progressive pruning. One-time pruning refers to pruning as many redundant structures of the network as possible for a fully pre-trained DCNN model, and then fine-tuning the pruned DCNN; progressive pruning refers to the DCNN model that is fully pre-trained. The model continues to be trained during the pruning process, and the pruning rate gradually increases during the pruning process. Under the condition of meeting an acceptable loss of model accuracy, as many redundant structures of the network as possible are pruned, and the training ends when the pruning is completed.
[0100] In the embodiment of the present invention, the structured pruning of DCNN refers to a pruning method with a convolution kernel as the minimum granularity, including structured pruning methods such as convolution kernel pruning and filter pruning. There are many methods of this type of structured pruning methods currently published in various academic journals, which will not be repeated here.
[0101] In the actual structured pruning process, for the convenience of implementation, the pruned DCNN network structure is generally not directly removed during the process, but a set of pruning marks are used to prune the network structure. Usually, the retained structure is marked as 1, and the pruned structure is marked as 0. After the pruning is completed, the pruning marks are used to compress the structure to form a new streamlined network.
[0102] In the present invention, convolution kernel is used as the basic unit for pruning marking. Figure 4 As shown, assuming that the convolutional layer l (or one of its branches) has M input channel feature maps ( Figure 4 M=4), there are N output channel feature maps ( Figure 4 In the example, N = 3), there are M*N convolution kernels involved in the mapping between the input channel and the output channel. We use the N*M logical matrix K i,i To mark the effectiveness of the convolution kernel after pruning.
[0103]
[0104] Where i represents the overall number of the input channel ( Figure 4 where i=1-1); Indicates that the mth kernel of the nth filter in the convolutional layer is pruned, and Indicates that the convolution kernel is not pruned. In the present invention, K i,I It is called the channel topology matrix of convolutional layer l. Figure 4 Middle K I-1,l As follows:
[0105]
[0106] Among them, K l--1,l The first row of K is all 0, indicating that the first filter of convolutional layer l is pruned as a whole; l-1,l The second row indicates that the second and third kernels in the second filter are pruned; K l-1,l The third line indicates that the second kernel of the third filter is pruned.
[0107] For all convolutional layers or convolutional layer branches of the pruned DCNN network, their channel topology matrices are established according to the pruning results, that is, the convolution kernel-based pruning marking of the DCNN network is realized.
[0108] Step S2, based on the connection relationship between each node in the DCNN network after pruning marking, construct the corresponding DCNN full network connection topology map.
[0109] Figure 5 It is a flow chart of constructing a DCNN full network connection topology graph (CTG) provided by an embodiment of the present invention, such as Figure 5 As shown, the process includes the following steps:
[0110] Step S201, determine all nodes in the DCNN network and generate a node set.
[0111] The overall connection structure of the DCNN network is a unidirectional, acyclic, and non-feedback structure from input to output. The input of the DCNN network consists of a set of channel feature maps, and the output of the DCNN network consists of another set of channel feature maps. From the network input to the network output, it consists of a series of single-branch or multi-branch convolutional layers. Some DCNN networks are composed of pure single-branch convolutional layers, such as the VGG network; a large number of extended DCNN networks are composed of multi-branch convolutional layers, such as ResNet, DenseNet, GoogLeNet, Unet network, etc. The single-branch convolutional layer maps the input channel feature map of the convolutional layer to the output channel feature map by a set of filters. The multi-branch convolutional layer includes two structures: aggregation and distribution. The aggregation structure includes multiple sets of input channel sets, each set of input channels contains a set of different filters, which respectively map their own input channel feature maps to a set of feature maps, and combine multiple sets of feature maps to form a unified output channel feature map; the distribution structure means that the same set of channel feature maps are used as the input of different branch convolutional layers, and are mapped to different branch output channel feature maps by different filter groups, and propagated forward.
[0112] For the convenience of modeling, the present invention takes the entire output channel feature map in each convolutional layer (single branch or multi-branch) as a single CTG graph node, and numbers all nodes in the DCNN network.
[0113] The entire output channel feature map of the convolutional layer number l in the DCNN network is denoted as node v l , assuming that the DCNN network contains N T convolutional layer nodes, then the network connection topology graph G contains N T +1 node, and the set of all nodes in the DCNN network is denoted as V:
[0114]
[0115] Among them, the input channels of the DCNN network are all recorded as nodes v 0 , N T is the number of convolutional layers of DCNN. In particular, we call v l , 1≤l≤N T -1 is the support node. 0 is the input node of the DCNN network, It is the output node of the DCNN network and is composed of all the output channels of the DCNN network.
[0116] Step S202, constructing an edge set of the DCNN network according to the directed edges between the nodes, and associating each directed edge in the edge set with a corresponding attribute matrix.
[0117] In the DCNN network, if two nodes vi With v j There is a direct connection between two nodes, then there is a directed edge e i,j :v i →v j , the set of all edges connecting nodes in the DCNN network to be analyzed is taken as the edge set E:
[0118] E={e i,j :v i →v j |v i , v j ∈V,0≤i <N T , 0 <j≤N T}
[0119] Edge i,j Associated with node v i With node v j The channel mapping relationship between them, if the node v i With node v j If there is a convolutional layer or a convolutional layer branch between them, then the edge e is defined i,j An attribute matrix is as follows:
[0120]
[0121] in, is node v i Inside and v j The number of associated channels, is node v j Inside and v i The number of channels associated with it. Specifically, is node v i Participate in convolution or participate in shortcut direct connection to form node v j The number of channels in the inner feature map; is node v i The number of feature maps involved in convolution or shortcut direct connection, these feature maps are node v j All or part of the channels.
[0122] In the present invention, K i,j is the channel topology matrix, matrix K i,j This is the pruning label matrix obtained in step S1, whose elements like It represents node v i The mth input channel in the node v j The convolution kernels corresponding to the n-th output channels in are not pruned; if It means that the corresponding convolution kernel is pruned.
[0123] If the edge i,j It is a shortcut direct connection relationship similar to ResNet. At this time, e i,j The attribute matrix K i,j Expressed as a unit matrix:
[0124]
[0125] For ease of understanding, Figure 6 1 is an example diagram of one of the convolutional layers, the retained convolution kernel labels, the channel topology matrix, and the connection topology map after DCNN pruning provided by an embodiment of the present invention. Figure 6 As shown in the figure, after the convolution layer is pruned, the black circles represent valid convolution kernels, and the white circles represent invalid convolution kernels. i is the input channel of the convolution layer, node v j is the output channel of the convolutional layer, node v i With node v j Between the edges i,j The channel topology matrix is K i,j :
[0126]
[0127] By K i,j Discontinuous channels in the input channels and output channels of the convolutional layer can be judged intuitively. Figure 6 Invalid filters and groups appear in the convolutional layer, namely filter2 and group2, and the corresponding matrix K i,j The second row and the second column are all 0, and the node v i The second channel and node v j The second channels of are all discontinuous channels. These channels prevent the feature signal from being transmitted backwards and do not contribute to the final output of the DCNN network. Therefore, discontinuous channels are also called invalid channels.
[0128] For further understanding, Figure 7a to Figure 7b is an example diagram of multiple convolutional layers, channel topology matrix and connection topology diagram connected sequentially after DCNN pruning provided by an embodiment of the present invention. Figure 7a to Figure 7b As shown, Figure 7a This is a schematic diagram of the network connection topology of a single-branch multi-layer convolution. Figure 7b Schematic diagram of the network connection topology of multi-branch convolution.
[0129] Figure 7b In the example, node v i With a distributed multi-branch structure, nodes It has a convergent multi-branch structure.
[0130] Step S203, constructing a supporting node connection topology subgraph corresponding to each node based on the directed edges and connection modes between each node and the corresponding predecessor node and successor node.
[0131] v l The supporting node connection topology subgraph (sCTG) of G l is the supporting node v l , and in DCNN with v l The local connection graph composed of directly connected nodes, G l As follows:
[0132]
[0133] Among them, 0≤l≤N T , v l ∈V is a graph G l The supporting node connection topology subgraph includes: a predecessor configuration, a successor configuration, a predecessor node set, a successor node set, a predecessor edge set and a corresponding predecessor edge update flag set, a successor edge set and a corresponding successor edge update flag set, and each update flag in the update flag set is used to indicate whether the attribute matrix of the corresponding edge can be updated.
[0134] The specific instructions are as follows:
[0135] Yes l The leading node subset of l A subset of the fan-in nodes. is in V and converges directly to v l The set of all nodes of . Assume YesN B nodes.
[0136] Yes 1 The leading node subset of The nodes in v l The connected edge set, i.e. The internal structure of edge ebi,l is described by its attribute matrix Kbi,l, which represents the intra-layer channel connection characteristics between node vbi and node vl. That is the node With node v l The pruned label matrix between .
[0137] Yes l The subset of successor nodes, also called v l A subset of fan-out nodes. is in V by v l The set of all directly connected nodes pointed to by . Assume YesN A nodes.
[0138] Yes l and the successor node subset The set of edges connected by the nodes in The side The internal structure of describe, Characterizes the node v l With Node In the present invention, the intra-layer channel connection characteristics between That is, node v l With Node The pruned label matrix between .
[0139] yes The nodes inside and the nodes v l The convergence form between Called v l The leading configuration. It is set to one of the three constants {'CONV', 'ADD', 'CONCAT'}, which respectively correspond to the three convergence connection modes of single branch connection and multi-branch connection in the DCNN network. Figures 8a to 8d The instructions are as follows:
[0140] ①If correspond Figure 8a situation, There is only one predecessor node Only one leading edge The connection relationship corresponding to the single-branch convolutional layer in the DCNN network.
[0141] ②If correspond Figure 8b The situation corresponds to the situation in the DCNN network where multiple branches converge and then perform channel addition, such as the residual aggregation node of the ResNet network. Figure 8b middle, The node inside points to the node v l There are N edges in total B If the node With node v l The connection relationship between them is a convolution relationship, then the edge Representation Node A set of feature maps are obtained by performing convolution operations; if the node With node v lThere is a shortcut direct connection between them, then the edge Representation Node The feature map in is used as the mapped feature map. B N obtained by different edges B The group feature maps are added correspondingly to obtain a unified group of channel feature maps to form node v l .
[0142] ③If correspond Figure 8c The situation corresponds to the situation where channels are stacked after multi-branch convergence in the DCNN network, such as the multi-branch node convergence in the Unet network. Figure 8c middle, The node inside points to the node v l There are N edges in total B If the node With node v l The connection relationship between them is a convolution relationship, then the edge Representation Node A set of feature maps are obtained by performing convolution operations; if the node With node v l There is a shortcut direct connection between them, then the edge Representation Node The feature map in is used as the mapped feature map. B N obtained by different edges B The group feature maps are stacked in the channel dimension to obtain a group of extended channel feature maps whose total number is the sum of the number of mapping channels of each branch, forming a node v l .
[0143] is node v l and The distribution form between nodes within Called v l The subsequent configuration of . It is set to one of the two constants {'CONV', 'DIS'}, corresponding to the two distribution modes of single-branch connection and multi-branch connection in the DCNN network. Figures 8a to 8d The instructions are as follows:
[0144] ①If correspond Figure 8a situation, There is only one successor node There is only one successor edge The connection relationship corresponding to the single-branch convolutional layer in the DCNN network.
[0145] ②If correspond Figure 8d The situation corresponds to the situation in which a convolutional layer node in the DCNN network generates multiple subsequent branches after distribution, such as the distribution of the Inception structure in the GoogLeNet network. Figure 8d middle, Each successor edge in Indicates that the nodes are convolved by different filter groups or directly connected by shortcuts to form different nodes. That is, multiple convolution branches are formed.
[0146] is the set of leading edge update flags, is the leading edge subset The edge The attribute matrix If Indicates the current Cannot be updated yet; if Indicates the current Can be updated.
[0147] is the set of successor edge update flags, is a subset of successor edges The edge The attribute matrix If Indicates the current Cannot be updated yet; if Indicates the current Can be updated.
[0148] From the above G l Definition, for each node v in the DCNN network node set V l , l=0,1,…,N T , establish the supporting node connection topology subgraph (sCTG) G l , as follows:
[0149] First, in the DCNN network, find the nodes directly connected to node v l All nodes connected to v are divided into two groups: l The set of nodes that pass the signal, marked as Receive v l The node set of the signal, labeled At the same time, record The nodes in and the nodes v l of the aggregation form, and give Assign the corresponding aggregation type constant value; record v l and The distribution form of the nodes in Assign the corresponding distribution type constant value.
[0150] Secondly, establish The inner node points to the node v l The edge set Node v l point to The edge set of the internal node And for each edge and Create attribute matrix and The DCNN network pruning label matrix in step S1 is compared with Establish a synchronous index relationship.
[0151] Finally, the update flag set for the attribute matrix and Set each element in to 0.
[0152] In particular, for the input node v of the DCNN network 0 , since there is no predecessor node, For the output nodes of the DCNN network Since there is no successor node,
[0153] Step S204, constructing a DCNN full-network connection topology graph based on the node set, edge set, and supporting node connection topology subgraphs corresponding to each node.
[0154] The node set V of the DCNN network is constructed using step S201 as follows:
[0155]
[0156] The edge set E of the DCNN network is constructed using step S202 as follows:
[0157] E={e i,j :v i →v j |v i , v j ∈V,0≤i <N T , 0 <j≤N T}
[0158] Use step S203 to construct the support node v of DCNN l The connection topology subgraph G l , l = 0, 1, ..., N T .
[0159] The full network connection topology graph G of the DCNN network is composed of NT +1 supporting node connected to form a topological subgraph, defined as follows:
[0160] G={V,E,G l , l=0,1,…,N T}
[0161] Among them, G l is the v constructed in step S203 1 The supporting node connection topology subgraph, G l Describes the DCNN with node v l The local connection structure is centered on G. 0 is the DCNN network input node v 0 The supporting node connection topology subgraph, It is the output node of the DCNN network The supporting nodes connect the topological subgraph.
[0162] The node set V, edge set E, and supporting node connection topology subgraph G of the aforementioned DCNN l From the construction of G l Medium l , Each node and Each edge of is an element in the node set V and the edge set E. Leading configuration and subsequent configurations is related to the local connection structure of the DCNN network l Internal attributes of node v i With node v j The edge between i,j :v i →v j , and the edge attribute matrix K i,j , describes the node v i With node v j The specific channel connection characteristics between the edges i,j The attribute matrix K i,j Update flag u i,j , describes the subsequent attribute matrix K i,j Status of the cleaning process.
[0163] In the subsequent steps of the present invention, in the cleaning of the redundant structure of the DCNN pruning network, in the subgraph G l The edge attribute matrix changed and updated in G, as well as the update flag of the attribute matrix, will be passed to l In the specific implementation, various data structures can be used to realize the transmission of this information. A more commonly used one can be: for each node v in the node set V in the graph G l , and its configuration Establish a global data structure for edge e in edge set E i,j , edge attribute matrix K i,j , and update flag u i,j Also establish a global type data structure. Subgraph G l The node set in Edge Set The data in the __INITIALIZATION_STRUCTURE is created in the form of an indirect reference data structure in a similar way to pointer variables in C language. The specific implementation method varies in different programming languages, so I will not go into details here.
[0164] The labeling information of the redundant structure of the DCNN pruned network is recorded on the edge e in graph G. i,j The attribute matrix K i,j In the present invention, these redundant structures refer to those structures marked as valid in which the signal in the leading convolution layer or the subsequent convolution layer is interrupted due to the invalid channel of the convolution layer input or output during the transmission process of the characteristic signal flow of the pruned DCNN network. Such invalid structures will continue to be transmitted forward or backward.
[0165] The cleaning of redundant structures in DCNN pruning network is to eliminate the edge i,j The attribute matrix K i,j The channel connection marks in the convolution layer are marked as 1, but are actually invalid due to the interruption of the signal in the adjacent convolution layer, and are set to 0 to mark them as invalid.
[0166] The cleaning of redundant structures of the DCNN pruning network is divided into two stages: forward and backward, such as the subsequent steps S3 and S4.
[0167] Step S3, forward cleaning the DCNN full network connection topology graph according to the supporting node connection topology subgraph corresponding to each node in the DCNN full network connection topology graph.
[0168] Fig. 9 is a forward cleaning flow chart of a pruned DCNN network provided by an embodiment of the present invention, such as Fig. 9 As shown, the process includes the following steps:
[0169] Step S301, setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value.
[0170] Reset the update flag of the attribute matrix of all edges in graph G to 0, that is, u i,j =0,e i,j ∈E, that is, all edges e i,j ∈E's attribute matrix K i,j Set to the waiting state and wait for cleaning.
[0171] Step S302: construct a forward queue to be cleaned according to the successor nodes of the input node of the DCNN network, and set the update flag of the successor edge of the input node to a second preset value.
[0172] Establish a queue Q of supporting nodes to be cleaned.
[0173] In the node set V of graph G, take the DCNN network input node v 0 The supporting node connection topology subgraph G 0 :
[0174]
[0175] Subset the successor nodes of G0 All nodes in are put into queue Q in turn, and the successor edge update flag set All update flags in are set to 1, that is,
[0176] Step S303 , judging in sequence whether the nodes in the forward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the leading edge update flag set of the node.
[0177] Take out the frontmost supporting node in the queue Q. Assume it is v l , and take v from graph G l The corresponding subgraph G l , to determine v l Whether it is a cleanable node. Delete node v in queue Q l .
[0178] Specifically, step S303 includes: sequentially determining whether all update flags in the leading edge update flag set of each node in the forward queue to be cleaned are second preset values;
[0179] If yes, the corresponding node is determined as a cleanable node.
[0180] Pair of subgraphs G l To judge the status, check subgraph G l The leading edge update flag set All Are all values set to 1?
[0181] like Existence Indicates G l If the leading edge attribute matrix of has not been updated yet, then v l Put it at the end of the queue Q to be cleaned and go to the beginning of step S303.
[0182] like All And l≠N T , then the current supporting node can be cleaned and go to step S304.
[0183] like All And l = N T , then node v l It is the output node of DCNN Go to step S306.
[0184] Step S304: based on the predecessor configuration and the successor configuration of the cleanable node and the attribute matrices of the predecessor edge and the successor edge, the attribute matrix of each successor edge of the cleanable node is updated.
[0185] Specifically, Fig.10 is a forward cleaning flow chart of a support node connection topology subgraph of a pruned DCNN network provided by an embodiment of the present invention, such as Fig.10 As shown, the process includes the following steps:
[0186] Step S30401, constructing a predecessor stacking matrix and a successor stacking matrix of the cleanable node according to the predecessor configuration and successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge.
[0187] According to v l The leading configuration as well as The edge attribute matrix set
[0188]
[0189] Among them, N B yes The number of nodes, construct v l The leading stacking matrix of The details are as follows:
[0190] ①If Predecessor node subset Each node in the convolution or direct connection (shortcut) forms a set of channel feature maps, and the node v l It is obtained by adding these different channel feature map groups together, so Inner Edge The attribute matrix of the channel topology matrix have the same number of rows, and the number of rows is equal to v l The number of channels in Each node in the shared node v l The channel is used as the output channel. In this case, v l The leading stacking matrix of The form is as follows:
[0191]
[0192] Right now, By N B Sub-matrices The horizontal block matrix composed of Number of columns for:
[0193]
[0194] in, Is a node The number of channels, that is Number of columns is the sum of the number of columns of each block matrix.
[0195] ②If v l The leading node subset of There is only one node in the leading stacking matrix It's the edge The attribute matrix Right now:
[0196]
[0197] ③If Each node in the network is convolved or directly connected (shortcut) to form a set of feature graphs, and the node v l It is formed by stacking these different channel feature map groups in the channel dimension. The node v l Number of channels for Inside each side The attribute matrix The sum of the number of rows. At this time, v l The leading stacking matrix of Has a block diagonal form:
[0198]
[0199] in,
[0200]
[0201] is node v l Inner and Node The number of channels associated with a node The number of feature maps formed by convolution or direct connection (shortcut), also a matrix number of rows; Is a node The number of channels is also the matrix The number of columns; Total number of columns is the sum of the number of columns of each block matrix, Total number of rows is the sum of the number of rows in each block matrix.
[0202] According to v l The subsequent configuration as well as The edge attribute matrix set
[0203]
[0204] Among them, N A yes The number of nodes, construct v l The successor stacking matrix The details are as follows:
[0205] ①If Node vl forms a set of channel feature maps through different convolution operations or direct connections (shortcuts). These different channel feature map groups constitute For each node in , in this case, Inner Edge The attribute matrix of the channel topology matrix have the same number of columns, and the number of columns is equal to v l The number of channels in Each node in the shared node v l The channel of v is used as the input channel. l The successor stacking matrix The form is as follows:
[0206]
[0207] Right now, By N A Sub-matrices The vertical block matrix composed of number of rows for:
[0208]
[0209] in, Is a node The number of channels, that is Total number of rows It is the sum of the number of rows in each block matrix.
[0210] ②If v l The subset of successor nodes There is only one node in the matrix, and the subsequent stacking matrix It's the edge The attribute matrix As shown below:
[0211]
[0212] Step S30402: Based on the leader stacking matrix and the preset operation formula, a channel validity leader indication matrix is established.
[0213] To establish v l Channel validity preamble indicator matrix First, we define three operation symbols: logical AND '∧', logical OR 'V', and logical matrix multiplication. The operation is as follows:
[0214] ∧: 0∧0=0, 0∧1=1∧0=0, 1∧1=1
[0215] V: 0∨0=0, 0∨1=1 V0=1, 1∨1=1
[0216] '∨' is a logical OR operation, aVb=max(a,b), which means taking the larger value of the two; '∧' is a logical AND operation, a∧b=min(a,b), which means taking the smaller value of the two.
[0217] If a given matrix A = [a ir ] N×R , matrix B = [b rj ] R×M , define matrix logical multiplication as follows:
[0218]
[0219]
[0220] Analogous to the algebraic multiplication of two matrices, the matrix logical multiplication defined here is The operation can be seen as the result of replacing the multiplication ‘×’ in matrix algebraic multiplication with the logical AND ‘∧’, and the addition ‘+’ in algebraic multiplication with the logical OR ‘V’.
[0221] make:
[0222]
[0223] yes A column vector with all elements set to 1, v l The leading stacking matrix of for:
[0224]
[0225] v l Channel validity preamble indicator matrix Is and Do matrix logical multiplication The diagonal matrix formed by The definition is as follows:
[0226]
[0227] in,
[0228]
[0229] is node v l The number of channels, is a matrix The number of columns, yes The element in row n and column m. When the nth row of is all 0, The value of node v is 0, that is, l The nth channel is an invalid channel.
[0230] v l Channel validity preamble indicator matrix is a Diagonal logical matrix, the element values on the off-diagonal are all 0, and the elements with a value of 1 on the diagonal represent the node v l The corresponding channel in is the valid channel; otherwise, the element node v with a value of 0 l The corresponding channel is an invalid channel.
[0231] Step S30403, clean the successor stacking matrix based on the channel validity predecessor indication matrix to obtain an updated attribute matrix of each successor edge of the cleanable node.
[0232] v l Channel validity preamble indicator matrix Indicates the node v l The valid channels and invalid channels of the subsequent stacking matrix With the leading indicator matrix Do matrix logical multiplication And reassign the result to The operation is as follows:
[0233]
[0234] v l Channel validity preamble indicator matrix The diagonal elements of represent v l The channel validity in . The diagonal element value of v is 1, then l The channel corresponding to the element is valid; otherwise, if the element value is 0, the corresponding channel is invalid. Matrix logical right multiplication Then The columns whose diagonal elements are 0 correspond to The column elements in are all set to 0, while keeping The columns whose diagonal elements are 1 correspond to The column elements in remain unchanged, so cleaning removes Zhong and v l Possible connection tags associated with the invalid channel.
[0235] According to v l The subsequent configuration For the updated Check out the build The time phase corresponds to the submatrix , and for v l The successor edge set of The corresponding edge in The attribute matrix Update, that is, replace with the updated The corresponding matrix sub-block in
[0236] Step S305, set the update flag of all the successor edges of the cleanable node to the second preset value, update the forward queue to be cleaned, and return to execute the step of sequentially determining whether the nodes in the forward queue to be cleaned are cleanable nodes until the determined node is the output node of the DCNN network.
[0237] Specifically, update the flag set for the successor edge All elements are set to 1, as follows:
[0238]
[0239] Find the successor node subset in the queue to be cleaned Q All nodes in j=1,2,…N A ,like If it is not in queue Q, Add to the end of queue Q.
[0240] Go to step S303.
[0241] Step S306, forward cleaning ends.
[0242] At this time, if the queue to be cleaned Q contains only the output nodes of the DCNN network Forward cleaning is completed.
[0243] Step S4, according to the supporting node connection topology subgraph corresponding to each node in the DCNN full network connection topology graph, the DCNN full network connection topology graph is back-cleaned.
[0244] Fig.11 is a backward cleaning flow chart of the pruned DCNN network provided by an embodiment of the present invention, such as Fig.11 As shown, the process includes the following steps:
[0245] Step S401, setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value.
[0246] The method for resetting the update flags of all edges in the graph G is the same as step S301 and will not be repeated here.
[0247] Step S402, constructing a backward queue to be cleaned according to the leading node of the output node of the DCNN network, and setting the update flag of the leading edge of the output node to a second preset value.
[0248] Establish a queue Q of supporting nodes to be cleaned.
[0249] In the node set V of graph G, take the node corresponding to the output channel group of the DCNN network The supporting node connection topology subgraph
[0250]
[0251] Will The leading node subset of All nodes in are put into queue Q in turn, and the leading edge is updated with the flag set All update flags in are set to 1, that is,
[0252] Step S403 , judging in sequence whether the nodes in the backward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the successor edge update flag set of the node.
[0253] Take out the frontmost supporting node in the queue Q. Assume it is v l , and take v from graph G l The corresponding subgraph G l , to determine v l Whether it is a cleanable node. Delete node v in queue Ql .
[0254] Specifically, step S403 includes: sequentially determining whether all update flags in the successor edge update flag set of each node in the backward to-be-cleaned queue are second preset values;
[0255] If yes, the corresponding node is determined as a cleanable node.
[0256] Pair of subgraphs G l To judge the status, check subgraph G l The successor edge update flag set All Are all values set to 1?
[0257] like Existence Indicates G l If the attribute matrix of the successor edge of v has not been updated yet, l Put it at the end of the queue Q to be cleaned and go to the beginning of step S403.
[0258] like All And l≠0, the current supporting node can be cleaned, and the process goes to step S404.
[0259] like All And l = 0, then node v 0 is the input node v of DCNN 0 , go to step S406.
[0260] Step S404: based on the predecessor configuration and successor configuration of the cleanable node and the attribute matrices of each predecessor edge and successor edge, the attribute matrix of each predecessor edge of the cleanable node is updated.
[0261] Specifically, Fig.12 is a backward cleaning flow chart of the support node connection topology subgraph of the pruned DCNN network provided by an embodiment of the present invention, such as Fig.12 As shown, the process includes the following steps:
[0262] Step S40401, constructing a predecessor stacking matrix and a successor stacking matrix of the cleanable node according to the predecessor configuration and successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge.
[0263] Leading stacking matrix With the successor stacking matrix The construction method is the same as step S30401 and will not be repeated here.
[0264] Step S40402, based on the successor stacking matrix and the preset operation formula, establish a channel validity successor indication matrix.
[0265] make:
[0266]
[0267] yes A row vector with all elements set to 1, v l The backward stacking matrix for:
[0268]
[0269] v l The channel validity successor indicator matrix Is and Do matrix logical multiplication The diagonal matrix formed by The definition is as follows:
[0270]
[0271] in,
[0272]
[0273] is node v l The number of channels, is a matrix the number of rows, yes The element in row n and column m. When the mth column of is all 0, The value of node v is 0, that is, l The mth channel is an invalid channel.
[0274] v l The channel validity successor indicator matrix is a Diagonal logical matrix, the values of the elements on the off-diagonal are all 0, and the elements with a value of 1 on the diagonal represent the node v l The corresponding channel in is a valid channel; otherwise, the element with value 0 is the node v l The corresponding channel is an invalid channel.
[0275] Step S40403, clean the predecessor stacking matrix based on the channel validity successor indication matrix to obtain an updated attribute matrix of each predecessor edge of the cleanable node.
[0276] v l The channel validity successor indicator matrix Indicates the node v 1 The valid channels and invalid channels of the subsequent indicator matrix Stacking matrices with leading Do matrix logical multiplication And reassign the result to The operation is as follows:
[0277]
[0278] v l The channel validity successor indicator matrix The diagonal elements of represent v l The channel validity in . The diagonal element value of v is 1, then l The channel corresponding to the element is valid; otherwise, if the element value is 0, the corresponding channel is invalid. Matrix logical left multiplication Then The rows whose diagonal elements are 0 correspond to The row elements in are all set to 0, while keeping The rows whose diagonal elements are 1 correspond to The row elements in remain unchanged, so cleaning removes Zhong and v l Possible connection tags associated with the invalid channel.
[0279] According to v l The leading configuration For the updated Check out the build The time phase corresponds to the submatrix , and for v l The leading edge set The corresponding edge in The attribute matrix Update, that is, replace with the updated The corresponding matrix sub-block in
[0280] Step S405, setting the update flag of all leading edges of the cleanable node to a second preset value, updating the backward queue to be cleaned, and returning to execute the step of sequentially determining whether the nodes in the backward queue to be cleaned are cleanable nodes until the determined node is an input node of the DCNN network.
[0281] Specifically, update the flag set for the leading edge All elements are set to 1, as follows:
[0282]
[0283] Find the leading node subset in the queue to be cleaned Q All nodes in i=1,2,…N B ,like If it is not in queue Q, Add to the end of queue Q.
[0284] Go to step S403.
[0285] Step S406, the backward cleaning is completed.
[0286] At this time, if the queue to be cleaned Q contains only the input node v of the DCNN network 0 , the backward cleaning is completed.
[0287] It should be noted that the order of the above steps S3 and S4 is not limited. In some specific cases, the DCNN network can also be cleaned only in the forward direction or in the backward direction, which can also play the role of cleaning the redundant structure of the DCNN network. Preferably, forward cleaning and backward cleaning of the DCNN network can have the best cleaning effect.
[0288] Step S5, generating a simplified DCNN model based on the cleaned DCNN full network connection topology graph.
[0289] The DCNN full network connection topology diagram G in the present invention is as follows:
[0290] G={V,E,G l , l=0,1,…,N T}
[0291] Node v in graph G l ∈V is a set of channels (feature maps) in the DCNN network. In a single-branch DCNN network, v l is the entirety of the input or output channels of a convolutional layer; in a multi-branch DCNN network, v l It can also be represented as a set of convolution input or output channels that are aggregated or distributed by multiple branches. i,j :v i →v j , e i,j ∈E represents two different channel groups v i With v j There is a direct signal transmission relationship between them, and the attribute matrix K of the edge i,j reflects two different channel groups v i With v j The convolution kernel in the filter is used as a bridge to connect the internal channels. The supporting node connection topology subgraph G 1 Describes the node vl The local connection relationship of DCNN centered on .
[0292] In the present invention, for the unpruned DCNN network, the pruned DCNN network and the pruned and cleaned DCNN network of the present invention, the node set V, the edge set E and the subgraph G of the full network connection topology graph G of the three are l The only difference between the graph G before and after pruning and before and after cleaning is that each edge e in the edge set E i,j The attribute matrix of the channel topology matrix K i,j different.
[0293] K without structured pruning i,j is a matrix with all elements equal to 1; K after structured pruning i,j It is a 0 / 1 logic matrix with some elements being 1 and the rest being 0. Since the feature signals of adjacent convolutional layers are interrupted after structured pruning, redundant connection marks are formed. i,j is the channel topology matrix with redundant connection flags cleared to 0.
[0294] After cleaning, K i,j Only node v is retained i With node v j The effective channel connection marks between the two channels are cleared, and the redundant channel connection marks are cleared. i,j A simplified DCNN model with an equivalent mapping relationship to the original pruned DCNN network is constructed, so as to better compress the parameters of the DCNN network and reduce the computational load of the DCNN network, so that it can be more effectively deployed on edge devices with limited resources.
[0295] There are many ways to use the cleaned channel topology matrix to generate a streamlined DCNN network model. Fig.13 This is an intuitive mapping flow chart for generating a simplified DCNN model using a cleaned channel topology matrix provided by an embodiment of the present invention, such as Fig.13 As shown, the process includes the following steps:
[0296] Step S501, simplify the attribute matrix corresponding to each directed edge in the cleaned DCNN full network connection topology graph to obtain a simplified channel topology matrix.
[0297] Assume that node v i The number of channels is Node v j The number of channels is Then the channel topology matrix K i,j is a Row× Matrix of columns.
[0298] Delete K i,j All 0 rows and all 0 columns are obtained to obtain the simplified channel topology matrix Assumptions For one Row× Column matrix. Record The rows and columns in the original matrix K i,j The row and column numbers in .
[0299] Step S502: retaining valid channels in each node according to the simplified channel topology matrix.
[0300] Keep node v i Zhong and K i,j The channel corresponding to the non-0 column, that is, the channel recorded in step S501 is retained The channel corresponding to the column number; retain the node v j Zhong and K i,j The channel corresponding to the non-0 row, that is, the channel recorded in step S501 is retained The channel corresponding to the row number.
[0301] Step S503: According to the simplified channel topology matrix, a simplified convolutional layer is established and parameters are initialized.
[0302] Each line corresponds to a filter, there are filters, the number of kernels in each filter is initialization The corresponding simplified convolutional layer fills the parameters of each kernel in the filter corresponding to each row with all 0s.
[0303] Step S504, fill in the effective convolution kernel parameters of the simplified convolution layer, and establish a simplified DCNN model in combination with the effective channels in each node.
[0304] Fill in the effective kernel parameters of each filter in the streamlined convolutional layer. Specifically, for each filter corresponding to Each element in the row vector of is judged. If the current element value is 1, it means that the corresponding kernel in the filter is a valid convolution kernel. Get the current element in the original channel topology matrix K i,j The row and column numbers in , and the corresponding kernel parameters of the corresponding filter in the DCNN pre-trained model corresponding to the row and column numbers are used to fill the corresponding kernel of the corresponding filter in the streamlined convolutional layer.
[0305] Step S505, establishing a simplified DCNN model.
[0306] The above steps are used to reconstruct the convolution structure of each convolution layer or convolution layer branch in the DCNN network, thereby generating a streamlined DCNN model, which has the same performance as the pre-trained pruned model.
[0307] The present invention proposes a cleaning and optimization method for a structured pruned deep convolutional neural network (DCNN). First, the DCNN network model is fully pre-trained and pruned with the convolution kernel as the minimum granularity to obtain a pruned DCNN model. Secondly, the input and output channels of the convolution layer in the pruned DCNN model are used as graph nodes to construct a DCNN full-network connection topology graph model. The pruned label matrix of the convolution layer is used as the edge attribute matrix of the DCNN full-network connection topology graph. The convolution layer pruned label matrix as the edge attribute matrix is cleaned with the help of the full-network connection topology graph, thereby deleting the redundant structure of the DCNN model corresponding to the redundant labels with discontinuous signal transmission in the pruned label matrix, and realizing the re-optimization of the pruned DCNN model structure. Finally, a more streamlined DCNN model with a higher compression rate is constructed using the cleaned and optimized convolution layer pruned label matrix, making it easier to deploy on resource-constrained edge devices.
[0308] The above method is applicable to many common neural networks, such as VGG, Unet, ResNet, GoogLeNet, DenseNet, etc.
[0309] In this embodiment, a cleaning optimization system for a deep convolutional neural network after structured pruning is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0310] This embodiment provides a cleaning optimization system for a deep convolutional neural network after structured pruning, such as Fig.14 As shown, including:
[0311] The pruning and marking module 601 is used to obtain a DCNN model and perform structured pruning and marking on the redundant structure of the DCNN network of the model;
[0312] A topology map generation module 602 is used to construct a corresponding DCNN full network connection topology map based on the connection relationship between each node in the DCNN network after pruning and marking;
[0313] A topology cleaning module 603 is used to perform forward cleaning and backward cleaning on the DCNN full network connection topology map according to the supporting node connection topology subgraphs corresponding to each node in the DCNN full network connection topology map;
[0314] The simplified model generation module 604 is used to generate a simplified DCNN model based on the cleaned DCNN full network connection topology graph.
[0315] In some optional implementations, the topology map generation module 602, when constructing a corresponding DCNN full network connection topology map based on the connection relationship between each node in the DCNN network after pruning marking, includes:
[0316] Determine all nodes in the DCNN network and generate a node set;
[0317] According to the directed edges between each node, the edge set of the DCNN network is constructed, and each directed edge in the edge set is associated with the corresponding attribute matrix;
[0318] Based on the directed edges and connection modes between each node and the corresponding predecessor node and successor node, a supporting node connection topology subgraph corresponding to each node is constructed;
[0319] Based on the node set, edge set and supporting node connection topology subgraph corresponding to each node, a DCNN full-network connection topology graph is constructed.
[0320] In some optional implementations, the supporting node connection topology subgraph includes: a predecessor configuration, a successor configuration, a predecessor node set, a successor node set, a predecessor edge set and a corresponding predecessor edge update flag set, a successor edge set and a corresponding successor edge update flag set, and each update flag in the update flag set is used to indicate whether the attribute matrix of the corresponding edge can be updated;
[0321] The topology cleaning module 603 performs forward cleaning on the DCNN full network connection topology map according to the supporting node connection topology subgraphs corresponding to each node in the DCNN full network connection topology map, including:
[0322] Setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value;
[0323] Construct a forward queue to be cleaned according to the successor node of the input node of the DCNN network, and set the update flag of the successor edge of the input node to a second preset value;
[0324] According to the values of each update flag in the node's leading edge update flag set, determine in turn whether the node in the forward to-be-cleaned queue is a cleanable node;
[0325] If yes, based on the predecessor configuration and successor configuration of the cleanable node and the attribute matrices of each predecessor edge and successor edge, the attribute matrix of each successor edge of the cleanable node is updated;
[0326] The update flags of all the successor edges of the cleanable node are set to the second preset value, the forward queue to be cleaned is updated, and the step of sequentially determining whether the nodes in the forward queue to be cleaned are cleanable nodes is returned to be executed until the determined node is the output node of the DCNN network.
[0327] In some optional implementations, the topology cleaning module 603 performs backward cleaning on the DCNN full network connection topology map according to the supporting node connection topology subgraphs corresponding to each node in the DCNN full network connection topology map, including:
[0328] Setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value;
[0329] Construct a backward queue to be cleaned according to the leading node of the output node of the DCNN network, and set the update flag of the leading edge of the output node to a second preset value;
[0330] According to the value of each update flag in the node's successor edge update flag set, determine in turn whether the node in the backward to-be-cleaned queue is a cleanable node;
[0331] If yes, based on the predecessor configuration and successor configuration of the cleanable node and the attribute matrices of each predecessor edge and successor edge, the attribute matrix of each predecessor edge of the cleanable node is updated;
[0332] The update flags of all leading edges of the cleanable nodes are set to the second preset value, the backward queue to be cleaned is updated, and the step of sequentially determining whether the nodes in the backward queue to be cleaned are cleanable nodes is returned to be executed until the determined node is an input node of the DCNN network.
[0333] In some optional implementations, the topology cleaning module 603 updates the attribute matrix of each successor edge of the cleanable node based on the predecessor configuration, successor configuration and attribute matrix of each predecessor edge and successor edge of the cleanable node, including:
[0334] According to the predecessor configuration and successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge, a predecessor stacking matrix and a successor stacking matrix of the cleanable node are constructed respectively;
[0335] Based on the leader stacking matrix and the preset operation formula, a channel validity leader indication matrix is established;
[0336] The successor stacking matrix is cleaned based on the channel validity predecessor indicator matrix to obtain an updated attribute matrix of each successor edge of the cleanable node.
[0337] In some optional implementations, the topology cleaning module 603 updates the attribute matrix of each leading edge of the cleanable node based on the leading configuration, the successor configuration and the attribute matrix of each leading edge and the successor edge of the cleanable node, including:
[0338] According to the predecessor configuration and successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge, a predecessor stacking matrix and a successor stacking matrix of the cleanable node are constructed respectively;
[0339] Based on the successor stacking matrix and the preset operation formula, a channel validity successor indication matrix is established;
[0340] The predecessor stacking matrix is cleaned based on the channel validity successor indication matrix to obtain an updated attribute matrix of each predecessor edge of the cleanable node.
[0341] In some optional implementations, the topology cleaning module 603 determines in sequence whether the nodes in the forward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the leading edge update flag set of the nodes, including:
[0342] Determine in sequence whether all update flags in the leading edge update flag set of each node in the forward queue to be cleaned are second preset values;
[0343] If yes, the corresponding node is determined as a cleanable node;
[0344] Alternatively, the topology cleaning module 603 determines in sequence whether the nodes in the backward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the successor edge update flag set of the node, including:
[0345] Determine in sequence whether all update flags in the successor edge update flag set of each node in the backward to-be-cleaned queue are second preset values;
[0346] If yes, the corresponding node is determined as a cleanable node.
[0347] In some optional implementations, the simplified model generation module 604, when generating a simplified DCNN model based on the cleaned DCNN full network connection topology graph, includes:
[0348] The attribute matrix corresponding to each directed edge in the cleaned DCNN full network connection topology graph is simplified to obtain a simplified channel topology matrix;
[0349] According to the simplified channel topology matrix, the valid channels in each node are retained;
[0350] According to the streamlined channel topology matrix, a streamlined convolution layer is established and parameters are initialized;
[0351] Fill in the effective convolution kernel parameters of the streamlined convolution layer, and build a streamlined DCNN model based on the effective channels in each node.
[0352] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0353] The cleaning optimization system of the structured pruned deep convolutional neural network in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.
[0354] The embodiment of the present invention also provides a computer device having the above Fig.14 The cleaning optimization system of the deep convolutional neural network after structured pruning shown.
[0355] See also Fig.15 , Fig.15 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.15 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.15 A processor 10 is taken as an example.
[0356] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0357] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0358] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0359] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0360] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.15 The example of connecting through bus is taken in the following.
[0361] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0362] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0363] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A cleaning and optimization method for a deep convolutional neural network after structured pruning, characterized in that: The method comprises: Obtaining a DCNN model, and performing structured pruning marking on a redundant structure of a DCNN network of the model; Based on the connection relationship between each node in the DCNN network after pruning and marking, the corresponding DCNN full network connection topology graph is constructed; According to the supporting node connection topology subgraphs corresponding to each node in the DCNN full network connection topology graph, forward cleaning and backward cleaning are performed on the DCNN full network connection topology graph respectively; A simplified DCNN model is generated based on the cleaned DCNN full-network connection topology graph.
2. The method according to claim 1, characterized in that The connection relationship between each node in the DCNN network after pruning marking is constructed to construct a corresponding DCNN full network connection topology diagram, including: Determine all nodes in the DCNN network and generate a node set; According to the directed edges between the nodes, an edge set of the DCNN network is constructed, and each directed edge in the edge set is associated with a corresponding attribute matrix; Based on the directed edges and connection modes between each node and the corresponding predecessor node and successor node, a supporting node connection topology subgraph corresponding to each node is constructed; Based on the node set, edge set and supporting node connection topology subgraphs corresponding to each node, a DCNN full-network connection topology graph is constructed.
3. The method according to claim 1, characterized in that: The supporting node connection topology subgraph includes: a predecessor configuration, a successor configuration, a predecessor node set, a successor node set, a predecessor edge set and a corresponding predecessor edge update flag set, a successor edge set and a corresponding successor edge update flag set, and each update flag in the update flag set is used to indicate whether the attribute matrix of the corresponding edge can be updated; According to the supporting node connection topology subgraph corresponding to each node in the DCNN full network connection topology graph, forward cleaning of the DCNN full network connection topology graph comprises: Setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value; Constructing a forward queue to be cleaned according to the successor node of the input node of the DCNN network, and setting the update flag of the successor edge of the input node to a second preset value; According to the values of each update flag in the node's leading edge update flag set, determine in turn whether the node in the forward to-be-cleaned queue is a cleanable node; If yes, based on the predecessor configuration and successor configuration of the cleanable node and the attribute matrices of each predecessor edge and successor edge, the attribute matrix of each successor edge of the cleanable node is updated; The update flags of all the successor edges of the cleanable node are set to a second preset value, the forward queue to be cleaned is updated, and the step of sequentially determining whether the nodes in the forward queue to be cleaned are cleanable nodes is returned to be executed until the determined node is an output node of the DCNN network.
4. The method according to claim 3, characterized in that According to the supporting node connection topology subgraph corresponding to each node in the DCNN full network connection topology graph, the DCNN full network connection topology graph is back-cleaned, including: Setting the update flags of all edges in the DCNN full network connection topology graph to a first preset value; Construct a backward queue to be cleaned according to the leading node of the output node of the DCNN network, and set the update flag of the leading edge of the output node to a second preset value; According to the value of each update flag in the node's successor edge update flag set, determine in turn whether the node in the backward to-be-cleaned queue is a cleanable node; If yes, based on the predecessor configuration and successor configuration of the cleanable node and the attribute matrices of each predecessor edge and successor edge, the attribute matrix of each predecessor edge of the cleanable node is updated; The update flags of all leading edges of the cleanable node are set to a second preset value, the backward queue to be cleaned is updated, and the steps of sequentially determining whether the nodes in the backward queue to be cleaned are cleanable nodes are returned to be executed until the determined node is an input node of the DCNN network.
5. The method according to claim 3, characterized in that: The updating of the attribute matrix of each successor edge of the cleanable node based on the predecessor configuration, the successor configuration and the attribute matrix of each predecessor edge and the successor edge of the cleanable node comprises: According to the predecessor configuration and the successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge, respectively construct a predecessor stacking matrix and a successor stacking matrix of the cleanable node; Based on the leader stacking matrix and a preset operation formula, a channel validity leader indication matrix is established; The successor stacking matrix is cleaned based on the channel validity predecessor indication matrix to obtain an updated attribute matrix of each successor edge of the cleanable node.
6. The method according to claim 4, characterized in that The updating of the attribute matrix of each leading edge of the cleanable node based on the leading configuration, the following configuration and the attribute matrix of each leading edge and the following edge of the cleanable node comprises: According to the predecessor configuration and the successor configuration of the cleanable node and the attribute matrix of each predecessor edge and successor edge, respectively construct a predecessor stacking matrix and a successor stacking matrix of the cleanable node; Based on the successor stacking matrix and the preset operation formula, a channel validity successor indication matrix is established; The predecessor stacking matrix is cleaned based on the channel validity successor indication matrix to obtain an updated attribute matrix of each predecessor edge of the cleanable node.
7. The method according to claim 3 or claim 4, characterized in that: The step of sequentially judging whether the nodes in the forward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the leading edge update flag set of the node includes: Determine in sequence whether all update flags in the leading edge update flag set of each node in the forward queue to be cleaned are second preset values; If yes, the corresponding node is determined as a cleanable node; Alternatively, judging in sequence whether the nodes in the backward to-be-cleaned queue are cleanable nodes according to the values of the update flags in the successor edge update flag set of the node includes: Determine in sequence whether all update flags in the successor edge update flag set of each node in the backward to-be-cleaned queue are second preset values; If yes, the corresponding node is determined as a cleanable node.
8. The method according to claim 1, characterized in that The method of generating a simplified DCNN model based on the cleaned DCNN full network connection topology graph includes: The attribute matrix corresponding to each directed edge in the cleaned DCNN full network connection topology graph is simplified to obtain a simplified channel topology matrix; According to the simplified channel topology matrix, retaining valid channels in each node; According to the simplified channel topology matrix, a simplified convolutional layer is established and parameters are initialized; Fill in the effective convolution kernel parameters of the streamlined convolution layer, and build a streamlined DCNN model based on the effective channels in each node.
9. A cleaning and optimization system for a deep convolutional neural network after structured pruning, characterized in that: The system comprises: A pruning and marking module is used to obtain a DCNN model and perform structured pruning and marking on the redundant structure of the DCNN network of the model; A topology map generation module is used to construct a corresponding DCNN full network connection topology map based on the connection relationship between each node in the DCNN network after pruning and marking; A topology cleaning module, for performing forward cleaning and backward cleaning on the DCNN full network connection topology map according to the supporting node connection topology subgraphs corresponding to each node in the DCNN full network connection topology map; The simplified model generation module is used to generate a simplified DCNN model based on the cleaned DCNN full network connection topology graph.
10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 8 by executing the computer instructions.
