Network structure optimization method and device for pruned convolutional neural network

Abstract

The invention relates to the technical field of deep learning, and discloses a network structure optimization method and device for a pruned convolutional neural network, and the method comprises the steps: obtaining an initial convolutional neural network model, and carrying out the pre-training, and obtaining a pre-training model; pruning each convolution layer of the pre-training model, and constructing a pruning mark matrix of the convolution layer based on a pruning mark of each convolution kernel of the convolution layer; clustering the plurality of filters in the convolutional layer based on the pruning mark matrix of the convolutional layer to obtain a target clustering matrix; and performing simplified mapping on the pre-training model based on the pruning mark matrix and the target clustering matrix of each convolutional layer to obtain a target convolutional neural network. According to the method, clustering is carried out by utilizing the similarity of effective convolution kernel distribution between filters, redundant parameters are reduced, the compression efficiency of the model is improved, the model is subjected to simplified mapping, the model structure is effectively compressed on the premise of keeping the model precision, and the calculation cost and the storage requirement of the model are reduced.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for optimizing the network structure of a pruned convolutional neural network. Background Art

[0002] Deep Convolutional Neural Networks (DCNNs) have been widely used in the field of computer vision and have achieved remarkable results. To further improve network performance, the number of convolutional layers in DCNNs has continued to increase, while the network size has also continued to expand. This has led to a sharp increase in storage and computing resource usage, severely restricting its deployment on resource-constrained edge devices. Therefore, structural compression optimization of DCNNs is crucial. This technology aims to significantly compress the network size, reduce floating-point operations and memory usage, while ensuring manageable network accuracy loss, thereby expanding the application scenarios of DCNNs.

[0003] Network pruning is one of the main techniques for DCNN structural compression optimization. It achieves compression by discarding useless parameters in the network, effectively reducing computational complexity and memory usage, and accelerating the model's inference process. DCNN network pruning is mainly divided into two categories: unstructured pruning and structured pruning. Structured pruning methods include convolution kernel pruning. In a DCNN, each convolutional layer consists of several filters, the number of which is consistent with the number of output channels (output feature maps) of the layer. Each filter contains a number of convolution kernels equal to the number of input channels (input feature maps). Specifically, a filter performs a convolution operation with the corresponding input channel feature map through its internal convolution kernel to generate a new feature map. After superposition and nonlinear activation, the output channel feature map corresponding to the filter is formed. Convolution kernel pruning uses a single convolution kernel as the smallest unit. Pruning a convolution kernel means removing the input channel mapping information transmitted by the kernel in the output channel of the filter where the kernel is located.

[0004] Existing DCNN convolution kernel pruning methods typically only soft-prune the convolution kernels of each convolutional layer. This means they don't actually remove the pruned kernels, but instead set their weights to zero. This pruning approach results in a large number of redundant zero-weight convolution kernels in the network structure. Therefore, optimizing the pruned DCNN structure to reduce computational resource waste and achieve effective DCNN compression remains a challenge. Summary of the Invention

[0005] In view of this, the present invention provides a network structure optimization method and device for a pruned convolutional neural network to solve the problem of structural optimization of the pruned convolutional neural network.

[0006] In a first aspect, the present invention provides a method for optimizing the network structure of a pruned convolutional neural network, the method comprising:

[0007] Obtain an initial convolutional neural network model, and pre-train the initial convolutional neural network model to obtain a pre-trained model;

[0008] Prune each convolutional layer of the pre-trained model and construct a pruning mark matrix for the convolutional layer based on the pruning mark of each convolution kernel in the convolutional layer. The pruning mark is used to indicate whether the convolution kernel is a valid convolution kernel to be retained or an invalid convolution kernel to be pruned.

[0009] Cluster multiple filters in the convolutional layer based on the pruned label matrix of the convolutional layer to obtain the target clustering matrix;

[0010] The pre-trained model is streamlined and mapped based on the pruned label matrix and target clustering matrix of each convolutional layer to obtain the target convolutional neural network.

[0011] The present invention pre-trains the initial convolutional neural network model to ensure that the model performance reaches the required accuracy, and prunes the convolutional layer of the pre-trained model to construct a pruned label matrix to accurately identify the effectiveness of each convolution kernel. The filters are clustered based on the pruned label matrix to obtain a target clustering matrix. The similarity of the effective convolution kernel distribution between filters is utilized to classify similar filters into one category, which can more deeply mine the redundant information in the network, further reduce redundant parameters, and improve the compression efficiency of the model. Finally, the pre-trained model is streamlined and mapped based on the pruned label matrix and the target clustering matrix. While maintaining the model accuracy, the model structure is effectively compressed, the computational cost and storage requirements of the model are greatly reduced, and the optimized target convolutional neural network structure is more compact and efficient, and easier to deploy and apply on resource-constrained edge devices.

[0012] In an optional embodiment, multiple filters in the convolutional layer are clustered based on the pruned labeling matrix of the convolutional layer to obtain a target clustering matrix, including:

[0013] For each filter in the convolutional layer, construct a filter sample of the filter based on the pruning mark of the convolution kernel included in the filter in the pruning mark matrix;

[0014] Determine the proportion of invalid convolution kernels in each filter sample, and take the filter samples whose proportion exceeds a preset proportion threshold as sparse samples;

[0015] Taking any sparse sample as the initial sample in the preset classification set, and obtaining the target classification set based on the distance between the preset classification set and other filter samples;

[0016] Each filter sample in the target classification set is clustered as a class center sample to obtain multiple initial clustering results;

[0017] An interactive benefit optimization clustering algorithm is used to optimize multiple initial clustering results and obtain the target clustering matrix.

[0018] The present invention can quickly identify filter samples with high redundancy by determining the proportion of invalid convolution kernels and screening sparse samples, providing a reasonable basis for the selection of initial samples, and determining the target classification set based on the distance between the preset classification set and other samples, so that the initial classification set can cover different feature distributions as much as possible, ensuring the comprehensiveness and rationality of clustering, clustering the samples in the target classification set as class centers, and making full use of the characteristics of these representative samples to obtain multiple initial clustering results. The interactive benefit optimization clustering algorithm is used to optimize the initial clustering results, and the attribution of samples can be continuously adjusted and optimized during the clustering process, thereby obtaining a more accurate and optimized target clustering matrix, further reducing redundancy and improving the compression effect and efficiency of the model.

[0019] In an optional embodiment, any sparse sample is used as an initial sample in a preset classification set, and a target classification set is obtained based on the distance between the preset classification set and other filter samples, including:

[0020] For each filter sample among the other filter samples, determining a distance between the filter sample and a preset classification set;

[0021] Add the filter sample with the largest distance to the preset classification set;

[0022] The sum of all filter samples in the preset classification set is used as the distribution vector;

[0023] Determine whether the number of non-zero elements of the distribution vector is equal to the total number of convolution kernels in the convolution layer;

[0024] When the number of non-zero elements of the distribution vector is not equal to the total number of convolution kernels in the convolution layer, return to the step of determining the distance between the filter sample and the preset classification set for each filter sample in other filter samples until the number of non-zero elements of the distribution vector is equal to the total number of convolution kernels in the convolution layer, and take the preset classification set obtained last time as the target classification set.

[0025] By selecting initial samples from sparse samples, the present invention can make the initial classification set representative from the beginning. By continuously calculating the distance between other filter samples and the preset classification set and adding the samples with the largest distance, the classification set can gradually cover more filter samples with different feature distributions, so that the target classification set can more comprehensively reflect the characteristics of the filters in the entire convolution layer. Whether the number of non-zero elements of the distribution vector is equal to the total number of convolution kernels is used as the stopping condition, which can ensure that the effective convolution kernel information contained in the samples in the target classification set can cover all convolution kernel dimensions of the entire convolution layer.

[0026] In an optional embodiment, an interactive benefit optimization clustering algorithm is used to optimize multiple initial clustering results to obtain a target clustering matrix, including:

[0027] Based on the filter samples in the multiple initial clustering results, constructing an initial clustering matrix of the multiple initial clustering results, the initial clustering matrix is ​​used to indicate the belonging class of the filter samples;

[0028] For each initial clustering result, determine the benefit obtained by removing each filter sample in the initial clustering result;

[0029] The filter sample with the largest benefit in each initial clustering result is removed from the initial clustering result, the filter sample removed from each initial clustering result is added to the set of samples to be classified, and the initial clustering matrix is ​​updated;

[0030] For each filter sample in the set of samples to be classified, determine the loss obtained by adding the filter sample to each initial clustering result;

[0031] Add the filter sample with the smallest loss in each initial clustering result to the initial clustering result, remove the filter sample added to each initial clustering result from the set of samples to be classified, and update the initial clustering matrix;

[0032] Merge all initial clustering results to obtain new initial clustering results, update the initial clustering matrix, and obtain the intermediate clustering matrix;

[0033] Determine whether the binary norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​less than a preset convergence threshold;

[0034] When the binary norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​not less than a preset convergence threshold, the intermediate clustering matrix is ​​used as a new initial clustering matrix, and the process returns to the step of constructing the initial clustering matrix of the multiple initial clustering results based on the filter samples in the multiple initial clustering results, until the binary norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​less than the preset convergence threshold, and the intermediate clustering matrix obtained by the last update is used as the target clustering matrix.

[0035] The present invention uses a bidirectional iterative gain-loss optimization mechanism to dynamically adjust sample attribution and merge similar clusters, and uses the two-norm convergence judgment to ensure optimization accuracy. It can not only accurately purify clusters and reduce redundant parameters, but also retain the distribution of key features, maintaining accuracy while achieving efficient model compression and acceleration.

[0036] In an optional embodiment, a pre-trained model is streamlined and mapped based on the pruned label matrix and the target clustering matrix of each convolutional layer to obtain a target convolutional neural network, including:

[0037] All input channels in the pre-trained model are used as input nodes, all output channels are used as output nodes, and each convolutional layer is used as a support node to construct a node set of the pre-trained model;

[0038] For any two nodes in the node set, when there is a convolution relationship between the two nodes, determine whether there is a directed edge between the two nodes, and build a queue to be subnetted based on all directed edges;

[0039] For each directed edge in the queue to be subnetted, multiple subnet topology matrices are constructed based on the pruned label matrix of the convolution output node and the target clustering matrix. The convolution output node represents the node located in the output direction of the directed edge.

[0040] Eliminate the all-zero columns in each subnet topology matrix to obtain the corresponding subnet simplified matrix;

[0041] Initialize each subnet simplified matrix to obtain the corresponding subnet initialization matrix;

[0042] For each subnet initialization matrix, extract the parameters of the effective convolution kernel at the corresponding position from the pruning mark matrix of the convolution output node, fill it into the subnet initialization matrix, and obtain the subnet parameter matrix;

[0043] Perform input-output channel mapping on each subnet parameter matrix to obtain a simplified mapping layer corresponding to the directed edge. Remove the directed edge from the queue to be subnetted and determine whether the queue to be subnetted is empty.

[0044] When the queue to be subnetted is not empty, the next directed edge is continued to be streamlined and mapped until the queue to be subnetted is empty, and the target convolutional neural network is obtained based on the streamlined mapping layers corresponding to all directed edges.

[0045] The present invention constructs a subnetwork topology matrix and removes all-zero columns therein, thereby reducing the number of invalid convolution kernels involved in the calculation, greatly reducing the total number of model parameters and computational complexity. By initializing the subnetwork streamlined matrix and extracting effective convolution kernel parameters from the original pruned labeling matrix for filling, the key feature mapping relationship of the original model is retained, and redundant parameters are eliminated through clustering optimization. Finally, by mapping the input and output channels separately, it is ensured that only the convolution structure is changed without changing the size and order of the input and output channels, thereby ensuring that the final output accuracy of the network is not affected.

[0046] In an optional embodiment, for each directed edge in the queue to be subnetted, multiple subnet topology matrices are constructed based on the pruning label matrix of the convolution output node and the target clustering matrix, including:

[0047] Extract the column index corresponding to the non-zero element from each row of the target clustering matrix of the convolution output node;

[0048] The row corresponding to each column index is extracted from the pruned label matrix of the convolution output node to obtain multiple subnetwork topology matrices.

[0049] The present invention constructs a subnetwork topology matrix through the collaborative processing of the target clustering matrix and the pruning labeling matrix, so that each subnetwork only contains parameters related to a specific cluster, which greatly reduces the computational complexity and the number of parameters while maintaining the model accuracy.

[0050] In an optional implementation, input-output channel mapping is performed on each subnetwork parameter matrix to obtain a simplified mapping layer corresponding to the directed edges, including:

[0051] Determine all channels of the convolution input node to form a total channel vector. The convolution output node represents the node located in the input direction of the directed edge.

[0052] Determine the input channel vector based on the correspondence between each subnetwork parameter matrix and all channels;

[0053] For each subnetwork parameter matrix, perform element-by-element multiplication of the total channel vector and the input channel vector to obtain the channel inflow vector;

[0054] Determine the output channel vector of each subnetwork parameter matrix;

[0055] Determine the reduction matrix of the subnetwork parameter matrix based on the corresponding column index of the subnetwork parameter matrix in the target clustering matrix;

[0056] Multiply the output channel vector by the restoration matrix to obtain the channel restoration vector of the subnetwork parameter matrix;

[0057] The sum of the channel restoration vectors of all subnetwork parameter matrices is determined as the output channel vector at the convolution output node;

[0058] Based on the output channel vector and the channel inflow vector of each subnetwork parameter matrix, a simplified mapping layer corresponding to the directed edge is obtained.

[0059] The present invention realizes the precise diversion of the original input channels through input channel mapping, so that each subnet obtains the input channels it needs. The output channel mapping realizes the organic combination of input channel mapping and output channel restoration, ensuring the equivalent transformation of the entire convolution structure and realizing efficient simplification and optimization of the network.

[0060] In a second aspect, the present invention provides a network structure optimization device for a pruned convolutional neural network, the device comprising:

[0061] A pre-training module is used to obtain an initial convolutional neural network model and pre-train the initial convolutional neural network model to obtain a pre-trained model;

[0062] The pruning module is used to prune each convolutional layer of the pre-trained model and construct a pruning mark matrix for the convolutional layer based on the pruning mark of each convolution kernel in the convolutional layer. The pruning mark is used to indicate whether the convolution kernel is a valid convolution kernel to be retained or an invalid convolution kernel to be pruned;

[0063] A clustering module is used to cluster multiple filters in the convolutional layer based on the pruned label matrix of the convolutional layer to obtain a target clustering matrix;

[0064] The optimization module is used to simplify the mapping of the pre-trained model based on the pruned label matrix and target clustering matrix of each convolutional layer to obtain the target convolutional neural network.

[0065] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the network structure optimization method of the pruned convolutional neural network according to the first aspect or any corresponding embodiment thereof.

[0066] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the network structure optimization method for a pruned convolutional neural network according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 is a diagram illustrating a structure of a deep convolutional neural network according to an embodiment of the present invention;

[0069] Figure 2 is a schematic diagram of a convolution kernel, a convolution kernel set, and a filter in a convolution layer according to an embodiment of the present invention;

[0070] Figure 3 is an example diagram of a convolutional layer after convolution kernel pruning according to an embodiment of the present invention;

[0071] Figure 4 is a flow chart of a method for optimizing the network structure of a pruned convolutional neural network according to an embodiment of the present invention;

[0072] Figure 5 is an example diagram of a subnet topology matrix according to an embodiment of the present invention;

[0073] Figure 6 is an example diagram of a subnet reduction matrix according to an embodiment of the present invention;

[0074] Figure 7 is a schematic diagram of input channel mapping inflow provided by an embodiment of the present invention;

[0075] Figure 8 Schematic diagram of output channel mapping restoration provided by an embodiment of the present invention;

[0076] Figure 9 2 is a comparative schematic diagram of the structural optimization of the convolutional neural network provided by an embodiment of the present invention;

[0077] Figure 10 is a structural block diagram of a network structure optimization device for a pruned convolutional neural network according to an embodiment of the present invention;

[0078] Figure 11 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0080] Figure 1 is a structural example diagram of a deep convolutional neural network according to an embodiment of the present invention, such as Figure 1 As shown in the figure, a DCNN consists of multiple convolutional layers. The input channels of layer l are the output channels of layer l-1, and the output channels of layer l are the input channels of layer l+1. Assuming that the convolutional layer l of a DCNN has M input channels and N output channels, the input and output channels of the convolutional layer l are mapped by N filters. A filter contains M convolution kernels, and its function is to map the information of the M input channels to a specific output channel through the corresponding convolution kernels, performing convolution, stacking, nonlinear activation, and pooling operations. A convolution kernel convolutionally maps the information of a specific input channel and sends it to the output channel of the filter containing the convolution kernel. In a convolutional layer consisting of M input channels and N output channels, there are a total of M*N convolution kernels. Figure 2 Schematic diagram of a convolution kernel, a convolution kernel set, and a filter in a convolution layer according to an embodiment of the present invention, as shown in FIG. Figure 2 As shown in the figure, the convolution kernel set (Group) associated with the input channel is the set of convolution kernels divided according to the input channels in the convolution layer. In a convolution layer composed of M input channels and N output channels, there are M Groups, each containing N convolution kernels. For example, the i-th Group is a set of M convolution kernels associated with the i-th input channel among all N filters. In summary, a convolution layer is associated with multiple input channels and multiple output channels, one input channel is associated with one convolution kernel set, one output channel is associated with one filter, and one convolution kernel establishes a connection between a specific input channel and a specific output channel.

[0081] Existing DCNN convolution kernel pruning methods typically only soft-prune the convolution kernels of each convolutional layer. This means that the pruned kernels are not actually removed, but instead their weights are set to zero. This pruning approach results in a large number of redundant zero-weight convolution kernels in the network structure, resulting in a waste of computing resources and making it difficult to achieve effective compression of the DCNN. Figure 3 is an example diagram of a convolution layer after convolution kernel pruning according to an embodiment of the present invention, as shown in FIG. Figure 3As shown, the square blocks represent the input feature maps on the input channel and the output feature maps on the output channel; the middle dots represent the convolution kernels, the black solid dots represent the valid convolution kernels retained after pruning, and the white hollow dots represent the invalid convolution kernels that are set to zero after pruning. The present invention obtains a target clustering matrix by clustering the filters based on the pruning label matrix, and utilizes the similarity of the distribution of effective convolution kernels between filters to classify similar filters into one category, which can more deeply mine the redundant information in the network, further reduce redundant parameters, and improve the compression efficiency of the model. Finally, the pre-trained model is streamlined and mapped based on the pruning label matrix and the target clustering matrix. While maintaining the model accuracy, the model structure is effectively compressed, the computational cost and storage requirements of the model are greatly reduced, and the optimized target convolutional neural network structure is more compact and efficient, and easier to deploy and apply on resource-constrained edge devices.

[0082] According to an embodiment of the present invention, an embodiment of a method for optimizing the network structure of a pruned convolutional neural network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0083] In this embodiment, a method for optimizing the network structure of a pruned convolutional neural network is provided. Figure 4 is a flow chart of a method for optimizing the network structure of a pruned convolutional neural network according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0084] Step S401, obtain the initial convolutional neural network model, pre-train the initial convolutional neural network model, and obtain a pre-trained model. Specifically, the user selects or designs a suitable DCNN according to his or her specific task. For example, a single-branch network can select the VGG (Visual Geometry Group Network) series to obtain stable feature extraction capabilities; densely connected networks such as DenseNet enhance feature reuse; multi-branch structures such as ResNet (Residual Network), GoogleNet or Unet are suitable for tasks that require cross-layer information interaction. After selecting the network, use large-scale public data sets such as ImageNet or proprietary business data sets for sufficient pre-training, and iteratively optimize the loss function to make the model accuracy meet the task requirements. Mainstream deep learning frameworks (such as PyTorch and TensorFlow) provide standardized network construction and training interfaces, which can efficiently implement operations such as convolutional layer parameter initialization and batch normalization. The relevant processes belong to the existing technology and will not be repeated here.

[0085] Step S402: Prune each convolution layer of the pre-trained model. Based on the pruning mark of each convolution kernel of the convolution layer, construct a pruning mark matrix of the convolution layer. The pruning mark is used to indicate whether the convolution kernel is a valid convolution kernel to be retained or an invalid convolution kernel to be pruned. Specifically, assuming that the convolution layer 1 has 4 input channels and 3 output channels (i.e., 3 filters), there are a total of 4*3 convolution kernels involved in the mapping between the input channel and the output channel. A 3*4 dimensional binary matrix is ​​used. This is also called the pruning marking matrix, which marks the validity of the convolution kernel after pruning. Where i represents the number of the input channel, taking l-1 as an example; Indicates that the mth convolution kernel of the nth filter in the lth convolution layer is an invalid convolution kernel set to 0. Indicates that the mth convolution kernel is a valid convolution kernel that is not set to 0. Optionally, the following formula (1) is an example of a pruning label matrix. By constructing a pruning label matrix, it is convenient to further analyze and remove invalid convolution kernels.

[0086]

[0087] In step S403, multiple filters in the convolutional layer are clustered based on the pruned label matrix of the convolutional layer to obtain a target cluster matrix. Specifically, by clustering filters based on the pruned label matrix, filters with similar sparsity patterns are grouped together, and the final clustering results are used to form the target cluster matrix, thereby removing more invalid convolution kernels while reducing network complexity.

[0088] In step S404, the pre-trained model is streamlined based on the pruned label matrix and the target clustering matrix of each convolutional layer to obtain the target convolutional neural network. Specifically, the streamlined mapping based on the target clustering matrix and the pruned label matrix converts the single convolutional structure into a parallel subnetwork structure. While maintaining the same number of output channels, redundant and invalid convolution kernels are removed, resulting in a more streamlined network structure. This further compresses and accelerates the network, achieving network structure optimization for the DCNN.

[0089] The present invention pre-trains the initial convolutional neural network model to ensure that the model performance reaches the required accuracy, and prunes the convolutional layer of the pre-trained model to construct a pruned label matrix to accurately identify the effectiveness of each convolution kernel. The filters are clustered based on the pruned label matrix to obtain a target clustering matrix. The similarity of the effective convolution kernel distribution between filters is utilized to classify similar filters into one category, which can more deeply mine the redundant information in the network, further reduce redundant parameters, and improve the compression efficiency of the model. Finally, the pre-trained model is streamlined and mapped based on the pruned label matrix and the target clustering matrix. While maintaining the model accuracy, the model structure is effectively compressed, the computational cost and storage requirements of the model are greatly reduced, and the optimized target convolutional neural network structure is more compact and efficient, and easier to deploy and apply on resource-constrained edge devices.

[0090] In this embodiment, a method for optimizing the network structure of a pruned convolutional neural network is provided, and the method specifically comprises the following steps:

[0091] Step S501: Obtain an initial convolutional neural network model and pre-train the initial convolutional neural network model to obtain a pre-trained model. Figure 4 Step S401 of the illustrated embodiment will not be described in detail here.

[0092] Step S502: Prune each convolutional layer of the pre-trained model. Based on the pruning mark of each convolution kernel in the convolution layer, a pruning mark matrix of the convolution layer is constructed. The pruning mark is used to indicate whether the convolution kernel is a valid convolution kernel to be retained or an invalid convolution kernel to be pruned. For details, please refer to Figure 4 Step S402 of the illustrated embodiment will not be described in detail here.

[0093] Step S503: Cluster multiple filters in the convolutional layer based on the pruned labeling matrix of the convolutional layer to obtain a target clustering matrix.

[0094] Specifically, the above step S503 includes:

[0095] Step S5031, for each filter in the convolution layer, based on the pruning mark of the convolution kernel included in the filter in the pruning mark matrix, construct the filter sample of the filter. Specifically, assuming that the convolution layer 1 of the DCNN has N filters, each filter has D convolution kernels, the valid convolution kernel in the pruning mark matrix is ​​represented by 1, and the invalid convolution kernel is represented by 0, then each filter is mapped to a filter sample of a D-dimensional binary vector according to the convolution kernel it has. By constructing the filter sample of the binary vector, the sparse pattern of the filter is intuitively represented. Optionally, the nth filter sample is: in, The amount Take binary value, Indicates that the d-th convolution kernel of the n-th filter is a valid convolution kernel, Indicates that the d-th convolution kernel of the n-th filter is an invalid convolution kernel.

[0096] Step S5032, determine the proportion of invalid convolution kernels in each filter sample, and use the filter samples whose proportion exceeds the preset ratio threshold as sparse samples. Specifically, for each filter sample, calculate the proportion of invalid convolution kernels in the filter sample, that is, the proportion of 0 elements in the vector. If the proportion exceeds the preset ratio threshold, for example 80%, it is used as a sparse sample. The invalid kernels of such samples are densely distributed, indicating that the corresponding filters have low dependence on the input channel and are suitable as the class center of the initial clustering, which helps to maintain a high input channel utilization rate in the cluster initialization stage and lays the foundation for subsequent differentiated clustering between classes.

[0097] Step S5033: taking any sparse sample as an initial sample in a preset classification set, and obtaining a target classification set based on the distance between the preset classification set and other filter samples.

[0098] In some optional implementations, the above step S5033 includes:

[0099] In step a1, for each of the other filter samples, the distance between the filter sample and the preset classification set is determined. Specifically, when step a2 is first executed, there is only one initial sample in the preset classification set. In this case, the Jaccard distance between each of the other filter samples and the initial sample is calculated using the following formula (2). The closer this distance is to 1, the greater the distance between the two filter samples, the fewer common elements, and the more different elements.

[0100]

[0101] in, Respectively represent the nth filter sample and the mth filter sample in the lth convolutional layer; represents the distance between the nth filter sample and the mth filter sample; Indicates the number of elements in the nth filter sample and the mth filter sample whose equal bits are all 1, that is, the number of elements whose values ​​are all 1; Indicates the number of elements in the nth filter sample and the mth filter sample where at least one bit is 1.

[0102] If this is not the first time step a2 is executed, then at least one filter sample already exists in the preset classification set. In this case, the joint distance between each other filter sample and all samples in the preset classification set is calculated using the following formula (3).

[0103]

[0104] in, represents any filter sample among the other filter samples; Represents a preset classification set; represents the joint distance; Indicates the number of filter samples in the preset classification set; It represents the distance between the nth filter sample and the jth filter sample in the preset classification set, which can be calculated by the above formula (2).

[0105] In step a2, the filter sample with the largest distance is added to the preset classification set. Specifically, the distance between each filter sample and the preset classification set can be obtained according to equation (2) or (3). The sample with the largest distance, that is, the sample with the greatest difference from the existing set, is selected and added to the preset classification set to ensure that each expansion maximizes the diversity of categories.

[0106] Step a3: The sum of all filter samples in the preset classification set is used as the distribution vector. Specifically, all filter samples in the preset classification set are added together to obtain the distribution vector:

[0107] Step a4, determine whether the number of effective convolution kernels of the distribution vector is equal to the total number of convolution kernels in the convolution layer. Specifically, when When it is equal to the total number D of convolution kernels in the l-th layer filter, that is, the number of non-zero elements in the distribution vector is equal to the total number of convolution kernels, it indicates that the samples in the current preset classification set have covered the distribution of all dimensions. If new samples are added, it will only increase the complexity of the dimensional distribution, reduce the utilization of the input channel, and increase the complex structure of the subnetwork.

[0108] In step a5, when the number of non-zero elements in the distribution vector is not equal to the total number of convolution kernels in the convolution layer, the process returns to the step of determining the distance between each filter sample and the preset classification set for each filter sample in the other filter samples, until the number of non-zero elements in the distribution vector is equal to the total number of convolution kernels in the convolution layer, and the preset classification set obtained last is used as the target classification set. Specifically, When the total number of convolution kernels is equal to the total number of convolution kernels, there is no need to find new samples. The preset classification set obtained last time is used as the final target classification set. If the two are not equal, return to step a1 and continue to add samples to the preset classification set until By determining the target classification set, the input channel utilization of subsequent clustering can be maximized.

[0109] In step S5034, each filter sample in the target classification set is clustered as a cluster center sample to obtain multiple initial clustering results. Specifically, any existing clustering algorithm is used to cluster each filter sample in the target classification set as a cluster center to obtain multiple initial clustering results. The above clustering process is conventional and will not be described in detail here.

[0110] Step S5035: using an interactive benefit optimization clustering algorithm to optimize the multiple initial clustering results to obtain a target clustering matrix.

[0111] In some optional implementations, the above step S5035 includes:

[0112] Step b1, based on the filter samples in the multiple initial clustering results, construct an initial clustering matrix of the multiple initial clustering results, and the initial clustering matrix is ​​used to indicate the class to which the filter samples belong. Specifically, in the initial clustering stage, although the preliminary clustering of the filter samples is completed by the preset classification set, so that the difference in feature distribution between classes is large and the input channel utilization rate is high, there is still room for optimization in the removal rate of invalid convolution kernels within the class. In order to improve the removal rate of invalid convolution kernels within the class while maintaining a high input channel utilization rate, the interactive benefit optimization clustering algorithm is used through steps b1 to b8 to optimize the initial clustering results so that the samples within each class are optimally allocated. First, the initial clustering matrix shown in the following formula (4) is constructed based on the current initial clustering results to indicate the class to which each filter sample belongs.

[0113]

[0114] Among them, U l Represents the initial clustering matrix; C represents the number of initial clustering results; N represents the number of all filter samples; represents the kth initial clustering result of the lth convolutional layer, represents the nth filter sample of the lth convolutional layer, When it belongs to the kth initial clustering result, the corresponding element in the matrix is ​​1. If it does not belong to the kth initial clustering result, the corresponding element in the matrix is ​​0; It means that the sum of the elements in each column of the initial clustering matrix is ​​1, that is, each filter sample will only belong to one initial clustering result.

[0115] Step b2: For each initial clustering result, determine the benefit obtained by removing each filter sample in the initial clustering result. Specifically, use the maximum benefit elimination method to remove samples from the initial clustering result, and determine the benefit obtained after removing each sample using the following formula (5).

[0116]

[0117] in, represents the kth initial clustering result of the lth convolutional layer; Represents the nth filter sample of the lth convolutional layer; N represents the benefit obtained by removing the nth filter sample from the kth initial clustering result; k Indicates the number of samples in the kth initial clustering result; Represents the distribution vector of the kth initial clustering result; Represents the number of non-zero elements in the distribution vector of k initial clustering results; Indicates the number of non-zero elements in the k-th initial clustering result after removing the n-th filter sample from the k-th initial clustering result.

[0118] Step b3, remove the filter sample with the largest benefit in each initial clustering result from the initial clustering result, add the filter sample removed from each initial clustering result to the set of samples to be classified, and update the initial clustering matrix. Specifically, remove the sample with the largest benefit in each initial clustering result from the initial clustering result to which it belongs, add it to the set of samples to be classified, and update the initial clustering matrix shown in formula (4) based on the initial clustering result after removal. The removed samples are set to zero in the updated initial clustering matrix to indicate the class to which each sample belongs after the removal operation. By quantifying the benefits, the samples that contribute the least to the invalid core removal rate within the class are preferentially removed, that is, the samples that can add more zero elements after removal, thereby improving the sparsity within the class, making the invalid core distribution of the remaining samples more concentrated, and improving the homogeneity within the class.

[0119] Step b4: For each filter sample in the set of samples to be classified, determine the loss obtained by adding the filter sample to each initial clustering result. Specifically, a minimum loss addition system is used to add each sample in the set of samples to be classified to the initial clustering result, and the loss obtained by adding each sample to each initial clustering result is determined by the following formula (6).

[0120]

[0121] in, represents the kth initial clustering result of the lth convolutional layer; Represents the nth filter sample of the lth convolutional layer; N represents the loss obtained by adding the nth filter sample to the kth initial clustering result; k Indicates the number of samples in the kth initial clustering result; Represents the distribution vector of the kth initial clustering result; Represents the number of non-zero elements in the distribution vector of k initial clustering results; Indicates the number of non-zero elements in the kth initial clustering result after the nth filter sample is added to the kth initial clustering result.

[0122] In step b5, the filter samples with the smallest loss in each initial clustering result are added to the initial clustering result, the filter samples added to each initial clustering result are removed from the set of samples to be classified, and the initial clustering matrix is ​​updated. Specifically, for each sample, the sample is added to the initial clustering result corresponding to the minimum loss and removed from the set of samples to be classified. Then, based on the added initial clustering result, the initial clustering matrix obtained in step b3 is updated again to indicate the class to which each sample belongs after the addition operation is performed. Through the minimum loss strategy, samples with the smallest impact on the sparsity within the class are preferentially added, which can avoid destroying the existing invalid kernel distribution within the class while supplementing the samples and maintain the homogeneity within the class.

[0123] In step b6, all initial clustering results are merged to obtain a new initial clustering result, and the initial clustering matrix is ​​updated to obtain an intermediate clustering matrix. Specifically, for any two initial clustering results, the effective channel utilization benefit brought about by merging the two can be regarded as the benefit of the same bit of 1 contained in the distribution vectors of the two, which can be calculated by the following formula (7).

[0124]

[0125] in, They represent the i-th initial clustering result and the j-th initial clustering result of the l-th convolutional layer respectively; represents the benefit of merging the i-th initial clustering result and the j-th initial clustering result; N represents the number of elements whose equal bits are 1 in the distribution vector of the i-th initial clustering result and the distribution vector of the j-th initial clustering result; i , N j They represent the number of samples in the i-th initial clustering result and the j-th initial clustering result respectively.

[0126] The loss of redundant convolution kernel removal rate caused by the combination of the two can be regarded as the loss of zero bits reduced after the combination of the two, which can be calculated by the following formula (8).

[0127]

[0128] in, They represent the i-th initial clustering result and the j-th initial clustering result of the l-th convolutional layer respectively; represents the loss obtained by merging the i-th initial clustering result and the j-th initial clustering result; Represents the distribution vector of the i-th initial clustering result and the j-th initial clustering result; Indicates The middle bit is 0, and The number of elements with a 1 in the middle bit; Indicates The middle bit is 0, and The number of elements with 1 in the middle position; N i , N j They represent the number of samples in the i-th initial clustering result and the j-th initial clustering result respectively.

[0129] Based on the difference in gains and losses from merging the two initial clustering results, a merging function is derived. If this merging function is greater than 0, the two can be merged; if it is not, it cannot be merged. This evaluation continues for other initial clustering results until all pairwise combinations of all initial clustering results have been merged. This results in a merged initial clustering result, and the clustering matrix from step b5 is updated again to indicate the class assigned to each sample after the merge operation. This merging operation reduces the number of clustering results, simplifies the network structure, and maintains or even improves effective channel utilization, achieving a balance between compression and efficiency.

[0130] Step b7: Determine whether the delta-norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​less than a preset convergence threshold. Specifically, the difference between the intermediate clustering matrix obtained after a single elimination, addition, and merging operation and the initial clustering matrix is ​​calculated. The delta-norm of this difference is compared with the preset convergence threshold to determine whether the clustering has reached convergence.

[0131] Step b8: When the binary norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​not less than the preset convergence threshold, the intermediate clustering matrix is ​​used as the new initial clustering matrix, and the process returns to the step of constructing the initial clustering matrix of the multiple initial clustering results based on the filter samples in the multiple initial clustering results, until the binary norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​less than the preset convergence threshold, and the intermediate clustering matrix obtained by the last update is used as the target clustering matrix. Specifically, if the binary norm of the difference is not less than the preset convergence threshold, the process returns to step b1 and performs elimination, addition, and merging again until the binary norm of the difference is less than the preset convergence threshold, for example, the preset convergence threshold is 10. -30 , that is, ||U l -U l ||2<10 -30, the clustering reaches convergence, and each sample is now in the optimal class. The matrix obtained from the last update is used as the target clustering matrix to represent the optimal class for each sample. Through iterative optimization, continuous adjustment of sample attribution and class merging ensures that each sample is in the class that optimizes intra-class sparsity and inter-class differentiation. The final target clustering matrix maximizes the removal rate of invalid cores within a class and differentiates the distribution of valid cores between classes, laying the foundation for generating efficient subnetworks and achieving deep network compression and acceleration.

[0132] Step S504: Simplify and map the pre-trained model based on the pruned label matrix and the target clustering matrix of each convolutional layer to obtain a target convolutional neural network.

[0133] Specifically, the above step S504 includes:

[0134] Step S5041: Treat all input channels in the pre-trained model as input nodes, all output channels as output nodes, and each convolutional layer as a supporting node to construct a node set for the pre-trained model. Specifically, to implement graph-theoretic modeling of the network structure, the structure of the pre-trained model is abstracted into nodes. Optionally, treat all input channels in the pre-trained model as a single input node, all output channels as a single output node, and each convolutional layer as a supporting node to obtain the node set: Among them, v 0 represents the input node, N T represents the number of convolutional layers of the pre-trained model, Represents the output node, 1≤l≤N T , v l Indicates a support node.

[0135] In step S5042, for any two nodes in the node set that have a convolutional relationship between them, a directed edge is determined between the two nodes, and a queue for subnetting is constructed based on all directed edges. Specifically, since the pre-trained model contains multiple structures besides convolutional layers, and convolution kernel pruning only targets convolutional layers, only structures with convolutional relationships need to be optimized. More specifically, directed edges with convolutional relationships are identified and a queue for subnetting is formed to accurately locate all convolutional layers that require optimization.

[0136] Step S5043: For each directed edge in the queue to be subnetted, multiple subnet topology matrices are constructed based on the pruning label matrix of the convolution output node and the target clustering matrix. The convolution output node represents the node located in the output direction of the directed edge.

[0137] In some optional implementations, step S5043 includes:

[0138] Step c1, extract the column index corresponding to the non-zero element from each row of the target clustering matrix of the convolution output node. Specifically, assuming that any directed edge e i,j :v i →v j , v i Located in the input direction of the directed edge, v i As the convolution input node, v j Located in the output direction of the directed edge, v j As the convolution output node. From the convolution output node, that is, from the target clustering matrix of convolution layer j, extract the column index of the non-zero elements in each row. Each row of the matrix can correspond to a subnet, and the column index of the non-zero element represents the output channel of the subnet. Assume that the target clustering result is as shown in the following formula (9). For the first row of the target clustering result, the non-zero elements are located in the first column and the second column, that is, the column index is {1, 2}. Similarly, the column index of the second row is {3, 4}, and the column index of the third row is {5, 6}.

[0139]

[0140] In step c2, the rows corresponding to each column index are extracted from the pruned label matrix of the convolution output node to obtain multiple subnetwork topology matrices. Specifically, assuming that the pruned label matrix of convolution layer j is shown in Equation (10), the rows are extracted according to the column index of each row of the target clustering result.

[0141]

[0142] For column index {1, 2}, the first and second rows are extracted from the pruned label matrix to form a subnet topology matrix; for column index {3, 4}, the third and fourth rows are extracted from the pruned label matrix to form a subnet topology matrix; for column index {5, 6}, the fifth and sixth rows are extracted from the pruned label matrix to form a subnet topology matrix, and the three subnet topology matrices shown in the following formula (11) are obtained, where c represents the number of subnet topology matrices.

[0143]

[0144] Step S5044, remove the all-zero columns in each subnet topology matrix to obtain the corresponding subnet simplified matrix. Specifically, remove the all-zero columns in the subnet topology matrix, that is, remove the invalid convolution kernels of all zeros to increase the compression rate of the network. Taking the above formula (11) as an example, the 2-4 columns of the first subnet topology matrix are all-zero columns, which are removed; the 1, 3 and 4 columns of the second subnet topology matrix are all-zero columns, which are removed; the 1 and 2 columns of the third subnet topology matrix are all-zero columns, which are removed. By removing the all-zero columns, the invalid convolution kernels are reduced and the structure is simplified.

[0145] In some optional embodiments, Figure 5 is an example diagram of a subnet topology matrix according to an embodiment of the present invention. Figure 5 As shown, the left side is the pruned label matrix of the convolution layer and the corresponding 7 filter samples, and the target clustering result of the convolution layer ( Figure 5 (not shown), three subnetwork topology matrices and their corresponding filter samples are extracted from the pruned label matrix. Figure 6 is an example diagram of a subnet reduction matrix according to an embodiment of the present invention, such as Figure 6 As shown, for Figure 5 For each subnet topology matrix in , remove the all-zero columns. For the first subnet topology matrix, remove its 1st, 2nd, and 5th columns; for the second subnet topology matrix, remove its 1st, 3rd, 4th, and 5th columns; for the third subnet topology matrix, remove its 2nd, 3rd, and 4th columns, to obtain the three subnet simplified matrices on the right.

[0146] Step S5045: Initialize each subnet simplified matrix to obtain a corresponding subnet initialized matrix. Specifically, initialize the simplified matrix, that is, set all elements in the matrix to 0.

[0147] In step S5046, for each subnet initialization matrix, the parameters of the valid convolution kernel at the corresponding position are extracted from the pruning mark matrix of the convolution output node and filled into the subnet initialization matrix to obtain the subnet parameter matrix. Specifically, when pruning and initializing the matrix, the index of each element is not changed. That is, if the index of an element in the subnet topology matrix is ​​3x1, that is, the element is the first input channel of the third filter, even if the position of the element in the matrix may change after the all-zero column is removed, the value of the element changes during initialization, but its index does not change. Therefore, according to the index of each element, the corresponding position is found in the pruning mark matrix. If the convolution kernel at that position is valid, its parameter 1 is filled into the element. If the convolution kernel at that position is invalid, no filling is required, and the element can be kept at the initialization value of 0. Through matrix filling, only valid parameters after pruning are retained, avoiding redundant weights occupying memory, while maintaining the spatial position mapping relationship of the parameters, ensuring network functional equivalence, and avoiding accuracy loss due to structural adjustment.

[0148] Step S5047 , performing input and output channel mapping on each subnet parameter matrix to obtain a simplified mapping layer corresponding to the directed edge, removing the directed edge from the queue to be subnetted, and determining whether the queue to be subnetted is empty.

[0149] In some optional implementations, step S5047, performing input-output channel mapping on each subnetwork parameter matrix to obtain a simplified mapping layer corresponding to directed edges, includes:

[0150] Step d1, determine all channels of the convolution input node to form a total channel vector, and the convolution output node represents the node located in the input direction of the directed edge. Specifically, since the single convolution structure is transformed into a subnet parallel convolution structure, the order of the output channels obtained will be different from the original model, which will cause a loss of network accuracy or an increase in network complexity. To avoid the above problems, the embodiment of the present invention only changes the convolution structure without changing the size and order of its input channels and output channels to ensure that the final output accuracy of the network is not affected. Since the input channels received by each subnet streamlined matrix are different from each other, it is necessary to map the input channels received by each subnet through steps d1-step d3 to obtain the input channels required by each subnet; at the same time, it is necessary to map and restore the output channels of each subnet output through steps d4-step d7 to ensure that the output channels remain unchanged. In order to better i Analyze and process the input channels in the node to define all channels of the node, that is, the total channel vector: in, Represents node v i The u-th channel of Represents node v i The number of channels in .

[0151] Step d2: Determine the input channel vector based on the correspondence between each subnet parameter matrix and all channels. Specifically, define node v i The input channel vector of : If node v is required in the cth subnet parameter matrix i When the u-th channel in The value is 1, otherwise it is 0.

[0152] Step d3: For each subnet parameter matrix, perform element-by-element multiplication on the total channel vector and the input channel vector to obtain the channel inflow vector. Specifically, for node v i It is the input channel of the convolution layer, and the input channels required by each subnet are different. i The total channel vector is multiplied element-by-element by the input channel vector to obtain the channel inflow vector. This element-by-element multiplication allows for precise diversion of the original input channels, allowing each subnetwork to obtain the input channels it needs.

[0153] In some optional embodiments, Figure 7 Schematic diagram of input channel mapping flow provided by an embodiment of the present invention, such as Figure 7 As shown, for each subnet parameter matrix, according to the total channel vector H iThe element-by-element multiplication is performed with the corresponding input channel vector to realize the diversion of the original input channel so that each subnetwork obtains the input channel it needs.

[0154] Step d4, determine the output channel vector of each subnet parameter matrix. Specifically, the output channel mapping restoration function is to restore the output channel mapping of each subnet parameter matrix output to the original output channel order. Simply merging the channels output by each subnet cannot restore them to the original output channels. Therefore, it is necessary to perform mapping restoration on the channels output by all subnets to ensure that the characteristic signal transmission of the network is not affected. First, for each subnet corresponding to the subnet parameter matrix, define its output channel vector: in, represents the number of output channels of the cth subnet parameter matrix, and the total number of channels of all subnet parameter matrices is the same as the number of nodes v j The total number of channels in the same.

[0155] Step d5, based on the corresponding column index of the subnet parameter matrix in the target clustering matrix, determine the restoration matrix of the subnet parameter matrix. Specifically, in order to restore the output channels of all subnet parameter matrices to their original order, the output channels of each subnet are expanded so that the number of channels is the same as that of the node v. j The total number of channels of . Assume that node v j The total number of output channels before the unreduced mapping is 6, and the node v j The number of output channels (equivalent to the number of filters, which is also equivalent to the number of rows) of the u-th subnet parameter matrix is ​​2, and the reduction matrix of the subnet parameter matrix is The dimension is 2x6. For the subnetwork parameter matrix, it is assumed that it is constructed based on the column index {2, 5} of the first row of the target clustering result. When constructing the reduction matrix, each row in the reduction matrix corresponds to a value in the column index. More specifically, for each element in the reduction matrix, if the element is in the first row, its row index is 1. The first element is found from the column index, that is, output channel 2. The element in the second column is found from all elements in the first row of the reduction matrix. The element in (1, 2) is set to 1, and the elements in other columns in the first row are set to 0. Similarly, for the second row of the reduction matrix, the second element is found from the column index, that is, output channel 5. The element in the fifth column is found from all elements in the second row of the reduction matrix. The element in (2, 5) is set to 1, and the elements in other columns in the second row are set to 0. Through the above process, the reduction matrix of each subnetwork parameter matrix can be obtained to represent the corresponding relationship between the output channels before and after the simplified mapping.

[0156] In step d6, the output channel vector is multiplied by the restoration matrix to obtain the channel restoration vector of the subnetwork parameter matrix. Specifically, through matrix operations, the subnetwork output channels are restored to their original channel positions, ensuring that the characteristic signals output by each subnetwork are accurately integrated into the output structure of the original network, maintaining the consistency and accuracy of the network output.

[0157] Step d7, the sum of the channel restoration vectors of all subnet parameter matrices is determined as the output channel vector at the convolution output node. Specifically, the channel restoration vectors of all subnets are superimposed to obtain a channel restoration vector to represent the node v j The correct order of the output channels.

[0158] In some optional embodiments, Figure 8 Schematic diagram of output channel mapping restoration provided by an embodiment of the present invention, such as Figure 8 As shown in Figure 1, the output channels of each subnetwork parameter matrix are restored and mapped through the restoration matrix, and finally all the restoration vectors are superimposed to obtain the output channels in the correct order.

[0159] Step d8, based on the output channel vector and the channel inflow vector of each subnet parameter matrix, obtain the simplified mapping layer corresponding to the directed edge. Specifically, by j The input channel and output channel of the node v are mapped separately to obtain j The simplified mapping layer realizes the organic combination of input channel mapping and output channel restoration, ensures the equivalent transformation of the entire convolution structure, and realizes efficient simplification and optimization of the network.

[0160] Step S5048, when the queue to be subnetted is not empty, continue to perform simplified mapping on the next directed edge until the queue to be subnetted is empty, and obtain the target convolutional neural network based on the simplified mapping layers corresponding to all directed edges. Specifically, the queue to be subnetted stores all directed edges that need to be simplified mapped. After the simplified mapping of a directed edge is completed through the above steps, it is removed and the process returns to step S5043 to perform simplified mapping on the next directed edge again until all directed edges are completed. At this time, all structures with convolutional relationships in the model have completed structural optimization, and the final target convolutional neural network is obtained. All convolutional structures that need to be optimized are processed in order through the queue mechanism to ensure that each convolution layer of the entire network can be effectively simplified and optimized, and finally a target convolutional neural network with high compression rate, high computational efficiency and unaffected accuracy is obtained, making it more suitable for deployment on resource-constrained edge devices.

[0161] In some optional embodiments, Figure 9 : is a comparative schematic diagram of the structural optimization of the convolutional neural network provided by the embodiment of the present invention, such as Figure 9As shown in the figure, the convolution layer is divided into multiple subnet parameter matrices, that is, multiple subnets, and the single convolution layer structure is transformed into a subnet branch parallel computing structure. In addition, input channel mapping and output channel mapping are used to ensure that the input and output channels remain unchanged, thereby achieving improvements in the network's compression ratio and acceleration ratio while maintaining the accuracy of the network's final output.

[0162] The present invention pre-trains the initial convolutional neural network model to ensure that the model performance reaches the required accuracy, and prunes the convolutional layer of the pre-trained model to construct a pruned label matrix to accurately identify the effectiveness of each convolution kernel. The filters are clustered based on the pruned label matrix to obtain a target clustering matrix. The similarity of the effective convolution kernel distribution between filters is utilized to classify similar filters into one category, which can more deeply mine the redundant information in the network, further reduce redundant parameters, and improve the compression efficiency of the model. Finally, the pre-trained model is streamlined and mapped based on the pruned label matrix and the target clustering matrix. While maintaining the model accuracy, the model structure is effectively compressed, the computational cost and storage requirements of the model are greatly reduced, and the optimized target convolutional neural network structure is more compact and efficient, and easier to deploy and apply on resource-constrained edge devices.

[0163] In this embodiment, a network structure optimization device for a pruned convolutional neural network is also provided. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the details that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0164] This embodiment provides a network structure optimization device for a pruned convolutional neural network, such as Figure 10 Shown, including:

[0165] The pre-training module 1001 is used to obtain an initial convolutional neural network model, pre-train the initial convolutional neural network model, and obtain a pre-trained model.

[0166] The pruning module 1002 is used to prune each convolutional layer of the pre-trained model, and construct a pruning mark matrix of the convolutional layer based on the pruning mark of each convolution kernel of the convolutional layer. The pruning mark is used to indicate whether the convolution kernel is a valid convolution kernel to be retained or an invalid convolution kernel to be pruned.

[0167] The clustering module 1003 is configured to cluster multiple filters in the convolutional layer based on the pruned labeling matrix of the convolutional layer to obtain a target clustering matrix.

[0168] The optimization module 1004 is used to simplify the mapping of the pre-trained model based on the pruned label matrix and the target clustering matrix of each convolutional layer to obtain a target convolutional neural network.

[0169] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0170] The network structure optimization device of the pruned convolutional neural network in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0171] The embodiment of the present invention also provides a computer device having the above Figure 10 The network structure optimization device of the pruned convolutional neural network is shown.

[0172] See also Figure 11 , Figure 11 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 11 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 11 A processor 10 is taken as an example.

[0173] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0174] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0175] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0177] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 11 The bus connection is taken as an example.

[0178] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0179] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0180] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0181] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for optimizing the network structure of a pruned convolutional neural network, characterized in that: The method comprises: Obtaining an initial convolutional neural network model, and pre-training the initial convolutional neural network model to obtain a pre-trained model; Pruning each convolutional layer of the pre-trained model, and constructing a pruning tag matrix for the convolutional layer based on a pruning tag of each convolution kernel of the convolutional layer, wherein the pruning tag is used to indicate whether the convolution kernel is a valid convolution kernel to be retained or an invalid convolution kernel to be pruned; Clustering multiple filters in the convolutional layer based on the pruned label matrix of the convolutional layer to obtain a target clustering matrix; The pre-trained model is streamlined and mapped based on the pruned label matrix and the target clustering matrix of each convolutional layer to obtain a target convolutional neural network.

2. The method according to claim 1, characterized in that The pruning labeling matrix based on the convolutional layer is used to cluster multiple filters in the convolutional layer to obtain a target clustering matrix, including: For each filter in the convolutional layer, construct a filter sample of the filter based on a pruning mark of a convolution kernel included in the filter in the pruning mark matrix; Determine the proportion of invalid convolution kernels in each filter sample, and take the filter samples whose proportion exceeds a preset proportion threshold as sparse samples; Taking any sparse sample as an initial sample in a preset classification set, and obtaining a target classification set based on the distance between the preset classification set and other filter samples; Clustering each filter sample in the target classification set as a class center sample to obtain multiple initial clustering results; An interactive benefit optimization clustering algorithm is used to optimize the multiple initial clustering results to obtain the target clustering matrix.

3. The method according to claim 2, characterized in that The method of using any sparse sample as an initial sample in a preset classification set and obtaining a target classification set based on the distance between the preset classification set and other filter samples includes: For each filter sample in the other filter samples, determining a distance between the filter sample and the preset classification set; Add the filter sample with the largest distance to the preset classification set; Taking the sum of all filter samples in the preset classification set as a distribution vector; Determining whether the number of non-zero elements of the distribution vector is equal to the total number of convolution kernels in the convolution layer; When the number of non-zero elements of the distribution vector is not equal to the total number of convolution kernels in the convolution layer, return to the step of determining the distance between the filter sample and the preset classification set for each filter sample in other filter samples until the number of non-zero elements of the distribution vector is equal to the total number of convolution kernels in the convolution layer, and use the preset classification set obtained last time as the target classification set.

4. The method according to claim 2, characterized in that The interactive benefit optimization clustering algorithm is used to optimize the multiple initial clustering results to obtain the target clustering matrix, including: Based on the filter samples in the multiple initial clustering results, constructing an initial clustering matrix of the multiple initial clustering results, wherein the initial clustering matrix is ​​used to indicate the class to which the filter samples belong; For each initial clustering result, determining a benefit obtained by removing each filter sample in the initial clustering result; Eliminate the filter sample with the greatest benefit in each initial clustering result from the initial clustering result, add the filter sample eliminated from each initial clustering result to the set of samples to be classified, and update the initial clustering matrix; For each filter sample in the set of samples to be classified, determining a loss obtained by adding the filter sample to each initial clustering result; Adding the filter sample with the smallest loss in each initial clustering result to the initial clustering result, removing the filter sample added to each initial clustering result from the set of samples to be classified, and updating the initial clustering matrix; Merging all initial clustering results to obtain a new initial clustering result, updating the initial clustering matrix to obtain an intermediate clustering matrix; Determining whether a binary norm of a difference between the initial clustering matrix and the intermediate clustering matrix is ​​less than a preset convergence threshold; When the binary norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​not less than a preset convergence threshold, the intermediate clustering matrix is ​​used as a new initial clustering matrix, and the process returns to the step of constructing the initial clustering matrix of the multiple initial clustering results based on the filter samples in the multiple initial clustering results, until the binary norm of the difference between the initial clustering matrix and the intermediate clustering matrix is ​​less than a preset convergence threshold, and the intermediate clustering matrix obtained by the last update is used as the target clustering matrix.

5. The method according to claim 1, wherein The pre-trained model is simplified and mapped based on the pruned label matrix and the target clustering matrix of each convolutional layer to obtain a target convolutional neural network, including: All input channels in the pre-trained model are used as input nodes, all output channels are used as output nodes, and each convolutional layer is used as a support node to construct a node set of the pre-trained model; For any two nodes in the node set, when a convolution relationship exists between the two nodes, determining that there is a directed edge between the two nodes, and constructing a queue to be subnetted based on all directed edges; For each directed edge in the queue to be subnetted, construct multiple subnet topology matrices based on the pruning label matrix of the convolution output node and the target clustering matrix, where the convolution output node represents the node located in the output direction of the directed edge; Eliminate the all-zero columns in each subnet topology matrix to obtain the corresponding subnet simplified matrix; Initialize each subnet simplified matrix to obtain the corresponding subnet initialization matrix; For each subnet initialization matrix, extract the parameters of the effective convolution kernel at the corresponding position from the pruning mark matrix of the convolution output node, and fill them into the subnet initialization matrix to obtain the subnet parameter matrix; Performing input and output channel mapping on each subnet parameter matrix to obtain a simplified mapping layer corresponding to the directed edge, removing the directed edge from the queue to be subnetted, and determining whether the queue to be subnetted is empty; When the queue to be subnetted is not empty, continue to perform simplified mapping on the next directed edge until the queue to be subnetted is empty, and obtain the target convolutional neural network based on the simplified mapping layers corresponding to all directed edges.

6. The method according to claim 5, characterized in that For each directed edge in the queue to be subnetted, multiple subnet topology matrices are constructed based on the pruning label matrix of the convolution output node and the target clustering matrix, including: Extracting the column index corresponding to the non-zero element from each row of the target clustering matrix of the convolution output node; The row corresponding to each column index is extracted from the pruned label matrix of the convolution output node to obtain the multiple subnetwork topology matrices.

7. The method according to claim 6, characterized in that The step of performing input-output channel mapping on each subnet parameter matrix to obtain a simplified mapping layer corresponding to the directed edge includes: Determine all channels of the convolution input node to form a total channel vector, wherein the convolution output node represents a node located in the input direction of the directed edge; Determine an input channel vector based on a correspondence between each subnetwork parameter matrix and all the channels; For each subnetwork parameter matrix, performing element-by-element multiplication on the total channel vector and the input channel vector to obtain a channel inflow vector; Determine the output channel vector of each subnetwork parameter matrix; Determining a reduction matrix of the subnetwork parameter matrix based on a column index corresponding to the subnetwork parameter matrix in the target clustering matrix; Performing matrix left multiplication on the output channel vector and the restoration matrix to obtain a channel restoration vector of the subnetwork parameter matrix; Determine the sum of the channel restoration vectors of all subnetwork parameter matrices as the output channel vector at the convolution output node; Based on the output channel vector and the channel inflow vector of each subnetwork parameter matrix, a simplified mapping layer corresponding to the directed edge is obtained.

8. A network structure optimization device for a pruned convolutional neural network, characterized in that: The device comprises: A pre-training module is used to obtain an initial convolutional neural network model and pre-train the initial convolutional neural network model to obtain a pre-trained model; A pruning module is used to prune each convolutional layer of the pre-trained model, and construct a pruning mark matrix of the convolutional layer based on the pruning mark of each convolution kernel of the convolutional layer, wherein the pruning mark is used to indicate whether the convolution kernel is a valid convolution kernel to be retained or an invalid convolution kernel to be pruned; A clustering module, configured to cluster the multiple filters in the convolutional layer based on the pruned labeling matrix of the convolutional layer to obtain a target clustering matrix; An optimization module is used to simplify and map the pre-trained model based on the pruned label matrix and the target clustering matrix of each convolutional layer to obtain a target convolutional neural network.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the network structure optimization method of the pruned convolutional neural network according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the network structure optimization method of the pruned convolutional neural network according to any one of claims 1 to 7.

Images