Network model lightweight method, device, equipment, medium and program product

Abstract

The invention discloses a network model lightweight method and device, equipment, a medium and a program product, and relates to the technical field of neural network compression, and the method comprises the steps: analyzing channel similarity measurement between models based on curvelet transformation feature similarity and structural similarity, merging matched channels, and avoiding the loss of image recognition knowledge; according to the method, the fusion matrix is generated based on the matched channel pairs, model fusion is performed through channel similarity measurement, and the method does not depend on a specific model architecture, so that the method can be suitable for fusion of heterogeneous models, and the lightweight capability and effect of the network model are improved.

Description

Technical Field

[0001] The present application relates to the field of neural network compression technology, and in particular to a network model lightweight method, device, equipment, medium and program product. Background Art

[0002] In order to enable the deployment of neural networks on edge devices, the network model needs to be lightweight, which specifically involves model pruning and model fusion.

[0003] In the related art, the scaling process based on the core set theory is used to scale the original network layer, but this method will cause the loss of some image recognition knowledge. When fusion of models is used, fusion methods such as ZipIt are used, but this method is only applicable to the fusion of isomorphic network models, that is, each model to be fused has an equal number of channels in the corresponding layer. Summary of the invention

[0004] The present application provides a network model lightweight method to at least solve the problem of network knowledge loss in related technologies and only supporting the fusion of homogeneous network models.

[0005] This application provides a network model lightweight method, including:

[0006] Performing random channel pruning on the output channels of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model;

[0007] Based on the feature maps outputted by the original network model and the compressed model at the target layer respectively, analyzing the similarity metric between channels according to the target similarity index to determine the channel matching result; the target similarity index includes structure and curvelet transform features, and the channel matching result represents a set of matched channel pairs;

[0008] Generate a fusion matrix according to the channel matching result, and determine the fusion weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model;

[0009] A fused model is obtained based on the fused weights corresponding to different network layers, and the fused model is used as the final compressed model of the original network model.

[0010] The present application also provides a network model lightweight device, including:

[0011] A random channel pruning module is used to perform random channel pruning on the output channel of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model;

[0012] A channel matching module, configured to analyze the inter-channel similarity metric according to a target similarity index based on the feature maps outputted at the target layer by the original network model and the compressed model respectively, to determine a channel matching result; the target similarity index includes structure and curvelet transform features, and the channel matching result represents a set of matched channel pairs;

[0013] A post-fusion weight determination module, used to generate a fusion matrix according to the channel matching result, and determine the post-fusion weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model;

[0014] The final compressed model determination module is used to obtain a fused model based on the fused weights corresponding to different network layers, and use the fused model as the final compressed model of the original network model.

[0015] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned network model lightweighting methods when executing the computer program.

[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned network model lightweighting methods are implemented.

[0017] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned network model lightweighting methods when executed by a processor.

[0018] In this application, by analyzing the channel similarity measurement between models based on curvelet transform feature similarity and structural similarity, matching channels are merged, which can avoid the loss of image recognition knowledge and improve the accuracy of image recognition; a fusion matrix is ​​generated based on matching channel pairs, and model fusion is performed through channel similarity measurement. It does not depend on the specific model architecture and is therefore suitable for the fusion of heterogeneous models, thereby improving the ability and effect of lightweight network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of a network model lightweight method provided in this application;

[0021] Figure 2A specific flow chart of a network model lightweight method provided in this application;

[0022] Figure 3 A schematic diagram of a specific channel fusion method under weak compression rate constraint provided in this application;

[0023] Figure 4 A schematic diagram of a specific channel fusion method under strong compression rate constraints provided in this application;

[0024] Figure 5 A schematic diagram of the structure of a network model lightweight device provided in this application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0028] The embodiment of the present application provides a network model lightweight method, see Figure 1 As shown, the method may include the following steps:

[0029] Step S11: Perform random channel pruning on the output channels of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model.

[0030] Pruning is a commonly used method to lightweight image recognition networks. It reduces the number of parameters, storage, and floating-point calculations of image recognition networks during inference by removing neurons or parameter groups in the image recognition network, thereby enabling the deployment and inference acceleration of image recognition networks on resource-limited edge devices. Specifically, pruning can be performed during training or inference. For example, during inference, pruning uses heuristic rules to select low-importance parameter groups for removal, and directly reconstructs the compressed model parameter matrix through the least squares method, without relying on retraining to restore the accuracy of the compressed image recognition network, greatly improving the construction efficiency of the model lightweight process itself. The above original network model is an image recognition network model.

[0031] In some embodiments, random channel pruning is performed on the output channels of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model, including: generating a random mask vector for the output channel of the target layer of the original network model, and performing channel pruning on the output channel of the target layer according to the random mask vector; that is, random output channel pruning is performed on the target layer of the original network model, that is, a random mask vector gr is generated for the output channel of the target layer of the original network. According to the feature map output by the target layer of the original network model and the feature map output by the target layer of the compressed model, the least squares method is used to solve the minimization error to determine the weight of the target layer of the compressed model. For example, the least squares method can be used to solve the optimization function of the feature map reconstruction error to obtain the weight of the target layer of the compressed network model, that is, by minimizing the feature map output by the target layer of the compressed model. And the feature map output by the target layer of the original model By solving the difference between Get the compressed network layer weights , X represents the input data of the target layer of the original model.

[0032] Step S12: Based on the feature maps outputted by the original network model and the compressed model at the target layer respectively, the channel similarity metric is analyzed according to the target similarity index to determine the channel matching result; the target similarity index includes structure and curvelet transform features, and the channel matching result represents a set of matched channel pairs.

[0033] In this embodiment, similarity metric learning is used to quantify the similarity between the output feature maps of the compressed model and the original model, and by calculating the similarity index, the channels with high similarity in the compressed model and the original model are found, so as to retain the most important feature information in the subsequent fusion process. Specifically, the two indicators of structural similarity and similarity of curvelet transform features can be used to quantify the similarity between the output feature maps of the compressed model and the original model for subsequent channel fusion or other optimization operations. In this way, the most important feature information can be retained during the model compression process, while reducing the computational complexity and storage requirements of the model.

[0034] Among them, the channel similarity measurement includes cross-model similarity and internal similarity. Cross-model similarity is to calculate the feature map of the compressed model And the feature map of the original model The channel similarity between the compressed feature maps is used to evaluate whether the compressed feature maps retain the characteristics of the original feature maps. Internal similarity, including the feature maps of the compressed model Internally, the similarity between channels is calculated to evaluate the redundancy within the compressed feature map; in the feature map of the original model Internally, the similarity between channels is calculated to evaluate the redundancy within the original feature map. That is, the target layer output feature map of the compressed model and the original model , Perform similarity metric learning, that is, calculate the output feature map of the target layer of the compressed model respectively The target layer output feature map of the original model The similarity index between each channel is calculated in the target layer of the compressed model to output the feature map Similarity index between channels, and calculation of output feature map inside the target layer of the original model Similarity index between channels.

[0035] The curvelet transform can perform multi-scale and multi-directional analysis on the texture features of an image. In this embodiment, the curvelet transform can be embedded into the neural network as a pooling layer for feature extraction. That is, before model fusion, the output feature map of the current network layer is firstly subjected to the curvelet transform, and the low-frequency coefficients of the curvelet transform, i.e., the coarse-scale layer coefficients, are used as the data for subsequent calculation of the similarity between channels, so as to achieve data compression while retaining the main information of the feature map. Assume that the target layer (the first Layer) Weight is a tensor of M×C×h×h, where M is the number of output channels, C is the number of input channels, and h is the size of the convolution kernel; the first Layer Weights is a tensor of N×C×h×h, where N is the number of output channels after compression; The output feature map of the layer is , compressed network model The output feature map of the layer is , X represents the original model The input data of the layer, Represents convolution calculation. The similarity between the output feature maps is used as the basis for measuring the similarity between weights, and the output feature maps and Perform curvelet transform to reduce the amount of data for metric learning.

[0036] Specifically, the feature maps outputted at the target layer by the original network model and the compressed model are analyzed according to the target similarity index to determine the channel matching result, including: the output of the target layer of the original network model is the first feature map, and the output of the target layer of the compressed model is the second feature map; the structural similarity between the channels of the first feature map and the channels of the second feature map is calculated, the structural similarity between different channels in the first feature map, and the structural similarity between different channels in the second feature map are calculated to generate a structural similarity matrix; the normalized mutual information between the channel feature map of the first feature map and the channel feature map of the second feature map on the coarse scale coefficient of the curvelet transform is calculated, the normalized mutual information of different channel feature maps of the first feature map on the coarse scale coefficient of the curvelet transform, and the normalized mutual information of different channel feature maps of the second feature map on the coarse scale coefficient of the curvelet transform are calculated to generate a normalized mutual information matrix; a mean matrix is ​​generated based on the structural similarity matrix and the normalized mutual information matrix, and the channel matching result is determined according to the mean matrix. It can be understood that the feature map output by the target layer is a multi-channel tensor, and the channel feature map is a two-dimensional feature map output by a single channel.

[0037] That is to say, calculate and The target similarity index between the channels of Each channel bi and For each channel ai in , the structural similarity and the normalized mutual information of the coarse-scale coefficients of the curvelet transform are calculated. Inside the target layer of the compressed model, the output feature map is calculated The similarity index between channels, that is, For any two channels bi and bj in the compressed feature map, we can calculate the similarity between them, thereby evaluating the redundancy or correlation between channels in the compressed feature map. In the target layer of the original model, we calculate the output feature map The similarity index between channels, that is, For any two channels ai and aj in the original feature map, we can calculate the similarity between them, thereby evaluating the redundancy or correlation between channels in the original feature map. Based on the structural similarity of each channel, we can get the structural similarity matrix ; Based on the normalized mutual information between the coarse-scale coefficients of the curvelet transform of each channel feature map, a normalized mutual information matrix is ​​formed , N is the number of output channels of the target layer of the compressed model, and M is the number of output channels of the target layer of the original network model. Finally, the mean matrix (SSIM+NMI) / 2 is obtained. The mean matrix can comprehensively represent the similarity of different channels in structure and coarse-scale coefficients, and the channel matching results are screened out according to the mean matrix.

[0038] By using normalized mutual information NMI (Normal mutual information) and structural similarity SSIM (Structural Similarity) as the similarity evaluation indicators between channels of feature maps, the value range of both can be . Among them, the normalized mutual information is equal to the ratio of the average mutual information between the two channels to the arithmetic mean of their respective entropies. Taking into account that the channel similarity matching results obtained based on the two evaluation indicators will produce different results, this application uses the mean of the two evaluation indicators as the comprehensive evaluation indicator, selects the largest similarity comprehensive evaluation indicator within the model and across models as the similarity matching result, and then generates a fusion matrix.

[0039] Further, determining the channel matching result according to the mean matrix includes: screening out the maximum value in each row of the mean matrix, and obtaining the channel matching result according to the row and column index corresponding to the maximum value. That is, from the mean matrix, selecting the maximum similarity row by row to determine the channel matching result (u, v).

[0040] Step S13: Generate a fusion matrix according to the channel matching result, and determine the fused weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model.

[0041] All channel matching results are represented by generating a fusion matrix, and the fused weight of the target layer is determined based on the fusion matrix, the weight of the target layer of the original network model, and the weight of the target layer of the compressed model.

[0042] In some embodiments, before generating a fusion matrix according to the channel matching results, it may also include: obtaining a network lightweight constraint scenario; the network lightweight constraint scenario includes a strong compression rate constraint and a weak compression rate constraint; the strong compression rate constraint indicates that the channel compression rate of the model is at least N / M, M is the number of output channels of the target layer of the original network model, and N is the ratio of the number of output channels of the target layer of the compressed model; the weak compression rate constraint indicates that the channel compression rate of the model is at least (N+M) / 2M.

[0043] It is understandable that, for example Figure 2 As shown in the figure, in order to meet the model capabilities under different compression requirements, two different processes are provided in the subsequent matrix fusion and channel fusion. That is, in order to improve the flexibility of compression, different model fusion processes are used for the two scenarios of weak compression rate constraints and strong compression rate constraints to achieve a better balance between compression rate and accuracy loss. Specifically, the fusion process used under the weak compression rate constraint considers the channel similarity between models, the channel similarity within the compressed model, and the channel similarity within the original model when generating the fusion matrix, and updates the channel pruning mask vector according to the channel matching result after the network layer fusion, that is, when the channel compression rate allows, more channels participate in the fusion. The fusion process used under the strong compression rate constraint omits the calculation of the channel similarity within the original model to ensure that the channel compression rate of the fused weight is equal to the target value.

[0044] Step S14: obtaining a fused model based on the fused weights corresponding to different network layers, and using the fused model as the final compressed model of the original network model.

[0045] Finally, a fused model is obtained based on the fused weights corresponding to different network layers, and the fused model is used as the final compressed model of the original network model. In other words, the target layer refers to a certain layer in the network model. By taking each layer in the network model as the target layer in turn and performing steps S11-S13 on each layer, the fused weights corresponding to each layer are obtained. The above-mentioned final compressed model is deployed on a server or edge device for image recognition.

[0046] In some embodiments, the fused weight of the target layer is determined based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model, and the fused model is obtained based on the fused weights corresponding to different network layers, including: determining the current fused weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model; obtaining the target fused weight through moving average processing based on the current fused weight and the historical fused weight generated by historical random channel pruning of the target layer of the original network model, so as to obtain the fused model based on the target fused weights corresponding to different network layers. For example Figure 2 As shown, multiple random pruning, reconstruction and fusion will be performed for each layer of the network, that is, the operations of steps S11-S13. After each fusion weight is obtained, a moving average process will be performed based on the historical fusion weight of the layer until the random pruning, reconstruction and fusion of the layer reach the preset number of times, and the fusion weight obtained by the latest moving average process will be used as the target fusion weight. That is, the fusion weight:

[0047] ; where r represents the cumulative number of processing times for the current layer, Represents the fused weight obtained by the last moving average processing.

[0048] In the related art, the pruning method during inference uses a scaling process based on the kernel set theory to scale the parameter matrix of the original network layer when reconstructing the compressed parameter matrix, and adds the retained channels in the original network model to the compressed network model in the order of the channels; and then solves the compressed parameter matrix by optimizing the feature map reconstruction error. However, this method only scales the channels retained after pruning, without considering retaining the knowledge contained in the pruned channels, which will cause the loss of some image recognition knowledge. However, the present application considers the channel similarity matching between models when reconstructing the compressed weight matrix, superimposes and merges the paired channels, that is, the fusion of the original network model and the compressed network model, and retains the knowledge learned by the original network model on a large-scale data set through such fusion. Similarity metric learning and channel matching are performed in the channel space composed of each channel of the feature map of the compressed network layer and each channel of the feature map of the original network layer, and the paired channels are superimposed and fused to avoid knowledge loss, which is conducive to the deployment and application of image recognition tasks based on deep learning on edge devices with limited computing and storage resources. In addition, in the present application, the similarity of each channel of the activation feature map is also measured based on the curvelet transform and the comprehensive similarity of the structure.

[0049] Deep model fusion technology can merge the parameters or predictions of multiple network models that solve different tasks and have different structures into one model. In related technologies, a graph-based fusion method (ZipIt) is often used, but this method is only applicable to isomorphic network models, that is, each model to be fused has an equal number of channels in the corresponding layer. However, this application, through model fusion based on channel similarity, is not restricted by the model structure and can be applied to heterogeneous model fusion. By reducing the accuracy loss caused by network compression through model fusion, there is no need to rely on model retraining to restore the accuracy of the network model, and the construction efficiency of the image recognition network compression process itself is optimized.

[0050] As can be seen from the above, in this embodiment, the original network model is pruned layer by layer during inference, random input channel pruning is performed in sequence, and then the weights of the compressed model network layer are solved with the goal of optimizing the output feature map reconstruction error, and then the weights of the original network layer are fused with the weights of the compressed network layer through the model fusion process. Therefore, by analyzing the channel similarity measurement between models based on the curvelet transform feature similarity and structural similarity, matching channels are merged to avoid the loss of image recognition knowledge; generating a fusion matrix based on matching channel pairs, and performing model fusion through channel similarity measurement, it does not depend on the specific model architecture, so it can be applied to the fusion of heterogeneous models, improving the ability and effect of lightweight network models.

[0051] The embodiment of the present application discloses a specific method for lightweighting a network model under weak compression rate constraints, which may include the following steps:

[0052] Step S21: original network model The output channels of the layer (i.e., the target layer) are randomly pruned for the rth time to determine the compressed model corresponding to the original network model. The weight of the layer.

[0053] Step S22: Based on the original network model and the compressed model, The feature map output by the layer is analyzed according to the target similarity index to determine the channel matching result; the target similarity index includes structure and curvelet transform features, and the channel matching result represents a set of matched channel pairs.

[0054] Step S23: removing duplicate channel matching results from all channel matching results, and generating a fusion matrix based on the removed duplicate channel matching results.

[0055] For example Figure 3 As shown, after the channel similarity is calculated, the maximum similarity is selected row by row from the generated mean matrix (SSIM+NMI) / 2 to determine the channel matching result (u, v), and the fusion matrix T is generated based on the channel matching result after deduplication. After removing the duplicate matching results, the similarity vector S={si} with a dimension of (N+M) / 2 is obtained, that is, the similarity vector S is used to store the maximum similarity value selected from each row of the mean matrix (SSIM+NMI) / 2. , M is the original network model The number of output channels of the layer, N is the compressed model The number of output channels of the layer. The i-th channel matching result corresponds to the i-th row, u-th column, and v-th column of the fusion matrix T, with values ​​of 1 / 2, and the rest of the columns in the row have values ​​of 0.

[0056] That is, the channel matching result includes a first index and a second index, where u and v are the indexes of the two channels, used to identify similar or matching channel pairs found in the channel matching process, and the index is an integer from 0 to N+M-1, where N and M represent the sizes of the two channel sets. A fusion matrix T is generated based on the channel matching results. In the fusion matrix T, the values ​​of the elements in the i-th row, the u-th column, and the v-th column are 1 / 2, indicating that the two channels are matched in the i-th matching result. The values ​​of the elements in the remaining columns of the row are 0, indicating that these channels are not matched in the i-th matching result.

[0057] Step S24: Determine a new pruning mask vector according to the channel matching result, and prune the original network model according to the new pruning mask vector. Channel pruning is performed on the input channels of the next layer of the layer.

[0058] Since the number of output channels of the compressed network layer with weak compression rate is relaxed to (M+N) / 2, while the number of input channels of the next layer is M, the pruning mask vector must be re-determined according to the matching result in the case of weak compression rate to ensure that the number of output channels of the compressed network layer is equal to the number of input channels of the next layer.

[0059] In some embodiments, determining a new pruning mask vector based on the channel matching result includes: creating a zero vector for the target layer of the original network model; the dimension of the zero vector is half the sum of N and M; the channel matching result includes a first index and a second index; if the first index is less than N, and the second index is greater than or equal to N, then the difference between the second index and N is written into the zero vector; if the first index and the second index are both greater than or equal to N, then the difference between the first index and N, or the difference between the second index and N is written into the zero vector. If both the first index and the second index are less than N, then the non-zero value corresponding to the first index in the original mask vector, or the non-zero value corresponding to the second index in the original mask vector is written into the zero vector to obtain a new pruning mask vector; both the first index and the second index are less than N, indicating that they both belong to the first set and since they are both valid indexes, and the corresponding values ​​may all be non-zero, it is necessary to randomly select one of the two values, and the random selection is to break the symmetry or avoid preferring a specific index. The original mask vector is a pruning mask vector of the original network model. The random mask vector used for random channel pruning of the output channels of the layer.

[0060] That is, first, generate a zero vector of dimension (N+M) / 2 , which will be used to store the values ​​calculated based on the channel matching results. Update the vector based on each channel matching result (u,v) , u and v in the matching results are the indices of channels, and they may belong to two different sets, namely the first set of N channels and the second set of M channels. There are three cases according to the values of u and v: 1. u < N and N ≤ v, that is, u belongs to the first set and v belongs to the second set; at this time, is stored in the vector , and the index of v is adjusted to the relative index of the second set through . 2. u, v ≥ N, the indices are both greater than or equal to N, that is, both u and v belong to the second set; randomly select and and store one of the values in the vector . 3. u, v < N, both u and v belong to the first set (the indices are both less than N), randomly select one of the u-th value or the v-th value of the non-zero elements in the mask vector gr and store it in the vector . Traverse all channel matching results, and update the vector according to the above rules, as the new mask vector.

[0061] Step S25: Divide the fusion matrix into a first fusion matrix corresponding to the original network model and a second fusion matrix corresponding to the compressed model; the number of columns of the first fusion matrix is M, and the number of columns of the second fusion matrix is N; according to the product of the first fusion matrix and the weights of the layer of the original network model, the first product is obtained; according to the product of the second fusion matrix and the weights of the layer of the compressed model, the second product is obtained; according to the sum of the first product and the second product, the layer of the fused weights is obtained.

[0062] Divide the fusion matrix T into the second fusion matrix of the compressed model and the first fusion matrix of the original model respectively according to the number of output channels of the layer of the compressed model and the original model, that is, divide the fusion matrix T into columns and , , represents concatenation, , . The fused weights , are the weights of the layer of the compressed model, are the weights of the layer of the original model.

[0063] It is understandable that the goal of the weak compression constraint is to retain the image recognition knowledge of the original network model to the maximum extent, rather than strictly requiring the channel compression rate to reach a predetermined target value. Therefore, the channel compression rate is allowed to have a certain degree of flexibility and can be relaxed to a higher value. At this time, the fusion process not only considers the channel matching between models, but also the channel matching within the model. Directly multiplying the weight matrices of the original model and the compressed model with the corresponding fusion matrix through matrix multiplication, and then adding them to obtain the fused weight matrix, can make full use of the matching information within the model and across models, and retain the characteristics of the original model to the maximum extent.

[0064] Step S26: Obtain a fused model based on the fused weights corresponding to different network layers, and use the fused model as the final compressed model of the original network model.

[0065] Determine whether r has reached the preset number R, if not, continue to The layer is processed for the r+1th time. If the preset number of times is reached, the fused weight is used as the new compressed model. Layer weight. Traverse each network layer of the original network model until the fused weights of each network layer are determined to obtain the final compressed network model.

[0066] Among them, the specific processes of the above steps S21, S22, and S26 can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0067] As can be seen from the above, in this embodiment, duplicate channel matching results are removed from all channel matching results, and a fusion matrix is ​​generated based on the channel matching results after deduplication; the matching information within the model and across models is fully utilized to maximize the retention of the features of the original model. A new pruning mask vector is determined based on the channel matching results, and the original network model is pruned based on the new pruning mask vector. Channel pruning is performed on the input channels of the next layer of the layer; ensuring that the number of output channels of the compressed network layer is equal to the number of input channels of the next layer.

[0068] The embodiment of the present application discloses a specific method for lightweighting a network model under a strong compression rate constraint, which may include the following steps:

[0069] Step S31: original network model The output channels of the layer (i.e., the target layer) are randomly pruned to determine the compressed model corresponding to the original network model. The weight of the layer.

[0070] Step S32: Based on the original network model and the compressed model, The feature maps output by the layer analyze the similarity metric between channels according to the target similarity metric to determine the channel matching result; the target similarity metric includes structural and wavelet transform features, and the channel matching result represents a set of matching channel pairs.

[0071] Step S33: Classify the channel matching results, which are divided into cross-model matching results, in-model matching results of the original model, and in-model matching results of the compressed model; based on the cross-model matching results and the in-model matching results of the compressed model, generate a fusion matrix.

[0072] For example Figure 4 As shown, to ensure that the number of channels in the fused network layer is equal to the target number of compressed channels N, only fuse according to the channel matching results between models. For the channels involved in the in-model matching results of the compressed model, directly copy them to the fused model without fusion processing, and directly ignore the in-model matching results of the original model.

[0073] According to the mean matrix of the two similarity metrics (SSIM + NMI) / 2, select the maximum similarity row by row to determine the channel matching result (u, v), obtain the similarity vector S = {si} of dimension N, and generate the fusion matrix T. Among them , . If the v-th channel matching result is a cross-model match, that is, N ≤ u, the values of the u-th and v-th columns in the v-th row of the corresponding fusion matrix T are 1 / 2, and the remaining columns in this row are 0. If the v-th channel matching result is an in-model match of the compressed model, that is, u < N, the values of the u-th column in the u-th row and the v-th column in the v-th row of the corresponding fusion matrix T are 1, and the remaining columns in this row are 0. That is, in the case of a strong compression rate, relevant channels are selected from the original model for fusion with the compressed model, and irrelevant channels are directly ignored to avoid interference.

[0074] The following is an example. The following data is only for example and not actual data.

[0075] Let N = 3 (number of channels in the compressed model), M = 4 (number of channels in the original model), and the mean matrix (SSIM + NMI) / 2 be:

[0076] ;

[0077] Determine the channel matching results. The maximum value in the first row is 0.82, and the channel matching result is (u = 0, v = 0); the maximum value in the second row is 0.77, and the channel matching result is (u = 1, v = 1); the maximum value in the third row is 0.79, and the channel matching result is (u = 4, v = 2). For the first row, u = 0 < N, which is an internal match, T[0,0] = T[0,0] = 1, and the values of the remaining columns are 0. For the second row, u = 1 < N, T[1,1] = T[1,1] = 1, and the values of the remaining columns are 0. For the third row, u = 4 > N, T[2,2] = T[2,4] = 0.5, and the values of the remaining columns are 0. The final fusion matrix T is as follows:

[0078] .

[0079] Step S34: Divide the fusion matrix into a first fusion matrix corresponding to the original network model and a second fusion matrix corresponding to the compressed model; the number of columns of the first fusion matrix is M, and the number of columns of the second fusion matrix is N; based on the first fusion matrix, the second fusion matrix, the weights of the layers of the original network model, the weights of the layers of the compressed model, the feature maps output by the layers of the original network model, and the feature maps output by the layers of the compressed model, construct an optimization function representing the reconstruction error of the feature maps; obtain the fusion coefficients by solving the optimization function; based on the fusion coefficients, the first fusion matrix, the weights of the layers of the original network model, and the weights of the layers of the compressed model, determine the layer's fused weights.

[0080] First, divide the fusion matrix T into the second fusion matrix of the compressed model and the first fusion matrix of the original model respectively according to the number of output channels of the layer of the compressed model and the original model, that is, divide the fusion matrix by columns into and , , is a diagonal matrix.

[0081] Furthermore, solve the fusion coefficients by minimizing the reconstruction error of the feature maps, specifically, the optimization function can be solved by the least squares method, and the optimization function is:

[0082] , represents the norm, and finally the fusion coefficient vector is obtained.

[0083] Weight after fusion , the compressed model Layer weights Compared with the original model Layer weights Convert to fusion weight .

[0084] That is, the goal of the strong compression constraint is to strictly ensure that the number of channels in the fused network layer is equal to the target number of channels after compression N. In this case, the fusion process only considers the channel matching results between models. The matching results within the compressed model are directly copied to the fused model without fusion processing; the matching results within the original model are directly ignored. Since the strong constraint conditions limit the structure of the fusion matrix, the direct use of matrix multiplication may not meet the strict channel number requirements. Therefore, it is necessary to solve the fusion coefficient in an optimized way to minimize the feature map reconstruction error. By optimizing the fusion coefficient, it is ensured that the fused network retains the characteristics of the original model as much as possible while meeting the channel number requirements.

[0085] Step S35: Obtain a fused model based on the fused weights corresponding to different network layers, and use the fused model as the final compressed model of the original network model.

[0086] Among them, the specific processes of the above steps S31, S32, and S35 can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0087] As can be seen from the above, in this embodiment, a fusion matrix is ​​generated based on the cross-model matching results and the internal matching results of the compressed model; the compression rate of the fusion model is ensured to meet the standard. By constructing an optimization function that characterizes the reconstruction error of the feature map; obtaining the fusion coefficient by solving the optimization function; based on the fusion coefficient, the first fusion matrix, the original network model The weight of the layer, the compressed model The weight of the layer determines The fusion weights of the layers are optimized to ensure that the fused network retains the characteristics of the original model as much as possible while meeting the channel number requirements.

[0088] Through the description of the above implementation modes, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform.

[0089] The embodiment of the present application also provides a network model lightweight device, see Figure 5 As shown, the device comprises:

[0090] A random channel pruning module 11 is used to perform random channel pruning on the output channel of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model;

[0091] A channel matching module 12 is used to analyze the similarity metric between channels according to a target similarity index based on the feature maps outputted by the original network model and the compressed model at the target layer to determine a channel matching result; the target similarity index includes structure and curvelet transform features, and the channel matching result represents a set of matched channel pairs;

[0092] A post-fusion weight determination module 13 is used to generate a fusion matrix according to the channel matching result, and determine the post-fusion weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model;

[0093] The final compressed model determination module 14 is used to obtain a fused model based on the fused weights corresponding to different network layers, and use the fused model as the final compressed model of the original network model.

[0094] As can be seen from the above, in this embodiment, by analyzing the channel similarity measurement between models based on curvelet transform feature similarity and structural similarity, matching channels are merged, which can avoid the loss of image recognition knowledge; a fusion matrix is ​​generated based on matching channel pairs, and model fusion is performed through channel similarity measurement. This does not depend on the specific model architecture and is therefore suitable for the fusion of heterogeneous models, thereby improving the ability and effect of lightweight network models.

[0095] In some specific embodiments, the output of the original network model target layer is a first feature map, and the output of the compressed model target layer is a second feature map; the channel matching module 12 may specifically include:

[0096] a structural similarity calculation unit, configured to calculate the structural similarity between the channels of the first feature map and the channels of the second feature map, calculate the structural similarity between different channels in the first feature map, and calculate the structural similarity between different channels in the second feature map, and generate a structural similarity matrix;

[0097] a coarse-scale coefficient similarity calculation unit, used to calculate the normalized mutual information between the channel feature map of the first feature map and the channel feature map of the second feature map on the coarse-scale coefficient of the curvelet transform, calculate the normalized mutual information of different channel feature maps of the first feature map on the coarse-scale coefficient of the curvelet transform, and the normalized mutual information of different channel feature maps of the second feature map on the coarse-scale coefficient of the curvelet transform, and generate a normalized mutual information matrix;

[0098] A mean matrix generating unit is used to generate a mean matrix based on the structural similarity matrix and the normalized mutual information matrix, and determine a channel matching result according to the mean matrix.

[0099] In some specific embodiments, the mean matrix generating unit may specifically include:

[0100] The channel matching result generating unit is used to screen out the maximum value in each row of the mean matrix, and obtain the channel matching result according to the row and column indexes corresponding to the maximum value.

[0101] In some specific embodiments, the random channel pruning module 11 may specifically include:

[0102] A random mask vector generating unit, used to generate a random mask vector for an output channel of a target layer of the original network model, and perform channel pruning on the output channel of the target layer according to the random mask vector;

[0103] A weight determination unit is used to determine the weight of the compressed model target layer by solving the minimum error using the least squares method based on the feature map output by the target layer of the original network model and the feature map output by the target layer of the compressed model.

[0104] In some specific embodiments, the network model lightweight device may specifically include:

[0105] A constraint scenario acquisition unit is used to acquire a network lightweight constraint scenario before generating a fusion matrix according to the channel matching result; the network lightweight constraint scenario includes a strong compression rate constraint and a weak compression rate constraint; the strong compression rate constraint indicates that the channel compression rate of the model is at least N / M, where M is the number of output channels of the target layer of the original network model and N is the ratio of the number of output channels of the target layer of the compressed model; the weak compression rate constraint indicates that the channel compression rate of the model is at least (N+M) / 2M;

[0106] In some specific embodiments, if the network lightweight constraint scenario is the weak compression rate constraint, the post-fusion weight determination module 13 may specifically include:

[0107] The first fusion matrix generating unit is used to remove duplicate channel matching results from all channel matching results, and generate a fusion matrix based on the channel matching results after removal of duplicates.

[0108] In some specific embodiments, if the network lightweight constraint scenario is the weak compression rate constraint, the post-fusion weight determination module 13 may specifically include:

[0109] A matrix division unit, used to divide the fusion matrix into a first fusion matrix corresponding to the original network model and a second fusion matrix corresponding to the compressed model; the number of columns of the first fusion matrix is ​​M, and the number of columns of the second fusion matrix is ​​N;

[0110] A first product determining unit, configured to obtain a first product according to the product of the first fusion matrix and the weight of the target layer of the original network model;

[0111] A second product determining unit, configured to obtain a second product according to the product of the second fusion matrix and the weight of the target layer of the compressed model;

[0112] A fused weight determination unit is used to obtain the fused weight of the target layer according to the sum of the first product and the second product.

[0113] In some specific embodiments, if the network lightweight constraint scenario is the weak compression rate constraint, the network model lightweight device may specifically include:

[0114] A new pruning mask vector determining unit is used to determine a new pruning mask vector according to the channel matching result after generating a fusion matrix, and perform channel pruning on the input channel of the next layer of the target layer of the original network model according to the new pruning mask vector.

[0115] In some specific embodiments, the new pruning mask vector determining unit may specifically include:

[0116] A zero vector creation unit, used to create a zero vector for the target layer of the original network model; the dimension of the zero vector is half of the sum of N and M; the channel matching result includes a first index and a second index;

[0117] a first writing unit, configured to write a difference between the second index and N into the zero vector if the first index is less than N and the second index is greater than or equal to N;

[0118] a second writing unit, configured to write a difference between the first index and N, or a difference between the second index and N, into the zero vector if both the first index and the second index are greater than or equal to N;

[0119] The third writing unit is used to write the non-zero value corresponding to the first index in the original mask vector, or the non-zero value corresponding to the second index in the original mask vector, into the zero vector if both the first index and the second index are less than N, so as to obtain a new pruning mask vector; the original mask vector is a random mask vector used when performing random channel pruning on the output channels of the target layer of the original network model.

[0120] In some specific embodiments, if the network lightweight constraint scenario is the strong compression rate constraint, the post-fusion weight determination module 13 may specifically include:

[0121] A classification unit, used to classify the channel matching results, wherein the channel matching results are divided into cross-model matching results, original model internal matching results, and compressed model internal matching results;

[0122] The second fusion matrix generating unit is used to generate a fusion matrix based on the cross-model matching result and the internal matching result of the compressed model.

[0123] In some specific embodiments, if the network lightweight constraint scenario is the strong compression rate constraint, the post-fusion weight determination module 13 may specifically include:

[0124] A matrix division unit, used to divide the fusion matrix into a first fusion matrix corresponding to the original network model and a second fusion matrix corresponding to the compressed model; the number of columns of the first fusion matrix is ​​M, and the number of columns of the second fusion matrix is ​​N;

[0125] An optimization function construction unit, used to construct an optimization function representing a feature map reconstruction error based on the first fusion matrix, the second fusion matrix, the weight of the target layer of the original network model, the weight of the target layer of the compressed model, the feature map output by the target layer of the original network model, and the feature map output by the target layer of the compressed model;

[0126] A fusion coefficient solving unit, used for obtaining a fusion coefficient by solving the optimization function;

[0127] A fused weight determination unit is used to determine the fused weight of the target layer based on the fusion coefficient, the first fusion matrix, the weight of the target layer of the original network model, and the weight of the target layer of the compressed model.

[0128] In some specific embodiments, the post-fusion weight determination module 13 may specifically include:

[0129] A current fused weight determining unit, used to determine the current fused weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model;

[0130] The moving average processing unit is used to obtain the target fused weight through moving average processing based on the current fused weight and the historical fused weight generated by the historical random channel pruning of the target layer of the original network model, so as to obtain the fused model based on the target fused weights corresponding to different network layers.

[0131] For the description of the features in the embodiment corresponding to the network model lightweight device, please refer to the relevant description of the embodiment corresponding to the network model lightweight method, which will not be repeated here.

[0132] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned network model lightweight method embodiments.

[0133] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned network model lightweight method embodiments when running.

[0134] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0135] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned network model lightweight method embodiments are implemented.

[0136] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned network model lightweight method embodiments are implemented.

[0137] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] The above is a detailed introduction to a network model lightweight method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A network model lightweight method, characterized in that: include: Performing random channel pruning on the output channels of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model; Based on the feature maps outputted by the original network model and the compressed model at the target layer respectively, analyzing the similarity metric between channels according to the target similarity index to determine the channel matching result; the target similarity index includes structure and curvelet transform features, and the channel matching result represents a set of matched channel pairs; Generate a fusion matrix according to the channel matching result, and determine the fusion weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model; A fused model is obtained based on the fused weights corresponding to different network layers, and the fused model is used as the final compressed model of the original network model.

2. The network model lightweight method according to claim 1, characterized in that: Based on the feature maps outputted by the original network model and the compressed model at the target layer, respectively, the channel similarity metric is analyzed according to the target similarity index to determine the channel matching result, including: The output of the target layer of the original network model is a first feature map, and the output of the target layer of the compressed model is a second feature map; Calculating the structural similarity between the channels of the first feature map and the channels of the second feature map, calculating the structural similarity between different channels in the first feature map, and calculating the structural similarity between different channels in the second feature map, and generating a structural similarity matrix; Calculating the normalized mutual information of the channel feature map of the first feature map and the channel feature map of the second feature map on the coarse-scale coefficient of the curvelet transform, calculating the normalized mutual information of different channel feature maps of the first feature map on the coarse-scale coefficient of the curvelet transform, and the normalized mutual information of different channel feature maps of the second feature map on the coarse-scale coefficient of the curvelet transform, and generating a normalized mutual information matrix; A mean matrix is ​​generated based on the structural similarity matrix and the normalized mutual information matrix, and a channel matching result is determined according to the mean matrix.

3. The network model lightweight method according to claim 2, characterized in that: Determining a channel matching result according to the mean matrix includes: The maximum value in each row of the mean matrix is ​​screened out, and the channel matching result is obtained according to the row and column indexes corresponding to the maximum value.

4. The network model lightweight method according to claim 1, characterized in that: Performing random channel pruning on the output channels of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model includes: Generate a random mask vector for the output channel of the target layer of the original network model, and perform channel pruning on the output channel of the target layer according to the random mask vector; According to the feature map output by the target layer of the original network model and the feature map output by the target layer of the compressed model, the least squares method is used to minimize the error to determine the weight of the target layer of the compressed model.

5. The network model lightweight method according to claim 1, characterized in that: Before generating the fusion matrix according to the channel matching result, the method further includes: Obtain a network lightweight constraint scenario; the network lightweight constraint scenario includes a strong compression rate constraint and a weak compression rate constraint; the strong compression rate constraint indicates that the channel compression rate of the model is at least N / M, where M is the number of output channels of the target layer of the original network model, and N is the ratio of the number of output channels of the target layer of the compressed model; the weak compression rate constraint indicates that the channel compression rate of the model is at least (N+M) / 2M; If the network lightweight constraint scenario is the weak compression rate constraint, a fusion matrix is ​​generated according to the channel matching result, including: The duplicate channel matching results are removed from the channel matching results, and a fusion matrix is ​​generated based on the duplicated channel matching results.

6. The network model lightweight method according to claim 5, characterized in that: If the network lightweight constraint scenario is the weak compression rate constraint, the fused weight of the target layer is determined based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model, including: Dividing the fusion matrix into a first fusion matrix corresponding to the original network model and a second fusion matrix corresponding to the compressed model; the number of columns of the first fusion matrix is ​​M, and the number of columns of the second fusion matrix is ​​N; Obtaining a first product according to the product of the first fusion matrix and the weight of the target layer of the original network model; Obtaining a second product according to the product of the second fusion matrix and the weight of the target layer of the compressed model; A fused weight of the target layer is obtained according to the sum of the first product and the second product.

7. The network model lightweight method according to claim 5, characterized in that: If the network lightweight constraint scenario is the weak compression rate constraint, after generating the fusion matrix, the following is also included: A new pruning mask vector is determined according to the channel matching result, and channel pruning is performed on the input channels of the next layer of the target layer of the original network model according to the new pruning mask vector.

8. The network model lightweight method according to claim 7, characterized in that: Determining a new pruning mask vector according to the channel matching result includes: creating a zero vector for the target layer of the original network model; the dimension of the zero vector is half of the sum of N and M; The channel matching result includes a first index and a second index; If the first index is less than N, and the second index is greater than or equal to N, then the difference between the second index and N is written into the zero vector; If both the first index and the second index are greater than or equal to N, the difference between the first index and N, or the difference between the second index and N, is written into the zero vector; If both the first index and the second index are less than N, the non-zero value corresponding to the first index in the original mask vector, or the non-zero value corresponding to the second index in the original mask vector, is written into the zero vector to obtain a new pruning mask vector; the original mask vector is a random mask vector used for random channel pruning of the output channels of the target layer of the original network model.

9. The network model lightweight method according to claim 5, characterized in that: If the network lightweight constraint scenario is the strong compression rate constraint, a fusion matrix is ​​generated according to the channel matching result, including: Classifying the channel matching results, wherein the channel matching results are divided into cross-model matching results, original model internal matching results, and compressed model internal matching results; A fusion matrix is ​​generated based on the cross-model matching results and the compressed model internal matching results.

10. The network model lightweight method according to claim 5, characterized in that: If the network lightweight constraint scenario is the strong compression rate constraint, the fused weight of the target layer is determined based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model, including: Dividing the fusion matrix into a first fusion matrix corresponding to the original network model and a second fusion matrix corresponding to the compressed model; the number of columns of the first fusion matrix is ​​M, and the number of columns of the second fusion matrix is ​​N; Based on the first fusion matrix, the second fusion matrix, the weight of the target layer of the original network model, the weight of the target layer of the compressed model, the feature map output by the target layer of the original network model, and the feature map output by the target layer of the compressed model, an optimization function representing the reconstruction error of the feature map is constructed; Obtaining a fusion coefficient by solving the optimization function; Based on the fusion coefficient, the first fusion matrix, the weight of the target layer of the original network model, and the weight of the target layer of the compressed model, the fused weight of the target layer is determined.

11. The network model lightweight method according to any one of claims 1 to 10, characterized in that: Determining the fused weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model, and obtaining the fused model based on the fused weights corresponding to different network layers, including: Determine the current fused weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model; Based on the current fused weights and the historical fused weights generated by historical random channel pruning of the target layer of the original network model, the target fused weights are obtained by moving average processing, so as to obtain a fused model based on the target fused weights corresponding to different network layers.

12. A network model lightweight device, characterized in that: include: A random channel pruning module, used to perform random channel pruning on the output channel of the target layer of the original network model to determine the weight of the target layer of the compressed model corresponding to the original network model; A channel matching module, configured to analyze the inter-channel similarity metric according to a target similarity index based on the feature maps outputted at the target layer by the original network model and the compressed model respectively, to determine a channel matching result; the target similarity index includes structure and curvelet transform features, and the channel matching result represents a set of matched channel pairs; A post-fusion weight determination module, used to generate a fusion matrix according to the channel matching result, and determine the post-fusion weight of the target layer based on the weight of the target layer of the original network model and the weight of the target layer of the compressed model; The final compressed model determination module is used to obtain a fused model based on the fused weights corresponding to different network layers, and use the fused model as the final compressed model of the original network model.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the network model lightweight method as described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the network model lightweighting method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the network model lightweighting method as described in any one of claims 1 to 11 are implemented.

Images