Convolutional neural network pruning method and device based on gram matrix orthogonality

Abstract

This invention provides a method and apparatus for pruning convolutional neural networks based on the orthogonality of Gram matrices, relating to the technical field of convolutional neural network pruning. The method includes: inputting feature map training samples into the current convolutional neural network to obtain output features; for each convolutional layer, determining the channel-dimensional parameter Gram matrix (elements representing the correlation between channel convolutional kernels), and calculating the saliency score of its diagonal elements; calculating a comprehensive loss function based on the saliency score and optimizing network parameters; iterating the above steps until the comprehensive loss function meets expectations, obtaining the target saliency score and the target pruned features; and performing pruning on the trained network based on both. This invention measures the importance of channels by using the saliency score determined by the diagonal elements of the Gram matrix, and guides training by combining it with the comprehensive loss function, effectively improving the trainability of the pruned network.

Description

Technical Field

[0001] This invention relates to the technical field of convolutional neural network pruning, and in particular to a convolutional neural network pruning method and apparatus based on the orthogonality of Gram matrices. Background Technology

[0002] Deep convolutional neural networks (CNNs) have achieved remarkable success in fields such as computer vision and natural language processing due to their powerful feature extraction and pattern recognition capabilities. As model architectures become increasingly complex, modern deep neural networks typically contain millions to billions of parameters. While models of this scale exhibit outstanding performance in tasks such as image classification and object detection, they also bring high computational costs, massive storage requirements, and significant energy consumption issues. This resource consumption makes it difficult to deploy large neural networks in resource-constrained environments such as mobile devices and embedded systems.

[0003] To address this challenge, model compression techniques have emerged, with neural network pruning becoming a research hotspot due to its effectiveness. Pruning techniques analyze the importance of network parameters, identifying and removing redundant weights, neurons, or entire channels to generate a more streamlined model structure. Channel pruning, in particular, has become a widely adopted method in practice due to its ability to produce structured sparse models and its good hardware compatibility and versatility. Existing pruning methods have demonstrated that by appropriately removing redundant structures, computational and storage overhead can be significantly reduced while maintaining model performance.

[0004] However, pruning techniques face a key problem: pruned models often require complex retraining processes to recover their performance. This is because pruning not only removes redundant parameters but also profoundly alters the network structure and parameter distribution, destroying the optimized characteristics acquired through careful training of the original model. This structural abrupt change leads to a significant alteration in the loss function surface of the pruned model, making it difficult to effectively optimize using conventional training methods, resulting in a substantial drop in model accuracy. Current research generally observes that even using the same hyperparameters and optimization strategies as the original training, the trainability of the pruned model is still significantly reduced. This phenomenon severely restricts the practical application of pruning techniques, especially in scenarios with stringent accuracy requirements. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method and apparatus for pruning convolutional neural networks based on the orthogonality of Gram matrices, so as to alleviate the technical problem of low trainability of pruned convolutional neural networks.

[0006] In a first aspect, the present invention provides a convolutional neural network pruning method based on the orthogonality of Gram matrices, comprising: The feature map training samples are input into the current convolutional neural network to obtain the output features; Based on each convolutional layer of the current convolutional neural network, a parameter Gram matrix of the channel dimension is determined, and the saliency score of the diagonal elements in the parameter Gram matrix is ​​calculated; wherein, the elements in the parameter Gram matrix are used to characterize the correlation of the channel convolution kernels of each convolutional layer; The comprehensive loss function corresponding to the current convolutional neural network is calculated based on the saliency score, and the corresponding parameters of the current convolutional neural network are optimized. The optimized convolutional neural network is used as the new current convolutional neural network, and the above steps are repeated until the comprehensive loss function meets the expectations, and then the target saliency score and target pruned features corresponding to the trained convolutional neural network are obtained. The trained convolutional neural network is pruned based on the target saliency score or the target pruned features.

[0007] In an optional implementation, the step of determining the channel-dimensional parameter Gram matrix for each convolutional layer of the current convolutional neural network includes: The parameter gram matrix is ​​calculated for each convolutional layer of the current convolutional neural network according to the channel dimension. The elements in each row of the parameter gram matrix, excluding the diagonal elements, are used to characterize the correlation between each channel convolutional kernel of each convolutional layer and other channel convolutional kernels. The diagonal elements are used to characterize the correlation between each channel convolutional kernel and itself.

[0008] In an optional implementation, the step of calculating the significance score of the diagonal elements in the parameter Gram matrix includes: Normalize the elements in each row of the parameter Gram matrix to obtain the element score of each row of the parameter Gram matrix excluding the diagonal elements, as well as the significance score of the diagonal elements.

[0009] In an optional implementation, the step of calculating the significance scores of the diagonal elements in the parameter Gram matrix further includes: The significance scores of the diagonal elements are input into the activation function and multiplied by the amplification factor corresponding to the current step size for progressive discretization to obtain the significance scores; wherein the amplification factor corresponding to the next step size is greater than the amplification factor of the current step size.

[0010] In an optional implementation, the step of calculating the comprehensive loss function corresponding to the current convolutional neural network based on the saliency score includes: Based on the comparison results between the saliency score and the preset threshold range, the differential control strategy for pruning channels and retaining channels in the current convolutional neural network is determined; If the significance score is higher than the upper limit of the preset threshold range, then the first trainable protection loss function used to constrain the off-diagonal elements of the channel corresponding to the significance score in the parameter Gram matrix to be 0 and the diagonal elements to be 1, and the original task loss function are selected for weighted calculation. If the significance score is lower than the lower limit of the preset threshold range, then the second trainable protection loss function used in the comprehensive loss function to constrain all elements of the channel corresponding to the significance score in the parameter Gram matrix to be 0, and the original task loss function are selected for weighted calculation. If the significance score is within the preset threshold range, then the original task loss function in the comprehensive loss function is selected for calculation.

[0011] In an optional implementation, the step of optimizing the corresponding parameters of the current convolutional neural network includes: Based on the pruned features obtained by multiplying the saliency score and the output features, the pruned channels and retained channels of the current convolutional neural network are determined. The parameters of the current convolutional neural network are optimized based on the pruned channels and the retained channels.

[0012] In an optional implementation, the step of pruning the trained convolutional neural network based on the target saliency score or the target pruned features includes: Based on the target saliency score or the target pruned features, determine the parameters corresponding to the pruned channels of the trained convolutional neural network; Based on the parameters corresponding to the pruned channels, pruning operations are performed on the corresponding channels of the trained convolutional neural network.

[0013] Secondly, the present invention provides a current convolutional neural network pruning device based on the orthogonality of Gram matrices, comprising: The training module inputs feature map training samples into the current convolutional neural network to obtain output features; based on each convolutional layer of the current convolutional neural network, it determines the parameter Gram matrix of the channel dimension and calculates the saliency score of the diagonal elements in the parameter Gram matrix; wherein, the elements in the parameter Gram matrix are used to characterize the correlation of the channel convolutional kernels of each convolutional layer; based on the saliency score, it calculates the comprehensive loss function corresponding to the current convolutional neural network and optimizes the corresponding parameters of the current convolutional neural network; the optimized convolutional neural network is used as the new current convolutional neural network, and the above steps are repeated until the comprehensive loss function meets the expectation, thus terminating to obtain the target saliency score and target pruned features corresponding to the trained convolutional neural network; The pruning module performs pruning operations on the trained convolutional neural network based on the target saliency score or the target pruned features.

[0014] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the method described in any of the foregoing embodiments.

[0015] Fourthly, the present invention provides a machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the steps of the method described in any of the foregoing embodiments.

[0016] This invention provides a convolutional neural network pruning method and apparatus based on the orthogonality of Gram matrices. Each convolutional layer generates a channel-dimensional Gram matrix. The orthogonality of this matrix reflects the correlation between the channel kernels of the convolutional layer and themselves, as well as with other channel kernels. Based on this, the saliency scores of the diagonal elements of the Gram matrix are obtained, which are used to calculate the comprehensive loss function and optimize the parameters of the convolutional network until the comprehensive loss meets the expected training outcome. At this point, the pruned features and target saliency scores of the trained convolutional neural network are used to prune the corresponding channels in the trained network. This process yields a convolutional neural network with better trainability after pruning.

[0017] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0018] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart of a convolutional neural network pruning method based on the orthogonality of Gram matrices provided in this embodiment of the invention; Figure 2A schematic diagram illustrating the training process of a convolutional neural network based on the orthogonality of Gram matrices, provided as an embodiment of the present invention; Figure 3 A functional block diagram of a convolutional neural network pruning device based on the orthogonality of Gram matrices provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The current pruning operation itself significantly changes the structure and parameter distribution of the model, destroys the original network's optimization characteristics, and leads to a decrease in the trainability of the pruned model, which in turn leads to a significant decrease in the accuracy of the compressed model.

[0023] Based on this, the present invention provides a current convolutional neural network pruning method and apparatus based on the orthogonality of Gram matrices. By establishing a Gram matrix of the channel dimension of the convolutional neural network, the orthogonality of the matrix is ​​used to decouple the pruned channels and the retained channels, improve the trainability of the model parameters after pruning, and thus improve the model accuracy.

[0024] To facilitate understanding of this embodiment, a current convolutional neural network pruning method based on the orthogonality of Gram matrices, disclosed in this embodiment of the invention, will first be described in detail. This method can be applied to intelligent control devices such as host computers, servers, and controllers.

[0025] Figure 1 The flowchart illustrates a current convolutional neural network pruning method based on the orthogonality of Gram matrices, as provided in this embodiment of the invention.

[0026] like Figure 1 As shown, the method includes the following steps: Step S102: Input the feature map training samples into the current convolutional neural network to obtain the output features.

[0027] Here, as Figure 2 As shown, training samples with feature map patterns are input into the current convolutional neural network, and the features are output after being processed by each convolutional layer of the current convolutional neural network.

[0028] Step S104: Based on each convolutional layer of the current convolutional neural network, determine the parameter gram matrix of the channel dimension, and calculate the saliency score of the diagonal elements in the parameter gram matrix.

[0029] It should be noted that the elements in the parameter Gram matrix in this embodiment of the invention are used to characterize the correlation of the channel convolution kernels of each convolutional layer. Specifically, step S104, which involves determining the channel-dimensional parameter Gram matrix for each convolutional layer of the current convolutional neural network, includes: The parameter Gram matrix of the current convolutional neural network is calculated according to the channel dimension for each convolutional layer.

[0030] For example, each convolutional layer of a convolutional neural network calculates a parameter gram matrix for each channel dimension. Each row element in the gram matrix (excluding diagonal elements) represents the correlation between the corresponding channel convolutional kernel and other channel convolutional kernels, while the diagonal elements correspond to the correlation between the convolutional kernel and itself.

[0031] To protect the trainability of the pruned model and improve its accuracy, the pruned and retained channels need to be decoupled to prevent the retained channels from being affected by the pruned channels. This is achieved by controlling the orthogonality of the channel Gram matrix, that is, ensuring that all elements in the Gram matrix except for the diagonal elements are constrained to 0, while ensuring that the elements corresponding to the retained channels in the diagonal elements are constrained to 1, and the elements corresponding to the pruned channels in the diagonal elements are constrained to 0.

[0032] The Gram matrix has positive intersection. This embodiment of the invention utilizes this property to associate the Gram matrix with each convolutional layer of the convolutional neural network. The elements in each row of the parameter Gram matrix, excluding the diagonal elements, are used to characterize the correlation between each channel convolutional kernel of each convolutional layer and other channel convolutional kernels, while the diagonal elements are used to characterize the correlation between each channel convolutional kernel and itself.

[0033] In some embodiments, the step S104 of calculating the significance scores of the diagonal elements in the parameter Gram matrix includes: Step 1.1: Normalize the elements of each row in the parameter Gram matrix to obtain the element score of each row excluding the diagonal elements and the significance score of the diagonal elements.

[0034] To determine the parameters of redundant channels, this embodiment of the invention normalizes the elements of the Gram matrix row by row using the softmax method, so that the sum of the scores of the elements in each row of the Gram matrix is ​​1. The larger the significance score of the diagonal elements, the lower the correlation between the corresponding channel and other channels, and the stronger the significance.

[0035] It should be noted that the saliency of a channel is defined as its "correlation" or "independence" with other channels by calculating the Gram matrix between the channel convolution kernels. The higher the saliency score of the diagonal elements, the more unique the channel is, the lower its information redundancy, and therefore the more important it is. Simultaneously, the rows of the Gram matrix are normalized using softmax, transforming the saliency score into a probability distribution, making the importance judgment smoother and more stable, and facilitating integration with subsequent optimization processes.

[0036] Step 1.2: Input the significance score of the diagonal elements into the activation function and multiply it by the amplification factor corresponding to the current step size to gradually discretize and obtain the significance score; where the amplification factor corresponding to the next step size is greater than the amplification factor of the current step size.

[0037] To achieve adaptive pruning based on saliency scores and optimize trainability while preserving parameters, the saliency scores of channels need to be gradually discretized into binary values ​​of 0 and 1 during training. However, to avoid gradient backpropagation failure caused by directly using discrete activation functions, a gradually discretized sigmoid activation function is adopted. The saliency score of the activation input is multiplied by an amplification factor, and the amplification factor is gradually increased with a fixed step size during training to achieve the gradual discretization of the saliency scores into binary values ​​of 0 and 1.

[0038] This invention provides a differentiable and adaptive discrete pruning mechanism that uses a gradually discretized sigmoid activation function to handle saliency scores. An amplification factor is controlled to gradually increase with each training iteration, smoothly and progressively discretizing the saliency score from a continuous value to a binary state of 0 or 1. This process is fully differentiable, allowing gradient backpropagation, thus achieving joint optimization of pruning decisions and network parameter updates. Furthermore, the final pruning rate is not a pre-set fixed hyperparameter but is adaptively learned through model training, making it more adaptable to different network structures and datasets.

[0039] Step S106: Calculate the comprehensive loss function corresponding to the current convolutional neural network based on the saliency score, and optimize the corresponding parameters of the current convolutional neural network.

[0040] Based on the aforementioned embodiments, network parameters can be adjusted based on the comprehensive loss calculated from the saliency score to improve network performance, including: Step 2.1: Based on the comparison results of the saliency score and the preset threshold range, determine the differential control strategy for pruning and retaining channels in the current convolutional neural network.

[0041] This invention protects the trainability of pruning by using Gram matrix orthogonal constraints. Based on the significance scores of the diagonal elements after activation, the parameters of the corresponding retained and pruned channels in the Gram matrix can be constrained according to their discretized binary states, thus maintaining the orthogonality of the matrix, decoupling the pruned and retained channels, and improving the trainability of the parameters.

[0042] Step 2.2: If the significance score is higher than the upper limit of the preset threshold range, then the first trainable protective loss function, which is used to constrain the off-diagonal elements of the channel corresponding to the significance score in the Gram matrix of the constraint parameters, is selected in the comprehensive loss function and the original task loss function is weighted for calculation; wherein, the first trainable protective loss function is as follows:

[0043] in, The first trainability protection loss function term in the comprehensive loss function is used to constrain the network to maintain trainability during channel pruning, and corresponds to the constraint logic of different significance score intervals. The weight matrix to be pruned in the l-th layer (where "1" can be understood as the identifier of the weights in this layer, or it can refer to a specific weight submatrix). yes The transpose of the matrix; Channel saliency score matrix, where elements This represents the saliency score of the c-th channel and the i-th position (or dimension); L represents the number of layers in the network, and this constraint is calculated and summed for each layer.

[0044] The core objective is to constrain the feature independence of "potentially reserved channels". When the channel significance score... Exceeding the threshold At that time, the channel was considered a "potentially reserved channel". By constraining the off-diagonal elements (cross-channel correlation) of a channel in the Gram matrix to approach 0, while ensuring the stability of the diagonal elements (its own feature strength), the features of the retained channels are made more independent (reducing redundant correlations with other channels), thus avoiding a decrease in feature expressive power after pruning.

[0045] Step 2.3: If the significance score is lower than the lower limit of the preset threshold range, then the second trainable protective loss function, which is used to constrain all elements of the channel corresponding to the significance score in the Gram matrix of the parameter, is selected in the comprehensive loss function, and the original task loss function is weighted and calculated; wherein, the first trainable protective loss function is as follows:

[0046] in, The second trainability protection loss function term in the comprehensive loss function is used to constrain the network to maintain trainability during channel pruning, and corresponds to the constraint logic of different significance score intervals; The weight matrix to be pruned in the l-th layer (where "1" can be understood as the identifier of the weights in this layer, or it can refer to a specific weight submatrix). yes The transpose of the matrix; Channel saliency score matrix, where elements This represents the saliency score of the c-th channel and the i-th position (or dimension); L represents the number of layers in the network, and this constraint is calculated and summed for each layer.

[0047] The core objective is to constrain the decay of the characteristic correlation of "potential pruning channels"; when the channel significance score Below At that time, the channel was considered a "potential pruning channel". By constraining all elements of a channel in the Gram matrix (including itself and cross-channels) to approach 0, its feature representation ability and correlation with other channels are gradually weakened, achieving a "soft pruning" effect and avoiding training oscillations caused by directly deleting channels.

[0048] Step 2.4: If the significance score is within the preset threshold range, then select the original task loss function in the comprehensive loss function for calculation.

[0049] The channel saliency score is within the preset threshold range < < In this case, the weight coefficients of the first and second trainable protection loss function terms in the comprehensive loss function can be lowered or set to 0 simultaneously to achieve the goal of selecting only the original task loss function in the comprehensive loss function for calculation. Similarly, the method of selecting the first or second trainable protection loss function term in the previous steps can be achieved by adjusting the weight coefficients of the first or second trainable protection loss function term to select or filter the function term. The selected function term is then weighted and calculated with the original task loss function to achieve joint optimization of the trainable protection constraint and the task loss function of the original network, thus achieving end-to-end adaptive completion of the entire process.

[0050] In some embodiments, the step of optimizing the corresponding parameters of the current convolutional neural network in step S106 includes: Step 3.1: Based on the saliency score and the output features, the pruned features are obtained by multiplying them to determine the pruned and retained channels of the current convolutional neural network.

[0051] Here, as Figure 2 As shown, multiplying the saliency score by the original channel output features can determine the pruned channel.

[0052] Step 3.2: Optimize the corresponding parameters of the current convolutional neural network based on the pruned and preserved channels.

[0053] Here, by removing the parameters corresponding to the pruned channels and outputting the pruned features, the propagation of the pruned channel features is prevented.

[0054] Step S108: Use the optimized convolutional neural network as the new current convolutional neural network, and repeat steps S102-S106 until the comprehensive loss function meets expectations, and then terminate to obtain the target saliency score and target pruned features corresponding to the trained convolutional neural network.

[0055] It is understandable that a trained convolutional neural network can output a target saliency score or target pruned features.

[0056] Step S110: Prune the trained convolutional neural network based on the target saliency score or the target pruned features.

[0057] For example, step S110 can be implemented by the following steps, which ensure the trainability of the network model after pruning, specifically including: Step 4.1: Based on the target saliency score or the target pruned features, determine the corresponding parameters of the pruned channels of the trained convolutional neural network.

[0058] Step 4.2: Prune the corresponding channels of the trained convolutional neural network according to the parameters of the pruned channels.

[0059] The embodiments of the present invention have at least the following significant advantages in neural network-based model pruning techniques: 1. Improved model accuracy after pruning: By controlling the orthogonality of the weight Gram matrix, the decoupling of the pruned and retained channels can be effectively achieved, improving the trainability of the model parameters after pruning and thus improving the model accuracy.

[0060] 2. Enhanced Adaptability: By utilizing a differentiable discrete activation function, the saliency scores of the channels are gradually discretized during training. Under the joint optimization of loss constraints based on Gram matrix orthogonality and the original task loss function, the pruning and retention channels of each layer are adaptively determined without manual intervention.

[0061] 3. Low overall training cost: By integrating the pruning process with the training process, the significant training cost of the three steps of pre-training the model, pruning, and fine-tuning in traditional pruning methods is avoided.

[0062] In some embodiments, such as Figure 3 As shown, this embodiment of the invention also provides a current convolutional neural network pruning device 200 based on the orthogonality of Gram matrices, comprising: Training module 201 inputs feature map training samples into the current convolutional neural network to obtain output features; based on each convolutional layer of the current convolutional neural network, it determines the parameter Gram matrix of the channel dimension and calculates the saliency score of the diagonal elements in the parameter Gram matrix; wherein, the elements in the parameter Gram matrix are used to characterize the correlation of the channel convolutional kernels of each convolutional layer; based on the saliency score, it calculates the comprehensive loss function corresponding to the current convolutional neural network and optimizes the corresponding parameters of the current convolutional neural network; the optimized convolutional neural network is used as the new current convolutional neural network, and the above steps are repeated until the comprehensive loss function meets the expectation, and then terminates to obtain the target saliency score and target pruned features corresponding to the trained convolutional neural network; The pruning module 202 performs pruning operations on the trained convolutional neural network based on the target saliency score or the target pruned features.

[0063] Figure 4 This is a schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. See also... Figure 4 As shown, the electronic device 300 includes a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304, and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303, and the communication interface 304 communicate with each other through the bus 305. The processor 302 can execute the convolutional neural network pruning method based on Gram matrix orthogonality described in the above embodiments by reading and executing the machine-executable instructions for Gram matrix orthogonality-based convolutional neural network pruning in the machine-readable storage medium 301.

[0064] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0065] Non-volatile media can be non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar non-volatile storage media, or combinations thereof.

[0066] It is understood that the specific operation methods of each functional module in this embodiment can be referred to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0067] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program. When the computer program code is executed, it can implement the convolutional neural network pruning method based on the orthogonality of Gram matrix described in any of the above embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0070] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0071] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A convolutional neural network pruning method based on Gram matrix orthogonality, characterized in that, Applied to image classification tasks, including: The feature map training samples are input into the current convolutional neural network to obtain the output features; Based on each convolutional layer of the current convolutional neural network, a parameter Gram matrix of the channel dimension is determined, and the saliency score of the diagonal elements in the parameter Gram matrix is ​​calculated; wherein, the elements in the parameter Gram matrix are used to characterize the correlation of the channel convolution kernels of each convolutional layer; The comprehensive loss function corresponding to the current convolutional neural network is calculated based on the saliency score, and the corresponding parameters of the current convolutional neural network are optimized. The optimized convolutional neural network is used as the new current convolutional neural network, and the above optimization steps are repeated until the comprehensive loss function meets the expectations. The target saliency score and target pruned features corresponding to the trained convolutional neural network are obtained. The trained convolutional neural network is pruned based on the target saliency score or the target pruned features. The steps for calculating the significance scores of the diagonal elements in the parameter Gram matrix include: Normalize each row of the parameter Gram matrix to obtain the element score of each row of the parameter Gram matrix excluding the diagonal elements and the significance score of the diagonal elements. The significance scores of the diagonal elements are input into the activation function and multiplied by the amplification factor corresponding to the current step size for progressive discretization to obtain the significance score; wherein, the amplification factor corresponding to the next step size is greater than the amplification factor of the current step size; The step of optimizing the corresponding parameters of the current convolutional neural network includes: Based on the pruned features obtained by multiplying the saliency score and the output features, the pruned channels and retained channels of the current convolutional neural network are determined. The parameters of the current convolutional neural network are optimized based on the pruned channels and the retained channels.

2. The method of claim 1, wherein, The step of determining the channel-dimensional parameter Gram matrix for each convolutional layer of the current convolutional neural network includes: The parameter gram matrix is ​​calculated for each convolutional layer of the current convolutional neural network according to the channel dimension. The elements in each row of the parameter gram matrix, excluding the diagonal elements, are used to characterize the correlation between each channel convolutional kernel of each convolutional layer and other channel convolutional kernels. The diagonal elements are used to characterize the correlation between each channel convolutional kernel and itself.

3. The method of claim 1, wherein, The step of calculating the comprehensive loss function corresponding to the current convolutional neural network based on the saliency score includes: Based on the comparison results between the saliency score and the preset threshold range, the differential control strategy for pruning channels and retaining channels in the current convolutional neural network is determined; If the significance score is higher than the upper limit of the preset threshold range, then the first trainable protection loss function used to constrain the off-diagonal elements of the channel corresponding to the significance score in the parameter Gram matrix to be 0 and the diagonal elements to be 1, and the original task loss function are selected for weighted calculation. If the significance score is lower than the lower limit of the preset threshold range, then the second trainable protection loss function used in the comprehensive loss function to constrain all elements of the channel corresponding to the significance score in the parameter Gram matrix to be 0, and the original task loss function are selected for weighted calculation. If the significance score is within the preset threshold range, then the original task loss function in the comprehensive loss function is selected for calculation.

4. The method of claim 1, wherein, The steps of pruning the trained convolutional neural network based on the target saliency score or the target pruned features include: Based on the target saliency score or the target pruned features, determine the parameters corresponding to the pruned channels of the trained convolutional neural network; Based on the parameters corresponding to the pruned channels, pruning operations are performed on the corresponding channels of the trained convolutional neural network.

5. A current convolutional neural network pruning device based on Gram matrix orthogonality, characterized in that, Applied to image classification tasks, including: The training module inputs feature map training samples into the current convolutional neural network to obtain output features; based on each convolutional layer of the current convolutional neural network, it determines the parameter Gram matrix of the channel dimension and calculates the saliency score of the diagonal elements in the parameter Gram matrix; wherein, the elements in the parameter Gram matrix are used to characterize the correlation of the channel convolutional kernels of each convolutional layer; based on the saliency score, it calculates the comprehensive loss function corresponding to the current convolutional neural network and optimizes the corresponding parameters of the current convolutional neural network; the optimized convolutional neural network is used as the new current convolutional neural network, and the above optimization steps are repeated until the comprehensive loss function meets the expectation, thus terminating to obtain the target saliency score and target pruned features corresponding to the trained convolutional neural network; The pruning module performs pruning operations on the trained convolutional neural network based on the target saliency score or the target pruned features; The training module is further configured to: normalize the elements of each row in the parameter Gram matrix to obtain the element score of each row excluding the diagonal elements and the significance score of the diagonal elements; input the significance score of the diagonal elements into the activation function and multiply it by the amplification factor corresponding to the current step size for progressive discretization to obtain the significance score; wherein the amplification factor corresponding to the next step size is greater than the amplification factor of the current step size; The training module is further configured to determine the pruned and retained channels of the current convolutional neural network based on the pruned features obtained by multiplying the saliency score and the output features; and to optimize the corresponding parameters of the current convolutional neural network according to the pruned and retained channels.

6. An electronic device comprising a memory, a processor, the memory having stored therein a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 4.

7. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 4.

Images