Bayesian optimization-based structured pruning method and device, terminal and medium

Abstract

The application provides a structured pruning method and device based on Bayesian optimization, a terminal and a medium. Compared with other manual pruning methods, the application does not need to manually set a pruning threshold for each convolutional layer to identify candidate filter parameters. Compared with a random pruning method, a more systematic method is used to find candidate pruning parameters, avoiding the possibility that important parameters of the model are pruned. Compared with other group-based methods, a set of solutions is not generated in each iteration. The proposed method benefits from the exploration strategy of Bayesian optimization, so the calculation time is less. The proposed method has a novel objective function, which considers resource constraints and model performance when finding the best parameters of the model.

Description

Technical Field

[0001] This application relates to the field of structured pruning technology for neural network models, and in particular to structured pruning methods, devices, terminals and media based on Bayesian optimization. Background Technology

[0002] Convolutional neural networks (CNNs) are an important tool in many disciplines for solving various problems. However, due to their high resource requirements, redundant weights, and large number of parameters, and limited by the execution resources available on devices, CNNs often require a significant amount of energy to perform inference on resource-constrained devices (such as mobile phones, wearable devices, and other terminal devices). In recent years, pruning has become an important technique for reducing irrelevant parameters and accelerating model performance. However, finding the optimal number of parameters to prune without significantly affecting model performance is a time-consuming and tedious task that requires a lot of manual tuning.

[0003] Over the years, deep neural networks have significantly improved performance on massive datasets like ImageNet, and the deep learning community has witnessed numerous successful applications of deep neural networks in various fields, such as object detection, classification, speech recognition, and image captioning. Over time, the number of layers and nodes per layer in deep neural networks has increased; for example, these networks have expanded from 5-layer LeNets to 152-layer ResNets. While this significant increase in the number of layers helps deep neural networks achieve advanced performance in various computer vision applications, it also requires more resources. For instance, the VG166 model, used for various classification and object detection tasks, has 138 million learnable parameters and 15.5 billion cumulative multiply-accumulate operations for a single inference process on a single image. Although neural network training can be completed on a GPU in one go, once the model is trained and deployed to end devices (such as wearables and mobile phones), it requires a significant amount of battery power during inference. For example, running a network with 1 billion connections at 20Hz requires approximately 12.8W of DRAM access alone, and end devices may not have the same computational, energy, and storage capacity available during model training. These limitations hinder the deployment of deep learning models to edge devices for real-world applications. Furthermore, over-parameterized networks can lead to overfitting; even popular benchmark convolutional networks like AlexNet, VGG16, and ResNet suffer from over-parameterization and computational complexity issues. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a structured pruning method, apparatus, terminal and medium based on Bayesian optimization to solve the technical problem of low pruning efficiency in the prior art.

[0005] To achieve the above and other related objectives, a first aspect of this application provides a structured pruning method based on Bayesian optimization, comprising: receiving a pre-trained initial convolutional neural network model, the model's accuracy data, the cumulative multiply-accumulate operation data of convolutional and fully connected layers, and the number of parameters generated during the iteration process via a communication interface, and writing them into a memory; setting an objective function; the objective function being associated with the accuracy, cumulative multiply-accumulate operation data, and number of parameters of the initial convolutional neural network model, as well as with the accuracy, cumulative multiply-accumulate operation data, and number of parameters of the convolutional and fully connected layers used in the current iteration; pruning the filters of the convolutional and fully connected layers of the convolutional neural network model in each iteration; calculating the objective function in the current iteration, using a Gaussian process to obtain the minimum value of the objective function as a convergence criterion for the iterative iteration, and correspondingly obtaining the optimal convolutional neural network model with the lowest cost.

[0006] In some embodiments of the first aspect of this application, the objective function is expressed as:

[0007] Where λ1, λ2, λ3 and α are hyperparameter constants, λ1+λ2+λ3=1; A0 is the accuracy of the initial convolutional neural network model; M is the accuracy of the convolutional neural network model at the i-th iteration. c_i and M d_i These are the cumulative multiply-accumulate operations of the convolutional layer and the fully connected layer at the i-th iteration; NP i It represents the number of model weight parameters in the i-th iteration.

[0008] In some embodiments of the first aspect of this application, the cumulative multiply-accumulate operations M of the convolutional layer and the fully connected layer during the i-th iteration c_i and M d_i They are represented as: Among them, M c_0 and M d_0 These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers in the initial convolutional neural network model, respectively. and These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers pruned in the i-th iteration, respectively.

[0009] In some embodiments of the first aspect of this application, using the minimum value of the objective function as the convergence criterion for the iterative loop includes: assuming that the best candidate currently found is t + This indicates that the next new candidate selected by the expected improvement function EI is... The convergence determination process is described as follows: like The calculated cost is greater than the optimal cost f(t) + ), then it means The cost compared to f(t) + The cost of ) has not improved, and the best candidate remains t. + At this point, the expected improvement function will be set to 0.

[0010] In some embodiments of the first aspect of this application, the process of pruning the filters of the convolutional and fully connected layers of the convolutional neural network model in each iteration includes: using absolute and L1 norms to quantify the importance of different filters, and the absolute and L1 norms are positively correlated with the importance of the filters; in each iteration, pruning away a preset number or proportion of filters with low absolute and L1 norm values.

[0011] In some embodiments of the first aspect of this application, the process of pruning the filter during each iteration of the convolutional layer includes: for any convolutional layer k, the input image received from layer (k-1) is (u k-1 ×v k-1 ×n k-1 ), where u and v are the height and width of the input, and n is the number of channels or depth; in the k-th layer, F filters are applied, and the shape of the filters is determined by (a k ×b k ×n k-1 ×F) is given, where a k and b k It is the height and width of the filter, n k-1 It is the filter depth; in the input image (u k-1 ×v k-1 ×n k-1 ) and filters (a k ×b k ×n k-1 Convolution operations are performed between (×F) to generate activation layer A. k (h×w×F) is used as the input image for layer (k+1), where h and w are the height and width of the (k+1)th layer, and h = (u k-1 -a k +2p) / s+1, w=(vb) k +2p) / s+1, where s and p are the stride and fill, respectively.

[0012] In some embodiments of the first aspect of this application, the pruned convolutional neural network model is evaluated by any one or a combination of the following metrics: Model accuracy: The proportion of model parameters reduced: The percentage reduction in floating-point operations per second: Where NP0 represents the number of weight parameters in the initial convolutional neural network model, NP p This represents the number of weight parameters in the pruned model; FLOPs0 represents the number of floating-point operations per second in the initial convolutional neural network model. p This represents the number of floating-point operations per second in the model after pruning.

[0013] To achieve the above and other related objectives, a second aspect of this application provides a structured pruning device based on Bayesian optimization, comprising: a parameter receiving module, configured to receive, via a communication interface, a pre-trained initial convolutional neural network model, the model's accuracy data, the cumulative multiply-accumulate operands of convolutional and fully connected layers, and the number of parameters generated during the iteration process, and write these data into a memory; an objective function module, configured to set an objective function; the objective function is associated with the accuracy of the initial convolutional neural network model, the cumulative multiply-accumulate operands of convolutional and fully connected layers, the number of parameters, and the accuracy of the convolutional neural network model used in the current iteration, the cumulative multiply-accumulate operands of convolutional and fully connected layers, and the number of parameters; and an iterative pruning module, configured to prune the filters of the convolutional and fully connected layers of the convolutional neural network model in each iteration; calculate the objective function in the current iteration, use a Gaussian process to obtain the minimum value of the objective function as a convergence criterion for the iterative iteration, and obtain the optimal convolutional neural network model with the lowest cost.

[0014] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the Bayesian-optimized structured pruning method.

[0015] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic terminal, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the Bayesian-optimized structured pruning method.

[0016] As described above, the structured pruning method, apparatus, terminal, and medium based on Bayesian optimization of this application have the following advantages:

[0017] Beneficial effects:

[0018] (1) Compared with other manual pruning methods, the present invention does not require manually setting a pruning threshold for each convolutional layer to identify candidate filter parameters.

[0019] (2) Compared with the random pruning method, the present invention uses a more systematic method to find candidate pruning parameters, thus avoiding the possibility of pruning important parameters of the model.

[0020] (3) Compared with other population-based methods, the present invention does not generate a set of solutions in each iteration. The proposed method benefits from the Bayesian optimization exploration strategy, thus requiring less computation time.

[0021] (4) The method proposed in this invention has a novel objective function that takes into account resource constraints and model performance when finding the optimal parameters of the model. Attached Figure Description

[0022] Figure 1 The diagram shown is a flowchart of a structured pruning method based on Bayesian optimization in one embodiment of this application.

[0023] Figure 2A This is a schematic diagram of the network state before the k-th layer trimming filter in one embodiment of this application.

[0024] Figure 2B This is a schematic diagram of the network state after the k-th layer trimming filter in one embodiment of this application.

[0025] Figure 3 The pruning ratios are shown as determined by the lowest cost in four experiments in one embodiment of this application.

[0026] Figure 4 The pruning ratios are shown as determined by the highest TOP-1 accuracy in four experiments in one embodiment of this application.

[0027] Figure 5 The figure shows the inference acceleration effect obtained by the pruning ratio determined by the lowest cost in four experiments in one embodiment of this application.

[0028] Figure 6 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application.

[0029] Figure 7 The diagram shown is a structural schematic of a Bayesian-optimized structured pruning device according to an embodiment of this application. Detailed Implementation

[0030] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0031] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of this application. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of this application. The following detailed description should not be considered limiting, and the scope of the embodiments of this application is defined only by the claims of the published patent. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0032] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," and "holding" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0033] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the invention.

[0035] Generally, there are two ways to improve the inference efficiency of neural networks on end devices. The first is to increase the computing power of the end device, making it more powerful enough to run resource-intensive neural networks. However, this approach is clearly not cost-effective, and GPU availability is limited on resource-constrained devices. The second approach is to improve the efficiency of neural networks by reducing the high resource requirements during inference. To reduce the resource requirements of network runtime, many model compression and acceleration methods have been developed, such as quantization and weight sharing, knowledge distillation, pruning, and low-rank approximation. Among these, pruning has attracted much attention in recent years. It reduces the storage and computational requirements of trained deep learning models by removing unimportant nodes, filters, convolutional layers, or weights. It also helps reduce the well-known overfitting problem in DNNs. Since not all network parameters are important during inference, these unimportant parameters have little impact on improving network accuracy. Therefore, less important parameters can be eliminated with almost no impact on model accuracy, reducing the number of learnable parameters and cumulative multiply-accumulate operations, thereby improving model performance on resource-scarce devices. Finding the optimal number of parameters that can be removed from a network remains an important research direction. Although many methods have been developed to make trained deep learning models efficient for inference, an automatic and general method is still needed to fully optimize the model's performance.

[0036] There are typically three ways to find filters with less important parameters: manual, random, and using optimization methods. Specifically, manually tuning the filters of a convolutional neural network and trying all possible combinations is a time-consuming process; random searches, which disregard previous results and even if the optimal solution is known to lie in a small, already explored region, still involve searching the entire design space, thus wasting considerable time. Furthermore, randomly deleting filters can significantly impact the results.

[0037] In view of this, the present invention provides a structured pruning method, apparatus, terminal and medium based on Bayesian optimization, which constructs a new objective function by combining model performance and resource constraints (such as accuracy, cumulative multiply-accumulate operations and the number of parameters), thereby achieving the purpose of automatically finding and deleting minor filters and their respective feature maps of each convolutional layer.

[0038] This invention provides a Bayesian-optimized structured pruning method, a system for implementing the Bayesian-optimized structured pruning method, and a storage medium storing an executable program for implementing the Bayesian-optimized structured pruning method. Regarding the implementation of the Bayesian-optimized structured pruning method, this invention will describe exemplary implementation scenarios.

[0039] like Figure 1 The diagram illustrates a flowchart of a Bayesian-optimized structured pruning method according to an embodiment of the present invention. This Bayesian-optimized structured pruning method is applied to the terminal side or the server side and mainly includes the following steps:

[0040] Step S11: Receive the pre-trained initial convolutional neural network model, the model's accuracy data, the cumulative multiply-accumulate operand data of the convolutional and fully connected layers, and the number of parameters generated during the iteration process through the communication interface, and write them into the memory.

[0041] Specifically, the communication interface refers to the interface between the central processing unit and the standard communication subsystem, including but not limited to standard serial ports (RS232), GPIB interfaces, Ethernet interfaces, USB interfaces, or wireless interfaces. The memory includes volatile memory and / or non-volatile memory; among which, non-volatile memory includes, for example, read-only memory (ROM) and programmable read-only memory (PROM), which are used as external caches.

[0042] The pre-trained initial convolutional neural network model can be obtained by training with historical data, including historical model accuracy data, cumulative multiply-accumulate operand data of convolutional layers and fully connected layers, and data on the number of parameters in the iteration process. The trained model is used as the pre-trained initial convolutional neural network model.

[0043] Since the entire Bayesian optimization algorithm is an iterative process, each iteration requires the training model, its prediction accuracy data, the cumulative multiply-accumulate operation data of convolutional and fully connected layers, and the number of parameters in the iteration process. For the first iteration, the initial model's accuracy data, the cumulative multiply-accumulate operation data of convolutional and fully connected layers, and the number of parameters in the iteration process are required.

[0044] Step S12: Set the objective function; the objective function is related to the accuracy of the initial convolutional neural network model, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, the number of parameters, and the accuracy of the convolutional neural network model used in the current iteration, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, and the number of parameters.

[0045] Specifically, the objective function used in this embodiment can be expressed as Equation 1:

[0046]

[0047] Where λ1, λ2, λ3, and α are hyperparameter constants, λ1 + λ2 + λ3 = 1, meaning λ1, λ2, and λ3 are three hyperparameter constants that add up to 1; A0 is the accuracy of the initial convolutional neural network model, and... M is the accuracy of the convolutional neural network model at the i-th iteration. c_i and M d_i These are the cumulative multiply-accumulate operations of the convolutional layer and the fully connected layer at the i-th iteration; NP i It represents the number of parameters in the i-th iteration.

[0048] It is worth noting that, through multiple experiments with different values ​​of λ1, λ2, λ3 and α, the optimal values ​​of λ1, λ2, λ3 and α were found to be 0.5, 0.25, 0.25 and 0.5, respectively, while simultaneously satisfying λ1+λ2+λ3=1.

[0049] The cumulative multiply-accumulate operations of the convolutional layer and the fully connected layer at the i-th iteration are M. c_i and M d_i The calculation method is as follows:

[0050]

[0051]

[0052] Among them, M c_0 and M d_0 These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers in the initial convolutional neural network model, respectively. and These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers pruned in the i-th iteration, respectively.

[0053] The number of parameters in the i-th iteration is NP. i The calculation method is as follows:

[0054]

[0055] Where NP0 represents the number of parameters in the initial convolutional neural network model. This represents the number of parameters removed in the i-th iteration.

[0056] Step S13: In each iteration, prune the filters of the convolutional and fully connected layers of the convolutional neural network model; calculate the objective function in the current iteration, use a Gaussian process to find the minimum value of the objective function as a convergence criterion for the loop iteration, and obtain the optimal convolutional neural network model with the minimum cost.

[0057] Specifically, this invention proposes a Bayesian optimization-based method to solve the optimal filter selection problem and prune corresponding filters from convolutional and fully connected layers. In other words, the pruning of filters in convolutional and fully connected layers is determined through Bayesian optimization. To obtain the optimal number of filters for the trained model, this invention represents filter pruning as a minimization problem. In each Bayesian optimization iteration, the cost of each iteration is calculated using the trained model, its cumulative multiply-accumulate operations, accuracy, and number of parameters through a performance prediction model. Finally, the optimal model with the minimum cost is obtained as the output through continuous iteration.

[0058] It should be understood that Bayesian optimization is an optimization method that optimizes a non-convex black-box objective function. It uses previously computed information (prior) to find the optimal point for the next attempt, continuously updating the probabilistic model through the objective function to be optimized. The probabilistic model of Bayesian optimization relies on Gaussian processes (GPs), which are extensions of multivariate Gaussian distributions. More specifically, a Gaussian process is a generalization of Gaussian distributions to functions that have a consistent joint Gaussian distribution. A Gaussian process is described by its mean function g(t) and covariance function h(t,t').

[0059] f(t)~GP(g(t),h(t,t′));

[0060] In this context, Generalized Progressive (GP) provides the distribution of the objective function f(t); in other words, the objective function is modeled using GP. Bayesian optimization fits the observation points to a GP distribution and selects the next possible optimal value using a selection function. Commonly used selection functions include Probabilistic Improvement (PI), Upper Confidence Limit (UCB), Entropy Search, and Expected Improvement (EI), among others. Currently, EI is generally considered the most effective selection function, as it considers both exploration and development. During exploration, EI explores unexplored regions in the design space; during development, EI searches the already explored space for further improvement.

[0061] In this embodiment, the minimum value of the objective function is used as the convergence criterion for the iterative loop, resulting in the optimal convolutional neural network model, which can be expressed as:

[0062] argmint∈T f(t) = {t1, t2, ..., t} n}; Formula 5)

[0063] Where f(t) represents the pruning performance prediction model; T is the set of solutions; t i (i = 1, 2, 3, ..., n) is one of the possible solutions to T. i It is a vector representing the set of filters in all convolutional layers considered by the convolutional network N.

[0064] The objective function (Formula 1 above) is modeled using a Gaussian process (GP). In this embodiment, the next possible optimal value is selected through a data acquisition function (expected improvement in EI). GP is described by the mean function g(t) and the covariance function h(t,t'), which are defined as follows:

[0065]

[0066]

[0067] in, It represents the mathematical expectation of the function.

[0068] For a given set T = {t1, t2, ..., t...} n} and the evaluation function f(T)={f(t 1) ,f(t2),...,f(t n The objective function f(t) at the new candidate position The posterior confidence level is:

[0069]

[0070] in, Denotes the mean function at the posterior. Let represent the covariance function at the posterior. Further:

[0071]

[0072]

[0073] in, Indicates that the input of the independent variable is The covariance function; T is a set (i.e., the input is a matrix), h(T,T) -1 f(T) represents the inverse of a matrix; f(T) represents the objective function with matrix T as input.

[0074] Assume that the best candidate found so far is t + This indicates that the new candidate The EI function is:

[0075]

[0076] in, It can be obtained more efficiently using the following approximate formula 12:

[0077]

[0078] Where Z represents the pair of pairs ... The value after standardization is shown in Equation 13 below; φ represents the standard normal probability density function, while Φ represents the standard normal cumulative distribution function.

[0079]

[0080] As can be seen from Formula 11, if The calculated cost is greater than the previously obtained optimal cost f(t) + ), A negative value means Compared with the previously estimated best observation value f(t) + Compared to the previous version, there was no significant improvement; therefore, the expected improvement function is... It will be set to zero. In other words, the selection... It did not reduce costs.

[0081] It is worth noting that the generalization and convergence capabilities of neural networks depend on a large number of weight parameters. Compared to high-amplitude weights, small-amplitude weight parameters have a smaller impact on network error. Therefore, this invention determines which filters have little impact on the results and can be pruned based on the size of the filter weights. It is also noteworthy that this invention does not use a manual setting of pruning thresholds for each layer, as this is not only inefficient but also computationally intensive. This invention represents filter pruning as an optimization problem and uses Bayesian optimization to find the correct number of filters for each convolutional and fully connected layer of the convolutional neural network model, thereby eliminating the need for manual intervention and greatly improving pruning efficiency.

[0082] In this embodiment, the absolute sum (L1 norm) is used to quantify the importance of different filters in convolutional and fully connected layers. Filters with smaller absolute sums (L1 norms) are considered less important than those with larger absolute sums (L1 norms). Therefore, in each model iteration, all filters in convolutional and fully connected layers are ranked according to their absolute sums (L1 norms), and low-ranked filters are marked as less important. After identifying less important filters, these filters are pruned, and a new neural network architecture is built using the remaining filters, replicating the weights from the original neural network architecture. The pruned convolutional neural network model is then retrained to reduce the accuracy loss caused by pruning.

[0083] In this embodiment, the process of pruning the filter during each iteration is illustrated using a convolutional layer as an example: For any convolutional layer k, the input image it receives from layer (k-1) is (u k-1 ×v k-1 ×n k-1 ), where u and v are the height and width of the input, and n is the number of channels or depth. In the k-th layer, F filters are applied, and the shape of the filters is determined by (a... k ×b k ×n k-1 ×F) is given, where a k and b k It is the height and width of the filter, n k-1 This refers to the filter depth (i.e., the number of input channels). In the input image (u... k-1 ×v k-1 ×n k-1 ) and filters (a k ×b k ×n k-1 Convolution operations are performed between (×F) to generate activation layer A. k (h×w×F) is used as the input image for layer (k+1), where h and w are the height and width of the (k+1)th layer, and h = (u k-1 -a k +2p) / s+1, w=(vb) k +2p) / s+1, where s and p are the stride and fill, respectively.

[0084] For ease of understanding by those skilled in the art, the following is used: Figure 2A and Figure 2B Provide supplementary explanations. Figure 2A This shows the network state before the k-th layer pruning filter. Figure 2BThe diagram illustrates the network state after pruning the filters in the k-th layer. Before pruning, the activation shape generated in the k-th layer is (h×w×F), where F = 5. After pruning two filters, the activation shape becomes h×w×(F-2). This means that the two filters and their corresponding feature images are removed, and the number of input channels in the next layer is also reduced, thereby improving the inference efficiency of the convolutional neural network.

[0085] It should be noted that the implementation process of this invention can be achieved using Python programming language and the PyTorch deep learning framework. To verify the advanced performance of the Bayesian-optimized structured pruning method provided by this invention, corresponding experimental data will be provided below. In this experiment, an NVIDIA DGX V100 supercomputer was used as the computing power source; this supercomputer is equipped with 128GB of RAM and 8 GPUs.

[0086] The experiment used three datasets in total: CIFAR10, CIFAR100, and ImageNet. For the CIFAR10 and CIFAR100 datasets with smaller sample sizes, this embodiment of the invention selected VGG16 as the network model. For the ImageNet dataset with larger sample sizes, this embodiment of the invention used both VGG16 and ResNet34 models for the experiment.

[0087] For each experiment, the results are presented in two different ways: the first shows the maximum accuracy achievable across all Bayesian iterations; the second shows the pruned configuration and final accuracy obtained by minimizing the objective function. The pruned model is represented by accuracy (Acc) and the reduction in the proportion of weight parameters (NP). red and the reduction in floating-point operations per second (FLOPs) red The evaluation is conducted using three criteria.

[0088]

[0089]

[0090]

[0091] Where NP0 represents the number of weight parameters in the initial convolutional neural network model, NP p This represents the number of weight parameters in the pruned model; FLOPs0 represents the number of floating-point operations per second in the initial convolutional neural network model. p This represents the number of floating-point operations per second in the model after pruning.

[0092] Table 1 below shows the accuracy and performance of VGG16 on the CIFAR10 and CIFAR100 datasets after adopting the pruning scheme provided in this embodiment of the invention, and compares it with several other pruning schemes:

[0093]

[0094]

[0095] As can be seen from Table 1, the pruning scheme provided by the embodiments of the present invention can tolerate a decrease in accuracy, greatly reducing weights and the number of operations per second, thereby accelerating the inference process of the convolutional neural network.

[0096] Table 2 below shows the accuracy and performance of VGG16 and ResNet on the ImageNet dataset after adopting the pruning scheme provided in this embodiment of the invention, and compares it with several other pruning schemes:

[0097]

[0098] As can be seen from Table 2 above, although the accuracy rate has decreased to some extent, the pruning scheme proposed in the embodiments of the present invention can achieve the maximum pruning ratio.

[0099] It should be noted that: The source of other scheme 1 is: H. Li, A. Kadav, I. Durdanovic, H. Samet, HPGraf, Pruning filters for efficient convnets, 2016, arXiv preprint arXiv:1608.08710. The source of other scheme 2 is: Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, C. Zhang, Learning efficient convolutional networks through network slimming, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp.2736–2744. Source of other scheme 3: C. Zhao, B. Ni, J. Zhang, Q. Zhao, W. Zhang, Q. Tian, ​​Variational convolutional neural network pruning, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2780–2789. Source of other scheme 4: T. Choudhary, V. Mishra, A. Goswami, J. Sarangapani, Heuristic-based automatic pruning of deep neural networks, Neural Comput. Appl. (2022) 1–15. Other sources for Scheme 5: JHLuo, H. Zhang, HYZhou, CWXie, J. Wu, W. Lin, Thinnet: Pruning CNN filters for a thinnernet, IEEE Trans. Pattern Anal. Mach. Intell. 41(10)(2018) 2525–2538.Sources for other solutions: Scheme 6: Y. Zhou, Y. Zhang, Y. Wang, Q. Tian, ​​Accelerate cnn via recursive bayesian pruning, in: Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019, pp. 3306–3315. Scheme 7: BOAyinde, T. Inanc, JM Zurada, Redundant feature pruning for accelerated inference in deep neural networks, Neural Netw. 118(2019) 148–158. Scheme 8: R. Yu, A. Li, CFChen, JHLai, VIMorariu, X. Han, et al., Nisp: Pruning networks using neuron importance score propagation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9194–9203.

[0100] Figure 3 The experiment shows the pruning ratio determined by the lowest cost in four experiments: VGG16 on the CIFAR10 dataset, VGG16 on the CIFAR100 dataset, VGG16 on the ImageNet dataset, and ResNet34 on the ImageNet dataset.

[0101] Figure 4 The pruning ratios determined by the highest TOP-1 accuracy in four experiments are shown: VGG16 on the CIFAR10 dataset, VGG16 on the CIFAR100 dataset, VGG16 on the ImageNet dataset, and ResNet34 on the ImageNet dataset.

[0102] Figure 5The inference speedup achieved by the pruning ratio determined by the lowest cost is demonstrated in four experiments: VGG16 on the CIFAR10 dataset, VGG16 on the CIFAR100 dataset, VGG16 on the ImageNet dataset, and ResNet34 on the ImageNet dataset.

[0103] The Bayesian-optimized structured pruning method provided in this invention can be implemented on the terminal side or the server side. For the hardware structure of the Bayesian-optimized structured pruning terminal, please refer to [link to relevant documentation]. Figure 6 This is a schematic diagram of an optional hardware structure of an electronic terminal 600 provided in an embodiment of the present invention. The terminal 600 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The Bayesian-optimized structured pruning terminal 600 includes: at least one processor 601, a memory 602, at least one network interface 604, and a user interface 606. The various components in the device are coupled together through a bus system 605. It is understood that the bus system 605 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 605 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general will label all buses as bus systems.

[0104] The user interface 606 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0105] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0106] In this embodiment of the invention, the memory 602 is used to store various types of data to support the operation of the Bayesian-optimized structured pruning terminal 600. Examples of this data include: any executable program for operating on the Bayesian-optimized structured pruning terminal 600, such as the operating system 6021 and application program 6022; the operating system 6021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 6022 may include various applications, such as media players, browsers, etc., for implementing various application services. The implementation of the Bayesian-optimized structured pruning method provided in this embodiment of the invention can be included in the application program 6022.

[0107] The methods disclosed in the above embodiments of the present invention can be applied to processor 601, or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in the form of software. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 601 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0108] In an exemplary embodiment, the Bayesian-optimized structured pruning terminal 600 can be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0109] like Figure 7The diagram illustrates a structural schematic of a Bayesian optimization-based structured pruning device according to an embodiment of the present invention. In this embodiment, the Bayesian optimization-based structured pruning device 700 includes a parameter receiving module 701, an objective function module 702, and an iterative pruning module 703.

[0110] The parameter receiving module 701 is used to receive the pre-trained initial convolutional neural network model, the model's accuracy data, the cumulative multiply-accumulate operand data of the convolutional layers and fully connected layers, and the parameter count data generated during the iteration process through the communication interface, and write them into the memory.

[0111] The objective function module 702 is used to set an objective function; the objective function is related to the accuracy of the initial convolutional neural network model, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, the number of parameters, and the accuracy of the convolutional neural network model used in the current iteration, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, and the number of parameters.

[0112] The iterative pruning module 703 is used to prune the filters of the convolutional and fully connected layers of the convolutional neural network model in each iteration; calculate the objective function in the current iteration, use a Gaussian process to find the minimum value of the objective function as a convergence criterion for the iterative iteration, and obtain the optimal convolutional neural network model with the minimum cost.

[0113] It should be noted that the Bayesian optimization-based structured pruning device provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the Bayesian optimization-based structured pruning device and the Bayesian optimization-based structured pruning method embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0114] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0115] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0116] In summary, this application provides a structured pruning method, apparatus, terminal, and medium based on Bayesian optimization. Compared with other manual pruning methods, this invention eliminates the need to manually set pruning thresholds for each convolutional layer to identify candidate filter parameters. Compared with random pruning methods, it uses a more systematic approach to find candidate pruning parameters, avoiding the possibility of pruning important model parameters. Compared with other population-based methods, it does not generate a set of solutions in each iteration; the proposed method benefits from the Bayesian optimization exploration strategy, thus requiring less computation time. The method proposed in this invention has a novel objective function that considers resource constraints and model performance when finding the optimal model parameters. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial applicability.

[0117] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A structured pruning method based on Bayesian optimization, characterized in that, include: The system receives the pre-trained initial convolutional neural network model, the model's accuracy data, the cumulative multiply-accumulate operand data of the convolutional and fully connected layers, and the number of parameters generated during the iteration process through the communication interface, and writes them into the memory. Define an objective function; the objective function is related to the accuracy of the initial convolutional neural network model, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, the number of parameters, and the accuracy of the convolutional neural network model used in the current iteration, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, and the number of parameters. The objective function is expressed as: ； in, , , and It is a hyperparameter constant. ; It is the accuracy of the initial convolutional neural network model; It represents the accuracy of the convolutional neural network model at the i-th iteration. and These are the cumulative multiply-accumulate operations of the convolutional layer and the fully connected layer, respectively, during the i-th iteration; It represents the number of model weight parameters in the i-th iteration; The cumulative multiply-accumulate operations of the convolutional layer and the fully connected layer at the i-th iteration and They are represented as: ; ;in, and These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers in the initial convolutional neural network model, respectively. and These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers pruned in the i-th iteration, respectively. The number of model weight parameters in the i-th iteration The calculation method is as follows: ； in, This indicates the number of parameters in the initial convolutional neural network model. This represents the number of parameters removed in the i-th iteration; In each iteration, the filters of the convolutional and fully connected layers of the convolutional neural network model are pruned; the objective function in the current iteration is calculated, and the minimum value of the objective function is obtained using a Gaussian process as a convergence criterion for the iterative cycle, thus obtaining the optimal convolutional neural network model with the lowest cost; using the minimum value of the objective function as a convergence criterion for the iterative cycle includes: assuming that the best candidate currently found is... This indicates that the next new candidate selected by the expected improvement function EI is... The convergence determination process is described as follows: ;like The calculated cost is greater than the optimal cost. , then it means The cost compared to The cost has not improved, and the best candidate remains [missing information]. At this point, the expected improvement function will be set to 0; Indicates new candidate The expected improvement function; [] denotes the mathematical expectation of the function; The process of pruning the filter during each iteration of the convolutional layer includes: for any convolutional layer k, the input image received from layer (k-1) is ( ), where u and v are the height and width of the input, and n is the number of channels or depth; in the k-th layer, F filters are applied, and the shape of the filters is determined by ( ) is given, where and These are the height and width of the filter. It is the filter depth; in the input image ( ) and filters ( Convolution operations are performed between the layers to generate activation layers. As the input image for layer (k + 1), where h, w are the height and width of layer (k + 1), , , where s and p are the stride and fill, respectively.

2. The structured pruning method based on Bayesian optimization according to claim 1, characterized in that, The process of pruning the filters of the convolutional and fully connected layers of the convolutional neural network model in each iteration includes: using absolute and L1 norms to quantify the importance of different filters, and the absolute and L1 norms are positively correlated with the importance of the filters; in each iteration, a preset number or proportion of filters with low absolute and L1 norm values ​​are pruned.

3. The structured pruning method based on Bayesian optimization according to claim 1, characterized in that, The pruned convolutional neural network model can be evaluated using any one or a combination of the following metrics: Model accuracy: % The proportion of model parameters reduced: % The percentage reduction in floating-point operations per second: ; in, This represents the number of weight parameters in the initial convolutional neural network model. This indicates the number of weight parameters in the model after pruning. This represents the number of floating-point operations per second in the initial convolutional neural network model. This represents the number of floating-point operations per second in the model after pruning.

4. A structured pruning device based on Bayesian optimization, characterized in that, include: The parameter receiving module is used to receive the pre-trained initial convolutional neural network model, the model's accuracy data, the cumulative multiply-accumulate operand data of the convolutional layers and fully connected layers, and the parameter count data generated during the iteration process through the communication interface, and write them into the memory. The objective function module is used to set the objective function; the objective function is related to the accuracy of the initial convolutional neural network model, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, the number of parameters, and the accuracy of the convolutional neural network model used in the current iteration, the cumulative number of multiply-accumulate operations of the convolutional and fully connected layers, and the number of parameters. The objective function is expressed as: ； in, , , and It is a hyperparameter constant. ; It is the accuracy of the initial convolutional neural network model; It represents the accuracy of the convolutional neural network model at the i-th iteration. and These are the cumulative multiply-accumulate operations of the convolutional layer and the fully connected layer, respectively, during the i-th iteration; It represents the number of model weight parameters in the i-th iteration; The cumulative multiply-accumulate operations of the convolutional layer and the fully connected layer at the i-th iteration and They are represented as: ; ;in, and These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers in the initial convolutional neural network model, respectively. and These are the cumulative multiply-accumulate operations of the convolutional and fully connected layers pruned in the i-th iteration, respectively. The number of model weight parameters in the i-th iteration The calculation method is as follows: ； in, This indicates the number of parameters in the initial convolutional neural network model. This represents the number of parameters removed in the i-th iteration; An iterative pruning module is used to prune the filters of the convolutional and fully connected layers of the convolutional neural network model in each iteration; it calculates the objective function in the current iteration, uses a Gaussian process to find the minimum value of the objective function as a convergence criterion for the iterative cycle, and obtains the optimal convolutional neural network model with the lowest cost; using the minimum value of the objective function as a convergence criterion for the iterative cycle includes: assuming that the best candidate currently found is... This indicates that the next new candidate selected by the expected improvement function EI is... The convergence determination process is described as follows: ;like The calculated cost is greater than the optimal cost. , then it means The cost compared to The cost has not improved, and the best candidate remains [missing information]. At this point, the expected improvement function will be set to 0; Indicates new candidate The expected improvement function; [] denotes the mathematical expectation of the function; The process of pruning the filter during each iteration of the convolutional layer includes: for any convolutional layer k, the input image received from layer (k-1) is ( ), where u and v are the height and width of the input, and n is the number of channels or depth; in the k-th layer, F filters are applied, and the shape of the filters is determined by ( ) is given, where and These are the height and width of the filter. It is the filter depth; in the input image ( ) and filters ( Convolution operations are performed between the layers to generate activation layers. As the input image for layer (k + 1), where h, w are the height and width of layer (k + 1), , , where s and p are the stride and fill, respectively.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the Bayesian optimization-based structured pruning method as described in any one of claims 1 to 3.

6. An electronic terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the terminal to perform the Bayesian-optimized structured pruning method as described in any one of claims 1 to 3.

Images