Pruning method based on neural architecture search

Through the pruning method based on neural architecture search, the pseudo-pruning algorithm and pheromone concentration matrix are used to accurately determine the importance of parameters in complex networks, and the problems of poor adaptability and reduced accuracy in the existing technology are solved, and efficient model pruning is achieved Branches and hardware-friendly structured pruning.

CN118862997BActive Publication Date: 2024-11-29CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411344627.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-25
Publication Date
2024-11-29
Estimated Expiration
2044-09-25

AI Technical Summary

Technical Problem

The existing pruning methods have poor adaptability when dealing with complex network structures, and the basis for judgment is unreasonable, resulting in a decrease in model accuracy and making it difficult to meet high-precision requirements.

Method used

The pruning method based on neural architecture search is adopted, and the parameters are reasonably judged through the pseudo-pruning algorithm and pheromone concentration matrix, and pruning is grouped to ensure that the model's accuracy remains high after pruning.

Benefits of technology

It realizes the precise determination of the importance of weights in complex models, maintains high accuracy, improves inference efficiency, reduces memory usage and computing volume, and is suitable for hardware implementation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118862997B_ABST
    Figure CN118862997B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of computer technology, and in particular to a pruning method based on neural architecture search. First, an initial weight matrix and an initial pheromone concentration matrix of a network model are obtained; then, pseudo-pruning is performed on the initial weight matrix, and the pheromone concentration matrix of the network model is updated to obtain weights and pheromone concentrations of all neurons in the network model; all neurons are grouped to obtain a pheromone concentration set of each neuron, and an importance coefficient of each neuron is determined according to the pheromone concentration set of each neuron; finally, the importance coefficients of all neurons are sorted, and neurons are removed according to the sorting results, thereby completing network pruning; the present invention optimizes the pruning algorithm through a method based on neural architecture search, so that the pruning algorithm can adapt to complex network structures; then, the importance of parameters is reasonably determined according to the pheromone concentration, so as to ensure that the pruned model still maintains high precision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular to a pruning method based on neural architecture search. Background Art

[0002] Most modern deep learning models have the problem of over-parameterization, and pruning can compress the original model by cutting out unimportant weights or channels. The advantages of pruning are mainly to minimize the number of actual parameters and reduce inference time. Pruning is usually divided into two categories: structured pruning and unstructured pruning. Unstructured pruning is a fine-grained pruning that considers every parameter in the model. Its advantage is that it can compress the model to a large extent, but it will also lead to irregular calculations and memory access, thereby limiting the parallelism of hardware calculations. In the paper "Unstructured Pruning Method Based on Neural Architecture Search" published by Wang Xianbao et al. in the journal "Pattern Recognition and Artificial Intelligence", structured pruning is a coarser-grained pruning, with all parameters in the entire convolution kernel as the minimum pruning unit. Structured pruning is more friendly to hardware implementation, which not only improves the compactness of the network, but also reduces the storage space requirements. Therefore, it is increasingly favored by researchers in the field.

[0003] At present, the mainstream pruning methods include importance-based pruning, regularization-based pruning and energy-aware pruning. Importance-based pruning can improve the convergence speed. In the paper "Learning both weights and connections for efficient neural network" published by Han et al. in the journal "Neural Information Processing Systems", importance-based pruning is applied to each layer. After the pruning is completed, retraining is used to compensate for the impact of weight removal. The remaining weights will also be adjusted during the retraining process, and finally AlexNet is reduced by 9 times the number of weights and 3 times the number of multiplications and additions. Regularization-based pruning uses the regularization part of the loss function equation to obtain the required sparsity. In the paper "Structured pruning for efficient convnets via incremental regularization" published by Wang et al. in the journal "International Joint Conference on Neural Networks", the original network converges to a sub-network with the desired sparsity rate and minimizes the loss of accuracy. The main advantage of this method is that it will obtain better accuracy, but it requires many iterations to achieve convergence. Energy-aware pruning was proposed by Yang et al. in their paper Designing energy-efficient convolutional neural networks using energy-aware pruning published in the journal 2017 IEEE Conference on Computer Vision and Pattern Recognition. It is a method for pruning weights by evaluating energy. Energy evaluation is achieved by evaluating the number of multiplications and additions, data sparsity, and memory usage. Compared with importance-based pruning, this method can improve energy efficiency by 1.74 times.

[0004] Most existing technologies are oriented towards image classification networks. Although they perform well in these networks, they have obvious shortcomings when dealing with more complex network structures. These shortcomings include:

[0005] (1) Poor adaptability: Existing technologies are optimized for image classification networks. When directly applied to networks with complex structures, they often lead to a significant drop in performance.

[0006] (2) Unreasonable judgment basis: The existing methods use unreasonable judgment basis for parameter importance, which leads to a decrease in the accuracy of the model after pruning and makes it difficult to meet high-precision requirements. Summary of the invention

[0007] In view of this, the present invention aims to provide a pruning method based on neural architecture search. Through the method based on neural architecture search, a pseudo-pruning algorithm is introduced to enable it to adapt to complex network structures. At the same time, the importance of parameters is reasonably determined through pheromone concentration to ensure that the pruned model still maintains high accuracy.

[0008] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0009] A pruning method based on neural architecture search is used to prune a deep learning network, and specifically includes the following steps:

[0010] S1: Pre-train the network model of the deep learning network to obtain the initial weight matrix corresponding to each layer of the network model; at the same time, create the initial pheromone concentration matrix of the network model;

[0011] S2: Pseudo-prune the initial weight matrix corresponding to each layer of the network to obtain the weight matrix corresponding to each layer of the network; update the pheromone concentration matrix of the network model, and then obtain the weights and pheromone concentrations of all neurons in the network model;

[0012] S3: Group the neurons in the network model to obtain a weight set of each neuron and a pheromone concentration set of each neuron, and determine the importance coefficient of each neuron according to the pheromone concentration set of each neuron;

[0013] S4: Determine the number of weights that need to be pruned according to the set pruning rate; sort the importance coefficients of all neurons, and remove neurons according to the sorting results until the number of removed neurons reaches the number of weights that need to be pruned;

[0014] S5: Input the training samples into the network model obtained in step S4, and perform training for no more than 10 times, and select the network model with the highest accuracy; the training samples are image training samples.

[0015] Furthermore, in step S1, the initial pheromone concentration matrix The shape of is the same as that of the network model.

[0016] Furthermore, step S2 specifically includes the following steps:

[0017] S21: Calculate the total number of parameters of the first layer of the network model , and according to the set pseudo pruning rate Determine the number of parameters to be pruned in the first layer of the network :

[0018]

[0019] S22: Initial mask of all 1s Random selection Set the parameters to zero to obtain the model mask M corresponding to the current layer network;

[0020] S23: The initial weight matrix of the current layer network Multiply the model mask M corresponding to the current layer network element by element to obtain the weight matrix W of the current layer network, completing pseudo pruning;

[0021] S24: Update the pheromone concentration matrix of the current network model for the kth time using the following formula:

[0022]

[0023] in, , K represents the total number of updates, represents the pheromone concentration matrix of the current network model after the k-1th update, decay is the pheromone volatility factor, represents the new pheromone concentration matrix generated when the current network model is updated for the kth time, represents the weight hyperparameter of the pheromone concentration matrix of the current network model after the k-1th update, Represents the weight hyperparameter of the new pheromone concentration matrix generated when the current layer network is updated for the kth time;

[0024] S25: Replace the initial mask with the model mask M corresponding to the current layer network , repeat steps S22 to S25 K times to obtain the pheromone concentration matrix P of the current network model;

[0025] S26: Replace the first layer of the network and repeat steps S21 to S25 until the weight matrices corresponding to all layers of the network model and the final pheromone concentration matrix are obtained, and then the weights and pheromone concentrations of the neurons in the network model are obtained.

[0026] Furthermore, in step S3, the weight set corresponding to the i-th neuron includes the weights between all neurons connected to the i-th neuron, that is:

[0027]

[0028] in, represents the weight set corresponding to the i-th neuron, represents the weight between the ith neuron and the jth neuron connected to it, and n represents the number of neurons connected to the ith neuron;

[0029] The pheromone concentration set corresponding to the i-th neuron includes the pheromone concentrations corresponding to the weights of all neurons connected to neuron i, that is:

[0030]

[0031] in, represents the pheromone concentration set corresponding to the i-th neuron, represents the pheromone concentration corresponding to the weight between the i-th neuron and the j-th neuron connected to it; each pheromone concentration With each weight One to one correspondence;

[0032] Add up all pheromone concentrations in the pheromone concentration set corresponding to the i-th neuron to obtain the importance coefficient of the i-th neuron, that is:

[0033]

[0034] in, Represents the importance coefficient of the i-th neuron.

[0035] Furthermore, in step S4, the number of weights to be pruned is obtained by the following formula:

[0036]

[0037] in, represents the number of weights, represents the set pruning rate, Represents the initial weight matrix The total number of parameters.

[0038] Compared with the prior art, the invention can achieve the following beneficial effects:

[0039] (1) The pruning method based on neural architecture search described in the present invention first uses a weight importance search method based on neural architecture search to perform pseudo pruning, which can accurately determine the importance of each weight in the model, and the accuracy will not drop significantly even in complex models;

[0040] (2) The pruning method based on neural architecture search described in the invention adopts reasonable pruning standards to better maintain accuracy. The pheromone concentration-based neuron pruning method uses the pheromone concentration matrix obtained by the search as the basis for judgment, ensuring that the model accuracy will not drop significantly after removing the less important parameters;

[0041] (3) The pruning method based on neural architecture search described in the invention adopts structured pruning, which groups parameters by neurons to ensure that parameters can be reasonably removed without significantly damaging the model structure, which is more friendly to hardware implementation;

[0042] (4) The pruning method based on neural architecture search created by the present invention improves the original model reasoning efficiency, speeds up the reasoning time, and reduces memory usage. By reasonably removing less important parameters, the model size is reduced, the amount of calculation is reduced, thereby speeding up the reasoning time and improving the model reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0044] Figure 1 A flowchart of a pruning method based on neural architecture search according to an embodiment of the present invention;

[0045] Figure 2 A flowchart of a pruning method based on neural architecture search according to an embodiment of the present invention;

[0046] Figure 3 A flowchart of step S2 of the present invention is created in the embodiment;

[0047] Figure 4 A schematic diagram of neuron grouping in step S3 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0049] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0050] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0051] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0052] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0053] like Figure 1 to Figure 2 As shown, the pruning method based on neural architecture search described in the embodiment of the present invention is used to prune a deep learning network, and specifically includes the following steps:

[0054] S1: Pre-train the network model of the deep learning network to obtain the initial weight matrix corresponding to each layer of the network model; at the same time, create the initial pheromone concentration matrix of the network model. Among them, the initial pheromone concentration matrix It is used to record the pheromones left by ants on the path when they move forward in the network model. Its shape is the same as that of the network model.

[0055] S2: Pseudo-prune the initial weight matrix corresponding to each layer of the network to obtain the weight matrix corresponding to each layer of the network; update the pheromone concentration matrix of the network model to obtain the weights and pheromone concentrations of all neurons in the network model.

[0056] Step S2 uses an improved ant colony algorithm combined with residual pheromone concentration to gradually search the network model, which can effectively determine the importance of weights on complex network models, such as Figure 3 As shown, the specific steps include:

[0057] S21: Calculate the total number of parameters of the first layer of the network model , and according to the set pseudo pruning rate Determine the number of parameters to be pruned in the first layer of the network :

[0058]

[0059] Among them, the pseudo pruning rate It is negatively correlated with the number of ants in the ant colony. The more ants there are, the higher the pseudo-pruning rate. The smaller the pseudo pruning rate Usually set to [0.05,0.3]. The total number of parameters of the current layer network It is obtained by the following formula:

[0060]

[0061] in, Indicates the number of inputs of this layer of network, Indicates the number of outputs of this layer of network. The size of the convolution kernel in this layer of network is k×k. In this embodiment of the present invention, in order to meet the needs of structured pruning, the total number of parameters actually calculated is is the number of convolution kernels, that is:

[0062]

[0063] S22: Initial mask of all 1s Random selection The parameters are set to zero to obtain the model mask M corresponding to the current layer network, that is:

[0064]

[0065] in, Represents the elements in the model mask M corresponding to the current layer network.

[0066] S23: The initial weight matrix of the current layer network Multiply the model mask M corresponding to the current layer network element by element to obtain the weight matrix W of the current layer network, completing pseudo pruning, that is:

[0067]

[0068] Among them, ⊙ represents the bit-by-bit multiplication of elements. At this time, the weight to be removed in the weight matrix W of the current layer network becomes 0, the corresponding convolution kernel in the network model is completely removed, the weight to be retained remains unchanged, and the corresponding convolution kernel in the network model is retained.

[0069] S24: The pheromone concentration matrix of the network model is updated using the improved ant colony algorithm. That is, when the ant colony moves forward, each ant passes through the network model one by one. The next ant can refer to the pheromone left on the path by the previous ant to select a path. The higher the pheromone, the greater the probability of the path being selected. The pheromone released by the ant is determined by the test accuracy of each ant after pseudo-pruning. Therefore, the specific process of updating the pheromone concentration matrix is ​​as follows:

[0070] The pheromone concentration matrix of the current network model is updated for the kth time by the following formula, that is, when the kth ant passes through the network model:

[0071]

[0072] in, , K represents the total number of updates, and also represents the total number of ants in the ant colony; It represents the pheromone concentration matrix of the current network model after the k-1th update, and also represents the pheromone concentration matrix composed of the residual pheromone when the k-1th ant passes through the network model; decay is the pheromone volatility factor, which is used to reduce the residual pheromone concentration to prevent it from falling into the local extreme value. It is set in [0, 1]. For complex structures such as deep learning networks, it is generally empirically suitable to set a lower value [0, 0.5], which can promote global search; It represents the new pheromone concentration matrix generated when the current network model is updated for the kth time, and also represents the pheromone concentration matrix composed of the pheromones released when the kth ant passes through the network model; represents the weight hyperparameter of the pheromone concentration matrix of the current network model after the k-1th update, represents the weight hyperparameter of the new pheromone concentration matrix generated when the current layer network is updated for the kth time, and They also represent the weight hyperparameters of the residual pheromone and the currently released pheromone, and different values ​​can be set according to different models.

[0073] S25: Replace the initial mask with the model mask M corresponding to the current layer network , repeat steps S22~S25 K times to obtain the pheromone concentration matrix P of the current network model.

[0074] Each element in the weight matrix W of the current layer network corresponds one-to-one to the weight of each neuron in the current layer network; each element in the pheromone concentration matrix P of the current network model corresponds one-to-one to the pheromone concentration of each neuron in the current network model.

[0075] S26: Replace the first layer of the network and repeat steps S21 to S25 until the weight matrices corresponding to all layers of the network model and the final pheromone concentration matrix are obtained, and then the weights and pheromone concentrations of the neurons in the network model are obtained.

[0076] The pseudo code for implementing step S2 is shown in Table 1.

[0077] Table 1:

[0078]

[0079] S3: Group the neurons in the network model to obtain a weight set of each neuron and a pheromone concentration set of each neuron, and determine the importance coefficient of each neuron according to the pheromone concentration set of each neuron.

[0080] In step S3, the weight set corresponding to the i-th neuron includes the weights between all neurons connected to the i-th neuron, that is:

[0081]

[0082] in, represents the weight set corresponding to the i-th neuron, represents the weight between the ith neuron and the jth neuron connected to it, and n represents the number of neurons connected to the ith neuron.

[0083] The i-th neuron (e.g. Figure 4 The set of pheromone concentrations corresponding to the red, white, and blue neurons in the figure includes the pheromone concentrations corresponding to the weights between neuron i and all neurons connected to it (such as Figure 4 red, blue and black lines), namely:

[0084]

[0085] in, represents the pheromone concentration set corresponding to the i-th neuron, represents the pheromone concentration corresponding to the weight between the ith neuron and the jth neuron connected to it, that is, Figure 4 The pheromone concentration set of red neurons includes the pheromone concentrations of all red lines, the pheromone concentration set of white neurons includes the pheromone concentrations of all black lines, and the pheromone concentration set of blue neurons includes the pheromone concentrations of all blue lines. With each weight One to one correspondence.

[0086] Add up all pheromone concentrations in the pheromone concentration set corresponding to the i-th neuron to obtain the importance coefficient of the i-th neuron, that is:

[0087]

[0088] in, Represents the importance coefficient of the i-th neuron.

[0089] S4: Determine the number of weights that need to be pruned according to the set pruning rate; sort the importance coefficients of all neurons by size, and remove the neurons starting from the neurons with the smallest importance coefficient according to the sorting results, until the number of removed neurons reaches the number of weights that need to be pruned, and the neuron pruning is completed.

[0090] In step S4, the neurons are sorted by importance coefficient, and the neurons with the smallest importance coefficient are removed according to the sorting result. The number of weights to be pruned is obtained by the following formula:

[0091]

[0092] in, represents the number of weights, represents the set pruning rate, Represents the initial weight matrix The total number of parameters.

[0093] S5: After pruning, due to the convergence point offset, several rounds of fine-tuning are required to restore the accuracy. Therefore, the training samples need to be input into the network model obtained after pruning in step S4, and no more than 10 trainings are performed to select the network model with the highest accuracy. The training samples are image training samples. In the embodiment of the present invention, a total of 10 trainings are performed, and then the network model with the highest accuracy is selected.

[0094] In the embodiment of the present invention, the Deformable DETR network in the paper "Deformable DETR: Deformable Transformers for End-to-End Object Detection" provided by Zhu et al. is used as a network model, and the HRSC2016 dataset in the paper "A high resolution optical satellite image datasetfor ship recognition and some new baselines" by Liu et al. is used as a training sample to train the Deformable DETR network model. The trained Deformable DETR network model is pruned and tested by the pruning method based on neural architecture search described in the embodiment of the present invention. The test results are shown in Table 2.

[0095] Table 2:

[0096]

[0097] The first column in Table 2 indicates the pruning rate, base indicates the initial weight without pruning, the second column indicates the test accuracy after several rounds of fine-tuning after pruning, and the test accuracy is tested on the HRSC2016 dataset, and the third column indicates the accuracy comparison with the model before pruning, ↑ indicates the accuracy is improved, ↓ indicates the accuracy is reduced, and if the accuracy after the test is n, then the accuracy change is 90.6% - n. It can be seen from Table 2 that the pruning method based on neural architecture search described in the embodiment of the present invention can maintain more than 98% of the original accuracy even for complex models such as Deformable DETR when pruning 40%, that is, greater than 90.6% × 98% = 88.788%. Here, 90.6% represents the accuracy of the model before pruning.

[0098] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0099] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A pruning method based on neural architecture search for pruning a deep learning network, characterized in that: The following steps are involved: S1: Pre-training the network model of the deep learning network to obtain the initial weight matrix corresponding to each layer of the network in the network model; and simultaneously creating the initial pheromone concentration matrix of the network model; S2: Pseudo-pruning the initial weight matrix corresponding to each layer of the network to obtain the weight matrix corresponding to each layer of the network; updating the pheromone concentration matrix of the network model, thereby obtaining the weights and pheromone concentrations of all neurons in the network model; S3: Grouping the neurons in the network model to obtain a weight set of each neuron and a pheromone concentration set of each neuron, and determining an importance coefficient of each neuron according to the pheromone concentration set of each neuron; S4: Determine the number of weights that need to be pruned according to the set pruning rate; sort the importance coefficients of all neurons, and remove neurons according to the sorting results until the number of removed neurons reaches the number of weights that need to be pruned; S5: Input the training samples into the network model obtained in step S4, and perform training for no more than 10 times, and select the network model with the highest accuracy; the training samples are image training samples.

2. The pruning method based on neural architecture search according to claim 1, characterized in that: In step S1, the initial pheromone concentration matrix The shape of is the same as that of the network model.

3. The pruning method based on neural architecture search according to claim 2, characterized in that: Step S2 specifically includes the following steps: S21: Calculate the total number of parameters of the first layer network in the network model , and according to the set pseudo pruning rate Determine the number of parameters to be pruned in the first layer of the network : S22: Initial mask of all 1s Random selection Set the parameters to zero to obtain the model mask M corresponding to the current layer network; S23: The initial weight matrix of the current layer network Multiply the model mask M corresponding to the current layer network element by element to obtain the weight matrix W of the current layer network, completing pseudo pruning; S24: Update the pheromone concentration matrix of the current network model for the kth time using the following formula: in, , K represents the total number of updates, represents the pheromone concentration matrix of the current network model after the k-1th update, decay is the pheromone volatility factor, represents the new pheromone concentration matrix generated when the current network model is updated for the kth time, represents the weight hyperparameter of the pheromone concentration matrix of the current network model after the k-1th update, Represents the weight hyperparameter of the new pheromone concentration matrix generated when the current layer network is updated for the kth time; S25: Replace the initial mask with the model mask M corresponding to the current layer network , repeat steps S22 to S25 K times to obtain the pheromone concentration matrix P of the current network model; S26: Replace the first layer of network and repeat steps S21 to S25 until the weight matrices corresponding to all layers of network in the network model and the final pheromone concentration matrix are obtained, and then the weights and pheromone concentrations of neurons in the network model are obtained.

4. The pruning method based on neural architecture search according to claim 3, characterized in that: In step S3, the weight set corresponding to the i-th neuron includes the weights between all neurons connected to the i-th neuron, that is: in, represents the weight set corresponding to the i-th neuron, represents the weight between the ith neuron and the jth neuron connected to it, and n represents the number of neurons connected to the ith neuron; The pheromone concentration set corresponding to the i-th neuron includes the pheromone concentrations corresponding to the weights of all neurons connected to neuron i, that is: in, represents the pheromone concentration set corresponding to the i-th neuron, represents the pheromone concentration corresponding to the weight between the i-th neuron and the j-th neuron connected to it; each pheromone concentration With each weight One to one correspondence; Add up all pheromone concentrations in the pheromone concentration set corresponding to the i-th neuron to obtain the importance coefficient of the i-th neuron, that is: in, Represents the importance coefficient of the i-th neuron.

5. The pruning method based on neural architecture search according to claim 4, characterized in that: In step S4, the number of weights that need to be pruned is obtained by the following formula: in, represents the number of weights, represents the set pruning rate, Represents the initial weight matrix The total number of parameters.

Citation Information

Patent Citations

  • Teacher blackboard writing action recognition method based on multi-granularity convolutional neural network pruning

    CN112800977A

  • Sparse pruning method based on double attention mechanism

    CN115359318A