Model processing method, electronic device and medium
By identifying and growing important weights in the pruned model and retraining it, the problem of model sparsity and accuracy reduction caused by unstructured pruning is solved, and model accuracy optimization is achieved for efficient deployment on small edge devices.
Patent Information
- Application Number
- PCT/CN2024/084957
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-14
AI Technical Summary
Existing unstructured pruning methods result in highly sparsity in the model, which reduces the model's representational ability and prevents the extraction of some features from the feature map, thus leading to a decrease in model accuracy.
By extracting the important weights from the pruned weights of the model, growth is performed, and the model is retrained until it converges again, thus optimizing model memory while ensuring model accuracy.
While reducing the number of model parameters, we can improve the model's accuracy and sparsity, ensuring the feasibility of deploying the model on small edge devices.
Smart Images

Figure CN2024084957_14112024_PF_FP_ABST
Abstract
Description
Model processing method, electronic device and medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 8, 2023, with application number 202310514419.8 and application name “A model processing method, electronic device and medium”. The entire contents of the above application are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a model processing method, electronic equipment, and medium. Background Art
[0003] In recent years, with the development of convolutional neural networks (CNNs), the network depth has become deeper and deeper, and the number of parameters has increased, making it difficult to deploy large convolutional neural networks on small edge devices. Pruning, as a commonly used method for compressing models, is mainly divided into structured pruning and unstructured pruning. Among them, structured pruning uses the filters in the convolutional layer of the convolutional neural network as the pruning unit, changing the filter group and the number of feature channels in the network to reduce the width of the convolutional neural network. Unstructured pruning uses a single weight in the convolutional neural network as the pruning unit and performs fine-grained pruning.
[0004] Traditional unstructured pruning methods minimize model parameters, enabling the pruned model to be deployed on small edge devices. However, this large-scale reduction in parameters can lead to high sparsity in the model, reducing its representational capabilities. For example, some features in the feature map cannot be extracted, resulting in reduced model accuracy.
[0005] Summary of the Invention
[0006] In order to solve the problem mentioned above that the existing unstructured pruning method will cause the model to become highly sparse, resulting in a reduction in the model's representation ability, for example, some features of the feature graph cannot be extracted, thereby reducing the accuracy of the model, the present application provides a model processing method, electronic device and medium.
[0007] In a first aspect, the present application provides a model processing method, which can be used in electronic devices, the model processing method comprising obtaining a first neural network model, wherein the first neural network model is obtained by pruning a neural network model to be pruned, and pruning comprises modifying the first weight set of the neural network model to be pruned into a second weight set of the first neural network model, wherein the non-zero first weight in the first weight set corresponds to a value of zero in the second weight set. A target weight set that meets the weight condition is selected from the second weight set. Parameter adjustment processing is performed on the target weight set in the second weight set to obtain a third weight set, wherein the value of the first weight in the third weight set is non-zero. The third weight set is used for the first neural network model.
[0008] Based on the above scheme, after obtaining the pruned model, the third set of weights from the pruned weights of the converged model can be determined, namely the important weights that can improve model accuracy. These important weights are then grown and reapplied to the pruned model. The model is then retrained based on the restored important weights and the unpruned weights from the pruned model until the model converges again, obtaining a converged model. This can optimize model memory to a certain extent while effectively ensuring model accuracy.
[0009] It can be understood that modifying the first weight set of the neural network model to be pruned to the second weight set of the first neural network model may refer to setting the first weight set of the pruned neural network model to zero, that is, achieving pruning of the first weight set.
[0010] It can be understood that the target weight set in the second weight set may refer to the important weights screened out from the pruned weights mentioned in this application.
[0011] It can be understood that growing the important weights may be restoring the important weights. For example, the weights that were set to zero may be adjusted to a positive number, or the weights that were set to zero may be adjusted to a negative number.
[0012] In some optional instances, a target weight set that meets the weight conditions is selected from the second weight set, including: restoring any weight in the second weight set to obtain a candidate neural network model; obtaining the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model; and determining the target weight set based on the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model after restoring each weight in the second weight set.
[0013] It can be understood that the weight condition can be that when the weight is applied to the pruned model alone, the current loss function corresponding to the model has a larger gradient (i.e., a decrease) than the loss function corresponding to the pruned model before application.
[0014] In some optional instances, obtaining the loss function corresponding to the first neural network model includes: determining the first loss function based on the real information corresponding to the training data set and the predicted information output by the first neural network model; determining the second loss function based on the first neural network model and the candidate neural network model; and determining the loss function corresponding to the first neural network model based on the first loss function and the second loss function.
[0015] It can be understood that the first loss function can represent the accuracy of the first neural network model in achieving the preset task, and the second loss function can represent the degree of difference between the weight matrix in the first network model to be pruned and the target low-rank approximation matrix corresponding to the weight matrix.
[0016] It is understood that when the preset network model to be pruned is a neural network model used to implement different tasks, the first loss function used to represent the accuracy of the first neural network model in implementing the preset task can be different. Commonly used first loss functions can include a 0-1 loss function, an absolute value loss function, a logarithmic loss function, a square loss function, an exponential loss function, a cost loss function, a cross-entropy loss function, etc.
[0017] In some optional instances, obtaining a first loss function corresponding to the first neural network model includes: inputting a training data set into the first neural network model to obtain prediction information output by the first neural network model; and obtaining the first loss function based on real information and prediction information corresponding to the training data set.
[0018] In some optional instances, obtaining a second loss function corresponding to the first neural network model includes: obtaining a set of weight matrices in the network model to be pruned; determining a target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set; and obtaining the second loss function corresponding to the first neural network model based on each weight matrix and the target low-rank approximation matrix corresponding to each weight matrix.
[0019] In some optional instances, the target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set is determined, including: normalizing each weight matrix to obtain a weight matrix to be decomposed corresponding to each weight matrix; performing low-rank decomposition processing on the weight matrix to be decomposed corresponding to each weight matrix to obtain a target low-rank approximation matrix corresponding to each weight matrix.
[0020] It can be understood that the target low-rank approximation matrix can be a low-rank decomposition of each weight matrix in the pruned network model, and a matrix composed of the multiplication of multiple sub-weight matrices with small parameters, simplicity and low rank can be obtained.
[0021] It can be understood that by determining the target low-rank approximation matrix corresponding to each weight matrix, the degree of difference between each weight matrix and the target low-rank approximation matrix in the first neural network model is determined to determine the important weights, which can improve the rank of the weight matrix, reduce the omission of feature extraction, and thus improve the accuracy of the model.
[0022] In some optional instances, pruning the neural network model to be pruned to obtain a first neural network model includes: pruning the network model to be pruned based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate to obtain the first neural network model.
[0023] In some optional instances, the network model to be pruned is pruned based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate to obtain a first neural network model, including: based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate, determining the weights corresponding to the preset sparsity rate in the network model to be pruned as pruning weights; and setting the values corresponding to the pruning weights to zero to obtain the first neural network model.
[0024] For example, the network model to be pruned has m weights. When the preset sparsity rate corresponding to this process is α, the m weights can be sorted from large to small based on their amplitudes (absolute values), and the m×α weights in the sequence are determined as the weights in the first weight subset in descending order. The weights in the first weight set are pruned, that is, all the weights in the first weight set are set to zero, thereby obtaining a first neural network model after the network model to be pruned is pruned.
[0025] It can be understood that by pruning the network model to be pruned, the number of parameters of the network model to be pruned can be reduced. For example, reducing the parameter amount m to mm×α can improve the model sparsity rate, so that the pruned model can be deployed on a small edge device.
[0026] In some optional instances, pruning the neural network model to be pruned to obtain a first neural network model includes: pruning the network model to be pruned based on the amplitude of each weight in the network model to be pruned, a preset sparsity rate, and a preset growth rate to obtain the first neural network model.
[0027] In some optional instances, a neural network model to be pruned is pruned to obtain a first neural network model, including: based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate, determining the weights corresponding to the preset sparsity rate in the network model to be pruned as a first weight subset; based on the amplitude of each weight other than the first weight subset in the network model to be pruned and a preset growth rate, determining the weights corresponding to the preset growth rate in the network model to be pruned other than the first weight subset as a second weight subset; and setting the values corresponding to the weights in the first weight subset and the second weight subset to zero to obtain the first neural network model.
[0028] For example, the network model to be pruned has m weights, and the preset sparsity rate corresponding to the processing is α, and the preset growth rate corresponding to the processing is β. The m weights can be sorted from large to small based on the amplitude (absolute value) of the m weights, and the m×α weights in the sequence are determined as the weights in the first weight subset in order from large to small. Then, the non-zero weights can be sorted from large to small based on the amplitude (absolute value) of the mm×α non-zero weights, and the m×β weights in the sequence are also determined as the weights in the second weight subset in order from large to small. Then, the union of the first weight subset and the second weight subset can be determined as the first weight set, and the weights in the first weight set can be pruned, that is, all the weights in the first weight set are set to zero, so that on the basis of pruning the first weight subset corresponding to the preset sparsity rate, the second weight subset is additionally pruned to obtain the first neural network model after the network model to be pruned is pruned.
[0029] It can be understood that by pruning the network model to be pruned, the number of parameters of the network model to be pruned can be reduced. For example, reducing the parameter m to mm×α-m×β can further improve the model sparsity rate, so that the pruned model can be deployed on a small edge device.
[0030] In a second aspect, the present application provides an electronic device comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the model processing method mentioned in the present application.
[0031] In a third aspect, the present application provides a readable storage medium having instructions stored thereon. When the instructions are executed on an electronic device, the electronic device executes the model processing method mentioned in the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] FIG1 shows a schematic diagram of an application scenario according to some embodiments of the present application;
[0033] FIG2 is a schematic diagram of a framework of a pruning method for a model;
[0034] FIG3 is a flow chart of a pruning method that can be applied to model processing;
[0035] FIG4 is a schematic diagram of a model processing method provided in an embodiment of the present application;
[0036] FIG5 is a flow chart of a method for determining a target weight set provided in an embodiment of the present application;
[0037] FIG6 is a schematic diagram of a process for obtaining a second loss function of a network model to be pruned, provided in an embodiment of the present application;
[0038] FIG7 is a graph showing the computing power consumption and accuracy of the comprehensive model of the model processing method provided in an embodiment of the present application;
[0039] FIG8 shows a schematic diagram of the hardware structure of an electronic device. DETAILED DESCRIPTION
[0040] The illustrative embodiments of the present application include, but are not limited to, a model-based processing method, an electronic device, and a medium.
[0041] It can be understood that the model processing method provided in the embodiments of the present application can be used in neural network models used in tasks such as image recognition, target detection, reinforcement learning, and semantic analysis. For example, the types of neural network models are not limited to convolutional neural networks, Transformer (a model based on a multi-head attention mechanism), recurrent neural networks (RNN), long short-term memory neural networks (LSTM), and other arbitrary neural network models.
[0042] As shown in Figure 1, in a specific implementation, the server 10 can prune the neural network model 11 for image recognition and then send it to the small edge device 20, so that the small edge device 20 can process data based on the pruned neural network model. The small edge device can be any implementable electronic device such as a mobile phone or a camera.
[0043] The following introduces the pruning methods of the models mentioned in some embodiments.
[0044] Figure 2 is a schematic diagram of a framework of a model pruning method. As shown in Figure 2, the pruning method may include:
[0045] First, a feature map is obtained. For example, a sampled image can be input into a convolutional neural network, and feature extraction processing can be performed on the sampled image to obtain a feature map. A convolutional neural network can include convolutional layers and fully connected layers.
[0046] Then, a convolutional layer is selected and pruned. For example, for the current convolutional layer in the convolutional neural network, each filter filters the input first feature map and outputs a second feature map. The first feature map is the feature map output by the convolutional layer above the current convolutional layer. The filters, first feature map, and second feature map are in one-to-one correspondence. The matrix rank corresponding to the second feature map is determined based on the parameters in the second feature map. It can be understood that when a matrix is an m×n matrix, the maximum rank of the matrix is the smaller of m and n, expressed as min(m,n). When the matrix rank is relatively large, it can be proved that there is a lot of redundant information in the feature map. The matrix ranks corresponding to the second feature maps output by each filter are sorted, and the convolutional neural network is pruned based on the sorting results. The filters corresponding to the second feature maps whose matrix ranks are less than the preset matrix rank are pruned. Optionally, all parameters in the filters can be set to zero. By pruning the filters corresponding to the feature maps with small matrix ranks, that is, setting all parameters in the weight matrix corresponding to the filters that extract redundant information to zero, the extraction of redundant information can be reduced in the subsequent process of extracting image features using the model with pruned filters.
[0047] Next, the pruned convolutional neural network is fine-tuned to obtain the final convolutional neural network for prediction.
[0048] Since this pruning scheme uses the feature graph as a whole to determine the matrix rank to prune the filter, it can only be applied to the structured pruning of the model and cannot be applied to the unstructured pruning of the model. In other words, it cannot achieve fine-grained pruning of the model and cannot achieve optimal memory optimization of the model.
[0049] FIG3 is a flow chart of a pruning method for a model. As shown in FIG3 , the pruning method may include:
[0050] Get the weight of each filter in each convolutional layer of the convolutional neural network. For example, you can get the weight of the filter in the lth convolutional layer W = {W l 1 ,...,W n 1}. Calculate the rank of the three-dimensional tensor of each filter weight, and obtain the rank of the three-dimensional tensor of the filter weight R = {R l 1 ,...,R n 1}, sort the weights of the filters according to the rank of the three-dimensional tensor of the filters to obtain the sorting result, for example, the weight of the l-th layer filter is W'={W l 1 ,...,W n 1Based on the sorting results, filters with ranks less than the preset rank are pruned to obtain the pruned convolutional neural network W = {W m 1 ,...,W n 1 After pruning, the convolutional neural network is fine-tuned before pruning the next convolutional layer. For example, after pruning the lth convolutional layer, the l+1th convolutional layer can be pruned. Because this pruning scheme uses the three-dimensional tensor of filter weights as the overall rank to prune filters, it cannot be applied to unstructured model pruning. This means that fine-grained pruning of the model is not possible, and the model's memory optimization cannot be achieved.
[0051] In order to solve the problem that the above-mentioned method of determining the rank based on the feature map as a whole and the method of determining the rank based on the three-dimensional tensor of the filter weights as a whole to prune the filter cannot be applied to unstructured pruning, and thus cannot be applied to fine-grained unstructured pruning, in some embodiments, two unstructured pruning methods are provided, namely, a method based on a preset pruning process and a method based on a preset pruning standard.
[0052] Among them, the method based on the preset pruning process can achieve better pruning effects through the pruning process of preset weight removal and preset growth weight. Preset pruning processes include gradual pruning, "from sparse to sparse" pruning that randomly initializes a sparse network, and "compression / decompression" pruning that alternates between dense and sparse network states. For example, gradual pruning performs pruning every ΔT training iterations, slowly increasing the sparsity of the model until the target sparsity is reached and pruning ends. The preset pruning standard selects redundant weights from the weight matrix for pruning by presetting the weight importance.
[0053] The method based on preset pruning standards implements pruning through preset pruning standards, wherein the pruning standards generally include those based on the absolute value of the weight, based on the weight gradient, based on the importance score of the weight obtained by self-learning, etc.
[0054] However, both the method based on a preset pruning process and the method based on a preset pruning standard achieve pruning by pruning the model parameters to the greatest extent possible. A large-scale reduction in the number of parameters will cause the model to become highly sparse, resulting in a reduction in the model's representational ability. For example, some features of the feature map cannot be extracted, thereby reducing the accuracy of the model.
[0055] To solve the above problems, the present application provides a model processing method. After obtaining a converged model and pruning the weights in the converged model, the pruned model can be used to determine the important weights in the pruned weights of the converged model that can improve the model accuracy. Then, the important weights are grown, that is, the important weights are restored and reapplied to the pruned model. The model is retrained based on the restored important weights and the unpruned weights in the pruned model until the model converges again, and the converged model is obtained. In this way, the model memory can be optimized to a certain extent while effectively ensuring the model accuracy.
[0056] Among them, each important weight can be a weight with a larger gradient (i.e., a decrease range) of the current loss function corresponding to the model and the loss function corresponding to the pruned model before the weight is applied alone when the weight is applied alone to the pruned model. It can be understood that the larger the gradient, the smaller the loss function value obtained after the weight is applied alone to the pruned model, that is, the higher the degree of model convergence, that is, the higher the accuracy of the model. For example, selecting a preset number of weights with larger corresponding gradients among the pruned weights as important weights can effectively ensure the accuracy of the model.
[0057] The important weights can be determined by restoring each of the pruned weights in turn. For example, first randomly select a weight A from each of the pruned weights, restore the weight A, and perform model training based on the currently restored weight A and the unpruned weights to obtain the current loss function corresponding to the model, and obtain the gradient (i.e., the degree of descent) of the current loss function and the loss function corresponding to the aforementioned pruned model, and use the gradient as the gradient corresponding to the weight A.
[0058] Then randomly select the next weight B for recovery, and perform model training based on the currently recovered weight B and the unpruned weights to obtain the current loss function corresponding to the model, and obtain the gradient of the current loss function and the loss function corresponding to the aforementioned pruned model, and obtain the gradient corresponding to the weight B; that is, obtain the gradient corresponding to each weight in the pruned weights in turn based on the above method, and sort the pruned weights based on the size of the gradient, and select the preset number of weights with larger corresponding gradients as important weights.
[0059] It can be understood that the larger the gradient, the smaller the loss function value obtained by model training based on the currently restored weights and unpruned weights, that is, the higher the degree of model convergence, the higher the accuracy of the model. In this way, selecting the preset number of weights corresponding to the larger gradient as important weights can effectively ensure the accuracy of the model.
[0060] In some embodiments, during the above-mentioned model pruning process, a certain number of weights can be additionally pruned based on the number of weights pruned to meet the preset sparsity rate of the model, and the number of important weights grown can be ensured to be the same as the number of additional pruned weights. In this way, the model accuracy can be guaranteed without changing the sparsity rate of the model.
[0061] In some embodiments, during the above-mentioned model pruning process, only a certain number of weights may be pruned so that the model preset sparsity rate is met after pruning, and no additional weights are pruned. However, after growing some important weights, for example, when growing important weights for the first time to obtain a converged model, the model sparsity rate increases due to the growth of some important weights and does not reach the preset sparsity rate. At this time, the above-mentioned pruning steps and weight growing steps may be re-executed until the model converges again and the sparsity rate of the model reaches the preset sparsity rate. In this way, the model sparsity rate can also be guaranteed.
[0062] For example, if the preset model sparsity is 60%, and 20% of important weights are added, the model sparsity increases to 80%. At this point, after the model converges, a second pruning can be performed. For example, the number of weights pruned reduces the model sparsity to 40%. At this point, the model sparsity reaches 40%. Then, 20% of important weights are added. When the model converges again, the preset model sparsity reaches 60%. The pruning ratio and the important weight addition ratio of each model can be determined according to actual needs.
[0063] The following is a detailed introduction to the model processing method mentioned in the embodiment of the present application. Figure 4 is a schematic diagram of a model processing method provided in the embodiment of the present application. The model processing method shown in Figure 4 can be executed by an electronic device. As shown in Figure 4, the model processing method may include:
[0064] 401: Obtain the network model to be pruned and the training dataset.
[0065] In an embodiment of the present application, the network model to be pruned may include multiple convolutional layers, and each convolutional layer in the multiple convolutional layers may have layer identification information, such as identity identification information that characterizes the identity of the convolutional layer. The layer identification information of each convolutional layer may be unique, that is, the layer identification information of each convolutional layer is different. For example, the network model to be pruned may include n convolutional layers, and the layer identification information of the convolutional layer may be Conv1, Conv2...Conv n. Each convolutional layer may include multiple convolution kernels, and each convolution kernel in the multiple convolution kernels may have kernel identification information, such as identity identification information that characterizes the identity of the convolution kernel. The kernel identification information of each convolution kernel may be unique, that is, the kernel identification information of each convolution kernel is different. For example, the convolution layer Conv1 may include m convolution kernels, and the kernel identification information of the convolution kernel may be W1, W2...Wm. Each convolution kernel may include multiple weights (parameters), and the weights in each convolution kernel may be all the same, all different, or partially the same. The multiple weights in each convolution kernel can be represented in the form of a matrix, that is, the multiple weights in each convolution kernel can be represented by a weight matrix, and each weight in the weight matrix can be used to extract and enhance the features of image data and audio data.
[0066] In some optional instances, the network model to be pruned may include, but is not limited to, any one of a convolutional neural network, a Transformer (a model based on a multi-head attention mechanism), a recurrent neural network (RNN), and a long short-term memory neural network (LSTM). It is understood that the network model to be pruned may be a neural network model used to implement tasks such as image recognition, target detection, reinforcement learning, and semantic analysis. In some optional implementations, the network model to be pruned may be a visual model used to implement various visual tasks.
[0067] In some optional examples, the training dataset can be an image dataset or an audio dataset. The data in the training dataset can have annotation information, and the annotation information can be information that matches the task of the network model to be pruned. For example, when the network model to be pruned is a neural network model used to implement an image recognition task, the annotation information can be the annotated category information. For another example, when the network model to be pruned is a neural network model used to implement an object detection task, the annotation information can be the annotated category information and location information.
[0068] 402: Modify the first weight set of the network model to be pruned into a second weight set to obtain a first neural network model.
[0069] It can be understood that when the network model to be pruned has a large number of parameters and is difficult to deploy on a small edge device, it is necessary to minimize the number of parameters of the network model to be pruned (that is, reset the largest number of weights in the network model to be pruned to zero) to achieve maximum compression of the network model to be pruned, so that the pruned model can be deployed on a small edge device.
[0070] In some optional instances, a first weight set is determined from the network model to be pruned based on the amplitude (absolute value) of each weight in the network model to be pruned, the preset sparsity rate corresponding to the current training process, and the preset growth rate corresponding to the current training process. The weights in the first weight set can be located in the same convolution kernel, that is, belong to the same weight matrix, or can be located in different convolution kernels, that is, belong to different weight matrices. After determining the first weight set, the weights in the first weight set can be pruned, that is, the weights in the first weight set are reset to zero to obtain a second weight set.
[0071] In some optional examples, a first subset of weights can be determined from the network model to be pruned based on the magnitude (absolute value) of each weight in the network model to be pruned and a preset sparsity rate corresponding to the current training process, and a second subset of weights can be determined from the network model to be pruned based on the magnitude (absolute value) of each weight in the network model to be pruned and a preset growth rate corresponding to the current training process. Subsequently, the union of the first subset of weights and the second subset of weights can be determined as a first weight set, and the first weight set can be modified into a second weight set to obtain a first neural network model.
[0072] For example, the network model to be pruned has m weights, and the preset sparsity rate corresponding to the processing is α, and the preset growth rate corresponding to the processing is β. The m weights can be sorted from large to small based on the amplitude (absolute value) of the m weights, and the m×α weights in the sequence are determined as the weights in the first weight subset in order from large to small. Then, the non-zero weights can be sorted from large to small based on the amplitude (absolute value) of the mm×α non-zero weights, and the m×β weights in the sequence are also determined as the weights in the second weight subset in order from large to small. Then, the union of the first weight subset and the second weight subset can be determined as the first weight set, and the weights in the first weight set can be pruned, that is, all the weights in the first weight set are set to zero, so that on the basis of pruning the first weight subset corresponding to the preset sparsity rate, the second weight subset is additionally pruned to obtain the first neural network model after the network model to be pruned is pruned.
[0073] 403: Determine a target weight set that meets a preset condition from the second weight set.
[0074] It is understandable that in the process of pruning the network model to be pruned, not only is it necessary to reset the maximum number of weights to zero, but it is also necessary to increase the rank of the weight matrix and reduce the omission of feature extraction, so as to achieve the accuracy of the network model to be pruned in achieving the preset task while compressing the network model to be pruned. Therefore, it is necessary to determine the target weight set from the second weight set and adjust it in the subsequent weight adjustment process to ensure that important weights are not mistakenly pruned, so as to ensure that the model achieves the preset task accurately.
[0075] In some optional instances, a target weight set that meets preset conditions can be determined from the second weight set based on the first loss function and the second loss function. The first loss function can be a loss function that represents the accuracy of the first network model to be pruned in achieving a preset task, and the second loss function can be a loss function that represents the different degrees of the weight matrix in the first network model to be pruned and the target low-rank approximation matrix corresponding to the weight matrix. For example, the degree of influence of the change of each weight in the second weight set on the change amplitude of the first loss function value and the second loss function value can be determined, and then the weight with the largest influence can be determined as the target weight to obtain the target weight set.
[0076] The specific method of determining the target weight set that meets the preset conditions from the second weight set based on the first loss function and the second loss function is detailed in FIG5 .
[0077] 404: Adjust the target weights in the target weight set to a third weight set to obtain a second neural network model.
[0078] It can be understood that adjusting the target weights in the target weight set to the third weight set can be to expand the absolute value of the target weights in the target weight set, that is, adjusting the zeroed weights to positive numbers, or adjusting the zeroed weights to negative numbers, thereby ensuring that important weights that have a significant impact on the accuracy of the pruned network model can be reactivated.
[0079] It is understood that after obtaining the second neural network model, the training data set can be input into the second neural network model, and the second neural network model can be trained to obtain a target neural network model. For example, the training data set can be input into the second neural network model, and feature extraction processing can be performed on the training data set based on the weights in the second neural network model, and prediction information can be output. The weights in the second neural network model include a third weight set. After the second neural network model outputs the prediction information of the training data set, the weights in the third weight set in the second neural network model are adjusted based on the annotation information and prediction information of the training data set until the number of iterations exceeds a preset number, thereby obtaining the target neural network model.
[0080] Below, the method for determining the target weight set mentioned in the embodiment of the present application is described in detail. Figure 5 is a flow chart of a method for determining the target weight set provided by the embodiment of the present application. As shown in Figure 5, the steps of determining the target weight set may include:
[0081] 501: Obtain a first loss function of a first neural network model.
[0082] It is understood that after obtaining the first neural network model, the accuracy of the first neural network model in achieving the preset task can be determined, that is, the accuracy of the prediction information output by the first neural network model when predicting the training data set can be determined. This accuracy can be represented by the error between the labeled information of the training data set and the predicted information. Optionally, a first loss function can be used to represent the accuracy of the first neural network model in achieving the preset task. The smaller the value of the first loss function, that is, the smaller the error between the labeled information of the training data set and the predicted information, the higher the accuracy of the first neural network model in achieving the preset task.
[0083] It can be understood that when the network model to be pruned is preset as a neural network model used to implement different tasks, the first loss function used to represent the accuracy of the first neural network model in implementing the preset task can be different. Commonly used first loss functions can include 0-1 loss function, absolute value loss function, logarithmic loss function, square loss function, exponential loss function, cost loss function, cross entropy loss function, etc. For example, when the network model to be pruned is a neural network model used to implement an image recognition task, the cross entropy loss function can be obtained as the first loss function. When the network model to be pruned is a neural network model used to implement an object detection task, the cost loss function can be obtained as the first loss function.
[0084] 502: Obtain a second loss function of the first neural network model.
[0085] It can be understood that the network model to be pruned, as a neural network model used to achieve the preset task, has a large number of weights (parameters) and many redundant weights. Therefore, the redundant weights in the network model to be pruned can be reduced (for example, the network model to be pruned is pruned, that is, some weights in the network model to be pruned are reset to zero) to reduce the number of parameters of the network model to be pruned and achieve compression of the network model to be pruned. When the network model to be pruned is compressed, the redundant weights in the network model to be pruned are reduced, so the weight matrix in the network model to be pruned before and after compression changes. The rank of the weight matrix after compression may be smaller than the rank of the weight matrix before compression (for example, the weights of a certain row of a certain weight matrix in the network model to be pruned are all set to zero). When the low-rank weight matrix is used to extract features from the data in the training dataset, feature extraction will be missed, which will lead to errors in the prediction information output by the network model to be pruned before and after compression. The accuracy of the preset task achieved by the compressed network model to be pruned is lower than the accuracy of the preset task achieved by the network model to be pruned before compression.
[0086] In some optional examples, the weight matrices in the pruned network model can be subjected to low-rank decomposition to obtain a target low-rank approximation matrix composed of the multiplication of multiple sub-weight matrices with a small number of parameters, simplicity, and low rank. For example, for an m×n weight matrix A, it can be subjected to low-rank decomposition to obtain a sub-weight matrix U m×k , sub-weight matrix ∑ k×k And the sub-weight matrix V T k×n The target low-rank approximation matrix multiplied by k<<n, the weight matrix A m×nThe number of parameters is m×n, and the number of parameters of the target low-rank approximation matrix is k×(m+n+1). Although the number of parameters of the target low-rank approximation matrix corresponding to the weight matrix is much smaller than the corresponding weight matrix, due to the low rank of the sub-weight moments in the target low-rank approximation matrix, when the target low-rank approximation matrix corresponding to the weight matrix is used to extract features from the data in the training dataset, feature extraction omissions will occur. Therefore, when compressing the network model to be pruned, the difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix can be increased, the rank of the compressed weight matrix can be increased, and feature extraction omissions can be reduced. This can reduce the error between the prediction information output by the network model to be pruned before and after compression to predict the training dataset, thereby ensuring that the compressed network model to be pruned achieves the accuracy of the preset task. Optionally, a second loss function can be used to represent the difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix. The smaller the value of the second loss function, that is, the greater the difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix, the higher the rank of the weight matrix, and the fewer omissions in feature extraction. In turn, the smaller the error between the prediction information output by predicting the training data set using the network model to be pruned before and after compression, the higher the accuracy of the compressed network model to be pruned in achieving the preset task.
[0087] 503: Determine a target weight set from the second weight set based on the first loss function and the second loss function.
[0088] It can be understood that in the process of pruning the network model to be pruned, it is necessary not only to reduce the amount of parameters, but also to reduce the omission of feature extraction. Therefore, in the process of pruning the network model to be pruned, it is necessary not only to reset the maximum number of weights to zero, but also to increase the rank of the weight matrix, so as to achieve the maximum compression of the network model to be pruned while ensuring the accuracy of the preset task of the network model to be pruned.
[0089] In some optional instances, by pruning the network model to be pruned to obtain the first neural network model, the error between the labeled information and the predicted information of the training data set can be adjusted, that is, the value of the first loss function is changed, and the degree of difference between the weight matrix and the target low-rank approximation matrix corresponding to the weight matrix can be adjusted to improve the rank of the weight matrix. Therefore, by pruning the network model to be pruned to obtain the first neural network model, the first loss function value and the second loss function value can be adjusted simultaneously to reduce the number of parameters of the network model to be pruned while improving the rank of the weight matrix in the network model to be pruned, reducing the omission of feature extraction, and ensuring the accuracy of the network model to be pruned in achieving the preset task.
[0090] It can be understood that after the first weight set in the pruned network model is set to zero, if there are important weights in the first weight set that have a greater impact on the change amplitude of the first loss function value and the change amplitude of the second loss function value, that is, important weights that have a significant impact on the accuracy of the prediction information output by the first neural network model, if the important weights are kept at zero, in the subsequent process of adjusting the weights (non-zeroed weights) of the first neural network model, the change amplitude of the first loss function value and the change amplitude of the second loss function value are small, and it is difficult to make the first neural network model converge by adjusting the non-zeroed weights. Therefore, after pruning the first weight set in the network model to be pruned to obtain the second weight set, the target weight set (i.e., important weights) can be determined from the second weight set for growth processing to ensure that important weights that have a greater impact on the variation range of the first loss function value and the variation range of the second loss value will not be pruned incorrectly, so that in the subsequent process of weight adjustment of the first neural network model, the important weights can be adjusted to change the variation range of the first loss function value and the second loss function value, so that the first neural network model converges, so as to reduce the number of parameters of the network model to be pruned while improving the rank of the weight matrix in the network model to be pruned, reducing the omission of feature extraction, and ensuring the accuracy of the network model to be pruned in achieving the preset task.
[0091] In some optional instances, the target loss function can be determined based on the sum of the first loss function and the second loss function, and the important weights (i.e., the target weight set) that have a greater impact on the magnitude of the change in the target loss function value can be determined, that is, the important weights that have a greater impact on the magnitude of the change in the first loss function value and the magnitude of the change in the second loss function value. When the weights in the network model to be pruned are subsequently adjusted, the target weights in the target weight set are also adjusted, so that while reducing the number of parameters of the network model to be pruned, the rank of the weight matrix in the network model to be pruned is improved, the omission of feature extraction is reduced, and the accuracy of the network model to be pruned in achieving the preset task is ensured.
[0092] In some optional examples, the objective loss function may be in the form shown in Formula 1: L = L task +λL rank Formula 1
[0093] Among them, in formula 1, L can represent the target loss function, L task The first loss function can be expressed as rank can represent the second loss function, and λ can represent the introduced linear combination hyperparameter.
[0094] It can be understood that the target loss function can be a function with multiple weights in the network model to be pruned as independent variables. By using the target loss function to perform chain derivation on each weight, the derivative corresponding to each weight can be obtained. The derivative can reflect the degree of influence of the change in weight on the change in the value of the target loss function. The larger the derivative of the weight, the greater the influence of the change in weight on the change in the value of the target loss function.
[0095] In some optional instances, after determining the derivative corresponding to each weight in the second weight set, the amplitudes (absolute values) of the derivatives corresponding to each weight in the second weight set can be sorted from large to small, and the weights corresponding to the m×β derivatives in the sequence can be determined as the target weight set in order from large to small.
[0096] Below, the method for determining the second loss function mentioned in the embodiment of the present application is described in detail. Figure 6 is a schematic diagram of a process for obtaining the second loss function of the first neural network model provided by the embodiment of the present application. As shown in Figure 6, the steps of obtaining the second loss function of the first neural network model may include:
[0097] 601: performing normalization processing on each weight matrix in the network model to be pruned to obtain a weight matrix to be decomposed corresponding to each weight matrix.
[0098] It can be understood that for each weight matrix in the network model to be pruned, the Frobenius norm of each weight matrix in the network model to be pruned can be determined based on the Frobenius norm calculation formula, and then each weight matrix in the network model to be pruned can be divided by the Frobenius norm of each weight matrix to achieve L2 normalization of each weight matrix in the network model to be pruned, and obtain the weight matrix to be decomposed corresponding to each weight matrix in the network model to be pruned.
[0099] In some optional examples, the Frobenius Norm calculation formula may be in the form shown in Formula 2:
[0100] In Formula 2, ||W|| F It can represent the Frobéus norm of each weight matrix in the network model to be pruned, W can represent each weight matrix in the network model to be pruned, W T It can represent the transpose of each weight matrix in the network model to be pruned, and tr(·) can represent the trace of ·.
[0101] In some optional examples, the weight matrix to be decomposed corresponding to each weight matrix in the network model to be pruned can be in the form shown in Formula 3:
[0102] In Formula 3, ||W|| F It can represent the Frobéus norm of each weight matrix in the network model to be pruned, and W can represent each weight matrix in the network model to be pruned. It can represent the weight matrix to be decomposed.
[0103] 602: Perform singular value decomposition on the weight matrices to be decomposed corresponding to the weight matrices in the network model to be pruned, and determine the target low-rank approximation matrix corresponding to the weight matrices in the network model to be pruned.
[0104] It can be understood that in order to improve the rank of the compressed weight matrix, the weight matrix to be decomposed corresponding to each weight matrix can be subjected to singular value decomposition to obtain the eigenvector set and singular value set corresponding to each weight matrix in the network model to be pruned. Then, the left singular matrix, singular value matrix and right singular matrix can be determined based on the eigenvector set and singular value set corresponding to each weight matrix in the network model to be pruned, and the left singular matrix, singular value matrix and right singular matrix are multiplied together as the target low-rank approximation matrix corresponding to the weight matrix.
[0105] In some optional examples, for the weight matrix to be decomposed corresponding to each weight matrix in the network model to be pruned, the transpose of the weight matrix to be decomposed can be determined, and multiple eigenvalues and eigenvectors can be determined based on the weight matrix to be decomposed and the transpose of the weight matrix to be decomposed. Then, the left singular matrix and the right singular matrix corresponding to the weight matrix can be determined based on the eigenvectors, and the singular value matrix can be determined based on the eigenvalues. For example, multiple eigenvalues and eigenvectors can be determined based on the product of the weight matrix to be decomposed and the transpose of the weight matrix to be decomposed, and the left singular matrix can be determined based on the determined multiple eigenvectors. Multiple eigenvalues and eigenvectors can be determined based on the transpose of the weight matrix to be decomposed and the product of the weight matrix to be decomposed, and the right singular matrix can be determined based on the determined multiple eigenvectors. The square roots of all eigenvalues can be determined as the singular values corresponding to each weight matrix in the network model to be pruned, thereby obtaining a set of singular values corresponding to each weight matrix in the network model to be pruned. Then, the singular value matrix can be determined based on the singular value set.
[0106] In some optional examples, after obtaining the singular value set corresponding to each weight matrix in the network model to be pruned, a candidate singular value set can be determined from the singular value set corresponding to each weight matrix of the network model to be pruned. For example, the largest k singular values can be selected from the singular value set corresponding to each weight matrix of the network model to be pruned according to the matrix low rank approximation principle (Eckart-Young) as the candidate singular value set corresponding to each weight matrix in the network model to be pruned. Then, a singular value matrix corresponding to each weight matrix in the network model to be pruned can be generated based on the candidate singular value set corresponding to each weight matrix in the network model to be pruned.
[0107] 603: Determine a second loss function according to each weight matrix in the network model to be pruned and a target low-rank approximation matrix corresponding to each weight matrix.
[0108] Next, the second loss function is introduced by taking a weight matrix in the network model to be pruned and the target low-rank approximation matrix corresponding to the weight matrix as an example. Specifically, the calculation formula of the second loss function can be shown in Formula 4: U T U=IV T V=I
[0109] In Formula 4, It can represent the weight matrix to be decomposed after the weight matrix is normalized, U can represent the left singular matrix, V can represent the transpose of the right singular matrix, ∑ can represent the singular value matrix, U and V can represent the unitary matrix concatenated by the orthogonal basis vectors, Trun can represent the target low-rank approximation matrix, which can be specifically the product of the transpose of the left singular matrix, the singular value matrix, and the right singular matrix, ||·|| F can represent the Frobeus norm of ·, and I can represent the identity matrix.
[0110] It can be understood that in actual applications, the model processing flow method can be implemented and executed in the form of software code.
[0111] In some optional implementations, the pseudo code for operations related to the model processing flow method (referred to as pseudo code M1) is as follows:
[0112] Input: An n-layer dense network W; target sparsity f at iteration iter s (iter); the growth weight ratio f in iter iterations deacy (iter); update frequency of pruning weights Δ; total pruning time T = T prune +T finetune ; learning rate α
[0113] Out:A Sparse Model W⊙M
[0114] :
[0115] 1 Initialize a matrix of all 1s;
[0116] 2 M←1;
[0117] 3 / / Phase 1: weight update (pruning and growing);
[0118] 4 for iter←1to T prune do
[0119] 5. Input the training data into the network for forward propagation and calculate the loss L;
[0120] 8 if iter%Δ=0then
[0121] 7 / / Update weight position mask M;
[0122] 8
[0123] 9 Prune to the target sparsity rate f of iter iterations according to the weight amplitude |W| s (iter)%;
[0124] 10 Again cut off the remaining weight f according to the weight amplitude |W| deacy (iter)%;
[0125] 11 Gradient size generated by combined loss Grow the same amount of weights f deacy (iter)%
[0126] 12 end if
[0127] 13 else
[0128] 14. Training Sparse Networks Using Gradient Descent
[0129] 15 end if
[0130] 18 end for
[0131] 17 / / Stage 2: Fine-tune the network without changing the position of the weights;
[0132] 18 for iter←T prune +1 to T do
[0133] 19. Without changing the weight positions, train the sparse network until convergence;
[0134] 20 end for
[0135] 21 return sparse network W⊙M Next, explain each line of code in pseudocode M1:
[0136] First, the fifth line of code in M1 is used to determine the task loss. The training dataset can be input into the dense network W. In the dense network W, the processed weight matrices of each convolutional layer forward propagate the training data. The dense network W can output the prediction information of the data in the training dataset, and then the task loss can be determined based on the labeling information and prediction information of the data in the training dataset.
[0137] The 8th line in M1 is used to determine the adversarial loss function. This can be done by determining the low-rank approximation corresponding to each weight matrix to be processed in the dense network W and performing convolution processing on the training data using each target low-rank approximation matrix. The dense network W can output the prediction information of the data in the training dataset, and then the adversarial loss function can be determined based on the labeling information and prediction information of the data in the training dataset.
[0138] Line 9 in M1 is used to prune a certain number of weights based on the preset weights that need to be pruned to improve the sparsity of the model and reduce the memory usage of the model.
[0139] Line 10 in M1 is used to prune a certain number of remaining unpruned weights based on the preset weights that need to be pruned.
[0140] Line 11 in M1 is used to sort the pruned weights according to the absolute value of the gradient and grow the weights with larger gradients, so as to ensure that important weights that were mistakenly pruned can be reactivated.
[0141] It can be understood that the above pseudocode, after pruning the weights, can determine the derivatives of all pruned weights based on the target loss function, and grow the weights with larger derivatives to ensure that important weights are not mistakenly pruned, thereby ensuring the accuracy of the model.
[0142] To verify the pruning effect of the model processing method provided in the embodiments of the present application, in some examples, the pruning effect was verified based on the small dataset CIFAR-10. Table 1 shows the classification accuracy of the pruned models obtained based on different pruning algorithms at different sparsity rates. Based on Table 1, it can be seen that the pruning effect of the model processing method provided in the embodiments of the present application at high sparsity rates is significantly better than other unstructured pruning methods.
[0143] Table 1
[0144] In other examples, the effectiveness of pruning was verified based on the large ImageNet dataset. Table 2 shows the classification accuracy of the pruned models obtained using different pruning algorithms at different sparsity rates. Based on Table 2, it can be seen that the model processing method provided in the embodiments of the present application significantly outperforms other unstructured pruning methods in pruning at high sparsity rates.
[0145] Table 2
[0146] Figure 7 is a curve diagram of the computing power consumption and accuracy of the comprehensive model of the model processing method provided in the embodiment of the present application. Based on Figure 7, it can be seen that the model processing method provided in the embodiment of the present application is significantly better than other unstructured pruning methods in terms of computing power consumption and accuracy of the comprehensive model.
[0147] In other examples, downstream visual tasks such as object detection and instance segmentation were verified on the COCO val2017 dataset using the Mask R-CNN framework with a ResNet-50 FPN as the backbone network. Table 3 shows the positioning accuracy and segmentation accuracy of the pruned object detection model obtained based on different pruning algorithms at different sparsity rates. Based on Table 3, it can be seen that the model processing method provided in the embodiment of the present application has a significantly better pruning effect at high sparsity rates than other unstructured pruning methods.
[0148] Table 3
[0149] In other examples, unstructured pruning was performed on the DeiT-S model of the transformer structure, and then verified on the ImageNet large-scale image classification dataset. Table 4 shows the classification accuracy of the transformer model obtained based on different pruning algorithms at different sparsity rates. Based on Table 4, it can be seen that the model processing method provided in the embodiment of the application is also relatively effective on the transformer architecture.
[0150] Table 4
[0151] The model processing method provided in the embodiments of this application can efficiently compress visual models, reduce the computing power of visual models, and help deploy large models on small edge devices. For example, it can be deployed to a cloud service or to a terminal device. The following example illustrates the effect of deploying a pruned model on a small edge device.
[0152] Challenging visual tasks like object detection often require the support of large visual models. In some cases, smartphones are small devices due to power and energy constraints. Deploying the object detection model after unstructured pruning on the phone significantly reduces the phone's energy consumption without compromising model performance, extending its operating life.
[0153] In other cases, in video surveillance, cameras are typically small, on-device devices. For surveillance systems in remote areas, where power supply infrastructure is difficult to deploy, solar power is often used. Therefore, the energy consumption of the object detection model deployed on the camera side is highly demanding. By deploying the object detection model after unstructured pruning on the camera side, we can significantly reduce video surveillance energy consumption without compromising model performance.
[0154] The electronic devices mentioned in this application are introduced below. It is understood that the electronic devices may include, but are not limited to: mobile phones (including foldable screen mobile phones and straight-screen mobile phones), tablet computers, desktop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), portable Android devices (PADs), personal digital assistants (PDAs), handheld devices with wireless communication functions, computing devices, vehicle-mounted devices or wearable devices, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and other mobile terminals or fixed terminals, power banks and other electronic devices with data transmission synchronization requirements.
[0155] FIG8 is a schematic diagram of the hardware structure of an electronic device. It is understood that the electronic device of the present application can be a server, a desktop computer, a handheld computer, a notebook computer, or other electronic device. The structure of the electronic device is described below using a server as an example.
[0156] Figure 8 is a block diagram of a server provided in an embodiment of the present application, and Figure 8 schematically illustrates an example server according to multiple embodiments. In one embodiment, the server may include one or more processors 804, system control logic 806 connected to at least one of the processors 804, system memory 812 connected to the system control logic 808, non-volatile memory (NVM) 816 connected to the system control logic 808, and a network interface 820 connected to the system control logic 808.
[0157] In some embodiments, the processor 804 may include one or more single-core or multi-core processors. In some embodiments, the processor 804 may include any combination of general-purpose processors and specialized processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments where the server utilizes an eNB (Evolved Node B) 101 or a RAN (Radio Access Network) controller 102, the processor 804 may be configured to execute various embodiments.
[0158] In some embodiments, system control logic 808 may include any suitable interface controller to provide any suitable interface to at least one of processors 804 and / or any suitable device or component in communication with system control logic 808 .
[0159] In some embodiments, the system control logic 808 may include one or more memory controllers to provide an interface to the system memory 812. The system memory 812 may be used to load and store data and / or instructions. In some embodiments, the server's memory 812 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).
[0160] The non-volatile memory (NVM) 816 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory (NVM) 816 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of an HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.
[0161] The non-volatile memory (NVM) 816 may include a portion of storage resources on the device where the server is installed, or it may be accessible by the device but not necessarily be part of the device. For example, the non-volatile memory (NVM) 816 may be accessed via the network interface 820 over a network.
[0162] In particular, system memory 812 and non-volatile memory (NVM) 816 may each include a temporary copy and a permanent copy of instructions 824. Instructions 824 may include instructions that, when executed by at least one of processors 804, cause the server to implement the model quantization method described in the embodiments of the present application. In some embodiments, instructions 824, hardware, firmware, and / or software components thereof may be additionally or alternatively located in system control logic 808, network interface 820, and / or processor 804.
[0163] The network interface 820 may include a transceiver for providing a radio interface for the server, thereby communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 820 may be integrated with other components of the server. For example, the network interface 820 may be integrated with at least one of the processor 804, the system memory 812, the non-volatile memory (NVM) 816, and a firmware device (not shown) having instructions. When at least one of the processors 804 executes the instructions, the server implements the model quantization method mentioned in the embodiments of the present application.
[0164] The network interface 820 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 820 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.
[0165] In one embodiment, at least one of the processors 804 may be packaged together with logic for one or more controllers of the system control logic 808 to form a system-in-package (SiP). In one embodiment, at least one of the processors 804 may be integrated on the same die with logic for one or more controllers of the system control logic 808 to form a system-on-chip (SoC).
[0166] The server may further include input / output (I / O) devices 832. The I / O devices 832 may include a user interface that enables a user to interact with the server, and peripheral component interfaces that enable peripheral components to interact with the server. In some embodiments, the server may also include sensors for determining at least one of environmental conditions and location information related to the server.
[0167] In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.
[0168] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.
[0169] In some embodiments, the sensors may include, but are not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be part of or interact with the network interface 820 to communicate with components of a positioning network (e.g., Global Positioning System (GPS) satellites).
[0170] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.
[0171] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.
[0172] While the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.
Claims
1. A model processing method, characterized in that: For an electronic device, the method comprises: Obtaining a first neural network model, wherein the first neural network model is obtained by pruning the neural network model to be pruned, and the pruning includes modifying a first weight set of the neural network model to be pruned into a second weight set of the first neural network model, wherein a non-zero first weight in the first weight set has a corresponding value of zero in the second weight set; Selecting a target weight set that meets a weight condition from the second weight set; Performing parameter adjustment processing on the target weight set in the second weight set to obtain a third weight set, wherein the value of the first weight in the third weight set is non-zero; The third set of weights is used for the first neural network model.
2. The method according to claim 1, characterized in that The selecting a target weight set satisfying a weight condition from the second weight set comprises: Performing recovery processing on any weight in the second weight set to obtain a candidate neural network model; Obtaining the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model; The target weight set is determined based on the gradient of the loss function corresponding to the first neural network model and the loss function corresponding to the candidate neural network model after each weight in the second weight set is restored.
3. The method according to claim 2, characterized in that The obtaining the loss function corresponding to the first neural network model includes: Determining a first loss function based on real information corresponding to the training data set and predicted information output by the first neural network model; Determine a second loss function based on the first neural network model and the candidate neural network model; A loss function corresponding to the first neural network model is determined based on the first loss function and the second loss function.
4. The method according to claim 3, characterized in that The obtaining a first loss function corresponding to the first neural network model includes: Inputting the training data set into the first neural network model to obtain prediction information output by the first neural network model; The first loss function is obtained based on the real information corresponding to the training data set and the predicted information.
5. The method according to claim 3, characterized in that: The obtaining a second loss function corresponding to the first neural network model includes: Obtaining a weight matrix set in the network model to be pruned; Determine a target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set; Based on the weight matrices and the target low-rank approximation matrices corresponding to the weight matrices, the second loss function corresponding to the first neural network model is obtained.
6. The method according to claim 5, characterized in that The determining of a target low-rank approximation matrix corresponding to each weight matrix in the weight matrix set includes: Performing standardization processing on the weight matrices to obtain weight matrices to be decomposed corresponding to the weight matrices; Perform low-rank decomposition processing on the weight matrices to be decomposed corresponding to the weight matrices to obtain the target low-rank approximation matrices corresponding to the weight matrices.
7. The method according to any one of claims 1 to 6, characterized in that: The step of pruning the neural network model to be pruned to obtain a first neural network model includes: The network model to be pruned is pruned based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate to obtain the first neural network model.
8. The method according to claim 7, characterized in that The method of pruning the network model to be pruned based on the amplitude of each weight in the network model to be pruned and a preset sparsity rate to obtain the first neural network model includes: Based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate, determining the weights of the number corresponding to the preset sparsity rate in the network model to be pruned as pruning weights; The value corresponding to the pruning weight is set to zero to obtain the first neural network model.
9. The method according to any one of claims 1 to 6, characterized in that: The step of pruning the neural network model to be pruned to obtain a first neural network model includes: Based on the amplitude of each weight in the network model to be pruned, a preset sparsity rate and a preset growth rate, the network model to be pruned is pruned to obtain the first neural network model.
10. The method according to claim 9, characterized in that The step of pruning the neural network model to be pruned to obtain a first neural network model includes: Based on the amplitude of each weight in the network model to be pruned and the preset sparsity rate, determining the weights of the number corresponding to the preset sparsity rate in the network model to be pruned as a first weight subset; Based on the amplitude and preset growth rate of each weight in the network model to be pruned except the first weight subset, the weights corresponding to the preset growth rate in the network model to be pruned except the first weight subset are determined as a second weight subset; The values corresponding to the weights in the first weight subset and the second weight subset are set to zero to obtain the first neural network model.
11. An electronic device, characterized in that: include: A memory, used to store instructions executed by one or more processors of the electronic device, and the processor, which is one of the one or more processors of the electronic device, is used to execute the model processing method described in any one of claims 1-10.
12. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the model processing method according to any one of claims 1 to 10.
Citation Information
Patent Citations
A compression method of a deep neural network
CN109726799A
Neural network model compression method, system and device and medium
CN111079899A
Compression method of convolutional neural network model
CN112836817A
Pruning method and device of neural network, electronic equipment and storage medium
CN115374926A
Method, apparatus, and system for providing data-driven selection of machine learning training observations
US20200167689A1
Cited By
Model pruning method and device for heterogeneous computing cluster and storage medium
CN119227769A
Unstructured temporal bone image report semantic tag modeling and extracting method based on large model instruction fine tuning
CN120011564A
Light-weight large-model intelligent customer service deployment method for edge calculation
CN120386534A
Distributed algorithm model compression method for micro-grid group collaborative optimization, electronic equipment and storage medium
CN121257631A
Pruning lightweight-based SPR connection quality tracing method and system
CN121903644A
