Unstructured pruning method based on gradient and weight change

Abstract

The invention discloses an unstructured pruning method based on gradient and weight change, which performs pruning operation in a neural network training process, does not need to perform retraining after pruning of a neural network, and reduces computing resources in a process of performing actual prediction by using the neural network. Meanwhile, in the process of pruning the neural network, according to the weight and gradient of each node of the neural network, a connection coefficient, a sorting coefficient and a comprehensive score of each node are obtained, and the nodes in each layer of the neural network are cut through the sorted connection coefficients, so that the neural network is pruned. The cut nodes in the neural network are recovered through the sorting coefficient and the comprehensive score, so that excessive fluctuation of the neural network is avoided, the training stability and robustness are ensured, premature parameter fixing is avoided, and the generalization ability of the network is enhanced; and meanwhile, the recovered nodes can optimize a gradient updating process in back propagation of the neural network, so that the accuracy of the clipped neural network is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unstructured pruning, and in particular relates to an unstructured pruning method based on gradient and weight changes. Background Art

[0002] The measurement of vital signs has evolved from contact to non-contact, reducing the inconvenience of contact and expanding the scope of application. The cumbersome way of measuring vital signs in hospitals in the past has now become within reach. This transformation stems from a new technology - remote photoplethysmography (rPPG). Subtle changes in facial skin color due to heart activity can be detected by digital cameras, and vital signs can be reliably extracted with the help of computer vision and optical / physiological principles. For example, heart rate, respiratory rate, heart rate activity, blood pressure, blood oxygen saturation, etc.

[0003] Neural networks are one of the most successful AI methods today. They have made major breakthroughs in particle physics, deep reinforcement learning, speech recognition, computer vision and other fields. In recent years, neural networks have set new records for many computer tasks, and the measurement accuracy of remote photoplethysmography (rPPG) has also been significantly improved after the introduction of artificial neural networks. However, neural networks bring huge computing loads and storage consumption. As we all know, network performance depends on a large amount of data and powerful hardware computing power. In order to meet the needs of rapid design and edge device deployment of neural networks in the field of rPPG, reducing the number of network parameters and compressing the model size have become the primary goals of many researchers. At the same time, no unstructured pruning method for rPPG neural networks has been found, and there is still a gap in this field.

[0004] At present, the pruning methods for neural networks are mainly divided into two categories. One is structured pruning, which targets the entire structural unit in the network, such as the filter in the convolution layer or the neuron in the fully connected layer; the other is unstructured pruning, which targets a single weight in the network, rather than the entire structural unit. Since rPPG networks usually use complex structures such as 3D convolution or LSTM to extract spatiotemporal features, the number of parameters is huge, which is not conducive to edge deployment; in addition, the training data of rPPG neural networks is relatively limited, which makes it difficult to support the design of large-scale networks; at the same time, the application scenarios of rPPG networks are diverse, such as driver monitoring, sleep monitoring, fitness, etc., and these scenarios usually have limited computing resources. Based on the above problems of rPPG neural networks, pruning them by using the high flexibility and fine-grainedness of unstructured pruning can reduce the number of network parameters, reduce computing and storage costs, and make rPPG networks more suitable for edge deployment; make the network structure simpler and easier to train, and improve the generalization ability of the model under limited data; enable rPPG neural networks to achieve high performance under limited computing resources and meet the needs of different scenarios.

[0005] From the perspective of rPPG applications in the field of physiological signal detection, the accuracy of the pruned model is undoubtedly the most important indicator. Therefore, compared with structured pruning of the entire structural unit in the network structure, unstructured pruning has the characteristics of fine-grainedness and high flexibility. It can compress the volume of the model while maintaining the accuracy of the original model to a large extent, so as to reduce the operation and deployment costs of the model and achieve more convenient and efficient practical applications. Although many methods for unstructured pruning of neural networks use the idea of ​​regeneration after pruning, there is still some room for improvement in important indicators such as accuracy and model compression rate. Summary of the invention

[0006] The object of the present invention is to provide an unstructured pruning method based on gradient and weight changes.

[0007] The present invention provides an unstructured pruning method based on gradient and weight changes, which comprises the following steps: Step 1: Collect face videos of different subjects and construct a dataset after preprocessing; Step 2: Use some samples in the data set to train the pruned neural network; Step 3: Get the connection coefficient of each node based on the weight and gradient of each node in the neural network ; Connection coefficient after sorting Prune the nodes in each layer of the neural network; Step 4: Get the ranking coefficient of each node based on the weight and gradient of each node in the neural network and comprehensive score ; By sorting the sorting coefficient and comprehensive score Restore some of the nodes that were pruned in step 3; Step 5: Repeat steps 2 and 4 until all samples in the data set complete the training of the neural network and obtain the pruned neural network model.

[0008] Preferably, in the step 1, the pretreatment process is as follows: Set the time window and step size to segment the face video, use the Viola-Jones face detector and the Lucas-Canard method to track the face in the segmented face video, crop the bounding box containing the face part in each frame image, and unify the size of the bounding box; take the image cropped in each time window as a sample of the dataset.

[0009] Preferably, in step 3, the connection coefficient The method to obtain is as follows: in, The index of the gradient after sorting in descending order; For Layer The total number of nodes; For the The weight of the i-th node in the layer; For the The gradients of the nodes in the layer; ; .

[0010] Preferably, in step 3, the specific method for determining the cutting node is: according to the connection coefficient The absolute values ​​are sorted in descending order, keeping the first Connection coefficient The corresponding node; among them, For the The sparsity of the layer.

[0011] As a preference, in step 4, the sorting coefficient The method to obtain is as follows: in, Indicates the weight after descending sorting The index of Presentation Layer The total number of nodes, For the The gradient of the i-th node in the layer; For the The weights of the nodes in the layer; ; .

[0012] As a preference, in step 4, according to the sorting coefficient The specific method for determining the restored nodes is: according to the sorting coefficient The absolute values ​​of the nodes are sorted in descending order, and the front Sorting coefficient The corresponding node; among them, is the regeneration rate.

[0013] As a preferred embodiment, in step 4, the comprehensive scoring The method to obtain is as follows: in, and Respectively, all nodes are sorted according to node gradient and node weight and the first The index of the element; Presentation Layer The total number of nodes, For the The gradients of the nodes in the layer; For the The weights of the nodes in the layer; ; .

[0014] As a preferred embodiment, in the step 4, according to the comprehensive score The specific method for determining the restored nodes is as follows: The absolute values ​​of the nodes are sorted in descending order, and the front Comprehensive rating The corresponding node; among them, is the repair rate.

[0015] Preferably, in step 4, the number of restored nodes is half of the number of pruned nodes in step 3.

[0016] Preferably, in step 5, the method for obtaining the node sparsity of the neural network in each pruning process is as follows: in, is the sparsity at the current time t; , They are the initial sparsity and the target sparsity respectively; is the pruning frequency; n is the number of iterations; To start training time; .

[0017] The present invention has the following beneficial effects: 1. The present invention prunes nodes according to the gradients and weights of the nodes in the neural network. After pruning the nodes with large weights and small gradients, the pruned nodes with small weights and large gradients and small weights and small gradients are restored respectively, thereby avoiding excessive fluctuations of the neural network, ensuring the stability and robustness of the training, avoiding premature fixing of parameters, and enhancing the generalization ability of the network. At the same time, the restored nodes can also optimize the gradient update process in the back propagation of the neural network, thereby improving the accuracy of the pruned neural network.

[0018] 2. The present invention performs a pruning operation on the neural network while training the neural network using samples, eliminating the need for retraining the neural network after pruning, thereby reducing computing resources in the actual prediction process using the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the present invention.

[0020] Figure 2 Schematic diagram of different pruning processes in the present invention.

[0021] Figure 3 Schematic diagram of comparison results of various evaluation indicators of the ablation experiment on the pruning method applied to the PhysNet model of the present invention; wherein, (a) is a schematic diagram of the signal-to-noise ratio comparison results; (b) is a schematic diagram of the mean absolute error comparison results; (c) is a schematic diagram of the root mean square error comparison results; and (d) is the mean absolute percentage error.

[0022] Figure 4 Schematic diagram of comparison results of various evaluation indicators of the ablation experiment on the pruning method applied to the DeeprPPG model of the present invention; wherein, (a) is a schematic diagram of signal-to-noise ratio comparison results; (b) is a schematic diagram of mean absolute error comparison results; (c) is a schematic diagram of root mean square error comparison results; and (d) is mean absolute percentage error.

[0023] Figure 5 Schematic diagram of comparison results of evaluation indicators of pruning the PhysNet model by the present invention and the existing pruning method; wherein, (a) is a schematic diagram of signal-to-noise ratio comparison results; (b) is a schematic diagram of mean absolute error comparison results; (c) is a schematic diagram of root mean square error comparison results; and (d) is mean absolute percentage error.

[0024] Figure 6Schematic diagram of comparison results of evaluation indicators of pruning the DeeprPPG model by the present invention and the existing pruning method; wherein, (a) is a schematic diagram of signal-to-noise ratio comparison results; (b) is a schematic diagram of mean absolute error comparison results; (c) is a schematic diagram of root mean square error comparison results; and (d) is mean absolute percentage error. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, an unstructured pruning method based on gradient and weight changes includes the following steps: Step 1: Collect face videos of different subjects; segment the face videos using a 5-second time window and a 1-second step size, use the Viola-Jones face detector to locate the face in the first frame of the face video, and use the KLT (Lucas-Kanadian) method to track the face in subsequent frames, crop the bounding box containing the face part in each frame, and resize the bounding box to 192×128; use the cropped image in each time window as a sample to construct a dataset.

[0027] Step 2: Use the samples in the dataset to train PhysNet and DeeprPPG respectively. The model structures of PhysNet and DeeprPPG are shown in Table 1.

[0028] Table 1 Model structure of PhysNet and DeeprPPG Use the samples in the dataset to train the pruned neural network once (i.e., perform a back-propagation parameter update on the weights of the neural network).

[0029] Step 3: Pruning like Figure 2 As shown, the trained neural network is pruned, and each layer in the neural network is pruned. The weights of all nodes in , and obtain its corresponding gradient ;in, ; is the number of layers of the neural network. According to the gradient The absolute value of the elements in the , and all nodes are sorted in descending order. Through this sorting, the relative size of the gradient of each node is obtained, and a connection coefficient is assigned to each node. , connection coefficient The expression is:

[0030] in, The index of the gradient after sorting in descending order; For Layer The total number of nodes; .

[0031] Connection coefficient reflects the influence of the node's gradient size on its importance; due to the index The range is 0~ ,gradient The larger the node, the closer its corresponding index is to 0; the gradient The smaller the node, the closer the corresponding index is. Therefore, nodes with small gradients and large weights will get higher connection coefficients.

[0032] According to the connection coefficient The importance of nodes in each layer is evaluated. Nodes with small gradients and large weights have a greater impact on the performance and structure of the network, so they should be retained first. The absolute values ​​are sorted in descending order, keeping the first Connection coefficient Corresponding nodes, the set of retained node sequences It is expressed as:

[0033] in, For the The sparsity of the layer; For the front Connection coefficient The index of .

[0034] After completing the selection of retained nodes, the remaining nodes are removed from the network. Through this pruning method, redundant nodes in the network are reduced while retaining those nodes that are critical to the network output and structure as much as possible.

[0035] Step 4: Regeneration and repair phase After the neural network is pruned in the pruning phase, the pruned nodes are restored through the regeneration phase and the repair phase respectively. The process is as follows: 4-1. Through the regeneration phase, the nodes that were pruned during the pruning phase due to large gradients but small weights are restored. These nodes have an important impact on network training. By combining the node's gradient information with the weight information, these nodes can be effectively restored, ensuring that the network remains flexible in subsequent training and improving the model's robustness and learning ability.

[0036] Get each layer Each node in Gradient-weight ranking coefficient , whose expression is: in, Indicates the weight after descending sorting The index of Presentation Layer The total number of nodes, For the The gradient of the i-th node in the layer.

[0037] By sorting coefficient It is possible to quantify the degree to which each node is neglected during training. Nodes with larger gradients, even if they have smaller weights, may have a greater impact on the network update during backpropagation. By calculating the ranking coefficient , evaluate the role of the node in the back-propagation process, and decide whether to restore the node.

[0038] To avoid restoring nodes that have been pruned, select nodes that were not retained during the pruning phase for regeneration. The absolute value of the larger Nodes are restored, among which, is the regeneration rate (range 0~1), which is used to control the number of nodes recovered. The node sequence set recovered during the regeneration phase It is expressed as:

[0039] in, Indicates that the gradient-weight sorting coefficient with the largest absolute value is selected Node index.

[0040] In this embodiment, the regeneration rate r is set to 0.1.

[0041] These nodes will be reactivated in the subsequent training process to play a greater role in gradient updates. Rejoin the network. Although the initial weights of the regenerated nodes are small, they play an important role in gradient updating in back propagation, especially in subsequent training, they can help the network adjust and optimize the learning process.

[0042] Regeneration rate It is a key parameter controlling the regeneration process, which determines how many pruned nodes will be restored. By setting the regeneration rate appropriately, a balance can be found between pruning and restoration, reducing redundant calculations while maintaining the flexibility and adaptability of the network. A reasonable regeneration rate helps prevent the model from fixing certain nodes too early, thereby improving the robustness and learning ability of the model. The choice of regeneration rate needs to be adjusted according to the specific task and training stage.

[0043] 4-2. Restore the nodes that were pruned during the pruning phase. Although nodes with small weights and small gradients may not be valued in the initial training, they help the network avoid excessive fluctuations and maintain the stability of network parameters by providing stable gradient feedback, which is crucial to the robustness and generalization ability of the model. In order to ensure that these nodes are not over-pruned and maintain the integrity of the network structure, it is very necessary to restore these nodes, especially those lost during the training process. The specific method is as follows:

[0044] Combine the weights of each node and gradient Get the comprehensive score of each node , based on the comprehensive rating Evaluate the importance of nodes and give a comprehensive score The calculation formula is: in, and Respectively, all nodes are sorted according to node gradient and node weight and the first The index of the element.

[0045] By comprehensive rating The node gradient and weight are considered comprehensively, and nodes with small weight but important role in training can be extracted. Sort the nodes that were pruned in the pruning phase and not restored in the regeneration phase. nodes; among them, is the repair rate, which controls the number of recovery nodes, and its value is; is the pruning rate. The node sequence set restored in the repair phase It is expressed as:

[0046] in, Indicates that the maximum absolute value of the comprehensive score is selected Node index.

[0047] Through this process, the nodes that still play an important role in gradient feedback are restored. Although their weights are small, they can provide stable feedback during training, avoid excessive fluctuations in the network, and ensure the stability of training. The nodes restored in the patching phase are reintroduced into the network. These restored nodes help the network maintain stability and robustness during training through stable gradient feedback, avoid premature fixing of certain parameters, and enhance the generalization ability of the network. Restoring these nodes helps maintain the structural integrity of the network, especially during long-term training, enabling the network to better adapt to complex tasks.

[0048] By restoring these nodes, the network can maintain more flexibility and stability, avoid over-reliance on nodes that were fixed prematurely during the initial training process, and further improve the robustness and adaptability of the model. Although these nodes are not conspicuous in the early stage, they can have a significant positive impact on the learning and generalization of the network during the later training process.

[0049] Step 5: Repeat steps 2 and 4. During each training process, the sparsity of the neural network at the current time t is The expression is: in, , They are the initial sparsity and the target sparsity respectively; is the pruning frequency; n is the number of iterations; Time to start training; .

[0050] Time from start of training Initially, the network prunes at a frequency of Iterate once to make the sparsity increase from the initial sparsity Gradually reduce to the target sparsity , that is, all samples in the data set complete the training of the neural network and realize the pruning of the neural network.

[0051] Step 6: The results of the ablation experiment are as follows Figure 3 and Figure 4 As shown. The effect of each step in WGJ (pruning, regeneration and patching) is tested separately. The subscript numbers are related to the three components of pruning, regeneration and patching respectively. The number "000" represents naive pruning, in which the network is pruned directly to the final sparsity. The results show that naive pruning ("000") has a clear correlation with the final sparsity, that is, for PhysNet, the SNR (signal-to-noise ratio) decreases with the decrease of the final sparsity, while for PhysNet, as the final sparsity decreases, the SNR decreases and the error metric increases. For DeeprPPG, when the target sparsity s fWhen s is larger, the performance is better than that of dense networks, while when the target sparsity s f When is smaller, the performance gets worse. This fully reflects the result of indiscriminate removal of neurons. The performance is improved by gradual pruning ('100'), and it is further improved when regeneration is added ('110'). The performance is best when all three components are present ('111').

[0052] The proposed WGJ pruning method is compared with existing methods. The network starts from the initial sparsity s i = 1:0 (dense network) to target sparsity s f The training was performed in the range of 0:9~0:1. The results are as follows Figure 5 and Figure 6 As shown in Figure 2. It can be observed that after applying the pruning method to PhysNet, the SNR increases overall compared to the original network, while three methods (WGJ, DSR, RigL) show an increase in SNR and two methods (SET and GraNet) show a decrease in SNR after applying the pruning method to DeeprPPG. This can be explained by the following two reasons. First, PhysNet contains high redundancy in terms of training data, and the pruning method accurately reduces redundant neurons to achieve the best performance. In contrast, DeeprPPG itself does not have a large number of redundant neurons, and the pruning method can easily prune away the originally important neurons. Second, the existing pruning methods cannot distinguish between important and unimportant neurons, resulting in a decrease in performance compared to the original network, while the proposed WJG algorithm always prunes unimportant neurons, resulting in superior performance for both PhysNet and DeeprPPG networks. For MAE (mean absolute error), RMSE (root mean square error), and MAPE (mean absolute percentage error), WGJ also obtains the lowest scores in most cases.

Claims

1. An unstructured pruning method based on gradient and weight changes, characterized in that: The following steps are involved: Step 1: Collect face videos of different subjects and construct a dataset after preprocessing; Step 2: Use some samples in the data set to train the pruned neural network; Step 3: Get the connection coefficient of each node based on the weight and gradient of each node in the neural network ; Connection coefficient after sorting Prune the nodes in each layer of the neural network; Step 4: Get the ranking coefficient of each node based on the weight and gradient of each node in the neural network and comprehensive score ; By sorting the sorting coefficient and comprehensive score Restore some of the nodes that were pruned in step 3; Step 5: Repeat steps 2 and 4 until all samples in the data set complete the training of the neural network and obtain the pruned neural network model.

2. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In the step 1, the pre-processing process is as follows: Set the time window and step size to segment the face video, use the Viola-Jones face detector and the Lucas-Canard method to track the face in the segmented face video, crop the bounding box containing the face part in each frame image, and unify the size of the bounding box; take the image cropped in each time window as a sample of the dataset.

3. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In step 3, the connection coefficient The method to obtain is as follows: in, The index of the gradient after sorting in descending order; For Layer The total number of nodes; For the The weight of the i-th node in the layer; For the The gradients of the nodes in the layer; ; .

4. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In step 3, the specific method for determining the cutting node is: according to the connection coefficient The absolute values ​​are sorted in descending order, keeping the first Connection coefficient The corresponding node; among them, For the The sparsity of the layer.

5. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In step 4, the sorting coefficient The method to obtain is as follows: in, Indicates the weight after descending sorting The index of Presentation Layer The total number of nodes, For the The gradient of the i-th node in the layer; For the The weights of the nodes in the layer; ; .

6. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In the step 4, according to the sorting coefficient The specific method for determining the restored nodes is: according to the sorting coefficient The absolute values ​​of the nodes are sorted in descending order, and the front Sorting coefficient The corresponding node; among them, is the regeneration rate.

7. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In step 4, the comprehensive score The method to obtain is as follows: in, and Respectively, all nodes are sorted according to node gradient and node weight and the first The index of the element; Presentation Layer The total number of nodes, For the The gradients of the nodes in the layer; For the The weights of the nodes in the layer; ; .

8. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In step 4, according to the comprehensive score The specific method for determining the restored nodes is as follows: The absolute values ​​of the nodes are sorted in descending order, and the front Comprehensive rating The corresponding node; among them, The repair rate.

9. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In the step 4, the number of restored nodes is half of the number of pruned nodes in the step 3.

10. The unstructured pruning method based on gradient and weight change according to claim 1, characterized in that: In the step 5, the method for obtaining the node sparsity of the neural network during each pruning process is as follows: in, is the sparsity at the current time t; , They are the initial sparsity and the target sparsity respectively; is the pruning frequency; n is the number of iterations; To start training time; .

Images