An unstructured pruning method based on gradient and weight changes

Abstract

The present invention discloses an unstructured pruning method based on gradient and weight changes. The method performs pruning operations during the training of a neural network, eliminates the need for retraining the neural network after pruning, and reduces computing resources in the actual prediction process using the neural network. Simultaneously, during the pruning of the neural network, the connection coefficient, ranking coefficient, and comprehensive score of each node are obtained according to the weight and gradient of each node of the neural network. The nodes in each layer of the neural network are pruned using the sorted connection coefficients, and the pruned nodes in the neural network are restored using the ranking coefficients and comprehensive scores. This avoids excessive fluctuations in the neural network, ensures the stability and robustness of the training, avoids premature parameter fixing, and enhances the generalization ability of the network. Simultaneously, the restored nodes can also optimize the gradient update process in the back propagation of the neural network, thereby improving the accuracy of the pruned neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of unstructured pruning, and in particular relates to an unstructured pruning method based on gradient and weight changes. Background Art

[0002] Vital sign measurement has evolved from contact-based to contactless methods, reducing the inconvenience of contact-based methods and expanding its application. Measuring vital signs, once cumbersome in hospitals, is now readily accessible. This shift stems from a new technology: remote photoplethysmography (rPPG). Digital cameras can detect subtle changes in facial skin tone due to cardiac activity. Using computer vision and optical / physiological principles, vital signs can be reliably extracted. Examples include heart rate, respiratory rate, heart rate activity, blood pressure, and blood oxygen saturation.

[0003] Neural networks are one of the most successful AI methods today, achieving significant breakthroughs in fields such as particle physics, deep reinforcement learning, speech recognition, and computer vision. In recent years, neural networks have set new records for many computer tasks, and the measurement accuracy of remote photoplethysmography (rPPG) has been significantly improved with the introduction of artificial neural networks. However, neural networks impose enormous computational loads and storage requirements. As is well known, network performance relies on large amounts of data and powerful hardware computing power. To meet the demands of rapid design and edge device deployment for practical applications of neural networks in the rPPG field, reducing network parameters and compressing model size have become primary goals for many researchers. However, no unstructured pruning methods for rPPG neural networks have been discovered, leaving some gaps in this area.

[0004] Current neural network pruning methods fall into two main categories: structured pruning, which targets entire structural units within the network, such as filters in a convolutional layer or neurons in a fully connected layer; and unstructured pruning, which targets individual weights rather than entire structural units. Because rPPG networks typically use complex structures like 3D convolutions or LSTMs to extract spatiotemporal features, they have a large number of parameters, making them unsuitable for edge deployment. Furthermore, training data for rPPG networks is relatively limited, making it difficult to design large-scale networks. Furthermore, rPPG networks have diverse application scenarios, such as driver monitoring, sleep monitoring, and fitness, which often have limited computing resources. To address these challenges with rPPG networks, unstructured pruning, leveraging its high flexibility and fine-grained nature, can reduce the number of network parameters, lower computational and storage costs, and make them more suitable for edge deployment. It also makes the network structure simpler and easier to train, improving the model's generalization ability with limited data. It also enables rPPG networks to achieve high performance with limited computing resources, meeting the needs of diverse scenarios.

[0005] In the application of rPPG in physiological signal detection, the accuracy of the pruned model is undoubtedly the most important indicator. Therefore, compared to structured pruning, which targets entire structural units in the network structure, unstructured pruning offers fine-grainedness and high flexibility. It can compress the model while largely maintaining the accuracy of the original model, thereby reducing the model's operating and deployment costs and enabling more convenient and efficient practical applications. Although many methods for unstructured pruning of neural networks currently use the concept of post-pruning regeneration, there is still room for improvement in important indicators such as accuracy and model compression rate. Summary of the Invention

[0006] The purpose of the present invention is to provide an unstructured pruning method based on gradient and weight changes.

[0007] The present invention provides an unstructured pruning method based on gradient and weight changes, which comprises the following steps:

[0008] Step 1: Collect facial videos of different subjects and construct a dataset after preprocessing;

[0009] Step 2: Use some samples in the data set to train the pruned neural network;

[0010] Step 3: Get the connection coefficient of each node based on the weight and gradient of each node in the neural network ; Connection coefficient after sorting Prune the nodes in each layer of the neural network;

[0011] Step 4: Get the ranking coefficient of each node based on the weight and gradient of each node in the neural network and comprehensive score ; By sorting the sorting coefficient and comprehensive score Restore some of the nodes that were pruned in step 3;

[0012] Step 5: Repeat steps 2 and 4 until all samples in the data set complete the training of the neural network and obtain the pruned neural network model.

[0013] Preferably, in step 1, the pretreatment process is as follows:

[0014] Set the time window and step size to segment the face video, use the Viola-Jones face detector and the Lucas-Canard method to track the face in the segmented face video, crop the bounding box containing the face part in each frame image, and unify the size of the bounding box; the image cropped in each time window is used as a sample of the dataset.

[0015] As a preference, in the step 3, the connection coefficient The method to obtain is as follows:

[0016]

[0017] in, The index of the gradient after descending sorting; For layer The total number of midpoints; For the The weight of the i-th node in the layer; For the The gradients of the nodes in the layer; ; .

[0018] As a preference, in step 3, the specific method for determining the cutting nodes is: according to the connection coefficient The absolute values ​​are sorted in descending order, and the top Connection coefficient The corresponding node; among them, For the The sparsity of the layer.

[0019] As a preference, in the step 4, the sorting coefficient The method to obtain is as follows:

[0020]

[0021] in, Indicates the weight after descending sorting 's index; Presentation Layer The total number of nodes, For the The gradient of the i-th node in the layer; For the The weights of the nodes in the layer; ; .

[0022] As a preference, in the step 4, according to the sorting coefficient The specific method for determining the restored nodes is: according to the sorting coefficient Arrange in descending order the absolute value of the pruned nodes and restore the front Sorting coefficient The corresponding node; among them, is the regeneration rate.

[0023] As a preference, in step 4, the comprehensive scoring The method to obtain is as follows:

[0024]

[0025] in, and Respectively, it means sorting all nodes according to node gradient and node weight and returning the The index of the element; Presentation Layer The total number of nodes, For the The gradients of the nodes in the layer; For the The weights of the nodes in the layer; ; .

[0026] As a preference, in step 4, according to the comprehensive score The specific method for determining the restored nodes is: based on the comprehensive score Arrange in descending order the absolute value of the pruned nodes and restore the front Comprehensive rating The corresponding node; among them, is the repair rate.

[0027] Preferably, in step 4, the number of restored nodes is half of the number of pruned nodes in step 3.

[0028] Preferably, in step 5, the method for obtaining the node sparsity of the neural network during each pruning process is as follows:

[0029]

[0030] in, is the sparsity at the current moment t; , are the initial sparsity and target sparsity respectively; is the pruning frequency; n is the number of iterations; Time to start training; .

[0031] The present invention has the following beneficial effects:

[0032] 1. The present invention prunes nodes in a neural network based on their gradients and weights. After pruning nodes with large weights and small gradients, the pruned nodes with small weights and large gradients and those with small weights and small gradients are restored. This avoids excessive fluctuations in the neural network, ensures the stability and robustness of training, avoids premature parameter fixing, and enhances the generalization ability of the network. At the same time, the restored nodes can also optimize the gradient update process in the back propagation of the neural network, thereby improving the accuracy of the pruned neural network.

[0033] 2. The present invention performs a pruning operation on the neural network while training the neural network using samples, eliminating the need for retraining the neural network after pruning, thereby reducing computing resources in the actual prediction process using the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flowchart of the present invention.

[0035] Figure 2 Schematic diagram of different pruning processes in the present invention.

[0036] Figure 3 Schematic diagram of the comparison results of various evaluation indicators of the ablation experiment on the pruning method applied to the PhysNet model of the present invention; among them, (a) is a schematic diagram of the signal-to-noise ratio comparison results; (b) is a schematic diagram of the mean absolute error comparison results; (c) is a schematic diagram of the root mean square error comparison results; and (d) is the mean absolute percentage error.

[0037] Figure 4 Schematic diagram of the comparison results of various evaluation indicators of the ablation experiment on the pruning method applied to the DeeprPPG model of the present invention; among them, (a) is a schematic diagram of the signal-to-noise ratio comparison results; (b) is a schematic diagram of the mean absolute error comparison results; (c) is a schematic diagram of the root mean square error comparison results; and (d) is the mean absolute percentage error.

[0038] Figure 5Schematic diagram of the comparison results of various evaluation indicators for pruning the PhysNet model using the present invention and existing pruning methods; among them, (a) is a schematic diagram of the signal-to-noise ratio comparison results; (b) is a schematic diagram of the mean absolute error comparison results; (c) is a schematic diagram of the root mean square error comparison results; and (d) is the mean absolute percentage error.

[0039] Figure 6 Schematic diagram of the comparison results of various evaluation indicators of the DeeprPPG model pruning performed by the present invention and the existing pruning method; among them, (a) is a schematic diagram of the signal-to-noise ratio comparison results; (b) is a schematic diagram of the mean absolute error comparison results; (c) is a schematic diagram of the root mean square error comparison results; and (d) is the mean absolute percentage error. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings.

[0041] like Figure 1 As shown in FIG, an unstructured pruning method based on gradient and weight changes includes the following steps:

[0042] Step 1: Collect face videos of different subjects; segment the face videos using a 5-second time window and a 1-second step size, use the Viola-Jones face detector to locate the face in the first frame of the face video, and use the KLT (Lucas-Kanadian) method to track the face in subsequent frames, crop the bounding box containing the face part in each frame image, and resize the bounding box to 192×128; use the cropped image in each time window as a sample to construct the dataset.

[0043] Step 2: Use the samples in the dataset to train PhysNet and DeeprPPG respectively. The model structures of PhysNet and DeeprPPG are shown in Table 1.

[0044] Table 1 Model structure of PhysNet and DeeprPPG

[0045]

[0046] Use the samples in the dataset to train the pruned neural network once (that is, perform a backpropagation parameter update on the weights of the neural network).

[0047] Step 3: Pruning

[0048] like Figure 2 As shown, the trained neural network is pruned, and each layer in the neural network is pruned. The weights of all nodes in , and obtain its corresponding gradient ;in, ; is the number of layers of the neural network. According to the gradient The absolute value of the elements in the array is used to sort all nodes in descending order. Through this sorting, the relative size of the gradient of each node is obtained, and a connection coefficient is assigned to each node. , connection coefficient The expression is:

[0049]

[0050] in, The index of the gradient after descending sorting; For layer The total number of midpoints; .

[0051] Connection coefficient Reflects the influence of the node's gradient size on its importance; due to the index The range is 0~ ,gradient The larger the node, the closer its corresponding index is to 0; the gradient The smaller the node, the closer the corresponding index is. Therefore, nodes with small gradients and large weights will receive higher connection coefficients.

[0052] According to the connection coefficient The importance of nodes in each layer is evaluated. Nodes with small gradients and large weights have a greater impact on the performance and structure of the network, so they should be retained first. The absolute values ​​are sorted in descending order, and the top Connection coefficient Corresponding nodes, retained node sequence set Expressed as:

[0053]

[0054] in, For the the sparsity of the layer; For the front Connection coefficient The index of .

[0055] After the selection of retained nodes is completed, the remaining nodes are removed from the network. Through this pruning method, redundant nodes in the network are reduced while retaining as many nodes as possible that are critical to the network output and structure.

[0056] Step 4: Regeneration and repair phase

[0057] After the neural network is pruned in the pruning phase, the pruned nodes are restored through the regeneration phase and the repair phase respectively. The process is as follows:

[0058] 4-1. The regeneration phase restores nodes that were pruned during the pruning phase due to large gradients but small weights. These nodes are crucial for network training. By combining node gradient information with weight information, these nodes can be effectively restored, ensuring the network remains flexible during subsequent training and improving the model's robustness and learning ability.

[0059] Get each layer Each node in Gradient-weight ranking coefficient , whose expression is:

[0060]

[0061] in, Indicates the weight after descending sorting 's index; Presentation Layer The total number of nodes, For the The gradient of the i-th node in the layer.

[0062] By sorting coefficient It can quantify the degree to which each node is neglected during training. Nodes with larger gradients, even if their weights are smaller, may have a greater impact on the network update during backpropagation. By calculating the ranking coefficient , evaluate the role of the node in the back-propagation process, and decide whether to restore the node.

[0063] To avoid restoring nodes that have been pruned, select nodes that were not retained during the pruning phase for regeneration. The absolute value is larger Nodes are restored, among which, is the regeneration rate (range 0~1), which is used to control the number of nodes recovered. The node sequence set recovered during the regeneration phase Expressed as:

[0064]

[0065] in, Indicates that the gradient-weight sorting coefficient with the largest absolute value is selected Node index.

[0066] In this embodiment, the regeneration rate r is set to 0.1.

[0067] These nodes will be reactivated in the subsequent training process to play a greater role in gradient updates. Rejoin the network. Although the initial weights of the regenerated nodes are small, they play an important role in gradient updates in backpropagation, especially in subsequent training, as they can help the network adjust and optimize the learning process.

[0068] Regeneration rate The regeneration rate is a key parameter controlling the regeneration process, determining how many pruned nodes are restored. By appropriately setting the regeneration rate, a balance can be found between pruning and restoration, reducing redundant computation while maintaining network flexibility and adaptability. A reasonable regeneration rate helps prevent the model from prematurely fixing certain nodes, thereby improving the model's robustness and learning ability. The choice of regeneration rate needs to be adjusted based on the specific task and training stage.

[0069] 4-2. Restore the nodes that were pruned during the pruning phase. Although nodes with small weights and small gradients may not be valued in the initial training, they help the network avoid excessive fluctuations and maintain the stability of network parameters by providing stable gradient feedback, which is crucial to the robustness and generalization ability of the model. In order to ensure that these nodes are not over-pruned and maintain the integrity of the network structure, it is very necessary to restore these nodes, especially those lost during training. The specific method is as follows:

[0070] Combine the weights of each node and gradient Get the comprehensive score of each node , based on the comprehensive score Evaluate the importance of nodes and give a comprehensive score The calculation formula is:

[0071]

[0072] in, and Respectively, it means sorting all nodes according to node gradient and node weight and returning the The index of the element.

[0073] Through comprehensive scoring Taking into account the gradient and weight of the node, it is possible to extract nodes with small weight but important role in training. Sort the nodes that were pruned in the pruning phase and not recovered in the regeneration phase. nodes; among them, is the repair rate, which controls the number of recovery nodes and its value is; is the pruning rate. The node sequence set restored in the repair phase Expressed as:

[0074]

[0075] in, Indicates that the maximum absolute value of the comprehensive score is selected Node index.

[0076] Through this process, nodes that still play an important role in gradient feedback are restored. Although their weights are small, they can provide stable feedback during training, avoiding excessive network fluctuations and ensuring training stability. The nodes restored during the repair phase are reintroduced into the network. These restored nodes, through stable gradient feedback, help maintain network stability and robustness during training, avoid premature parameter fixation, and enhance the network's generalization ability. Restoring these nodes helps maintain the structural integrity of the network, especially during long training sessions, enabling the network to better adapt to complex tasks.

[0077] By restoring these nodes, the network can maintain greater flexibility and stability, avoiding over-reliance on nodes that were prematurely fixed during initial training, further improving the model's robustness and adaptability. Although these nodes are not noticeable in the early stages, they can have a significant positive impact on the network's learning and generalization during later training.

[0078] Step 5: Repeat steps 2 and 4. During each training process, the sparsity of the neural network at the current time t is The expression is:

[0079]

[0080] in, , are the initial sparsity and target sparsity respectively; is the pruning frequency; n is the number of iterations; Time to start training; .

[0081] Time from start of training Initially, the network prunes at a frequency of Iterate once to make the sparsity increase from the initial sparsity Gradually reduce to the target sparsity , that is, all samples in the data set complete the training of the neural network and realize the pruning of the neural network.

[0082] Step 6: The results of the ablation experiment are as follows Figure 3 and Figure 4As shown. The effect of each step in WGJ (pruning, regeneration and patching) is tested separately. The subscript numbers are related to the three components of pruning, regeneration and patching respectively. The number "000" represents naive pruning, in which the network is pruned directly to the final sparsity. The results show that naive pruning ("000") has a clear correlation with the final sparsity, that is, for PhysNet, the SNR (signal-to-noise ratio) decreases with the decrease of the final sparsity, while for PhysNet, the SNR decreases and the error metric increases with the decrease of the final sparsity. For DeeprPPG, when the target sparsity s f When the target sparsity s is large, the performance is better than that of dense networks. f When is smaller, performance deteriorates. This reflects the indiscriminate removal of neurons. Performance improves with gradual pruning ('100'), and further increases when regeneration is added ('110'). Performance is best when all three components are present ('111').

[0083] The proposed WGJ pruning method is compared with existing methods. The network starts from the initial sparsity s i = 1:0 (dense network) to target sparsity s f The training was conducted in the range of 0:9~0:1, and the results were as follows Figure 5 and Figure 6 As shown in Figure 3, after applying pruning methods to PhysNet, the SNR generally increases compared to the original network. While three methods (WGJ, DSR, and RigL) show an increase in SNR, two methods (SET and GraNet) show a decrease in SNR after applying pruning methods to DeeprPPG. This can be explained by two reasons. First, PhysNet contains a high level of redundancy in its training data, and pruning methods precisely reduce redundant neurons to achieve optimal performance. In contrast, DeeprPPG itself does not have a large number of redundant neurons, making pruning methods easily prune away otherwise important neurons. Second, existing pruning methods fail to distinguish between important and unimportant neurons, resulting in performance degradation compared to the original network. The proposed WJG algorithm consistently prunes unimportant neurons, resulting in superior performance for both PhysNet and DeeprPPG networks. WGJ also achieves the lowest scores in most cases for MAE (mean absolute error), RMSE (root mean square error), and MAPE (mean absolute percentage error).

Claims

1. An unstructured pruning method based on gradient and weight changes, characterized by: The following steps are involved: Step 1: Collect facial videos of different subjects and construct a dataset after preprocessing; Step 2: Use some samples in the dataset to train the pruned neural network; Step 3: Get the connection coefficient of each node based on the weight and gradient of each node in the neural network ; Connection coefficient after sorting Prune the nodes in each layer of the neural network; In the step 3, the connection coefficient The method to obtain is as follows: ； in, Gradient after descending sorting 's index; For layer The total number of midpoints; For the The weight of the i-th node in the layer; For the The gradients of the nodes in the layer; ; ; Step 4: Get the ranking coefficient of each node based on the weight and gradient of each node in the neural network and comprehensive score ; By sorting the sorting coefficient and comprehensive score Restore some of the nodes that were pruned in step 3; In the step 4, the sorting coefficient The method to obtain is as follows: ； in, Indicates the weight after descending sorting 's index; For the The gradient of the i-th node in the layer; For the The weights of the nodes in the layer; Step 5: Repeat steps 2 to 4 until all samples in the data set complete the training of the neural network and obtain the pruned neural network model.

2. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In the step 1, the pre-processing process is as follows: Set the time window and step size to segment the face video, use the Viola-Jones face detector and the Lucas-Canard method to track the face in the segmented face video, crop the bounding box containing the face part in each frame image, and unify the size of the bounding box; the image cropped in each time window is used as a sample of the dataset.

3. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In step 3, the specific method for determining the cutting node is: according to the connection coefficient The absolute values ​​are sorted in descending order, and the top Connection coefficient The corresponding node; among them, For the The sparsity of the layer.

4. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In the step 4, according to the sorting coefficient The specific method for determining the restored nodes is: according to the sorting coefficient Arrange in descending order the absolute value of the pruned nodes and restore the front Sorting coefficient The corresponding node; among them, is the regeneration rate; For the The sparsity of the layer.

5. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In step 4, the comprehensive score The method to obtain is as follows: 。 6. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In the step 4, according to the comprehensive score The specific method for determining the restored nodes is: based on the comprehensive score Arrange in descending order the absolute value of the pruned nodes and restore the front Comprehensive rating The corresponding node; among them, is the repair rate; For the The sparsity of the layer.

7. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In step 4, the number of restored nodes is half of the number of pruned nodes in step 3.

8. The unstructured pruning method based on gradient and weight changes according to claim 1, characterized in that: In step 5, the node sparsity of the neural network is obtained in each pruning process as follows: ； in, is the sparsity at the current moment t; , are the initial sparsity and target sparsity respectively; is the pruning frequency; n is the number of iterations; Time to start training; .

Images