Motion recognition model lightweight method based on hierarchical dynamic fusion network pruning

Abstract

The invention discloses an action recognition model lightweight method based on hierarchical dynamic fusion network pruning. The method comprises the following steps: uniformly sampling an original video to obtain an RGB frame sequence, and constructing an initial action recognition model; designing a multi-granularity computational graph coding module, weighting and randomly selecting pruning granularity according to the redundancy of the current granularity network parameters, and dynamically describing an interlayer parameter relationship by utilizing the frequency domain similarity of each granularity parameter output feature graph, so as to generate a corresponding hierarchical graph structure; thirdly, searching a key path in the graph by utilizing a self-adaptive pruning fusion module to dynamically prune redundant parameter nodes; and finally, performing fine tuning optimization on the pruned action recognition model to obtain a lightweight action recognition network. According to the method, the capability of describing time sequence dependence by network parameters can be evaluated, the pruning position, the pruning granularity and the pruning proportion can be adaptively adjusted, the reasoning speed of the action recognition model is increased, the quantity of model parameters is reduced, and lightweight deployment of the action recognition model is facilitated.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, in particular to the field of visual model compression and action recognition, and relates to a lightweight method for action recognition models based on hierarchical dynamic fusion network pruning. Background Art

[0002] Action recognition is an important computer vision task that aims to identify single action types contained in videos, such as "running" and "jumping," by analyzing the spatial and temporal information in video frames. It has widespread application in the field of autonomous driving. Early approaches, whether combining two-stream networks (processing single-frame images) and temporal streams (processing optical flow images) or 3D convolutional networks (convolutional networks) that perform convolution operations on the spatiotemporal dimensions of videos to capture spatiotemporal information, have exponentially increased the amount of computation required to capture long-term temporal dependencies and high-resolution video samples. In recent years, with the remarkable success of deep neural networks based on the Transformer architecture in computer vision, researchers have proposed video vision Transformer models for action recognition tasks. These models begin to leverage self-attention mechanisms to capture long-range dependencies, significantly improving the model's ability to model temporal relationships. However, deep learning neural networks based on the video Transformer architecture typically have a large number of parameters, significantly increasing computational cost and training difficulty, extending inference time, and making them difficult to deploy on edge and end-device devices with limited computing resources. Therefore, how to achieve efficient computing and accurate recognition at the same time has become a key issue in actual deployment. The lightweight video action recognition model has become an urgent need. How to reduce the complexity and number of parameters of the network model, improve training efficiency, and shorten inference time without losing accuracy as much as possible, so that it can be deployed on small devices with limited computing resources and efficiently complete real-time recognition tasks, has important application value.

[0003] Action recognition models aim to assign category labels to videos containing a single action. Early research primarily used CNNs (Convolutional Neural Networks) as the backbone network and extracted spatiotemporal features in two ways: one was to use a two-stream network to extract RGB features in the spatial dimension and optical flow features in the temporal dimension; the other was to use a 3D CNN to directly model spatiotemporal information. To fully combine the advantages of two-stream networks and 3D CNNs, the classic SlowFast network focuses on capturing spatial semantic information in the slow pathway, while the fast pathway is used to extract motion features at a fine temporal resolution. In recent years, researchers have proposed deep neural networks for action recognition based on the Transformer architecture, and further improved their performance by pre-training models on large-scale datasets. The classic work, the Swin Transformer, achieves information interaction between different windows through window shifting and extracts multi-scale spatiotemporal features using a hierarchical structure, significantly improving video action recognition.

[0004] In autonomous driving, motion recognition models can improve driving safety and interaction capabilities by analyzing the behavior of pedestrians, drivers, and other traffic participants. For example, identifying actions such as pedestrians "suddenly running" or cyclists "falling" can help vehicles avoid obstacles in advance; monitoring the driver's "yawning" or "looking down at the phone" and other states can provide timely warnings of fatigue driving. How to ensure the real-time performance of model recognition in these complex real-world scenarios is a key technical challenge in the application of motion recognition models to autonomous driving.

[0005] Network pruning, a classic method for model compression and lightweighting, aims to reduce the number of network weight parameters, improve inference efficiency, and lower model storage requirements, resulting in lightweight models. This allows for fast, real-time inference on resource-constrained learning tasks. Existing pruning methods can be categorized into two main categories: structured pruning and unstructured pruning, depending on the pruning granularity. Structured pruning removes redundant components from the network at a larger granularity (e.g., filters, channels, or layers). For example, channel pruning compresses the model by deleting entire feature channels, while layer pruning directly removes entire network layers. These methods typically maintain a regular network structure, are compatible with general-purpose hardware, and do not require specialized library support. Unstructured pruning prunes action recognition models at a finer granularity (e.g., individual weights or neurons). For example, weight pruning achieves sparsification by zeroing out unimportant connections. While this method achieves significant compression, the resulting lightweight model exhibits an irregular coefficient structure and requires the use of sparse computing libraries for acceleration.

[0006] The shortcomings of the above methods are mainly manifested in the following aspects: (1) Most existing network pruning methods adopt a fixed pruning granularity strategy, and the pruning ratio depends on manual setting. However, the fixed pruning granularity cannot adapt to the model network structure that changes dynamically during the pruning process. During the model pruning process, the network structure changes dynamically with each round of pruning operations. The fixed pruning granularity (such as fixed channel, filter or layer pruning) cannot adapt to this dynamic change, resulting in unsatisfactory pruning effect; (2) Existing network pruning methods almost assume that the pruning ratio of each layer is the same, but the parameter redundancy of different layers is not consistent. If the network layers are pruned uniformly, the parameters of the key layers will inevitably be over-pruned. Some layers still have many redundant parameters, resulting in a low upper limit of model compression and serious performance loss; (3) Existing pruning methods for video action models have similarities in the ability of parameters to depict temporal dependencies, which makes the pruning process destroy the key temporal feature extraction ability of the action recognition model, resulting in the difficulty of the pruned lightweight model to accurately capture the dynamic changes of the action. Therefore, based on these considerations, there is an urgent need to design a pruning method for action recognition models that can fully evaluate the ability of parameters to characterize temporal dependencies and adaptively adjust the pruning position, pruning granularity, and pruning ratio. Summary of the Invention

[0007] The purpose of the present invention is to address the shortcomings of the existing technology and provide a lightweight method for action recognition models based on hierarchical dynamic fusion network pruning. According to the redundancy of each granularity in the pruning process, the pruning granularity of each round is dynamically adjusted; according to the parameters that characterize the temporal dependence of video samples and the degree of similarity of the capabilities, the graph structure is used to screen redundant network parameters layer by layer, perform network pruning, and combine network fine-tuning and other methods to ultimately achieve lightweight action recognition models.

[0008] The lightweight method of action recognition model based on hierarchical dynamic fusion network pruning includes the following steps:

[0009] Step (1): Convert the video containing action category labels into a number of video clips consisting of frame images; construct an original action recognition network with the video clips as input and action categories as output initialization The redundancy of pruning granularity is different according to the pruning granularity. The weights of each layer in the network are refined separately to obtain the refined weights;

[0010] Step (2): Select the pruning granularity according to the redundancy; select a video clip input In the above example, we get the output feature map F of the lth layer. l ; l represents Any layer in the feature map F according to the selected pruning granularityl Group and get the feature map of the i-th weight output i represents any weight at the pruning granularity given by the lth layer; the optimization objective function is constructed based on the variational mode decomposition method to obtain the feature map The eigenmode of With center frequency If the pruning granularity is the filter granularity or the channel granularity, Construct the graph layer by layer. For the first layer, the refined weights corresponding to the pruning granularity are used as nodes in the graph. With center frequency Get the weight A of the edge in the graph pq , and then get the adjacency matrix A of the current layer graph l ;

[0011] Step (3): Initialize an empty forest Each node is independently treed to construct the maximum spanning tree; the adjacency matrix A l Sort the capacity c between all nodes in the network from large to small, and select the edge with the largest capacity. Join the Forest until all nodes are connected by trees; for forest Every tree in Search the farthest node twice and get the tree All nodes on the diameter of the node are taken as key nodes; the refined weights corresponding to other nodes except the key nodes are proportionally integrated into the corresponding weights of the nodes adjacent to the other nodes according to the capacity c between all nodes, and the pruned action recognition model is obtained.

[0012] Preferably, in step (1), the conversion specifically comprises the following steps:

[0013] Uniformly sampling the video containing the action category label to obtain a frame stream image sequence in RGB format, and uniformly sampling the frame stream image sequence to obtain a plurality of video clips composed of frame images;

[0014] described It includes: a spatial feature extractor based on the ResNet101 network, a spatiotemporal feature fusion module based on three-layer multi-head self-attention, a fully connected layer and a softmax(·) function;

[0015] described The redundancy of the pruning granularity includes the redundancy of the three pruning granularities of filter, channel, and layer. The redundancy values of the three pruning granularities of filter, channel, and layer are equal and between 0 and 1.

[0016] Preferably, in step (2), selecting the pruning granularity according to the redundancy size specifically includes:

[0017] . According to the redundancy of the pruning granularity in step (1), the selection probability of the pruning granularity is set according to the redundancy. The higher the redundancy, the higher the selection probability of the granularity. A pruning granularity is randomly selected according to the selection probability.

[0018] Preferably, in step (2),

[0019] According to the eigenmode With center frequency Get the weight A of the edge in the graph pq , and then get the adjacency matrix A of the current layer graph l , specifically including:

[0020] Based on the center frequency of adjacent nodes Calculate the frequency similarity of adjacent nodes;

[0021] Eigenmodes based on adjacent nodes Calculate modal component similarity;

[0022] The weight A of the edge in the graph is obtained by weighting the frequency similarity and modal component similarity. pq ;

[0023] The adjacency matrix A l , whose diagonal elements c pp =1, according to weight A pq Set the threshold of off-diagonal elements to filter out weight A pq The connection is less than the threshold, the adjacency matrix A l The distance A between node p and node q pq =c pq , the capacity between node p and node q in the graph

[0024] Preferably, in step (3), the maximum spanning tree is constructed using the Kruskal algorithm; the step of connecting all nodes by trees further includes: if node p and node q do not belong to the same tree, merging the trees corresponding to the two nodes until all nodes are connected by one or more trees.

[0025] Preferably, in step (3),

[0026] The search for the farthest node twice yields the tree All nodes on the diameter are regarded as key nodes, which specifically includes the following steps:

[0027] Select any starting node p, use the breadth-first algorithm to find the node q farthest from the starting node, use node q as the starting node, and perform the breadth-first algorithm again to find the node s farthest from node q. The path length between node q and node s is used as the length of each tree. diameter All nodes on the diameter are critical nodes, and their corresponding paths are critical paths. The pruning granularity redundancy is updated to the ratio of other nodes except the critical nodes.

[0028] Preferably, after step (3), the method further comprises the following steps:

[0029] Step (4) Input the video clips into the pruned action recognition model and the original action recognition network

[0030] Calculate the action category label and the pruned action recognition model Cross entropy loss between predicted action categories;

[0031] Compute the pruned action recognition model and the original action recognition network The distance loss between the predicted action category probabilities,

[0032] Construct a loss function based on cross entropy loss and distance loss, and optimize the pruned action recognition model based on the loss function

[0033]

[0034] Step (5) Repeat steps (2) to (4) until the pruned action recognition model is convergence.

[0035] Preferably, the distance loss is calculated based on Kullback-Leibler divergence. Preferably, the action category is: the action category of a pedestrian or a driver; and the video is a video of a road condition or a driver's status when a vehicle is driving.

[0036] The present invention aims to solve the problem that fixed pruning granularity cannot adapt to the model network structure that changes dynamically during the pruning process, and has the following advantages: existing work such as DepGraph has attempted to convert the network structure into a graph structure for pruning, explicitly simulating the mutual dependence between paired layers in the neural network, but this method focuses on the pruning strategy of a single granularity or a fixed granularity, lacks the design of a specific model, and pruning is only performed from a hierarchical perspective, with a large pruning granularity. Different from this, step (2) of this solution constructs a graph network (Filter-wise, Layer-wise, channel-wise) for the original model based on different pruning granularities, and uses weights of different granularities as nodes. The granularity is smaller than that of DepGraph, and the similarity of the focus area of the generated feature graph is used as the edge between the weights, dynamically models the action recognition model globally, and adaptively adjusts the pruning strategy to adapt to the model network structure that changes dynamically during the pruning process; at the same time, in this step (2-3), the features of the original action model (such as feature graph or feature vector) are regarded as time series signals, and the features are decomposed into multiple intrinsic mode components (IMF: Intrinsic Mode Decomposition) using variational mode decomposition. Function), each IMF represents a certain level of information in the feature (such as high-frequency details, low-frequency structure, etc.), which is convenient for accurately measuring the similarity of the temporal relationship described by the weights of each granularity, and avoiding the pruning process to destroy the key temporal feature extraction ability of the action recognition model, resulting in the difficulty of the pruned lightweight model to accurately capture the dynamic changes of the action; at the same time, in this step (2-2), a weighted random algorithm based on redundancy is used to adaptively select the pruning granularity according to the redundancy of the network parameters of each granularity, avoiding the phenomenon of excessive pruning in one dimension and redundancy in other dimensions;

[0037] Existing work such as Token Fusion can fuse weights to maintain a higher accuracy of the original action recognition model. However, this method focuses on how to evaluate the importance of a single node and does not decouple the dependencies between nodes during pruning. In addition, the determination of pruning ratios at different granularities depends on manual hyperparameter settings. In contrast, step (3) of this solution constructs a maximum spanning tree through the Kruskal algorithm. While ensuring the overall connectivity of the graph, it prioritizes retaining the edges with the largest weights, thereby retaining the strongest connection relationships in the network and building a stable and efficient backbone structure. It further extracts the diameter of the maximum spanning tree as the key path, accurately identifying the path nodes with the longest propagation and the widest influence in the network, thereby identifying the key nodes and redundant nodes in the graph network, and dynamically adjusting the fusion ratio according to the weight (capacity) of the edges connecting the redundant nodes.

[0038] Almost all existing network pruning works will fine-tune the pruned action recognition model after pruning. However, this fine-tuning often only uses the labels of the video samples to guide the pruned action recognition model. Based on this, step (4) of this method draws on the solutions in the related fields of knowledge distillation, uses the original action recognition model as the teacher model, and the pruned action recognition model as the student model. The patterns or regularities captured by the teacher model on the training data are transferred to the student model, which can significantly improve the performance of the student model.

[0039] This method has been verified on the UCF101 video dataset and the Joint Attention in Autonomous Driving dataset. It can obtain a lightweight action recognition model and complete real-time action recognition tasks on the computing power of a single 2080Ti graphics card. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the main steps in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0041] The method of the present invention performs the following operations in sequence on a given video data set containing category labels:

[0042] Step (1) uniformly sampling the original video of the road condition of the vehicle while driving to obtain an RGB frame image sequence, and randomly sampling video clips as model input, constructing the original action recognition network, and initializing the redundancy of each pruning granularity (such as filter, channel, layer);

[0043] Step (2) constructs a multi-granularity computational graph encoding module, which takes as input a uniformly sampled RGB frame image sequence and the original action recognition network. The pruning granularity of this round is randomly selected based on the redundancy of each pruning granularity. Using variational mode decomposition, the similarity of the ability of each layer to depict the temporal relationship with different parameters at the same granularity is used as the edge of the graph, and the output is the graph structure corresponding to the pruning granularity of this round.

[0044] Step (3) construct an adaptive network flow pruning and fusion module, input the graph structure at this round of granularity, search for key paths through network flow theory for pruning and fusion, and output the pruned action recognition model and the model redundancy at this granularity;

[0045] Step (4) fine-tuning the pruned action recognition model, inputting the pruned action recognition model and the original action recognition model, calculating the cross-entropy loss (Cross-Entropy Loss) between the real action category in the road condition video sample when the vehicle is driving and the action category predicted by the lightweight network, as well as the distance loss between the lightweight action recognition model and the original action recognition model output of the pedestrian or driver action category probability, and optimizing the pruned action recognition model using the stochastic gradient descent method;

[0046] Step (5) repeats steps (2) to (4) until the pruned action recognition model converges, and finally obtains a lightweight action recognition model. The road condition video sample of the new vehicle is uniformly sampled to obtain an RGB frame image sequence, which is input into the compressed lightweight action recognition model to obtain the predicted probability of the roadside pedestrian or driver action category, and the category with the largest probability is selected as the roadside pedestrian or driver action category in the road condition video of the new vehicle.

[0047] Furthermore, step (1) is specifically:

[0048] (1-1) The original vehicle driving road condition video is sampled at a rate of 30 frames per second Perform uniform sampling to obtain a frame stream image sequence Wherein, the subscript I is the abbreviation of Image, n' represents the number of frame stream images, is the field of real numbers, I k represents the kth RGB image in the sequence with a width of w0, a height of h0, and a number of channels of 3. t' is the duration of the original video sample. w, h, and c are the width, height, and number of channels of each frame in the original road condition video when the vehicle is driving, respectively.

[0049] (1-2) Get the frame stream image sequence in (1-1) Uniformly sample n video clips, that is, Among them, the four-dimensional tensor X i' represents the i'th video clip, is the real number domain, t is the duration of the video clip;

[0050] (1-3). The frame stream image sequence obtained in (1-2) A randomly selected video clip Input, after ImageNet dataset pre-training, contains 101 layers of convolution ResNet101 network composed of 33 Bottleneck Blocks, the action recognition network model spatial feature extractor, obtains the original action recognition network spatial encoding features Where w', h', c' are the width, height and number of channels of the spatial encoding feature respectively, and the subscript s is the abbreviation of spatial;

[0051] (1-4). Substitute (1-3) to obtain the spatial encoding feature F of the original action recognition network s After three layers of multi-head self-attention (Multi-Head), the spatiotemporal fusion features of the action recognition model are obtained Where m is the abbreviation of feature fusion (Merge);

[0052] (1-5). Finally, the original action recognition network The fusion feature F m The video action is classified through the fully connected layer and the softmax(·) function to output the video action type C', completing the original action recognition network. Construct, where l∈[1,N] is the index of the model layer number, N is the original action recognition network The total number of layers, according to the different pruning granularity, the weights of each layer are refined as follows Where W, C, and L represent the filter granularity, channel granularity, and layer granularity, respectively. is the total number of filters in layer l, is the total number of channels in the lth layer, and the redundancy of the filter granularity, channel granularity, and layer granularity are R w 、R c 、R l Initialized to 1.

[0053] Furthermore, step (2) is specifically:

[0054] (2-1) The multi-granularity computational graph encoding module mainly consists of a granularity dynamic selection module and a temporal variational graph encoding module. The former is used to determine the pruning granularity of the current round based on the redundancy of each granularity of the action recognition network, while the latter is used to construct the graph structure of the original action recognition model layer by layer and complete the modeling of the relationship between the weights of each granularity.

[0055] (2-2) Construct a dynamic granularity selection module, based on the redundancy R of each pruning granularity in step (1-5) w 、R c 、R l , calculate the selection probability of filter granularity, channel granularity, and layer granularity. The higher the redundancy, the higher the selection probability of the granularity. Random sampling is performed based on the selection probability, and the pruning granularity of this round is output. The filter granularity will be used as an example for discussion later.

[0056] (2-3). Construct a temporal variational graph encoding module, with the input being the original action recognition network M constructed in step (1-5), and the video clip set in step (1-2) Frame stream image sequence A randomly selected video clip Input original action recognition network In the pruning process, the feature map F is pruned according to the selected pruning granularity. l Grouping, original action recognition network The weights of each layer are refined into or in is the total number of filters in the lth layer, is the total number of channels in the lth layer. If the pruning granularity is the filter granularity, the i-th weight output feature map in the lth layer is at this time If the pruning granularity is the channel granularity, then at this time If the pruning granularity is layer granularity, The layer granularity weight output feature map is the original intermediate feature map, at this time i = 1, where t l 、w l 、h l 、c l Represent the time dimension, width, height and number of channels of the feature map of the lth layer respectively. The superscripts W, C and L represent the filter granularity, channel granularity or layer granularity respectively. Get the eigenmode Its dimensions and consistent with its center frequency express The modal component in the time dimension t l Above The index of δ(·) is the unit impulse function, j represents an imaginary number, and ||·||1 represents the l1 norm of the tensor. The constraint of this formula is that the input signal can be reconstructed from the modal components: To solve the optimization problem, the Lagrange multiplier λ and the penalty parameter α∈(0,1) are introduced, and we get:

[0057] Lagrangian function, alternating direction multiplier optimization target, output weight output feature map of the i-th layer The modal center frequency and modal components Its dimensions and consistent;

[0058] (2-4). If the pruning granularity is the filter granularity or the channel granularity, the action recognition network Build the graph layer by layer and transform the first layer action recognition network The refined weight As a node in the graph According to steps (2-3), the center frequency is obtained and modal components Calculate the frequency similarity of adjacent central nodes and modal component similarity If the pruning granularity is the filter granularity If the pruning granularity is the channel granularity Represents two different indexes in layer l, are the bandwidth parameters of the center frequency, which control the decay rate of the similarity, ||·||2 represents the l2 norm of the tensor, and the weight A of the edge in the graph pq A is composed of two weighted components: center frequency similarity and modal component similarity. pq =μ·S ω (p,q)+(1-μ)·S u (p,q), μ∈(0,∞) is a hyperparameter, and the adjacency matrix of the current layer graph is constructed. If the pruning granularity is the filter granularity If the pruning granularity is the channel granularity Its diagonal elements c pp =1, set the threshold τ∈[0,1] for the off-diagonal elements and filter out the weight A pq connections smaller than the threshold, Define the capacity between node p and node q in the graph as Output the adjacency matrix A corresponding to the graph structure at this round of granularity l , where the adjacency matrix A l The distance A between node p and node q pq =c pq If the pruning granularity is layer granularity, since the graph is constructed in each layer in the above way, each graph is only one node, so the action recognition network Overall construction diagram. The construction method of this diagram is exactly the same as the above.

[0059] Furthermore, step (3) is specifically:

[0060] (3-1) The adaptive network flow pruning and fusion module mainly includes a key path dynamic search module and an adaptive pruning and fusion module. The former is used to determine the location of redundant parameters in the action recognition model, and the latter is used to reduce the performance loss of the action recognition model during the pruning process.

[0061] (3-2) Construct a dynamic search module for the key path, and input the adjacency matrix A obtained in step (2-4) l , expect to get the path with the largest capacity between nodes as the key path. First, initialize an empty forest Each node is independently constructed into a tree, and the maximum spanning tree is constructed using the Kruskal algorithm. Specifically, the capacity c between all nodes in the graph is sorted from large to small, and the current capacity c is selected in turn. pq The largest side Join the Forest In the process, if node p and node q do not belong to the same tree, merge the trees corresponding to the two nodes and repeat this step until all nodes are connected by one or more trees (because the generated graph is not necessarily connected);

[0062] (3-3). For each tree By searching the farthest node twice, specifically, select any starting node p, use the breadth-first algorithm to find the node q farthest from the starting node, use node q as the starting node, and use the breadth-first algorithm again to find the node s farthest from node q. The path capacity between node q and node s is diameter All nodes on the diameter are key nodes in is the number of key nodes, the corresponding path is the key path, and the corresponding pruning granularity redundancy is updated where n l is the total number of nodes in the graph;

[0063] (3-4). Build an adaptive pruning fusion module, and input the key nodes obtained in step (3-3) For all other nodes Perform pruning fusion. Specifically, The i-th node in Weight corresponding to the granularity According to the capacity c of the edge, it is proportionally integrated into the corresponding weight of the adjacent node q middle, Where proj(·) is the mapping function from the index of the key node in the graph to the original weight index, express The capacity of the node to the adjacent node q, realizes adaptive pruning fusion, and outputs the pruned action recognition model

[0064] Furthermore, step (4) is specifically:

[0065] (4-1) Fine-tune the pruned action recognition model and convert the frame stream image sequence obtained in step (1-2) A randomly selected video clip Input the pruned action recognition model obtained in step (3-4) The original action recognition network obtained in steps (1-5) In the video sample, calculate the action category label of the pedestrian or driver on the roadside in the form of one-hot encoding The action category probability vector output by the pruned action recognition model The cross entropy loss between and distance loss Among them, C is the number of action categories, ce is the abbreviation of Cross-Entropy Loss, p is the probability value of the output of the correct action category in p'; the relative entropy between the action category probability vector p output by the original action recognition network and the probability vector p' output by the pruned action recognition network is calculated. As the distance loss, where p' i” Represents the logical value of the i-th class of the original action recognition network model, p i” represents the logical value of the i'th class of the pruned action recognition network model, 1≤i'≤C, the subscript kl is the abbreviation of Kullback-Leibler divergence, and log(·) is the logarithmic function with the natural base e as the base;

[0066] (4-2). Calculate the final loss function Among them, η>0 is a hyperparameter used to balance the weight of the loss function. The stochastic gradient descent method is used to optimize the pruned action recognition model to complete the pruned action recognition model. Fine-tuning.

[0067] Furthermore, step (5) is specifically:

[0068] Repeat steps (2)-(4) until the pruned action recognition model converges to obtain a lightweight action recognition model

[0069] Finally, we introduce the lightweight action recognition model The method for performing action recognition includes the following steps:

[0070] Video of road conditions when new vehicles are driving Uniform sampling to obtain RGB frame image sequence and, where the subscript I is the abbreviation of Image, n' represents the number of frame stream images, is the field of real numbers, I k Represents the kth RGB image in the sequence with width w0, height h0, and number of channels 3;

[0071] The RGB frame image sequence obtained by sampling Input the trained lightweight model In the equation, we get the new probability vector of the action category of pedestrians or drivers on the roadside Select the category with the highest probability As a new video , where argmax(·) is used to find the index of the maximum value, that is, the largest category in the action category probability vector is returned as the action category of the roadside pedestrian or driver in the road condition video when the new vehicle is driving.

[0072] The present invention proposes a lightweight method for action recognition models based on hierarchical dynamic fusion network pruning. This method has the following characteristics: 1) Utilizing the granularity dynamic selection module and the temporal variational graph encoding module, according to the redundancy of each granularity of the action recognition network, the graph structure of the original action recognition model is constructed layer by layer, and the relationship between the weights of each granularity is modeled; 2) Utilizing the key path dynamic search module and the adaptive pruning and fusion module, the core features of the key path-retaining network are found, thereby ensuring the accuracy and stability of the pruned model.

[0073] The present invention addresses the problem that fixed pruning granularity cannot adapt to the model network structure that changes dynamically during the pruning process, and has the following advantages: 1) Utilizing a weighted random algorithm based on redundancy, the pruning granularity is adaptively selected according to the redundancy of the network parameters of each current granularity, avoiding the phenomenon of excessive pruning in one dimension and redundancy in other dimensions; 2) The key path is searched layer by layer through the graph structure, and the pruning ratio is dynamically determined, solving the problem of inconsistency between the pruning ratio of each layer and the redundancy of the model parameters in the existing method; 3) The temporal variational mode decomposition technology is used to measure the similarity of the parameters' ability to characterize the temporal dependence of the video, avoiding the destruction of key temporal features in the action recognition network during the pruning process; the present invention can be used in actual scenarios of autonomous driving.

[0074] The present invention will be further described below with reference to the accompanying drawings.

[0075] like Figure 1 This lightweight action recognition model method based on hierarchical dynamic fusion network pruning first uniformly samples the original vehicle road condition video to obtain an RGB frame image sequence. The ResNet101 network model is used to extract video spatial features, and a spatiotemporal feature fusion module is constructed through the Transformer to obtain the original action recognition model and initialize the redundancy of each granularity. A multi-granularity computational graph encoding module is then constructed to obtain the graph corresponding to the current pruning granularity. An adaptive pruning and fusion module is then constructed to obtain the pruned action recognition model. Finally, the pruned action recognition model is fine-tuned until convergence, resulting in a lightweight action recognition model. This method utilizes a dynamic granularity selection module and a temporal variational graph encoding module to construct the graph corresponding to each layer of the action recognition network at the current pruning granularity. The critical path dynamic search module and the adaptive pruning and fusion module are then used to search for redundant parameters at each layer, perform pruning and fusion, and fine-tune the pruned action recognition network to obtain a highly accurate lightweight action recognition model.

[0076] The contents described in this embodiment are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as being limited to the specific forms described in the embodiment. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A lightweight action recognition model method based on hierarchical dynamic fusion network pruning, characterized by: The following steps are involved: Step (1): Convert the video containing action category labels into a number of video clips consisting of frame images; construct an original action recognition network with the video clips as input and action categories as output initialization The redundancy of pruning granularity is different according to the pruning granularity. The weights of each layer in the network are refined separately to obtain the refined weights; Step (2): Select the pruning granularity according to the redundancy; select a video clip input In the above example, we get the output feature map F of the lth layer. l ; l represents Any layer in the feature map F according to the selected pruning granularity l Group and get the feature map of the i-th weight output i represents any weight at the pruning granularity given by the lth layer; Based on the variational mode decomposition method, the optimization objective function is constructed to obtain the feature map The eigenmode of With center frequency If the pruning granularity is the filter granularity or the channel granularity, Construct the graph layer by layer. For the first layer, the refined weights corresponding to the pruning granularity are used as nodes in the graph. With center frequency Get the weight A of the edge in the graph pq , and then get the adjacency matrix A of the current layer graph l ; Step (3): Initialize an empty forest Each node is independently treed to construct the maximum spanning tree; the adjacency matrix A l Sort the capacity c between all nodes in the network from large to small, and select the edge with the largest capacity. Join the Forest until all nodes are connected by trees; for forest Every tree in Search the farthest node twice and get the tree All nodes on the diameter of the node are taken as key nodes; the refined weights corresponding to other nodes except the key nodes are proportionally integrated into the corresponding weights of the nodes adjacent to the other nodes according to the capacity c between all nodes, and the pruned action recognition model is obtained.

2. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to claim 1, characterized in that: In step (1), the conversion specifically includes the following steps: Uniformly sampling the video containing the action category label to obtain a frame stream image sequence in RGB format, and uniformly sampling the frame stream image sequence to obtain a plurality of video clips composed of frame images; described It includes: a spatial feature extractor based on the ResNet101 network, a spatiotemporal feature fusion module based on three-layer multi-head self-attention, a fully connected layer and a softmax(·) function; The original action recognition network The redundancy of the pruning granularity includes the redundancy of the three pruning granularities of filter, channel, and layer. The redundancy values of the three pruning granularities of filter, channel, and layer are equal and between 0 and 1.

3. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to claim 1, characterized in that: In step (2), the pruning granularity is selected according to the redundancy, specifically including: According to the redundancy of the pruning granularity in step (1), the selection probability of the pruning granularity is set according to the redundancy. The higher the redundancy, the higher the selection probability of the granularity. A pruning granularity is randomly selected according to the selection probability.

4. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to claim 1, wherein: In step (2), According to the eigenmode With center frequency Get the weight A of the edge in the graph pq , and then get the adjacency matrix A of the current layer graph l , specifically including: Based on the center frequency of adjacent nodes Calculate the frequency similarity of adjacent nodes; Eigenmodes based on adjacent nodes Calculate modal component similarity; The weight A of the edge in the graph is obtained by weighting the frequency similarity and modal component similarity. pq ; The adjacency matrix A l , whose diagonal elements c pp =1, according to weight A pq Set the threshold of off-diagonal elements to filter out weight A pq The connection is less than the threshold, the adjacency matrix A l The distance A between node p and node q pq =c pq , the capacity between node p and node q in the graph 5. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to claim 1, characterized in that: In step (3), the maximum spanning tree is constructed using the Kruskal algorithm; The step of connecting all nodes by trees further includes: if the node p and the node q do not belong to the same tree, merging the trees corresponding to the two nodes until all nodes are connected by one or more trees.

6. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to claim 1, wherein: In step (3), The search for the farthest node twice yields the tree All nodes on the diameter are regarded as key nodes, which specifically includes the following steps: Select any starting node p, use the breadth-first algorithm to find the node q farthest from the starting node, use node q as the starting node, and perform the breadth-first algorithm again to find the node s farthest from node q. The path length between node q and node s is used as the length of each tree. diameter All nodes on the diameter are critical nodes, and their corresponding paths are critical paths. The pruning granularity redundancy is updated to the ratio of other nodes except the critical nodes.

7. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to claim 1, characterized in that: After step (3), the following steps are also included: Step (4) Input the video clips into the pruned action recognition model and the original action recognition network Calculate the action category label and the pruned action recognition model Cross entropy loss between predicted action categories; Compute the pruned action recognition model and the original action recognition network The distance loss between the predicted action category probabilities, Construct a loss function based on cross entropy loss and distance loss, and optimize the pruned action recognition model based on the loss function Step (5) Repeat steps (2) to (4) until the pruned action recognition model is convergence.

8. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to claim 7, characterized in that: The distance loss is calculated based on the Kullback-Leibler divergence.

9. The method for lightweighting an action recognition model based on hierarchical dynamic fusion network pruning according to any one of claims 1 to 8, characterized in that: The action category is: the action category of pedestrians on the roadside or the driver; the video is a video of the road condition or the driver's status when the vehicle is driving.

Images