Action recognition model lightening method based on hierarchical dynamic fusion network pruning

Abstract

The application discloses a kind of action recognition model lightening methods based on hierarchical dynamic fusion network pruning.First, original video is uniformly sampled to obtain RGB frame sequence, and initial action recognition model is constructed;Multi-granularity computation graph coding module is designed, according to the redundancy of current each granularity network parameter, weighted random selects pruning granularity, utilizes the frequency domain similarity of each granularity parameter output feature map to dynamically describe interlayer parameter relationship, to generate corresponding hierarchical graph structure;Then, using adaptive pruning fusion module searches the key path in graph to dynamically prune redundant parameter nodes;Finally, the pruned action recognition model is fine-tuned and optimized to obtain a light-weight action recognition network.The application can not only evaluate the ability of network parameter description timing dependence, but also adaptively adjust the pruning position, pruning granularity and pruning proportion, improve the inference speed of action recognition model, reduce the model parameter amount, and facilitate the lightweight deployment of action recognition model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, especially the field of visual model compression and action recognition, and relates to a lightweight method for action recognition models based on hierarchical dynamic fusion network pruning. Background Technology

[0002] Action recognition is an important computer vision task that aims to identify single action types, such as "running" and "jumping," in video by analyzing spatial and temporal information in video frames. It has wide applications in the field of autonomous driving. Early methods, whether it was dual-stream networks combining spatial flow (processing single-frame images) and temporal flow (processing optical flow images) or 3D convolutional networks that performed convolutional operations in the spatiotemporal dimension of video to capture spatiotemporal information, resulted in an exponential increase in computational cost when attempting to characterize long-term temporal dependencies and high-resolution video samples. In recent years, with the significant success of deep neural networks based on the Transformer architecture in computer vision, researchers have proposed video vision Transformer models for action recognition tasks. These models utilize self-attention mechanisms to capture long-distance dependencies, significantly improving the model's ability to model temporal relationships. However, deep learning neural networks based on the video Transformer architecture typically have a large number of parameters, which greatly increases computational cost and training difficulty, prolongs inference time, and makes them difficult to deploy on edge and end-device devices with limited computing resources. Therefore, how to achieve both efficient computation and accurate recognition has become a key issue in practical deployment. Lightweighting of video action recognition models has become an urgent need. Without sacrificing accuracy as much as possible, how to reduce the complexity and number of parameters of the network model, improve training efficiency, and reduce inference time so that it can be deployed on small devices with limited computing resources and efficiently complete real-time recognition tasks has important application value.

[0003] Action recognition models aim to assign category labels to videos containing single actions. Early research primarily used Convolutional Neural Networks (CNNs) as the backbone network and extracted spatiotemporal features in two ways: one was to use a two-stream network to extract RGB features in the spatial dimension and optical flow features in the temporal dimension; the other was to use 3D CNNs to directly model spatiotemporal information. To fully combine the advantages of two-stream networks and 3D CNNs, the classic SlowFast network focuses on capturing spatial semantic information in the slow pathway, while the fast pathway is used to extract motion features at a fine temporal resolution. In recent years, researchers have proposed deep neural networks for action recognition based on the Transformer architecture and further improved their performance with pre-trained models on large-scale datasets. The classic work, the Swin Transformer, achieves information interaction between different windows through window shifting and extracts multi-scale spatiotemporal features using a hierarchical structure, significantly improving video action recognition.

[0004] In autonomous driving, action recognition models can improve driving safety and interaction capabilities by analyzing the behavior of pedestrians, drivers, and other traffic participants. For example, recognizing actions such as pedestrians "suddenly running" or cyclists "falling down" can help vehicles avoid obstacles in advance; monitoring states such as drivers "yawning" or "looking down at their phones" can provide timely warnings of fatigue driving. Ensuring the real-time performance of model recognition in these complex real-world scenarios is a key technical challenge for applying action recognition models to autonomous driving.

[0005] Network pruning, a classic method for model compression and lightweighting, aims to reduce the number of network weights and parameters, improve inference efficiency, and reduce model storage requirements, resulting in a lightweight model. This enables fast, real-time inference of learning tasks under resource constraints. Based on the pruning granularity, existing pruning methods can be broadly categorized into structured pruning and unstructured pruning. Structured pruning removes redundant parts of the network at a larger granularity (e.g., filters, channels, or layers). For example, channel pruning compresses the model by deleting entire feature channels, while layer pruning removes entire network layers. These methods typically maintain a regular network structure, are compatible with general-purpose hardware, and do not require specific library support. Unstructured pruning, on the other hand, trims action recognition models at a finer granularity (e.g., individual weights or neurons). For example, weight pruning achieves sparsity by nulling unimportant connections. While this results in significant compression, the resulting lightweight model exhibits an irregular coefficient structure, requiring sparse computation libraries for acceleration.

[0006] The shortcomings of the above methods are mainly reflected in the following aspects: (1) Most existing network pruning methods adopt a fixed pruning granularity strategy, and the pruning ratio depends on manual setting. However, the fixed pruning granularity cannot adapt to the dynamically changing model network structure during the pruning process. During the model pruning process, the network structure changes dynamically with each round of pruning operation. The fixed pruning granularity (such as fixed channel, filter or layer pruning) cannot adapt to this dynamic change, resulting in an unsatisfactory pruning effect; (2) Existing network pruning methods almost assume that the pruning ratio of each layer is the same. However, the parameter redundancy of different layers is not consistent. If each layer of the network is pruned uniformly, the parameters of the key layers will inevitably be over-pruned. Some layers still have a lot of redundant parameters, resulting in a low upper limit of model compression and serious performance loss; (3) Existing pruning methods for video action models have similar parameters that can characterize temporal dependencies. This makes the pruning process destroy the key temporal feature extraction ability of the action recognition model, making it difficult for the lightweight model after pruning to accurately capture dynamic changes in action. Therefore, based on these considerations, there is an urgent need to design an action recognition model pruning method that can fully evaluate the ability of parameters to characterize temporal dependencies and adaptively adjust the pruning position, pruning granularity, and pruning ratio. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a lightweight action recognition model based on hierarchical dynamic fusion network pruning. This method dynamically adjusts the pruning granularity of each round based on the redundancy of each granularity during the pruning process. It also utilizes graph structures to filter redundant network parameters layer by layer for network pruning, combined with network fine-tuning, ultimately achieving a lightweight action recognition model.

[0008] A lightweight action recognition model based on hierarchical dynamic fusion network pruning is proposed, comprising the following steps:

[0009] Step (1): Convert the video containing action category tags into several video segments composed of frame images; construct an original action recognition network with the video segments as input and the action category as output. initialization The redundancy of pruning granularity is determined based on the different pruning granularities. The weights of each layer are refined to obtain the refined weights;

[0010] Step (2): Select the pruning granularity based on the redundancy level; select a video segment as input. In the process, the output feature map F of the l-th layer is obtained. l ;l represents Any layer in the middle; adjust the feature map F according to the selected pruning granularity.l Grouping is performed to obtain the feature map of the i-th weight output. i represents any weight at the given pruning granularity of the l-th layer; an optimization objective function is constructed based on the variational mode decomposition method to obtain the feature map. eigenmodes With center frequency If the pruning granularity is filter granularity or channel granularity, for The graph is constructed layer by layer. For the l-th layer, the refined weights corresponding to the pruning granularity are used as nodes in the graph; based on the intrinsic modes... With center frequency Obtain the weight A of the edge in the graph. pq This leads to the adjacency matrix A of the current layer graph. l ;

[0011] Step (3): Initialize an empty forest Each node forms its own tree, constructing the maximum spanning tree; the adjacency matrix A is... l Sort all nodes and their associated capacities (c) in descending order, and then select the edge with the largest current capacity. Join the Forest In the process, until all nodes are connected by the tree; for a forest Every tree in Search the farthest node twice to obtain the tree. All nodes along the diameter are designated as key nodes; the refined weights corresponding to the other nodes (excluding the key nodes) are proportionally integrated into the weights of the nodes adjacent to these other nodes, based on the capacity c between all nodes, to obtain the pruned action recognition model.

[0012] Preferably, in step (1), the conversion specifically includes the following steps:

[0013] A video containing action category markers is uniformly sampled to obtain a frame stream image sequence in RGB format. The frame stream image sequence is then uniformly sampled to obtain a video segment composed of several frame images.

[0014] The It includes, in sequence: a spatial feature extractor based on ResNet101 network, a spatiotemporal feature fusion module based on three-layer multi-head self-attention, a fully connected layer and a softmax(·) function;

[0015] The The redundancy of the pruning granularity specifically includes the redundancy of the three pruning granularities: filter, channel, and layer. The redundancy values ​​of the three pruning granularities are equal and between 0 and 1.

[0016] Preferably, in step (2), selecting the pruning granularity based on the redundancy level specifically includes:

[0017] Based on the redundancy of the pruning granularity in step (1), the selection probability of the pruning granularity is set according to the redundancy. The higher the redundancy, the higher the selection probability of the granularity. A pruning granularity is randomly selected according to the selection probability.

[0018] Preferably, in step (2),

[0019] The intrinsic mode With center frequency Obtain the weight A of the edge in the graph. pq This leads to the adjacency matrix A of the current layer graph. l Specifically, it includes:

[0020] Based on the center frequency of adjacent nodes Calculate the frequency similarity between adjacent nodes;

[0021] Eigenmodes based on neighboring nodes Calculate modal component similarity;

[0022] The weight A of the edges in the graph is obtained by weighting frequency similarity and modal component similarity. pq ;

[0023] The adjacency matrix A l Its diagonal element c pp =1, according to weight A pq Set a threshold for off-diagonal elements to filter out elements with weight A. pq Connections less than the threshold, adjacency matrix A l The distance A between node p and node q pq =c pq The capacity between node p and node q in the graph

[0024] Preferably, in step (3), the Kruskal algorithm is used to construct the maximum spanning tree; the step of until all nodes are connected by trees also includes: if node p and node q do not belong to the same tree, then merge the trees corresponding to these two nodes until all nodes are connected by one or more trees.

[0025] Preferably, in step (3),

[0026] The tree is obtained by searching for the farthest node twice. All nodes on the diameter are considered critical nodes, specifically including the following steps:

[0027] Choose any starting node p, and use the breadth-first search algorithm to find the node q farthest from the starting node. Using node q as the starting node, repeat the breadth-first search algorithm to find the node s farthest from node q. Use the path length between node q and node s as the length of each tree. diameter All nodes on the diameter are critical nodes, and their corresponding paths are critical paths. The pruning granularity redundancy is then updated to the proportion of all nodes except the critical nodes.

[0028] Preferably, the step (3) is followed by the following steps:

[0029] Step (4) Input the video segments into the pruned action recognition model respectively. and the original action recognition network

[0030] Calculate the action category label and the pruned action recognition model. The cross-entropy loss between the predicted action categories;

[0031] Calculate the action recognition model after pruning and the original action recognition network Distance loss between the predicted action category probabilities

[0032] A loss function is constructed based on cross-entropy loss and distance loss, and the pruned action recognition model is optimized based on the loss function.

[0033]

[0034] Step (5) Repeat steps (2) to (4) until the action recognition model after pruning is established. convergence.

[0035] Preferably, the distance loss is calculated based on Kullback-Leibler divergence. Preferably, the action category is: the action category of a pedestrian or driver on the roadside; the video is a video of road conditions or driver status while the vehicle is in motion.

[0036] This invention addresses the problem that fixed pruning granularity cannot adapt to the dynamically changing model network structure during pruning, and has the following advantages: Existing work such as DepGraph has attempted to transform the network structure into a graph structure for pruning, explicitly simulating the interdependence between paired layers in the neural network. However, this method focuses on pruning strategies with single or fixed granularity, lacks specific model design, and pruning is only performed from the layer perspective, resulting in a large pruning granularity. In contrast, step (2) of this scheme constructs a graph network (Filter-wise, Layer-wise, Channel-wise) for the original model based on different pruning granularities, using weights of different granularities as nodes. The granularity is smaller than that of DepGraph, and the similarity of the regions of interest in the generated feature map is used as the edges between weights. This dynamically models the action recognition model globally and adaptively adjusts the pruning strategy to adapt to the dynamically changing model network structure during pruning. At the same time, in step (2-3), the features of the original action model (such as feature maps or feature vectors) are regarded as time series signals, and variational mode decomposition is used to decompose the features into multiple intrinsic mode components (IMF). Each IMF represents a certain level of information in the features (such as high-frequency details, low-frequency structures, etc.), which facilitates accurate measurement of the similarity of temporal relationships characterized by the weights of each granularity, and avoids the pruning process from destroying the key temporal feature extraction capability of the action recognition model, which would make it difficult for the lightweight model after pruning to accurately capture dynamic changes in actions; at the same time, in this step (2-2), a weighted random algorithm based on redundancy is used to adaptively select the pruning granularity according to the redundancy of the network parameters of each granularity, so as to avoid the phenomenon of over-pruning in one dimension and redundancy in other dimensions;

[0037] Existing work such as Token Fusion can maintain higher accuracy of the original action recognition model by fusing weights. However, this method focuses on how to evaluate the importance of a single node and does not decouple the dependencies between nodes during pruning. Moreover, the determination of the pruning ratio at different granularities depends on manual hyperparameter settings. In contrast, step (3) of this scheme constructs the maximum spanning tree through the Kruskal algorithm. Under the premise of ensuring the overall connectivity of the graph, the edge with the largest weight is retained first, thereby retaining the strongest connection relationship in the network and constructing a stable and efficient backbone structure. Furthermore, the diameter of the maximum spanning tree is extracted as the critical path, accurately identifying the path node that propagates the farthest and has the widest impact in the network, thereby identifying the key nodes and redundant nodes in the graph network. The fusion ratio is dynamically adjusted according to the weight (capacity) of the edge connected to the redundant node.

[0038] Almost all existing network pruning work will fine-tune the pruned action recognition model after pruning. However, this fine-tuning often only uses the labels of video samples to guide the pruned action recognition model. In step (4) of this method, based on this, we draw on the knowledge distillation related field schemes, use the original action recognition model as the teacher model and the pruned action recognition model as the student model. The transfer of the patterns or rules captured by the teacher model on the training data to the student model can significantly improve the performance of the student model.

[0039] This method has been validated on the video dataset UCF101 and the autonomous driving dataset Joint Attention in Autonomous Driving. It can obtain a lightweight action recognition model and complete real-time action recognition tasks on a single 2080Ti graphics card. Attached Figure Description

[0040] Figure 1 This is a flowchart of the main steps in a specific embodiment of the present invention. Detailed Implementation

[0041] The method of this invention performs the following operations sequentially on a given set of video data with category labels:

[0042] Step (1) uniformly sample the original video of the road conditions when the vehicle is driving to obtain an RGB frame image sequence, and randomly sample video segments as model input to construct the original action recognition network and initialize the redundancy of each pruning granularity (such as filter, channel, layer).

[0043] Step (2) Construct a multi-granularity computation graph encoding module. The input is a uniformly sampled RGB frame image sequence and the original action recognition network. The pruning granularity of this round is randomly selected based on the redundancy of each pruning granularity. The variational mode decomposition is used to decompose the similarity of the ability of each layer with the same granularity and different parameters to characterize the temporal relationship as the edge of the graph. The output is the graph structure corresponding to the pruning granularity of this round.

[0044] Step (3) Construct an adaptive network flow pruning and fusion module. Input the graph structure at the current granularity, search for the critical path through network flow theory to perform pruning and fusion, and output the pruned action recognition model and the model redundancy at this granularity.

[0045] Step (4) Fine-tune the pruned action recognition model. Input the pruned action recognition model and the original action recognition model. Calculate the cross-entropy loss between the real action category and the action category predicted by the lightweight network in the road condition video sample when the vehicle is driving, as well as the distance loss between the output probability of the action category of pedestrians or drivers on the roadside by the lightweight action recognition model and the original action recognition model. Optimize the pruned action recognition model using the stochastic gradient descent method.

[0046] Step (5) Repeat steps (2) to (4) until the action recognition model after pruning converges, and finally obtain the lightweight action recognition model. The RGB frame image sequence is obtained by uniformly sampling the road condition video sample when the new vehicle is driving, and the image is input into the compression to complete the lightweight action recognition model to obtain the predicted probability of the action category of the pedestrian or driver on the roadside. The category with the highest probability is selected as the action category of the pedestrian or driver on the roadside in the road condition video when the new vehicle is driving.

[0047] Furthermore, step (1) specifically involves:

[0048] (1-1). The original road condition video of the vehicle in motion was sampled at a rate of 30 frames per second. Uniform sampling is performed to obtain the frame stream image sequence. Where the subscript I is an abbreviation for Image, and n' represents the number of images in the frame stream. For the real number field, I k Let w' represent the k-th RGB image in the sequence with width w0, height h0, and number of channels 3, t' represent the duration of the original video sample, and w, h, and c represent the width, height, and number of channels of each frame in the original video of the road conditions when the vehicle is driving, respectively.

[0049] (1-2). Obtain the frame stream image sequence from (1-1). Uniformly sample n video segments, i.e. Among them, the four-dimensional tensor X i' This represents the i'-th video segment. Let be a real number field, and t be the duration of the video segment;

[0050] (1-3). The frame stream image sequence obtained in (1-2) A randomly selected video segment The input is a ResNet101 action recognition network model spatial feature extractor pre-trained on the ImageNet dataset, consisting of 101 convolutional layers and 33 Bottleneck Blocks, to obtain the original action recognition network spatial encoded features. Where w', h', and c' are the width, height, and number of channels of the spatial coding feature, respectively, and the subscript s is an abbreviation for Spatial.

[0051] (1-4). The spatial coding features F of the original action recognition network obtained from (1-3) are... s The spatiotemporal fusion features of the action recognition model are obtained through three layers of multi-head self-attention. Where m is an abbreviation for feature fusion (Merge);

[0052] (1-5). Finally, the original action recognition network... Fusion feature F m The video actions are classified and output as video action type C' by sequentially passing through fully connected layers and the softmax() function, thus completing the original action recognition network. Construct, where l∈[1,N] is the index of the model layer, and N is the original action recognition network. The total number of layers is further refined by weighting each layer according to the different pruning granularities. Where W, C, and L represent filter granularity, channel granularity, and layer granularity, respectively. This represents the total number of filters in layer l. Let R be the total number of channels in layer l, and R be the redundancy of filter granularity, channel granularity, and layer granularity, respectively. w R c R l Initialize to 1.

[0053] Furthermore, step (2) specifically involves:

[0054] (2-1). The multi-granularity computation graph encoding module mainly consists of a granularity dynamic selection module and a temporal variational graph encoding module. The former is used to determine the pruning granularity of the current round based on the redundancy of each granularity of the action recognition network, while the latter is used to construct the graph structure of the original action recognition model layer by layer and complete the modeling of the relationship between the weights of each granularity.

[0055] (2-2). Construct a dynamic granularity selection module, based on the redundancy R of each pruning granularity in step (1-5). w R c R l Calculate the selection probability of filter granularity, channel granularity, and layer granularity. The higher the redundancy of the granularity, the higher the selection probability. Based on random sampling according to the selection probability, the pruning granularity of this round is output, and the filter granularity will be used as an example for discussion later;

[0056] (2-3). Construct a temporal variational graph encoding module, with the input being the original action recognition network M constructed in step (1-5) and the video clip set obtained in step (1-2). Frame stream image sequence A randomly selected video segment Input raw action recognition network In the process, the feature map F is adjusted according to the selected pruning granularity. l Grouping, original action recognition network The weights of each layer are refined into: or in This represents the total number of filters in the l-th layer. Let be the total number of channels in the l-th layer. If the pruning granularity is the same as the filter granularity, then the output feature map of the i-th weight in the l-th layer is... at this time If the pruning particle size is the channel particle size, then at this time If the pruning grain size is layered, then The layer-level granularity weighted output feature map is the original intermediate feature map, where i = 1, and t l w l h l c l These represent the temporal dimension, width, height, and number of channels of the feature map at layer l, respectively. The superscripts W, C, and L represent the filter granularity, channel granularity, or layer granularity, respectively. The optimization objective... Obtaining intrinsic modes Its dimensions and Consistent with its center frequency express Modal components in the time dimension t l The above is index, δ(·) is the unit impulse function, j represents the imaginary number, and ||·||1 represents the l1 norm of the tensor; the constraint condition of this equation is that the input signal can be reconstructed from the modal components: To solve this optimization problem, we introduce the Lagrange multiplier λ and the penalty term parameter α∈(0,1), and obtain:

[0057] The Lagrangian function, the alternating direction multipliers, and the optimization objective are used to output the i-th weight feature map in the l-th layer. modal center frequency and modal components Its dimensions and Consistent;

[0058] (2-4) If the pruning granularity is at the filter granularity or channel granularity, for action recognition networks The graph is constructed layer by layer, and the action recognition network of layer l is constructed layer by layer. Refined weights As a node in the graph The center frequency is obtained according to steps (2-3). and modal components Calculate the frequency similarity of adjacent center nodes Similarity with modal components Where the pruning granularity is the filter granularity If the pruning particle size is the channel particle size This represents two different indices in layer l. Here, represents the bandwidth parameter of the center frequency, controls the decay rate of similarity, ||·||² represents the l2 norm of the tensor, and A represents the weight of the edge in the graph. pq A is composed of two weighted components: center frequency similarity and modal component similarity. pq =μ·S ω (p,q)+(1-μ)·S u (p,q), μ∈(0,∞) are hyperparameters. Construct the adjacency matrix of the graph at this layer. If the pruning granularity is the same as the filter granularity... If the pruning particle size is the channel particle size Its diagonal element c pp =1, set a threshold τ∈[0,1] for off-diagonal elements, and filter out weight A. pq Connections less than the threshold, Define the capacity between node p and node q in the graph as . Output the adjacency matrix A corresponding to the graph structure at this granularity level. l The adjacency matrix A l The distance A between node p and node q pq =c pq If the pruning granularity is layer-level, since each layer constructs a graph in the above manner, each graph contains only one node. Therefore, for the action recognition network... Overall construction diagram, which is constructed in the same way as the one above.

[0059] Furthermore, step (3) specifically involves:

[0060] (3-1). The adaptive network flow pruning and fusion module mainly includes a critical path dynamic search module and an adaptive pruning and fusion module. The former is used to determine the location of redundant parameters in the action recognition model, and the latter is used to reduce the performance loss of the action recognition model during the pruning process.

[0061] (3-2). Construct a dynamic search module for the critical path, with the adjacency matrix A obtained in step (2-4) as the input. l To obtain the path with the largest capacity between nodes as the critical path, we first initialize an empty forest. Each node forms a separate tree. Kruskal's algorithm is used to construct the maximum spanning tree. Specifically, all nodes in the graph are sorted in descending order of their respective capacities (c), and the current capacity (c) is selected sequentially. pq Largest edge Join the Forest In the process, if node p and node q do not belong to the same tree, then merge the trees corresponding to these two nodes, and repeat this step until all nodes are connected by one or more trees (since the generated graph is not necessarily connected).

[0062] (3-3). For each tree By searching for the farthest node twice, specifically, choosing any starting node p, using the breadth-first search algorithm to find the node q farthest from the starting node, and then using node q as the starting node again, performing the breadth-first search algorithm to find the node s farthest from node q, the path capacity between node q and node s is [value missing] for each tree. diameter All nodes on the diameter are critical nodes. in The number of critical nodes corresponds to the critical path, and the redundancy of the corresponding pruning granularity is updated. Where n l This represents the total number of nodes in the graph;

[0063] (3-4). Construct an adaptive pruning and fusion module, with the key nodes obtained in step (3-3) as the input. For all other nodes To perform pruning and fusion, specifically, to The i-th node Weights corresponding to granularity The weight of the edge is proportionally incorporated into the weight of its adjacent node q based on the edge's capacity c. middle, Where proj(·) is the mapping function that maps the index of the key node in the graph to the original weight index. express The capacity from a node to its neighboring node q is used to achieve adaptive pruning fusion, outputting a pruned action recognition model.

[0064] Furthermore, step (4) specifically involves:

[0065] (4-1). Fine-tune the action recognition model after pruning, and use the frame stream image sequence obtained in step (1-2). A randomly selected video segment Input the pruned action recognition model obtained in step (3-4) Compared with the original action recognition network obtained in steps (1-5) In the process, the one-hot encoded form of video samples is used to classify the actions of pedestrians or drivers on the roadside. The action category probability vector output by the pruned action recognition model Cross-entropy loss between and distance loss Where C is the number of action categories, ce is an abbreviation for Cross-Entropy Loss, and p is the probability value of the correct action class output in p'; it represents the relative entropy between the action class probability vector p output by the original action recognition network and the probability vector p' output by the pruned action recognition network. As the distance loss, where p' i” p represents the logical value of class i in the original action recognition network model. i” Let represent the i”th class logical value of the action recognition network model after pruning, 1≤i”≤C, the subscript kl is an abbreviation for Kullback-Leibler divergence, and log(·) is the logarithmic function with the natural base e;

[0066] (4-2). Calculate the final loss function. Where η>0 is a hyperparameter used to balance the weights of the loss function, the stochastic gradient descent method is used to optimize the pruned action recognition model, thus completing the optimization of the pruned action recognition model. Fine-tuning.

[0067] Furthermore, step (5) specifically involves:

[0068] Repeat steps (2)-(4) until the pruned action recognition model converges, thus obtaining the lightweight action recognition model.

[0069] Finally, we introduce the use of lightweight action recognition models. The method for performing action recognition includes the following steps:

[0070] Video of road conditions while the new vehicle is driving. Obtaining RGB frame image sequences through uniform sampling And, where the subscript I is an abbreviation for image, and n' represents the number of images in the frame stream. For the real number field, I k This represents the k-th RGB image in the sequence with width w0, height h0, and 3 channels;

[0071] The RGB frame image sequence obtained from sampling Input the trained lightweight model In this process, we obtain the probability vector of the action category of the new pedestrian or driver on the roadside. Choose the category with the highest probability. As a new video The action categories are defined by argmax(·), which is used to find the index of the maximum value. That is, it returns the largest category in the action category probability vector as the action category of the pedestrian or driver on the roadside in the road condition video when the new vehicle is driving.

[0072] This invention proposes a lightweight action recognition model based on hierarchical dynamic fusion network pruning. This method has the following characteristics: 1) Using a granularity dynamic selection module and a temporal variational graph encoding module, the graph structure of the original action recognition model is constructed layer by layer according to the redundancy of each granularity of the action recognition network, and the relationship between the weights of each granularity is modeled; 2) Using a critical path dynamic search module and an adaptive pruning fusion module, the core features of the critical path retention network are found, ensuring the accuracy and stability of the pruned model.

[0073] This invention addresses the problem that fixed pruning granularity cannot adapt to dynamically changing model network structures during pruning, and has the following advantages: 1) It utilizes a redundancy-based weighted random algorithm to adaptively select the pruning granularity according to the redundancy of network parameters at each granularity, avoiding over-pruning in one dimension while redundancy exists in other dimensions; 2) It dynamically determines the pruning ratio by searching the critical path layer by layer through a graph structure, solving the problem of inconsistency between the pruning ratio of each layer and the redundancy of model parameters in existing methods; 3) It uses temporal variational pattern decomposition technology to measure the similarity of the parameters' ability to characterize the temporal dependencies of the video, avoiding the destruction of key temporal features in the action recognition network during pruning; This invention can be applied to real-world autonomous driving scenarios.

[0074] The present invention will be further described below with reference to the accompanying drawings.

[0075] like Figure 1 A lightweight action recognition model based on hierarchical dynamic fusion network pruning is proposed. First, RGB frame image sequences are obtained by uniformly sampling the original road condition video during vehicle movement. The ResNet101 network model is used to extract video spatial features, and a spatiotemporal feature fusion module is constructed using Transformer to obtain the original action recognition model, initializing the redundancy at each granularity. Then, a multi-granularity computational graph encoding module is constructed to obtain the graph corresponding to the current pruning granularity. An adaptive pruning fusion module is constructed to obtain the pruned action recognition model. Finally, the pruned action recognition model is fine-tuned until it converges, resulting in a lightweight action recognition model. This method utilizes a granularity dynamic selection module and a temporal variational graph encoding module to construct the graph corresponding to each layer of the action recognition network at the current pruning granularity. A key path dynamic search module and an adaptive pruning fusion module are used to search for redundant parameters at each layer, perform pruning and fusion, and fine-tune the pruned action recognition network to obtain a high-accuracy lightweight action recognition model.

[0076] The content described in this embodiment is merely an enumeration of the implementation forms of the inventive concept. The scope of protection of this invention should not be regarded as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A lightweight method for action recognition models based on hierarchical dynamic fusion network pruning, characterized in that, Includes the following steps: Step (1): Convert the video containing action category tags into several video segments composed of frame images; construct an original action recognition network with the video segments as input and the action category as output. initialization The redundancy of pruning granularity is determined based on the different pruning granularities. The weights of each layer are refined to obtain the refined weights; Step (2): Select the pruning granularity based on the redundancy level; select a video segment as input. In the process, the output feature map F of the l-th layer is obtained. l ;l represents Any layer in the middle; adjust the feature map F according to the selected pruning granularity. l Grouping is performed to obtain the feature map of the i-th weight output. i represents any weight at the given pruning granularity of the l-th layer; The objective function is constructed based on the variational mode decomposition method, and the feature map is obtained. eigenmodes With center frequency If the pruning granularity is filter granularity or channel granularity, for The graph is constructed layer by layer. For the l-th layer, the refined weights corresponding to the pruning granularity are used as nodes in the graph; based on the intrinsic modes... With center frequency Obtain the weight A of the edge in the graph. pq This leads to the adjacency matrix A of the current layer graph. l ; Step (3): Initialize an empty forest Each node forms its own tree, constructing the maximum spanning tree; the adjacency matrix A is... l Sort all nodes and their associated capacities (c) in descending order, and then select the edge with the largest current capacity. Join the Forest In the process, until all nodes are connected by the tree; for a forest Every tree in Search the farthest node twice to obtain the tree. All nodes along the diameter are designated as key nodes; the refined weights corresponding to the other nodes (excluding the key nodes) are proportionally integrated into the weights of the nodes adjacent to these other nodes, based on the capacity c between all nodes, to obtain the pruned action recognition model.

2. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in claim 1, characterized in that, In step (1), the transformation specifically includes the following steps: A video containing action category markers is uniformly sampled to obtain a frame stream image sequence in RGB format. The frame stream image sequence is then uniformly sampled to obtain a video segment composed of several frame images. The It includes, in sequence: a spatial feature extractor based on ResNet101 network, a spatiotemporal feature fusion module based on three-layer multi-head self-attention, a fully connected layer and a softmax(·) function; The original action recognition network The redundancy of the pruning granularity specifically includes the redundancy of the three pruning granularities: filter, channel, and layer. The redundancy values ​​of the three pruning granularities are equal and between 0 and 1.

3. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in claim 1, characterized in that, In step (2), selecting the pruning granularity based on the redundancy level specifically includes: Based on the redundancy of the pruning granularity in step (1), the selection probability of the pruning granularity is set according to the redundancy. The higher the redundancy, the higher the selection probability of the granularity. A pruning granularity is randomly selected according to the selection probability.

4. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in claim 1, characterized in that, In step (2), The intrinsic mode With center frequency Obtain the weight A of the edge in the graph. pq This leads to the adjacency matrix A of the current layer graph. l Specifically, it includes: Based on the center frequency of adjacent nodes Calculate the frequency similarity between adjacent nodes; Eigenmodes based on neighboring nodes Calculate modal component similarity; The weight A of the edges in the graph is obtained by weighting frequency similarity and modal component similarity. pq ; The adjacency matrix A l Its diagonal element c pp =1, according to weight A pq Set a threshold for off-diagonal elements to filter out elements with weight A. pq Connections less than the threshold, adjacency matrix A l The distance A between node p and node q pq =c pq The capacity between node p and node q in the graph 5. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in claim 1, characterized in that, In step (3), the Kruskal algorithm is used to construct the maximum spanning tree; The phrase "until all nodes are connected by a tree" further includes: if node p and node q do not belong to the same tree, then merge the trees corresponding to these two nodes until all nodes are connected by one or more trees.

6. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in claim 1, characterized in that, In step (3), The tree is obtained by searching for the farthest node twice. All nodes on the diameter are considered critical nodes, specifically including the following steps: Choose any starting node p, and use the breadth-first search algorithm to find the node q farthest from the starting node. Using node q as the starting node, repeat the breadth-first search algorithm to find the node s farthest from node q. Use the path length between node q and node s as the length of each tree. diameter All nodes on the diameter are critical nodes, and their corresponding paths are critical paths. The pruning granularity redundancy is then updated to the proportion of all nodes except the critical nodes.

7. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in claim 1, characterized in that, The following steps are included after step (3): Step (4) Input the video segments into the pruned action recognition model respectively. and the original action recognition network Calculate the action category label and the pruned action recognition model. The cross-entropy loss between the predicted action categories; Calculate the action recognition model after pruning and the original action recognition network Distance loss between the predicted action category probabilities A loss function is constructed based on cross-entropy loss and distance loss, and the pruned action recognition model is optimized based on the loss function. Step (5) Repeat steps (2) to (4) until the action recognition model after pruning is established. convergence.

8. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in claim 7, characterized in that, The distance loss is calculated based on the Kullback-Leibler divergence.

9. The lightweight method for action recognition model based on hierarchical dynamic fusion network pruning as described in any one of claims 1 to 8, characterized in that, The action category is: the action category of pedestrians or drivers on the roadside; the video is the video of road conditions or driver status when the vehicle is in motion.

Images