A dynamic behavior recognition system based on key editing discrimination

Abstract

The present invention discloses a dynamic behavior recognition system based on key editing discrimination. The system achieves performance comparable to the latest performance in the field of efficient behavior recognition and obtains a 3D (video-based) method with the lowest computational consumption in the field of efficient behavior recognition. In the efficient behavior recognition scenario, by expanding the 3DRNNPooling technology, a multi-model framework based on the mutual distillation of sub-networks of different scales in the video data scenario is successfully built, that is, by setting multiple models of different scales for online mutual knowledge distillation, effective effect improvement is obtained. A key editing learning method based on a double-layer gating mechanism is applied, and the model is supervised to learn to distinguish important clips by further scoring the criticality of the clips in the entire video through the double-layer gating unit. The final model only needs to infer the most important clip action sequence to obtain the inference result of the entire video, which saves a great deal of computational effort.

Description

Technical Field

[0001] The present invention relates to the technical field of video behavior recognition and efficient behavior recognition integrating static and dynamic methods in a model compression method, in particular to a dynamic behavior recognition system based on key editing discrimination. Background Art

[0002] Specifically, behavior recognition technology requires the establishment of a model that can classify the behaviors of people in a video input by the user, and give a relatively accurate distribution of action behavior prediction results. The category with the highest probability in the prediction distribution is the model's inferred category, which is used as the prediction result of this segment of input data. For example, in the large-scale human dynamic behavior recognition dataset Kinetics, the behavior categories include various ball sports, crying, making cakes, making tea, making beds, and other human dynamic behaviors. Finally, the most likely prediction category is output. Behavior recognition can be regarded as a mirror problem in the video field similar to image classification.

[0003] Since static inference unifies the data processing flow that never changes, it is not friendly to data at both ends of the data complexity. For example, for simpler data, static inference will uniformly assign too much computational complexity to simpler and more obvious data, resulting in a waste of resources. For more complex data, this type of model cannot give a more considerable confidence at best, making the final prediction distribution less reliable. The inference result is unreliable.

[0004] Static inference has a negative impact on the compatibility and adaptability of the model due to the unified static data processing process. Deleting neurons that are rarely activated during the training inference process may have a destructive effect on most samples of average difficulty, resulting in a decrease in model performance. At the same time, the model cannot be uniformly deployed on multiple target devices and edge devices, and multiple different tasks cannot be executed according to a unified computing strategy, which reduces the versatility of the model.

[0005] Therefore, relatively more flexible dynamic methods can alleviate the above two problems. For example, for two different video samples, such as doing yoga and Tai Chi, the computing resources or convolutional features required to distinguish their categories may be different, but for static methods, all data with inference difficulty are often divided into a static and unchanging inference route, resulting in a waste of resources. The dynamic method can assign shallow neural networks to samples that are easy to infer, and match deeper neural networks to data that is difficult to distinguish, thereby reducing resource waste and improving the network's adaptability to data. Summary of the invention

[0006] The purpose of the present invention is to propose a dynamic behavior recognition system based on key clip discrimination in response to the defects of the prior art. The Gumbel-Softmax mechanism is introduced into the system to design another branch selection network to further reduce the computational overhead. Finally, the key segment efficient dynamic network integrating three modules obtains a performance that is still relatively competitive compared with the sota model in the field of efficient behavior recognition. And it significantly reduces the lower limit of the computational overhead in 3D methods in this field.

[0007] The specific technical scheme for achieving the purpose of the present invention is as follows:

[0008] A dynamic behavior recognition system based on key clip identification is characterized in that the system includes a key clip identification module, a branch strategy module and a branch network module, and the three modules are connected to each other; data is input from the key clip module, and the key clip is output to the branch strategy module after extracting the key clip. The branch strategy module matches the key clip to different sub-branch networks in the subsequent branch network module, and dynamically matches the key clip to sub-networks of different sizes and different computational amounts through the GumbelSoftmax algorithm, thereby saving and optimizing the computational amount, and greatly reducing the number of floating point operations (FLOPs) consumed by the entire system. Among them:

[0009] The key clip discrimination module consists of a neural network f g and two MLP gating units. After receiving the input data, it passes through the neural network f g After extracting the features, the features are calculated through two gating units to obtain the criticality ranking of the key clips; the k most important key clips in the criticality ranking (the ones with the highest criticality ranking) are selected, and the parameters are updated through the supervised loss during training;

[0010] The branch strategy module is composed of a neural network π, which receives the key clips transmitted by the key clip identification module. The neural network π extracts the key clip data features and then uses the GumbelSoftmax algorithm to compare each key clip with the branch network module f l The sub-networks of different depths are matched, and the parameters are updated through supervision loss and computational (FLOPs) constraint loss during training;

[0011] The branch network module f l, consists of 4 sub-neural networks of different depths, including the original network (most behavior recognition networks can be used as the original network) and three modified sub-networks of the original network. The three modified sub-networks use the extended 3DRNNPooling technology to replace different computing units in the original network to achieve sub-networks of different depths; each key clip is matched by the branch sub-network to perform behavior recognition calculations to obtain behavior recognition results; parameters are updated through supervision loss and distillation loss during training.

[0012] The neural network f g After extracting the features, the features are calculated through two gating units to obtain the criticality ranking of the key clips, and the most important (most critical) k key clips in the criticality ranking are selected, including:

[0013] Step 2.1: For the input action recognition video v containing hundreds of frames t , divided into multiple clips in units of 16 frames. If the last clip has less than 16 frames, return to the first frame of the clip to fill in the copy frame order until 16 frames are met; use {c1,c2…,c p} represents all clips of the video, the number of clips is p; each clip is passed through the neural network f g , extract clipping features; define the g After feature extraction, it is called local feature

[0014] Step 2.2: Input the local features into the first gating unit ψ α Get the critical weight of the clip information:

[0015] Step 2.3: Take α i The weighted sum of local features is used as weight to obtain the global features of the entire video

[0016] Step 2.4: Concatenate the local features and global features to obtain And use the second gating unit ψ β Extract the concatenated features to obtain the purified critical weight β i ;

[0017] Step 2.5: β of all the clips obtained iSort from large to small, take the top k β (topk) in the sorting, and take the clips corresponding to their serial numbers as the k most critical clips of this video; for example, if β3 is the first element after sorting, it means that its third clip c3 is the most important (critical), then the corresponding third clip c3 is the most critical (most important) clip; the same applies to the remaining k-1 key clips. k is used as a hyperparameter of this module.

[0018] The branch strategy module mainly uses the neural network π to perform sub-network matching on the output results of the key editing discrimination module. Assuming that the number of sub-networks of the branch network module is B=4, each key editing is matched to the sub-network in the subsequent module branch network module through the branch strategy module. After the neural network π extracts the key editing data features, the GumbelSoftmax algorithm is used to match the key editing with the branch network module f l The different branch sub-networks in the input are matched, and each key clip has the following steps for the k key clips, including:

[0019] Step 3.1: Extract features from neural network π; obtain the probability distribution q of the output of neural network π t , t∈[1,B];

[0020] Step 3.2: By using the Gumbel_Softmax algorithm to calculate the output probability distribution q of the neural network π t Calculate the one-hot distribution using V i , i∈[1,B] to indicate that one unit of this distribution (V) is 1 and the others are all 0;

[0021] Step 3.3: The sub-network aligned with the maximum value and the only 1 in the one-hot distribution V obtained for the key clip is the matching sub-network of the current key clip in the branch network module.

[0022] Branch network module l In the example, each key clip is subjected to behavior recognition calculation by the matched branch sub-network to obtain a behavior recognition result, which is specifically:

[0023] Step 4.1: perform the behavior recognition calculation of the key clip in the branch sub-network matched by the key clip, and record the behavior recognition result;

[0024] Step 4.2 obtains k behavior recognition results from different branch sub-neural networks in the branch network module that matches the k key clips, and averages the k behavior recognition results as the final behavior recognition result of the entire video, which is the behavior recognition result of the entire video.

[0025] The training described in the key clip identification module is performed through supervised loss to update parameters, including:

[0026] Step 5.1: The weights α learned for the two gating units i β i Perform weighted summation to obtain the final integrated features of the entire video

[0027] Step 5.2: The supervision method is cross entropy loss. Through cross entropy loss, the neural network parameters in the module change in the direction of retaining the key features of the category: for a total number of samples N, the current category is j, and the label of the sample number i is y ij , the integrated feature f′ of the current sample in the jth category finally obtained by the two gating units v , for clarity, we use f′ ij Replace f′ v , use the following formula as the cross entropy supervision loss:

[0028]

[0029] The training described in the branch strategy module is performed by updating the parameters through the supervision loss and the computational (FLOPs) constraint loss, specifically including: Step 6.1: N is the total number of samples, the number of behavior recognition categories is set to C, Vo is the one-hot distribution under the o-th branch; in order to optimize the sub-network of the branch network module that matches the key clip to the correct classification as much as possible, set the key clip x i Pass in the subsequent matched branch network module f l The probability distribution of the sub-network in the key clip is calculated as h i , the corresponding label of category j of sample i is y ij For the category of the current key clip sample, there are cross entropy loss functions of i∈[1,N], j∈[1,C], o∈[1,4]:

[0030]

[0031] Step 6.2: Use computational constraint loss supervision for the branch strategy module: Set the number of sub-branches to B = 4, the current branch to o, Vo to be the one-hot distribution under the o branch, and the number of floating-point operations (FLOPs) for the o-th branch to FLOPs(o); all have loss functions:

[0032] Step 6.3: The branch strategy module can be trained by calculating the gradient descent of the above two loss functions: λ is the balancing term of the two loss functions.

[0033] Branch network module l The extended 3DRNNPooling technology described in is the extension of RNNPooling technology to 3DRNNPooling. The specific steps are as follows:

[0034] Step 7.1: For a 3D data feature map with an input size of I*C*F*H*W, i.e., batch number*channel number*frame number*length*width, slice the entire data according to the pre-set hyperparameters such as stride and kernel size of the 3DRNNPooling layer to obtain slice units of multiple consecutive frames, and the number of frames can be customized;

[0035] Step 7.2: Perform RNNPooling operation on each frame of the slice unit;

[0036] Step 7.3: Perform a third-dimensional RNNPooling operation on the RNNPooling result obtained for each frame in the time dimension to obtain the output.

[0037] The sub-networks in the branch network module can use the 3DRNNPooling technology to transform the original network (most of the behavior recognition neural networks) into multiple neural networks of different depths, especially the three modified sub-networks of the original network, specifically:

[0038] Step 8.1: Perform 3DRNNPooling modification on the original network (resnet18 or other popular behavior recognition neural networks) to obtain the following three sub-networks, so that the original large number of convolution blocks that consume computing power in the model are replaced by RNN (recurrent neural network) to reduce FLOPs consumption.

[0039] For the 2x downsampling subnetwork: 3DRNNPooling-2x replaces the original maxpool3D and the first conv3D computing unit and saves the related computing consumption. The replaced module downsamples the input data by two times; a 3DRNNPooling module also downsamples the input data by two times;

[0040] For the 4x downsampling subnetwork: 3DRNNPooling-4x replaces the original maxpool3D and the first two conv3D computing units and saves the related computing consumption. The replaced module downsamples the input data by four times; a 3DRNNPooling module also downsamples the input data by four times;

[0041] For the 8x downsampling subnetwork: 3DRNNPooling-8x replaces the original maxpool3D and the first three conv3D computing units and saves the related computing consumption. The replaced module downsamples the input data eight times; a 3DRNNPooling module also enables the input data to be downsampled eight times.

[0042] The above three modified sub-neural networks using 3DRNNPooling technology and the original neural network form four sub-neural networks with different depths, forming the branch network module f l .

[0043] For branch network module f l The training method of the four sub-neural networks is as follows:

[0044] Step 9.1: For a batch of input data, which contains N video clips; for the jth output unit of the ith sample, the output of each subnetwork classifier, i.e., the predicted distribution, is set to Its softened form is It is the output of the fc layer and is softened by the hyperparameter T; the feature data g of the jth unit under the i-th sample ij It is the input data of the neural network Fc layer;

[0045] Step 9.2: Correlation calculation: The feature data between sub-network a and sub-network b are calculated using g a g b Indicates that the correlation is defined as α ab :

[0046]

[0047] The projection matrices K and Q are used to determine the relevance of different branch sub-networks by self-attention to high-level features;

[0048] Step 9.3: For multiple sub-networks of different depths, based on the self-attention ratio α ab The process of transferring knowledge to other branches is to distill the features t of the teacher of the a-th sub-network. a Defined as

[0049]

[0050] It is the soft logarithm of the predicted distribution of the enumerated sub-network (see 9.1); in order to perform distillation learning between the sub-networks of the branch network module, the distillation loss calculation process for this module is defined as follows:

[0051]

[0052] ti represents the teacher distilled features of the i-th sub-network, q′ i represents the soft logarithm of the prediction distribution of the sub-network (see 9.1); through the two formulas in this step, by reducing t i and q′ i The gap between them is used for distillation;

[0053] Step 9.4: Some high-level semantic features learned from other sub-networks of different complexity can be transferred to other sub-networks through the above steps; at the same time, each sub-network is guided by the cross entropy objective y represents the label, q i represents the sample prediction distribution under the i-th branch sub-network (unsoftened, see 9.1); the cross entropy target is Indicates, i represents the sub-network number of the branch network module; finally, the loss function is calculated for all sub-networks of different depths (scales) by the following formula;

[0054]

[0055] T 2 To balance the terms, a multi-scale (depth) mutual distillation is performed in this way. Experimental verification shows that this module distillation algorithm can effectively reduce the improvement of generalization effect caused by overfitting.

[0056] Beneficial effects of the present invention:

[0057]

[0058] From the data in the above table, we can see that: by comparing the performance with the latest models today, we can find that the present invention, as one of the few 3D models, has obtained indicators that are still more competitive than most other 2D models. In the table, MN represents mobilenet, RN represents resnet, and SN represents shufflenet. Since most 2D networks (image-based inference models) consume significantly less computational power than 3D networks (clip-based inference models) in the field of behavior recognition, and the latter's prediction performance is better than the former as a whole, the best (sota) methods in the field of efficient behavior recognition in recent years are mostly applications of 2D models. It can also be seen from the table that most of the sota models in recent years are based on 2D baselines, such as image-based FrameExit, which consumes 7.8G per video average inference FLOPs and achieves the best performance of 75.3%. Therefore, from the above chart, we can see that KCEDNet (SN2+X3D-M) has achieved the best performance among 3D methods, and its consumption of FLOPs is also lower than that of most of the best methods. This system is equivalent to optimizing the lower limit of algorithm computational consumption for clip-based models in the field of EfficientInference, which effectively proves that 3D models can be compared with 2D models from the perspective of inferring important data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Calculate the flow chart for the complete module of the system;

[0060] Figure 2 Training flow chart for branch network module;

[0061] Figure 3 Training flowchart for the key clip discrimination module;

[0062] Figure 4 This is a complete data inference (calculation) diagram for this system. DETAILED DESCRIPTION

[0063] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0064] See also Figure 4 The present invention includes three modules, namely two lightweight network modules, namely a branch strategy module π and a key editing discrimination module fg and a branch network module f lThe network in the branch network module consists of four sub-networks, one original network, three original networks and a 3DRNNPooling replacement sub-network. A set of clip data is input and passed through the key clip identification module. After the key clip is identified, a large number of redundant calculations of non-key clips are reduced; the key clip is sent to the subsequent branch strategy module: through calculation, the probability distribution is transformed based on the Gumbel-Sotmax algorithm to obtain a one-hot distribution, and the subsequent branch network sub-network with a match of 1 is the matching network of the key clip, so that all key clips do not need to be inferred according to the same amount of calculation. Finally, the branch network module modified by the matching 3DRNNPooling technology is used for behavior recognition to obtain more reliable results.

[0065] Figure 4 The parameter details of each sub-network of the branch network implemented by 3DRNNPooling technology, taking resnet18 as an example, are shown.

[0066] The present invention provides an efficient behavior recognition process based on key clip importance scoring, which specifically includes:

[0067] Step 1 Expand the multi-scale mutual distillation behavior recognition group network of RNNPooling3D

[0068] 1.1) Using RNNPooling3D for efficient video scene recognition

[0069] For general RNNPooling3D, the input is a 3D data feature map of size I*C*F*H*W. Similar to RNNPooling2D, the entire volume is sliced ​​according to the stride and convolution kernel size of the pre-set RNNPooling3D layer. The input 3D features are sliced ​​according to a specific kernel size and stride size to keep the shape change the same as the original backbone parameters, and the input feature size is downsampled by different multiples. The similar operation of RNNPooling in each slice is to take the feature extraction result of the entire individual frame as an element of an output feature map, and the set of all sequence frames is input into the final RNN3 for feature fusion of temporal information. After this, an additional RNN3 is applied to traverse the consecutive frames fi to output δ,i∈[1,n] as the only last unit in the following formula.

[0070] o i ←RNN3(δ),δ∈[1,n] (1)

[0071] In order to aggregate sequence information on the time axis, a unidirectional RNN3 is applied, that is, the time dimension of the continuous aggregated features δ of several frames is aggregated. RNN3 is used here for unidirectional processing because the performance of RNN3 after bidirectional feature aggregation is worse. Finally, the entire input feature is jointly pooled by the RNNPooling3D layer. The intermediate feature size of the coupled model to which the RNNPooling3D layer is applied also remains unchanged as the original model tensor changes. Therefore, the front-end layer of the 3D CNN is replaced with the RNNPooling3D layer to reduce FLOPs. Compared with the basic model, the rest of the layers are the same. From Figure 4 It can be seen in the branch network module that the large amount of computation concentrated in the front convolutional layer is replaced by a lightweight RNNPooling3D module to accelerate the down sampling of intermediate features.

[0072] 1.2) Knowledge Distillation between Online Branch Networks at Multiple Computational Scales

[0073] To further improve the versatility of the compression model, a multi-branch complexity-boosting network architecture for mutual learning is proposed. The online distillation architecture deploys a set of student networks to learn from each other, which has better performance than the traditional distillation setting, which is a pre-trained bulky teacher model and the student model simulates the predicted output of the teacher model. Online distillation is an end-to-end training setting, which is easy to deploy without pre-training. In general, the softened distribution q of the probability distribution of each classifier ij , which is the output of the fc layer and is softened by the hyperparameter T, as shown in formula (2). However, a similar approach is to use high-level features g before obtaining the fc layer. ij . These are called “soft features”.

[0074]

[0075] In order to collect knowledge from other branches and avoid learning unimportant features of other branches, the self-attention module is used to calculate the correlation of high-level features g between B branches according to the correlation of the following formula (B is used to represent the number of branches). The benefit of self-attention is that it avoids the loss of diversity to a certain extent.

[0076]

[0077] Each student model transfers knowledge to other branches based on self-attention association learning and aggregates together as shown in Equation (4). Therefore, the overall distillation by KL divergence is summarized in Equation (5).

[0078]

[0079]

[0080] Therefore, in order to let each branch learn from other branches and transfer knowledge better, four branches are deployed to transfer knowledge from different complexities of the RNNPooling3D network.

[0081]

[0082] 2.1) Efficient Inference Framework for Key Clips Based on Double-layer Gating

[0083] The main inspiration of the framework is to try to detect decisive action segments in the entire video to save computation. This module first divides the input video vi into multiple clips evenly, and then passes a lightweight network f g The features of multiple clips are obtained and then the features are gated by the gate module to extract the importance of clip information. The input video vi is uniformly divided into p clip data {c1, c2…, c p There is no frame overlap in the action recognition scenario. A gating mechanism is used to learn the correlation of action recognition actions. For the input video segment ci, a lightweight network f g To extract the features of all clips.

[0084] α i =ψ α (f g (c i )),i∈[1,p] (7)

[0085] Let ψ represent a gate module that measures the importance of clip ci. Therefore, the entire video feature can be easily weighted by formula (8) as the overall attention feature of the entire video after importance extraction. However, the attention score obtained at this stage is not robust enough. In order to obtain a more robust importance judgment, try to introduce more features. The features of the local clip are spliced ​​with the features of the global clip as new features for learning importance. Then, the clip data segments that are more discriminative of the more important semantic information as a whole will be strengthened in the process of neural network optimization. Therefore, the double feature size is thrown into another gating unit again to participate in importance learning. In the following formula, the overall feature f aggregated by the gate mechanism global (v t )as follows:

[0086]

[0087] In order to measure the resolution of a fragment, the splicing of local and global features provides a comparative learning process (Formula 9), and the gate ψβ will learn to distinguish more discriminative splicing data during the optimization process.

[0088] ψ β Learning weighted connections It measures Compared with the entire video in equation (10) of relevance.

[0089]

[0090]

[0091] Use multi-classification cross entropy loss to supervise the key clips learning network. For a total number of samples N, the current category is j, and the label of the sample number i is y ij , the feature of the current sample in the jth category finally obtained through the two-level attention gating mechanism is f′ ij To conduct supervision:

[0092]

[0093] The above formula makes the network change in the direction of reducing the classification loss so that the attention gating mechanism in the model can learn to extract more representative features. In order to make the loss of the salient segments smaller, formula 11 calculates the overall features of the entire video. Finally, the overall features enhanced by the secondary attention gating are supervised by cross entropy as the classification loss of this module.

[0094] 3.1) Dynamic inference through Gumbel Softmax technology

[0095] Another lightweight policy network π is used as the policy network for this task to judge the complexity. Assuming that the distribution of the final output of the policy network is to let the network output the selection of the sub-network, the size of the final output must be aligned with the number of sub-networks. However, selection is a non-differentiable problem and it is difficult to optimize using gradient descent. In order to facilitate training, Gumbel Softmax technology is considered here. For a model with k sub-networks, the probability distribution pt,t∈[1,k] of the output of network π is as follows:

[0096]

[0097] In order to use this probability distribution qt as the basis for branch decision making, the final choice should be

[0098] x π = arg max(log(q i )) (13)

[0099] In order to make it differentiable and retain a certain degree of prediction randomness, GumbelSoftmax is used to add a Gumbel distribution Gi to this distribution. The Gumbel distribution function is F(x) = eex, so the problem becomes:

[0100] x π = arg max(log(q i )+G i ) (14)

[0101] Therefore, since each output probability qi must be added to Gi, and Gi is randomly sampled within the distribution function, the entire Softmax becomes random, thus achieving the purpose of randomizing Softmax. The probability distribution qt becomes:

[0102]

[0103] In order to make the prediction distribution of each network similar at the beginning of training, τ is used in the above formula to smooth the distribution gap as much as possible. As the training iterations gradually deepen, τ is continuously reduced, which eventually makes the entire goal more and more sensitive, and finally matches the corresponding computing resources as much as possible.

[0104]

[0105] For the policy network, the cross entropy loss function and the computational consumption loss are used for supervision. The policy network should first be guided by the accuracy rate when allocating data to which branch for calculation. That is, it should first consider how to make the network correctly classify the data, and then allocate the branch with the smallest amount of calculation as much as possible under this premise. Therefore, the prediction distribution q is defined t ′ is used to represent the output distribution of the policy network, that is, formula (15), which directly distributes q t The maximum value of the predicted probability in ′ As the selection of this data. Therefore, assuming that the target distribution is as shown in formula (15), the FLOPs computational consumption of the i branch is set to FLOPs(^q). In order to optimize the network towards smaller branches, for each input data, the following regularization term is added as the loss function of the constraint computation amount]:

[0106]

[0107] B is the number of branches in the network, and Vi is the prediction probability of the i-th branch according to the Gumbel Softmax technique in the previous section. Therefore, by gradient descent on LF LOPs, the network can be optimized in the direction of reducing FLOPs. In order to optimize the model in the direction of more likely correct classification, the prediction distribution p obtained for each branch isi , both have:

[0108]

[0109] Example 1

[0110] See also Figure 1 The present invention obtains the final calculation prediction through the collaborative inference of the three sub-models. The specific operation steps are as follows:

[0111] S100: the input video vi is evenly divided into multiple clips, and all of them are input into the key clip identification module;

[0112] S110: The fg model is used to determine the key clips, that is, the importance of the image.

[0113] S130: discard non-critical clips, and retain the topk clips with the highest critical weights to pass to the next module;

[0114] S140: The branch strategy module calculates (infers) the topk clips respectively, and processes the probability distribution of each clip output with the Gumbel-softmax algorithm to match the sub-network in the subsequent branch network module;

[0115] S150: For each branch sub-network matched by the key clip, the branch sub-network calculates and outputs a discrimination result.

[0116] See also Figure 2 , the specific steps of the key editing identification module proposed in this system are described as follows:

[0117] S200: Randomly sample batch-size of I videos from the input, where k 16-frame clips of different starting frames are randomly sampled from each video, for a total of I*k clips;

[0118] S210: Input the I*k clipping data in the previous step into the fg network to extract features respectively and obtain their high-level features;

[0119] S220: After calculating all high-level features, input them to the first-level gating unit to output the clipped part (local) critical weight α i ;

[0120] S230: performing a weighted sum of the importance of all I*k clips and the corresponding clip features to obtain the overall (global) features of the k clips after the critical weighting of k samples of the entire video;

[0121] S240: concatenate the overall features of the k clips with the sampled clips in each video, and send them to the next calculation unit;

[0122] S250: Send the concatenated features of the previous step to the second-level gating unit to obtain the second-level criticality (importance) weight β through calculation;

[0123] S260: calculating the overall secondary overall features of the k clips by performing a dot product of the secondary importance and the concatenated features and normalizing the results;

[0124] S270: Perform supervision loss on the overall features obtained by the secondary gating unit obtained in the previous step, supervise the overall features processed by each secondary gating unit of I samples by calculation, establish a mapping relationship, and update the network parameters to improve the feature extraction ability of the model.

[0125] See also Figure 3 :

[0126] S300: For the branch network module, batchsize video sample clips are input, and each sub-network model calculates (infers) the data;

[0127] S310: retain the high-level features of all models, i.e., the features before the FC layer, and obtain the probability distribution through the FC layer;

[0128] S320: Divide all high-level features by T to smooth the distribution, and calculate the softmax of each sub-model through the exp function to obtain the softened high-level features;

[0129] S330: Project the high-level features of the previous step through the two matrices of self-attention, query and key, obtain the projected data, perform matrix dot product multiplication, and obtain the attention value after matrix multiplication through softmax;

[0130] S340: According to the dot multiplication of the attention value and the probability distribution obtained in S310, the distilled distribution of each sub-model for other branches is obtained after comparing the self-attention similarity between its own high-level features and the high-level features of other models;

[0131] S350: Calculate the KL divergence for the distillation distribution and the predicted distribution of each sub-model, calculate the supervision loss for the predicted distribution, and update the parameters of each model to reduce the distribution difference.

Claims

1. A dynamic behavior recognition system based on key editing discrimination, characterized in that: The system includes a key editing identification module, a branch strategy module and a branch network module, and the three modules are connected to each other; The data is input from the key clip module. After extracting the key clip, the key clip is output to the branch strategy module. The branch strategy module matches the key clip to different sub-branch networks in the subsequent branch network module. The GumbelSoftmax algorithm is used to dynamically match the key clip to sub-networks of different sizes and different computational amounts, where: The key clip discrimination module consists of a neural network f g and two MLP gating units. After receiving the input data, it passes through the neural network f g After extracting the features, the features are calculated through two gating units to obtain the criticality ranking of the key clips; the most important k key clips in the criticality ranking, i.e., the ones with the highest criticality ranking, are selected, and the parameters are updated through the supervision loss during training; The branch strategy module is composed of a neural network π, which receives the key clips transmitted by the key clip identification module. The neural network π extracts the key clip data features and then uses the GumbelSoftmax algorithm to compare each key clip with the branch network module f l The sub-networks of different depths are matched, and the parameters are updated through supervision loss and computational constraint loss during training; The branch network module f l , consists of 4 sub-neural networks of different depths, including the original network, namely the behavior recognition network, and three modified sub-networks of the original network. The three modified sub-networks use the extended 3DRNNPooling technology to replace different computing units in the original network to achieve sub-networks of different depths; each key clip is matched by the branch sub-network to perform behavior recognition calculations to obtain the behavior recognition results; during training, parameters are updated through supervision loss and distillation loss.

2. The dynamic behavior recognition system according to claim 1, characterized in that: The neural network f g After extracting the features, the features are calculated through two gating units to obtain the criticality ranking of the key clips, and the most important k key clips in the criticality ranking are selected, including: Step 2.1: For the input action recognition video v containing hundreds of frames t , divided into multiple clips in units of 16 frames, if the last clip is less than 16 frames, return to the first frame of the clip to fill in the copy frame order until 16 frames are met; use {c1, c2…, c p } represents all clips of the video, the number of clips is p; each clip is passed through the neural network f g , extract clipping features; define the g After feature extraction, it is called local feature Step 2.2: Input the local features into the first gating unit ψ α Get the critical weight of the clip information: Step 2.3: Take α i The weighted sum of local features is used as weight to obtain the global features of the entire video Step 2.4: Concatenate the local features and global features to obtain And use the second gating unit ψ β Extract the concatenated features to obtain the purified critical weight β i ; Step 2.5: β of all the clips obtained i Sort them from large to small, take the first k βs, and use the clips corresponding to their serial numbers as the k most critical clips of this video; k is a hyperparameter.

3. The dynamic behavior recognition system according to claim 1, characterized in that: After the neural network π extracts the key clip data features, the GumbelSoftmax algorithm is used to compare the key clips with the branch network module f l The different branch sub-networks in the input are matched, and each key clip has the following steps for the k key clips, including: Step 3.1: Extract features from neural network π; obtain the probability distribution q of the output of neural network π t , t∈[1,B]; Step 3.2: By using the Gumbel_Softmax algorithm to calculate the output probability distribution q of the neural network π t Calculate the one-hot distribution and use V i , i∈[1,B] to indicate that one unit of this distribution (V) is l and the others are all 0; Step 3.3: The sub-network aligned with the maximum value and the only l in the one-hot distribution V obtained for the key clip is the matching sub-network of the current key clip in the branch network module.

4. The dynamic behavior recognition system according to claim 1, characterized in that: Branch network module l In the example, each key clip is subjected to behavior recognition calculation by the matched branch sub-network to obtain a behavior recognition result, which is specifically: Step 4.1: perform the behavior recognition calculation of the key clip in the branch sub-network matched by the key clip, and record the behavior recognition result; In step 4.2, k behavior recognition results are obtained by different branch sub-networks in the branch network module that matches the k key clips, and the average of the k behavior recognition results is taken as the final behavior recognition result of the entire video, that is, the behavior recognition result of the entire video.

5. The dynamic behavior recognition system according to claim 1, characterized in that: The training described in the key clip identification module is performed through supervised loss to update parameters, including: Step 5.1: The weights α learned for the two gating units i β i Perform weighted summation to obtain the final integrated features of the entire video Step 5.2: The supervision method is cross entropy loss. Through cross entropy loss, the neural network parameters in the module change in the direction of retaining the key features of the category: for a total number of samples N, the current category is j, and the label of the sample number i is y ij , the integrated feature f′ of the current sample in the jth category finally obtained by the two gating units v , for clarity, we use f′ ij Replace f′ v , use the following formula as the cross entropy supervision loss:

6. The dynamic behavior recognition system according to claim 1, characterized in that: During training, the parameters are updated through supervision loss and computational constraint loss as described in the branch strategy module, including: Step 6.1: N is the total number of samples, the number of behavior recognition categories is set to C, Vo is the one-hot distribution under the oth branch; in order to optimize the sub-network of the branch network module that matches the key clip to the correct classification, let the key clip x i Pass in the subsequent matched branch network module f l The probability distribution of the key clips calculated by the sub-network in is h i , the corresponding label of category j of sample i is y ij For the category of the current key clip sample, there are cross entropy loss functions of i∈[1,N],j∈[1,C],o∈[1,4]: Step 6.2: Use computational constraint loss supervision for the branch strategy module: Set the number of sub-branches to B = 4, the current branch to o, Vo to be the one-hot distribution under the o branch, and the number of floating-point operations for the o-th branch to FLOPs(o); all have loss functions: Step 6.3: The branch strategy module can be trained by calculating the gradient descent of the above two loss functions: λ is the balancing term of the two loss functions.

7. The dynamic behavior recognition system according to claim 1, characterized in that: Branch network module l The extended 3DRNNPooling technology described in is an extension of RNNPooling to 3DRNNPooling. The specific steps are as follows: Step 7.1: For a 3D data feature map with an input size of I*C*F*H*W, i.e., batch number*channel number*frame number*length*width, slice the entire data according to the pre-set stride and kernel size of the 3DRNNPooling layer to obtain slice units of multiple consecutive frames, and the number of frames can be customized; Step 7.2: Perform RNNPooling operation on each frame of the slice unit; Step 7.3: Perform a third-dimensional RNNPooling operation on the RNNPooling result obtained for each frame in the time dimension to obtain the output.

8. The dynamic behavior recognition system according to claim 1, characterized in that: The three modified sub-networks of the original network are specifically: For the 2x downsampling subnetwork: 3DRNNPooling-2x replaces the original maxpool3D and the first conv3D computing unit, and the replaced module downsamples the input data by two times; a 3DRNNPooling module also downsamples the input data by two times; For the 4x downsampling subnetwork: 3DRNNPooling-4x replaces the original maxpool3D and the first two conv3D computing units. The replaced module downsamples the input data four times; a 3DRNNPooling module also downsamples the input data four times; For the 8x downsampling subnetwork: 3DRNNPooling-8x replaces the original maxpool3D and the first three conv3D computing units. The replaced module downsamples the input data eightfold; a 3DRNNPooling module also downsamples the input data eightfold.

9. The dynamic behavior recognition system according to claim 1, characterized in that: The training method of the branch network module fl is as follows: Step 9.1: For a set of input data, which contains N video clips; for the jth output unit of the ith sample, the output of each subnetwork classifier - that is, the predicted distribution is set to Its softened form is It is the output of the fc layer and is softened by the hyperparameter T; the feature data g of the jth unit under the i-th sample ij It is the input data of the neural network Fc layer; Step 9.2: Correlation calculation: The feature data between sub-network a and sub-network b are calculated using g a g b Indicates that the correlation is defined as α ab : The projection matrices K and Q are used to determine the relevance of different branch sub-networks by self-attention to high-level features; Step 9.3: For sub-networks of different depths, based on the self-attention ratio α ab The process of transferring knowledge to other branches is to distill the features t of the teacher of the a-th sub-network. a Defined as q′ b It is the soft logarithm of the predicted distribution of the enumerated sub-network; in order to perform distillation learning between the sub-networks of the branch network module, the distillation loss calculation process for this module is defined as follows: t i represents the teacher distilled features of the i-th sub-network, q′ i Represents the soft logarithm of the prediction distribution of the sub-network; through the two formulas in this step, by reducing t i and q′ i The gap between them is used for distillation; Step 9.4: Some high-level semantic features learned from other sub-networks of different complexity are transferred to other sub-networks of this module through the above steps; at the same time, each sub-network is guided by the cross entropy objective y represents the label, q i represents the sample prediction distribution under the i-th branch sub-network; the cross entropy target is Indicates, i represents the sub-network number of the branch network module; finally, the loss function is calculated for all sub-networks of different depths by the following formula; T 2 To balance the terms, a multi-depth mutual distillation is performed in this way, which effectively reduces the improvement of generalization effect caused by overfitting.

Images