A training method and computer device for video skeleton action recognition model

Abstract

The present invention discloses a training method and computer equipment for a video skeleton action recognition model. The video skeleton action recognition model includes a block encoder, an action classifier, and a self-attention transformer that internally introduces a parameter efficient fine-tuning framework built by a self-attention mechanism, an expert modality adapter, and a cross-modality adapter. The method includes: inputting a video sequence and a skeleton sequence into a block encoder to obtain a video feature vector and a skeleton feature vector; inputting the position-encoded video feature vector and the skeleton feature vector into a self-attention transformer to obtain a video deep feature vector and a skeleton deep feature vector; inputting the video deep feature vector and the skeleton deep feature vector into an action classifier, determining the classification loss according to the predicted label and the true label of the video sequence and the predicted label and the true label of the skeleton sequence, and then fine-tuning the parameters of the expert modality adapter and the cross-modality adapter. The present invention can reduce resource consumption during model training.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method and computer equipment for a video skeleton action recognition model. Background Art

[0002] With the development of technology, vision-based human action recognition technology has shifted from using a single modality to using multiple modalities to improve recognition accuracy and robustness. These modalities include RGB images, depth images, skeleton information, infrared images, point cloud data, etc. RGB images can provide rich scene details, including the shape, color, and texture of objects. These details are crucial for understanding the semantic content of the action, while skeleton information can effectively eliminate background interference and focus on capturing the motion characteristics of the action.

[0003] When processing video data, the encoding of temporal information is particularly critical for distinguishing various actions, but the addition of the temporal dimension also brings challenges to human action recognition technology. In response to this, some scholars have proposed using a variety of deep learning architectures including three-dimensional convolutional neural networks, recurrent neural networks, and long short-term memory networks to encode temporal information in video data, but they usually face limitations when processing complex actions. Later, some scholars proposed introducing the Transformer model into the field of human action recognition to better process sequence data, but due to the complex structure of the Transformer model, it usually requires a lot of computing resources for training, resulting in high computing costs and resource consumption. Finding more efficient training methods and model optimization techniques has become the focus of current research. Summary of the invention

[0004] The present invention provides a training method and computer equipment for a video skeleton action recognition model to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0005] In a first aspect, a training method for a video skeleton action recognition model is provided, wherein the video skeleton action recognition model includes a block encoder, a self-attention transformer and an action classifier, and a parameter efficient fine-tuning framework constructed by a self-attention mechanism with a time induction bias, an expert modality adapter and a cross-modality adapter is introduced into the self-attention transformer, and the method includes:

[0006] Get video sequence and skeleton sequence;

[0007] Processing the video sequence and the skeleton sequence by using the block encoder to obtain a video feature vector and a skeleton feature vector;

[0008] Position encoding is performed on the video feature vector and the skeleton feature vector, and then the position-encoded video feature vector and the skeleton feature vector are processed by the self-attention transformer to obtain a video deep feature vector and a skeleton deep feature vector;

[0009] Using the action classifier to process the video deep feature vector and the skeleton deep feature vector, to obtain a prediction label of the video sequence and a prediction label of the skeleton sequence;

[0010] The classification loss is determined based on the predicted labels and true labels of the video sequence and the predicted labels and true labels of the skeleton sequence, and then the parameters of the expert modality adapter and the cross-modality adapter are fine-tuned in combination with the back-propagation algorithm to train the video skeleton action recognition model.

[0011] Furthermore, the obtaining of the video sequence and the skeleton sequence comprises:

[0012] Obtain the original video sequence and the original skeleton sequence;

[0013] Sampling and data augmenting the original video sequence to obtain the video sequence;

[0014] The original skeleton sequence is sampled and data augmented to obtain the skeleton sequence.

[0015] Further, the self-attention transformer includes a plurality of self-attention transformation modules connected in sequence, each of which includes a video feature extraction module, a skeleton feature extraction module and a cross-modal adapter;

[0016] The video feature extraction module includes a first multi-layer perceptron, a first modality adapter, and a first attention module formed by a convolutional modality adapter and a first self-attention network based on partial time perception, and the skeleton feature extraction module includes a second multi-layer perceptron, a second modality adapter, and a second attention module formed by a graph convolutional modality adapter and a second self-attention network based on partial time perception;

[0017] After layer-normalizing the input video feature vector, the first attention module and the first modality adapter are sequentially used to process the first video feature vector, the input video feature vector and the first video feature vector are added to obtain a second video feature vector, the second video feature vector is layer-normalized to obtain a third video feature vector, and the third video feature vector is processed by the first multi-layer perceptron to obtain a fourth video feature vector;

[0018] After layer-normalizing the input skeleton feature vector, the first skeleton feature vector is processed by the second attention module and the second modality adapter in sequence to obtain a first skeleton feature vector, the input skeleton feature vector and the first skeleton feature vector are added to obtain a second skeleton feature vector, the second skeleton feature vector is layer-normalized to obtain a third skeleton feature vector, and the third skeleton feature vector is processed by the second multi-layer perceptron to obtain a fourth skeleton feature vector;

[0019] The third video feature vector and the third bone feature vector are cross-processed by the cross-modal adapter to obtain a fifth video feature vector and a fifth bone feature vector, the second video feature vector, the fourth video feature vector and the fifth bone feature vector are added to obtain a video deep feature vector, and the second bone feature vector, the fourth bone feature vector and the fifth video feature vector are added to obtain a bone deep feature vector.

[0020] Furthermore, the first self-attention network and the second self-attention network have the same structure, and either self-attention network includes a multi-head self-attention layer;

[0021] A key matrix, a value matrix and a query matrix are determined according to the first input feature vector, and different time masks are introduced through the multi-head self-attention layer to process the key matrix, the value matrix and the query matrix to obtain multiple attention vectors, and then the multiple attention vectors are concatenated and linearly mapped to obtain a first output feature vector.

[0022] Further, the convolutional modality adapter includes a first activation function, a depthwise separable convolutional layer, a first fully connected layer for compressing feature dimensions, and a second fully connected layer for expanding feature dimensions;

[0023] The first input video feature vector is processed sequentially through the first fully connected layer, the first activation function, the depthwise separable convolutional layer and the second fully connected layer to obtain a sixth video feature vector, and the first input video feature vector and the sixth video feature vector are added to obtain a first output video feature vector.

[0024] Further, the graph convolution modality adapter includes a second activation function, a graph convolution network layer, a third fully connected layer for compressing feature dimensions, and a fourth fully connected layer for expanding feature dimensions;

[0025] The first input bone feature vector is processed sequentially through the third fully connected layer, the second activation function, the graph convolutional network layer and the fourth fully connected layer to obtain a sixth bone feature vector, and the first input bone feature vector and the sixth bone feature vector are added to obtain a first output bone feature vector.

[0026] Further, the cross-modal adapter includes a first cross-attention layer, a second cross-attention layer, a third cross-attention layer, a fourth cross-attention layer, a third modality adapter, and a fourth modality adapter;

[0027] Processing the second input skeleton feature vector and the first learnable variable through the first cross attention layer to obtain a first cross feature vector, the first variable represents the latent space feature of the video modality, processing the second input video feature vector and the first cross feature vector through the second cross attention layer to obtain a seventh skeleton feature vector, and processing the seventh skeleton feature vector through the third modality adapter to obtain a second output skeleton feature vector;

[0028] The second input video feature vector and the learnable second variable are processed by the third cross-attention layer to obtain a second cross-feature vector, wherein the second variable represents the latent space feature of the skeletal modality. The second input skeletal feature vector and the second cross-feature vector are processed by the fourth cross-attention layer to obtain a seventh video feature vector. The seventh video feature vector is processed by the fourth modality adapter to obtain a second output video feature.

[0029] Further, the first modality adapter, the second modality adapter, the third modality adapter and the fourth modality adapter have the same structure, and any modality adapter includes a third activation function, a fifth fully connected layer for compressing feature dimensions, and a sixth fully connected layer for expanding feature dimensions;

[0030] The second input feature vector is processed sequentially through the fifth fully connected layer, the third activation function and the sixth fully connected layer to obtain an intermediate feature vector, and the second input feature vector and the intermediate feature vector are added to obtain a second output feature vector.

[0031] Furthermore, the method further comprises:

[0032] Obtain the video sequence to be tested and the skeleton sequence to be tested corresponding to the user during the movement;

[0033] The video sequence to be tested and the skeleton sequence to be tested are processed using a trained video skeleton action recognition model to obtain the action category information of the user.

[0034] In a second aspect, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the training method of the video skeletal action recognition model as described in the first aspect.

[0035] The present invention has at least the following beneficial effects: by adding a temporal inductive bias to the original spatial self-attention of the model to construct a self-attention network based on partial time perception, the ability of the original spatial self-attention to extract temporal information can be effectively improved; by setting different expert modality adapters in the model to extract features from different modalities, and by setting a cross-modality adapter in the model to realize feature transfer learning between video modality and skeletal modality, the accuracy and robustness of the model in recognizing actions can be improved; by introducing a parameter efficient fine-tuning framework constructed by a self-attention mechanism with added temporal inductive bias, an expert modality adapter and a cross-modality adapter into the model, resource consumption during model training can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0037] Figure 1 is a schematic diagram of the structure of a single self-attention transformation module in an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of the structure of any self-attention network in an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of the structure of a convolutional modality adapter in an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of the structure of a graph convolution modality adapter in an embodiment of the present invention;

[0041] Figure 5 is a schematic diagram of the structure of a cross-modal adapter in an embodiment of the present invention;

[0042] Figure 6 is a schematic diagram of the structure of any modal adapter in an embodiment of the present invention;

[0043] Figure 7 It is a flowchart of a training method for a video skeleton action recognition model in an embodiment of the present invention;

[0044] Figure 8 It is a schematic diagram of the hardware structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the system or the order in the flow chart. The terms "first", "second", "third", "fourth", etc. in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, such as processes, methods, systems, products or devices that include a series of steps or units, which are not necessarily limited to those steps or units clearly listed, but may include other steps or units inherent to these processes, methods, products or devices that are not clearly listed.

[0047] First, some of the terms involved in the present invention are explained as follows:

[0048] Parameter-Efficient Fine-Tuning (PEFT) is a technique for fine-tuning large pre-trained models, which aims to improve the performance of the model on new tasks while reducing the number of fine-tuning parameters and computational complexity.

[0049] The back propagation algorithm is an algorithm used to train artificial neural networks. It is based on the gradient descent method and updates the weights by calculating the partial derivatives of the loss function with respect to the network weights, thereby optimizing the performance of the network.

[0050] Layer Normalization is a common normalization technique in neural networks. It mainly normalizes the features of each sample independently, aiming to accelerate convergence and stabilize training during training, while improving the generalization ability of the model.

[0051] At present, the Transformer (self-attention transformer) model has been introduced into the field of action recognition due to its advantages in processing sequence data. However, due to the complex structure and numerous parameters of the Transformer model, a large amount of computing resources are usually required for training, which leads to high computing costs and resource consumption. At present, some scholars have proposed to use efficient parameter fine-tuning technology to alleviate the problems of large data requirements, model overfitting, and high training costs. Its core goal is to keep the parameters of large-scale pre-trained language models frozen and achieve excellent performance in downstream tasks by fine-tuning a small number of parameters. This efficient parameter fine-tuning technology usually adds additional time modules to large-scale pre-trained Transformer models to achieve spatiotemporal decoupling, but this decoupling method often destroys long-distance dependencies in space and time, and the time module cannot fully utilize the spatial self-attention pre-training weights to extract time information. Based on this, an embodiment of the present invention provides a training method and computer device for a video skeletal action recognition model, wherein the video skeletal action recognition model includes a block encoder, a self-attention transformer and an action classifier connected in sequence, an efficient parameter fine-tuning framework constructed by a cross-modal adapter, an expert modality adapter and a self-attention mechanism is introduced into the self-attention transformer, and a temporal induction bias is added to the self-attention mechanism, so as to overcome the relevant application defects of the current efficient parameter fine-tuning technology in the Transformer model as much as possible.

[0052] In an embodiment of the present invention, the self-attention transformer includes a plurality of self-attention transformation modules connected in sequence. Preferably, the number of the plurality of self-attention transformation modules is set to 12, and the network structure of each self-attention transformation module is the same.

[0053] Please refer to Figure 1 , Figure 1 is a schematic diagram of the structure of a single self-attention transformation module provided in an embodiment of the present invention, wherein the self-attention transformation module includes a cross-modality adapter (CrossModalityAdapter), a skeleton feature extraction module and a video feature extraction module, wherein:

[0054] The video feature extraction module includes a convolutional modality adapter (ConvExpert), a first modality adapter (Adapter1), a first multilayer perceptron (MLP1), and a first self-attention network (Modality-Agnostic1) based on partial temporal perception and independent of modality, wherein the first self-attention network (Modality-Agnostic1) and the convolutional modality adapter (ConvExpert) form a first attention module (PTP-MSA1);

[0055] The skeleton feature extraction module includes a graph convolution modality adapter (GCNExpert), a second modality adapter (Adapter2), a second multilayer perceptron (MLP2) and a second self-attention network (Modality-Agnostic2) based on partial temporal perception and independent of modality. The second self-attention network (Modality-Agnostic2) and the graph convolution modality adapter (GCNExpert) form a second attention module (PTP-MSA2). The convolution modality adapter (ConvExpert) and the graph convolution modality adapter (GCNExpert) are both referred to as the expert modality adapter.

[0056] In practical applications, a layer normalization operation is performed on the input video feature vector to obtain a first intermediate feature vector, the first intermediate feature vector is input into the first attention module (PTP-MSA1) for feature extraction to obtain a second intermediate feature vector, the second intermediate feature vector is input into the first modality adapter (Adapter1) for feature enhancement to obtain a first video feature vector, the first video feature vector and the input video feature vector are added to obtain a second video feature vector, a layer normalization operation is performed on the second video feature vector to obtain a third video feature vector, and the third video feature vector is input into the first multi-layer perceptron (MLP1) for feature aggregation to obtain a fourth video feature vector;

[0057] Performing a layer normalization operation on the input skeleton feature vector to obtain a third intermediate feature vector, inputting the third intermediate feature vector into the second attention module (PTP-MSA2) for feature extraction to obtain a fourth intermediate feature vector, inputting the fourth intermediate feature vector into the second modality adapter (Adapter2) for feature enhancement to obtain a first skeleton feature vector, adding the first skeleton feature vector and the input skeleton feature vector to obtain a second skeleton feature vector, performing a layer normalization operation on the second skeleton feature vector to obtain a third skeleton feature vector, and inputting the third skeleton feature vector into the second multilayer perceptron (MLP2) for feature aggregation to obtain a fourth skeleton feature vector;

[0058] The third bone feature vector and the third video feature vector are input into the cross-modality adapter (CrossModalityAdapter) for cross-processing to obtain a fifth video feature vector and a fifth bone feature vector, the fifth video feature vector, the fourth bone feature vector and the second bone feature vector are added to obtain a bone deep feature vector, and the fifth bone feature vector, the fourth video feature vector and the second video feature vector are added to obtain a video deep feature vector.

[0059] On this basis, the internal implementation process of the self-attention transformation module is expressed by the following formula:

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] in, is the second video feature vector, is the input video feature vector, is the first intermediate eigenvector, is the second bone feature vector, is the input bone feature vector, is the third intermediate eigenvector, is the fifth bone eigenvector, is the fifth video feature vector, is the third video feature vector, is the third bone feature vector, is the deep feature vector of the video, is the deep feature vector of the skeleton, and s is a hyperparameter.

[0068] In an embodiment of the present invention, the first self-attention network (Modality-Agnostic1) and the second self-attention network (Modality-Agnostic2) have the same structure. Mainly by adding different time masks to different pre-trained spatial self-attention heads, some attention heads can maintain the spatial information in the pre-trained model, and at the same time, some attention heads can be adjusted to perform domain adaptation to learn spatiotemporal features, so as to solve the problem that temporal modeling in a decoupled manner will destroy long-distance dependencies in the video.

[0069] Please refer to Figure 2 , Figure 2It is a structural diagram of any self-attention network provided by an embodiment of the present invention, and the self-attention network includes a multi-head self-attention layer; in actual application, a query encoder is used to process a first input feature vector to obtain a query matrix, a key encoder is used to process the first input feature vector to obtain a key matrix, and a value encoder is used to process the first input feature vector to obtain a value matrix, different time masks are set in different heads in the multi-head self-attention layer, and then the multi-head self-attention layer is used to perform self-attention operations on the query matrix, the key matrix and the value matrix to obtain multiple attention vectors, and finally the multiple attention vectors are spliced ​​and linearly mapped to obtain a first output feature vector.

[0070] On this basis, the internal implementation process of the self-attention network is expressed by the following formula:

[0071] Q=Z l1 W Q ,K=Z l1 W K ,V=Z l1 W V ,

[0072] Modality-Agnostic(Z l1 )=MultiHead(Q,K,V)

[0073] =Concat(head1,head2,...,head H )W O ,

[0074]

[0075] Where Q is the query matrix, K is the key matrix, V is the value matrix, and Z l1 is the first input feature vector, W Q , W K , W V are all learnable linear transformation parameters, W O is the linear mapping weight, head i is the i-th attention vector, H is the number of heads contained in the multi-head self-attention layer and is preferably set to 8, d is the feature dimension in the self-attention network, M is the time mask used for each standard scaled dot product attention, T is the transpose symbol, and Concat represents the concatenation operation on the dimension.

[0076] Please refer to Figure 3 , Figure 3It is a structural diagram of a convolutional modality adapter provided by an embodiment of the present invention, wherein the convolutional modality adapter (ConvExpert) mainly includes a first fully connected layer (FC Down1, which is used to compress feature dimensions), a first activation function (GELU1), a depthwise separable convolutional layer (DWConv) and a second fully connected layer (FC Up1, which is used to expand feature dimensions); in practical applications, the first fully connected layer (FC Down1) is used to perform dimension compression on a first input video feature vector to obtain a fifth intermediate feature vector, the first activation function (GELU1) is used to perform nonlinear mapping on the fifth intermediate feature vector to obtain a sixth intermediate feature vector, the depthwise separable convolutional layer (DWConv) is used to process the sixth intermediate feature vector to obtain a seventh intermediate feature vector, the second fully connected layer (FC Up1) is used to perform dimension expansion on the seventh intermediate feature vector to obtain a sixth video feature vector, and then the sixth video feature vector and the first input video feature vector having the same dimension are added to obtain a first output video feature vector.

[0077] On this basis, the internal implementation process of the convolutional modality adapter (ConvExpert) is expressed by the following formula:

[0078]

[0079] in, is the learnable linear mapping parameter matrix, r adapter is a manually set hyperparameter and is preferably set to 0.25, d m1 is the feature dimension in the convolutional modality adapter, R represents a real number set, is the first input video feature vector.

[0080] Please refer to Figure 4 , Figure 4It is a structural schematic diagram of a graph convolution modality adapter provided by an embodiment of the present invention, and the graph convolution modality adapter (GCNExpert) mainly includes a third fully connected layer (FC Down2, which is used to compress feature dimensions), a second activation function (GELU2), a graph convolution network layer (GCN) and a fourth fully connected layer (FC Up2, which is used to expand feature dimensions); in practical applications, the third fully connected layer (FC Down2) is used to perform dimension compression on the first input bone feature vector to obtain an eighth intermediate feature vector, the second activation function (GELU2) is used to perform nonlinear mapping on the eighth intermediate feature vector to obtain a ninth intermediate feature vector, the graph convolution network layer (GCN) is used to process the ninth intermediate feature vector to obtain a tenth intermediate feature vector, the fourth fully connected layer (FC Up2) is used to perform dimension expansion on the tenth intermediate feature vector to obtain a sixth bone feature vector, and then the sixth bone feature vector and the first input bone feature vector with the same dimension are added to obtain a first output bone feature vector.

[0081] On this basis, the internal implementation process of the graph convolutional modality adapter (GCNExpert) is expressed by the following formula:

[0082]

[0083] in, is the learnable linear mapping parameter matrix, d m2 is the feature dimension in the graph convolutional modality adapter, is the first input bone feature vector.

[0084] Please refer to Figure 5 , Figure 5 : is a schematic diagram of the structure of a cross-modality adapter provided by an embodiment of the present invention, wherein the cross-modality adapter (CrossModalityAdapter) mainly includes a third modality adapter (Adapter3), a fourth modality adapter (Adapter4), a first cross attention layer (CrossAttention1), a second cross attention layer (CrossAttention2), a third cross attention layer (CrossAttention3) and a fourth cross attention layer (CrossAttention4);

[0085] In practical applications, a learnable first variable and a second variable are determined, wherein the first variable is used to characterize the latent spatial features of the video modality, and the second variable is used to characterize the latent spatial features of the skeletal modality;

[0086] The first cross-attention layer (CrossAttention1) is used to perform a cross-attention operation on the first variable and the second input bone feature vector to obtain a first cross-feature vector about the bone modality, the second cross-attention layer (CrossAttention2) is used to perform a cross-attention operation on the first cross-feature vector and the second input video feature vector to obtain a seventh bone feature vector, and the third modality adapter (Adapter3) is used to perform feature enhancement on the seventh bone feature vector to obtain a second output bone feature vector; the third cross-attention layer (CrossAttention3) is used to perform a cross-attention operation on the second variable and the second input video feature vector to obtain a second cross-feature vector about the video modality, the fourth cross-attention layer (CrossAttention4) is used to perform a cross-attention operation on the second cross-feature vector and the second input bone feature vector to obtain a seventh video feature vector, and the fourth modality adapter (Adapter4) is used to perform feature enhancement on the seventh video feature vector to obtain a second output video feature.

[0087] On this basis, the internal implementation process of the cross-modality adapter (CrossModalityAdapter) is expressed by the following formula:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] in, is the second input video feature vector, is the second input bone feature vector, is the seventh bone eigenvector, is the first cross eigenvector, g is the learnable proportional control parameter, is the seventh video feature vector, is the second cross eigenvector, is the first variable, is the second variable.

[0094] In an embodiment of the present invention, the first modality adapter, the second modality adapter, the third modality adapter and the fourth modality adapter have the same structure.

[0095] Please refer to Figure 6 , Figure 6 is a schematic diagram of the structure of any modality adapter provided in an embodiment of the present invention, the modality adapter mainly includes a fifth fully connected layer (FC Down3, which is used to compress feature dimensions), a third activation function (GELU3) and a sixth fully connected layer (FC Up3, which is used to expand feature dimensions);

[0096] In practical applications, the fifth fully connected layer (FC Down3) is used to perform dimension compression on the second input feature vector to obtain an eleventh intermediate feature vector, the third activation function (GELU3) is used to perform nonlinear mapping on the eleventh intermediate feature vector to obtain a twelfth intermediate feature vector, the sixth fully connected layer (FC Up3) is used to perform dimension expansion on the twelfth intermediate feature vector to obtain the final required intermediate feature vector, and the intermediate feature vector and the second input feature vector with the same dimension are added to obtain a second output feature vector.

[0097] On this basis, the internal implementation process of the modal adapter is expressed by the following formula:

[0098] Adapter(Z l2 )=Z l2 +GELU3(Z l2 W down3 )W up3 ;

[0099] in, is the learnable linear mapping parameter matrix, d m3 is the characteristic dimension in the modal adapter, Z l2 is the second input feature vector.

[0100] Please refer to Figure 7 , Figure 7 1 is a flow chart of a method for training a video skeleton action recognition model provided by an embodiment of the present invention, the method comprising the following steps:

[0101] Step S110, obtaining a video sequence and a skeleton sequence;

[0102] Step S120, using the block encoder to process the video sequence and the skeleton sequence to obtain a video feature vector and a skeleton feature vector;

[0103] Step S130, position encoding the video feature vector and the skeleton feature vector, and then using the self-attention transformer to process the position-encoded video feature vector and the skeleton feature vector to obtain a video deep feature vector and a skeleton deep feature vector;

[0104] Step S140: Process the video deep feature vector and the skeleton deep feature vector using the action classifier to obtain the predicted label of the video sequence and the predicted label of the skeleton sequence;

[0105] Step S150: Determine the classification loss according to the predicted label and the true label of the video sequence and the predicted label and the true label of the skeleton sequence, and then fine-tune the parameters of the expert modality adapter and the cross-modal adapter in combination with the backpropagation algorithm to train the video-skeleton action recognition model.

[0106] In some embodiments of the present invention, the implementation process of the above step S110 includes but is not limited to the following:

[0107] Step S111: Obtain the original skeleton sequence and the original video sequence;

[0108] Specifically, the original skeleton sequence and the original video sequence are mainly obtained from the existing NTU60 dataset. Each data included in the NTU60 dataset is collected by a Microsoft Kinect v2 camera with three lenses. The middle lens is an RGB color camera, and the lenses on its left and right sides are both 3D structured light depth sensors composed of an infrared emitter and an infrared CMOS camera. The RGB color camera is used to capture video modality data, and the 3D structured light depth sensor is used to capture depth data and extract the position data of the skeleton modality from it;

[0109] Step S112: Perform sampling and data augmentation operations on the original video sequence to obtain the final required video sequence;

[0110] Step S113: Perform sampling and data augmentation operations on the original skeleton sequence to obtain the final required skeleton sequence.

[0111] More specifically, the sampling operations mentioned in the above step S112 and the above step S113 are based on the same principle. Here, only the sampling process of the original video sequence is further described as follows:

[0112] The first optional sampling method is dense sampling. Assume that the total number of frames included in the original video sequence is T′, and the original video sequence is denoted as If the number of frames to be sampled currently is T (T < T′), set the sampling interval as τ, and the sampled video sequence is The advantage of this dense sampling method is that it can ensure the absolute position information in the original video sequence.

[0113] The second optional sampling method is uniform sampling. Assume that the total number of frames contained in the original video sequence is T′, and denote the original video sequence as If the number of frames to be sampled currently is T (T < T′), first evenly divide the original video sequence into τ = T′ / T intervals, and then randomly select one frame in each divided interval inside (n - 1)τ < t n-1 < nτ, and the sampled video sequence is The advantage of this uniform sampling method is that it can ensure the relative position information in the original video sequence.

[0114] More specifically, in the above step S112, the data augmentation operations performed on the sampled video sequence include image scaling, image cropping, and image flipping; image scaling refers to scaling the image along the time dimension using the same ratio, which can be implemented using algorithms such as the nearest neighbor interpolation algorithm, bilinear interpolation algorithm, or bicubic interpolation algorithm, etc.; image cropping refers to randomly selecting a rectangular area in the image and deleting other unnecessary parts. This operation only reduces the size of the image but does not affect the resolution of the image; image flipping refers to flipping the image horizontally, vertically, or diagonally.

[0115] More specifically, in the above step S113, the data augmentation operation performed on the sampled skeleton sequence includes normalization processing. For each piece of skeleton data contained in the sampled skeleton sequence, since the frame length of the depth data related to this piece of skeleton data collected by the 3D structured light depth sensor is Δx and the frame width is Δy, update the position information (x, y) of each skeleton node contained in this piece of skeleton data to (x / Δx, y / Δy), and then normalize the updated position information of each skeleton node to the range [0, 1] to obtain the corresponding augmented skeleton data.

[0116] It should be noted that for the above image cropping operation, a specific and compact square area can also be selected on the image and other parts can be deleted, which is mainly obtained after analyzing the sampled skeleton sequence. The specific description is as follows:

[0117] For any RGB image data contained in the sampled video sequence, obtain a piece of skeleton data corresponding to the acquisition time of this RGB image data from the sampled skeleton sequence. According to the position information of all skeleton nodes contained in this piece of skeleton data, determine the position range jointly covered by all skeleton nodes, which is composed of x min 、x max 、y min 、y maxThese four position parameters define and form a rectangular frame, and the center point position (x center ,y center ), width w half and high h half :

[0118] x min =x center -w half ,

[0119] x max =x center +w half ,

[0120] y min =y center -h half ,

[0121] y max =y center +h half ;

[0122] From this width half and the high h half Extract the minimum value, and then use the minimum value as the side length, and according to the center point position (x center ,y center ) to determine a square frame and obtain the positions of its four corner points. The square frame is equivalent to clipping a set of parallel edges on the rectangular frame. Finally, the four corner points and the center point position (x center ,y center ) performs cropping on the RGB image data.

[0123] In some embodiments of the present invention, the processing method of the block encoder on different sequences mentioned in the above step S120 is specifically described as follows:

[0124] For the video sequence, the shape when it is input to the block encoder is (N, C in ,S,H,W), move the data in the D direction to the Batch dimension and perform a 2D convolution operation:

[0125]

[0126] Among them, out(N i ,C out ) is the video feature vector, N i is the index on the Batch dimension, C outIt is the feature dimension of the convolution output. By controlling Padding and Stride, the size of the final output feature vector is indirectly controlled, and a non-overlapping convolution method is used.

[0127] For the skeleton sequence, the shape when it is input to the block encoder is (C in ,M,D,V), C in The data in the direction is moved to the last dimension and then linear operations are performed:

[0128] y=xW+b;

[0129] Among them, y is the bone feature vector, x is the bone sequence, W is the weight of the linear layer, and b is the additive bias.

[0130] In some embodiments of the present invention, the position encoding operation mentioned in the above step S130 is specifically described as follows:

[0131] According to the length L of the video sequence rgb , the length L of the skeleton sequence ske And the feature dimension d in the self-attention transformer model , initialize the relative position encoding:

[0132]

[0133] Then the relative position code P of the video sequence is rgb Add the video feature vector and encode the relative position of the skeleton sequence P ske It is added to the bone feature vector to complete the position encoding operation.

[0134] In some embodiments of the present invention, the classification loss mentioned in the above step S150 is calculated using the following formula:

[0135] Loss total =λ rgb Loss rgb +λ ske Loss ske ,

[0136] Loss rgb =-∑y′ rgb log(y rgb ),Loss ske =-∑y′ ske log(y ske );

[0137] Among them, Loss total is the classification loss, Lossrgb is the classification loss for the video sequence, Loss ske is the classification loss for the skeleton sequence. These two classification losses are determined by the cross entropy loss function. rgb , ske are all hyperparameters, y′ rgb is the probability distribution of the true label of the video sequence, y rgb is the probability distribution of the predicted label of the video sequence, y′ ske is the probability distribution of the true label of the skeleton sequence, y ske is the probability distribution of the predicted labels for the bone sequence.

[0138] In some embodiments of the present invention, after obtaining the trained video skeleton action recognition model, it can be put into formal application, specifically by obtaining the corresponding bone sequence to be tested and the video sequence to be tested during the user's movement, and then inputting the bone sequence to be tested and the video sequence to be tested into the trained video skeleton action recognition model for analysis to obtain the user's action category information.

[0139] In an embodiment of the present invention, by adding a temporal inductive bias to the original spatial self-attention of the model to construct a self-attention network based on partial time perception, the ability of the original spatial self-attention to extract temporal information can be effectively improved; by setting different expert modality adapters in the model to perform feature extraction on different modalities, and setting a cross-modal adapter in the model to realize feature transfer learning between video modality and skeletal modality, the accuracy and robustness of the model in recognizing actions can be improved; by introducing a parameter efficient fine-tuning framework constructed by a self-attention mechanism with added temporal inductive bias, an expert modality adapter and a cross-modal adapter into the model, resource consumption during model training can be reduced.

[0140] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, a training method for a video skeletal action recognition model in the above embodiment is implemented. Wherein, the computer-readable storage medium includes but is not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. In other words, the storage device includes any medium that stores or transmits information in a readable form by a device (such as a computer, a mobile phone, etc.), which can be a read-only memory, a disk or an optical disk, etc.

[0141] also, Figure 8 2 is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention, wherein the computer device includes a processor 220, a memory 230, an input unit 240, a display unit 250 and other devices. Those skilled in the art can understand that Figure 8 The device structure device shown does not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 230 can be used to store the computer program 210 and various functional modules, and the processor 220 runs the computer program 210 stored in the memory 230, thereby executing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include an internal memory and an external memory. The internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory may include a hard disk, a floppy disk, a U disk, a magnetic tape, etc. The memory 230 disclosed in the embodiment of the present invention includes but is not limited to the above-mentioned types of memory. The memory 230 disclosed in the embodiment of the present invention is only an example and not a limitation.

[0142] The input unit 240 is used to receive the input of the signal and the keyword input by the user. The input unit 240 may include a touch panel and other input devices. The touch panel can collect the user's touch operation on or near it (such as the user's operation on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program; other input devices may include but are not limited to one or more of a physical keyboard, a function key (such as a playback control key, a switch key, etc.), a trackball, a mouse, a joystick, etc. The display unit 250 can be used to display the information input by the user or the information provided to the user and various menus of the terminal device. The display unit 250 can be in the form of a liquid crystal display, an organic light emitting diode, etc. The processor 220 is the control center of the terminal device, which uses various interfaces and lines to connect various parts of the entire device, and executes various functions and processes data by running or executing software programs and / or modules stored in the memory 230, and calling data stored in the memory 230.

[0143] As an embodiment, the computer device includes a processor 220, a memory 230 and a computer program 210, wherein the computer program 210 is stored in the memory 230 and is configured to be executed by the processor 220, and the computer program 210 is configured to execute a training method for a video skeletal action recognition model in the above-mentioned embodiment.

[0144] Although the description of the present application has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims by reference to the attached claims, taking into account the prior art, so as to effectively cover the intended scope of the present application. In addition, the above description of the present application is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present application that have not yet been foreseen may still represent equivalent changes to the present application.

Claims

1. A training method for a video skeleton action recognition model, characterized in that: The video skeleton action recognition model includes a block encoder, a self-attention transformer and an action classifier. A parameter efficient fine-tuning framework constructed by a self-attention mechanism with a time induction bias, an expert modality adapter and a cross-modality adapter is introduced into the self-attention transformer. The method includes: Get video sequence and skeleton sequence; Processing the video sequence and the skeleton sequence by using the block encoder to obtain a video feature vector and a skeleton feature vector; Position encoding is performed on the video feature vector and the skeleton feature vector, and then the position-encoded video feature vector and the skeleton feature vector are processed by the self-attention transformer to obtain a video deep feature vector and a skeleton deep feature vector; Using the action classifier to process the video deep feature vector and the skeleton deep feature vector, to obtain a prediction label of the video sequence and a prediction label of the skeleton sequence; Determine the classification loss according to the predicted label and the true label of the video sequence and the predicted label and the true label of the skeleton sequence, and then fine-tune the parameters of the expert modality adapter and the cross-modality adapter in combination with the back propagation algorithm to train the video skeleton action recognition model; The self-attention transformer comprises a plurality of self-attention transformation modules connected in sequence, each of which comprises a video feature extraction module, a skeleton feature extraction module and a cross-modal adapter; The video feature extraction module includes a first multi-layer perceptron, a first modality adapter, and a first attention module formed by a convolutional modality adapter and a first self-attention network based on partial time perception, and the skeleton feature extraction module includes a second multi-layer perceptron, a second modality adapter, and a second attention module formed by a graph convolutional modality adapter and a second self-attention network based on partial time perception; After layer-normalizing the input video feature vector, the first attention module and the first modality adapter are sequentially used to process the first video feature vector, the input video feature vector and the first video feature vector are added to obtain a second video feature vector, the second video feature vector is layer-normalized to obtain a third video feature vector, and the third video feature vector is processed by the first multi-layer perceptron to obtain a fourth video feature vector; After layer-normalizing the input skeleton feature vector, the first skeleton feature vector is processed by the second attention module and the second modality adapter in sequence to obtain a first skeleton feature vector, the input skeleton feature vector and the first skeleton feature vector are added to obtain a second skeleton feature vector, the second skeleton feature vector is layer-normalized to obtain a third skeleton feature vector, and the third skeleton feature vector is processed by the second multi-layer perceptron to obtain a fourth skeleton feature vector; The third video feature vector and the third bone feature vector are cross-processed by the cross-modal adapter to obtain a fifth video feature vector and a fifth bone feature vector, the second video feature vector, the fourth video feature vector and the fifth bone feature vector are added to obtain a video deep feature vector, and the second bone feature vector, the fourth bone feature vector and the fifth video feature vector are added to obtain a bone deep feature vector.

2. The training method of the video skeleton action recognition model according to claim 1, characterized in that: The obtaining of the video sequence and the skeleton sequence comprises: Obtain the original video sequence and the original skeleton sequence; Sampling and data augmenting the original video sequence to obtain the video sequence; The original skeleton sequence is sampled and data augmented to obtain the skeleton sequence.

3. The training method of the video skeleton action recognition model according to claim 1, characterized in that: The first self-attention network and the second self-attention network have the same structure, and either self-attention network includes a multi-head self-attention layer; A key matrix, a value matrix and a query matrix are determined according to the first input feature vector, and different time masks are introduced through the multi-head self-attention layer to process the key matrix, the value matrix and the query matrix to obtain multiple attention vectors, and then the multiple attention vectors are concatenated and linearly mapped to obtain a first output feature vector.

4. The training method of the video skeleton action recognition model according to claim 1, characterized in that: The convolutional modality adapter includes a first activation function, a depthwise separable convolutional layer, a first fully connected layer for compressing feature dimensions, and a second fully connected layer for expanding feature dimensions; The first input video feature vector is processed sequentially through the first fully connected layer, the first activation function, the depthwise separable convolutional layer and the second fully connected layer to obtain a sixth video feature vector, and the first input video feature vector and the sixth video feature vector are added to obtain a first output video feature vector.

5. The training method of the video skeleton action recognition model according to claim 1, characterized in that: The graph convolution modality adapter includes a second activation function, a graph convolution network layer, a third fully connected layer for compressing feature dimensions, and a fourth fully connected layer for expanding feature dimensions; The first input bone feature vector is processed sequentially through the third fully connected layer, the second activation function, the graph convolutional network layer and the fourth fully connected layer to obtain a sixth bone feature vector, and the first input bone feature vector and the sixth bone feature vector are added to obtain a first output bone feature vector.

6. The training method of the video skeleton action recognition model according to claim 1, characterized in that: The cross-modal adapter includes a first cross-attention layer, a second cross-attention layer, a third cross-attention layer, a fourth cross-attention layer, a third modality adapter, and a fourth modality adapter; Processing the second input skeleton feature vector and the first learnable variable through the first cross attention layer to obtain a first cross feature vector, the first variable represents the latent space feature of the video modality, processing the second input video feature vector and the first cross feature vector through the second cross attention layer to obtain a seventh skeleton feature vector, and processing the seventh skeleton feature vector through the third modality adapter to obtain a second output skeleton feature vector; The second input video feature vector and the learnable second variable are processed by the third cross-attention layer to obtain a second cross-feature vector, wherein the second variable represents the latent space feature of the skeletal modality. The second input skeletal feature vector and the second cross-feature vector are processed by the fourth cross-attention layer to obtain a seventh video feature vector. The seventh video feature vector is processed by the fourth modality adapter to obtain a second output video feature.

7. The training method of the video skeleton action recognition model according to claim 6, characterized in that: The first modality adapter, the second modality adapter, the third modality adapter and the fourth modality adapter have the same structure, and any modality adapter includes a third activation function, a fifth fully connected layer for compressing feature dimensions, and a sixth fully connected layer for expanding feature dimensions; The second input feature vector is processed sequentially through the fifth fully connected layer, the third activation function and the sixth fully connected layer to obtain an intermediate feature vector, and the second input feature vector and the intermediate feature vector are added to obtain a second output feature vector.

8. The training method of the video skeleton action recognition model according to claim 1, characterized in that: The method further comprises: Obtain the video sequence to be tested and the skeleton sequence to be tested corresponding to the user during the movement; The video sequence to be tested and the skeleton sequence to be tested are processed using a trained video skeleton action recognition model to obtain the action category information of the user.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: The processor executes the computer program to implement the training method for the video skeleton action recognition model as described in any one of claims 1 to 8.

Images