A classroom complete meta-motion recognition method based on dynamic position embedding

Abstract

The application discloses a kind of based on dynamic position embedding classroom complete meta-motion recognition method, comprising: obtaining single student classroom meta-motion video, key frame sampling is carried out to video;Position coding module, Video Swin Transformer basic module, ViT3D basic module consisting of dynamic position embedding are constructed into classroom complete meta-motion recognition model, dynamic position feature representation, multiple local attention vectors, meta-motion class probability vector are obtained in turn, classroom complete meta-motion recognition model is iteratively optimized;Student classroom meta-motion video after pre-processing is input into the model to obtain classroom student meta-motion class.This method not only uses dynamic position embedding method to carry out conditional position coding combined with deep convolutional network to improve the analytical ability to meta-motion space structure, and adopts the whole attention mechanism of ViT3D to extract the potential space features of action, captures the global space-time information of meta-motion, to improve the student complete meta-motion recognition accuracy of classroom video.

Description

Technical Field

[0001] This invention relates to video understanding and analysis technology, particularly action recognition technology in video understanding, specifically a complete meta-action recognition method for classrooms based on dynamic position embedding. Background Technology

[0002] Offline classrooms are the primary venue for student learning and teacher instruction, and the quality of teaching directly impacts student learning outcomes. With the development of smart classrooms, acquiring student video data in the classroom is becoming increasingly easy. Therefore, video action recognition technology is introduced to identify student actions in the classroom, instantly sensing students' learning status and providing intelligent analysis reports reflecting classroom quality. The classroom action recognition task uses a sequence of student action video frames as input and outputs student action categories, finding wide application in scenarios such as classroom teaching, self-study management, and unsupervised examinations. For example, in an unsupervised examination environment, classroom action recognition technology can identify examinees' actions in real time; if suspected cheating occurs, the examinee can be investigated, thus ensuring examination discipline. Currently, the main challenges are: the varying amplitude and frequency of the same actions among different students increases the difficulty of recognition; and recognizing actions from multiple students requires significant computational overhead.

[0003] Currently, practical applications of action recognition in classroom scenarios are limited. Traditional action recognition technologies employ an end-to-end learning framework, automatically extracting feature representations that reflect the latent semantic information of video content. However, traditional methods require encoding video frames into handcrafted features (such as HOG3D and 3Dsurf features), which have significant limitations and are slow to extract, failing to meet real-time requirements. In recent years, action recognition methods based on Convolutional Neural Networks (CNNs) have emerged, enabling end-to-end learning of feature representations reflecting the latent semantic information of videos, thus significantly improving the accuracy of action recognition. To extract more effective visual features, Residual Networks (ResNets) use residual connections to link different layers, mitigating problems such as overfitting, vanishing gradients, or exploding gradients during the training of deeper neural network models. Non-Local Networks (NLONs) utilize non-local operations to capture long-range dependencies, establishing connections between pixel blocks at different distances in video frames through attention mechanisms to mine semantic information between them. In addition, the Transformer model, which originated from the field of natural language processing, has recently become popular in the field of computer vision. Its multi-head attention mechanism is used to extract diverse key temporal information in video frame sequences, enabling the model to learn more discriminative feature representations.

[0004] Existing technologies for complete classroom action recognition still have many shortcomings: First, calculating spatiotemporal attention across all video frames during feature extraction ignores the local characteristics of spatiotemporal features, limiting recognition rates and incurring excessive computational costs at high video resolutions. Second, many techniques extract spatiotemporal features from single-scale blocks, failing to adapt to the spatiotemporal relationships involved in actions. To address the lack of mechanisms for exchanging local spatiotemporal feature information, there is an urgent need for a method that can improve both the understanding of local spatial structures and the capture of global spatiotemporal feature information, thereby enhancing the accuracy of complete classroom action recognition for students. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a classroom-based complete meta-action recognition method based on dynamic position embedding. This method can improve the understanding of local spatial structure, capture global spatiotemporal feature information, and improve the accuracy of student classroom complete meta-action recognition.

[0006] The technical solution to achieve the objective of this invention is:

[0007] A complete meta-action recognition method for classrooms based on dynamic location embedding includes the following steps:

[0008] 1) First, acquire high-definition classroom student video data, preprocess the high-definition classroom student video data, and obtain individual student classroom meta-action videos;

[0009] 2) Sample the video of a single student’s classroom actions at a frame rate of 8 to 32 frames per video. The input is a sequence of student action video frames, and the output is key video frames of student actions.

[0010] 3) Construct a conditional location encoding module based on dynamic location embedding. The input is key video frames of student actions, and the output is a dynamic location feature representation, including:

[0011] 3-1) Dynamic position embedding is as shown in formula (1):

[0012] DPE(X) = DWConv(X) (1);

[0013] 3-2) In formula (1), DWConv indicates that zero padding is used during the convolution process, so that each element in the data can better understand its absolute position in the data by gradually exploring its neighbor information, thereby enhancing the model's ability to understand the spatial structure. X represents the input video frame.

[0014] 4) Construct the basic blocks of the Video Swin Transformer: The input is a dynamic positional feature representation, and the output is multiple local attention vectors, including:

[0015] 4-1) Video Swin Transformer introduces window attention into the Transformer encoder to achieve hierarchy, locality and transformation invariance. It combines the basic Transformer unit with powerful modeling capabilities and reduces computational complexity.

[0016] 4-2) The basic blocks of the Video Swin Transformer first undergo window self-attention 3D W-MSA calculation, then pass through the normalization layer LN, and perform residual connections. Then, feature filtering is performed on the MLP layer and the normalization layer. The attention calculation based on the sliding window is a shift window partitioning attention calculation method. Two partitioning configurations are used alternately in the continuous Video Swin Transformer modules. The principle is shown in formulas (2)-(5):

[0017]

[0018] In formulas (2)-(5), z l-1 This represents the basic block input of the Video Swin Transformer; z l and z l+1 It represents the features of the sliding window self-attention calculation and the output of the MLP (Multilayer Perceptron) layer. The attention calculation using the sliding window can associate the features of the previous and next frames; 3DW-MSA represents 3D window attention calculation; 3DSW-MSA represents 3D attention calculation with sliding window; LN represents normalization layer;

[0019] 5) Construct the ViT3D basic block: The input is all the local attention vectors output from step 4), and the output is the meta-action category probability vector, including:

[0020] 5-1) The ViT3D module is ViT (Visio Transformer, or ViT for short) which, after parameter inflation, stitches together a video or multiple images into a set of inputs, so that all small patches in the deep feature map of the model can be fully attentioned.

[0021] 5-2) After the Video Swin Transformer module outputs n feature maps, they are input into the Tansformer Encoder in the ViT3D module for full attention calculation, and the attention vector of each pixel block is extracted. The principle is as shown in formula (6):

[0022]

[0023] 5-3) In formula (6), Q is the Query matrix, representing the current query or question, usually related to the current processing position or element; K is the Key matrix, representing all possible keys in the dataset; V is the Value matrix, representing the value associated with each key; K T d represents the transpose of the K matrix; k The dimension of the key; the Softmax function transforms the input vector into a probability distribution, ensuring that the weights of all outputs sum to 1;

[0024] 5-4) The attention vector of each pixel block is classified through the MLP HEAD layer to obtain the recognized classroom meta-actions;

[0025] 6) Construct a complete meta-action recognition model for the classroom, consisting of a location encoding module with dynamic location embedding, a basic module of Video Swin Transformer, and a basic module of ViT3D. Train the model using the Video Swin Transformer framework. The total number of iterations is 10 to 1000. The AdamW optimizer is used for gradient descent optimization. The learning rate of the backbone network is set to 3e-5, and the batch size is set to 8 to 512.

[0026] 7) Perform human detection on the new classroom student meta-action videos to obtain individual student classroom meta-action videos, input them into the already trained complete classroom meta-action recognition model, and output the classroom student meta-action categories.

[0027] This technical solution is applied to student meta-action recognition in offline smart classrooms and has the following characteristics: 1) A classroom meta-action recognition network architecture combining VideoSwin Transformer and ViT3D. This architecture inputs the deep features extracted by the VideoSwin Transformer network into ViT3D for comprehensive attention calculation, which not only enhances the model's ability to capture action details but also improves its understanding of the overall structure of meta-actions; 2) Dynamic position embedding is used in position encoding, employing deep convolution technology and introducing zero padding during the convolution process. This allows each element in the data to better understand its absolute position in the data by gradually exploring its neighbor information, enhancing the model's understanding of the spatial structure of action videos and improving its ability to capture local spatiotemporal features of actions; 3) This technical solution has high effectiveness and reliability in various action recognition scenarios, providing a new technical means for the fields of smart education and behavior analysis.

[0028] This method can improve the understanding of local spatial structure, capture global spatiotemporal feature information, and improve the accuracy of students' complete meta-action recognition in the classroom. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the method used in this embodiment. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0031] Example:

[0032] Reference Figure 1 A complete meta-action recognition method for classrooms based on dynamic location embedding includes the following steps:

[0033] 1) First, acquire high-definition classroom student video data, preprocess the high-definition classroom student video data, and obtain individual student classroom meta-action videos;

[0034] 2) Sample the video of a single student’s classroom actions at a frame rate of 8 to 32 frames per video. The input is a sequence of student action video frames, and the output is key video frames of student actions.

[0035] 3) Construct a conditional location encoding module based on dynamic location embedding. The input is key video frames of student actions, and the output is a dynamic location feature representation, including:

[0036] 3-1) Dynamic position embedding is as shown in formula (1):

[0037] DPE(X) = DWConv(X) (1);

[0038] 3-2) In formula (1), zero padding is used in the DWConv convolution process, so that each element in the data can better understand its absolute position in the data by gradually exploring its neighbor information, thereby enhancing the model's ability to understand the spatial structure; X represents the input video frame;

[0039] 4) Construct the basic blocks of the Video Swin Transformer: the input is a dynamic positional feature representation, and the output is multiple local attention vectors, including:

[0040] 4-1) Video Swin Transformer introduces window attention into the Transformer encoder to achieve hierarchy, locality and transformation invariance. It combines the basic Transformer unit with powerful modeling capabilities and reduces computational complexity.

[0041] 4-2) The basic blocks of the Video Swin Transformer first undergo window self-attention 3D W-MSA calculation, then pass through the normalization layer LN, and perform residual connections. Then, feature filtering is performed on the MLP layer and the normalization layer. The attention calculation based on the sliding window is a shift window partitioning attention calculation method. Two partitioning configurations are used alternately in the continuous Video Swin Transformer modules. The principle is shown in formulas (2)-(5):

[0042]

[0043] In formulas (2)-(5), z l-1 This represents the basic block input of the Video Swin Transformer; z l and z l+1 It represents the features of the sliding window self-attention calculation and the output of the MLP (Multilayer Perceptron) layer. The attention calculation using the sliding window can associate the features of the previous and next frames; 3DW-MSA represents 3D window attention calculation; 3DSW-MSA represents 3D attention calculation with sliding window; LN represents normalization layer;

[0044] 5) Construct the ViT3D basic block: The input is all the local attention vectors output from step 4), and the output is the meta-action category probability vector, including:

[0045] 5-1) The ViT3D module is ViT with parameter dilation, which stitches a video or multiple images into a set of inputs, so that all small patches in the deep feature map of the model can be fully attentioned.

[0046] 5-2) After the Video Swin Transformer module outputs n feature maps, they are input into the Tansformer Encoder in the ViT3D module for full attention calculation, and the attention vector of each pixel block is extracted. The principle is as shown in formula (6):

[0047]

[0048] 5-3) In formula (6), Q is the Query matrix, representing the current query or question, usually related to the current processing position or element; K is the Key matrix, representing all possible keys in the dataset; V is the Value matrix, representing the value associated with each key; K T d represents the transpose of the K matrix; k The dimension of the key; the Softmax function transforms the input vector into a probability distribution, ensuring that the weights of all outputs sum to 1;

[0049] 5-4) The attention vector of each pixel block is classified through the MLP HEAD layer to obtain the recognized classroom meta-actions;

[0050] 6) Construct a complete meta-action recognition model for the classroom, consisting of a location encoding module with dynamic location embedding, a basic module of Video Swin Transformer, and a basic module of ViT3D. Train the model using the Video Swin Transformer framework. The total number of iterations is 10 to 1000. The AdamW optimizer is used for gradient descent optimization. The learning rate of the backbone network is set to 3e-5, and the batch size is set to 8 to 512.

[0051] 7) Perform human detection on the new classroom student meta-action videos to obtain individual student classroom meta-action videos, input them into the already trained complete classroom meta-action recognition model, and output the classroom student meta-action categories.

[0052] In this example, ViT3D is used for full attention computation, which effectively improves the understanding of the overall structure of meta-actions. Conditional position encoding with dynamic position embedding is adopted, combined with deep convolution technology, and zero padding is introduced during the convolution process. This allows each element in the data to better understand its absolute position in the data by gradually exploring its neighbor information, which enhances the model's understanding of the spatial structure of action videos and improves the model's ability to capture the local spatiotemporal features of students' classroom meta-actions. The method in this example can be applied to various sports scenarios and provides technical support for subsequent smart education and behavior analysis fields.

Claims

1. A classroom-based complete meta-action recognition method based on dynamic position embedding, characterized in that, Includes the following steps: 1) First, acquire high-definition classroom student video data, perform human body detection on the high-definition classroom student video data, and obtain individual student classroom meta-action videos; 2) Sample the video of a single student’s classroom actions at a frame rate of 8 to 32 frames per video. The input is a sequence of student action video frames, and the output is key video frames of student actions. 3) Construct a conditional location encoding module based on dynamic location embedding. The input is key video frames of student actions, and the output is a dynamic location feature representation, including: 3-1) Dynamic position embedding is as shown in formula (1): DPE(X) = DWConv(X) (1); 3-2) In formula (1), DWConv means that zero padding is used during the convolution process, so that each element in the data can understand its absolute position in the data by gradually exploring its neighbor information, and X represents the input video frame; 4) Construct the basic blocks of the Video Swin Transformer: The input is a dynamic positional feature representation, and the output is multiple local attention vectors, including: 4-1) Video Swin Transformer introduces window attention in the Transformer encoder to achieve hierarchy, locality, and transformation invariance; 4-2) The basic block of Video Swin Transformer first undergoes window self-attention 3D W-MSA calculation, then passes through the normalization layer LN, and performs residual connection. Then, it performs feature filtering of MLP layer and normalization layer. Two partition configurations are used alternately in the continuous Video Swin Transformer module, as shown in formula (2)-formula (5): In formulas (2)-(5), z l-1 This represents the input of the Video Swin Transformer basic block, z. l and z l+1 These are the features of sliding window self-attention calculation and multilayer perceptron (MLP) layer output. 3DW-MSA represents 3D window attention calculation, 3DSW-MSA represents 3D window attention calculation with sliding window, and LN represents normalization layer. 5) Construct the ViT3D basic block: The input is all the local attention vectors output from step 4), and the output is the meta-action category probability vector, including: 5-1) The ViT3D module is ViT with parameter dilation, which stitches a video or multiple images into a set of inputs, so that all small patches in the deep feature map of the model can be fully attentioned. 5-2) After the Video Swin Transformer module outputs n feature maps, they are input into the Tansformer Encoder in the ViT3D module for full attention calculation, and the attention vector of each pixel block is extracted. The principle is as shown in formula (6): 5-3) In formula (6), Q is the Query matrix, representing the current query or question, related to the current processing position or element; K is the Key matrix, representing all possible keys in the dataset; V is the Value matrix, representing the value associated with each key; K T Denotes the transpose of matrix K, d k Given the dimension of the key, the Softmax function transforms the input vector into a probability distribution, ensuring that the weights of all outputs sum to 1. 5-4) The attention vector of each pixel block is classified through the MLP HEAD layer to obtain the recognized classroom meta-actions; 6) Construct a complete meta-action recognition model for the classroom, consisting of a location encoding module with dynamic location embedding, a basic module of Video Swin Transformer, and a basic module of ViT3D. Train the model using the Video Swin Transformer framework. The total number of iterations is 10 to 1000. The AdamW optimizer is used for gradient descent optimization. The learning rate of the backbone network is set to 3e-5, and the batch size is set to 8 to 512. 7) Perform human detection on the new classroom student meta-action videos to obtain individual student classroom meta-action videos, input them into the already trained complete classroom meta-action recognition model, and output the classroom student meta-action categories.

Images