An interactive teaching behavior recognition method for classroom videos

Abstract

This invention discloses a method for recognizing interactive teaching behaviors in classroom videos. At the network structure level, it employs a spatiotemporal feature extraction module comprising spatial and temporal feature extraction branches. The spatial feature branch embeds a spatial reconstruction unit and uses a channel-dimensional weight adaptive mechanism to perform hierarchical filtering of feature maps, strengthening key spatial features while suppressing redundant information. The temporal feature branch constructs a temporal focusing unit and introduces a multi-head self-attention module to establish an inter-frame long-term correlation model, achieving dynamic temporal attention allocation to capture the temporal patterns of teaching interactions. The refined spatial features and long-term temporal features extracted by the two branches are complementaryly integrated through a lightweight fusion module to complete the recognition and classification of classroom teaching interaction states. This invention significantly enhances robustness to changes in classroom lighting and complex scenes, effectively reducing the cost of using traditional methods such as questionnaires and interviews for teaching behavior research.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence education technology, and in particular relates to a method for recognizing interactive teaching behaviors in classroom videos. Background Technology

[0002] Traditional classroom teaching scenarios typically include multimodal interactive elements: verbal communication between teachers and students, body movements, blackboard writing, and operation of teaching aids constitute a dynamic teaching semantic space. However, the application of existing visual behavior recognition technology in classroom teaching scenarios faces the following core challenges: (1) Complex environmental interference: The dense distribution of people, high-frequency changing interactive subjects, and multiple static background targets in the classroom environment make it difficult for traditional 3D convolutional models to effectively distinguish between the behavioral subjects and environmental noise, and they are easily affected by redundant features; (2) Defects in dynamic temporal modeling: When processing non-continuous interactive behaviors, traditional 3D convolutional networks lack the ability to model long-term dependencies, making it difficult to capture key frame association features distributed in the time dimension; (3) Computational efficiency bottleneck: The high parameter design of existing behavior recognition networks cannot meet the needs of real-time classroom analysis.

[0003] Traditional classroom settings typically involve a large number of teachers and students, along with environmental elements such as blackboards, desks, walls, and windows. These elements combine to create a highly complex and dynamically changing scene, posing a significant challenge to the identification of interactive teaching behaviors. Therefore, there is an urgent need to develop a method for identifying interactive teaching behaviors in classroom videos to effectively identify these behaviors within the complex classroom environment. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes an interactive teaching behavior recognition method for classroom videos. This method significantly enhances robustness to changes in lighting and complex scenarios in the classroom, providing technical support for digital classroom analysis and teaching effectiveness evaluation. It also effectively reduces the cost of using traditional methods such as questionnaires and interviews to study teaching behavior.

[0005] To achieve the above objectives, the present invention provides a method for recognizing interactive teaching behaviors in classroom videos, comprising:

[0006] S1. Construct a classroom interaction behavior recognition model;

[0007] S2. Obtain video datasets of classroom interactive teaching behaviors;

[0008] S3. Process the video dataset of the classroom interactive teaching behavior to obtain a processed video dataset, and divide the processed video dataset into a training set and a validation set in a 4:1 ratio.

[0009] S4. Perform shallow feature extraction on the training set to obtain shallow features of the video data;

[0010] S5. Extract the temporal and spatial features from the shallow features of the video data to obtain the temporal and spatial features of the video data;

[0011] S6. Perform lightweight fusion processing on the temporal and spatial features of the video data to obtain the deep spatiotemporal features of the video data;

[0012] S7. The deep spatiotemporal features of the video data are used to identify classroom interactive behaviors through a behavior classifier;

[0013] S8. Set the loss function, train the classroom interaction behavior recognition model, and obtain the training results;

[0014] S9. Validate the training results using the validation set and record the model parameters with the best performance.

[0015] S10. Repeat S4-S9 until the set number of iterations is reached to obtain the optimal classroom interaction behavior recognition model.

[0016] S11. Obtain the video dataset of classroom interactive teaching behaviors to be tested, and input it into the optimal classroom interactive behavior recognition model to perform interactive teaching behavior recognition.

[0017] Optionally, shallow feature extraction is performed on the processed video dataset to obtain shallow features of the video data, including: inputting the input 5D tensor into a first sequence consisting of a 7×7×7 separable convolutional layer, a max pooling layer, and a 1×1×1 basic convolutional layer.

[0018] The first sequence is input into the second sequence, which consists of a 3×3×3 separable convolutional layer and a max pooling layer.

[0019] Optionally, in the process of extracting spatiotemporal features from the shallow features of the video data, the temporal feature extraction branch includes: downsampling the input features in the spatial dimension and generating a query matrix Q, a key matrix K, and a value matrix V;

[0020] Q, K, and V are split into h attention heads and time-dimensional self-attention is computed in parallel.

[0021] The spatial dimension is compressed by global average pooling, and a temporal weight vector A is generated.

[0022] Dynamic weighting of temporal features is achieved by multiplying A element-by-element with the input feature map.

[0023] Optionally, the computation of self-attention is split into 8 independent attention heads, and the attention output is restored to the original dimension after being concatenated with cross-head features.

[0024] Optionally, in the process of extracting spatiotemporal features from the shallow features of the video data, the spatial feature extraction branch includes: a sequentially connected 1×1×1 basic convolutional layer, a 1×3×3 basic spatial convolutional layer, a spatial reconstruction unit, and a 3×1×1 basic temporal convolutional layer. The spatial reconstruction unit suppresses spatial redundancy information through channel dimension weight separation and cross-reconstruction strategies.

[0025] Optionally, the implementation process of the spatial reconstruction unit includes:

[0026] The input features are group normalized, and the channel weights are calculated based on the trainable scaling factor.

[0027] The channel weights are decomposed into high-weight feature maps and low-weight feature maps using a threshold gating strategy.

[0028] The original feature map is multiplied by the high-weight feature map and the low-weight feature map respectively to generate information-rich features and information-redundant features;

[0029] Perform cross-channel segmentation and feature summation operations on the information-rich features and the information-redundant features to generate a first fusion feature and a second fusion feature;

[0030] The first fusion feature and the second fusion feature are concatenated along the channel dimension to generate the reconstructed output feature.

[0031] Optionally, the temporal and spatial features of the video data are subjected to lightweight fusion processing to obtain deep spatiotemporal features of the video data, including:

[0032] The first feature processing branch performs local feature dimensionality reduction on the temporal and spatial features of the video data to obtain a first result.

[0033] The second feature processing branch performs spatiotemporal joint feature extraction on the temporal and spatial features of the video data to obtain a second result.

[0034] The video data is downsampled and compressed using the third feature processing branch to obtain a third result.

[0035] The first result, the second result, and the third result are merged according to the channel dimension to generate deep spatiotemporal features of the video data.

[0036] Optionally, the deep spatiotemporal features of the video data are used to identify classroom interaction behaviors through a behavior classifier, which includes: performing a global average pooling operation on the input feature map to compress the feature tensor into a pooling vector;

[0037] The pooling vector is input into the first fully connected layer, and after ReLU nonlinear activation, a hidden layer feature vector is generated.

[0038] The hidden layer features are mapped to a classification space with a preset number of categories through a second fully connected layer;

[0039] The Softmax function is used to generate probability distributions of behavior categories to achieve interactive behavior recognition.

[0040] Technical Effects of this Invention: This invention discloses an interactive teaching behavior recognition method for classroom videos. It identifies interactive teaching behaviors in classroom videos, effectively improving recognition accuracy with low computational complexity and minimal memory overhead. Through a spatiotemporal feature extraction module and a lightweight fusion module, this invention ensures efficient use of computational resources while fully exploring the spatiotemporal contextual relationships within the teaching scenario. In the shallow network stage, the multi-branch collaborative spatiotemporal feature extraction module captures details of local body movements such as students raising their hands and teachers writing on the blackboard, and utilizes the multi-head self-attention mechanism in the temporal focusing unit to establish long-term temporal relationships for intermittent teaching events. In the deep network stage, the lightweight fusion module, through structural reparameterization technology, compresses the feature interaction process of spatial and temporal flows into an equivalent single-path computation, while preserving key interactive semantics in the global features. In summary, this invention significantly enhances robustness to changes in lighting and complex scenarios in the classroom, providing technical support for digital classroom analysis and teaching effectiveness evaluation, and effectively reducing the cost of using traditional methods such as questionnaires and interviews for teaching behavior research. Attached Figure Description

[0041] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0042] Figure 1 This is a flowchart illustrating an interactive teaching behavior recognition method for classroom videos according to an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0045] This embodiment provides a method for recognizing interactive teaching behaviors in classroom videos, including:

[0046] S1. Construct a classroom interaction behavior recognition model;

[0047] S2. Obtain video datasets of classroom interactive teaching behaviors;

[0048] S3. Process the video dataset of the classroom interactive teaching behavior to obtain a processed video dataset, and divide the processed video dataset into a training set and a validation set in a 4:1 ratio.

[0049] S4. Perform shallow feature extraction on the training set to obtain shallow features of the video data;

[0050] S5. Extract the temporal and spatial features from the shallow features of the video data to obtain the temporal and spatial features of the video data;

[0051] S6. Perform lightweight fusion processing on the temporal and spatial features of the video data to obtain the deep spatiotemporal features of the video data;

[0052] S7. The deep spatiotemporal features of the video data are used to identify classroom interactive behaviors through a behavior classifier;

[0053] S8. Set the loss function, train the classroom interaction behavior recognition model, and obtain the training results;

[0054] S9. Validate the training results using the validation set and record the model parameters with the best performance.

[0055] S10. Repeat S4-S9 until the set number of iterations is reached to obtain the optimal classroom interaction behavior recognition model.

[0056] S11. Obtain the video dataset of classroom interactive teaching behaviors to be tested, and input it into the optimal classroom interactive behavior recognition model to perform interactive teaching behavior recognition.

[0057] Furthermore, shallow feature extraction is performed on the processed video dataset to obtain shallow features of the video data, including: inputting the input 5D tensor into a first sequence consisting of a 7×7×7 separable convolutional layer, a max pooling layer, and a 1×1×1 basic convolutional layer.

[0058] The first sequence is input into the second sequence, which consists of a 3×3×3 separable convolutional layer and a max pooling layer.

[0059] Furthermore, in the process of extracting spatiotemporal features from the shallow features of the video data, the temporal feature extraction branch includes: downsampling the input features in the spatial dimension and generating a query matrix Q, a key matrix K, and a value matrix V;

[0060] Q, K, and V are split into h attention heads and time-dimensional self-attention is computed in parallel.

[0061] The spatial dimension is compressed by global average pooling, and a temporal weight vector A is generated.

[0062] Dynamic weighting of temporal features is achieved by multiplying A element-by-element with the input feature map.

[0063] Furthermore, the self-attention computation is split into 8 independent attention heads, and the attention output is restored to the original dimension after being concatenated with cross-head features.

[0064] Furthermore, in the process of extracting spatiotemporal features from the shallow features of the video data, the spatial feature extraction branch includes: a sequentially connected 1×1×1 basic convolutional layer, a 1×3×3 basic spatial convolutional layer, a spatial reconstruction unit, and a 3×1×1 basic temporal convolutional layer. The spatial reconstruction unit suppresses spatial redundancy information through channel dimension weight separation and cross-reconstruction strategies.

[0065] Furthermore, the implementation process of the spatial reconstruction unit includes:

[0066] The input features are group normalized, and the channel weights are calculated based on the trainable scaling factor.

[0067] The channel weights are decomposed into high-weight feature maps and low-weight feature maps using a threshold gating strategy.

[0068] The original feature map is multiplied by the high-weight feature map and the low-weight feature map respectively to generate information-rich features and information-redundant features;

[0069] Perform cross-channel segmentation and feature summation operations on the information-rich features and the information-redundant features to generate a first fusion feature and a second fusion feature;

[0070] The first fusion feature and the second fusion feature are concatenated along the channel dimension to generate the reconstructed output feature.

[0071] Furthermore, the temporal and spatial features of the video data are subjected to lightweight fusion processing to obtain deep spatiotemporal features of the video data, including:

[0072] The first feature processing branch performs local feature dimensionality reduction on the temporal and spatial features of the video data to obtain a first result.

[0073] The second feature processing branch performs spatiotemporal joint feature extraction on the temporal and spatial features of the video data to obtain a second result.

[0074] The video data is downsampled and compressed using the third feature processing branch to obtain a third result.

[0075] The first result, the second result, and the third result are merged according to the channel dimension to generate deep spatiotemporal features of the video data.

[0076] Furthermore, the deep spatiotemporal features of the video data are used to identify classroom interaction behaviors through a behavior classifier, which includes: performing a global average pooling operation on the input feature map to compress the feature tensor into a pooling vector;

[0077] The pooling vector is input into the first fully connected layer, and after ReLU nonlinear activation, a hidden layer feature vector is generated.

[0078] The hidden layer features are mapped to a classification space with a preset number of categories through a second fully connected layer;

[0079] The Softmax function is used to generate probability distributions of behavior categories to achieve interactive behavior recognition.

[0080] Specifically, such as Figure 1 As shown, the implementation process of this embodiment includes the following steps:

[0081] S1: Construct a classroom interaction behavior recognition network based on 3D convolution;

[0082] S2: Obtain a video training dataset of classroom interactive teaching behaviors;

[0083] S3: Preprocessing and data augmentation of the dataset;

[0084] S4: The shallow feature extraction module performs preliminary processing on the video data and extracts shallow features, as follows:

[0085] For a given instructional video input I, a shallow feature extraction module H is used, consisting of a 7×7×7 separating convolutional layer, a 3×3×3 max pooling layer, a 1×1×1 basic convolutional layer, a 3×3×3 separating convolutional layer, and a 3×3×3 max pooling layer. SFE Preliminary feature extraction and dimensionality reduction are performed on the input data to generate shallow features F′. The process is as follows:

[0086] F′=H SFE (I);

[0087] S5: Input the shallow features into the spatiotemporal feature extraction module to extract the temporal and spatial features of the video. The spatiotemporal feature extraction module includes spatial feature extraction branches and temporal feature extraction branches, as detailed below:

[0088] First branch H b1The input features are reduced in dimensionality using a 1×1×1 basic convolutional layer, extracting low-dimensional local features while preserving the main spatial and temporal information. The second branch H... b2 The main task is to extract spatial information. Specifically, this branch first uses a 1×1×1 basic convolutional layer for dimensionality reduction, then uses a 1×3×3 basic spatial convolutional layer to extract features in the spatial dimension using a separable convolution method. It captures local spatial information in a single frame of the image frame by frame, and uses a spatial reconstruction unit to separate and reconstruct the feature map space, thereby suppressing redundant information in space and enhancing the feature representation of key regions. Finally, a 3×1×1 basic temporal convolutional layer is used to integrate the spatial information and capture local short-term actions. The third branch, H... b3 The focus is on capturing contextual information in the temporal dimension. First, a 1×1×1 basic convolutional layer is used for dimensionality reduction. Then, a 3×3×3 separable convolutional layer is used to extract complex spatiotemporal features. Finally, a temporal focusing unit is used to dynamically adjust the feature weights of keyframes, thereby capturing long-duration action sequences in the video. The fourth branch, H... b4 The feature map is downsampled using max pooling to retain the most salient features, and then further reduced in dimensionality using a 1×1×1 convolutional layer. Finally, the features extracted from all branches are concatenated, capturing spatiotemporal information at different scales and levels, enhancing the model's ability to handle complex dynamic scenes while maintaining reasonable computational complexity. The process can be represented as follows:

[0089] F b1 =H b1 (F′1);

[0090] F b2 =H b2 (F′2);

[0091] F b3 =H b3 (F′3);

[0092] F b4 =H b4 (F′4);

[0093] F b =Concat{F b1 ,F b2 ,F b3 ,F b4};

[0094] a) Spatial reconstruction unit:

[0095] i. Input X to the feature map of the upper layer iThe trainable scaling factor γ in the group-normalized GN is used to evaluate the information richness of different feature maps, thus separating information-rich feature maps from information-poor feature maps:

[0096]

[0097] ii. Normalize the parameters of γ to obtain the relevant weights W γ It can effectively characterize the importance of different feature maps:

[0098]

[0099] iii. Using W γ Xg is reweighted and mapped to the range (0, 1) using the Sigmoid function, then gating is performed using a threshold (set to 0.5 in the experiment). Weights above the threshold are assigned α (set to 0.8 in the experiment) and are considered information-rich weights W1; weights below the threshold are assigned (1-α) and are considered information-poor weights W2. The specific formula is as follows:

[0100] W = Gate(σ(W) γ ×GN(X g )));

[0101] iv. Input feature X i Multiplying X1 and W2 respectively generates a weighted, information-rich feature X1 and an information-redundant feature X2. After the separation operation, a reconstruction operation is performed. Specifically, SRU uses a cross-reconstruction method to segment X1 and X2 into X1 and X2 along the channel dimension. 11 X 12 X 21 and X 22 Then, X 11 and X 22 Adding them together gives Y1 and X 21 and X 12 The sums are Y1 and Y2. Y1 and Y2 are then concatenated along the channel dimension to obtain the output feature Y.

[0102] b) Timing-focused unit:

[0103] i. Input X to the feature map of the upper layer i The feature representations of each video frame are uniformly distributed after group normalization, and the spatial information of each frame is linearly combined through a 1×1 convolution operation:

[0104] Q = XW Q K = XW K V = XW V ;

[0105] Among them W Q W K and W V This is the learning matrix used to generate QKV. Then, the dot product of Q and K (Softmax) is calculated to obtain their similarity, and this similarity is weighted and summed with V to obtain the output O. i :

[0106] O i =Attention(Q,K,V)=Softmax(Q·K)·V;

[0107] ii. Further, Q, K, and V are split into 8 heads using a multi-head self-attention (MHSA) mechanism. Attention A is computed in parallel by multiple heads to capture temporal dependencies at different scales:

[0108] A=σ(GAP(MHSA(Q,K,V)))∈R BC×T×1×1 ;

[0109] iii. The generated temporal attention A is applied to the feature map X. i The weighted operation is calculated using the following formula:

[0110] X A =A·X i ;

[0111] S6: Input the spatial and temporal features of the video into the lightweight fusion module to extract the deep spatiotemporal features of the video, as follows:

[0112] First branch F l1 and the third branch F l3 The processing ideas and methods of the first and fourth branches of the spatiotemporal feature extraction module are the same, and the second branch F l2 This is a lightweight merged branch of the second and third branches of the spatiotemporal feature extraction module. Specifically, this branch uses a 1×1×1 base convolutional layer to reduce the dimensionality of the input features, and then uses a 3×3×3 separating convolutional layer to directly extract the spatiotemporal information of the feature map. Finally, the three branches are concatenated, which can simultaneously capture local details and global contextual information. The process is represented as follows:

[0113] F l1 =H l1 (F b1 );

[0114] F l2 =H l2 (F b2 );

[0115] F l3 =Hl3 (F b3 );

[0116] F l =Concat{F l1 ,F l2 F l3};

[0117] S7: Using a behavior classifier to identify classroom interaction behaviors based on deep spatiotemporal features. Specifically:

[0118] This module first uses a global average pooling layer to process the 3D feature map F. l The feature vector is compressed into a one-dimensional vector, and then two fully connected layers are used to map the one-dimensional feature vector to the teaching behavior category space. During this process, the ReLU activation function is introduced to enhance its non-linear expressive power. Finally, the Softmax function is applied to generate the behavior category probability distribution to realize interactive behavior recognition Y.

[0119] S8: Set the loss function and train the classroom interaction behavior recognition network. Specifically, the loss function is the cross-entropy loss function:

[0120]

[0121] S9: Validate the network training results using the validation dataset and record the parameters of the network model with the best performance.

[0122] S10: Repeat S4-S9 until the set number of iterations is reached to obtain the network model parameters with optimal performance.

[0123] S11: Use the best-performing network model to identify classroom interaction behaviors.

[0124] This invention discloses an interactive teaching behavior recognition method for classroom videos. It identifies interactive teaching behaviors in classroom videos, effectively improving recognition accuracy with low computational complexity and minimal memory overhead. Through a spatiotemporal feature extraction module and a lightweight fusion module, this invention ensures efficient use of computational resources while fully exploring the spatiotemporal contextual relationships within the teaching scenario. In the shallow network stage, the multi-branch collaborative spatiotemporal feature extraction module captures details of local body movements such as students raising their hands and teachers writing on the blackboard, and utilizes the multi-head self-attention mechanism in the temporal focusing unit to establish long-term temporal relationships for intermittent teaching events. In the deep network stage, the lightweight fusion module uses structural reparameterization technology to compress the feature interaction process of spatial and temporal flows into an equivalent single-path computation, while preserving key interactive semantics in the global features. In summary, this invention significantly enhances robustness to changes in lighting and complex scenarios in the classroom, providing technical support for digital classroom analysis and teaching effectiveness evaluation, and effectively reducing the cost of traditional methods such as questionnaires and interviews for researching teaching behaviors.

[0125] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An interactive teaching behavior recognition method for classroom videos, characterized in that, include: S1. Construct a classroom interaction behavior recognition model; S2. Obtain video datasets of classroom interactive teaching behaviors; S3. Process the video dataset of the classroom interactive teaching behavior to obtain a processed video dataset, and divide the processed video dataset into a training set and a validation set in a 4:1 ratio. S4. Perform shallow feature extraction on the training set to obtain shallow features of the video data; S5. Extract the temporal and spatial features from the shallow features of the video data to obtain the temporal and spatial features of the video data; S6. Perform lightweight fusion processing on the temporal and spatial features of the video data to obtain the deep spatiotemporal features of the video data; S7. The deep spatiotemporal features of the video data are used to identify classroom interactive behaviors through a behavior classifier; S8. Set the loss function, train the classroom interaction behavior recognition model, and obtain the training results; S9. Validate the training results using the validation set and record the model parameters with the best performance. S10. Repeat S4-S9 until the set number of iterations is reached to obtain the optimal classroom interaction behavior recognition model. S11. Obtain the video dataset of classroom interactive teaching behaviors to be tested, and input it into the optimal classroom interactive behavior recognition model to recognize the interactive teaching behaviors; During the process of extracting spatiotemporal features from the shallow features of the video data, the spatial feature extraction branch includes: a sequentially connected 1×1×1 basic convolutional layer, a 1×3×3 basic spatial convolutional layer, a spatial reconstruction unit, and a 3×1×1 basic temporal convolutional layer. The spatial reconstruction unit suppresses spatial redundancy information through channel dimension weight separation and cross-reconstruction strategies. The implementation process of the spatial reconstruction unit includes: The input features are group normalized, and the channel weights are calculated based on the trainable scaling factor. The channel weights are decomposed into high-weight feature maps and low-weight feature maps using a threshold gating strategy. The original feature map is multiplied by the high-weight feature map and the low-weight feature map respectively to generate information-rich features and information-redundant features; Perform cross-channel segmentation and feature summation operations on the information-rich features and the information-redundant features to generate a first fusion feature and a second fusion feature; The first fusion feature and the second fusion feature are concatenated along the channel dimension to generate the reconstructed output feature; The temporal and spatial features of the video data are subjected to lightweight fusion processing to obtain deep spatiotemporal features of the video data, including: The first feature processing branch performs local feature dimensionality reduction on the temporal and spatial features of the video data to obtain a first result. The second feature processing branch performs spatiotemporal joint feature extraction on the temporal and spatial features of the video data to obtain a second result. The video data is downsampled and compressed using the third feature processing branch to obtain a third result. The first result, the second result, and the third result are merged according to the channel dimension to generate deep spatiotemporal features of the video data.

2. The method for interactive teaching behavior recognition of classroom video of claim 1, wherein, The shallow feature extraction of the processed video dataset is performed to obtain the shallow features of the video data, which includes: inputting the input 5D tensor into a first sequence consisting of a 7×7×7 separable convolutional layer, a max pooling layer, and a 1×1×1 basic convolutional layer. The first sequence is input into the second sequence, which consists of a 3×3×3 separable convolutional layer and a max pooling layer.

3. The interactive teaching behavior recognition method for classroom videos as described in claim 1, characterized in that, In the spatio-temporal feature extraction process of the shallow layer features of the video data, the time feature extraction branch comprises: performing spatial dimension down-sampling on the input features and generating a query matrix Q , a key matrix K and a value matrix V ; Will Q , K , V Split into h Each attention head computes self-attention in parallel over time. Global average pooling is used to compress the spatial dimension and generate a time weight vector. A ; Will A Dynamic weighting of temporal features is achieved by multiplying the input feature map element by element.

4. The interactive teaching behavior recognition method for classroom videos as described in claim 3, characterized in that, The computation of self-attention is split into 8 independent attention heads, and the attention output is restored to the original dimension after being concatenated with cross-head features.

5. The interactive teaching behavior recognition method for classroom videos as described in claim 1, characterized in that, Using a behavior classifier to identify classroom interaction behavior by applying the deep spatiotemporal features of the video data includes: performing a global average pooling operation on the input feature map to compress the feature tensor into a pooling vector; The pooling vector is input into the first fully connected layer, and after ReLU nonlinear activation, a hidden layer feature vector is generated. The hidden layer features are mapped to a classification space with a preset number of categories through a second fully connected layer; The Softmax function is used to generate probability distributions of behavior categories to achieve interactive behavior recognition.

Images