A teacher teaching intention detection method and system based on an interleaved attention mechanism

Abstract

The present application takes the teacher's hand gesture and upper limb movement as the research object, and discloses a teacher teaching intention detection method and system based on staggered attention for the auxiliary role of computer vision technology in the teaching scene. The method comprises the following steps: 1) respectively acquiring the teacher's hand gesture and upper limb movement video stream under the RGB-D camera and 3D structured light camera in the teaching environment; 2) respectively pre-processing the image data stream. The collected video stream is converted into teacher gesture, teacher upper limb movement key point heat map data stream and teacher gesture, teacher upper limb movement structured light image data stream; 3) the pre-processed data stream is respectively input into the teacher hand gesture recognition model and the upper limb movement recognition model. 4) after model prediction, the teacher hand gesture recognition result and the teacher upper limb movement recognition result are respectively output. The teacher gesture and the upper limb movement recognition result are combined to judge the teaching gesture. The present application uses a teacher teaching gesture estimation model based on a staggered attention mechanism, so that the network can recognize 3D teaching gestures according to the key point heat map when predicting the teacher gesture. At the same time, by combining the teacher upper limb movement recognition result with the gesture recognition result, the detection of the teacher's teaching intention is realized, and reference support is provided for the non-verbal ability evaluation of the teacher's teaching process.

Description

Technical Field

[0001] This application relates to the field of pattern recognition technology, and more specifically, to a method and system for detecting teacher instructional intentions based on interleaved attention. Background Technology

[0002] Sign language is an indispensable teaching aid; appropriate hand gestures can guide students to improve their attention and enhance their classroom engagement. Utilizing hand gesture estimation for intelligent measurement and evaluation of teacher teaching quality can provide teachers and educational administrators with efficient feedback, promoting the overall improvement of teachers' abilities. Therefore, detecting teachers' hand gestures to reflect their teaching intentions can serve as an effective technical means of assessing teaching quality.

[0003] Gesture recognition technology initially relied on wearable devices like data gloves to acquire hand data, enabling accurate gesture recognition. However, the high cost of this approach led researchers to shift their focus to optical tagging methods for acquiring gesture information. Subsequently, to explore the broader applications of natural hand gesture detection and recognition, researchers gradually turned their attention to wireless signal-based and vision-based gesture recognition methods. Current vision-based hand pose recognition technologies for educational environments estimate hand posture by extracting hand joint features; however, factors such as lighting conditions, complex backgrounds, and mutual occlusion between the left and right hands generally result in low prediction accuracy and poor generalization ability. Therefore, researchers are using multi-angle or depth cameras to reconstruct 3D gestures and employing multi-feature fusion to achieve accurate capture of key hand points.

[0004] Although gesture recognition has achieved significant results and been implemented in digital teaching, teacher evaluation in the teaching environment still largely relies on traditional manual assessment. This method is resource-intensive and inefficient. Therefore, it is urgent to address the issue of using gesture recognition in conjunction with teacher evaluation. Summary of the Invention

[0005] To address at least one deficiency or improvement need in existing technologies, this invention provides a method and system for detecting teacher instructional intentions based on an interleaved attention mechanism. Its aim is to achieve high gesture recognition accuracy and generalization ability even when there is significant occlusion during left-right hand interaction in a teaching environment.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method and system for detecting teacher instructional intentions based on interleaved attention are provided, comprising the following steps:

[0007] The teacher's hand pose RGB video stream, teacher's upper limb movement RGB video stream, teacher's hand pose structured light image video stream, and teacher's upper limb movement structured light image video stream were acquired from the RGB-D camera and the 3D structured light camera in the teaching scene, respectively.

[0008] The video streams collected in the teaching scene were preprocessed to obtain single-channel key point heatmaps of teacher's hand posture, single-channel key point heatmaps of teacher's upper limb movements, structured light images of teacher's hand posture, and structured light images of teacher's upper limb movements. The images were then divided into training set and test set according to the proportions.

[0009] The preprocessed single-channel keypoint heatmap gesture splicing vector of the teacher's gestures and the structured light feature vector of the teacher's hand posture are fused together and then input into a pre-trained teaching gesture estimation model to obtain the teacher's gesture recognition result. Similarly, the RGB feature vector of the teacher's upper limb movements and the structured light feature vector of the teacher's upper limb movements are fused together and then input into a pre-trained upper limb movement convolutional neural network model to obtain the teacher's upper limb movement recognition result.

[0010] The teacher's gesture recognition results and upper limb movement recognition results are combined to finally determine the type of teacher's teaching gestures;

[0011] The teaching gesture recognition model includes a feature fusion module, a non-sequential Transformer module, an interleaved attention layer, and a fully connected layer. The feature fusion module includes a U-Net network to extract and reduce the dimensionality of the teacher's gesture image appearance vector captured by an RGB-D camera, and then combines it with the location-encoded keypoint heatmap features to form a gesture concatenation vector. The feature fusion module fuses the gesture concatenation vector and the gesture structured light image vector to output a gesture fusion vector. The non-sequential Transformer takes the concatenation vector (formed from the teacher's gesture appearance vector and single-channel keypoint heatmap vector) and the gesture fusion vector (formed from the gesture structured light vector) as input to extract the joint features of the teacher's hand in the heatmap and performs gesture recognition on the extracted features. The non-sequential self-attention layer in the non-sequential Transformer module avoids inference dependencies on the estimated gesture from the reference extractor. The interleaved attention layer predicts the value of each joint to predict hand posture. The fully connected layer outputs the final, refined gesture recognition result.

[0012] According to the above scheme, the process of extracting and fusing teacher gesture features is as follows:

[0013] Step 1.1.1: Feature extraction from the input data. The U-Net network is used to reduce the dimensionality of the hand pose appearance vector in the RGB-D data stream. Specifically, two 3D network vectors are used. A 3x3 convolution kernel is used for several unpadded convolutions, reducing the number of pixels by two each time. Then, several convolutions with a 2x2 kernel are performed. The deconvolution operation (2) stitches together the cropped images from the convolution operation. The weights for this process are:

[0014] , (1)

[0015] Dimensionality reduction process yields gestures and appearance feature vectors. = .

[0016] Step 1.1.2 involves positional encoding of the heatmap vectors, including positional feature encoding and learnable embeddings. The learned embedding variables can be represented as: .in, This represents the learnable positional embedding. The learnable positional embedding of the next higher level is represented as: Each single-channel keypoint heatmap feature vector in the spatial encoder is processed by a matrix. , , The purpose is to obtain a linear mapping. , , .in, c represents the number of attention heads in a multi-head attention layer. This indicates the number of attention heads, and LN represents layer normalization. The output keypoint heatmap is used for location feature encoding. The output keypoint heatmap location encoding vector is represented as follows: .

[0017] Step 1.1.3 involves preprocessing the 3D structured light image data stream. The input structured light image is subjected to grayscale averaging to obtain the grayscale values ​​of 64 frames of pixel images. The grayscale averaging process is as follows: .in, This represents the maximum grayscale value in the image. This represents the minimum grayscale value in the image. This represents the learning parameters for mean-based averaging.

[0018] Step 1.1.4: The 3D coordinates of the hand joint are obtained through projection transformation and normalization using a structured light camera. The relationship between the 2D planar coordinates and the 3D coordinates is calculated as follows:

[0019] , (2)

[0020] in, , These represent the camera's focal length, , , Represents 3D coordinates. , This represents 2D coordinates. The formula for obtaining 3D coordinates is as follows:

[0021] (3)

[0022] in, This represents the learning parameters.

[0023] Step 1.1.5: Fuse the input RGB-D image vectors. The gesture splicing vector is formed by combining the appearance feature vector of the teacher's hand gesture and the spatially encoded keypoint heatmap vector: The gesture concatenation vector is fused with the 3D coordinates obtained from the structured light image feature extraction. This can be represented as: .in, This represents the balance parameter for feature fusion.

[0024] According to the above scheme, the training of the teaching gesture recognition model is as follows:

[0025] Step 1.2.1: Input the gesture fusion vector as the input vector into the non-sequential Transformer model. The pre-estimation results of the gesture reference extractor provide reference information for the prediction.

[0026] Step 1.2.2: Calculate the difference between the predicted keypoint heatmap output by the gesture recognition model and the labeled keypoint heatmap, using this difference as the loss value of the gesture recognition model. For the predicted keypoint heatmap in the image, labels are obtained using the keypoint data. The loss value for the corresponding photo is calculated, and the network weights are updated using the total loss value of the dataset. For each pixel in the keypoint heatmap, the unique coordinates of the pixel under correct labeling can be obtained. The coordinates of the current predicted key point heatmap ( The formula for calculating the loss value of this pixel is as follows:

[0027] = , (4)

[0028] Set threshold =3 pixels, the pixel coordinates of the keypoint heatmap satisfy... If so, then the key point is classified as a nearby joint. Then the key point and the background are grouped together.

[0029] Step 1.2.3 uses a single interleaved attention layer with learned joint queries to predict the relationship between keypoints and joints. The aim is to improve prediction accuracy by gaining a forward-backward perception of the relationship between keypoints and joints. The calculation can be expressed as follows:

[0030] , (5)

[0031] Among them, learning joint query , They represent 256 respectively A learning matrix of 256.

[0032] Step 1.2.4, the loss function for the actual hand joint coordinates and the estimated hand joint coordinates during model training is expressed as follows:

[0033] , (6)

[0034] in This represents the true value of the hand joint coordinates. Let J represent the predicted coordinates of the hand joints, and J represent the number of samples. The loss function for calculating the true keypoints and hand joints is as follows:

[0035] , (7)

[0036] Where KL represents the KL divergence loss. , These represent the predicted and actual hand joint keypoints, respectively; the loss functions for the actual and estimated hand joint coordinates in the fully connected layer are as follows:

[0037] , (8)

[0038] in This represents the actual coordinates of the hand joints in the fully connected layer.

[0039] Step 1.2.5, during the model training process, the end-to-end learning process of the model is supervised by minimizing the loss function, as shown below:

[0040] , (9)

[0041] According to the above scheme, the gesture recognition results obtained by the fully connected layer are divided into four categories: figurative gestures, speech aid gestures, basic gestures, and non-essential gestures.

[0042] The non-sequential Transformer comprises a multi-head self-attention layer, joint feature union, a non-sequential self-attention layer, and a gesture reference extractor. The multi-head self-attention layer models the relationships between keypoints and creates global context-aware features for each keypoint. The joint union encodes keypoint identity information along with localized image features. The non-sequential self-attention layer avoids inference dependencies on the gesture estimation by the reference extractor; the gesture reference extractor exposes a reference pose to the decoder.

[0043] According to the above scheme, the training process of the non-sequential Transformer model is as follows:

[0044] Step 1.3.1: Input the gesture fusion vector obtained from feature extraction and feature fusion into the multi-head self-attention mechanism and the multilayer perceptron, and then output the vector. The attention mechanism adaptively aggregates a set of input values ​​based on attention weights that consider the compatibility of a given query with a set of keys. Let the dimension of each query and key be... The dimension of the value is In the multi-head self-attention layer, attention weights are calculated on the L, C, and H matrices, and the multi-head attention mapping for hand joint features is as follows:

[0045] , (10)

[0046] in, , , In a multilayer perceptron, the signal propagates unidirectionally from the input layer to the output layer. ,in .

[0047] Step 1.3.2, the non-sequential structured decoding process eliminates the sequential dependency on previously generated tags and converts the input... The modified copy is directly passed to the decoder to obtain the output sequence. The decoding process is as follows:

[0048] , (11)

[0049] in These are model parameters.

[0050] Step 1.3.3: Input the normalized 3D key point information into the structured reference to generate a reference hand pose in the form of hand joint features. By passing a reference hand pose to the decoder, the decoder can capture the correlation between reference joints as an approximation of the target hand pose and use this information to better constrain the output space.

[0051] The upper limb extraction module includes a backbone network, feature extraction, and fusion.

[0052] According to the above scheme, the steps for upper limb movement extraction are as follows:

[0053] Step 2.1.1: 3D structured light video sequence of the teacher's gestures. As an input feature vector, K, Q, V, and T represent the number of channels, video length, height, and width, respectively.

[0054] Step 2.1.2: Extract upper limb feature maps. Feature maps are extracted using a 3D CNN with two [3,3,3] convolutional kernels. 3D CNNs can enhance learning information in the time dimension; feature mappings are obtained by deconvolution using a 2D CNN with two [3,3] convolutional kernels. 2D CNNs focus more on video sequences Hand features are extracted from the image, where D is the image batch size.

[0055] Step 2.1.3: Use the feature fusion function to merge the feature maps into short-frequency feature maps. Where W is the number of short frequencies. The calculation formula is: ,in Using convolution filter parameters A function representing a 3D CNN or a 2D CNN;

[0056] According to the above scheme, the teacher's upper limb movement recognition model in the teaching scenario is characterized in that the teacher's upper limb movement recognition model includes a 3D structured light image acquisition module, upper limb feature extraction, a convolutional neural network, and a fully connected layer.

[0057] According to the above scheme, the training of the upper limb movement recognition model is as follows:

[0058] Step 2.2.1 involves preprocessing the structured light image data stream transformed from the video information acquired by the 3D structured light camera. Preprocessing includes image denoising of the structured light image using a Gaussian filter and uniformization of upper limb joint features. The pixel coordinates in the image before denoising are defined as follows: Pixel value The pixel value of the pixel in the image after noise reduction is denoised as... , The calculation formula is:

[0059] , (12)

[0060] , (13)

[0061] in, For pixels Normal distribution equation For pixels The pixel coordinates within, r pixel values, The value is a preset normal number. The feature vector normalization process yields the teacher's upper limb joint input vector. .

[0062] Step 2.2.2: Perform convolution operations to extract deep features from the input layer. The length and convolution kernel are then compared. Same teacher upper limb joint input vector data Convolution operations were performed to obtain the joint feature values ​​of the teacher's upper limbs. The output after activation function processing is as follows:

[0063] , (14)

[0064] in Represents the convolution kernel weight parameters. Represents the bias parameter, and ReLU represents the activation function.

[0065] Step 2.2.3 involves performing Softmax classification to output the recognition probability of each class in multi-class classification. The estimated probability of each sample belonging to a certain class can be expressed as:

[0066] = , (15)

[0067] in Represents the features of the input. This represents the class label, i.e., the sample label. This represents the parameters in the Softmax regression model, where k is the number of nodes in the Softmax layer. Therefore, the confidence formula for this probability is as follows:

[0068] , (16)

[0069] Step 2.2.4: Filter the output of the convolutional layer and linearly map it to the sample space to achieve the purpose of classification.

[0070] Step 2.2.5, the loss function calculated by both during model training is expressed as follows: ,in This represents the sample data of teachers. This indicates the number of targets that teachers can predict. A true label for teachers This represents the loss constraint parameters. The framework is trained using a minimum estimation method, and the model parameter values ​​are finally obtained when the loss function reaches its minimum value.

[0071] This invention employs a teacher instruction gesture recognition network based on an interleaved attention mechanism, designing a non-sequential decoding method that replaces the sequential decoding of traditional Transformers. This improves the inference order and provides constraint information for reference gesture postures. The interleaved attention module enhances prediction accuracy by perceiving the relationships between key points and joints.

[0072] This invention fuses RGB images with 3D structured light images; specifically, RGB-D refers to an RGB camera combined with a Time-of-Flight (TOF) camera. The RGB camera acquires 2D images of the teacher's hands and upper limbs, while the TOF and 3D structured light cameras acquire depth information. This combination of depth cameras overcomes the limitations of structured light due to its limited distance and the low accuracy of the TOF camera. By employing multi-camera fusion, more comprehensive and accurate 3D gesture and upper limb feature data are obtained. Attached Figure Description

[0073] Figure 1 This is a flowchart of a method and system for detecting teacher instructional intent based on interleaved attention, as implemented in this invention.

[0074] Figure 2 This is a diagram illustrating data acquisition in a teaching environment;

[0075] Figure 3 This is a network diagram of the teacher gesture recognition model according to an embodiment of the present invention;

[0076] Figure 4 This is a network diagram of the teacher's upper limb movement recognition model according to an embodiment of the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features described in the various embodiments of the invention below can be combined with each other as long as they do not conflict with each other.

[0078] like Figure 1 As shown, this embodiment of the invention is a method and system for detecting teacher instructional intentions based on interleaved attention, including the following steps:

[0079] Step 1: Acquire the RGB video stream of the teacher's hand posture, the RGB video stream of the teacher's upper limb movements, the structured light image video stream of the teacher's hand posture, and the structured light image video stream of the teacher's upper limb movements from the RGB-D camera and the 3D structured light camera in the teaching scene, respectively;

[0080] Step 2: Preprocess the video streams collected in the teaching scene to obtain single-channel key point heatmaps of teacher's hand posture, single-channel key point heatmaps of teacher's upper limb movements, structured light images of teacher's hand posture, and structured light images of teacher's upper limb movements.

[0081] Step 3: The preprocessed single-channel keypoint heatmap data stream of the teacher's gestures and the structured light image data stream of the teacher's hand posture are fused, and then input into the pre-trained teaching gesture estimation model to output the teacher's gesture recognition result. The single-channel keypoint heatmap video stream of the teacher's upper limb movements and the structured light image of the teacher's upper limb movements are fused, and then input into the pre-trained upper limb movement convolutional neural network model to output the teacher's upper limb movement recognition result.

[0082] like Figure 2 As shown, in the teaching scenario, video sequences of the teacher's gestures and upper limb movements are captured by an RGB-D camera and a 3D structured light camera located in the center of the scene. The RGB and structured light video images captured by the cameras are the data source for the teacher gesture recognition model and the teacher upper limb movement recognition model.

[0083] like Figure 3 As shown, in this embodiment, the teacher gesture recognition model includes feature fusion, a non-sequential Transformer, an interleaved attention layer, and a fully connected layer. The feature fusion includes a U-Net network, used to extract and reduce the dimensionality of the RGB-D acquired teacher gesture image appearance vector and combine it with the location-encoded keypoint heatmap features to form a gesture concatenation vector. The feature fusion module fuses the gesture concatenation vector and the gesture structured light image vector to output a gesture fusion vector. The interleaved attention layer predicts the value of each joint, thereby predicting hand posture. The fully connected layer outputs the final, refined gesture recognition result.

[0084] The non-sequential Transformer comprises a multi-head self-attention layer, joint feature union, a non-sequential self-attention layer, and a gesture reference extractor. The multi-head self-attention layer models the relationships between keypoints and creates global context-aware features for each keypoint. The joint union encodes keypoint identity information along with localized image features; the non-sequential self-attention layer avoids inference dependencies on the gesture estimation from the reference extractor; and the gesture reference extractor exposes reference poses to the decoder.

[0085] According to the above scheme, the feature extraction and feature fusion process is as follows:

[0086] Step 3.1.1: Feature extraction from the input data. The U-Net network is used to reduce the dimensionality of the hand pose appearance vector from the RGB-D data stream, obtaining the appearance feature vector. = .

[0087] Step 3.1.2 involves positional encoding of the heatmap vectors, including positional feature encoding and learnable embeddings. The learned embedding variables can be represented as: .in, This represents the learnable positional embedding. The learnable positional embedding of the next higher level is represented as: Each single-channel keypoint heatmap feature vector in the spatial encoder is processed by a matrix. , , The purpose is to obtain a linear mapping. , (17)

[0088] in, c represents the number of attention heads in a multi-head attention layer. This indicates the number of attention heads, and LN represents layer normalization. The output keypoint heatmap is used for location feature encoding. The output keypoint heatmap location encoding vector is represented as follows: .

[0089] Step 3.1.3 involves preprocessing the 3D structured light image data stream. The input structured light image is subjected to grayscale averaging to obtain the grayscale values ​​of 64 frames of pixel images. The grayscale averaging calculation process is as follows:

[0090] , (18)

[0091] in, This represents the maximum grayscale value in the image. This represents the minimum grayscale value in the image. This represents the learning parameters for mean-based averaging.

[0092] Step 3.1.4: Obtain the 3D coordinates of the hand joint through projection transformation and normalization using a structured light camera. The calculation between the 2D planar coordinates and the 3D coordinates is as follows:

[0093] , (19)

[0094] in, , These represent the camera's focal length, , , Represents 3D coordinates. , This represents 2D coordinates. 3D coordinates can be obtained and represented as follows:

[0095] , (20)

[0096] in, This represents the learning parameters.

[0097] Step 3.1.5: Fuse the input RGB-D image vectors. The gesture splicing vector is formed by combining the appearance feature vector of the teacher's hand gesture and the spatially encoded keypoint heatmap vector: .

[0098] Step 3.1.6 involves fusing the gesture splicing vector with the 3D coordinates obtained from the structured light image feature extraction. This can be represented as: .

[0099] The collected RGB-D key point heatmap of the teacher's hand is A= The acquired 3D structured light image of the teacher's hand is D= The RGB-D keypoint heatmap and structured light image are input into the U-Net network for dimensionality reduction, sample combination, and multilayer perceptron processing before outputting the result. Simultaneously, the RGB-D keypoint heatmap and structured light image are embedded with positional encoding. Then, feature fusion is performed to output a gesture fusion vector.

[0100] According to the above scheme, the teacher gesture recognition model is trained as follows:

[0101] Step 3.2.1: Perform feature fusion between the gesture stitching vector and the hand pose structured light image vector to obtain the gesture fusion vector. As input.

[0102] Step 3.2.2: Input the gesture fusion vector as the input vector into the non-sequential Transformer model. The pre-estimation results of the gesture reference extractor provide reference information for the prediction.

[0103] Step 3.2.3: For each pixel contained in the keypoint heatmap, the unique coordinates of the pixel under correct labeling conditions can be obtained. The coordinates of the current predicted key point heatmap ( The loss value of this pixel satisfies:

[0104] = , (twenty one)

[0105] Set threshold =3 pixels, the pixel coordinates of the keypoint heatmap satisfy... If so, then the key point is classified as a nearby joint. Then the key point and the background are grouped together.

[0106] Step 3.2.4 uses a single interleaved attention layer with learned joint queries to predict the relationship between keypoints and joints. The aim is to improve prediction accuracy by gaining a forward-backward perception of the relationship between keypoints and joints. The calculation can be expressed as:

[0107] , (twenty two)

[0108] Among them, learning joint query , They represent 256 respectively A learning matrix of 256.

[0109] Step 3.2.5, the loss function for the actual hand joint coordinates and the estimated hand joint coordinates is calculated during model training as follows:

[0110] , (twenty three)

[0111] in This represents the true value of the hand joint coordinates. Let J represent the predicted coordinates of the hand joints, and J represent the number of samples. The loss function for calculating the true keypoints and hand joints is as follows:

[0112] , (twenty four)

[0113] Where KL represents the KL divergence loss. , These represent the predicted and actual hand joint keypoints, respectively; the loss function for the actual and estimated hand joint coordinates in the fully connected layer is calculated as follows:

[0114] , (25)

[0115] in This represents the actual coordinates of the hand joints in the fully connected layer.

[0116] Step 3.2.6, during the model training process, the end-to-end learning process of the model is supervised by minimizing the loss function, calculated as follows:

[0117] , (26)

[0118] According to the above scheme, the non-sequential Transformer training process is as follows:

[0119] Step 3.3.1 involves inputting the gesture fusion vector, which has undergone feature extraction and feature fusion, into the multi-head self-attention mechanism and the multilayer perceptron, and then outputting the vector. The attention mechanism adaptively aggregates a set of input values ​​based on attention weights that consider the compatibility of a given query with a set of keys. Let the dimension of each query and key be... The dimension of the value is In the multi-head self-attention layer, attention weights are calculated on the L, C, and H matrices, and the multi-head attention mapping for hand joint features is as follows:

[0120] , (27)

[0121] in, , , In a multilayer perceptron, the signal propagates unidirectionally from the input layer to the output layer.

[0122] , (28)

[0123] in .

[0124] Step 3.3.2, the non-sequential structured decoding process eliminates the sequential dependency on previously generated tags and converts the input... The modified copy is directly passed to the decoder to obtain the output sequence. The decoding process is as follows:

[0125] , (29)

[0126] in These are model parameters.

[0127] Step 3.3.3: Input the normalized 3D key point information into the structured reference to generate a reference hand pose in the form of hand joint features. By passing a reference hand pose to the decoder, the decoder can capture the correlation between reference joints as an approximation of the target hand pose and use this information to better constrain the output space.

[0128] like Figure 4 As shown in this embodiment, the teacher's upper limb movement prediction method is characterized in that the teacher's upper limb movement recognition model includes a 3D structured light image acquisition module, a backbone network, upper limb feature extraction, a convolutional neural network, and a fully connected layer.

[0129] The backbone network includes a structured light image regression model, which designs a joint keypoint image data generator. The backbone network uses the joint keypoint images generated by the structured light image regression model to train the entire network. After training, it directly outputs the joint keypoint positions in the input image. The upper limb feature extraction includes 2D convolution, 3D convolution, and a feature fusion function. The convolutional neural network mainly consists of convolutional layers and a Softmax classifier. The fully connected layer is used to output the recognized upper limb movement results.

[0130] According to the above scheme, the upper limb feature extraction steps are as follows:

[0131] Step 3.4.1: 3D structured light video sequence of the teacher's gestures. As an input feature vector, K, Q, V, and T represent the number of channels, video length, height, and width, respectively.

[0132] Step 3.4.2: Extract upper limb feature maps. Feature maps are extracted using a 3D CNN with two [3,3,3] convolutional kernels. 3D CNNs can enhance learning information in the time dimension; feature mappings are obtained by deconvolution using a 2D CNN with two [3,3] convolutional kernels. 2D CNNs focus more on video sequences Hand features are extracted from the image, where D is the image batch size.

[0133] Step 3.4.3: Use the feature fusion function to merge the feature maps into short-frequency feature maps. Where W is the number of short frequencies. The calculation formula is: ,in Using convolution filter parameters A function representing a 3D CNN or a 2D CNN;

[0134] According to the above scheme, the upper limb movement recognition model is trained as follows:

[0135] Step 3.5.1: Determine the pixel coordinates in the image before noise reduction processing as follows: Pixel value The pixel value of the pixel in the image after noise reduction is denoised as... , The calculation formula is as follows:

[0136] , (30)

[0137] , (31)

[0138] in, For pixels Normal distribution equation For pixels The pixel coordinates within, r pixel values, The value is a preset normal number. The feature vector normalization process yields the teacher's upper limb joint input vector.

[0139] Step 3.5.2: Perform convolution operations to extract deep features from the input layer. The length and convolution kernel are then compared. Same teacher upper limb joint input vector data Convolution operations were performed to obtain the joint feature values ​​of the teacher's upper limbs. The output after processing by the activation function is:

[0140] , (32)

[0141] in Represents the convolution kernel weight parameters. Represents the bias parameter, and ReLU represents the activation function.

[0142] Step 3.5.3 involves performing Softmax classification to output the recognition probability of each class in multi-class classification. The formula for calculating the estimated probability of each sample belonging to a certain class is as follows:

[0143] , (33)

[0144] in Represents the features of the input. This represents the class label, i.e., the sample label. This represents the parameters in the Softmax regression model, where k is the number of nodes in the Softmax layer. Therefore, the confidence level for this probability is calculated as follows:

[0145] , (34)

[0146] Step 3.5.4: Filter the output of the convolutional layer and linearly map it to the sample space to achieve the purpose of classification.

[0147] Step 3.5.5, the loss function calculated during model training is as follows:

[0148] , (35)

[0149] in This represents the sample data of teachers. This indicates the number of targets that teachers can predict. A true label for teachers This represents the loss constraint parameters. The framework is trained using a minimum estimation method, and the model parameter values ​​are finally obtained when the loss function reaches its minimum value.

[0150] Step 4: Based on the teacher's gestures and upper limb movements in the teaching environment, feature fusion is performed to determine the teacher's teaching gestures and detect the teacher's teaching intentions.

[0151]

[0152] Step 5: Provide reference support for teacher teaching evaluation based on the results of the teacher teaching intention detection.

[0153] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting teacher instructional intentions based on an interleaved attention mechanism, characterized in that, Including the following steps: The teacher's hand pose RGB video stream, teacher's upper limb movement RGB video stream, teacher's hand pose structured light image video stream, and teacher's upper limb movement structured light image video stream were acquired from the RGB-D camera and the 3D structured light camera in the teaching scene, respectively. The video streams collected in the teaching scene were preprocessed to obtain single-channel key point heatmaps of teacher's hand posture, single-channel key point heatmaps of teacher's upper limb movements, structured light images of teacher's hand posture, and structured light images of teacher's upper limb movements. The preprocessed single-channel keypoint heatmap gesture splicing vector of teacher gestures and the structured light feature vector of teacher hand posture are fused together and then input into the pre-trained teaching gesture estimation model to obtain the teacher gesture recognition result; the RGB feature vector of teacher upper limb movement and the structured light feature vector of teacher upper limb movement are fused together and then input into the pre-trained upper limb movement convolutional neural network model to obtain the teacher upper limb movement recognition result. The teacher's gesture recognition results and upper limb movement recognition results are combined using mean normalization to determine the type of teacher's teaching gestures. The teacher gesture recognition model includes an RGB-D image acquisition module, a feature fusion module, a non-sequential Transformer, an interleaved attention module, and a fully connected layer; The feature fusion module includes a U-Net network, used to extract and reduce the appearance vector of the teacher's gesture image acquired in RGB-D, perform position encoding on the key point heatmap features to obtain the gesture position encoding vector, and perform grayscale uniformization on the structured light image to obtain the 3D coordinate gesture structured light feature vector; the gesture appearance vector and the gesture position encoding vector form a gesture splicing vector; the feature fusion module fuses the gesture splicing vector and the gesture structured light image feature vector to output a gesture fusion vector; the non-sequential Transformer is used to take the gesture fusion vector as input, extract the joint features of the teacher's hand in the teaching hand heatmap, and perform gesture recognition on the extracted features; staggered attention layers are used to predict the value of each joint and then to predict the hand posture. The fully connected layer is used to output the final, fine-grained gesture recognition result; The teacher gesture recognition results are used as an assessment reference for evaluating teachers' ability to convey thoughts and feelings, organize teaching and learning, and demonstrate their good mental outlook and professional ethics through gestures in the classroom.

2. The method for detecting teacher instructional intent based on interleaved attention mechanism as described in claim 1, characterized in that, The training of the teaching gesture recognition model is as follows: Step 1.2.1: Input the gesture fusion vector as the input vector into the non-sequential Transformer model. The pre-estimation results of the gesture reference extractor provide reference information for the prediction. Step 1.2.2: Calculate the difference between the predicted keypoint heatmap and the labeled keypoint heatmap output by the gesture recognition model, using this difference as the loss value of the gesture recognition model. Then, for the predicted keypoint heatmap in the image, obtain the label using the keypoint data. The loss value for the corresponding photo is calculated, and the network weights are updated using the total loss value of the dataset. For each pixel in the keypoint heatmap, the unique coordinates of the pixel under correct labeling can be obtained. The coordinates of the current predicted key point heatmap ( The loss value of this pixel satisfies: = Set threshold The pixel coordinates of the key point heatmap satisfy If so, then the key point is classified as a nearby key point. Then the key point and the background are grouped into one category; Step 1.2.3 uses a single interleaved attention layer with learned joint queries to predict the relationship between keypoints and joints. The purpose is to improve the prediction accuracy by gaining a forward-backward perception of the relationship between keypoints and joints; the calculation can be expressed as: Among them, learning joint query , They represent 256 respectively A learning matrix of 256; Step 1.2.4, the loss function for the actual hand joint coordinates and the estimated hand joint coordinates during model training is expressed as: ,in This represents the true value of the hand joint coordinates. Let J represent the predicted coordinates of the hand joints, and J represent the number of samples. The loss function for calculating the true keypoints and hand joints is expressed as: Where KL represents the KL divergence loss. , These represent the predicted and actual hand joint keypoints, respectively; the loss function for the fully connected layer between the actual and estimated hand joint coordinates is expressed as: ,in This represents the actual values ​​of the hand joint coordinates in the fully connected layer; Step 1.2.5: During the model training process, the end-to-end learning process of the model is supervised by minimizing the loss function, as shown in: ; The framework is trained by minimization, and the model parameter values ​​can be obtained when the loss function reaches its minimum value. The non-sequential Transformer includes a multi-head self-attention layer, joint joints, a non-sequential self-attention layer, and a gesture reference extractor; the multi-head self-attention layer is used to model the relationships between key points and create global context-aware features for each key point. The joint joint encodes key point identity information together with localized image features; the non-sequential self-attention layer avoids inference dependence on the gesture estimation of the gesture reference extractor; The gesture reference extractor provides reference hand gestures to the decoder.

3. The method for detecting teacher instructional intent based on interleaved attention mechanism as described in claim 1, characterized in that, The gesture recognition results obtained by the fully connected layer are divided into four categories: figurative gestures, speech aid gestures, basic gestures, and non-essential gestures.

4. The method for detecting teacher instructional intent based on interleaved attention mechanism as described in claim 1, characterized in that, The teacher's upper limb movement recognition model includes a 3D structured light image acquisition module, a backbone network, upper limb feature extraction, a convolutional neural network, and a fully connected layer; The backbone network includes a structured light image regression model, which designs a joint key point image data generator. The backbone network uses the joint key point images generated by the structured light image regression model to train the entire network. After training, it directly outputs the joint key point positions in the input image. The upper limb feature extraction includes 2D convolution, 3D convolution, and feature fusion functions. The convolutional neural network mainly consists of convolutional layers and a Softmax classifier. The fully connected layer is used to output the recognized upper limb movement results. The convolutional neural network is used to take the structured light image converted from the image of the teacher's upper limb movements as input, extract the features of the teacher's upper limb movements, and perform action recognition on the extracted features.

5. The method for detecting teacher instructional intent based on interleaved attention mechanism as described in claim 4, characterized in that, The steps for extracting upper limb movements are as follows: Step 2.1.1: 3D structured light video sequence of the teacher's gestures. As an input feature vector, K, Q, V, and T represent the number of channels, video length, height, and width, respectively; Step 2.1.2: Extract upper limb feature maps using a 3D CNN with two [3,3,3] convolutional kernels. 3D CNNs can enhance the learning information in the time dimension; Feature mapping is obtained by deconvolution using a 2D CNN with two layers of [3,3] convolutional kernels. 2D CNNs focus more on video sequences Hand features are extracted from the image, where D is the image batch size. Step 2.1.3: Use the feature fusion function to merge the feature maps into short-frequency feature maps. Where W is the number of short frequencies, calculated using the following formula: ,in Using convolution filter parameters A function representing a 3D CNN or a 2D CNN.

6. The method for detecting teacher instructional intent based on interleaved attention mechanism as described in claim 5, characterized in that, The training of the aforementioned upper limb motion recognition model is as follows: Step 2.2.1: The structured light image data stream transformed from the video information acquired by the 3D structured light camera is preprocessed. This preprocessing includes image denoising using a Gaussian filter and uniformization of upper limb joint features. The pixel coordinates in the image before denoising are then set as follows: Pixel value The pixel value of that pixel in the image after noise reduction is denoised as... , The calculation formulas are as follows: , ,in, For pixels Normal distribution equation For pixels The pixel coordinates within, r pixel values, The preset normal numbers; the teacher's upper limb joint input vector obtained by the feature vector normalization process. ; Step 2.2.2: Perform convolution operation to extract deep features from the input layer, and then adjust the length of the convolution kernel. Same teacher upper limb joint input vector data Perform convolution operations to obtain the joint feature values ​​of the teacher's upper limbs. The output after processing by the activation function is: ,in Represents the convolution kernel weight parameters. ReLU represents the bias parameter, and ReLU represents the activation function. Step 2.2.3: Perform Softmax classification to output the recognition probability of each class in multi-class classification. The confidence formula for this probability is: The output of the convolutional layer is filtered and linearly mapped to the sample space to achieve the purpose of classification. Represents the features of the input. Indicates sample label, These represent the parameters in the Softmax regression model, where k is the number of nodes in the Softmax layer. The output of the convolutional layer is filtered and linearly mapped to the sample space to achieve the purpose of classification. Step 2.2.4: The final upper limb movement recognition result is obtained by calculating through a fully connected layer; Step 2.2.5, the loss function calculated by both during model training is expressed as follows: ,in This represents a sample of teacher data. This indicates the number of targets that teachers can predict. A true label for teachers The loss constraint parameters are used to train the framework using the minimum estimation method. The model parameter values ​​can be obtained when the loss function reaches its minimum value.

Images