A Classroom Teaching Gesture Recognition Method Based on Chessboard Self-Attention Mechanism

Abstract

This invention addresses the issue of the limited range of methods in current smart classroom teaching by disclosing a gesture recognition method based on a chessboard self-attention mechanism. The method includes the following steps: 1) acquiring teacher video data from a four-sensor color linear scan camera and point cloud information from a 3D LiDAR scanner in the smart classroom; 2) dividing the teacher video data from the four-sensor color linear scan camera into RGB image frames in chronological order; 3) inputting the teacher's RGB image frames into a gesture detection model to obtain the teacher's gesture detection results in the smart classroom; 4) fusing the teacher's gesture detection images with the point cloud information, and further inputting the data into a 3D gesture recognition model to identify the teacher's gestures in the smart classroom; 5) analyzing the teacher's gesture recognition results at different times and feeding them back into the smart classroom for human-computer interaction, thereby controlling the teaching equipment. This invention, by recognizing and analyzing the teacher's gestures at different times in the smart classroom, can determine the teacher's gestures in real time, convey the teacher's intended actions, and efficiently control the teaching equipment.

Description

Technical Field

[0001] This invention relates to the field of classroom teaching, specifically to a method for recognizing classroom gestures based on a chessboard self-attention mechanism. Background Technology

[0002] In the era of "Artificial Intelligence + Education," the integration of information technology and teaching has given rise to smart classrooms. With the development of educational information, from the traditional "blackboard and chalk" classroom to the transitional era of electronic classrooms, this teaching method is primarily teacher-centered and textbook-centered, emphasizing standardization over innovation, relying mainly on blackboard writing and lecturing, resulting in a relatively monotonous teaching style. Traditional teaching methods rarely involve modern teaching equipment, failing to adequately meet the demands of comprehensively improving teachers' overall quality; they lack access to the most advanced teaching methods, hindering exposure to ever-changing knowledge and preventing the broadening of both teachers' and students' horizons; and the monotonous nature of traditional teaching relegates students to supporting roles, lacking two-way interaction between teachers and students, thus hindering teaching efficiency. It can be said that traditional teaching methods need to be transformed. In traditional teaching, many teaching contents requiring on-site demonstrations or experiments cannot be conducted, teachers cannot provide students with the information they need to see or hear, and the completeness of the teaching content is far from guaranteed.

[0003] The emergence of smart classrooms has garnered widespread praise from teachers and students alike for its personalized, data-driven, intelligent, contextualized, interactive, collaborative, open, and experiential features. Smart classrooms enable students to apply knowledge to solve problems in contextualized, mobile, and sensory learning activities, fostering intellectual growth through practical experiences. Teachers, as facilitators in the classroom, determine the quality of teaching. Therefore, it is crucial for teachers to motivate students and effectively organize instruction. However, current gesture recognition remains challenging due to complex posture changes, global orientation variations, self-similarity between fingers, and severe occlusion issues, making accurate and robust hand gesture recognition a pressing problem that needs to be addressed. Summary of the Invention

[0004] The technical problem this invention aims to solve is to address the shortcomings of existing methods by providing a classroom teaching gesture recognition method based on a chessboard self-attention mechanism. This method can accurately capture and recognize teachers' teaching gestures in real time in smart classrooms. By interpreting the teacher's body language, the method can determine the quality of teaching, thereby deciding whether to enliven the classroom atmosphere, organize teaching, and motivate students' learning.

[0005] The technical solution adopted by this invention to solve its technical problem is: a classroom teaching gesture recognition method based on a chessboard self-attention mechanism, comprising the following steps:

[0006] The video data of the teacher under the four-sensor color linear scan camera and the point cloud information of the smart classroom under the 3D LiDAR were acquired respectively.

[0007] The teacher's video data from the four-sensor color linear scan camera was divided into RGB image frames in chronological order.

[0008] The teacher's RGB image frame is input into the gesture detection model to obtain the gesture detection results of the teacher in the smart classroom;

[0009] The teacher's gesture detection image is fused with point cloud information, and the data is further fed into a 3D gesture recognition model to identify the teacher's gesture results in the smart classroom.

[0010] The results of the teacher's gesture recognition at different times are analyzed and fed back into the smart classroom for human-computer interaction, which can then be used to control the teaching equipment.

[0011] The data acquisition module includes the acquisition of image data and the acquisition of point cloud data. The image data is acquired by a four-sensor color linear scan camera; the point cloud data is acquired by a 3D LiDAR.

[0012] The gesture detection model follows a typical encoder-decoder framework, including image preprocessing, a backbone feature extraction Transformer, a gesture detection Transformer, feedforward neural networks (FFNs), and loss calculation. The backbone feature extraction Transformer (BF-Transformer) module consists of L feature extraction encoders; the gesture detection Transformer (HD-Transformer) consists of N sets of gesture detection encoders and decoders.

[0013] The image preprocessing segmentes the image into blocks and processes the image dimensions accordingly; the backbone feature extraction Transformer uses a chessboard self-attention mechanism to extract multi-scale teacher image features; the gesture detection Transformer uses the teacher image features of all frames as input to detect and locate the position of the teacher's gesture; the feedforward neural network is used to predict the gesture position result; the loss calculation is used for gradient backpropagation to update the model parameters, thereby obtaining the final detection result.

[0014] According to the above scheme, the image preprocessing module steps are as follows:

[0015] Step 1.1.1: Obtain the teacher video data stream from the smart classroom camera, select the teacher's RGB image frame in the smart classroom as input, and perform a series of image enhancement and noise reduction operations.

[0016] Step 1.1.2: Select a set of RGB images of teacher gestures as input, with an image size of h×w. First, segment the images into non-overlapping image patches using a patch partitioning layer. The image patch size is M×M, typically M=4. Then, project the original value features onto an arbitrary dimension c using a linear embedding layer, typically c=56.

[0017] According to the above scheme, the training process of the Transformer model for backbone feature extraction is as follows:

[0018] Step 1.2.1: Calculate the weighted sum of the self-attention coefficient matrix and the value matrix. The self-attention coefficient matrix Ω = Γ·Δ T ,in To compute the input feature representation of the (l+1)th layer, a weighted sum of the coefficient matrix and value matrix of each self-attention layer is required.

[0019] Step 1.2.2: Calculate multi-head self-attention separately. The feature image is divided using a chessboard-like staggered window strategy starting from the top-left pixel. Therefore, the recursive formula for calculating continuous chessboard self-attention is:

[0020]

[0021]

[0022]

[0023]

[0024] in, This represents the output features of the W_MSA multi-head self-attention and MLP modules, z l LN(·) represents the output features of the B_MSA multi-head self-attention and MLP modules, LN(·) represents layer normalization, and l∈{1,…,2n} represents the number of attention heads.

[0025] Step 1.2.3: Perform relative position encoding for each head. When calculating self-attention, there are two different self-attention calculations; relative position encoding is then performed based on these. Therefore, the head of each multi-head self-attention:

[0026]

[0027] in It is the query matrix in each feature extraction encoder. It is the keyword matrix in each feature extraction encoder. It is the value matrix in each feature extraction encoder, where d represents the dimensions of Γ, Ψ, and Δ, and M... 2 Indicates batch size;

[0028] Step 1.2.4: Obtain multi-scale features. Utilizing local prior knowledge, aggregate adjacent small patches to obtain multi-scale features x. t =Merging(g(L) r In this context, Merging represents the aggregation operation, and g(·) represents the convolutional layer. Steps 1.2.1 and 1.2.2 are repeated, primarily to leverage spatial dimensions to obtain more channels, resulting in finer-grained global features. The final output multi-scale feature dimension is...

[0029] According to the above scheme, the training process of the gesture detection Transformer model is as follows:

[0030] Step 1.3.1, Feature Preprocessing. The high-dimensional features f at multiple scales are reduced in dimensionality using 1×1 convolutions to create new feature maps. The gesture detection encoder takes the expected sequence as input and flattens the feature map f0 into a one-dimensional sequence. Therefore, the preprocessed features can be represented by the model in a vectorized form.

[0031] Step 1.3.2: Calculate the query matrix, keyword matrix, and value matrix for each gesture detection encoder. Given a query... Keywords Sum Therefore, the query matrix Γ = W in each gesture detection encoder i (γ) γ, therefore the keyword matrix Ψ = W in each gesture detection encoder i (ψ) ψ, therefore the value matrix Δ = W in each gesture detection encoder i (δ) δ, where the learnable parameter is... and Here the output feature matrix can be transformed into:

[0032]

[0033] Step 1.3.3, calculate multi-head self-attention. Each self-attention head h i The calculation method for (i = 1, ..., h) is as follows:

[0034] MUTH(Γ,Ψ,Δ)=Concat(h1,…,hh W o (7)

[0035] Each self-attention head h here i ∈ATN(Γ,Ψ,Δ);

[0036] Step 1.3.4, the decoding process of the gesture detection Transformer. The decoder follows the standard Transformer architecture, with input represented as embeddings of size C×N, meaning each decoder layer can decode N objects in parallel. Here, N is typically 50, and C represents the dimension of the encoder's output features. These input embeddings are learnable positional codes, which, along with global features from the encoder, compute self-attention, transforming the N input embeddings into N output embeddings.

[0037] According to the above scheme, the steps of the feedforward neural network are as follows:

[0038] Step 1.4.1: Since the final features predicted by the decoder are a set of N fixed-size bounding boxes, where N is typically much larger than the actual number of objects of interest in the image, Used to indicate that there are no objects inside the prediction box;

[0039] Step 1.4.2: The final prediction result is calculated by a 3-layer perceptron with an ELU activation function, hidden layers, and a linear projection layer. FFNs predict the normalized center coordinates, height, and width of the input image bounding box. The linear layer uses the Softmax function to predict the class label.

[0040] According to the above scheme, the steps for calculating Loss are as follows:

[0041] Step 1.5.1, let y represent the set of true values ​​of the object. To represent a set of N predictions, additionally use This indicates that no object was detected, or that it belongs to the background category. Therefore, the optimal assignment is calculated as follows:

[0042]

[0043] And the corresponding matching loss is:

[0044]

[0045] in It is the true value y i Matching loss with the predicted value index σ(i);

[0046] Step 1.5.2: To better fit the loss between predicted and true values ​​and establish a one-to-one correspondence between the true and predicted bounding boxes, the Hungarian algorithm is used to calculate the loss function. The Hungarian loss function is:

[0047]

[0048] in Gradient backpropagation is performed to update the model parameters, thereby obtaining the final detection result.

[0049] According to the above scheme, the data fusion module includes teacher images and point cloud information with detected gesture parts; the 3D gesture recognition model includes a point cloud processing Transformer module, a joint estimation Transformer module, and a fully connected layer; the point cloud processing Transformer module includes an input embedding layer, a reference gesture extractor, four cascaded self-attention modules, and a Concat function; the joint estimation Transformer module follows a standard encoder-decoder architecture, including L1 gesture recognition encoders, L2 gesture recognition decoders, NC multi-head self-attention layers, and ED multi-head attention layers.

[0050] The data fusion module is used to fuse the teacher's gesture detection image with point cloud information; the point cloud processing Transformer module is used to extract the point cloud features of the teacher's gesture; the joint estimation Transformer module; and the fully connected layer are used to obtain the coordinates of the gesture joints.

[0051] According to the above scheme, the data fusion module steps are as follows:

[0052] Step 2.1.1: The calibration teacher has detected the hand image frame data and the point cloud information acquired by the LiDAR. The calibration coordinate system equation is:

[0053]

[0054] Where U and V represent the rotation and translation matrices of the camera coordinate system relative to the lidar coordinate system, respectively, and g x g represents the camera's lateral focal length. y Let (α0, β0) represent the camera's longitudinal focal length, (α, β0) represent the original image coordinates, and (α, β) represent the transformed coordinates in the pixel coordinate system. x Indicates the offset between coordinate system transformations;

[0055] Step 2.1.2, transform the (H) coordinate system of the camera into a coordinate system. c J c ,K c Transformed into the lidar coordinate system (H) L J L ,K LUsing images of the teacher's hands that have already been detected in the image frame reduces the recognition space for the 3D gesture recognition module, decreases redundant background points in the point cloud, and improves operating efficiency.

[0056] According to the above scheme, the steps of the point cloud processing Transformer module are as follows:

[0057] Step 2.2.1: Point Cloud Input. Select a set of unordered spatial coordinate points of the teacher's gestures. These spatial coordinate points can be modeled as a series of vector representations:

[0058] Step 2.2.2: Extraction of the reference gesture. The gesture is parameterized into a set of spatial joint coordinates in the camera coordinate system. Where J represents the number of hand joints. A reference gesture is generated using a reference gesture extractor based on the relevant input points.

[0059] Step 2.2.3: Extract point cloud features. Input point cloud P∈R N×d First, the input embedding layer outputs a value with d. e 3D eigenvectors Then, the outputs of each self-attention layer are concatenated using the feature dimension:

[0060]

[0061] Among them, AT i W represents the i-th self-attention layer, where each self-attention layer has the same output dimension as its input; o These are the weights of the linear layer.

[0062] According to the above scheme, the steps of the joint estimation Transformer module are as follows:

[0063] Step 2.3.1: Obtain enhanced gesture keypoint features. Downsample and normalize the N point set in the camera coordinate system, represented as P = [p1, p2, ..., p...]. N The normalized point P is fed into the gesture recognition encoder to generate enhanced gesture key points C = {c1, ..., c2}. N};

[0064] Step 2.3.2: Fusion of enhanced gesture joint features with reference gesture. The reference gesture is further input into the gesture recognition decoder, and its structural dependencies between joints are captured through an NC multi-head self-attention layer, which is used as an approximation of the target gesture;

[0065] Step 2.3.3: Associate the point-by-point augmented gesture joint C. Using the ED multi-head attention module, the captured pose relevance is associated with the point-by-point augmented gesture joint C to discover specific jointly generated relevance points from different subspaces. The decoding process can be formulated as follows:

[0066]

[0067] Where θ are the model parameters.

[0068] According to the above scheme, the fully connected layer is used to obtain the coordinates of each joint, and the steps are as follows:

[0069] Step 2.4.1: Under the supervision of the captured reference gesture pose correlation, the correlation between the output joint and the input joint is better captured. The weighted enhanced gesture joint features are aggregated with the pose correlation, and then passed point by point through a feedforward neural network (FFN) to obtain the decoder output. The decoder output is then passed through a fully connected layer to obtain the coordinates (x, y, y) of each gesture joint. i ,y i );

[0070] Step 2.4.2: For training, regarding the loss function, based on the true key points in each training sample. Compared with the estimated key points (x) i ,y i Using the squared difference loss function:

[0071]

[0072] Point-by-point features of the joints of the reference gesture Joint coordinates (x) regressed from reference gesture j ‘ ,y j ‘ Using the squared difference loss function

[0073]

[0074] Therefore, the total loss function is:

[0075]

[0076] Compared with existing technologies, the advantages of this invention are as follows: It provides a classroom teaching gesture recognition method based on a chessboard self-attention mechanism, enabling teachers to better operate teaching equipment; the various modules proposed in this invention make the network structure more hierarchical and scalable; the method fuses image data with point clouds, uses the detection results to reduce the space of 3D gesture recognition, reduces redundant background points in the point cloud data, and greatly improves operating efficiency; in gesture recognition, it utilizes the inherent dependencies between reference gesture joints and uses the globality of Transformer for modeling. The proposed invention has a very good promoting effect on teachers' teaching in smart classrooms. Attached Figure Description

[0077] Figure 1 This is a flowchart of a classroom teaching gesture recognition method based on a chessboard self-attention mechanism according to an embodiment of the present invention;

[0078] Figure 2 This is a diagram illustrating data acquisition in a smart classroom environment;

[0079] Figure 3 This is a network diagram illustrating a classroom teaching gesture recognition method based on a chessboard self-attention mechanism, according to an embodiment of the present invention. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0081] like Figure 1 As shown, this embodiment of the invention is a classroom teaching gesture recognition method based on a chessboard self-attention mechanism, including the following steps:

[0082] Step 1: Acquire the teacher's video data from the four-sensor color linear array scanning camera and the point cloud information of the smart classroom from the 3D LiDAR.

[0083] Step 2: Divide the teacher's video data from the four-sensor color linear scan camera into RGB image frames in chronological order;

[0084] Step 3: Input the teacher's RGB image frame into the gesture detection model to obtain the teacher's gesture detection results in the smart classroom;

[0085] like Figure 2As shown, a teacher is giving a lesson in a smart classroom. In this scenario, a four-sensor color linear scan camera captures video image frames of the teacher's gestures, and a 3D LiDAR is used to obtain the point cloud information of the teacher in the smart classroom. The RGB images and point cloud information of the teacher's gestures collected in this scenario provide important data sources for the gesture detection module and the 3D gesture recognition module.

[0086] like Figure 3 As shown, in this embodiment, the gesture detection model includes image preprocessing, a backbone feature extraction Transformer, a gesture detection Transformer, feedforward neural networks (FFNs), and loss calculation. The backbone feature extraction Transformer (BF-Transformer) module consists of L feature extraction encoders; the gesture detection Transformer (HD-Transformer) consists of N sets of gesture detection encoders and decoders. The feature extraction encoder consists of a merging layer and a multi-head attention mechanism; the gesture detection encoders and gesture detection decoders are composed of a multi-head self-attention mechanism, and the output of the gesture detection decoder can be input into the feedforward network in parallel.

[0087] The backbone feature extraction Transformer uses a chessboard self-attention mechanism to extract multi-scale teacher image features; the gesture detection Transformer uses the teacher image features of all frames as input to detect and locate the position of the teacher's gesture; the feedforward neural network is used to predict the gesture detection result; the loss calculation is used for gradient backpropagation to update the model parameters, thereby obtaining the final detection result.

[0088] According to the above scheme, the image preprocessing steps are as follows:

[0089] Step 3.1.1: Select the RGB image frame of the classroom in the smart classroom as input, and perform a series of image enhancement and noise reduction operations;

[0090] Step 3.1.2: Select a set of processed RGB images of the teachers and set their image size to h×w. First, segment the images into non-overlapping image patches using a patch partitioning layer. The image patch size is M×M, usually M=4. Then, project the original value features onto an arbitrary dimension c using a linear embedding layer, usually c=56.

[0091] According to the above scheme, the training process of the Transformer model for backbone feature extraction is as follows:

[0092] Step 3.2.1: Calculate the weighted sum of the self-attention coefficient matrix and the value matrix. The self-attention coefficient matrix Ω = Γ·Δ T , where Ω∈R M2×M2 To compute the input feature representation of the (l+1)th layer, a weighted sum of the coefficient matrix and value matrix of each self-attention layer is required, i.e.

[0093] Step 3.2.2: Calculate multi-head self-attention separately. The feature image is divided using a chessboard-like staggered window strategy starting from the top-left pixel. Therefore, the recursive formula for calculating continuous chessboard self-attention is:

[0094]

[0095]

[0096]

[0097]

[0098] in, This represents the output features of the W_MSA multi-head self-attention and MLP modules, z l LN(·) represents the output features of the B_MSA multi-head self-attention and MLP modules, LN(·) represents layer normalization, and l∈{1,…,2n} represents the number of attention heads.

[0099] Step 3.2.3: Perform relative position encoding for each head. When calculating self-attention, there are two different self-attention calculations; relative position encoding is then performed based on these two methods. Therefore, the head of each multi-head self-attention:

[0100]

[0101] in It is the query matrix in each feature extraction encoder. It is the keyword matrix in each feature extraction encoder. It is the value matrix in each feature extraction encoder, where d represents the dimensions of Γ, Ψ, and Δ, and M... 2 Indicates batch size;

[0102] Step 3.2.4: Obtain multi-scale features. Utilizing local prior knowledge, aggregate adjacent small patches to obtain multi-scale features x. t =Merging(g(L) rIn this context, Merging represents the aggregation operation, and g(·) represents the convolutional layer. Steps 1.2.1 and 1.2.2 are repeated, primarily to leverage spatial dimensions to obtain more channels, resulting in finer-grained global features. The final output multi-scale feature dimension is...

[0103] According to the above scheme, the training process of the gesture detection Transformer model is as follows:

[0104] Step 3.3.1, Feature Preprocessing. The high-dimensional features f at multiple scales are reduced in dimensionality using 1×1 convolutions to create new feature maps. The gesture detection encoder takes the expected sequence as input and flattens the feature map f0 into a one-dimensional sequence. Therefore, the preprocessed features can be represented by the model in a vectorized form.

[0105] Step 3.3.2: Calculate the query matrix, keyword matrix, and value matrix for each gesture detection encoder. Given a query... Keywords Sum Therefore, the query matrix Γ = W in each gesture detection encoder i (γ) γ, therefore the keyword matrix Ψ = W in each gesture detection encoder i (ψ) ψ, therefore the value matrix Δ = W in each gesture detection encoder i (δ) δ, where the learnable parameter is... and Here the output feature matrix can be transformed into:

[0106]

[0107] Step 3.3.3, calculate multi-head self-attention. Each self-attention head h i The calculation method for (i = 1, ..., h) is as follows:

[0108] MUTH(Q,K,V)=Concat(h1,…,h h W o , (twenty three)

[0109] Each self-attention head h here i ∈ATN(Q,K,V);

[0110] Step 3.3.4, the decoding process of the gesture detection Transformer. The decoder follows the standard Transformer architecture, with input represented as embeddings of size C×N, meaning each decoder layer can decode N objects in parallel. Here, N is typically 50, and C represents the dimension of the encoder's output features. These input embeddings are learnable positional codes, which, along with global features from the encoder, compute self-attention, transforming the N input embeddings into N output embeddings.

[0111] According to the above scheme, the feedforward neural network is used to predict the gesture detection result.

[0112] Step 3.4.1: Since the final features predicted by the decoder are a set of N fixed-size bounding boxes, where N is typically much larger than the actual number of objects of interest in the image, Used to indicate that there are no objects inside the prediction box;

[0113] Step 3.4.2: The final prediction result is calculated by a 3-layer perceptron with an ELU activation function, hidden layers, and a linear projection layer. FFNs predict the normalized center coordinates, height, and width of the input image bounding box. The linear layer uses the Softmax function to predict the class label.

[0114] According to the above scheme, the calculated Loss is used for gradient backpropagation to update the model parameters, thereby obtaining the final detection result.

[0115] Step 3.5.1, let y represent the set of true values ​​of the object. To represent a set of N predictions, additionally use This indicates that no object was detected, or that it belongs to the background category. Therefore, the optimal assignment is calculated as follows:

[0116]

[0117] And the corresponding matching loss is:

[0118]

[0119] in It is the true value y i Matching loss with the predicted value index σ(i);

[0120] Step 3.5.2: To better fit the loss between the predicted and true values ​​and establish a one-to-one correspondence between the true and predicted bounding boxes, the Hungarian algorithm is used to calculate the loss function. The Hungarian loss function is:

[0121]

[0122] in Gradient backpropagation is performed to update the model parameters, thereby obtaining the final detection result.

[0123] Step 4: The teacher's gesture detection image is fused with the point cloud information, and the data is further fed into the 3D gesture recognition model to identify the gesture results in the smart classroom.

[0124] like Figure 3 As shown, in this embodiment, according to the above scheme, the data fusion module includes the teacher image with detected gesture parts and point cloud information; the 3D gesture recognition model includes a point cloud processing Transformer module, a joint estimation Transformer module, and a fully connected layer. The point cloud processing Transformer module includes an input embedding layer, a reference gesture extractor, four cascaded self-attention modules, and a Concat function; the joint estimation Transformer module follows a standard encoder-decoder architecture, including L1 gesture recognition encoders, L2 gesture recognition decoders, NC multi-head self-attention layers, and ED multi-head attention layers.

[0125] The point cloud processing Transformer module is used to extract point cloud features of the teacher's gestures; the joint estimation Transformer module; and the fully connected layer are used to obtain the coordinates of the gesture joints.

[0126] Step 4.1.1: The calibration teacher has detected the hand image frame data and the point cloud information acquired by the LiDAR. The calibration coordinate system equation is:

[0127]

[0128] Where U and V represent the rotation and translation matrices of the camera coordinate system relative to the lidar coordinate system, respectively, and g x g represents the camera's lateral focal length. y Let (α0, β0) represent the camera's longitudinal focal length, (α, β0) represent the original image coordinates, and (α, β) represent the transformed coordinates in the pixel coordinate system. x Indicates the offset between coordinate system transformations;

[0129] Step 4.1.2, transform the (H) coordinate system of the camera into a coordinate system. c J c ,K c Transformed into the lidar coordinate system (H) L J L ,K L Using images of the teacher's hands that have already been detected in the image frame reduces the recognition space for the 3D gesture recognition module, decreases redundant background points in the point cloud, and improves operating efficiency.

[0130] According to the above scheme, the steps of the point cloud processing Transformer module are as follows:

[0131] Step 4.2.1: Point Cloud Input. Select a set of unordered spatial coordinate points of the teacher's gestures. These spatial coordinate points can be modeled as a series of vector representations:

[0132] Step 4.2.2: Extraction of the reference gesture. The gesture is parameterized into a set of spatial joint coordinates in the camera coordinate system. Where J represents the number of hand joints. A reference gesture is generated using a reference gesture extractor based on the relevant input points.

[0133] Step 4.2.3: Extract point cloud features. Input point cloud P∈R N×d First, the input embedding layer outputs a value with d. e 3D eigenvectors Then, the outputs of each self-attention layer are concatenated using the feature dimension:

[0134]

[0135] Among them, AT i W represents the i-th self-attention layer, where each self-attention layer has the same output dimension as its input; o These are the weights of the linear layer.

[0136] According to the above scheme, the steps of the joint estimation Transformer module are as follows:

[0137] Step 4.3.1: Obtain enhanced gesture keypoint features. Downsample and normalize the N point set in the camera coordinate system, represented as P = [p1, p2, ..., p N The normalized point P is fed into the gesture recognition encoder to generate enhanced gesture key points C = {c1, ..., c2}. N};

[0138] Step 4.3.2: Fusion of enhanced gesture joint features with reference gesture. The reference gesture is further input into the gesture recognition decoder, and its structural dependencies between joints are captured through an NC multi-head self-attention layer, which is used as an approximation of the target gesture;

[0139] Step 4.3.3: Associate the point-by-point augmented gesture joint C. Using the ED multi-head attention module, the captured pose relevance is associated with the point-by-point augmented gesture joint C to discover specific jointly generated relevance points from different subspaces. The decoding process can be formulated as follows:

[0140]

[0141] Where θ are the model parameters.

[0142] According to the above scheme, the fully connected layer is used to obtain the coordinates of each joint.

[0143] Step 4.4.1: Under the supervision of the captured reference gesture pose correlation, the correlation between the output joint and the input joint is better captured. The weighted enhanced gesture joint features are aggregated with the pose correlation, and then passed point by point through a feedforward neural network (FFN) to obtain the decoder output. The decoder output is then passed through a fully connected layer to obtain the coordinates (x, y, y) of each gesture joint. i ,y i );

[0144] Step 4.4.2: For training, regarding the loss function, based on the true key points in each training sample. Compared with the estimated key points (x) i ,y i Using the squared difference loss function:

[0145]

[0146] Point-by-point features of the joints of the reference gesture Joint coordinates (x) regressed from reference gesture j ‘ ,y j ‘ Using the squared difference loss function

[0147]

[0148] Therefore, the total loss function is:

[0149]

[0150] Step 5: Analyze the teacher's gesture recognition results at different times and feed them back into the smart classroom for human-computer interaction, enabling control of teaching equipment. Here, the teacher can operate devices such as the electronic blackboard, multimedia podium, microphone, and speakers. There are eight possible gesture recognition results, each corresponding to a different gesture command meaning. The specific rules are shown in Table 1 below, which illustrates the correspondence between teacher gesture recognition results and gesture command meanings.

[0151] Table 1. Correspondence Rules between Teacher Gesture Recognition Results and Gesture Command Meanings

[0152] Gesture recognition results Meaning of gesture commands "make a fist" Open the electronic blackboard "1” Guide students to learn attention "2” Multimedia screen page turning "5” Multimedia screen magnification "6” Turn on the microphone "7” Multimedia screen shrink "Like" Control the sound system to encourage student performance in class. Interlace your index fingers Turn off electronic screens

[0153] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A classroom teaching gesture recognition method based on a chessboard self-attention mechanism, characterized in that, Including the following steps: The video data of the teacher under the four-sensor color linear scan camera and the point cloud information of the smart classroom under the 3D LiDAR were acquired respectively. The teacher's video data from the four-sensor color linear scan camera was divided into RGB image frames in chronological order. The teacher's RGB image frame is input into the gesture detection model to obtain the teacher's gesture detection results in the smart classroom; The teacher's gesture detection image is fused with point cloud information, and the data is further fed into a 3D gesture recognition model to identify the teacher's gesture results in the smart classroom. The results of the teacher's gesture recognition at different times are analyzed and fed back into the smart classroom for human-computer interaction, which plays a role in controlling teaching equipment; The data fusion module is used to fuse the teacher's gesture detection image with point cloud information; the 3D gesture recognition model includes a point cloud processing Transformer module, a joint estimation Transformer module, and a fully connected layer; the point cloud processing Transformer module includes an input embedding layer, a reference gesture extractor, four cascaded self-attention modules, and a Concat function; the joint estimation Transformer module follows a standard encoder-decoder architecture, including... A gesture recognition encoder, A gesture recognition decoder, an NC multi-head self-attention layer, and an ED multi-head attention layer; Step 2.1.1: The calibrator teacher has detected the hand image frame data and the point cloud information acquired by the lidar; Step 2.1.2, transform the camera coordinate system... Transformed to lidar coordinate system The image of the teacher's hand that has already been detected is used to reduce the recognition space for the 3D gesture recognition module, thereby reducing redundant background points in the point cloud and improving operating efficiency. The steps of the point cloud processing Transformer module are as follows: Step 2.2.1: Point cloud input. Select a set of disordered spatial coordinate points of the teacher's gestures. The spatial coordinate points of the teacher's gestures can be modeled into a series of vector representations: ; Step 2.2.2: Extract the reference gesture and parameterize it into a set of spatial joint coordinates in the camera coordinate system. ,in This indicates the number of hand joints. Based on relevant input points, a reference gesture extractor is used to generate a reference gesture. ; Step 2.2.3: Extract point cloud features, input point cloud First, the input embedding layer outputs a... 3D eigenvectors Then, the outputs of each self-attention layer are concatenated through the feature dimension: in, Indicates the first There are 1 self-attention layer, and each self-attention layer has the same output dimension as its input; These are the weights of the linear layer; The steps of the joint estimation Transformer module are as follows: Step 2.3.1: Obtain enhanced gesture keypoint features by downsampling and normalizing the N point set in the camera coordinate system, represented as follows: The normalized point P is fed into the gesture recognition encoder to generate enhanced gesture key points. ; Step 2.3.2: Fusion of enhanced gesture joint features with reference gesture. The reference gesture is further input into the gesture recognition decoder and its structural dependencies between joints are captured by an NC multi-head self-attention layer and used as an approximation of the target gesture. Step 2.3.3: Associate point-by-point enhancement of gesture key points By using the ED multi-head attention module, the captured pose relevance is combined with point-by-point augmentation of gesture key points. Correlation is used to discover specific jointly generated correlation points from different subspaces, and its decoding process can be formulated as follows: ,in These are model parameters.

2. The classroom teaching gesture recognition method based on a chessboard self-attention mechanism as described in claim 1, characterized in that, The data acquisition module consists of a four-sensor color linear scan camera and a 3D LiDAR, including the acquisition of image data and point cloud data; the image data is acquired by the four-sensor color linear scan camera; the point cloud data is acquired by the 3D LiDAR; the gesture detection model follows a typical encoder-decoder framework, including image preprocessing, backbone feature extraction Transformer, gesture detection Transformer, feedforward neural networks (FFNs), and loss calculation; the backbone feature extraction Transformer, abbreviated as BF-Transformer, consists of L feature extraction encoders; the gesture detection Transformer, abbreviated as HD-Transformer, consists of N sets of gesture detection encoders and decoders; The backbone feature extraction Transformer uses a chessboard self-attention mechanism to extract multi-scale teacher image features; The gesture detection Transformer uses the features of all frames of the teacher's image as input to detect and locate the position of the teacher's gesture; the feedforward neural network is used to predict the gesture position result; the loss calculation is used for gradient backpropagation to update the model parameters, thereby obtaining the final detection result.

3. The classroom teaching gesture recognition method based on a chessboard self-attention mechanism as described in claim 2, characterized in that, The image preprocessing module is used to acquire image frames, and its steps are as follows: Step 1.1.1: Select the RGB image frame of the classroom in the smart classroom as input, and perform a series of image enhancement and noise reduction processes; Step 1.1.2: Select a set of RGB images of teacher gestures as input, with an image size of [size missing]. First, the image is segmented into non-overlapping image patches through a patch partition layer. The size of each image patch is M × M, usually M = 4. The original value features are then projected onto an arbitrary dimension c through a linear embedding layer, usually c = 56.

4. The classroom teaching gesture recognition method based on a chessboard self-attention mechanism as described in claim 2, characterized in that, The training process of the Transformer model for backbone feature extraction is as follows: Step 1.2.1: Calculate the weighted sum of the self-attention coefficient matrix and the value matrix. The coefficient matrix of the self-attention... ,in In order to calculate the first The input feature representation of the layer requires a weighted sum of the coefficient matrix and value matrix of each self-attention layer, i.e. ; Step 1.2.2: Calculate multi-head self-attention separately. For its feature image, use a chessboard-like staggered window partitioning strategy starting from the top-left pixel. Therefore, the recursive formula for calculating continuous chessboard self-attention is: , , ,in, This represents the output characteristics of the W_MSA multi-head self-attention and MLP modules. This represents the output characteristics of the B_MSA multi-head self-attention and MLP modules. Representation layer normalization, The number of heads that represent attention; Step 1.2.3: Perform relative position encoding for each head. When calculating self-attention, there are two different methods of self-attention calculation; relative position encoding is then performed based on these methods. Therefore, each multi-head self-attention head ,in It is the query matrix in each feature extraction encoder. It is the keyword matrix in each feature extraction encoder. It is the value matrix in each feature extraction encoder, where d represents Dimensions Indicates batch size; Step 1.2.4: Obtain multi-scale features. Utilize local prior knowledge to aggregate adjacent small patches to obtain multi-scale features. Where Merging represents the aggregation operation. This represents a convolutional layer. Steps 1.2.1 and 1.2.2 are then repeated. The main purpose is to leverage spatial dimensions to obtain more channels, resulting in finer-grained global features. The final output multi-scale feature dimension is... .

5. The classroom teaching gesture recognition method based on a chessboard self-attention mechanism as described in claim 2, characterized in that, The training process of the gesture detection Transformer model is as follows: Step 1.3.1, Feature preprocessing, which involves processing high-dimensional features across multiple scales. ,pass Convolutional dimensionality reduction creates new feature maps. The gesture detection encoder takes the expected sequence as input and the feature map... Straighten into a one-dimensional sequence Therefore, the preprocessed features can be represented by the model vectorization. ; Step 1.3.2: Calculate the query matrix, keyword matrix, and value matrix in each gesture detection encoder, given a query. Keywords Sum Therefore, the query matrix in each gesture detection encoder Therefore, the keyword matrix in each gesture detection encoder Therefore, the value matrix in each gesture detection encoder Here, the learnable parameters are... , and Here the output feature matrix can be transformed into... ; Step 1.3.3, calculate multi-head self-attention, for each self-attention head. The calculation method is as follows Each of these self-attention heads ; Step 1.3.4, the decoding process of the gesture detection Transformer. The decoder follows the standard architecture of Transformer, and the input is represented as a value of size [size missing]. Each decoder layer can decode N objects in parallel, where N is typically 50 and C represents the dimension of the encoder's output features. These input embeddings are learnable positional codes that compute self-attention with global features in the encoder, thereby transforming the N input embeddings into N output embeddings.

6. The classroom teaching gesture recognition method based on a chessboard self-attention mechanism as described in claim 2, characterized in that, The feedforward neural network is used to predict gesture detection results; Step 1.4.1: Since the final features predicted by the decoder are a set of N fixed-size bounding boxes, where N is typically much larger than the actual number of objects of interest in the image, Used to indicate that there are no objects inside the prediction box; In step 1.4.2, the final prediction result is calculated by a 3-layer perceptron with an ELU activation function, a hidden layer, and a linear projection layer. FFNs predict the normalized center coordinates, height, and width of the input image box, and the linear layer uses the Softmax function to predict the class label.

7. The classroom teaching gesture recognition method based on a chessboard self-attention mechanism as described in claim 2, characterized in that, The Loss calculation is used to calculate the loss between the actual value and the predicted value; Step 1.5.1, set To represent the set of true values ​​of an object. To represent a set of N predictions, additionally use To indicate that no object was detected, or that it is considered to belong to the background category, calculate the optimal assignment. ,in It is the true value Index of predicted values The matching loss, and ; Step 1.5.2, to better fit the loss between the predicted and true values ​​and to determine the one-to-one correspondence between the true and predicted bounding boxes, the Hungarian algorithm is used to calculate the loss function. Therefore, ,in Gradient backpropagation is performed to update the model parameters, thereby obtaining the final detection result.

Images