A classroom behavior recognition method based on bidirectional LSTM and spatiotemporal dual-stream network

Abstract

The present invention discloses a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network, the method comprising the following steps: S1: designing a student classroom behavior recognition model suitable for classroom behavior recognition; S2: training the designed student classroom behavior recognition model to obtain a trained student classroom behavior recognition model; S3: using the trained student classroom behavior recognition model to recognize a classroom behavior recognition video, and outputting a prediction result for the entire video. The present invention can effectively extract the time and space information in the video sequence by designing a dual-stream convolutional neural network combined with a bidirectional LSTM module and dynamically assigning weights through an attention module, and perform time series modeling on this information, so as to recognize more complex and longer-lasting actions, thereby realizing accurate recognition of classroom behaviors.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition, and in particular to a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network. Background Art

[0002] Classroom behavior recognition is an important research field, especially in the field of education, it helps teachers better understand students' behavior and learning status, so as to adjust teaching strategies and improve teaching effectiveness. Traditional classroom behavior recognition methods mainly rely on teachers' direct observation and experience judgment, which is not only inefficient but also easily affected by subjective factors, leading to doubts about the accuracy and objectivity of the recognition results.

[0003] In recent years, with the continuous advancement of computer vision, artificial intelligence and other technologies, classroom behavior recognition methods based on deep learning have emerged. Classroom behavior recognition methods based on deep learning can automatically extract key information from classroom videos or images and identify different classroom behaviors by training models. Compared with traditional recognition methods, classroom behavior recognition methods based on deep learning have a higher degree of automation and stronger robustness, and can better meet the needs of modern education.

[0004] At present, classroom behavior recognition methods based on deep learning have achieved certain research results. However, existing methods still have some shortcomings. For example, some methods often cannot accurately distinguish different types of behaviors when dealing with complex classroom environments; in addition, students' behaviors are often continuous and dynamic, and the impact of time series information on behavior recognition needs to be considered. However, traditional deep learning models may have certain limitations when processing time series information, and it is difficult to fully capture the dynamic changes of behavior. This may lead to inaccurate or untimely recognition of classroom behaviors.

[0005] The Chinese patent with publication number CN114708525A discloses "a method and system for identifying student classroom behavior based on deep learning". It detects student targets in the extracted video frame image through the Yolov5 target detection model, constructs student target frames, and uses the improved ShuffleNetV2 classification model to analyze student images to obtain student classroom behavior labels. This method can identify student classroom behavior with high accuracy, help teachers review classroom situations, improve teaching levels and improve teaching methods. However, this method lacks modeling of student behavior time series information. When performing behavior recognition in classrooms with more dynamic behaviors and longer durations, poor recognition performance may occur. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention provides a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network, which aims to effectively extract the time and space information in the video sequence by designing a two-stream convolutional neural network combined with a bidirectional LSTM module and dynamically assigning weights through an attention module, and perform time series modeling on this information, so as to identify more complex and longer-lasting actions, thereby realizing accurate recognition of classroom behaviors.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The first aspect of the present invention provides a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network, the method comprising the following steps:

[0009] S1: Design a student classroom behavior recognition model suitable for classroom behavior recognition;

[0010] The student classroom behavior recognition model includes a video preprocessing module, a spatiotemporal feature extraction module, an LSTM module, an attention module and a classroom behavior prediction module; the specific steps include:

[0011] S11: First, the input classroom behavior video is sent to the video preprocessing module to extract multiple RGB image frames and continuous optical flow frames;

[0012] S12: sending the acquired RGB image frame and continuous optical flow frame to a spatiotemporal feature extraction module to extract features, and sending the extracted features to a spatiotemporal feature fusion module for feature fusion to obtain fused features;

[0013] S13: sending the obtained fused features to the LSTM module for time series information modeling;

[0014] S14: Send the features processed by the LSTM module to the attention module to obtain the output state vector;

[0015] S15: inputting the obtained output state vector into a classroom behavior prediction module to obtain a final behavior recognition prediction result;

[0016] S2: training the designed student classroom behavior recognition model to obtain a trained student classroom behavior recognition model;

[0017] S3: Use the trained student classroom behavior recognition model to recognize the classroom behavior recognition video and output the prediction results for the entire video.

[0018] As an embodiment of the present application, the step S11 specifically includes:

[0019] S111: segmenting the input classroom behavior video, cropping it into parts containing different classroom behavior segments, and extracting continuous frames of each segment as RGB image frame graphs;

[0020] S112: graying the obtained RGB image frame, then calculating the motion vector field of pixels between adjacent frames by using the selected optical flow algorithm, and finally visualizing the direction and size of the motion vector by color coding to generate a series of continuous optical flow frame images;

[0021] S113: The sizes of the obtained RGB image frame and optical flow frame are preprocessed to 256×256. For the optical flow frame, every 10 frames are used as a time step. Then, the RGB image frame and optical flow frame are divided into a training set and a test set in a ratio of 8:2 and sent to the spatiotemporal feature extraction module for training.

[0022] As an embodiment of the present application, the spatiotemporal feature extraction module in step S12 includes a spatial stream convolutional neural network and a temporal stream convolutional neural network. The spatial stream convolutional neural network and the temporal stream convolutional neural network have the same network structure, both including 5 convolutional layers and 3 fully connected layers. The spatial stream convolutional neural network receives a single RGB image frame, and the temporal stream convolutional neural network receives a continuous optical flow frame.

[0023] As an embodiment of the present application, the step S12 specifically includes:

[0024] S121: First, the RGB image frame and the continuous optical flow frame are respectively sent to the first convolution layer of the spatial stream convolutional neural network and the temporal stream convolutional neural network, and 256 convolution kernels with a size of 7×7 and a step size of 2 are used to perform preliminary feature extraction on the image to obtain a feature map of the first stage;

[0025] S122: The feature map of the first stage is further processed by a second convolution layer, where the second convolution layer uses 256 convolution kernels of a size of 5×5 and a step size of 2 to obtain a feature map of the second stage;

[0026] S123: Then, the feature map of the second stage is sequentially passed to three consecutive convolutional layers, and the three consecutive convolutional layers are all equipped with 512 convolution kernels of size 3×3 and step size 1, so as to further deepen the feature hierarchy to obtain the processed feature map;

[0027] S124: In the temporal stream convolutional neural network, the continuous optical flow frame image is subjected to feature extraction in each convolutional layer, and then the output result of each layer is transmitted to the subsequent convolutional layer, and the feature map is also transmitted to the corresponding layer of the spatial stream convolutional neural network, so as to realize joint learning of two information flows;

[0028] S125: sending the feature maps processed by the spatial stream convolutional neural network and the temporal stream convolutional neural network respectively to the first fully connected layer containing 4096 neurons in their respective networks for feature integration;

[0029] S126: Then it is sent to the second fully connected layer containing 2048 neurons to further compress and optimize the feature representation;

[0030] S127: Continue to feed into the third fully connected layer containing 256 neurons and output features;

[0031] S128: Send the spatial features output by the spatial stream convolutional neural network and the temporal features output by the temporal stream convolutional neural network to a spatiotemporal feature fusion module for fusion to obtain a fused feature vector.

[0032] As an embodiment of the present application, the LSTM module in step S13 includes a forward propagation LSTM and a backward propagation LSTM, and the specific steps include:

[0033] S131: In each time step, the fused feature vector obtained by the spatiotemporal feature extraction module is It is fed into the forward propagation LSTM and the backward propagation LSTM. When it is fed into the forward propagation LSTM, the unit layer weight is set to , and obtain an LSTM unit consisting of a feature sequence from front to back , when the obtained feature sequence is sent to the unit layer at the next moment, the weight ratio is , when sent to the final output at that moment, the weight ratio is , when fed into the back-propagation LSTM, the unit layer weight is set to , and obtain an LSTM unit consisting of a feature sequence from front to back When the obtained feature sequence is sent to the unit layer of the previous moment, the weight ratio is , when sent to the final output at that moment, the weight ratio is , the calculation formulas of the two LSTM units are as follows:

[0034]

[0035]

[0036] in, For the The fused feature vector obtained by the spatiotemporal feature fusion module at each time step is is the activation function, , is bias;

[0037] S132: The feature vector of the corresponding moment of the forward propagation LSTM output , the back-propagated LSTM outputs the feature vector at the corresponding moment , the calculation formula is as follows:

[0038]

[0039]

[0040] in, is the activation function, is bias;

[0041] S133: The obtained feature vector With the eigenvector Average to get the output of the LSTM module at the corresponding time , and its corresponding formula is as follows:

[0042] .

[0043] As an embodiment of the present application, the attention module in step S14 specifically includes:

[0044] S141: Construct an input sequence from multiple outputs of the LSTM module ,in is the first The output of the time step is is the number of time steps;

[0045] S142: Use a multi-layer perceptron MLP to calculate an attention score for each element in the input sequence ;

[0046] Attention score The calculation formula is as follows:

[0047]

[0048] in, and is a learnable weight matrix, is a weight vector, is the number of time steps, is a context vector;

[0049] S143: Use the normalized exponential function to normalize the attention score to obtain the attention weight ; The attention weight The calculation formula is as follows:

[0050]

[0051] S144: Use these attention weights to perform weighted summation on the input sequence to obtain a weighted feature vector , and its calculation formula is as follows:

[0052] .

[0053] As an embodiment of the present application, the classroom behavior prediction module in step S15 includes two fully connected layers and one normalized index layer, and the specific steps include:

[0054] S151: The feature vector weighted by the attention module is input into the first fully connected layer containing 256 neurons for further feature extraction and integration;

[0055] S152: The output of the first fully connected layer is input into the second fully connected layer containing 128 neurons to extract high-level features;

[0056] S153: The output of the second fully connected layer is input into the normalized exponential layer, which converts the output into a probability distribution and takes the category with the highest probability value as the final prediction result. At this time, the number of neurons contained in the normalized exponential layer is the same as the number in the classroom behavior category, and the activation function used is the normalized exponential function.

[0057] As an embodiment of the present application, the loss function is used in step S2 to train the designed student classroom behavior recognition model, and the loss function includes a spatiotemporal feature extraction module loss function , LSTM module loss function , attention module loss function , loss function of classroom behavior prediction module ;

[0058] The loss function of the spatiotemporal feature extraction module The formula is as follows:

[0059]

[0060]

[0061]

[0062] in, is the spatial flow loss function, is the temporal flow loss function, is the sample size, It is a sample The real label, and The spatial stream and the temporal stream are samples The predicted probability of

[0063] The LSTM module loss function The formula is as follows:

[0064]

[0065]

[0066] in, is the number of behavior categories, is the true label in the category The one-hot encoded value on , is the probability distribution of the model predictions The probability of , which is obtained by passing the output of the LSTM module at the corresponding moment through a fully connected layer, For the LSTM module The loss of the output of time steps, is the number of time steps;

[0067] The attention module loss function The formula is as follows:

[0068]

[0069]

[0070] in, Yes Category The weight of For the LSTM module The loss of the output of time steps, is the number of time steps;

[0071] The classroom behavior prediction module loss function The formula is as follows:

[0072]

[0073] in, is the number of time steps, It is The actual behavior value output in each time step, is the behavioral value predicted by the model.

[0074] The beneficial effects of the present invention are:

[0075] 1. The present invention designs a dual-stream convolutional neural network, a time stream convolutional neural network and a space stream convolutional neural network to fully extract the time information (motion information) and space information (static appearance information) from the input video, and models the extracted time series information by designing a bidirectional LSTM module, so that the model can more accurately identify complex behaviors in the classroom;

[0076] 2. The present invention realizes the dynamic allocation of different weights to the output video frame sequence by designing an attention module, thereby highlighting the importance of certain frames and improving the accuracy of classroom behavior recognition;

[0077] 3. By designing an innovative loss function, the present invention enables faster fitting during model training, with better results and is more suitable for classroom behavior recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A technical solution flow chart of a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network provided in an embodiment of the present invention;

[0079] Figure 2 A classroom behavior recognition model structure diagram of a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network provided in an embodiment of the present invention;

[0080] Figure 3 A schematic diagram of a spatiotemporal feature extraction module of a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network provided in an embodiment of the present invention;

[0081] Figure 4 A schematic diagram of an LSTM module of a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network provided in an embodiment of the present invention;

[0082] Figure 5 A schematic diagram of an attention module of a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network provided in an embodiment of the present invention;

[0083] Figure 6 A schematic diagram of a classroom behavior prediction module of a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0086] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0087] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme that satisfies both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0088] Reference Figure 1-Figure 6 The first aspect of the present invention provides a classroom behavior recognition method based on a bidirectional LSTM and a spatiotemporal dual-stream network, the method comprising the following steps:

[0089] S1: Design a student classroom behavior recognition model suitable for classroom behavior recognition;

[0090] The student classroom behavior recognition model includes a video preprocessing module, a spatiotemporal feature extraction module, an LSTM module, an attention module and a classroom behavior prediction module; the specific steps include:

[0091] S11: First, the input classroom behavior video is sent to the video preprocessing module to extract multiple RGB image frames and continuous optical flow frames;

[0092] S12: sending the acquired RGB image frame and continuous optical flow frame to a spatiotemporal feature extraction module to extract features, and sending the extracted features to a spatiotemporal feature fusion module for feature fusion to obtain fused features;

[0093] S13: sending the obtained fused features to the LSTM module for time series information modeling;

[0094] S14: Send the features processed by the LSTM module to the attention module to obtain the output state vector;

[0095] S15: inputting the obtained output state vector into a classroom behavior prediction module to obtain a final behavior recognition prediction result;

[0096] S2: training the designed student classroom behavior recognition model to obtain a trained student classroom behavior recognition model;

[0097] S3: Use the trained student classroom behavior recognition model to recognize the classroom behavior recognition video and output the prediction results for the entire video.

[0098] As an embodiment of the present application, the step S11 specifically includes:

[0099] S111: segmenting the input classroom behavior video, cropping it into parts containing different classroom behavior segments, and extracting continuous frames of each segment as RGB image frame graphs;

[0100] S112: graying the obtained RGB image frame, then calculating the motion vector field of pixels between adjacent frames by using the selected optical flow algorithm, and finally visualizing the direction and size of the motion vector by color coding to generate a series of continuous optical flow frame images;

[0101] S113: The sizes of the obtained RGB image frame and optical flow frame are preprocessed to 256×256. For the optical flow frame, every 10 frames are used as a time step. Then, the RGB image frame and optical flow frame are divided into a training set and a test set in a ratio of 8:2 and sent to the spatiotemporal feature extraction module for training.

[0102] As an embodiment of the present application, the spatiotemporal feature extraction module in step S12 includes a spatial stream convolutional neural network and a temporal stream convolutional neural network. The spatial stream convolutional neural network and the temporal stream convolutional neural network have the same network structure, both including 5 convolutional layers and 3 fully connected layers. The spatial stream convolutional neural network receives a single RGB image frame, and the temporal stream convolutional neural network receives a continuous optical flow frame.

[0103] Specifically, the spatiotemporal feature extraction module can fully extract the temporal information (motion information) and spatial information (static appearance information) in the input video, and effectively fuse the extracted information to form a more comprehensive and richer feature representation, so that the model can more accurately identify complex behaviors in the classroom.

[0104] As an embodiment of the present application, the step S12 specifically includes:

[0105] S121: First, the RGB image frame and the continuous optical flow frame are respectively sent to the first convolution layer of the spatial stream convolutional neural network and the temporal stream convolutional neural network, and 256 convolution kernels with a size of 7×7 and a step size of 2 are used to perform preliminary feature extraction on the image to obtain a feature map of the first stage;

[0106] S122: The feature map of the first stage is further processed by a second convolution layer, where the second convolution layer uses 256 convolution kernels of a size of 5×5 and a step size of 2 to extract more refined features, thereby obtaining a feature map of the second stage;

[0107] S123: Then, the feature map of the second stage is sequentially passed to three consecutive convolutional layers, and the three consecutive convolutional layers are all equipped with 512 convolution kernels of size 3×3 and step size 1, so as to further deepen the feature hierarchy to obtain the processed feature map;

[0108] S124: In the temporal stream convolutional neural network, the continuous optical flow frame image is subjected to feature extraction in each convolutional layer, and then the output result of each layer is transmitted to the subsequent convolutional layer, and the feature map is also transmitted to the corresponding layer of the spatial stream convolutional neural network, so as to realize joint learning of two information flows;

[0109] S125: sending the feature maps processed by the spatial stream convolutional neural network and the temporal stream convolutional neural network to the first fully connected layer containing 4096 neurons in their respective networks, and the output dimension of the feature will become 4096, in order to perform higher-level feature integration;

[0110] S126: Then the data stream is sent to the second fully connected layer containing 2048 neurons, and the output dimension of the feature is changed from 4096 to 2048, further compressing and optimizing the feature representation;

[0111] S127: The data is continuously fed into the third fully connected layer containing 256 neurons and outputs features. The output dimension of the features changes from 2048 to 256 in preparation for feature fusion.

[0112] S128: Send the spatial features output by the spatial stream convolutional neural network and the temporal features output by the temporal stream convolutional neural network to the spatiotemporal feature fusion module for fusion to obtain a fused feature vector, providing a rich information basis for subsequent tasks.

[0113] As an embodiment of the present application, the LSTM module in step S13 includes a forward propagation LSTM and a backward propagation LSTM. Such a design can obtain more relevant information before and after the video, and more fully consider the timing information of the video frame. The specific steps include:

[0114] S131: In each time step, the fused feature vector obtained by the spatiotemporal feature extraction module is It is fed into the forward propagation LSTM and the backward propagation LSTM. When it is fed into the forward propagation LSTM, the unit layer weight is set to , and obtain an LSTM unit consisting of a feature sequence from front to back , when the obtained feature sequence is sent to the unit layer at the next moment, the weight ratio is , when sent to the final output at that moment, the weight ratio is , when fed into the back-propagation LSTM, the unit layer weight is set to , and obtain an LSTM unit consisting of a feature sequence from front to back When the obtained feature sequence is sent to the unit layer of the previous moment, the weight ratio is , when sent to the final output at that moment, the weight ratio is , the calculation formulas of the two LSTM units are as follows:

[0115]

[0116]

[0117] in, For the The fused feature vector obtained by the spatiotemporal feature fusion module at each time step is is the activation function, , is bias;

[0118] S132: The feature vector of the corresponding moment of the forward propagation LSTM output , the back-propagated LSTM outputs the feature vector at the corresponding moment , the calculation formula is as follows:

[0119]

[0120]

[0121] in, is the activation function, is bias;

[0122] S133: The obtained feature vector With the eigenvector Average to get the output of the LSTM module at the corresponding time , and its corresponding formula is as follows:

[0123] .

[0124] Specifically, the LSTM module can simultaneously consider past and future contextual information through processing in both forward and reverse directions, learn more complex sequence patterns and dependencies, more accurately model long-term dependencies in sequences, and capture multiple aspects and details of student behaviors, including previous behaviors, subsequent behaviors, and the relationships between them, thereby improving the accuracy of the model in identifying classroom behaviors.

[0125] As an embodiment of the present application, the attention module in step S14 specifically includes:

[0126] S141: Construct an input sequence from multiple outputs of the LSTM module ,in is the first The output of the time step is is the number of time steps;

[0127] S142: Use a multi-layer perceptron MLP to calculate an attention score for each element in the input sequence ;

[0128] Attention score The calculation formula is as follows:

[0129]

[0130] in, and is a learnable weight matrix, is a weight vector, is the number of time steps, is a context vector;

[0131] S143: Use the normalized exponential function to normalize the attention score to obtain the attention weight ; The attention weight The calculation formula is as follows:

[0132]

[0133] S144: Use these attention weights to perform weighted summation on the input sequence to obtain a weighted feature vector , and its calculation formula is as follows:

[0134] .

[0135] Specifically, the attention module dynamically assigns different weights to the output video frame sequence, thereby highlighting the importance of certain frames and improving the accuracy of classroom behavior recognition.

[0136] As an embodiment of the present application, the classroom behavior prediction module in step S15 includes two fully connected layers and a normalized index layer, which can accurately predict the classroom behavior results based on the previous output. The specific steps include:

[0137] S151: The feature vector weighted by the attention module is input into the first fully connected layer containing 256 neurons for further feature extraction and integration;

[0138] S152: The output of the first fully connected layer is input into the second fully connected layer containing 128 neurons to extract high-level features;

[0139] S153: The output of the second fully connected layer is input into the normalized exponential layer, which converts the output into a probability distribution and takes the category with the highest probability value as the final prediction result. At this time, the number of neurons contained in the normalized exponential layer is the same as the number in the classroom behavior category, and the activation function used is the normalized exponential function.

[0140] As an embodiment of the present application, the loss function is used in step S2 to train the designed student classroom behavior recognition model, and the loss function includes a spatiotemporal feature extraction module loss function , LSTM module loss function , attention module loss function , loss function of classroom behavior prediction module ;

[0141] The loss function of the spatiotemporal feature extraction module The formula is as follows:

[0142]

[0143]

[0144]

[0145] in, is the spatial flow loss function, is the temporal flow loss function, is the sample size, It is a sample The real label, and The spatial stream and the temporal stream are samples The predicted probability of

[0146] The LSTM module loss function The formula is as follows:

[0147]

[0148]

[0149] in, is the number of behavior categories, is the true label in the category The one-hot encoded value on , is the probability distribution of the model predictions The probability of , which is obtained by passing the output of the LSTM module at the corresponding moment through a fully connected layer, For the LSTM module The loss of the output of time steps, is the number of time steps;

[0150] The attention module loss function The formula is as follows:

[0151]

[0152]

[0153] in, Yes Category The weight of For the LSTM module The loss of the output of time steps, is the number of time steps;

[0154] The classroom behavior prediction module loss function The formula is as follows:

[0155]

[0156] in, is the number of time steps, It is The actual behavior value output in each time step (can be a continuous value or a measurement value), is the behavioral value predicted by the model.

[0157] Specifically, the present invention uses an innovative loss function to enable faster fitting during model training, while achieving better results and being more suitable for classroom behavior recognition scenarios.

[0158] Specifically, the present invention first pre-processes the classroom video data to be analyzed by loading pre-trained time stream and space stream convolutional neural networks, and inputs it into the network to extract its spatial information (static appearance information) and time information (motion information) in a time step of 10 frames; then the extracted information is feature fused to obtain a fused feature vector; then the fused feature vector is input into a bidirectional LSTM module for temporal information modeling to capture the bidirectional dependency in the video sequence and improve the accuracy of behavior recognition; then the output of the LSTM module is sequentially sent to the attention module to assign different weights to the sequence, thereby highlighting the importance of certain frames; finally, the output of the attention module is sent to the classroom behavior prediction module to obtain the classroom behavior recognition result, thereby improving the accuracy of the model for classroom behavior recognition.

[0159] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A classroom behavior recognition method based on bidirectional LSTM and spatiotemporal dual-stream network, characterized in that: The method comprises the following steps: S1: Design a student classroom behavior recognition model suitable for classroom behavior recognition; The student classroom behavior recognition model includes a video preprocessing module, a spatiotemporal feature extraction module, an LSTM module, an attention module and a classroom behavior prediction module; the specific steps include: S11: First, the input classroom behavior video is sent to the video preprocessing module to extract multiple RGB image frames and continuous optical flow frames; S12: sending the acquired RGB image frame and continuous optical flow frame to a spatiotemporal feature extraction module to extract features, and fusing the extracted features to obtain fused features; S13: sending the obtained fused features to the LSTM module for time series information modeling; S14: Send the features processed by the LSTM module to the attention module to obtain the output state vector; S15: inputting the obtained output state vector into a classroom behavior prediction module to obtain a final behavior recognition prediction result; S2: training the designed student classroom behavior recognition model to obtain a trained student classroom behavior recognition model; S3: Use the trained student classroom behavior recognition model to recognize the classroom behavior recognition video and output the prediction results for the entire video; The step S11 specifically includes: S111: segmenting the input classroom behavior video, cropping it into parts containing different classroom behavior segments, and extracting continuous frames of each segment as RGB image frame graphs; S112: graying the obtained RGB image frame, then calculating the motion vector field of pixels between adjacent frames by using the selected optical flow algorithm, and finally visualizing the direction and size of the motion vector by color coding to generate a series of continuous optical flow frame images; S113: preprocessing the size of the obtained RGB image frame and the optical flow frame to 256×256, and for the optical flow frame, every 10 frames are used as a time step, and then the RGB image frame and the optical flow frame are divided into a training set and a test set in a ratio of 8:2 and sent to the spatiotemporal feature extraction module for training; The spatiotemporal feature extraction module in step S12 includes a spatial stream convolutional neural network and a temporal stream convolutional neural network, the spatial stream convolutional neural network and the temporal stream convolutional neural network have the same network structure, both including 5 convolutional layers and 3 fully connected layers, the spatial stream convolutional neural network receives a single RGB image frame, and the temporal stream convolutional neural network receives a continuous optical flow frame; The step S12 specifically includes: S121: First, the RGB image frame and the continuous optical flow frame are respectively sent to the first convolution layer of the spatial stream convolutional neural network and the temporal stream convolutional neural network, and 256 convolution kernels with a size of 7×7 and a step size of 2 are used to perform preliminary feature extraction on the image to obtain a feature map of the first stage; S122: The feature map of the first stage is further processed by a second convolution layer, where the second convolution layer uses 256 convolution kernels of a size of 5×5 and a step size of 2 to obtain a feature map of the second stage; S123: Then, the feature map of the second stage is sequentially passed to three consecutive convolutional layers, and the three consecutive convolutional layers are all equipped with 512 convolution kernels of size 3×3 and step size 1, so as to further deepen the feature hierarchy to obtain the processed feature map; S124: In the temporal stream convolutional neural network, the continuous optical flow frame image is subjected to feature extraction in each convolutional layer, and then the output result of each layer is transmitted to the subsequent convolutional layer, and the feature map is also transmitted to the corresponding layer of the spatial stream convolutional neural network, so as to realize joint learning of two information flows; S125: sending the feature maps processed by the spatial stream convolutional neural network and the temporal stream convolutional neural network respectively to the first fully connected layer containing 4096 neurons in their respective networks for feature integration; S126: Then it is sent to the second fully connected layer containing 2048 neurons to further compress and optimize the feature representation; S127: Continue to feed into the third fully connected layer containing 256 neurons and output features; S128: Fusing the spatial features output by the spatial stream convolutional neural network and the temporal features output by the temporal stream convolutional neural network to obtain a fused feature vector; In step S2, the designed student classroom behavior recognition model is trained using a loss function, and the loss function includes a spatiotemporal feature extraction module loss function. , LSTM module loss function , attention module loss function , loss function of classroom behavior prediction module ; The loss function of the spatiotemporal feature extraction module The formula is as follows: in, is the spatial flow loss function, is the temporal flow loss function, is the sample size, It is a sample The real label, and The spatial stream and the temporal stream are samples The predicted probability of The LSTM module loss function The formula is as follows: in, is the number of behavior categories, is the true label in the category The one-hot encoded value on , is the probability distribution of the category predicted by the model The probability of , which is obtained by passing the output of the LSTM module at the corresponding moment through a fully connected layer, For the LSTM module The loss of the output of time steps, is the number of time steps; The attention module loss function The formula is as follows: in, Yes Category The weight of For the LSTM module The loss of the output of time steps, is the number of time steps; The classroom behavior prediction module loss function The formula is as follows: in, is the number of time steps, It is The actual behavior value output in each time step, is the behavioral value predicted by the model.

2. According to claim 1, a classroom behavior recognition method based on bidirectional LSTM and spatiotemporal dual-stream network is characterized in that: The LSTM module in step S13 includes a forward propagation LSTM and a backward propagation LSTM, and the specific steps include: S131: In each time step, the fused feature vector obtained by the spatiotemporal feature extraction module is It is fed into the forward propagation LSTM and the backward propagation LSTM. When it is fed into the forward propagation LSTM, the unit layer weight is set to , and obtain an LSTM unit consisting of a feature sequence from front to back , when the obtained feature sequence is sent to the unit layer at the next moment, the weight ratio is , when sent to the final output at that moment, the weight ratio is , when fed into the back-propagation LSTM, the unit layer weight is set to , and obtain an LSTM unit consisting of a feature sequence from front to back When the obtained feature sequence is sent to the unit layer of the previous moment, the weight ratio is , when sent to the final output at that moment, the weight ratio is , the calculation formulas of the two LSTM units are as follows: in, For the The fused feature vector obtained by the spatiotemporal feature fusion module at each time step is is the activation function, , is bias; S132: The feature vector of the corresponding moment of the forward propagation LSTM output , the back-propagated LSTM outputs the feature vector at the corresponding moment , the calculation formula is as follows: in, is the activation function, is bias; S133: The obtained feature vector With the eigenvector Average to get the output of the LSTM module at the corresponding time , and its corresponding formula is as follows: 。 3. According to claim 1, a classroom behavior recognition method based on bidirectional LSTM and spatiotemporal dual-stream network is characterized in that: The attention module in step S14 specifically includes: S141: Construct an input sequence from multiple outputs of the LSTM module ,in is the first The output of the time step is is the number of time steps; S142: Use a multi-layer perceptron MLP to calculate an attention score for each element in the input sequence ; Attention score The calculation formula is as follows: in, and is a learnable weight matrix, is a weight vector, is the number of time steps, is a context vector; S143: Use the normalized exponential function to normalize the attention score to obtain the attention weight ; The attention weight The calculation formula is as follows: S144: Use these attention weights to perform weighted summation on the input sequence to obtain a weighted feature vector , and its calculation formula is as follows: 。 4. According to claim 1, a classroom behavior recognition method based on bidirectional LSTM and spatiotemporal dual-stream network is characterized in that: The classroom behavior prediction module in step S15 includes two fully connected layers and a normalized index layer, and the specific steps include: S151: The feature vector weighted by the attention module is input into the first fully connected layer containing 256 neurons for further feature extraction and integration; S152: The output of the first fully connected layer is input into the second fully connected layer containing 128 neurons to extract high-level features; S153: The output of the second fully connected layer is input into the normalized exponential layer, which converts the output into a probability distribution and takes the category with the highest probability value as the final prediction result. At this time, the number of neurons contained in the normalized exponential layer is the same as the number in the classroom behavior category, and the activation function used is the normalized exponential function.

Images