A method and system for classroom student action recognition based on deep learning

Abstract

The present invention provides a method and system for classroom student action recognition based on deep learning in the field of action recognition technology, the method comprising: step S10, editing the captured classroom video to obtain several classroom sub-videos; step S20, sequentially detecting the faces in each classroom sub-video through a face detection algorithm, and then editing the classroom sub-video into several face sub-videos; step S30, respectively extracting static feature images and dynamic feature images of each face sub-video, and constructing a training data set based on each static feature image and dynamic feature image; step S40, creating an action recognition model based on deep learning, and training the action recognition model using the training data set; step S50, using the trained action recognition model to perform real-time action recognition on the classroom video captured by the camera. The advantages of the present invention are: realizing real-time and accurate recognition of student actions, and timely adjusting teaching strategies, thereby greatly improving the teaching quality.

Description

Technical Field

[0001] The present invention relates to the field of motion recognition technology, and in particular to a method and system for identifying student motions in a classroom based on deep learning. Background Art

[0002] The concentration of students in the classroom is often a key factor in judging the quality of classroom teaching, and the actions of students in the classroom can often reflect the degree of their concentration. For example, the action of students sleeping can reflect the students' lack of concentration. Therefore, it is very necessary to identify the actions of students in the classroom.

[0003] Traditionally, teachers judge the concentration of students in class. When teachers find that a student is absent-minded, they will call the roll or walk up to him / her. However, teachers' main focus during teaching is often on teaching content. It is unrealistic to rely solely on teachers to judge whether all students in a class are focused. There are often some students who slip through the net, especially in large-class teaching scenarios.

[0004] Therefore, how to provide a classroom student action recognition method and system based on deep learning to achieve real-time and accurate recognition of student actions has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for classroom student action recognition based on deep learning, so as to realize real-time and accurate recognition of student actions.

[0006] In a first aspect, the present invention provides a method for identifying student actions in a classroom based on deep learning, comprising the following steps:

[0007] Step S10, capturing a large number of classroom videos, and editing the classroom videos to obtain a number of classroom sub-videos;

[0008] Step S20, detecting faces in each of the classroom sub-videos in turn by using a face detection algorithm, and then editing the classroom sub-videos into a plurality of face sub-videos;

[0009] Step S30, respectively extracting static feature images and dynamic feature images of each of the face sub-videos, and constructing a training data set based on the static feature images and dynamic feature images;

[0010] Step S40: creating an action recognition model based on deep learning, and training the action recognition model using the training data set;

[0011] Step S50: Use the trained action recognition model to perform real-time action recognition on the classroom video captured by the camera.

[0012] Furthermore, the step S10 is specifically as follows:

[0013] A large number of classroom videos are captured by a camera installed in the classroom, and the classroom videos are sequentially edited at preset time intervals to obtain a number of classroom sub-videos.

[0014] Furthermore, the step S20 is specifically as follows:

[0015] First, each of the classroom sub-videos is subjected to noise reduction processing, and then the faces in each of the classroom sub-videos are detected in turn using the Viola-Jones face detection algorithm. The face area of ​​the detected face is marked by a rectangular frame, and the face area marked by each rectangular frame is edited to obtain a number of face sub-videos.

[0016] Furthermore, the step S30 specifically includes:

[0017] Step S31, randomly selecting a first picture frame from each of the face sub-videos as a static feature picture;

[0018] Step S32, after each of the static feature pictures, extract a second picture every 8 frames, and extract 4 frames of the second pictures in sequence;

[0019] Step S33, extracting the horizontal optical flow and the vertical optical flow from each of the static feature pictures and the second picture by using the TV_L1 method, and superimposing the horizontal optical flow and the vertical optical flow to obtain a dynamic feature picture;

[0020] Step S34: construct a training data set based on the static feature images and the dynamic feature images.

[0021] Further, in step S40, the action recognition model includes a spatial stream module for extracting static feature vectors, a temporal stream module for extracting dynamic feature vectors, and a first fully connected layer for concatenating the static feature vectors and the dynamic feature vectors; the output ends of the spatial stream module and the temporal stream module are connected to the input end of the first fully connected layer;

[0022] The spatial stream module consists of an Inception V3 unit pre-trained using ImageNet and a second fully connected layer;

[0023] The time series flow module consists of five convolutional layers and a third fully connected layer; each convolutional layer uses a convolution kernel with a size of 3×3 and a step size of 1; the time series flow module uses the ReLU function as an activation function.

[0024] In a second aspect, the present invention provides a classroom student action recognition system based on deep learning, comprising the following modules:

[0025] A classroom video editing module is used to capture a large number of classroom videos and edit the classroom videos to obtain a number of classroom sub-videos;

[0026] A face detection module, used to detect faces in each of the classroom sub-videos in turn by using a face detection algorithm, and then edit the classroom sub-videos into a plurality of face sub-videos;

[0027] A training data set construction module, used to extract the static feature pictures and dynamic feature pictures of each face sub-video respectively, and construct a training data set based on each static feature picture and dynamic feature picture;

[0028] An action recognition model training module is used to create an action recognition model based on deep learning, and train the action recognition model using the training data set;

[0029] The action recognition module is used to use the trained action recognition model to perform real-time action recognition on the classroom video captured by the camera.

[0030] Furthermore, the classroom video editing module is specifically:

[0031] A large number of classroom videos are captured by a camera installed in the classroom, and the classroom videos are sequentially edited at preset time intervals to obtain a number of classroom sub-videos.

[0032] Furthermore, the face detection module is specifically:

[0033] First, each of the classroom sub-videos is subjected to noise reduction processing, and then the faces in each of the classroom sub-videos are detected in turn using the Viola-Jones face detection algorithm. The face area of ​​the detected face is marked by a rectangular frame, and the face area marked by each rectangular frame is edited to obtain a number of face sub-videos.

[0034] Furthermore, the training data set construction module specifically includes:

[0035] A static feature image selection unit, used to randomly select a first image frame from each of the face sub-videos as a static feature image;

[0036] A second picture selection unit is used to extract a second picture frame every 8 frames after each of the static feature pictures, and sequentially extract 4 frames of the second picture;

[0037] A dynamic feature picture extraction unit, used for extracting a horizontal optical flow and a vertical optical flow from each of the static feature pictures and the second picture by using a TV_L1 method, and superimposing the horizontal optical flow and the vertical optical flow to obtain a dynamic feature picture;

[0038] The static and dynamic feature image integration unit is used to construct a training data set based on the static feature images and the dynamic feature images.

[0039] Furthermore, in the action recognition model training module, the action recognition model includes a spatial stream module for extracting static feature vectors, a temporal stream module for extracting dynamic feature vectors, and a first fully connected layer for splicing the static feature vectors and the dynamic feature vectors; the output ends of the spatial stream module and the temporal stream module are connected to the input end of the first fully connected layer;

[0040] The spatial stream module consists of an Inception V3 unit pre-trained using ImageNet and a second fully connected layer;

[0041] The time series flow module consists of five convolutional layers and a third fully connected layer; each convolutional layer uses a convolution kernel with a size of 3×3 and a step size of 1; the time series flow module uses the ReLU function as an activation function.

[0042] The advantages of the present invention are:

[0043] By pre-capturing a large number of classroom videos and dividing them into several classroom sub-videos, detecting the faces in each classroom sub-video and then editing the classroom sub-videos into several face sub-videos, and then extracting the static feature images and dynamic feature images of each face sub-video to construct a training data set, and then using the training data set to train the created action recognition model, finally using the trained action recognition model to perform real-time action recognition on the captured classroom videos, that is, using the trained two-stream convolutional network (spatial stream module, temporal stream module) for action recognition, which not only has fast and accurate classification speed, but also has strong generalization and robustness, and ultimately realizes real-time and accurate recognition of student actions, and timely adjusts teaching strategies, thereby greatly improving the teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0045] Figure 1 It is a flow chart of a method for identifying classroom student movements based on deep learning of the present invention.

[0046] Figure 2 It is a structural schematic diagram of a classroom student action recognition system based on deep learning of the present invention.

[0047] Figure 3 It is a schematic diagram of four rectangular features of Haar features in the Viola-Jones face detection algorithm of the present invention.

[0048] Figure 4It is a schematic diagram of the generation of dynamic feature pictures of the present invention.

[0049] Figure 5 It is a schematic diagram of the action recognition model of the present invention.

[0050] Figure 6 Schematic diagram of the Inception V3 unit of the present invention. DETAILED DESCRIPTION

[0051] The technical solution in the embodiments of the present application has the following overall idea: detect the faces in each classroom sub-video and then edit the classroom sub-video into several face sub-videos, then extract the static feature images and dynamic feature images of each face sub-video to construct a training data set, then use the training data set to train the created action recognition model, and finally use the trained action recognition model to perform real-time action recognition on the classroom video, so as to realize real-time and accurate recognition of student actions, and assist teachers in judging students' concentration.

[0052] Please refer to Figures 1 to 6 As shown, a preferred embodiment of a method for identifying student actions in a classroom based on deep learning of the present invention comprises the following steps:

[0053] Step S10: The social robot captures a large number of classroom videos, and edits the classroom videos to obtain a number of classroom sub-videos;

[0054] Step S20: The social robot detects faces in each of the classroom sub-videos in turn by using a face detection algorithm, and then edits the classroom sub-videos into a plurality of face sub-videos;

[0055] Step S30: The social robot extracts the static feature images and the dynamic feature images of each of the face sub-videos, and constructs a training data set based on the static feature images and the dynamic feature images;

[0056] Step S40: the social robot creates an action recognition model based on deep learning, and trains the action recognition model using the training data set; that is, the training data set is divided into a training set and a verification set according to a preset ratio, the action recognition model is trained using the training set, and the trained action recognition model is verified using the verification set;

[0057] Step S50: The social robot uses the trained action recognition model to perform real-time action recognition on the classroom video captured by the camera, and periodically sends the recognition results to the teaching terminal; that is, the static feature images and dynamic feature images in the classroom video are extracted and input into the trained action recognition model for action recognition.

[0058] The step S10 is specifically as follows:

[0059] The social robot captures a large amount of classroom videos through a camera installed in the classroom, and sequentially edits the classroom videos at preset time intervals to obtain several classroom sub-videos, for example, editing to generate several classroom sub-videos at intervals of every 4 seconds.

[0060] To ensure that students' movements can be captured, the camera's frame rate should be greater than 25fps.

[0061] The step S20 is specifically as follows:

[0062] First, each classroom sub-video is subjected to noise reduction processing to remove redundant information, and then the faces in each classroom sub-video are detected in turn by using the Viola-Jones face detection algorithm, and the face area of ​​the detected face is marked by a rectangular frame, and the face area marked by each rectangular frame is edited to obtain a number of face sub-videos. That is, the face is located, and then the video clip containing a single student is cut out to detect the actions of each student.

[0063] The steps of the Viola-Jones face detection algorithm to detect faces are as follows:

[0064] First use Haar features (such as Figure 3 The four rectangular features shown in the figure are used to describe the facial features. The Haar feature sums the pixels in the white area and the black area and takes the difference, and then an integral image is established. For any point in the integral image, the integral image value of the point is equal to the sum of all pixels located in the upper left corner of the point. Then, the weak classifier and the strong classifier are trained in sequence using the AdaBoost algorithm for the features of the integral image. Finally, a cascade classifier is used for face detection. Several AdaBoost classifiers are cascaded together. At the beginning, a small number of features are used to eliminate most of the non-face areas, and then more complex features are used to eliminate more complex non-face areas. When the face area is detected, a rectangular frame is used to frame the specific face area.

[0065] The step S30 specifically includes:

[0066] Step S31, randomly selecting a first picture frame from each of the face sub-videos as a static feature picture;

[0067] Step S32, after each of the static feature pictures, extract a second picture every 8 frames, and extract 4 frames of the second pictures in sequence;

[0068] Step S33, extracting the horizontal optical flow and the vertical optical flow from each of the static feature pictures and the second picture by using the TV_L1 method, and superimposing the horizontal optical flow and the vertical optical flow to obtain a dynamic feature picture;

[0069] Step S34: construct a training data set based on the static feature images and the dynamic feature images.

[0070] The specific steps for extracting static feature images and dynamic feature images are as follows:

[0071] Randomly select the τth frame from each face sub-video as the static feature image:

[0072] τ = random(1, N-32), where N represents the total number of frames;

[0073] Taking the static feature image as the starting point, extract one frame every d frames, and continuously extract L frames. Calculate the horizontal optical flow u and the vertical optical flow v for two adjacent frames in the sequence {τ, τ+d, ..., τ+(L-1)d} in turn. Finally, superimpose u and v of L sampling frames to form a dynamic feature image of 2L channels; the values ​​of d and L in this application are 8 and 4 respectively.

[0074] The lateral optical flow u and the longitudinal optical flow v are calculated based on the brightness constancy equation:

[0075]

[0076]

[0077] Where I(x,y,t) represents the image intensity function at point (x,y) at time t; Represents the spatial variation gradient; I t represents the time-varying gradient; (dx, dy) and dt represent the displacement vector of the point and time respectively; p and q represent the horizontal motion vector and the vertical motion vector respectively, so u and v can be calculated by the following formula:

[0078]

[0079]

[0080] In the step S40, the action recognition model includes a spatial stream module for extracting static feature vectors, a temporal stream module for extracting dynamic feature vectors, and a first fully connected layer for splicing the static feature vectors and the dynamic feature vectors; the output ends of the spatial stream module and the temporal stream module are connected to the input end of the first fully connected layer;

[0081] The spatial stream module is composed of an Inception V3 unit pre-trained using ImageNet and a second fully connected layer of 128 dimensions; a global average pooling layer is also included between the Inception V3 unit and the second fully connected layer;

[0082] The time-series flow module consists of five convolutional layers and a third fully connected layer; each convolutional layer uses a convolution kernel of size 3×3 and a step size of 1; the time-series flow module uses the ReLU function as the activation function. The five convolution kernels have 32, 64, 64, 128, and 128 output channels respectively. After the first, third, and fifth convolutional layers, there is a maximum pooling layer with a convolution kernel size of 3×3 and a 25% dropout layer.

[0083] The action recognition model finally transforms the spatial stream feature F s and the time series flow feature F t Concatenate them together to get a 256-dimensional fused feature vector F:

[0084] F=f cat (F s ,F t );

[0085] Then the fused feature vector F is sent to the fully connected layer, mapped to 128 dimensions, and then passed through a 7-dimensional softmax layer to generate the final prediction result. The 7 dimensions represent 7 types of actions: drinking water, sleeping, looking at the phone, looking at the computer, listening to a class, talking, and writing.

[0086] That is, the dual-stream convolutional network of this application has made the following two improvements compared to the traditional one. First, the Inception V3 unit is used to generate the feature map of the spatial stream, making the model converge easier and faster; second, the spatial stream and the temporal stream are fused based on the fully connected layer, and the fusion effect is better.

[0087] A preferred embodiment of a classroom student action recognition system based on deep learning of the present invention includes the following modules:

[0088] A classroom video editing module is used for the social robot to capture a large number of classroom videos and edit the classroom videos to obtain a number of classroom sub-videos;

[0089] A face detection module, used for the social robot to detect faces in each of the classroom sub-videos in turn through a face detection algorithm, and then edit the classroom sub-videos into a number of face sub-videos;

[0090] A training data set construction module, used for the social robot to extract the static feature image and the dynamic feature image of each face sub-video, and construct a training data set based on each static feature image and the dynamic feature image;

[0091] The action recognition model training module is used for the social robot to create an action recognition model based on deep learning, and use the training data set to train the action recognition model; that is, the training data set is divided into a training set and a verification set according to a preset ratio, the training set is used to train the action recognition model, and the verification set is used to verify the trained action recognition model;

[0092] The action recognition module is used for the social robot to use the trained action recognition model to perform real-time action recognition on the classroom video captured by the camera, and periodically send the recognition results to the teaching terminal; that is, extract the static feature images and dynamic feature images in the classroom video and input them into the trained action recognition model for action recognition.

[0093] The classroom video editing module is specifically:

[0094] The social robot captures a large amount of classroom videos through a camera installed in the classroom, and sequentially edits the classroom videos at preset time intervals to obtain several classroom sub-videos, for example, editing to generate several classroom sub-videos at intervals of every 4 seconds.

[0095] To ensure that students' movements can be captured, the camera's frame rate should be greater than 25fps.

[0096] The face detection module is specifically:

[0097] First, each classroom sub-video is subjected to noise reduction processing to remove redundant information, and then the faces in each classroom sub-video are detected in turn by using the Viola-Jones face detection algorithm, and the face area of ​​the detected face is marked by a rectangular frame, and the face area marked by each rectangular frame is edited to obtain a number of face sub-videos. That is, the face is located, and then the video clip containing a single student is cut out to detect the actions of each student.

[0098] The steps of the Viola-Jones face detection algorithm to detect faces are as follows:

[0099] First use Haar features (such as Figure 3The four rectangular features shown in the figure are used to describe the facial features. The Haar feature sums the pixels in the white area and the black area and takes the difference, and then an integral image is established. For any point in the integral image, the integral image value of the point is equal to the sum of all pixels located in the upper left corner of the point. Then, the weak classifier and the strong classifier are trained in sequence using the AdaBoost algorithm for the features of the integral image. Finally, a cascade classifier is used for face detection. Several AdaBoost classifiers are cascaded together. At the beginning, a small number of features are used to eliminate most of the non-face areas, and then more complex features are used to eliminate more complex non-face areas. When the face area is detected, a rectangular frame is used to frame the specific face area.

[0100] The training data set construction module specifically includes:

[0101] A static feature image selection unit, used to randomly select a first image frame from each of the face sub-videos as a static feature image;

[0102] A second picture selection unit is used to extract a second picture frame every 8 frames after each of the static feature pictures, and sequentially extract 4 frames of the second picture;

[0103] A dynamic feature picture extraction unit, used for extracting a horizontal optical flow and a vertical optical flow from each of the static feature pictures and the second picture by using a TV_L1 method, and superimposing the horizontal optical flow and the vertical optical flow to obtain a dynamic feature picture;

[0104] The static and dynamic feature image integration unit is used to construct a training data set based on the static feature images and the dynamic feature images.

[0105] The specific steps for extracting static feature images and dynamic feature images are as follows:

[0106] Randomly select the τth frame from each face sub-video as the static feature image:

[0107] τ = random(1, N-32), where N represents the total number of frames;

[0108] Taking the static feature image as the starting point, extract one frame every d frames, and continuously extract L frames. Calculate the horizontal optical flow u and the vertical optical flow v for two adjacent frames in the sequence {τ, τ+d, ..., τ+(L-1)d} in turn. Finally, superimpose u and v of L sampling frames to form a dynamic feature image of 2L channels; the values ​​of d and L in this application are 8 and 4 respectively.

[0109] The lateral optical flow u and the longitudinal optical flow v are calculated based on the brightness constancy equation:

[0110]

[0111]

[0112] Where I(x,y,t) represents the image intensity function at point (x,y) at time t; Represents the spatial variation gradient; I t represents the time-varying gradient; (dx, dy) and dt represent the displacement vector of the point and time respectively; p and q represent the horizontal motion vector and the vertical motion vector respectively, so u and v can be calculated by the following formula:

[0113]

[0114]

[0115] In the action recognition model training module, the action recognition model includes a spatial stream module for extracting static feature vectors, a temporal stream module for extracting dynamic feature vectors, and a first fully connected layer for splicing the static feature vectors and the dynamic feature vectors; the output ends of the spatial stream module and the temporal stream module are connected to the input end of the first fully connected layer;

[0116] The spatial stream module is composed of an Inception V3 unit pre-trained using ImageNet and a second fully connected layer of 128 dimensions; a global average pooling layer is also included between the Inception V3 unit and the second fully connected layer;

[0117] The time-series flow module consists of five convolutional layers and a third fully connected layer; each convolutional layer uses a convolution kernel of size 3×3 and a step size of 1; the time-series flow module uses the ReLU function as the activation function. The five convolution kernels have 32, 64, 64, 128, and 128 output channels respectively. After the first, third, and fifth convolutional layers, there is a maximum pooling layer with a convolution kernel size of 3×3 and a 25% dropout layer.

[0118] The action recognition model finally transforms the spatial stream feature F s and the time series flow feature F t Concatenate them together to get a 256-dimensional fused feature vector F:

[0119] F=f cat (F s ,F t );

[0120] Then the fused feature vector F is sent to the fully connected layer, mapped to 128 dimensions, and then passed through a 7-dimensional softmax layer to generate the final prediction result. The 7 dimensions represent 7 types of actions: drinking water, sleeping, looking at the phone, looking at the computer, listening to a class, talking, and writing.

[0121] That is, the dual-stream convolutional network of this application has made the following two improvements compared to the traditional one. First, the Inception V3 unit is used to generate the feature map of the spatial stream, making the model converge easier and faster; second, the spatial stream and the temporal stream are fused based on the fully connected layer, and the fusion effect is better.

[0122] In summary, the advantages of the present invention are:

[0123] By pre-capturing a large number of classroom videos and dividing them into several classroom sub-videos, detecting the faces in each classroom sub-video and then editing the classroom sub-videos into several face sub-videos, and then extracting the static feature images and dynamic feature images of each face sub-video to construct a training data set, and then using the training data set to train the created action recognition model, finally using the trained action recognition model to perform real-time action recognition on the captured classroom videos, that is, using the trained two-stream convolutional network (spatial stream module, temporal stream module) for action recognition, which not only has fast and accurate classification speed, but also has strong generalization and robustness, and ultimately realizes real-time and accurate recognition of student actions, and timely adjusts teaching strategies, thereby greatly improving the teaching quality.

[0124] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for identifying student actions in a classroom based on deep learning, characterized in that: The steps include: Step S10, capturing a large number of classroom videos, and editing the classroom videos to obtain a number of classroom sub-videos; Step S20, detecting faces in each of the classroom sub-videos in turn by using a face detection algorithm, and then editing the classroom sub-videos into a plurality of face sub-videos; Step S30, respectively extracting static feature images and dynamic feature images of each of the face sub-videos, and constructing a training data set based on the static feature images and dynamic feature images; Specifically include: Step S31, randomly selecting a first picture frame from each of the face sub-videos as a static feature picture; Step S32, after each of the static feature pictures, extract a second picture every 8 frames, and extract 4 frames of the second pictures in sequence; Step S33, extracting the horizontal optical flow and the vertical optical flow from each of the static feature pictures and the second picture by using the TV_L1 method, and superimposing the horizontal optical flow and the vertical optical flow to obtain a dynamic feature picture; Step S34, constructing a training data set based on the static feature pictures and the dynamic feature pictures; Step S40, creating an action recognition model based on deep learning, and using the training data set to train the action recognition model; the action recognition model includes a spatial stream module for extracting static feature vectors, a temporal stream module for extracting dynamic feature vectors, and a first fully connected layer for splicing the static feature vectors and the dynamic feature vectors; the output ends of the spatial stream module and the temporal stream module are connected to the input end of the first fully connected layer; The spatial stream module consists of an Inception V3 unit pre-trained using ImageNet and a second fully connected layer; The time series flow module consists of five convolutional layers and a third fully connected layer; each convolutional layer uses a convolution kernel of size 3×3 and a step size of 1; the time series flow module uses the ReLU function as the activation function; Step S50: Use the trained action recognition model to perform real-time action recognition on the classroom video captured by the camera.

2. A method for identifying student actions in a classroom based on deep learning as claimed in claim 1, characterized in that: The step S10 is specifically as follows: A large number of classroom videos are captured by a camera installed in the classroom, and the classroom videos are sequentially edited at preset time intervals to obtain a number of classroom sub-videos.

3. A method for identifying student actions in a classroom based on deep learning as claimed in claim 1, characterized in that: The step S20 is specifically as follows: First, each of the classroom sub-videos is subjected to noise reduction processing, and then the faces in each of the classroom sub-videos are detected in turn using the Viola-Jones face detection algorithm. The face area of ​​the detected face is marked by a rectangular frame, and the face area marked by each rectangular frame is edited to obtain a number of face sub-videos.

4. A classroom student action recognition system based on deep learning, characterized by: Includes the following modules: A classroom video editing module is used to capture a large number of classroom videos and edit the classroom videos to obtain a number of classroom sub-videos; A face detection module, used to detect faces in each of the classroom sub-videos in turn by using a face detection algorithm, and then edit the classroom sub-videos into a plurality of face sub-videos; A training data set construction module, used to extract the static feature pictures and dynamic feature pictures of each face sub-video respectively, and construct a training data set based on each static feature picture and dynamic feature picture; The training data set construction module specifically includes: A static feature image selection unit, used to randomly select a first image frame from each of the face sub-videos as a static feature image; A second picture selection unit is used to extract a second picture frame every 8 frames after each of the static feature pictures, and sequentially extract 4 frames of the second picture; A dynamic feature picture extraction unit, used for extracting a horizontal optical flow and a vertical optical flow from each of the static feature pictures and the second picture by using a TV_L1 method, and superimposing the horizontal optical flow and the vertical optical flow to obtain a dynamic feature picture; A static and dynamic feature image integration unit, used to construct a training data set based on the static feature images and the dynamic feature images; An action recognition model training module is used to create an action recognition model based on deep learning, and use the training data set to train the action recognition model; in the action recognition model training module, the action recognition model includes a spatial stream module for extracting static feature vectors, a temporal stream module for extracting dynamic feature vectors, and a first fully connected layer for splicing the static feature vectors and the dynamic feature vectors; the output ends of the spatial stream module and the temporal stream module are connected to the input end of the first fully connected layer; The spatial stream module consists of an Inception V3 unit pre-trained using ImageNet and a second fully connected layer; The time series flow module consists of five convolutional layers and a third fully connected layer; each convolutional layer uses a convolution kernel of size 3×3 and a step size of 1; the time series flow module uses the ReLU function as the activation function; The action recognition module is used to use the trained action recognition model to perform real-time action recognition on the classroom video captured by the camera.

5. A classroom student action recognition system based on deep learning as claimed in claim 4, characterized in that: The classroom video editing module is specifically: A large number of classroom videos are captured by a camera installed in the classroom, and the classroom videos are sequentially edited at preset time intervals to obtain a number of classroom sub-videos.

6. A classroom student action recognition system based on deep learning as claimed in claim 4, characterized in that: The face detection module is specifically: First, each of the classroom sub-videos is subjected to noise reduction processing, and then the faces in each of the classroom sub-videos are detected in turn using the Viola-Jones face detection algorithm. The face area of ​​the detected face is marked by a rectangular frame, and the face area marked by each rectangular frame is edited to obtain a number of face sub-videos.

Images