A classroom recording video-oriented teacher inappropriate gesture detection method and system

Abstract

The present application relates to the field of image recognition, and provides a teacher improper gesture detection method and system for classroom recording video, which comprises the following steps: (1) effective picture acquisition, loading classroom recording video resources for training, screening effective video segments and converting them into pictures, applying a character detection model to identify the identity of the teacher, screening and retaining effective pictures containing the teacher; (2) gesture detection model training, inputting original picture data into a neural network, extracting gesture features for model training, and saving the trained model as a gesture detection model; (3) improper gesture judgment, applying the trained gesture detection model to detect the pictures converted from the video one by one. The present application automatically identifies and intelligently judges the improper gesture behavior of the teacher in the teaching video resources, realizes intelligent monitoring and accurate evaluation in teaching in the teaching video resources, and provides support for high-quality digital course resource construction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to a teacher inappropriate gesture detection method and system for classroom recording video. BACKGROUND

[0002] Developing teacher inappropriate gesture detection for classroom recording video is conducive to the construction and development of high-quality digital course resources. At present, there are only a few studies on teacher classroom behavior based on classroom recording video in China, and they do not focus on the identification and judgment of teacher inappropriate gestures in classroom recording video. The systematic study of teacher inappropriate gesture detection for classroom recording video is still blank.

[0003] The current teacher inappropriate gesture detection for classroom recording video has the following difficulties: (1) the real classroom teaching environment factors and characteristics are complex, which leads to the formation of classroom recording video information that is diverse and difficult to accurately extract and identify the details of teacher gestures; (2) it is difficult for machines to automatically and accurately identify the identities of teachers and students, distinguish the differences between teacher and student gestures and postures, and accurately extract the features of teacher gestures in classroom recording video; (3) there is a lack of standardized process and method for teacher inappropriate gesture detection in classroom recording video, which makes it difficult to realize the large-scale, automated and objective teacher inappropriate gesture detection in classroom recording video. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a teacher inappropriate gesture detection method and system for classroom recording video, which aims to automatically identify and intelligently judge the inappropriate gesture behavior of teachers in teaching video resources, realize intelligent monitoring and accurate evaluation in teaching video resources, fill the gap of automatic supervision of teaching video resources, and provide support for high-quality digital course resource construction.

[0005] The purpose of the present application is achieved through the following technical measures.

[0006] A teacher inappropriate gesture detection method for classroom recording video, comprising the following steps:

[0007] (1) Effective picture acquisition. Load the classroom recording video resources for training, filter the effective video segments by video cutting and convert them into pictures, identify the teacher's identity by applying the character detection model MediaPipe, and apply the ResNet network to filter and retain the effective pictures containing the teacher.

[0008] (2) Gesture detection model training. Input the original picture data into the neural network, divide different feature regions by data enhancement and image segmentation, label the region belonging to the gesture category, extract the gesture features for model training, and save the trained model as a gesture detection model.

[0009] (3) Improper gesture judgment. The trained gesture detection model is applied to detect each picture converted from the video, to judge whether there is improper gesture, and to give visual feedback result.

[0010] The application also provides a teacher improper gesture detection system for classroom recording video, comprising the following modules:

[0011] A video information extraction module is used to extract relevant information of the teaching video resource, obtain frame rate, width, height, resolution and time length information of the video resource, and save them;

[0012] A video conversion module is used to convert the video into pictures, and the video is divided according to the length of the classroom teaching, and the divided video is converted into pictures according to different frame rates;

[0013] A teacher identity recognition module is used to identify the teacher identity in the image picture, and the existence of the teacher in all pictures is detected by using a person detection model and a residual neural network ResNet;

[0014] A data processing module is used to process picture data for training, filter the picture data, adaptively scale the picture to the size required by the neural network, and perform data enhancement on the scaled image data through random scaling, random cropping and random arrangement, and then label different gesture features;

[0015] A feature extraction module is used to input the picture, slice the input picture, aggregate and form image features at different image fine-grained levels, and mix and combine the image features;

[0016] A model training module is used to train the gesture detection model, train the model by using the neural network through parameter setting, and save the optimal model according to the feedback accuracy;

[0017] A gesture detection module is used to detect the improper gesture of the teacher in the teaching video resource, convert the teaching video into pictures and put them into the optimal model for detection by calling the trained optimal model, and give visual feedback result after detection.

[0018] The application has the following advantages:

[0019] The computer information processing and image recognition technology are applied to capture the classroom recording video and filter effective pictures, then the gesture detection processing is performed on the filtered pictures, and the processing result is represented by using the column chart visualization method, which fills the research blank of the teacher improper gesture detection for classroom recording video, supports the normalization and data of the real-time detection and result feedback of the teacher improper gesture in the classroom recording video, and helps the construction of high-quality teaching video resources. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is the total flow chart of the improper gesture detection method for teachers facing classroom recording video constructed by the embodiment of the application.

[0021] Figure 2 is the image data enhancement effect schematic diagram based on classroom recording video in the embodiment of the application.

[0022] Figure 3 is the image processing flow chart in the embodiment of the application.

[0023] Figure 4 is the main network structure schematic diagram of the neural network structure in the embodiment of the application.

[0024] Figure 5 is the Neck network structure schematic diagram of the neural network structure in the embodiment of the application.

[0025] Figure 6 is the focus structure schematic diagram of the neural network structure in the embodiment of the application.

[0026] Figure 7 is the SPP module structure schematic diagram of the neural network structure in the embodiment of the application.

[0027] Figure 8 is the CBL module structure schematic diagram of the neural network structure in the embodiment of the application.

[0028] Figure 9 is the CSP module structure schematic diagram of the neural network structure in the embodiment of the application.

[0029] Figure 10 is the improper gesture detection model training result schematic diagram in the teaching video resource in the embodiment of the application.

[0030] Figure 11 is the improper gesture detection result feedback schematic diagram in the teaching video resource in the embodiment of the application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0032] As shown in Figure 1 , the embodiment of the application provides an improper gesture detection method for teachers facing classroom recording video, which includes the following steps:

[0033] (1) Effective picture acquisition. Load the classroom recording video resources for training, filter the effective video segments through video cutting and convert them into pictures, apply the character detection model MediaPipe to identify the teacher's identity, and apply the ResNet network to filter and retain the effective pictures containing the teacher.

[0034] (1-1) Video information acquisition. Read the video resources in the database, and obtain the frame rate F, width W, height H, resolution P, and time length T of the video.

[0035] (1-2) Video cutting. According to the length of 45 minutes of class and 10 minutes of rest, the video resources are divided into classroom video segments and intermission video segments according to the time length T; the intermission video segments are removed, and the classroom video segments are retained as effective video resources.

[0036] (1-3) Picture conversion and cleaning. Convert the video segments into picture sets Imgs1 according to the frame rate F, and set the picture storage location path as source; then perform data cleaning on the obtained pictures, and the specific steps of picture cleaning are as follows:

[0037] (1-3-1) Picture type division. According to the picture content, the pictures are divided into four categories: classroom environment pictures, teacher pictures, student pictures, and teacher-student pictures; the classroom environment pictures refer to the pictures mainly in the classroom without teachers or students; the teacher pictures refer to the pictures only with teachers; the student pictures refer to the pictures only with students; the teacher-student pictures refer to the pictures with both students and teachers.

[0038] (1-3-2) Picture selection and storage. Apply the character detection model MediaPipe and the ResNet network to analyze the picture set Imgs1 one by one, label the picture types, and only retain the teacher pictures and teacher-student pictures as the cleaned picture set Imgs2, and save them to the picture path source as the effective content for subsequent gesture detection model training. The picture type labeling steps are as follows:

[0039] a. Apply the character detection model MediaPipe to analyze the picture set Imgs1 one by one, capture the human key points to record the human skeleton structure, calculate the number of human skeletons Num_skeleton in each picture, label the pictures with Num_skeleton as 0 as classroom environment pictures, and remove them;

[0040] b. For pictures with Num_skeleton greater than 0, calculate the human skeleton completeness Com_skeleton, the interval distance D_skeleton between each human skeleton and other skeleton groups; extract the human skeleton KEY_skeleton with the maximum Com_skeleton or D_skeleton in each picture, and mark the human skeleton with the most occurrences of KEY_skeleton in each picture as the teacher human skeleton Teacher_skeleton;

[0041] c. Use the ResNet network to obtain the face and clothing features of the person corresponding to the teacher human skeleton Teacher_skeleton, and mark them as teacher features Teacher_feature;

[0042] d. Use the ResNet network to search for teacher features Teacher_feature, and mark the picture where Teacher_feature cannot be found as a student picture;

[0043] e. For pictures with teacher features Teacher_feature and Num_skeleton of 1, mark them as teacher pictures; the remaining unmarked pictures are marked as teacher-student pictures.

[0044] (2) Image detection model training. Input the original picture data into the neural network, divide different feature regions through data enhancement and image segmentation, and mark the gesture category to which the region belongs, extract the gesture features for model training, and save the trained model.

[0045] (2-1) Picture input. Extract the picture Imgs2 in the source path to the neural network, process the picture data through picture scaling, data enhancement, and data segmentation.

[0046] (2-1-1) Picture scaling. To adaptively scale the picture to the size Size (416x416) required by the neural network, calculate the picture scaling factor Scaling;

[0047] Scaling = min{W scaling ,H scaling}

[0048] W scaling = 416 / W

[0049] H scaling = 416 / H

[0050] Where min is the minimum function, W scaling is the picture width scaling factor, and H scalingis the picture height scaling coefficient, W is the original picture width, and H is the original picture height.

[0051] The picture size is adjusted according to the scaling coefficient Scaling, and the black edges generated during image scaling are filled with RGB image data to form the scaled picture Imgs3.

[0052]

[0053]

[0054] wherein, is the scaled picture width, is the scaled picture height.

[0055] (2-2) Picture data preprocessing. The picture data is subjected to data enhancement, and the enhanced picture data is subjected to gesture picture segmentation and labeling to form a picture set Imgs4.

[0056] (2-2-1) Picture data enhancement. Four scaled pictures are taken as a group, and the four pictures are spliced into a picture with a size Size conforming to the requirements of the neural network through random extraction, scaling, distribution and splicing, so as to enrich the detection data set and improve the network robustness. The picture data enhancement effect is shown in Figure 2 , and the overall picture processing process is shown in Figure 3 .

[0057] (2-2-2) Gesture picture segmentation and labeling. The enhanced picture data is labeled according to different gesture categories, and an XML labeling coordinate file is generated. Different gesture categories are segmented and labeled by reading the XML file content. The gesture categories include improper gestures and reasonable gestures.

[0058] (2-3) Model training. The picture set Imgs4 is input, and after slice processing of the input picture, it is output to the backbone network (such as Figure 4 shown), which is aggregated and formed into an image feature at different image fine-grained levels, and then the picture is output to the Neck structure (such as Figure 5 shown), which mixes and combines the image features and transmits the features to the prediction layer. The image features are predicted in the prediction layer to generate a bounding box and predict the gesture category. The predicted result is compared with the labeled result to calculate the accuracy. The specific process is as follows:

[0059] (2-3-1) Backbone network Backbone processing. The neural network focus structure (such as Figure 6 shown) is used to perform slice operation on the input picture set Imgs4, and then perform convolution operation with 32 convolution kernels to become the feature map required by the system, and then put into the SPP module (such asFigure 7 The processed data is formed by the adaptive scaling process; the specific process is as follows:

[0060] The adaptive scaling original image is put into the focus structure, and a slice operation is performed to become 304*304*12 size image data, which is output by the CBL module (as shown in Figure 8 The image data of 152*152 is output to the CSP module (as shown in Figure 9 The image data of 76*76 is formed, and part of the data output by the CSP is transmitted to the Neck structure, and part is transmitted to the next CBL module to form 38*38 image data and then transmitted to the CSP module. The data output by the CSP is respectively transmitted to the Neck and the next CBL, CSP, CBL, and then output to the CSP structure of the Neck through the SPP module;

[0061] The SPP module is composed of a CBL module, a max pooling layer Maxpool, and a tensor concatenation layer Concat; the data is sequentially processed by the CBL module, the max pooling layer Maxpool, and then processed by the CBL module again to reduce student gesture interference, and finally enters the tensor concatenation layer Concat and outputs the result to the CBL module;

[0062] The CBL module includes a convolution layer, a normalization layer, and an activation function.

[0063] (2-3-2) Neck network processing. The data processed by the backbone network is imported into the Neck network structure, and different tensor concatenation layers Concat are accessed for parameter aggregation to further improve the feature extraction capability. The specific process is as follows:

[0064] The CSP in the Neck accepts the data from the SPP in the backbone network, and then outputs 38*38 image data to the CBL module. The data from the CBL is respectively down-sampled and transmitted to the tensor concatenation layer. The data after down-sampling is transmitted to the tensor concatenation layer. At this time, the data of the tensor concatenation layer comes from the backbone network. The tensor concatenation layer is then output to the CSP and CBL, and is up-sampled again. Then the up-sampled data and the data from the CSP in the backbone network are sent to the connection layer, which is output to the CSP. The CSP is output to the prediction layer on one hand, and to the CBL for the next tensor concatenation on the other hand. After multiple concatenations, the data is finally output to the prediction end.

[0065] (2-3-3) Prediction end processing. The data input by the Neck structure is predicted and labeled with a prediction box. The CIOU_LOSS loss function is used for the prediction end bounding box to improve the speed and accuracy of the prediction box regression.

[0066] CIOU_LOSS = 1-CIOU

[0067]

[0068]

[0069]

[0070] wherein, a and v are the aspect ratio, IOU is the ratio of the intersection and union of the predicted area and the labeled area, p represents the Euclidean distance between b and b(gt), w, h and w(gt), h(gt) represent the height and width of the predicted box and the true box respectively, b represents the parameter of the predicted center coordinate, b gt represents the parameter of the true target boundary box center.

[0071] (2-3-4) Model training. The model training parameter batchsize is set to 128, epochs is initially 1000, and the training is automatically stopped when the model is not changing. After the training is completed, the model with the highest precision is saved, and the model detection training detection result is as shown in Figure 10 .

[0072]

[0073] wherein, TP refers to the case where the predicted value is true when the true value is true; and FP refers to the case where the predicted value is true when the true value is false.

[0074] (3) Improper gesture judgment. The trained gesture detection model is applied to detect each picture converted from the video, to judge whether there is an improper gesture, and to give a visual feedback result, and the visualization result is as shown in Figure 11 .

[0075] (3-1) Detection picture conversion. Obtain the classroom recording video resource that needs to be detected for improper gestures of teachers, and then convert the video into pictures of corresponding quality frame by frame.

[0076] (3-2) Gesture detection. Read the saved optimal model, and put the pictures converted from the video into the model for gesture detection to obtain the gesture type judgment result.

[0077] (3-3) Visualization result display. According to the video period where the picture is located and the gesture type judgment result, the teacher improper gesture detection result chart is formed in the form of a column chart.

[0078] The contents not described in detail in the specification belong to the prior art known to the professional technical personnel.

[0079] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for teacher inappropriate gesture detection for classroom recorded video, characterized in that The method comprises the following steps: (1) Effective picture acquisition, load the classroom recording video resources for training, screen effective video segments through video cutting and convert them into pictures, apply the character detection model MediaPipe to identify the teacher's identity, and apply the ResNet network to screen and retain the effective pictures containing the teacher; The specific process is: (1-1) Video information acquisition, read the video resources in the database, and obtain the frame rate F, width W, height H, resolution P, and time length T of the video; (1-2) Video cutting, taking the classroom teaching time of 45 minutes and the rest time of 10 minutes as the unit, the video resources are cut into classroom video segments and intermission video segments according to the time length T; the intermission video segments are removed, and the classroom video segments are retained as effective video resources; (1-3) Picture conversion and cleaning, the video segments are converted into a picture set Imgs1 according to the frame rate F, and the picture storage location path is set as source; the obtained pictures are subjected to data cleaning processing, and the picture cleaning specific steps are as follows: (1-3-1) Picture type division, according to the picture content, the pictures are divided into four categories of classroom environment pictures, teacher pictures, student pictures and teacher-student pictures, wherein the classroom environment pictures refer to pictures mainly in the classroom without teachers or students; the teacher pictures refer to pictures only with teachers; the student pictures refer to pictures only with students; the teacher-student pictures refer to pictures with both students and teachers; (1-3-2) Picture screening and storage, the picture set Imgs1 is detected and analyzed one by one by applying the character detection model MediaPipe and the ResNet network, the picture types are labeled, only the teacher pictures and the teacher-student pictures are retained as the cleaned picture set Imgs2, and saved to the picture path source as the effective content for subsequent gesture detection model training, and the picture type labeling steps are as follows: a. The picture set Imgs1 is detected and analyzed one by one by applying the character detection model MediaPipe, the human key points are captured to record the human skeleton structure, the number of human skeletons Num_skeleton in each picture is calculated, the pictures with Num_skeleton of 0 are labeled as classroom environment pictures, and are removed; b. For pictures with Num_skeleton greater than 0, the completeness of human skeleton Com_skeleton and the interval distance D_skeleton of each human skeleton and other skeleton groups are calculated; the human skeleton KEY_skeleton with the maximum Com_skeleton or D_skeleton in each picture is extracted, and the human skeleton with the most occurrences KEY_skeleton in each picture is labeled as the teacher human skeleton Teacher_skeleton; c. The face and clothing features of the character corresponding to the teacher human skeleton Teacher_skeleton are obtained by using the ResNet network, and are labeled as the teacher features Teacher_feature; d. Use ResNet network to search teacher feature Teacher_feature, and mark the picture without Teacher_feature as student picture; e. Mark the picture with teacher feature Teacher_feature and Num_skeleton as 1 as teacher picture, and mark the remaining pictures as teacher-student picture; (2) Gesture detection model training, input the original picture data into the neural network, divide different feature regions through data enhancement and image segmentation, and label the gesture category to which the region belongs, extract the gesture feature for model training, and save the trained optimal model as the gesture detection model; (3) Improper gesture judgment, apply the trained gesture detection model to detect each picture converted from the video, judge whether there is improper gesture, and give visual feedback result. 2.The teacher inappropriate gesture detection method for class recording video according to claim 1, characterized in that The specific process of step (2) for gesture detection model training is: (2-1) Picture input, extract the picture Imgs2 in the source path to the neural network, process the picture data through picture scaling, data enhancement, data segmentation and convolution compression; (2-1-1) Picture scaling, adaptively scale the picture to the size Size required by the neural network, 416x416, calculate the picture scaling factor Scaling; Scaling = min{W scaling ,H scaling} W scaling = 416 / W H scaling = 416 / H wherein min is a minimum function, W scaling is a picture width scaling factor, H scaling is a picture height scaling factor, W is a picture original width, and H is a picture original height. Adjust the picture size according to the scaling factor Scaling, and fill the black edges generated during image scaling with RGB image data to form the scaled picture Imgs3; wherein, is the scaled picture width, is the scaled picture height; (2-2) Picture data preprocessing, data enhancement is performed on the picture data, and gesture picture segmentation and labeling are performed on the enhanced picture data to form the picture set Imgs4; (2-2-1) Picture data enhancement, take the four scaled pictures as a group, and through random extraction, scaling, distribution and splicing, splice the four pictures into a picture with a size Size required by the neural network, enrich the detection data set, and improve the network robustness; (2-2-2) Gesture picture segmentation and labeling, gesture labeling is performed on the enhanced picture data according to different gesture categories, XML annotation coordinate files are generated, and different gesture categories are segmented and labeled by reading the XML file content, wherein the gesture categories are improper gesture and reasonable gesture; (2-3) Model training, input the picture set Imgs4, after slicing the input picture, output to the backbone network, aggregate and form image features at different image fine-grained, then output the picture to the Neck structure, mix and combine the image features, and pass the features to the prediction layer, predict the image features in the prediction layer, generate the bounding box and predict the gesture category, compare the predicted result with the labeled result, calculate the accuracy; The specific process is as follows: (2-3-1) Backbone network Backbone processing, use the neural network focus structure, slice the input picture set Imgs4, perform convolution operation with 32 convolution kernels once, become the feature map required by the system, put into the SPP module for processing, form the processed data; The SPP module is composed of a CBL module, a maximum pooling layer Maxpool, and a tensor concatenation layer Concat; the data sequentially passes through the CBL module, the maximum pooling layer Maxpool, and then enters the CBL module again for processing, reduces student gesture interference, and finally enters the tensor concatenation layer Concat and outputs the result to the CBL module; The CBL module includes a convolution layer, a normalization layer, and an activation function; The (2-3-2) Neck network processes the data processed by the backbone network Backbone, imports the data into the Neck network structure, accesses different tensor concatenation layers Concat for parameter aggregation, and further improves the feature extraction capability; The (2-3-3) prediction end processing uses the CIOU_LOSS loss function for the prediction end bounding box, improves the speed and accuracy of the prediction box regression; CIOU_LOSS = 1-CIOU wherein a and v are the aspect ratio, IOU is the ratio of the intersection and union of the predicted area and the labeled area, p represents the Euclidean distance between b and b(gt), w, h and w(gt), h(gt) represent the height and width of the predicted box and the true box respectively, b represents the parameter of the predicted center coordinates, b gt represents the parameter of the center of the true target bounding box. The (2-3-4) model training sets the model training parameter batchsize to 128, the initial epochs to 1000, and automatically stops the training when the model is not changing, saves the model with the highest precision after the training is completed; Wherein, TP refers to the case where the predicted value is true when the true value is true; FP refers to the case where the predicted value is true when the true value is false. 3.The teacher inappropriate gesture detection method for class recording video according to claim 1, characterized in that The specific process of the (3) improper gesture judgment is as follows: (3-1) Detection picture conversion: obtaining the classroom recording video resource that needs to be detected for improper gestures of teachers, and converting the video into pictures with corresponding quality frame by frame; (3-2) Gesture detection: reading the saved optimal model, putting the pictures converted from the video into the model for gesture detection, and obtaining the gesture type judgment result; (3-3) Visualization result display: according to the video period and gesture type judgment result of the picture, the teacher improper gesture detection result picture is formed in the form of a column chart.

4. A teacher inappropriate gesture detection system for a classroom recording video, characterized by The improper gesture detection method for teachers in the classroom recording video resource is used to realize the improper gesture detection method for teachers in the classroom recording video resource in any one of claims 1-3, comprising: A video information extraction module is used to extract the related information of the teaching video resource, obtain the frame rate, width and height, resolution, and time length information of the video resource, and save them; A video conversion module is used to convert the video into pictures, and the video is divided according to the length of the class in the classroom, and the divided video is converted into pictures according to different frame rates; A teacher identity recognition module is used to identify the teacher identity in the image picture, and a person detection model and a residual neural network ResNet are used to detect the existence of the teacher in all pictures; A data processing module is used to process the picture data for training, filter the picture data, adaptively scale the picture to the size required by the neural network, and perform data enhancement on the scaled image data through random scaling, random cropping, and random arrangement, and then label different gesture features; A feature extraction module inputs the picture, aggregates and forms image features on different image fine-grained after slicing the input picture, and mixes and combines the image features; The model training module is configured to train a gesture detection model, perform model training by parameter setting using a neural network, and save an optimal model according to feedback accuracy. The gesture detection module is configured to detect inappropriate gestures of a teacher in a teaching video resource, convert the teaching video into pictures and put the pictures into the optimal model for detection by calling the trained optimal model, and give a visual feedback result after the detection.

Images