A classroom behavior analysis method and system based on video intelligent timing processing

Abstract

The application discloses a kind of based on video intelligent timing processing's classroom behavior analysis method, comprising: recording classroom teaching, obtains data source;Characteristic extraction module first divides long video, extracts space-time characteristics using feature encoder to each short segment, extracts the feature sequence of video;Timing segmentation module carries out timing modeling to feature sequence;Regression each frame's behavior category and position start time interval, obtain different teaching segments, and the teaching length of different segments is counted;Quality evaluation module uses human body detection model to detect the position coordinates of human body key points in each teaching segment, obtains gesture feature by decentralization calculation, obtains original feature using feature fusion model comprehensive feature sequence, using channel attention mechanism is weighted to feature channel, obtain final feature.The human body posture contained in the human body detection model detected by the present application contains the teaching state of teacher and the feedback information of student in classroom, and the quality of classroom teaching is automatically evaluated.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of video understanding, and in particular relates to a classroom behavior analysis method and system based on video intelligent time sequence processing. BACKGROUND

[0002] Teaching quality evaluation can provide teachers with objective feedback on the effectiveness of classroom teaching. Through systematic evaluation results, teachers can identify areas that need improvement in the teaching process, such as the clarity of instruction, the effectiveness of interaction, student engagement, etc. Objective and effective feedback can prompt teachers to reflect on and optimize their teaching methods, thereby improving overall teaching standards. For students, teaching quality evaluation not only improves learning effectiveness, but also enhances learning experience and promotes personalized growth. Therefore, teaching quality evaluation is an important link in promoting continuous improvement of teaching quality and cultivating high-quality talents.

[0003] Currently, traditional classroom teaching behavior analysis usually relies on observation and recording by multiple experts in the field, which has high time cost, low efficiency, and strong subjectivity. Intelligent classroom behavior analysis systems can automatically analyze and understand the time sequence of teaching videos and output more objective evaluation feedback. Compared with traditional evaluation methods, intelligent classroom behavior analysis systems can complete teaching quality evaluation tasks efficiently and at a lower cost.

[0004] Classroom behavior analysis technology based on video understanding can be defined as: recording the entire classroom teaching process to obtain digital video, pre-processing each frame of the video as a feature extraction object, and then performing time sequence segmentation and quality evaluation on the video. Time sequence segmentation of the video can be considered as consisting of two sub-tasks: one sub-task is to predict the start and end time interval of the behavior; the other sub-task is to predict the behavior category. The quality evaluation task often needs to extract the temporal and spatial features of the video segment and regress the evaluation score to reflect the pros and cons of the behavior quality in the video.

[0005] The existing video time sequence segmentation technology has the technical problem of fuzzy behavior boundaries, especially in the application background of classroom teaching. Teaching behaviors such as teacher-only instruction, teacher-student question and answer interaction, and classroom discussion have similarities, and it is easy to make mistakes in identifying the start and end of different behaviors. In addition, the evaluation of the teaching quality of teachers involves multiple dimensions, including the language expression of teachers, the interaction effect, and the participation of students, etc. It is difficult to comprehensively consider the above factors. SUMMARY

[0006] The application proposes a classroom behavior analysis method and system based on video intelligent time sequence processing to overcome the shortcomings of the prior art. A time sequence segmentation module is used to establish the long-term dependence of the video sequence, classify the teaching behavior frame by frame, and regress the start and end time interval. The teacher's teaching state and student's classroom feedback information contained in the human body posture detected by the human body detection model are used to automatically evaluate the classroom teaching quality.

[0007] A classroom behavior analysis method based on video intelligent time sequence processing, comprising the following steps:

[0008] Step 1: Build a shooting platform to record classroom teaching and obtain data sources;

[0009] Step 2: The feature extraction module first divides the long video, uses a feature encoder to extract the space-time features of each short segment, so that each discretized timestamp gets the corresponding feature vector, thereby extracting the feature sequence of the video;

[0010] Step 3: The time sequence segmentation module models the feature sequence obtained in step 2 in time sequence;

[0011] Step 4: Regress the behavior category of each frame and locate the start and end time interval to obtain different teaching segments and count the teaching duration of different segments;

[0012] Step 5: The quality evaluation module uses a human body detection model to detect the position coordinates of the human body key points in each teaching segment, decentralizes to obtain the posture features, then uses a feature fusion model to integrate the feature sequence obtained in step 2 to obtain the original features, uses a channel attention mechanism to weight the feature channels, and obtains the final features for regression of the teaching quality score.

[0013] Further, in step 1, the classroom teaching is recorded to obtain the original video, then the video is down-sampled, and each frame is cropped for preprocessing to obtain the data source.

[0014] Further, in step 3, the specific method for extracting the feature sequence of the video is as follows: an I3D model pre-trained on Kinetics data is used, a step of 10 is used to obtain discrete timestamps, 16 frames are used as the segment length, a number of short segments are obtained as I3D input, the space-time feature vector of each timestamp is obtained, and then the feature sequence of the entire video is obtained.

[0015] Further, in step 4, the specific method for counting the teaching duration of different segments is as follows: the feature pyramid obtained by modeling the feature sequence in time sequence is respectively passed through a classification head and a regression head; the classification head predicts the probability of each timestamp belonging to different teaching categories; the regression head predicts the distance from each timestamp to the start and end of the corresponding teaching behavior; that is, the output of each timestamp t is:

[0016]

[0017] where p(a t ) is the confidence of the teaching behavior, respectively, the offset of the behavior start and end time compared to the time stamp t, and further decoding gets a behavior instance:

[0018]

[0019] where e t , s t are the start and end time of the behavior respectively;

[0020] Finally, the behavior candidates are processed by Soft-NMS to remove highly overlapping instances, and the final output is generated, and the duration of different teaching behaviors is counted.

[0021] Further, the channel attention mechanism is used to weight the feature channels to obtain the final features, which are divided into three stages:

[0022] The compression stage uses a global pooling layer F sq (·) to aggregate global information to obtain the pooled features z c , where u c represents the three-dimensional original feature of size T×H×W of the cth channel:

[0023]

[0024] where T, H, and W represent time, frame height, and frame width, respectively;

[0025] The excitation stage F ex (·,W) generates different weights for each channel, which passes through two fully connected layers. The first fully connected layer is W1, and its channel number is C / r, where C is the original feature channel number, and r represents the scaling factor. After passing through a ReLu activation function layer, it passes through a fully connected layer W2, so that the output channel number is the same as the original feature channel number C. Finally, it passes through a sigmoid function to get the feature channel weight s c :

[0026] s c =F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0027] The weighting stage multiplies the weight generated in the excitation stage with the original feature to get the final feature fused by the channel attention module for the regression of the teaching quality score:

[0028]

[0029] A classroom behavior analysis system based on video intelligent time sequence processing, comprising a feature extraction module, a time sequence segmentation module and a quality evaluation module;

[0030] The feature extraction module is mainly responsible for extracting the space-time feature sequence of the input video as the input of the time sequence segmentation module and the quality evaluation module.

[0031] The time sequence segmentation module is mainly responsible for time sequence modeling, and regression classification to obtain video clips of different teaching behaviors.

[0032] The quality evaluation module is mainly responsible for combining human posture and space-time features to output quality scores for different video clips.

[0033] The advantages of the present application are:

[0034] (1) The time sequence segmentation module adopts an anchor-free single-stage time sequence action positioning model, which has lower training cost and more accurate and efficient segmentation positioning than the two-stage positioning model.

[0035] (2) The quality evaluation module no longer relies only on space-time features, but combines human posture information, reasonably introduces implicit expressions of teacher teaching level and student feedback in classroom teaching, so as to evaluate teaching quality in multiple dimensions, making the analysis system more intelligent, comprehensive and objective. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The method flowchart of the present application;

[0037] Figure 2 The 17 human body key point schematic diagram detected by Yolov8-Pose. DETAILED DESCRIPTION

[0038] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0039] In view of the problems of traditional classroom teaching behavior analysis methods, such as relying on manual observation and recording, low efficiency, strong subjectivity and the like, the application adopts video time sequence segmentation technology to locate and segment different teaching behaviors in the classroom, fragments the teaching process, and performs time constant statistics and quality evaluation on each fragment. The application specifically adopts a time sequence segmentation model to understand a video sequence, classifies each frame of the teaching video, and regresses the corresponding behavior boundary; a human body detection model is used to extract human posture information, reflect the teaching state of the teacher and the feedback of the students, evaluate the teaching quality score, and achieve the purpose of automatic and fine time axis analysis.

[0040] Reference Figure 1 The application provides a classroom behavior analysis method based on video intelligent time sequence processing, comprising the following steps:

[0041] Step 1, a shooting platform is built to record classroom teaching and obtain data sources;

[0042] Step 2, the feature extraction module first divides the long video, uses a feature encoder to extract the space-time features of each short fragment, so that each discretized timestamp obtains a corresponding feature vector, thereby extracting the feature sequence of the video;

[0043] Step 3, the time sequence segmentation module models the feature sequence obtained in step 2; the time sequence segmentation module adopts an anchor-free single-stage time sequence action positioning process to automatically segment different teaching fragments, which is simple and effective; the quality evaluation module introduces human posture information, and further combines the space-time features to guide the model to evaluate the teaching quality in multiple dimensions.

[0044] Step 4, the behavior category of each frame is regressed and the start and end time interval is located to obtain different teaching fragments, and the teaching time of different fragments is counted;

[0045] Step 5, the quality evaluation module uses a human body detection model to detect the position coordinates of the human body key points of each teaching fragment, and obtains posture features through decentralized calculation, then uses a feature fusion model to obtain the original features from the feature sequence obtained in step 2, uses a channel attention mechanism to weight the feature channels, and obtains the final features, which are used for regression of the teaching quality score.

[0046] Further optimization technical scheme, in step 1, the classroom teaching is recorded to obtain the original video, then the video is down-sampled, and each frame is cropped and preprocessed to obtain the data source.

[0047] Further optimization of the technical scheme, in step 3, the specific method for extracting the feature sequence of the video is: using the I3D model pre-trained on the Kinetics data, with a step size of 10, obtaining discrete timestamps, and taking 16 frames as the segment length, obtaining a plurality of short segments as the I3D input, obtaining the spatio-temporal feature vector of each timestamp, and then obtaining the feature sequence of the entire video.

[0048] Further optimization of the technical scheme, in step 3, the time sequence segmentation module first encodes the video features into a feature pyramid. Specifically, the excellent time sequence modeling capability of the Transformer is utilized, and in order to improve the efficiency on long videos, a local attention mechanism is adopted to exclude context information beyond a certain time range by limiting attention within a local window, achieving precise positioning and improving efficiency.

[0049] Further optimization of the technical scheme, in step 4, the specific method for counting the teaching duration of different segments is: the feature pyramid obtained by time sequence modeling of the feature sequence is respectively passed through a classification head and a regression head; the classification head predicts the probability of each timestamp belonging to different teaching categories such as teacher's individual teaching, teacher-student interactive questioning, and classroom discussion; the regression head predicts the distance from each timestamp to the start and end of the corresponding teaching behavior; that is, the output of each timestamp t is:

[0050]

[0051] where p(a t ) is the confidence of the teaching behavior, and are the offsets of the behavior start and end from timestamp t, respectively, and a behavior instance is further decoded:

[0052]

[0053] where e t , s t are the start and end times of the behavior, respectively;

[0054] Finally, the Soft-NMS processing result is used to remove highly overlapping instances to generate the final output, and the duration of different teaching behaviors is counted.

[0055] Further optimization of the technical scheme, in step 5, the quality evaluation module uses Yolov8-Pose to detect the position coordinates of the human key points in each teaching segment, and uses a channel attention mechanism to integrate the spatio-temporal features obtained in step 2 to obtain pose features, which specifically includes: detecting 17 human key points by Yolov8-Pose, as shown in Figure 2 , obtaining the coordinates of each key point (x i , y i), because the posture information is irrelevant to the absolute position of the video picture where the person is located, so the coordinates are decentralized to obtain the posture feature.

[0056] The spatiotemporal feature and the posture feature are spliced to obtain the original feature u c The channel attention mechanism is used to weight the feature channels to obtain the final feature, which is divided into three stages.

[0057] The compression stage uses a global pooling layer F sq (·) to aggregate global information to obtain the pooled feature z c , wherein u c represents a three-dimensional original feature of size T×H×W of the cth channel:

[0058]

[0059] Wherein, T, H, W represent time, frame height and frame width, respectively.

[0060] The excitation stage F ex (·,W) generates different weights for each channel, which passes through two fully connected layers, the first fully connected layer W1, whose channel number is C / r, wherein C is the original feature channel number, and r represents a scaling factor. The scaling factor used in the present application is 32. After passing through a ReLu activation function layer, it passes through a fully connected layer W2, so that the output channel number is the same as the original feature channel number C. Finally, it passes through a sigmoid function to obtain the feature channel weight s c :

[0061] s c =F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0062] The weighting stage multiplies the weight generated in the excitation stage with the original feature to obtain the final feature fused through the channel attention module for the regression of the teaching quality score:

[0063]

[0064] The present application also provides a classroom behavior analysis system based on video intelligent time sequence processing, comprising a feature extraction module, a time sequence segmentation module and a quality evaluation module.

[0065] The feature extraction module is mainly responsible for extracting the spatiotemporal feature sequence of the input video as the input of the time sequence segmentation module and the quality evaluation module.

[0066] The time sequence segmentation module is mainly responsible for time sequence modeling, and the classification regression obtains video clips of different teaching behaviors.

[0067] The quality evaluation module is mainly responsible for combining human posture and space-time features to output quality scores for different video clips.

[0068] The timing segmentation module of the application adopts an anchor-free single-stage timing action positioning model, compared with a two-stage positioning model, the timing segmentation module in the application has lower training cost and more accurate and efficient segmentation positioning. The quality evaluation module no longer relies only on space-time features, but combines human posture information, reasonably introduces the implicit expression of teacher teaching level and student feedback in classroom teaching, so as to evaluate the teaching quality in multiple dimensions, so that the analysis system is more intelligent, comprehensive and objective.

[0069] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A classroom behavior analysis method based on intelligent video temporal processing, characterized in that, Includes the following steps: Step 1: Set up a filming platform to record classroom teaching and obtain data sources; Step 2: The feature extraction module first divides the long video into segments and uses a feature encoder to extract spatiotemporal features from each segment, so that each discretized timestamp obtains a corresponding feature vector, thereby extracting the feature sequence of the video. Step 3: The temporal segmentation module performs temporal modeling on the feature sequences obtained in Step 2; Step 4: Regress the behavior category of each frame and locate the start and end time intervals to obtain different teaching segments, and count the teaching duration of different segments; The specific method for calculating the teaching duration of different segments is as follows: The feature pyramid obtained from the feature sequence is used for time-series modeling, and then processed by a classification head and a regression head. The classification head predicts the probability that each timestamp belongs to a different teaching category; the regression head predicts the distance from each timestamp to the start and end of the corresponding teaching activity. That is, the output for each timestamp t is: Where p(a) t () represents the confidence level of the teaching behavior. These are the offsets from the start and end of the action relative to the timestamp t, respectively. Further decoding yields an example of the action: Where e t s t These represent the start and end times of the action, respectively. Finally, the candidate behaviors are processed by Soft-NMS to remove highly overlapping instances, generate the final output, and count the duration of different teaching behaviors. Step 5: The quality assessment module uses a human detection model to detect the position coordinates of key human points in each teaching segment, calculates the posture features in a decentralized manner, and then uses a feature fusion model to synthesize the feature sequence obtained in Step 2 to obtain the original features. The feature channels are weighted using a channel attention mechanism to obtain the final features, which are used for the regression of teaching quality scores.

2. The classroom behavior analysis method based on intelligent video temporal processing according to claim 1, characterized in that, In step 1, the classroom teaching is recorded to obtain the original video, then the video is downsampled, and each frame is pre-processed by cropping to obtain the data source.

3. The classroom behavior analysis method based on intelligent video temporal processing according to claim 2, characterized in that, In step 2, the specific method for extracting the feature sequence of the video is as follows: using the I3D model pre-trained on the Kinetics data, with a step size of 10, discrete timestamps are obtained, and with 16 frames as the segment length, several short segments are obtained as I3D inputs to obtain the spatiotemporal feature vector of each timestamp, thereby obtaining the feature sequence of the entire video.

4. The classroom behavior analysis method based on intelligent video temporal processing according to claim 3, characterized in that, The feature channels are weighted using a channel attention mechanism to obtain the final features, which is divided into three stages: During the compression phase, a global pooling layer F is used. sq (·) Aggregate global information to obtain the pooled feature z. c , where u c Represents the original 3D feature of size T×H×W in the c-th channel: Where T, H, and W represent time, frame height, and frame width, respectively; Incentive phase, F ex (·,W) generates different weights for each channel, which are then passed through two fully connected layers. The first fully connected layer, W1, has C / r channels, where C is the original number of feature channels and r is a scaling factor. After passing through a ReLU activation function layer, it passes through another fully connected layer, W2, to make the number of output channels the same as the original number of feature channels, C. Finally, it passes through a sigmoid function to obtain the feature channel weights s. c : s c =F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) In the weighting phase, multiplying the weights generated in the activation phase with the original features yields the final features after fusion by the channel attention module. Regression used for teaching quality scores:

5. A classroom behavior analysis system based on intelligent video time-series processing, characterized in that, The method for implementing the classroom behavior analysis method based on intelligent video temporal processing as described in any one of claims 1-4 includes a feature extraction module, a temporal segmentation module, and a quality assessment module. The feature extraction module is mainly responsible for extracting the spatiotemporal feature sequence of the input video, which serves as the input for the temporal segmentation module and the quality assessment module. The temporal segmentation module is mainly responsible for temporal modeling, classification, and regression to obtain video segments of different teaching behaviors; The quality assessment module is mainly responsible for combining human posture and spatiotemporal features to output quality scores for different video clips.

Images