Classroom behavior analysis method, system, electronic device and storage medium

Abstract

The present invention provides a classroom behavior analysis method, system, electronic device and storage medium, which relate to the field of educational informatization technology. The method includes collecting video and audio, performing image enhancement, noise filtering and frame segmentation on the video, performing noise reduction, sound source localization and speech segmentation on the audio, and generating standardized image and speech sequences; using an improved MTCNN cascade network combined with posture estimation technology to extract student facial key points from the video, identify class status, head posture and facial micro-expressions, and analyze the audio through a bidirectional LSTM network to obtain speech duration and frequency; constructing individual behavior indicators based on student attendance, head movements and expressions; constructing interaction indicators based on teacher and student speech data; summarizing individual and interaction indicators to generate multidimensional classroom behavior portraits, and analyzing the relationship between the portraits and teaching effects, thereby improving the accuracy of classroom behavior analysis, enhancing data processing capabilities, and realizing multi-dimensional comprehensive evaluation of classroom behavior.

Description

Technical Field

[0001] The present invention relates to the field of educational information technology, and in particular to a classroom behavior analysis method, system, electronic equipment and storage medium. Background Art

[0002] In the field of educational informatization, with the development of smart devices and big data analysis technology, classroom behavior analysis has become an important means to improve teaching quality and optimize teaching management.

[0003] In existing technologies, classroom behavior analysis primarily uses video surveillance systems to capture students' facial expressions and body movements, combined with machine learning algorithms for behavioral recognition to assess their learning status and engagement. However, traditional classroom behavior analysis methods often rely on manual observation and subjective judgment, and suffer from numerous drawbacks. Specifically, recognition accuracy is limited, significantly declining in complex lighting conditions or when students' faces are obscured. Second, data processing capabilities are limited, making it difficult to achieve real-time analysis of large-scale data. Third, the analysis dimension is limited, lacking a comprehensive assessment of the quality of teacher-student interaction.

[0004] Therefore, it is necessary to provide classroom behavior analysis methods, systems, electronic equipment and storage media to solve the above technical problems. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a classroom behavior analysis method, system, electronic device and storage medium, which are used to solve the problems that the existing technology cannot effectively improve the accuracy of classroom behavior analysis, has poor data processing capabilities, and cannot achieve multi-dimensional comprehensive evaluation of classroom behavior.

[0006] The classroom behavior analysis method provided by the present invention comprises:

[0007] Obtain classroom video stream data and teacher-student audio stream data, and perform image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence; perform noise reduction, sound source localization, and voice segmentation on the teacher-student audio stream data to generate a standardized voice sequence;

[0008] A CNN is used to extract features from the standardized image sequence. Specifically, an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion is used to locate key points on the student's face. This is combined with a multi-joint head pose estimation model to identify the student's attendance status, head pose, and facial micro-expressions. A bidirectional LSTM network is then used to perform time series modeling on the standardized speech sequence to obtain the duration and frequency of the teacher's speech, as well as the duration and frequency of the student's speech.

[0009] Based on the students' attendance status, head posture, and facial micro-expressions, a student individual behavior indicator system is constructed, including student attendance rate, head-up rate, head-down duration, and frequency of expression changes. Based on the teacher's speaking time and frequency and the student's speaking time and frequency, a teacher-student interaction behavior indicator system is constructed, including the proportion of teacher-student speeches, the frequency of students' questions, and the duration of teacher feedback.

[0010] The student individual behavior indicator system and the teacher-student interaction behavior indicator system are aggregated to generate a multi-dimensional classroom behavior portrait, and the multi-dimensional classroom behavior portrait is associated with the classroom teaching effect.

[0011] Preferably, the acquisition of classroom video stream data and teacher-student audio stream data, and performing image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence, and performing noise reduction, sound source localization, and voice segmentation on the teacher-student audio stream data to generate a standardized voice sequence, specifically includes:

[0012] Deploy multi-angle cameras and array microphones in the target teaching area to form a distributed sensor network;

[0013] Based on the distributed sensor network, the classroom video stream data with a resolution of 720P or above is captured in real time through the RTSP protocol, and the classroom audio stream data with a sampling rate of 48kHz is synchronously collected to complete the spatiotemporal alignment of the classroom video stream data and the classroom audio stream data;

[0014] Performing image enhancement, noise filtering, and video frame segmentation on the classroom video stream data, i.e., improving scene contrast through an adaptive histogram equalization method; removing teacher image noise based on a target detection model, prior knowledge of seating layout, and a bilateral filtering algorithm, while retaining student image information, to generate the standardized image sequence;

[0015] The classroom audio stream data is subjected to noise reduction, sound source localization and speech segmentation, that is, the Wiener filtering algorithm is used to remove environmental noise, and the short-time energy detection method is combined to filter out silent segments; the direction of the sound source is located based on beamforming technology, and the teacher and student voiceprint features are trained through the GMM-UBM model to automatically separate the teacher's voice signal from the student's voice signal; the voice activity detection algorithm is used to segment the valid voice segments to generate the standardized voice sequence with timestamps.

[0016] Preferably, the method of removing teacher image noise based on the target detection model, prior knowledge of seating layout, and bilateral filtering algorithm, retaining student image information, and generating the standardized image sequence specifically includes:

[0017] For the classroom video stream data, the teacher's body area is identified based on the target detection model, and a teacher mask is generated. ;

[0018] Based on the prior knowledge of the seating layout, combined with the background difference method, the student body area is extracted and the student mask is generated. , completing the spatial separation of teacher and student images in the classroom video stream data;

[0019] The teacher mask is processed by the bilateral filtering algorithm Perform denoising to generate the denoised teacher area. The corresponding calculation formula is as follows:

[0020] Where, represents the pixel p in the teacher area after denoising; represents the normalization factor at pixel p; N(p) represents the spatial neighborhood of pixel p; Represents the spatial standard deviation. If the teacher mask is retained The details in ; If the teacher mask For enhanced denoising, ; represents the spatial domain Gaussian weight function, which is used to control the spatial influence range of the neighborhood pixels, and ; represents the standard deviation of the range, , Represents the standard deviation of pixel values in the teacher's facial area; represents the pixel p in the original teacher image region; represents the pixel q in the original teacher image region; represents the range Gaussian weight function, which is used to control the influence of color similarity of neighboring pixels, and ;

[0021] Directly retain the student mask Covered pupil original image area , and the denoised teacher region The original image area with the pupil The fusion is performed according to the corresponding mask to generate the standardized image sequence. The corresponding calculation formula is as follows:

[0022] Where, represents a normalized image sequence; represents the teacher mask; represents the teacher region after denoising; represents the student mask; Represents the student original image area.

[0023] Preferably, the teacher mask is processed by the bilateral filtering algorithm. During the denoising process, a motion compensation mechanism is introduced, which includes:

[0024] Masks for the teachers Motion blur noise in the spatial domain is added with a motion compensation term in the Gaussian weight function. ,in, represents the motion vector of pixel p; represents the motion vector of pixel q, then the updated spatial domain Gaussian weight function is .

[0025] Preferably, the feature extraction of the standardized image sequence by CNN is performed, that is, the facial key points of the students are located by an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion, and the multi-joint head posture estimation model is combined to identify the students' attendance status, head posture and facial micro-expressions, specifically including:

[0026] Through the improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion, the weights of each layer of the network are dynamically allocated according to the complexity of the facial area in the standardized image sequence, and the facial feature information at different scales is integrated to locate the 68 key points of the student's face;

[0027] Synchronously access the multi-joint head posture estimation model to calculate the three-dimensional posture parameters of the student's head in real time, and the multi-joint head posture estimation model at least includes a skeleton point detection module and an angle calculation module based on a convolutional neural network;

[0028] A two-layer detection mechanism of facial presence status and head posture matrix is established to identify the student's attendance status, head posture and facial micro-expressions based on the student's facial key points and the student's three-dimensional head posture parameters.

[0029] Preferably, the bidirectional LSTM network is used to perform time series modeling on the standardized speech sequence to obtain the teacher's speaking time and frequency and the student's speaking time and frequency, specifically including:

[0030] Converting the standardized speech sequence into a Mel-frequency cepstral coefficient sequence, and inputting the Mel-frequency cepstral coefficient sequence into the bidirectional LSTM network for time series modeling;

[0031] Extracting speech energy, zero-crossing rate, and fundamental frequency features from the Mel-frequency cepstral coefficient sequence through a preset sliding window, and combining the attention mechanism to obtain key speech periods;

[0032] Construct a teacher-student speech classifier to distinguish between the teacher's teaching period and the student's interaction period in the key speech period, and output the teacher's speech duration and frequency as well as the student's speech duration and frequency.

[0033] Preferably, the aggregating the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associating the multi-dimensional classroom behavior portrait with classroom teaching effectiveness specifically includes:

[0034] Performing feature normalization processing on the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a classroom behavior indicator set X;

[0035] Obtaining a preset number of clusters, grouping the classroom behavior indicator set X using a K-means algorithm to obtain multiple clusters, calculating the center vector of each cluster and constructing the multi-dimensional classroom behavior profile Y;

[0036] Establish a classroom teaching effect indicator set E, use a multiple linear regression model to fit the linear relationship between the multidimensional classroom behavior portrait Y and the classroom teaching effect indicator set E, and screen significant correlation indicators through t-test;

[0037] The SHAP value is introduced to calculate the contribution of each indicator in the classroom behavior indicator set X to the classroom teaching effect, and the indicators in the classroom behavior indicator set X are sorted from large to small according to the contribution, and the top A indicators are selected as key classroom behavior indicators, where A represents the number of preset key indicators;

[0038] Calculate the Pearson correlation coefficient between the key classroom behavior indicators and the classroom teaching effect and construct a behavior-effect correlation heat map.

[0039] A classroom behavior analysis system, the analysis system comprising:

[0040] A data acquisition module is used to acquire classroom video stream data and teacher-student audio stream data, and perform image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence; and perform noise reduction, sound source localization, and speech segmentation on the teacher-student audio stream data to generate a standardized speech sequence;

[0041] A data analysis module is used to extract features from the standardized image sequence using a CNN. Specifically, the module locates key points on the student's face using an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion. This module, combined with a multi-joint head pose estimation model, identifies the student's attendance status, head pose, and facial micro-expressions. Furthermore, the module uses a bidirectional LSTM network to perform time series modeling on the standardized speech sequence to obtain the duration and frequency of the teacher's speech and the duration and frequency of the student's speech.

[0042] An indicator construction module is used to construct a student individual behavior indicator system based on the student's attendance status, head posture, and facial micro-expressions, including student attendance rate, head-up rate, head-down duration, and frequency of expression changes; and to construct a teacher-student interaction behavior indicator system based on the teacher's speaking time and frequency and the student's speaking time and frequency, including the proportion of teacher-student speaking, the frequency of student questions, and the duration of teacher feedback;

[0043] The summary association module is used to summarize the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associate the multi-dimensional classroom behavior portrait with the classroom teaching effect.

[0044] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the steps of any one of the above-described classroom behavior analysis methods.

[0045] A readable storage medium stores a computer program, which, when executed by a processor, is used to implement the steps of any one of the classroom behavior analysis methods described above.

[0046] Compared with related technologies, the classroom behavior analysis method, system, electronic device, and storage medium provided by the present invention have the following beneficial effects:

[0047] The present invention obtains classroom video stream data and teacher-student audio stream data, and performs image enhancement, noise filtering and video frame segmentation on the classroom video stream data to generate a standardized image sequence, and performs noise reduction, sound source localization and voice segmentation on the teacher-student audio stream data to generate a standardized voice sequence; CNN is used to extract features from the standardized image sequence, that is, the improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion is used to locate the key points of the student's face, and the multi-joint head posture estimation model is combined to identify the student's attendance status, head posture and facial micro-expressions, and a bidirectional LSTM network is used to perform time series modeling on the standardized voice sequence to obtain the teacher's speaking time and frequency. and the duration and frequency of students' speeches; construct a student individual behavior indicator system based on students' attendance status, head posture and facial micro-expressions, including student attendance rate, head-raising rate, head-lowering duration and frequency of expression changes; and construct a teacher-student interaction behavior indicator system based on the teacher's speaking time and frequency and the student's speaking time and frequency, including the proportion of teacher and student speeches, the frequency of students' questions and the duration of teacher feedback; summarize the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associate the multi-dimensional classroom behavior portrait with the classroom teaching effect, so as to effectively improve the accuracy of classroom behavior analysis, enhance data processing capabilities, and realize multi-dimensional comprehensive evaluation of classroom behavior.

[0048] The present invention improves the MTCNN cascade network, combines posture estimation technology, and constructs a two-layer mechanism of face detection and posture assistance, which can effectively solve the recognition problems in complex scenes such as low light and facial occlusion, and provide a reliable basis for accurately evaluating learning status. The present invention uses a distributed sensor network, combined with a bilateral filtering algorithm and a motion compensation mechanism, to achieve high-quality collection and preprocessing of classroom data, while supporting real-time streaming and batch analysis of classroom data, improving data processing efficiency, and meeting the low-latency requirements of large-scale education scenarios. The present invention can construct a multi-dimensional indicator system that includes individual behavior and teacher-student interaction, generate student behavior portraits through cluster analysis, and quantify the relationship between classroom behavior and teaching effectiveness through statistical modeling, providing scientific guidance for teaching intervention. The present invention adopts a full-process automated system design, supports the rapid deployment of multiple scenarios such as smart classrooms and online education, is compatible with multi-terminal display and distributed computing mechanisms, and can adapt to different classroom environments without manual calibration, significantly improving the intelligence level and resource adaptation capabilities of education management. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the classroom behavior analysis method of the present invention;

[0050] Figure 2 is a system block diagram of the classroom behavior analysis system of the present invention;

[0051] Figure 3 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] like Figure 1 FIG. 1 is a flow chart of a classroom behavior analysis method according to an embodiment of the present invention. Figure 1The execution subject of the method shown may be a software and / or hardware device. The execution subject of the present application may include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, user equipment may include but is not limited to computers, smart phones, personal digital assistants (PDAs) and the electronic devices mentioned above. Network equipment may include but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. It includes steps S1 to S4, as follows:

[0054] S1, obtaining classroom video stream data and teacher-student audio stream data, and performing image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence, and performing noise reduction, sound source localization, and speech segmentation on the teacher-student audio stream data to generate a standardized speech sequence;

[0055] Classroom video stream data refers to the continuous dynamic video information of the classroom scene. It exists in the form of a data stream and contains various visual elements of the classroom, such as the teacher and students' movements, expressions, and classroom environment. Teacher and student audio stream data refers to the continuous audio information of the classroom scene. It is presented in the form of a data stream and includes audio content such as the teacher's lecture, the students' speeches, and various voice exchanges in the classroom. This audio stream data can reflect the verbal interaction between teachers and students in the classroom, providing a data foundation for analyzing language expression and teacher-student communication in the classroom.

[0056] Furthermore, a standardized image sequence refers to a series of images with uniform specifications and standards generated by performing image enhancement, noise filtering, and video frame segmentation on the acquired classroom video stream data. This standardization process ensures that the images are consistent in terms of resolution, format, brightness, and contrast, facilitating the subsequent analysis and processing of information in the images, such as students' facial features and head posture, using relevant algorithms and models, thereby improving the accuracy and efficiency of the analysis. A standardized speech sequence refers to a standardized and orderly sequence of speech data obtained by performing noise reduction, sound source localization, and speech segmentation on the teacher-student audio stream data. The standardization process ensures consistency in sampling rate, quantization accuracy, and duration segmentation of the speech data, facilitating the subsequent use of a bidirectional LSTM network for time series modeling of the speech data. This allows for the acquisition of valuable information such as the duration and frequency of teacher and student speeches, providing a clear and standardized data foundation for analyzing verbal interaction in the classroom.

[0057] In practical applications, for classroom video stream data, image enhancement algorithms can be used to improve visual quality such as image contrast and sharpness. Noise filtering algorithms can be used to remove interference such as Gaussian noise and salt-and-pepper noise. Video frame segmentation technology can be used to decompose continuous video streams into discrete image frames, generating standardized image sequences with uniform resolution and format. For teacher-student audio stream data, noise reduction algorithms can be used to suppress ambient noise. Sound source localization technology can be used to distinguish the locations of teacher and student voices. Speech segmentation algorithms can be used to segment continuous speech into meaningful segments, forming standardized speech sequences.

[0058] S2, using CNN to extract features from the standardized image sequence, namely, locating students' facial key points through an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion, and combining it with a multi-joint head posture estimation model to identify students' attendance status, head posture, and facial micro-expressions. A bidirectional LSTM network is then used to perform time series modeling on the standardized speech sequence to obtain the duration and frequency of the teacher's speech and the duration and frequency of the student's speech.

[0059] As you can understand, MTCNN (Multi-Task Cascaded Convolutional Neural Network) is a deep learning model commonly used for face detection and related feature point location. It is composed of multiple convolutional neural networks in cascade. It can quickly and accurately detect faces in images and locate key facial features such as eyes, nose, and mouth.

[0060] Among them, the adaptive weight adjustment operation can automatically adjust the weight parameters of each part of the network according to different input data and task requirements, making the network more flexible and accurate when handling different situations, thereby improving the network's performance and adaptability. For example, when faced with facial images with different lighting, angles, and postures, the adaptive weight adjustment operation can better capture facial features. The multi-scale feature fusion operation can take into account the fact that faces in the image may be of different sizes. By fusing feature information at different scales, the network can better handle faces of different sizes. For example, when detecting smaller faces, the multi-scale feature fusion operation can comprehensively utilize features at different resolutions to avoid missed detections or false detections due to scale differences, thereby improving the accuracy of face detection and key point localization. This improved MTCNN cascade network has higher accuracy and robustness in locating key points on students' faces.

[0061] It's important to note that student facial key points refer to representative, key locations on a student's face, such as the center of the eyes, the corners of the eyes, the tip of the nose, the corners of the mouth, and the endpoints of the eyebrows. These key points reflect the basic structure and expressive features of a student's face. By locating these key points, we can analyze changes in a student's facial expressions, such as smiles and frowns, and assess their attention state. For example, when a student's eyes or head posture change, the position of these key points will also change accordingly, providing important evidence for analyzing their behavior and emotions.

[0062] In the multi-joint head pose estimation model, the joints here refer to key parts of the head, such as the top of the head and the connection point between the neck and the head. These joints can help determine the overall posture of the head. A head pose estimation model is a model that determines head posture by analyzing head features and position information in images or videos. Incorporating information from multiple joints, the model can more accurately estimate a student's head posture, such as looking up, looking down, and turning the head. For example, by analyzing the relative position and movement of the head joints, it can be determined whether the student is paying attention, such as looking up and keeping their head relatively stable, or whether they are doing something else, such as looking down or turning their head frequently.

[0063] Bidirectional LSTM (Long Short-Term Memory) networks are a special type of recurrent neural network (RNN) that effectively handles long-term dependencies in sequential data. When processing time-series data such as speech, LSTM networks can store and update information through memory cells, avoiding the vanishing and exploding gradient problems that can occur in traditional RNNs. Bidirectionality means that the network processes information not only from the beginning to the end of the sequence, but also from the end to the beginning. This bidirectional processing allows for more comprehensive capture of contextual information in the sequence data. When processing standardized speech sequences, bidirectional LSTM networks can simultaneously consider information before and after the speech, more accurately modeling the temporal characteristics of speech. This allows for information such as the duration and frequency of teacher and student speeches.

[0064] Specifically, for image feature extraction, convolutional neural networks (CNNs) can be used to process standardized image sequences. By cascading multiple convolutional network modules using an improved multi-task cascaded convolutional neural network (MTCNN) based on an adaptive weight adjustment mechanism and a multi-scale feature fusion strategy, the system accurately locates key facial features of students, such as the corners of the eyes, the tip of the nose, and the corners of the mouth. In combination with a multi-joint head pose estimation model, computer vision techniques are used to analyze the relative position changes of head joints from multiple angles, enabling student attendance status recognition, head pose detection, and facial micro-expression analysis. Head poses can include looking up, looking down, and turning sideways, while facial micro-expressions can include smiling or frowning. For speech feature extraction, a bidirectional long short-term memory network (Bi-LSTM) is used to perform temporal modeling on standardized speech sequences. Its bidirectional processing capabilities capture contextual information, enabling accurate statistics of the duration and frequency of speech by both teachers and students.

[0065] S3: Build a student individual behavior indicator system based on the student's attendance status, head posture, and facial micro-expressions, including student attendance rate, head-up rate, head-down duration, and frequency of expression changes; and build a teacher-student interaction behavior indicator system based on the teacher's speaking time and frequency and the student's speaking time and frequency, including the teacher-student speaking ratio, student question frequency, and teacher feedback duration;

[0066] The individual student behavior indicator system is a set of indicators used to measure and evaluate individual student behavior in the classroom. The teacher-student interaction behavior indicator system is a set of indicators used to describe and analyze the interaction between teachers and students in the classroom. It can reflect the dynamic process of classroom teaching from the perspective of verbal communication and interaction.

[0067] In practice, student attendance rate is calculated by counting the ratio of the actual number of students present to the expected number of students. Head-up rate is measured by calculating the proportion of time students look up relative to the total class time. Head-down duration is measured by recording the length of time students continuously look down. Expression change frequency is determined by analyzing the number of dynamic changes in facial micro-expressions. The teacher-student speech ratio is the ratio of the teacher's and students' speaking time to the total class time. The frequency of student questions is determined by counting the number of questions students raise proactively. Teacher feedback duration is the length of time the teacher spends responding to student comments or questions.

[0068] S4, summarizing the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associating the multi-dimensional classroom behavior portrait with the classroom teaching effect.

[0069] It should be noted that the multi-dimensional classroom behavior portrait is a comprehensive portrayal and presentation of the behavior of teachers and students in the classroom. Classroom teaching effectiveness refers to the actual results achieved by teachers guiding students in learning during the classroom teaching process.

[0070] It's understandable that the data from the student behavior indicator system and the teacher-student interaction behavior indicator system can be integrated to generate a multidimensional classroom behavior profile that encompasses both individual student behavior characteristics and teacher-student interaction patterns. Using statistical methods such as correlation analysis and regression analysis, a correlation model between classroom behavior profiles and classroom teaching effectiveness can be established, providing data support for teaching quality assessment and teaching strategy optimization. Classroom teaching effectiveness includes student knowledge mastery, ability development, and changes in emotional attitudes.

[0071] In the specific implementation process, the acquisition of classroom video stream data and teacher-student audio stream data, and performing image enhancement, noise filtering and video frame segmentation on the classroom video stream data to generate a standardized image sequence, and performing noise reduction, sound source localization and voice segmentation on the teacher-student audio stream data to generate a standardized voice sequence, specifically includes:

[0072] Deploy multi-angle cameras and array microphones in the target teaching area to form a distributed sensor network;

[0073] Based on the distributed sensor network, the classroom video stream data with a resolution of 720P or above is captured in real time through the RTSP protocol, and the classroom audio stream data with a sampling rate of 48kHz is synchronously collected to complete the spatiotemporal alignment of the classroom video stream data and the classroom audio stream data;

[0074] Performing image enhancement, noise filtering, and video frame segmentation on the classroom video stream data, i.e., improving scene contrast through an adaptive histogram equalization method; removing teacher image noise based on a target detection model, prior knowledge of seating layout, and a bilateral filtering algorithm, while retaining student image information, to generate the standardized image sequence;

[0075] The classroom audio stream data is subjected to noise reduction, sound source localization and speech segmentation, that is, the Wiener filtering algorithm is used to remove environmental noise, and the short-time energy detection method is combined to filter out silent segments; the direction of the sound source is located based on beamforming technology, and the teacher and student voiceprint features are trained through the GMM-UBM model to automatically separate the teacher's voice signal from the student's voice signal; the voice activity detection algorithm is used to segment the valid voice segments to generate the standardized voice sequence with timestamps.

[0076] In actual teaching areas, a distributed sensor network can be built, deploying multiple high-definition cameras and array microphones with different viewing angles. Multi-angle cameras can provide panoramic coverage of the classroom, ensuring complete capture of teacher and student behavior. The array microphones utilize a multi-channel sound pickup design, enhancing the directionality and signal-to-noise ratio of sound collection through spatial array layout, providing a foundation for subsequent sound source localization and speech separation.

[0077] Furthermore, based on a distributed sensor network, classroom video streaming data can be captured in real time using the Real-Time Streaming Protocol (RTSP). The capture resolution must be at least 720p to ensure clear and discernible visual details. Simultaneously, classroom audio streaming data is collected at a 48kHz sampling rate, which meets CD-quality standards and fully preserves the high-frequency details of the speech signal. Hardware timestamp synchronization technology and a time calibration algorithm enable precise spatiotemporal alignment of the video and audio data streams, ensuring strict temporal correspondence between the two, avoiding audio and video desynchronization issues caused by transmission delays and laying the foundation for subsequent joint analysis.

[0078] During the preprocessing of video stream data, image enhancement uses the adaptive histogram equalization (CLAHE) method to process video frames. This algorithm dynamically adjusts image contrast through local area histogram equalization, effectively improving image clarity in scenes with uneven lighting and enhancing the visual distinction between people and background. Noise filtering uses deep learning object detection models, such as the YOLO series, to identify the teacher and student areas in the video frames and, combined with prior knowledge of seating layout, locate the student. A bilateral filtering algorithm is used to reduce noise in the teacher's area. This algorithm smooths noise while preserving image edge information, preventing excessive blur that can affect student facial recognition. Selective noise reduction removes noise interference from the teacher image while preserving the integrity of the student image. Video frame segmentation can segment a continuous video stream into independent frames at a fixed frame rate, such as 30 fps. This unifies the image size and format, generating standardized image sequences that provide standardized data input for subsequent computer vision analysis.

[0079] During preprocessing of the audio stream, noise reduction is performed using the Wiener filter algorithm. Based on the minimum mean square error criterion, this algorithm dynamically adjusts filter parameters according to the statistical characteristics of the signal and noise, effectively suppressing ambient background noise. Short-term energy detection automatically identifies and removes silent segments in the audio, reducing the amount of invalid data. Sound source localization utilizes beamforming technology to spatially process the audio signal. By adjusting the weighting coefficients of each channel in the array microphone, a beam pointing in a specific direction is formed, enabling precise localization of the sound source. Voiceprint features of the teacher and student speech are trained using the Gaussian mixture model-universal background model (GMM-UBM), building a personalized acoustic model and automatically separating speech signals from different sources. Speech segmentation utilizes the voice activity detection (VAD) algorithm. By analyzing features such as the audio signal's short-term energy and zero-crossing rate, it automatically detects speech start and end boundaries, segmenting the continuous speech stream into independent, valid segments. Each segment is then accurately timestamped to generate standardized speech sequences, facilitating subsequent time series modeling and interaction analysis.

[0080] Through the above multi-step, multi-technology integration processing flow, high-quality collection and standardized preprocessing of classroom video streams and teacher-student audio stream data can be achieved, providing a reliable data foundation for subsequent deep learning-based classroom behavior analysis.

[0081] The method of removing teacher image noise based on the target detection model, prior knowledge of seat layout, and bilateral filtering algorithm, retaining student image information, and generating the standardized image sequence specifically includes:

[0082] For the classroom video stream data, the teacher's body area is identified based on the target detection model, and a teacher mask is generated. ;

[0083] Based on the prior knowledge of the seating layout, combined with the background difference method, the student body area is extracted and the student mask is generated. , completing the spatial separation of teacher and student images in the classroom video stream data;

[0084] The teacher mask is processed by the bilateral filtering algorithm Perform denoising to generate the denoised teacher area. The corresponding calculation formula is as follows:

[0085] Where, represents the pixel p in the teacher area after denoising; represents the normalization factor at pixel p; N(p) represents the spatial neighborhood of pixel p; Represents the spatial standard deviation. If the teacher mask is retained The details in ; If the teacher mask For enhanced denoising, ; represents the spatial domain Gaussian weight function, which is used to control the spatial influence range of the neighborhood pixels, and ; represents the standard deviation of the range, , Represents the standard deviation of pixel values in the teacher's facial area; represents the pixel p in the original teacher image region; represents the pixel q in the original teacher image region; represents the range Gaussian weight function, which is used to control the influence of color similarity of neighboring pixels, and ;

[0086] Directly retain the student mask Covered pupil original image area , and the denoised teacher region The original image area with the pupil The fusion is performed according to the corresponding mask to generate the standardized image sequence. The corresponding calculation formula is as follows:

[0087] Where, represents a normalized image sequence; represents the teacher mask; represents the teacher region after denoising; represents the student mask; Represents the student original image area.

[0088] First, an object detection model can be used to identify the teacher's body region in classroom video stream data, generating a teacher mask that can mark the teacher's position. Second, based on pre-determined seating layout information and background subtraction, the student area can be extracted to form a student mask, thereby achieving spatial distinction between teacher and student images.

[0089] Then, a bilateral filtering algorithm can be used to denoise the teacher mask. This algorithm uses two Gaussian weighting functions to consider the spatial distance and color similarity between pixels, respectively. The spatial standard deviation is set based on different requirements. To preserve the teacher mask's details, a smaller value is used; to enhance the denoising effect, a larger value is used. The range standard deviation is determined based on the standard deviation of the pixel values in the teacher's facial region. The denoised teacher region is calculated using a formula.

[0090] Finally, the original student image area covered by the student mask can be directly retained, and the denoised teacher area and the original student image area can be accurately fused according to the teacher mask and the student mask to finally generate a standardized image sequence, providing a standardized and clear data basis for subsequent classroom behavior analysis.

[0091] The bilateral filtering algorithm is used to filter the teacher mask During the denoising process, a motion compensation mechanism is introduced, which includes:

[0092] Masks for the teachers Motion blur noise in the spatial domain is added with a motion compensation term in the Gaussian weight function. ,in, represents the motion vector of pixel p; represents the motion vector of pixel q, then the updated spatial domain Gaussian weight function is .

[0093] It's understandable that when using the bilateral filtering algorithm to denoise the teacher mask, a motion compensation mechanism can be introduced to effectively handle the motion blur noise. Bilateral filtering originally achieves denoising by using Gaussian weighting functions in the spatial and value domains, taking into account both the spatial distance and color similarity between pixels. The motion compensation mechanism adds a motion compensation term to the spatial Gaussian weighting function, which is constructed based on the pixel's motion vector.

[0094] Furthermore, the pixel's motion vector represents the direction and distance of the pixel's movement between video frames. By incorporating the pixel's motion vector into the calculation of the spatial domain Gaussian weighting function, the updated function is able to perceive and utilize pixel motion information. When processing teacher masks with motion blur noise, the motion-inclusive spatial domain Gaussian weighting function more accurately weights adjacent pixels, effectively preserving image details and edges while removing noise, thus improving denoising and image quality.

[0095] The CNN is used to extract features from the standardized image sequence, that is, the improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion is used to locate the key points of the student's face, combined with the multi-joint head posture estimation model, to identify the student's attendance status, head posture and facial micro-expressions, specifically including:

[0096] Through the improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion, the weights of each layer of the network are dynamically allocated according to the complexity of the facial area in the standardized image sequence, and the facial feature information at different scales is integrated to locate the 68 key points of the student's face;

[0097] Synchronously access the multi-joint head posture estimation model to calculate the three-dimensional posture parameters of the student's head in real time, and the multi-joint head posture estimation model at least includes a skeleton point detection module and an angle calculation module based on a convolutional neural network;

[0098] A two-layer detection mechanism of facial presence status and head posture matrix is established to identify the student's attendance status, head posture and facial micro-expressions based on the student's facial key points and the student's three-dimensional head posture parameters.

[0099] To localize student facial landmarks, an improved multi-task cascaded convolutional neural network based on adaptive weight adjustment and multi-scale feature fusion can be used. Traditional multi-task cascaded convolutional neural networks, when processing facial images of varying complexity, have fixed weights at each layer, potentially resulting in poor feature extraction. The improved network, however, has adaptive capabilities, dynamically assigning weights to each layer based on the complexity of the facial region in a standardized image sequence, such as varying lighting conditions, occlusions, or pose changes. When strong lighting creates complex facial contrast, or when obstructions such as glasses or hair are present, the network automatically increases the weights of key layers to better extract facial features. Furthermore, the network integrates facial feature information at different scales, capturing the overall facial outline at small scales and more detailed information at larger scales. Through multi-scale feature fusion, 68 student facial landmarks, including key coordinates of the eyes, eyebrows, nose, and mouth, can be accurately located for both small telephoto and large close-up shots.

[0100] When estimating head posture, a multi-joint head posture estimation model can be integrated simultaneously. This model consists of a convolutional neural network-based skeleton point detection module and an angle calculation module. Specifically, the skeleton point detection module uses the powerful feature extraction capabilities of the convolutional neural network to identify the position information of key points of the head skeleton. The angle calculation module then calculates the three-dimensional posture parameters of the student's head, such as pitch, yaw, and roll angles, in real time based on the detected skeleton points, thereby determining the head's posture in three-dimensional space.

[0101] To accurately identify student attendance, head posture, and facial micro-expressions, a dual-layer detection mechanism based on facial presence and head posture matrices can be established. This mechanism, on the one hand, determines whether the student's face is fully present in the image based on the located facial key points, thereby identifying the student's attendance status. On the other hand, head posture can be analyzed by combining the student's three-dimensional head posture parameters with facial key point information. For example, if the head pitch angle exceeds a certain threshold, it is determined to be in a lowered state, while if it falls below the threshold, it is in an raised state. Furthermore, by observing the relative position changes between facial key points, such as a raised corner of the mouth or a furrowed brow, facial micro-expressions can be identified. This allows for a comprehensive understanding of students' behavior and emotional state in the classroom, providing rich and accurate feature data for classroom behavior analysis.

[0102] The bidirectional LSTM network is used to perform time series modeling on the standardized speech sequence to obtain the teacher's speaking time and frequency and the student's speaking time and frequency, specifically including:

[0103] Converting the standardized speech sequence into a Mel-frequency cepstral coefficient sequence, and inputting the Mel-frequency cepstral coefficient sequence into the bidirectional LSTM network for time series modeling;

[0104] Extracting speech energy, zero-crossing rate, and fundamental frequency features from the Mel-frequency cepstral coefficient sequence through a preset sliding window, and combining the attention mechanism to obtain key speech periods;

[0105] Construct a teacher-student speech classifier to distinguish between the teacher's teaching period and the student's interaction period in the key speech period, and output the teacher's speech duration and frequency as well as the student's speech duration and frequency.

[0106] Mel-frequency cepstral coefficients are a widely used characteristic parameter in speech processing. They simulate the characteristics of the human auditory system, performing frequency domain analysis on the Mel-scale before calculating the cepstral coefficients. This conversion effectively extracts important features relevant to human perception and removes redundant information.

[0107] After the conversion is complete, the Mel-frequency cepstral coefficient sequence can be input into a bidirectional long short-term memory network for time series modeling. As a special type of recurrent neural network, the bidirectional long short-term memory network is unique in its ability to process information from both the beginning and end of the speech sequence, thereby more comprehensively capturing the contextual information before and after the speech. This overcomes the gradient vanishing or gradient exploding issues that traditional recurrent neural networks are prone to when processing long sequences of data, providing strong support for accurately analyzing speech time series features.

[0108] Secondly, a preset sliding window can be used to extract speech energy, zero-crossing rate, and fundamental frequency features from the Mel-frequency cepstral coefficient sequence. The sliding window slides across the sequence data at a fixed step size, intercepting a segment of data each time for feature extraction. Specifically, speech energy reflects the strength of the speech signal, the zero-crossing rate indicates the number of times the speech signal crosses the zero axis per unit time, and the fundamental frequency is related to the pitch of the speech. These features describe the characteristics of the speech signal from different perspectives. Combined with the attention mechanism, the bidirectional LSTM network can automatically focus on key speech segments that are important for analyzing the speech of teachers and students, ignoring irrelevant or minor speech segments, thereby improving model processing efficiency and analysis accuracy. For example, in a speech sequence containing a teacher-student conversation and classroom background noise, the attention mechanism can help the network identify the truly informative portions of the teacher-student speech while eliminating noise interference.

[0109] Finally, a teacher-student speech classifier can be constructed to categorize key speech periods. This classifier, trained using deep learning or machine learning algorithms, can accurately distinguish between teacher lectures and student interaction periods within key speech periods based on extracted speech features. During a teacher's lecture, speech features may exhibit specific rhythm and pitch patterns; while speech features during student interaction periods vary in volume and speed. By learning these differences, the classifier can achieve accurate classification and output the duration and frequency of teacher and student speeches. This data provides a quantitative basis for analyzing teacher-student interaction patterns and teaching engagement in the classroom, helping to gain a deeper understanding of classroom teaching dynamics.

[0110] The step of aggregating the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associating the multi-dimensional classroom behavior portrait with classroom teaching effectiveness specifically includes:

[0111] Performing feature normalization processing on the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a classroom behavior indicator set X;

[0112] Obtaining a preset number of clusters, grouping the classroom behavior indicator set X using a K-means algorithm to obtain multiple clusters, calculating the center vector of each cluster and constructing the multi-dimensional classroom behavior profile Y;

[0113] Establish a classroom teaching effect indicator set E, use a multiple linear regression model to fit the linear relationship between the multidimensional classroom behavior portrait Y and the classroom teaching effect indicator set E, and screen significant correlation indicators through t-test;

[0114] The SHAP value is introduced to calculate the contribution of each indicator in the classroom behavior indicator set X to the classroom teaching effect, and the indicators in the classroom behavior indicator set X are sorted from large to small according to the contribution, and the top A indicators are selected as key classroom behavior indicators, where A represents the number of preset key indicators;

[0115] Calculate the Pearson correlation coefficient between the key classroom behavior indicators and the classroom teaching effect and construct a behavior-effect correlation heat map.

[0116] In practical applications, due to the differences in the numerical range and dimension of different indicators, for example, the student attendance rate is a percentage value, while the teacher feedback time is measured in minutes. Directly using the original data will affect the accuracy of the analysis and the performance of the model. Therefore, through normalization processing, all indicators can be converted into a unified numerical range, eliminating the dimensional effect, and generating a set of classroom behavior indicators containing various types of classroom behavior data.

[0117] Next, the K-means algorithm can be used to perform cluster analysis on the set of classroom behavior indicators. Before clustering, the number of clusters must be predefined, determined by the complexity of the classroom behavior data and the purpose of the analysis. The K-means algorithm uses distance as a metric to group similar classroom behavior data into the same cluster. By iteratively calculating the center vector of each cluster, it ultimately generates multiple groups with similar behavioral characteristics. These groups and their center vectors together construct a multidimensional classroom behavior profile that can intuitively present different behavioral patterns and characteristic distributions within the classroom.

[0118] Subsequently, a set of classroom teaching effectiveness indicators can be established, encompassing multiple dimensions of teaching effectiveness evaluation, including knowledge mastery, ability development, and changes in emotional attitudes. Using a multivariate linear regression model, a linear relationship can be fitted between the multidimensional classroom behavior profile and the set of classroom teaching effectiveness indicators, with model parameter estimation used to describe the correlation trend between the two. Furthermore, t-tests can be used to perform significance tests on each indicator in the regression model, identifying classroom behavior indicators that have a significant impact on classroom teaching effectiveness while eliminating irrelevant or less influential indicators.

[0119] Furthermore, SHAP values can be introduced to quantify the contribution of each indicator in the classroom behavior indicator set to classroom teaching effectiveness. It should be noted that SHAP values are based on game theory principles and can fairly distribute the importance of each feature in model prediction. By calculating the SHAP value of each indicator and sorting the indicators in the classroom behavior indicator set from large to small according to their contribution, a preset number of indicators are selected as key classroom behavior indicators. These key indicators play an important role in influencing classroom teaching effectiveness.

[0120] Finally, we can calculate the Pearson correlation coefficient between key classroom behavior indicators and classroom teaching effectiveness. This coefficient measures the degree of linear correlation between two variables. Based on the calculated correlation coefficient, we can construct a behavior-effect correlation heat map, visually demonstrating the strength and direction of the correlation between key classroom behavior indicators and classroom teaching effectiveness. The color depth in the heat map represents the magnitude of the correlation coefficient, with darker colors indicating stronger correlations. This allows us to intuitively understand the classroom behaviors that have a significant impact on teaching effectiveness, providing a scientific basis for teaching improvement and optimization.

[0121] like Figure 2 FIG. 1 is a system block diagram of a classroom behavior analysis system provided by an embodiment of the present invention. The analysis system includes:

[0122] A data acquisition module is used to acquire classroom video stream data and teacher-student audio stream data, and perform image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence; and perform noise reduction, sound source localization, and speech segmentation on the teacher-student audio stream data to generate a standardized speech sequence;

[0123] A data analysis module is used to extract features from the standardized image sequence using a CNN. Specifically, the module locates key points on the student's face using an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion. This module, combined with a multi-joint head pose estimation model, identifies the student's attendance status, head pose, and facial micro-expressions. Furthermore, the module uses a bidirectional LSTM network to perform time series modeling on the standardized speech sequence to obtain the duration and frequency of the teacher's speech and the duration and frequency of the student's speech.

[0124] An indicator construction module is used to construct a student individual behavior indicator system based on the student's attendance status, head posture, and facial micro-expressions, including student attendance rate, head-up rate, head-down duration, and frequency of expression changes; and to construct a teacher-student interaction behavior indicator system based on the teacher's speaking time and frequency and the student's speaking time and frequency, including the proportion of teacher-student speaking, the frequency of student questions, and the duration of teacher feedback;

[0125] The summary association module is used to summarize the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associate the multi-dimensional classroom behavior portrait with the classroom teaching effect.

[0126] Figure 2 The apparatus of the embodiment shown can be used to perform Figure 1 The implementation principles and technical effects of the steps in the method embodiment shown are similar and will not be repeated here.

[0127] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the steps of any one of the above-described classroom behavior analysis methods.

[0128] like Figure 3 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. The electronic device 30 includes: a processor 31, a memory 32 and a computer program;

[0129] The memory 32 is used to store the computer program, which may also be a flash memory. The computer program is, for example, an application program or a functional module for implementing the above method.

[0130] The processor 31 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant description in the above method embodiment.

[0131] Optionally, the memory 32 may be independent or integrated with the processor 31 .

[0132] When the memory 32 is a device independent of the processor 31, the device may further include:

[0133] The bus 33 is used to connect the memory 32 and the processor 31 .

[0134] A readable storage medium stores a computer program, which, when executed by a processor, is used to implement the steps of any one of the classroom behavior analysis methods described above.

[0135] The readable storage medium may be a computer storage medium or a communication medium. Communication media include any medium that facilitates the transfer of computer programs from one location to another. Computer storage media may be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may be located in an application-specific integrated circuit (ASIC). In addition, the ASIC may be located in a user device. Of course, the processor and the readable storage medium may also exist as discrete components in a communication device. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0136] The present invention also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the methods provided in the various embodiments described above.

[0137] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0138] Through the introduction of the above embodiments, the present invention can obtain classroom video stream data and teacher-student audio stream data through classroom behavior analysis methods, systems, electronic devices and storage media, and perform image enhancement, noise filtering and video frame segmentation on classroom video stream data to generate standardized image sequences, perform noise reduction, sound source positioning and voice segmentation on teacher-student audio stream data to generate standardized voice sequences; use CNN to extract features from standardized image sequences, that is, locate student facial key points through an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion, combine with a multi-joint head posture estimation model to identify student attendance status, head posture and facial micro-expressions, and use a bidirectional LSTM network to perform feature extraction on standardized voice sequences. Perform time series modeling to obtain the length and frequency of teachers' speeches and the length and frequency of students' speeches; construct a student individual behavior indicator system based on students' attendance status, head posture and facial micro-expressions, including student attendance rate, head-up rate, head-down duration and frequency of expression changes; and construct a teacher-student interaction behavior indicator system based on the length and frequency of teachers' speeches and the length and frequency of students' speeches, including the proportion of teachers and students speaking, the frequency of students' questions and the duration of teacher feedback; summarize the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associate the multi-dimensional classroom behavior portrait with the classroom teaching effect, so as to effectively improve the accuracy of classroom behavior analysis, enhance data processing capabilities, and realize multi-dimensional comprehensive evaluation of classroom behavior.

[0139] The present invention improves the MTCNN cascade network, combines posture estimation technology, and constructs a two-layer mechanism of face detection and posture assistance, which can effectively solve the recognition problems in complex scenes such as low light and facial occlusion, and provide a reliable basis for accurately evaluating learning status. The present invention uses a distributed sensor network, combined with a bilateral filtering algorithm and a motion compensation mechanism, to achieve high-quality collection and preprocessing of classroom data, while supporting real-time streaming and batch analysis of classroom data, improving data processing efficiency, and meeting the low-latency requirements of large-scale education scenarios. The present invention can construct a multi-dimensional indicator system that includes individual behavior and teacher-student interaction, generate student behavior portraits through cluster analysis, and quantify the relationship between classroom behavior and teaching effectiveness through statistical modeling, providing scientific guidance for teaching intervention. The present invention adopts a full-process automated system design, supports the rapid deployment of multiple scenarios such as smart classrooms and online education, is compatible with multi-terminal display and distributed computing mechanisms, and can adapt to different classroom environments without manual calibration, significantly improving the intelligence level and resource adaptation capabilities of education management.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0142] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

Claims

1. Classroom behavior analysis method, characterized by: The analysis method comprises: Obtain classroom video stream data and teacher-student audio stream data, and perform image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence; perform noise reduction, sound source localization, and voice segmentation on the teacher-student audio stream data to generate a standardized voice sequence; A CNN is used to extract features from the standardized image sequence. Specifically, 68 key facial points of the students are located through an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion. A multi-joint head posture estimation model including a skeletal point detection module and an angle calculation module based on a convolutional neural network is combined to establish a two-layer detection mechanism of facial presence status and head posture matrix to identify students' attendance status, head posture, and facial micro-expressions. A bidirectional LSTM network is used to perform time series modeling on the standardized speech sequence. Speech energy, zero-crossing rate, and fundamental frequency features are extracted through a preset sliding window. The attention mechanism is used to obtain key speaking time periods, and the duration and frequency of teachers' speeches, as well as the duration and frequency of students' speeches, are obtained. Based on the students' attendance status, head posture, and facial micro-expressions, a student individual behavior indicator system is constructed, including student attendance rate, head-up rate, head-down duration, and frequency of expression changes. Based on the teacher's speaking time and frequency and the student's speaking time and frequency, a teacher-student interaction behavior indicator system is constructed, including the proportion of teacher-student speeches, the frequency of students' questions, and the duration of teacher feedback. The student individual behavior indicator system and the teacher-student interaction behavior indicator system are summarized to generate a multi-dimensional classroom behavior portrait, the SHAP value is introduced to calculate the contribution of each indicator to the classroom teaching effect, and the multi-dimensional classroom behavior portrait is associated with the classroom teaching effect.

2. The classroom behavior analysis method according to claim 1, characterized in that: The method of obtaining classroom video stream data and teacher-student audio stream data, performing image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence, and performing noise reduction, sound source localization, and voice segmentation on the teacher-student audio stream data to generate a standardized voice sequence specifically includes: Deploy multi-angle cameras and array microphones in the target teaching area to form a distributed sensor network; Based on the distributed sensor network, the classroom video stream data with a resolution of 720P or above is captured in real time through the RTSP protocol, and the teacher and student audio stream data with a sampling rate of 48kHz is synchronously collected. The classroom video stream data and the teacher and student audio stream data are temporally and spatially aligned through hardware timestamp synchronization technology and time calibration algorithm; The classroom video stream data is subjected to image enhancement, noise filtering, and video frame segmentation, that is, the scene contrast is improved through an adaptive histogram equalization method; the teacher and student areas in the video frame are identified based on the YOLO series of deep learning target detection models, and the student position is located by combining prior knowledge of the seating layout. Selective noise reduction processing is performed using a bilateral filtering algorithm, that is, noise is reduced in the teacher's area while retaining student image information. The continuous video stream is segmented into independent image frames at a fixed frame rate of 30fps to generate the standardized image sequence; The teacher-student audio stream data is subjected to noise reduction, sound source localization, and speech segmentation. That is, the Wiener filter algorithm is used to remove environmental noise. The Wiener filter algorithm is used to dynamically adjust the filter parameters according to the statistical characteristics of the signal and noise based on the minimum mean square error criterion to suppress environmental background noise. The short-time energy detection method is combined to filter out silent segments. The short-time energy detection method is used to automatically identify and remove silent segments in the audio, reducing the amount of invalid data. The sound source localization processing uses beamforming technology to perform spatial domain processing on the audio signal. By adjusting the weighting coefficients of each channel of the array microphone, the direction of the sound source is located and a beam pointing in the direction of the sound source is formed. The voiceprint features of the teacher and student voices are trained using the GMM-UBM model to construct a personalized acoustic model and automatically separate the teacher's voice signal from the student's voice signal. The voice activity detection algorithm is used to automatically detect the start and end boundaries of the speech by analyzing the short-time energy and zero-crossing rate of the audio signal, and the continuous voice stream is divided into independent valid voice segments. A timestamp is added to each valid voice segment to generate the standardized voice sequence with a timestamp.

3. The classroom behavior analysis method according to claim 2, characterized in that: Based on the target detection model, prior knowledge of the seating layout, and a bilateral filtering algorithm, the teacher image noise is removed, the student image information is retained, and the standardized image sequence is generated, specifically including: For the classroom video stream data, the teacher's body area is identified based on the target detection model, and a teacher mask M marked with the teacher's position is generated. T ; Based on the prior knowledge of the seating layout and combined with the background difference method, the student body area is extracted to generate the student mask M S =1-M T , completing the spatial separation of teacher and student images in the classroom video stream data; The bilateral filtering algorithm is used to filter the teacher mask M T Perform denoising to generate the denoised teacher area. The corresponding calculation formula is as follows: Where B T (p) represents the pixel p in the teacher area after denoising; w p represents the normalization factor at pixel p; N(p) represents the spatial neighborhood of pixel p; σ s Represents the spatial standard deviation. If the teacher mask M is retained T The details in , then σ s =2; if the teacher mask M T For enhanced denoising, σ s =8; represents the spatial domain Gaussian weight function, which is used to control the spatial influence range of the neighborhood pixels, and σ r represents the standard deviation of the range, Represents the standard deviation of the pixel values in the teacher's face area; I T (p) represents the pixel p in the original teacher image area; I T (q) represents pixel q in the teacher’s original image region; represents the range Gaussian weight function, which is used to control the influence of color similarity of neighboring pixels, and Directly retain the student mask M S Covered student original image area I S , and the denoised teacher region B T The original image region I S The fusion is performed according to the corresponding mask to generate the standardized image sequence. The corresponding calculation formula is as follows: I final =M T *B T +M S *I S Where, I final represents a standardized image sequence; M T Indicates the teacher mask; B T represents the teacher region after denoising; M S Indicates student mask; I S Represents the student original image area.

4. The classroom behavior analysis method according to claim 3, characterized in that: The bilateral filtering algorithm is used to filter the teacher mask M T During the denoising process, a motion compensation mechanism based on pixel motion vectors is introduced to handle motion blur noise, specifically including: For the teacher mask M T Motion blur noise in the spatial domain is added with a motion compensation term in the Gaussian weight function. Among them, V m (p) represents the motion vector of pixel p; V m (q) represents the motion vector of pixel q, then the updated spatial domain Gaussian weight function is Among them, the pixel's motion vector represents the pixel's movement direction and distance between video frames. By incorporating the pixel's motion vector into the calculation of the spatial domain Gaussian weight function, the updated spatial domain Gaussian weight function is used to perceive and utilize the pixel's motion information.

5. The classroom behavior analysis method according to claim 1, characterized in that: The CNN is used to extract features from the standardized image sequence, that is, the improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion is used to locate the key points of the student's face, combined with the multi-joint head posture estimation model, to identify the student's attendance status, head posture and facial micro-expressions, specifically including: Through the improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion, the weights of each layer of the network are dynamically allocated according to the complexity of the facial area in the standardized image sequence, combined with the lighting conditions, occlusion or posture changes. When the strong lighting causes complex light and dark contrast on the face, or when there are multiple occluders, the network weights of the key layers are automatically enhanced. At the same time, facial feature information at different scales is integrated, that is, the overall facial contour is captured at a small scale, and facial detail information is obtained at a large scale. The 68 key points of the student's face and the corresponding key coordinate points are located; Synchronously access the multi-joint head posture estimation model to calculate the three-dimensional posture parameters of the student's head in real time, and the multi-joint head posture estimation model at least includes a skeleton point detection module and an angle calculation module based on a convolutional neural network, the skeleton point detection module is used to use the feature extraction capability of the convolutional neural network to identify the position information of the key points of the head skeleton, and the angle calculation module is used to calculate the three-dimensional posture parameters of the student's head, namely the pitch angle, yaw angle and roll angle, in real time based on the detected skeleton key points, to determine the posture of the student's head in three-dimensional space; A double-layer detection mechanism of facial presence status and head posture matrix is established. Based on the student's facial key points and the student's three-dimensional head posture parameters, the student's attendance status, head posture and facial micro-expressions are identified. The double-layer detection mechanism is used to determine whether the student's face is completely presented in the picture based on the located student's facial key points, and identify the student's attendance status; combine the student's three-dimensional head posture parameters and the student's facial key points to analyze the head posture. When the head pitch angle exceeds the threshold, it is judged to be in a lowered state; when the head pitch angle is lower than the threshold, it is judged to be in a raised state; by observing the relative position changes between the facial key points, facial micro-expressions, including raised corners of the mouth and furrowed brows, are identified to determine the student's behavior and emotional state in the classroom.

6. The classroom behavior analysis method according to claim 1, characterized in that: The bidirectional LSTM network is used to perform time series modeling on the standardized speech sequence to obtain the teacher's speaking time and frequency and the student's speaking time and frequency, specifically including: Converting the standardized speech sequence into a Mel-frequency cepstral coefficient sequence, and inputting the Mel-frequency cepstral coefficient sequence into the bidirectional LSTM network for time series modeling; The speech energy, zero-crossing rate, and fundamental frequency features in the Mel-frequency cepstral coefficient sequence are extracted through a preset sliding window. The preset sliding window slides on the Mel-frequency cepstral coefficient sequence at a fixed step size, intercepting a section of data each time for feature extraction. The speech energy represents the strength of the speech signal, the zero-crossing rate represents the number of times the speech signal crosses the zero axis per unit time, and the fundamental frequency is related to the pitch of the speech signal. Combined with the attention mechanism, the bidirectional LSTM network is used to obtain key speech periods with effective information. A teacher-student speech classifier based on machine learning algorithm training is constructed to classify key speech periods. The teacher teaching period and student interaction period in the key speech period are distinguished based on the extracted speech features. The output is used to quantitatively analyze the teacher-student interaction pattern and teaching participation in the classroom. The speech features of the teacher teaching period are different in rhythm and tone patterns; the speech features of the student interaction period are different in volume and speaking speed.

7. The classroom behavior analysis method according to claim 1, characterized in that: The step of aggregating the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associating the multi-dimensional classroom behavior portrait with classroom teaching effectiveness specifically includes: Performing feature normalization processing on the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a classroom behavior indicator set X containing various types of classroom behavior data; Obtaining a preset number of clusters determined based on the complexity of the classroom behavior data and the purpose of the analysis, using the K-means algorithm and distance as a metric to group the classroom behavior indicator set X, dividing similar classroom behavior data into the same cluster, obtaining multiple clusters with similar behavioral characteristics, calculating the central vector of each cluster through continuous iteration, and jointly constructing the multidimensional classroom behavior portrait Y based on the clusters and their central vectors. The multidimensional classroom behavior portrait Y is used to intuitively present different behavioral patterns and characteristic distributions in the classroom; Establish a classroom teaching effect indicator set E, use a multiple linear regression model to fit the linear relationship between the multidimensional classroom behavior portrait Y and the classroom teaching effect indicator set E, describe the correlation trend between the multidimensional classroom behavior portrait Y and the classroom teaching effect indicator set E through model parameter estimation, and perform a significance test on each indicator in the regression model through a t-test to screen out classroom behavior indicators that have a significant impact on classroom teaching effects, namely, significant correlation indicators. Among them, the classroom teaching effect indicator set E covers teaching effect evaluation indicators in the dimensions of knowledge mastery, ability development, and emotional attitude changes; The SHAP value is introduced. Based on the principles of game theory, the contribution of each indicator in the classroom behavior indicator set X to the classroom teaching effect is calculated. The indicators in the classroom behavior indicator set X are sorted from large to small according to the contribution, and the top A indicators are selected as key classroom behavior indicators, where A represents the number of preset key indicators. Calculate the Pearson correlation coefficient between the key classroom behavior indicators and the classroom teaching effect. The Pearson correlation coefficient is used to measure the degree of linear correlation between the key classroom behavior indicators and the classroom teaching effect, and construct a behavior-effect correlation heat map. The behavior-effect correlation heat map is used to visually display the correlation strength and direction between the key classroom behavior indicators and the classroom teaching effect. The color depth in the behavior-effect correlation heat map represents the size of the Pearson correlation coefficient. The darker the color, the larger the Pearson correlation coefficient, and the stronger the correlation between the key classroom behavior indicators and the classroom teaching effect.

8. A classroom behavior analysis system, applied to the classroom behavior analysis method according to any one of claims 1 to 7, characterized in that: The analysis system comprises: A data acquisition module is used to acquire classroom video stream data and teacher-student audio stream data, and perform image enhancement, noise filtering, and video frame segmentation on the classroom video stream data to generate a standardized image sequence; and perform noise reduction, sound source localization, and speech segmentation on the teacher-student audio stream data to generate a standardized speech sequence; A data analysis module is used to extract features from the standardized image sequence using a CNN. Specifically, the module locates key points on the student's face using an improved MTCNN cascade network based on adaptive weight adjustment and multi-scale feature fusion. This module, combined with a multi-joint head pose estimation model, identifies the student's attendance status, head pose, and facial micro-expressions. Furthermore, the module uses a bidirectional LSTM network to perform time series modeling on the standardized speech sequence to obtain the duration and frequency of the teacher's speech and the duration and frequency of the student's speech. An indicator construction module is used to construct a student individual behavior indicator system based on the student's attendance status, head posture, and facial micro-expressions, including student attendance rate, head-up rate, head-down duration, and frequency of expression changes; and to construct a teacher-student interaction behavior indicator system based on the teacher's speaking time and frequency and the student's speaking time and frequency, including the proportion of teacher-student speaking, the frequency of student questions, and the duration of teacher feedback; The summary association module is used to summarize the student individual behavior indicator system and the teacher-student interaction behavior indicator system to generate a multi-dimensional classroom behavior portrait, and associate the multi-dimensional classroom behavior portrait with the classroom teaching effect.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor runs the computer program stored in the memory, the processor performs the steps of the classroom behavior analysis method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it is used to implement the steps of the classroom behavior analysis method according to any one of claims 1 to 7.

Images