Intelligent analysis and evaluation method and system for teacher teaching behavior based on multi-modal data

Abstract

The application provides a teacher teaching behavior intelligent analysis and evaluation method and system based on multi-modal data, relates to the technical field of intelligent analysis and evaluation, synchronously collects classroom audio stream, classroom video stream and teaching courseware image stream in a teaching scene to form a multi-modal data sequence; video behavior features, speech behavior features and courseware behavior features in the multi-modal data sequence are respectively extracted, cross-modal feature fusion is performed, and a fusion feature sequence is generated; time sequence behavior detection is performed based on the fusion feature sequence, and a teacher comprehensive behavior analysis result is output; and a structured teaching behavior evaluation report is generated according to the teacher comprehensive behavior analysis result. The application can realize intelligent analysis on teacher teaching behavior through hierarchical behavior detection and phased differentiated evaluation, and improve the accuracy of behavior recognition and the rationality of evaluation results.

Description

Technical Field

[0001] This application relates to the field of intelligent analysis and evaluation technology, and more specifically, to a method and system for intelligent analysis and evaluation of teachers' teaching behavior based on multimodal data. Background Technology

[0002] Current classroom teaching behavior analysis technologies primarily employ single-modal data for evaluation, such as teacher action recognition based on classroom video streams or voice sentiment analysis based on classroom audio streams. Some technologies attempt to integrate multimodal data, typically using a combination of video and audio modalities to classify and identify teacher behavior before outputting statistical results. In terms of behavior detection, existing technologies often employ a flat detection structure, uniformly detecting all behavior categories across the timeline without distinguishing between teaching stages. Regarding evaluation methods, current technologies typically use uniform evaluation indicators and scoring standards across the entire classroom timeframe to generate a holistic evaluation result.

[0003] However, existing technologies still have shortcomings in practical teaching scenarios. Current behavior detection typically adopts a flat structure, uniformly detecting all behavior categories across the entire lesson timeline, without considering the constraints imposed on teacher behavior at different teaching stages. This can lead to isolated false positives that do not conform to the teaching logic. Existing assessment methods often use uniform indicators and fixed standards for scoring across the entire classroom, ignoring the differences in teaching objectives and focuses at different teaching stages. The refinement and interpretability of assessment results need improvement. Therefore, how to achieve intelligent analysis of teacher teaching behavior through hierarchical behavior detection and phased differentiated assessment, thereby improving the accuracy of behavior recognition and the rationality of assessment results, is a challenge facing the industry. Summary of the Invention

[0004] This application provides a method and system for intelligent analysis and evaluation of teachers' teaching behavior based on multimodal data. It can achieve intelligent analysis of teachers' teaching behavior through hierarchical behavior detection and phased differentiated evaluation, thereby improving the accuracy of behavior recognition and the rationality of evaluation results.

[0005] Firstly, this application provides an intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data, the intelligent analysis and evaluation method comprising the following steps: The system simultaneously collects classroom audio streams, classroom video streams, and teaching courseware image streams from the teaching scenario, forming a multimodal data sequence. Video behavior features, speech behavior features, and courseware behavior features are extracted from the multimodal data sequence, and cross-modal feature fusion is performed to generate a fused feature sequence. Based on the fused feature sequence, temporal behavior detection is performed, and the comprehensive behavior analysis results of teachers are output. Based on the results of the comprehensive teacher behavior analysis, a structured teaching behavior evaluation report is generated according to the preset multi-dimensional evaluation indicators.

[0006] In this embodiment, the simultaneous acquisition of classroom audio streams, classroom video streams, and teaching courseware image streams in a teaching scenario to form a multimodal data sequence specifically includes: Classroom video streams are captured by video capture devices deployed in classrooms, classroom audio streams are captured by audio capture devices, and teaching courseware image streams are obtained by capturing the screen of the teaching computer. During the acquisition process, each frame of the classroom video stream, each frame of the classroom audio stream, and each frame of the teaching courseware image stream are timestamped based on the same clock source. After the data acquisition is completed, using the timeline of the classroom video stream as the reference timeline, frames in the classroom audio stream and the teaching courseware image stream with timestamp deviations within a preset threshold are established with frames in the classroom video stream corresponding to the timestamps, forming a frame-level aligned multimodal data sequence.

[0007] In this embodiment, video behavior features, speech behavior features, and courseware behavior features are extracted from the multimodal data sequence, and cross-modal feature fusion is performed to generate a fused feature sequence, specifically including: Teacher target detection and tracking are performed on the classroom video stream, the teacher's human skeleton sequence is extracted, the teacher's body movement categories are identified, and a video behavior feature vector is generated. Speech recognition is performed on the classroom audio stream to extract the text features of the teacher's speech. Based on the audio frame energy value and the detection results of silent segments, the time boundary of the teacher-student turn-taking is marked to generate a speech behavior feature vector. Inter-frame difference analysis is performed on the image stream of the teaching courseware to extract the timestamps of page turning events and element change events within the courseware, and to generate a courseware behavior feature vector; After aligning the video behavior feature vector, the speech behavior feature vector, and the courseware behavior feature vector by timestamp, they are input into a cross-modal feature fusion network, which outputs a fused feature sequence.

[0008] In this embodiment, the temporal behavior detection based on the fused feature sequence and the output of the teacher's comprehensive behavior analysis results specifically include: The fused feature sequence is input into a hierarchical behavior detection structure, which includes a first detection layer and a second detection layer. In the first detection layer, the fused feature sequence is scanned with a first time resolution to identify the teaching stage category of the teacher and output the start and end timestamps of each teaching stage. The teaching stage categories include the introduction stage, the lecture stage, the interaction stage, and the summary stage. In the second detection layer, based on the teaching stage category output by the first detection layer, a set of behavior categories and a behavior state transition probability matrix corresponding to the teaching stage category are loaded. Within the time interval defined by the start and end timestamps of each teaching stage, the fused feature sequence is scanned with a second time resolution higher than the first time resolution to identify specific teaching behavior instances within the stage and output the behavior recognition results within the stage. The behavior recognition results within the stage include behavior category labels and their start and end timestamps. The teaching stage identification results are combined with the behavior identification results within the stage to output the teacher comprehensive behavior analysis results. The teacher comprehensive behavior analysis results include teaching stage category labels, teaching behavior category labels, start and end timestamps corresponding to each label, and the teaching stage index to which each teaching behavior belongs.

[0009] In this embodiment, the method for obtaining the behavioral state transition probability matrix specifically includes: Collect multimodal classroom data from multiple teachers, and label the boundaries of teaching stages and the specific teaching behavior sequences within each teaching stage in the classroom data. The frequency of transitions between adjacent teaching behavior pairs within the same teaching stage is statistically analyzed to construct a teaching behavior transition frequency matrix; The teaching behavior transition frequency matrix is ​​normalized row by row so that the sum of the elements in each row is 1, thus obtaining the behavior state transition probability matrix.

[0010] In this embodiment, based on the results of the comprehensive teacher behavior analysis and according to preset multi-dimensional evaluation indicators, the generation of a structured teaching behavior evaluation report specifically includes: Based on the teaching stage category labels and their start and end timestamps in the teacher comprehensive behavior analysis results, the teacher comprehensive behavior analysis results for the whole lesson are divided into teaching stages to obtain the behavioral data subsets corresponding to each teaching stage. For each teaching stage, according to the corresponding preset assessment focus, a subset of assessment indicators activated in that stage is selected from the preset multi-dimensional assessment indicators, and behavioral instances related to the activated assessment indicator subset are extracted from the behavioral data subset of that stage for indicator calculation to obtain the stage assessment score for each teaching stage. The stage assessment scores of each teaching stage are weighted and combined with the global assessment scores calculated over the entire classroom time window to generate a structured teaching behavior assessment report.

[0011] In this embodiment, extracting behavioral instances from the subset of behavioral data for that stage that are related to the activated subset of evaluation indicators and calculating the indicators to obtain the stage evaluation score for each teaching stage specifically includes: Obtain the original calculated value of each indicator in the current teaching stage's activated subset of assessment indicators within that stage; Obtain the preset stage standard value of the corresponding indicator. The preset stage standard value is a reference value obtained by statistical analysis based on the calculation results of the corresponding indicators of multiple teachers in the same teaching stage. Calculate the degree of deviation between the original calculated value and the preset stage standard value within the teaching stage, map the degree of deviation to the preset scoring range, and obtain the stage evaluation score of the corresponding indicator in the current teaching stage.

[0012] In this embodiment, the weighted summation of the stage assessment scores for each teaching stage with the global assessment scores calculated over the entire class time window specifically includes: The proportion of the duration of each teaching stage to the total class time is used as the duration weighting coefficient for each stage. Obtain the pre-configured importance weight coefficients for each teaching stage, wherein the importance weight coefficients are set according to the degree of emphasis on each teaching stage in the teaching evaluation criteria; For each evaluation indicator, the global evaluation score calculated over the entire class time window and the stage evaluation scores for each teaching stage are weighted and summed in three layers according to the global preset weight, the duration weight coefficient, and the importance weight coefficient to obtain the weighted comprehensive score of the indicator. The weighted scores of each indicator are summed to generate the total evaluation score in the teaching behavior evaluation report.

[0013] In this embodiment, the teaching behavior evaluation report includes: the total evaluation score for the entire class, the global evaluation score for each multi-dimensional evaluation indicator, the stage evaluation score for each teaching stage, and the sub-item score for each indicator.

[0014] Secondly, this application provides an intelligent analysis and evaluation system for teacher teaching behavior based on multimodal data, used to execute an intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data. The intelligent analysis and evaluation system includes: The data acquisition module is used to simultaneously collect classroom audio streams, classroom video streams, and teaching courseware image streams in the teaching scenario, forming a multimodal data sequence; The feature fusion module is used to extract video behavior features, speech behavior features, and courseware behavior features from the multimodal data sequence, respectively, and perform cross-modal feature fusion to generate a fused feature sequence. The behavior analysis module is used to perform time-series behavior detection based on the fused feature sequence and output the comprehensive behavior analysis results of teachers. The behavior assessment module is used to generate a structured teaching behavior assessment report based on the comprehensive analysis results of the teachers' behavior and according to preset multi-dimensional assessment indicators.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The system synchronously collects classroom audio streams, classroom video streams, and teaching courseware image streams from the teaching scenario to form a multimodal data sequence. Video behavior features, speech behavior features, and courseware behavior features are extracted from the multimodal data sequence, and cross-modal feature fusion is performed to generate a fused feature sequence. Based on the fused feature sequence, temporal behavior detection is performed, and a comprehensive teacher behavior analysis result is output. Based on the comprehensive teacher behavior analysis result, a structured teaching behavior evaluation report is generated according to preset multi-dimensional evaluation indicators.

[0016] Therefore, this application firstly, through three-modal frame-level synchronous acquisition, provides a precisely aligned multi-source data foundation for subsequent hierarchical behavior detection and phased differentiated assessment, ensuring the data quality of intelligent analysis of teachers' teaching behavior from the source; secondly, by extracting teaching-specific features in the teaching scenario and generating a fused feature sequence through cross-modal attention fusion, it provides a semantically rich unified feature representation for subsequent hierarchical behavior detection and phased differentiated assessment; and thirdly, by first identifying the macro-level teaching stage and then detecting specific behavior instances under stage constraints using the transition probability matrix, it outputs a hierarchical comprehensive teacher behavior score containing stage attribution relationships. The analysis results suppressed isolated false positives and fragmented detections, improved the coherence and rationality of behavioral sequences, and established a subordinate relationship between behaviors and teaching links through stage indexing, providing a structured data foundation for phased differentiated assessment. Finally, by activating different assessment indicators in stages and comparing them with differentiated stage standard values, and then through three-level weighted synthesis, a structured teaching behavior assessment report containing global assessment scores, stage scores, and item scores is generated. This ensures that the assessment dimensions are consistent with the actual functional goals of each teaching stage, supports one-click backtracking from assessment scores to key behavioral segments, and provides teachers with precise basis for teaching improvement.

[0017] In summary, the technical solution adopted in this application can achieve intelligent analysis of teachers' teaching behavior through hierarchical behavior detection and phased differentiated evaluation, thereby improving the accuracy of behavior recognition and the rationality of evaluation results. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an exemplary flowchart of the intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data provided in this application; Figure 2 This is a schematic diagram of the hierarchical behavior detection structure provided in this application; Figure 3 This is a schematic diagram of multimodal data synchronous acquisition provided in this application; Figure 4 This is a module structure diagram of the intelligent analysis and evaluation system for teacher teaching behavior based on multimodal data provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a method and system for intelligent analysis and evaluation of teacher teaching behavior based on multimodal data. Its core is to simultaneously collect classroom audio streams, classroom video streams, and teaching courseware image streams in a teaching scenario, forming a multimodal data sequence; extract video behavior features, speech behavior features, and courseware behavior features from the multimodal data sequence, and perform cross-modal feature fusion to generate a fused feature sequence; perform temporal behavior detection based on the fused feature sequence to output a comprehensive teacher behavior analysis result; and generate a structured teaching behavior evaluation report based on the comprehensive teacher behavior analysis result and preset multi-dimensional evaluation indicators.

[0022] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown in the figure, this is an exemplary flowchart of an intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data according to this embodiment of the present application. The intelligent analysis and evaluation method includes the following steps: In step S1, classroom audio stream, classroom video stream, and teaching courseware image stream are collected synchronously in the teaching scenario to form a multimodal data sequence.

[0023] In this embodiment, the simultaneous acquisition of classroom audio streams, classroom video streams, and teaching courseware image streams in a teaching scenario to form a multimodal data sequence specifically includes: Classroom video streams are captured by video capture devices deployed in classrooms, classroom audio streams are captured by audio capture devices, and teaching courseware image streams are obtained by capturing the screen of the teaching computer. During the acquisition process, each frame of the classroom video stream, each frame of the classroom audio stream, and each frame of the teaching courseware image stream are timestamped based on the same clock source. After the data acquisition is completed, using the timeline of the classroom video stream as the reference timeline, frames in the classroom audio stream and the teaching courseware image stream with timestamp deviations within a preset threshold are established with frames in the classroom video stream corresponding to the timestamps, forming a frame-level aligned multimodal data sequence.

[0024] In practical implementation, firstly, a video capture system is deployed in the classroom to capture the classroom video stream, an audio capture system to capture the classroom audio stream, and a screen capture program is run on the teaching computer to capture the teaching courseware image stream. Specifically, a synchronous capture system is set up in the classroom: a high-definition camera is installed at the back of the classroom as the video capture device to capture the classroom video stream covering the podium area; a wireless microphone worn by the teacher at their collar serves as the audio capture device to capture the teacher's voice; and a screen capture program runs in the background of the teaching computer to capture the screen of the teaching courseware used by the teacher, forming the teaching courseware image stream. Then, during the capture process, each frame of the classroom video stream, each frame of the classroom audio stream, and each frame of the teaching courseware image stream are timestamped based on the same clock source. That is, the video capture device sends a timestamp to the backend control program when each frame is fully exposed. The process involves three data streams: First, the system requests the current clock reading and writes it into the metadata field of the video frame. Second, the audio capture device appends the current clock reading when packaging each audio frame. Third, the screen capture program appends the current clock reading after each screenshot is completed. Thus, each frame of the three data streams carries a timestamp based on the same clock source. Finally, after acquisition, using the timeline of the classroom video stream as the reference timeline, frames in the classroom audio stream and teaching material image stream with timestamp deviations within a preset threshold are linked to their corresponding timestamp frames in the classroom video stream. This forms a frame-level aligned multimodal data sequence. Specifically, during alignment, each video frame in the classroom video stream is traversed, and its timestamp is recorded as the video frame moment. In the timestamp sequence of the classroom audio stream, audio frames whose absolute difference from the video frame moment is less than a preset deviation threshold (set to half the video frame period) are searched. If a matching audio frame exists, a synchronization mapping pair is established between the video frame and the audio frame. If multiple matching audio frames exist, the audio frame with the smallest absolute difference from the video frame moment is selected. In the timestamp sequence of the teaching courseware image stream, courseware frames that meet the time deviation condition are also searched and synchronization mapping pairs are established. For video frames that do not match a courseware frame, the most recently successfully matched courseware frame is used as its synchronization mapping pair. After the above alignment process, all synchronization mapping pairs are arranged in the order of the video frame timestamps, thus forming a frame-level aligned multimodal data sequence.

[0025] In step S2, video behavior features, speech behavior features, and courseware behavior features are extracted from the multimodal data sequence, and cross-modal feature fusion is performed to generate a fused feature sequence.

[0026] In this embodiment, video behavior features, speech behavior features, and courseware behavior features are extracted from the multimodal data sequence, and cross-modal feature fusion is performed to generate a fused feature sequence. This can be achieved through the following steps: Teacher target detection and tracking are performed on the classroom video stream, the teacher's human skeleton sequence is extracted, the teacher's body movement categories are identified, and a video behavior feature vector is generated. Speech recognition is performed on the classroom audio stream to extract the text features of the teacher's speech. Based on the audio frame energy value and the detection results of silent segments, the time boundary of the teacher-student turn-taking is marked to generate a speech behavior feature vector. Inter-frame difference analysis is performed on the image stream of the teaching courseware to extract the timestamps of page turning events and element change events within the courseware, and to generate a courseware behavior feature vector; After aligning the video behavior feature vector, the speech behavior feature vector, and the courseware behavior feature vector by timestamp, they are input into a cross-modal feature fusion network, which outputs a fused feature sequence.

[0027] In practical implementation, firstly, teacher target detection and tracking are performed on the classroom video stream. The teacher's human skeleton sequence is extracted, and the categories of the teacher's body movements are identified to generate video behavior feature vectors. Specifically, a pre-trained teacher detector is used to locate the teacher in each video frame, obtaining the teacher's bounding box. A tracking algorithm is then used to match the bounding boxes between adjacent frames, forming the teacher's continuous motion trajectory. Based on the obtained teacher bounding boxes, the key point sequence of the teacher's human skeleton is extracted. The key points include seventeen joints: head, shoulders, elbows, wrists, hips, knees, and ankles. This yields the teacher's skeleton spatial coordinate sequence in each video frame. Based on the skeleton spatial coordinate sequence, the categories of the teacher's body movements are identified, including three types: blackboard writing, pointing at courseware, and movement in the podium area. The recognition criteria for blackboard writing actions are the duration for which the teacher's wrist is above shoulder height and remains in the blackboard area; the recognition criteria for pointing actions on courseware are the teacher's arm extension direction towards the screen area and the angle between the fingertips and the screen normal being less than a preset angle threshold; the recognition criteria for moving actions in the podium area are the displacement and speed of the teacher's hip joint center coordinates within the podium area. The above-mentioned skeleton spatial coordinate sequence, action category labels, and trajectory coordinates together constitute the video behavior feature vector. This video behavior feature vector is output with the video frame rate as the temporal resolution, and each video frame corresponds to a set of video behavior features. Next, speech recognition is performed on the classroom audio stream to extract the teacher's speech text features. Based on the audio frame energy value and the silent segment detection results, the time boundary of teacher-student turn-taking is marked, and a speech behavior feature vector is generated. That is, the teacher's speech is converted into text using a pre-trained speech recognition engine to obtain the teacher's speech text sequence and the timestamp of each word. The energy value of the audio frame is extracted, and segments with consecutive frames of energy values ​​lower than a preset energy threshold are marked as silent segments. Based on the distribution of silent segments and the timestamp of the teacher's speech text, the time boundary of teacher-student turn-taking is marked. The recognition rule for teacher-student turn-taking is as follows: If a silence segment occurs after the teacher's speech, and the duration of the silence segment exceeds the preset minimum waiting time but does not exceed the preset maximum waiting time, and then an audio energy rebound is detected, the start time of this silence segment is marked as the end boundary of the teacher's turn, and the time of the energy rebound is marked as the start boundary of the student's response. If another silence segment occurs after the student's response, the time when the teacher resumes speaking is marked as the start boundary of the next teacher's turn. Thus, each teacher's speech and the subsequent student response constitute a complete turn-taking pair. The speech behavior feature vector includes: the teacher's speech text, the timestamp of each word, the start and end timestamps of the silence segment, and the teacher-student turn-taking boundary information, output at audio frame rate with time resolution.

[0028] Furthermore, in the specific implementation, inter-frame differential analysis is performed on the teaching courseware image stream to extract the timestamps of page-turning events and in-course element change events, generating a courseware behavior feature vector. Specifically, pixel-level difference calculations are performed on adjacent frames of courseware images, and the proportion of the difference pixels to the total number of pixels is counted. When this proportion exceeds a preset change threshold, it is determined that the courseware content has changed. The characteristics of page-turning events and in-course element change events are as follows: the difference pixel ratio is close to 100%, meaning almost the entire image changes; and the changed image remains stable in subsequent frames. The characteristics of in-course element change events are: the difference pixel ratio is less than 50%, meaning the main subject of the image remains unchanged, and only local content changes. The timestamps of the two types of events are recorded separately to generate a courseware behavior feature vector. The courseware behavior feature vector is output at a temporal resolution based on the courseware image stream frame rate. Each courseware frame corresponds to a set of courseware behavior features, including whether a page-turning event occurred, whether an element change event occurred, and the specific event type label. Finally, the video behavior feature vector, speech behavior feature vector, and courseware behavior feature vector are aligned by timestamp and input into a cross-modal feature fusion network, which outputs a fused feature sequence. That is, based on the temporal resolution of the video behavior feature vector, the speech behavior feature vector and the courseware behavior feature vector are aligned to the corresponding video frame positions through timestamp matching. For cases where the timestamps do not match perfectly, nearest neighbor interpolation is used for alignment. The aligned trimodal feature vectors are input into a cross-modal feature fusion network, which employs a multi-head attention structure: First, the video behavior feature vector, speech behavior feature vector, and courseware behavior feature vector are mapped to a unified feature dimension space through their respective fully connected layers to obtain video mapping features, speech mapping features, and courseware mapping features. Then, the video mapping features are used as query vectors, and attention is calculated with the speech mapping features and courseware mapping features respectively to obtain supplementary information of speech features on video features and supplementary information of courseware features on video features. Finally, the video mapping features are concatenated with the above two supplementary information, and then feature compression and fusion are performed through a fully connected layer to output a fused feature sequence.

[0029] It should be noted that by extracting teaching-specific features in the teaching scenario and generating a fusion feature sequence through cross-modal attention fusion, a semantically rich unified feature representation is provided for subsequent hierarchical behavior detection and phased differentiated evaluation.

[0030] In step S3, time-series behavior detection is performed based on the fused feature sequence, and the comprehensive behavior analysis results of the teacher are output.

[0031] In this embodiment, the temporal behavior detection based on the fused feature sequence and the output of the teacher's comprehensive behavior analysis results can be achieved through the following steps: The fused feature sequence is input into a hierarchical behavior detection structure, which includes a first detection layer and a second detection layer. In the first detection layer, the fused feature sequence is scanned with a first time resolution to identify the teaching stage category of the teacher and output the start and end timestamps of each teaching stage. The teaching stage categories include the introduction stage, the lecture stage, the interaction stage, and the summary stage. In the second detection layer, based on the teaching stage category output by the first detection layer, a set of behavior categories and a behavior state transition probability matrix corresponding to the teaching stage category are loaded. Within the time interval defined by the start and end timestamps of each teaching stage, the fused feature sequence is scanned with a second time resolution higher than the first time resolution to identify specific teaching behavior instances within the stage and output the behavior recognition results within the stage. The behavior recognition results within the stage include behavior category labels and their start and end timestamps. The teaching stage identification results are combined with the behavior identification results within the stage to output the teacher comprehensive behavior analysis results. The teacher comprehensive behavior analysis results include teaching stage category labels, teaching behavior category labels, start and end timestamps corresponding to each label, and the teaching stage index to which each teaching behavior belongs.

[0032] In practical implementation, firstly, the fused feature sequence is input into a hierarchical behavior detection structure. In the first detection layer, the fused feature sequence is coarsely scanned at a first temporal resolution to identify the teaching stage category of the teacher and output the start and end timestamps of each teaching stage. That is, the fused feature sequence is input into the hierarchical behavior detection structure, and after passing through the first detection layer, the fused feature sequence is segmented in the time dimension. Using a preset first time window length as the basic detection unit, the fused feature sequence of the entire lesson is divided into multiple continuous detection segments. Stage discrimination features are extracted for each detection segment. The stage discrimination features include three dimensions: the average and variance of the teacher's motion energy in the video behavior features within the segment; the average speech rate, volume variance, and proportion of silent segments in the teacher's speech behavior features within the segment; and the frequency of page-turning events and the cumulative number of element change events in the courseware behavior features within the segment. The extracted stage-discriminating features are matched with preset feature templates for each teaching stage category. In this embodiment, the teaching stage categories include four types: introduction stage, lecture stage, interaction stage, and summary stage. The preset feature templates for each stage are obtained by statistical analysis of classroom data from multiple experienced teachers. For example: the feature template for the introduction stage: the teacher has moderate physical energy, moderate average speaking speed, and small volume variance; the frequency of page turning in the courseware is low; and the changes in courseware elements are mainly static images. The feature template for the lecture stage: the teacher has moderate physical energy, relatively high average speaking speed, and small volume variance; the frequency of page turning in the courseware is high; and the frequency of changes in courseware elements is high. The feature template for the interaction stage: the teacher has relatively high physical energy, large variation in average speaking speed, and large volume variance; and the frequency of page turning in the courseware is low. The feature template for the summary stage: the teacher has relatively low physical energy, relatively low average speaking speed, and moderate frequency of page turning in the courseware. The first detection layer determines the teaching stage category to which the segment belongs by calculating the matching similarity between the stage-discriminating features of each detection segment and the above four types of stage feature templates, and outputs the category label and start and end timestamps for each teaching stage.

[0033] Furthermore, in the specific implementation, in the second detection layer, based on the teaching stage category output by the first detection layer, a set of behavior categories and a behavior state transition probability matrix corresponding to that teaching stage category are loaded. Within the time interval defined by the start and end timestamps of each teaching stage, a fine-grained scan of the fused feature sequence is performed at a second time resolution higher than the first time resolution to identify specific teaching behavior instances within the stage, and the behavior recognition results within the stage are output. That is, within the time interval defined by the start and end timestamps of each teaching stage output by the first detection layer, the second detection layer performs a fine-grained scan of the fused feature sequence at a higher second time resolution to identify specific teaching behavior instances within the stage. Different teaching stages correspond to different sets of behavior categories, and the set of behavior categories for each stage is predefined according to teaching rules. The method of obtaining the behavior state transition probability matrix will be explained in detail in subsequent steps; here, its application in detection is described first. When the second detection layer performs a sliding window scan along the time axis, predicting the behavior category for the fused feature segment within the current window relies not only on the segment's own features but also on the behavior categories identified in the previous window. Specifically, the feature classifier calculates the initial confidence score of the current segment belonging to each candidate behavior category in the set of behavior categories for that stage. The feature classifier takes the current segment of the fused feature sequence as input and outputs the probability value of the segment belonging to each candidate behavior category, normalizing the sum of the probability values ​​for all candidate behavior categories to 100%. From the behavior state transition probability matrix, the probability value of transitioning from the behavior category identified in the previous window to each candidate behavior category is retrieved. Each row of the behavior state transition probability matrix corresponds to a previous behavior category, and each column corresponds to a possible current behavior category. The values ​​in the matrix represent the statistical probability of transitioning from the previous behavior to the current behavior in the current teaching stage. The initial confidence scores of each candidate behavior category are weighted and fused with their corresponding transition probabilities to obtain the corrected confidence scores. The specific calculation method for weighted fusion is as follows: for each candidate behavior category, the initial confidence score is multiplied by the transition probability value, and the square root of the product is taken. This result is the corrected confidence score for that candidate behavior category. This calculation method ensures that the corrected confidence score is significantly prominent only when both the initial confidence score and the transition probability are high; if either value is low, the corrected confidence score will be effectively suppressed. After calculating the corrected confidence scores for all candidate behavior categories, normalization is performed again to ensure that the sum of the corrected confidence scores for all candidate behavior categories is 100%. The candidate behavior category with the highest corrected confidence score is selected as the final behavior category prediction result for the current window. After completing the behavior category prediction for the current window, the prediction result of the current window is used as the previous behavior category for the next window. The above steps are repeated, recursively along the time axis, until all sliding windows within this teaching stage are traversed.For the first window at the start of each teaching stage, since there are no behavior categories identified in the previous window, the behavior category for this window is determined directly based on the initial confidence level, without any transition probability weighting. After predicting the behavior categories for all windows within this teaching stage along the timeline, the behavior category prediction results for each window are scanned sequentially along the timeline, and adjacent windows that are temporally continuous and have the same behavior category are merged into the same behavior instance. After merging, the start and end timestamps of each behavior instance are jointly determined by the start timestamp of the first window and the end timestamp of the last window in the merged window sequence. For isolated behavior instances with too short a duration after aggregation, if their duration is shorter than the preset minimum behavior duration threshold, the isolated behavior instance is merged into a temporally adjacent behavior instance with a longer duration. After aggregation, the boundaries of each behavior instance are finely adjusted, and fusion feature fragments near the start and end boundaries of the behavior instance are extracted. The start and end time offsets of the behavior instance are predicted by a pre-trained boundary regression network. The boundary regression network is trained by using manually labeled precise start and end frames of the behavior as supervision signals to learn the subtle change patterns of fusion features when transitioning from one behavior to another at the behavior boundary.Based on the offset prediction results, the start and end timestamps of the behavior instances are added with the start and end offsets respectively. After the above processing steps, the second detection layer outputs the behavior recognition results within the teaching stage. Each behavior instance in the result includes a behavior category label, a start and end timestamp accurate to the frame level, and the average confidence score of the behavior instance. Finally, the teaching stage recognition results are merged with the behavior recognition results within the stage to output the teacher's comprehensive behavior analysis results. That is, using the start and end timestamps of each teaching stage in the teaching stage recognition results as indexes, the start and end timestamps of each behavior instance in the behavior recognition results within the stage are traversed. If the start and end timestamps of a behavior instance fall completely within the range of the start and end timestamps of a teaching stage, the category label and stage index of the teaching stage to which the behavior instance belongs are added. If the start and end timestamps of a behavior instance cross the boundary between two teaching stages, the behavior instance is split into two instances at the boundary. For example, the teacher comprehensive behavior analysis results, categorized into two teaching stages, are output as a hierarchical data structure containing three levels of information: The classroom level records basic information about the entire lesson, including total lesson duration, classroom identifier, data collection time, and a stage sequence list, which lists all teaching stage indices in chronological order; the teaching stage level expands to include teaching stages as units, with each stage entry containing: a teaching stage category label, start and end timestamps of the teaching stage, duration of the teaching stage, and a behavior sequence list, which lists all teaching behavior indices within the teaching stage in chronological order; the teaching behavior level expands to include individual teaching behavior instances as the smallest unit, with each teaching behavior instance entry containing: a teaching behavior category label, start and end timestamps of the behavior instance, duration of the behavior instance, identification confidence, and the index of the teaching stage to which the behavior instance belongs.

[0034] In this embodiment, the method for obtaining the behavioral state transition probability matrix can be implemented using the following steps: Collect multimodal classroom data from multiple teachers, and label the boundaries of teaching stages and the specific teaching behavior sequences within each teaching stage in the classroom data. The frequency of transitions between adjacent teaching behavior pairs within the same teaching stage is statistically analyzed to construct a teaching behavior transition frequency matrix; The teaching behavior transition frequency matrix is ​​normalized row by row so that the sum of the elements in each row is 1, thus obtaining the behavior state transition probability matrix.

[0035] In practice, the first step is to collect multimodal classroom data from multiple teachers, and then label the boundaries of teaching stages and the specific teaching behavior sequences within each teaching stage. In other words, the multimodal classroom data from multiple teachers is used as the raw material for constructing the matrix. The selected teachers should be experienced teachers with excellent teaching evaluations, and the collected classroom data should cover different subjects and different lesson types. Classroom data for each lesson is annotated, with two levels of annotation. The first level is the annotation of teaching stage boundaries. Based on the actual progress of the class, the entire lesson is divided into four teaching stages: introduction, lecture, interaction, and summary. The start and end timestamps of each teaching stage are recorded. The division of teaching stages is based on the functional characteristics of classroom teaching. The second level is the annotation of specific teaching behavior sequences within each teaching stage, in seconds. Within the time range of each teaching stage, the category of teaching behavior currently being performed by the teacher is annotated second by second. The annotation system of behavior categories is consistent with the set of behavior categories corresponding to each teaching stage in the second detection layer. After annotation, each lesson generates an annotated data set containing teaching stage boundaries and behavior sequences within each stage. Then, the transition frequency of adjacent teaching behavior pairs within the same teaching stage is statistically analyzed to construct a teaching behavior transition frequency matrix. That is, for each teaching stage, all segments belonging to the corresponding teaching stage in all annotated data are traversed. Within each segment, the behavior category sequence is scanned second by second along the time axis. For two temporally adjacent positions, the transition frequency is obtained. For the behavior categories at the previous and next moments, the count value of the corresponding row and column in the behavior transition frequency matrix for that teaching stage is incremented by one. When the same behavior category continues for several seconds, this is treated as a transition of the behavior category to itself, and the count is incremented at the corresponding position on the diagonal of the frequency matrix. The above statistical process is executed independently for the introduction stage, lecture stage, interaction stage, and summary stage, and a behavior transition frequency matrix corresponding to each stage is established. The number of rows and columns of each frequency matrix is ​​equal to the number of categories in the behavior category set for that stage. The frequency value in the i-th row and j-th column of the matrix represents the cumulative number of times the behavior category shifts from the i-th behavior category to the j-th behavior category in that teaching stage. Finally, the teaching behavior transition frequency matrix is ​​normalized row by row so that the sum of the elements in each row is 1, resulting in the behavior state transition probability matrix. That is, for each row of the frequency matrix, the sum of all frequency values ​​in that row is calculated, and each frequency value in that row is divided by the sum. The result is the probability value of the behavior category shifting from the behavior category corresponding to that row to the behavior category corresponding to that column. After normalization, the sum of the probability values ​​in each row of the matrix is ​​100%. The probability value in the i-th row and j-th column of the matrix represents the statistical probability that the j-th behavior category will occur immediately after the i-th behavior category occurs in the current teaching stage.The probability values ​​at the diagonal positions reflect the probability of each behavior category transitioning to itself, i.e., the likelihood of the same behavior continuing; the probability values ​​at the off-diagonal positions reflect the probability of switching between different behavior categories, thus obtaining the behavior state transition probability matrix.

[0036] It should be noted that by first identifying the macro-level teaching stages and then using the transition probability matrix to detect specific behavioral instances under stage constraints, the system outputs hierarchical comprehensive teacher behavior analysis results that include stage affiliation relationships. This suppresses isolated false detections and fragmented detections, improves the coherence and rationality of behavioral sequences, and establishes a subordinate relationship between behavior and teaching links through stage indexing, providing a structured data foundation for phased differentiated assessment.

[0037] In step S4, based on the results of the comprehensive teacher behavior analysis, a structured teaching behavior evaluation report is generated according to the preset multi-dimensional evaluation indicators.

[0038] In this embodiment, based on the results of the comprehensive teacher behavior analysis, a structured teaching behavior evaluation report is generated according to preset multi-dimensional evaluation indicators. This can be achieved through the following steps: Based on the teaching stage category labels and their start and end timestamps in the teacher comprehensive behavior analysis results, the teacher comprehensive behavior analysis results for the whole lesson are divided into teaching stages to obtain the behavioral data subsets corresponding to each teaching stage. For each teaching stage, according to the corresponding preset assessment focus, a subset of assessment indicators activated in that stage is selected from the preset multi-dimensional assessment indicators, and behavioral instances related to the activated assessment indicator subset are extracted from the behavioral data subset of that stage for indicator calculation to obtain the stage assessment score for each teaching stage. The stage assessment scores of each teaching stage are weighted and combined with the global assessment scores calculated over the entire classroom time window to generate a structured teaching behavior assessment report.

[0039] In practical implementation, based on the teaching stage category labels and their start and end timestamps in the teacher comprehensive behavior analysis results, the teacher comprehensive behavior analysis results for the entire lesson are divided into teaching stages to obtain behavioral data subsets corresponding to each teaching stage. Specifically, each teaching stage recorded in the teaching stage hierarchy is traversed, and the time interval defined by its start and end timestamps is extracted. All teaching behavior instances whose start and end timestamps completely fall within this time interval are selected from the teaching behavior hierarchy. These behavior instances are then associated with the category label and stage index of that teaching stage to form the behavioral data subset corresponding to that teaching stage. Then, for each teaching stage, according to the corresponding preset evaluation focus, the data is extracted from the preset multi-dimensional evaluation indicators. A subset of assessment indicators activated in each teaching stage is selected, and behavioral instances related to these indicators are extracted from the behavioral data subset for that stage. These instances are then used to calculate the assessment score for each teaching stage. Specifically, the activation correspondence between each teaching stage and the assessment indicators is pre-defined, such as: the subset of assessment indicators activated in the introduction stage includes interaction effectiveness and teacher demeanor affinity; the subset of assessment indicators activated in the lecturing stage includes lecturing clarity, blackboard writing and courseware coordination, and teaching pace control; the subset of assessment indicators activated in the interaction stage includes interaction effectiveness and teacher demeanor affinity; and the subset of assessment indicators activated in the summary stage includes lecturing clarity and teaching pace control. For each indicator in the subset of assessment indicators activated in the current teaching stage, behavioral instances related to that indicator are extracted from the behavioral data subset for that stage to calculate the assessment score for each teaching stage. This will be explained in detail in subsequent steps.

[0040] In this embodiment, extracting behavioral instances related to the activated evaluation index subset from the behavioral data subset of this stage and calculating the index to obtain the stage evaluation score for each teaching stage can be achieved through the following steps: Obtain the original calculated value of each indicator in the current teaching stage's activated subset of assessment indicators within that stage; Obtain the preset stage standard value of the corresponding indicator. The preset stage standard value is a reference value obtained by statistical analysis based on the calculation results of the corresponding indicators of multiple teachers in the same teaching stage. Calculate the degree of deviation between the original calculated value and the preset stage standard value within the teaching stage, map the degree of deviation to the preset scoring range, and obtain the stage evaluation score of the corresponding indicator in the current teaching stage.

[0041] In practical implementation, firstly, the original calculated values ​​of each indicator in the subset of assessment indicators activated in the current teaching stage are obtained within that stage. Specifically, the duration sequence of lecturing behaviors is extracted from the behavioral data subset, and the average single duration, standard deviation of duration, and conversion frequency (the number of transitions from lecturing to non-lecturing behaviors divided by the total stage duration) are calculated. These three values—average single duration, standard deviation of duration, and frequency of transition from lecturing to non-lecturing—are used as the original calculated values ​​of the teaching rhythm control indicator within the current teaching stage. Then, the timing of blackboard writing actions is extracted from the behavioral data subset. The time intervals of blackboard writing actions and the time intervals of courseware pointing actions are used to extract the timestamps of courseware page-turning events from the courseware behavior features. The average time interval between each blackboard writing action and the most recent courseware page-turning event is calculated as the blackboard writing follow-up delay. The average time overlap ratio between each courseware pointing action and the corresponding courseware page-turning event is calculated as the pointing synchronization rate. The blackboard writing follow-up delay and pointing synchronization rate are used as the original calculated values ​​of the blackboard writing and courseware matching degree index within the current teaching stage. The end timestamp of the teacher's question is extracted from the speech behavior features, and the start and end times of the student's response are identified from the turn-taking boundary. The average waiting time for each question is calculated, and the proportion of waiting times within a reasonable range is used to obtain the question effectiveness rate. The proportion of student responses whose duration exceeds a minimum threshold is used to obtain the response quality rate. The question effectiveness rate and response quality rate are used as the raw calculated values ​​of the interaction effectiveness index within the current teaching stage. The time interval of lecturing behaviors is extracted from the behavioral data subset. Based on this, the teacher's speech text is extracted from the speech behavior features, and the average sentence length, the proportion of interjections, and the proportion of repeated segments are calculated. The average sentence length, the proportion of interjections, and the proportion of repeated segments are used as the raw calculated values ​​of the lecturing clarity index within the current teaching stage. The coverage area of ​​the podium area movement trajectory and the distribution of dwell time in each area are extracted from the behavioral data subset. The proportion of time the teacher faces the students directly is extracted from the video behavioral features. The average speech rate and the range of fundamental frequency variation of the teacher are extracted from the speech behavior features. The coverage area of ​​the movement trajectory, the distribution of dwell time in each area, the proportion of time facing the students directly, the average speech rate, and the range of fundamental frequency variation are used as the raw calculated values ​​of the teaching demeanor affinity index within the current teaching stage.

[0042] In addition, in specific implementation, the preset stage standard value of the corresponding indicator is obtained. That is, multiple teachers with excellent teaching evaluation levels are selected as a sample group, and their classroom multimodal data are collected. According to the calculation method of each indicator mentioned above, the original calculated value of each indicator for each sample teacher in each teaching stage is calculated. The average of the original calculated values ​​of the same indicator for the same sample group in the same teaching stage is taken as the preset stage standard value of the indicator in that teaching stage. Finally, the degree of deviation between the original calculated value and the preset stage standard value in the teaching stage is calculated, and the degree of deviation is mapped to the preset scoring interval to obtain the stage evaluation score of the corresponding indicator in the current teaching stage. For single-value indicators, the absolute value of the difference between the original calculated value and the preset stage standard value in the corresponding stage is calculated and divided by the preset stage standard value to obtain the deviation rate. For multi-value indicators, the deviation rate of each sub-item is calculated separately, and then the weighted average of the deviation rates of each sub-item is taken as the comprehensive deviation rate of the indicator. The weight of each sub-item is preset according to the importance of the sub-item in the actual teaching evaluation. The deviation rate is mapped to a preset scoring range, from zero to one hundred points. A deviation rate of zero corresponds to a full score of one hundred, indicating that the teacher's performance is completely consistent with the standard for an excellent teacher. The higher the deviation rate, the lower the corresponding score. The mapping function can be a piecewise linear function. The score obtained after mapping is the stage assessment score of that indicator in the current teaching stage. By iterating through all indicators in the subset of assessment indicators activated in the current teaching stage, the stage assessment score of each indicator in the current teaching stage can be obtained. Each teaching stage obtains its own stage assessment score by following the above steps.

[0043] In this embodiment, the weighted summation of the stage assessment scores for each teaching stage with the global assessment scores calculated over the entire class time window specifically includes: The proportion of the duration of each teaching stage to the total class time is used as the duration weighting coefficient for each stage. Obtain the pre-configured importance weight coefficients for each teaching stage, wherein the importance weight coefficients are set according to the degree of emphasis on each teaching stage in the teaching evaluation criteria; For each evaluation indicator, the global evaluation score calculated over the entire class time window and the stage evaluation scores for each teaching stage are weighted and summed in three layers according to the global preset weight, the duration weight coefficient, and the importance weight coefficient to obtain the weighted comprehensive score of the indicator. The weighted scores of each indicator are summed to generate the total evaluation score in the teaching behavior evaluation report.

[0044] In practical implementation, firstly, the proportion of the duration of each teaching stage to the total class time is used as the duration weight coefficient for each stage. Specifically, the start and end timestamps of each teaching stage are extracted from the teaching stage hierarchy of the teacher's comprehensive behavior analysis results. For each teaching stage, the start timestamp is subtracted from the end timestamp to obtain the duration of that teaching stage. The durations of all teaching stages are added together to obtain the total class time. For each teaching stage, its duration is divided by the total class time, and the quotient is the duration weight of that teaching stage. The first factor is the weighting coefficient, which reflects the proportion of each teaching stage's contribution to the entire lesson in terms of time. The second factor is the acquisition of pre-configured weighting coefficients for each teaching stage. That is, based on the specific objectives and application scenarios of the teaching evaluation, teaching evaluation experts assign weights to the relative importance of each teaching stage in the teaching evaluation. For example, if the teaching evaluation focuses on assessing the teacher's knowledge delivery ability, a higher weighting coefficient can be set for the delivery stage; if the teaching evaluation focuses on assessing the teacher's teacher-student interaction ability, a higher weighting coefficient can be set for the interaction stage. The sum of the importance weight coefficients for each teaching stage is 100%. Then, for each evaluation indicator, the global evaluation score calculated over the entire class time window and the stage evaluation scores for each teaching stage are weighted and summed in three layers according to the global preset weight, the duration weight coefficient, and the importance weight coefficient to obtain the weighted comprehensive score of that indicator. That is, the first layer of weight is the global preset weight. For each evaluation indicator, the weight of the corresponding global evaluation score in the comprehensive score and the total weight of the stage evaluation score in the comprehensive score are set. The sum of the global evaluation score weight and the total weight of the stage evaluation score is 100%. The allocation between the two is... The proportions can be preset based on expert experience. The second layer is the importance weighting coefficient. For each teaching stage, the stage assessment score is multiplied by the importance weighting coefficient for that stage, and then multiplied by the total weight of the stage assessment scores to obtain the importance-weighted contribution value for that stage. The importance-weighted contribution values ​​of all teaching stages are summed to obtain the comprehensive value of the stage assessment score after importance weighting. The third layer is the duration weighting coefficient. For each teaching stage, the stage assessment score is multiplied by the duration weighting coefficient for that stage, then multiplied by the importance weighting coefficient for that stage, and then multiplied by the total weight of the stage assessment scores to obtain the comprehensive contribution value for that stage. The comprehensive contribution values ​​of all teaching stages are summed to obtain the final comprehensive value of the stage assessment score. The global assessment score is multiplied by the global assessment score weight, and then added to the final comprehensive value of the stage assessment score to obtain the weighted comprehensive score of that indicator. The weighted comprehensive scores of all assessment indicators are summarized to generate the total assessment score in the teaching behavior assessment report. The summation method is to take the arithmetic mean of the weighted comprehensive scores of all assessment indicators, and the result is used as the total assessment score for the entire class.

[0045] Finally, a structured teaching behavior evaluation report can be generated. This report includes: the total evaluation score for the entire class, which is a summary of the weighted composite scores of all evaluation indicators, presented prominently at the top of the report as a percentage, directly reflecting the teacher's overall teaching performance level for the lesson. The total evaluation score is calculated by taking the arithmetic mean of the weighted composite scores of all evaluation indicators; the global evaluation score for each multi-dimensional evaluation indicator, which is an overall score for each indicator across the entire class time, regardless of the teaching stage. The report lists the name of each evaluation indicator and its corresponding global evaluation score in a list format. The global evaluation score for each indicator is generated by the indicator calculation process within the entire class time window and is completed in parallel with the calculation of the stage evaluation scores; and the stage evaluation scores for each teaching stage, displayed by teaching stage, listing the comprehensive stage evaluation scores for the introduction stage, lecture stage, interaction stage, and summary stage. The overall stage assessment score is derived from the sum of the sub-scores of each assessment indicator activated within that stage. The sub-scores for each indicator are the specific scores for each assessment indicator activated within each teaching stage. The report is grouped by teaching stage, with each stage listing the names of the activated assessment indicators and their corresponding stage assessment scores below each stage. The sub-scores for each indicator are calculated using a deviation rate mapping, the specific calculation process of which is already completed in the stage assessment score calculation steps.

[0046] like Figure 2 The diagram illustrates a hierarchical behavior detection structure. From top to bottom, it shows the processing flow of the fused feature sequence input, sequentially passing through the first and second detection layers. The first detection layer performs coarse-grained identification of teaching stages, outputting category labels and start and end timestamps for four teaching stages: introduction, lecture, interaction, and summary. The second detection layer, based on the stage categories output by the first layer, loads the corresponding behavior category set and behavior state transition probability matrix for that stage, and performs fine-grained identification of teaching behaviors within the time interval of each teaching stage, outputting behavior instances within each stage. The combined results of the two layers output a comprehensive teacher behavior analysis result, including teaching stage category labels, teaching behavior category labels, start and end timestamps for each label, and the index of the stage to which the behavior belongs. The single arrow in the diagram indicates the data flow, reflecting the hierarchical processing logic of first identifying macro-level stages and then detecting micro-level behaviors.

[0047] like Figure 3The diagram illustrates the synchronous acquisition of multimodal data. On the left, three rows of text are labeled to represent the classroom video stream, the teaching material image stream, and the classroom audio stream, signifying the three raw data inputs. Each of the three data streams passes through a separate data acquisition module before converging at the acquisition workstation. The workstation internally performs timestamp marking based on a common clock source and frame-level alignment of the three data streams, outputting a synchronized multimodal data sequence. A single arrow in the diagram indicates the data flow direction. This diagram schematically illustrates the overall process of multimodal data acquisition and synchronization alignment in this solution.

[0048] Therefore, this application firstly, through three-modal frame-level synchronous acquisition, provides a precisely aligned multi-source data foundation for subsequent hierarchical behavior detection and phased differentiated assessment, ensuring the data quality of intelligent analysis of teachers' teaching behavior from the source; secondly, by extracting teaching-specific features in the teaching scenario and generating a fused feature sequence through cross-modal attention fusion, it provides a semantically rich unified feature representation for subsequent hierarchical behavior detection and phased differentiated assessment; and thirdly, by first identifying the macro-level teaching stage and then detecting specific behavior instances under stage constraints using the transition probability matrix, it outputs a hierarchical comprehensive teacher behavior score containing stage attribution relationships. The analysis results suppressed isolated false positives and fragmented detections, improved the coherence and rationality of behavioral sequences, and established a subordinate relationship between behaviors and teaching links through stage indexing, providing a structured data foundation for phased differentiated assessment. Finally, by activating different assessment indicators in stages and comparing them with differentiated stage standard values, and then through three-level weighted synthesis, a structured teaching behavior assessment report containing global assessment scores, stage scores, and item scores is generated. This ensures that the assessment dimensions are consistent with the actual functional goals of each teaching stage, supports one-click backtracking from assessment scores to key behavioral segments, and provides teachers with precise basis for teaching improvement.

[0049] In summary, the technical solution adopted in this application can achieve intelligent analysis of teachers' teaching behavior through hierarchical behavior detection and phased differentiated evaluation, thereby improving the accuracy of behavior recognition and the rationality of evaluation results.

[0050] Example 2: This application provides an intelligent analysis and evaluation system for teacher teaching behavior based on multimodal data, referencing... Figure 4 As shown in the figure, this is a modular structure diagram of a multimodal data-based intelligent analysis and evaluation system for teacher teaching behavior according to this embodiment of the present application. The intelligent analysis and evaluation system includes: The data acquisition module 100 is used to simultaneously acquire classroom audio streams, classroom video streams, and teaching courseware image streams in the teaching scenario, forming a multimodal data sequence; The feature fusion module 200 is used to extract video behavior features, speech behavior features and courseware behavior features from the multimodal data sequence, respectively, and perform cross-modal feature fusion to generate a fused feature sequence; The behavior analysis module 300 is used to perform time-series behavior detection based on the fused feature sequence and output the comprehensive behavior analysis results of the teacher. The behavior assessment module 400 is used to generate a structured teaching behavior assessment report based on the results of the comprehensive teacher behavior analysis and according to preset multi-dimensional assessment indicators.

[0051] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0053] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A method for intelligent analysis and evaluation of teachers' teaching behavior based on multimodal data, characterized in that: The intelligent analysis and evaluation method includes the following steps: The system simultaneously collects classroom audio streams, classroom video streams, and teaching courseware image streams from the teaching scenario, forming a multimodal data sequence. Video behavior features, speech behavior features, and courseware behavior features are extracted from the multimodal data sequence, and cross-modal feature fusion is performed to generate a fused feature sequence. Based on the fused feature sequence, temporal behavior detection is performed, and the comprehensive teacher behavior analysis results are output. Based on the results of the comprehensive teacher behavior analysis, a structured teaching behavior evaluation report is generated according to the preset multi-dimensional evaluation indicators.

2. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 1, characterized in that, The simultaneous acquisition of classroom audio streams, classroom video streams, and teaching material image streams in a teaching scenario constitutes a multimodal data sequence, specifically including: Classroom video streams are captured using video capture devices deployed in classrooms, classroom audio streams are captured using audio capture devices, and teaching courseware image streams are obtained by capturing the screen feed of the teaching computer. During the acquisition process, each frame of the classroom video stream, each frame of the classroom audio stream, and each frame of the teaching courseware image stream are timestamped based on the same clock source. After the data acquisition is completed, using the timeline of the classroom video stream as the reference timeline, frames in the classroom audio stream and the teaching courseware image stream with timestamp deviations within a preset threshold are established with frames in the classroom video stream corresponding to the timestamps, forming a frame-level aligned multimodal data sequence.

3. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 1, characterized in that, The video behavior features, speech behavior features, and courseware behavior features are extracted from the multimodal data sequence respectively, and cross-modal feature fusion is performed to generate a fused feature sequence, specifically including: Teacher target detection and tracking are performed on the classroom video stream, the teacher's human skeleton sequence is extracted, the teacher's body movement categories are identified, and a video behavior feature vector is generated. Speech recognition is performed on the classroom audio stream to extract the text features of the teacher's speech. Based on the audio frame energy value and the detection results of silent segments, the time boundary of the teacher-student turn-taking is marked to generate a speech behavior feature vector. Inter-frame difference analysis is performed on the image stream of the teaching courseware to extract the timestamps of page turning events and element change events within the courseware, and to generate courseware behavior feature vectors. After aligning the video behavior feature vector, the speech behavior feature vector, and the courseware behavior feature vector by timestamp, they are input into a cross-modal feature fusion network, which outputs a fused feature sequence.

4. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 1, characterized in that, Based on the fused feature sequence, temporal behavior detection is performed, and the output of the comprehensive teacher behavior analysis results specifically includes: The fused feature sequence is input into a hierarchical behavior detection structure, which includes a first detection layer and a second detection layer. In the first detection layer, the fused feature sequence is scanned with a first time resolution to identify the teaching stage category of the teacher and output the start and end timestamps of each teaching stage. The teaching stage categories include the introduction stage, the lecture stage, the interaction stage, and the summary stage. In the second detection layer, based on the teaching stage category output by the first detection layer, a set of behavior categories and a behavior state transition probability matrix corresponding to the teaching stage category are loaded. Within the time interval defined by the start and end timestamps of each teaching stage, the fused feature sequence is scanned with a second time resolution higher than the first time resolution to identify specific teaching behavior instances within the stage and output the behavior recognition results within the stage. The behavior recognition results within the stage include behavior category labels and their start and end timestamps. The teaching stage identification results are combined with the behavior identification results within the stage to output the teacher comprehensive behavior analysis results. The teacher comprehensive behavior analysis results include teaching stage category labels, teaching behavior category labels, start and end timestamps corresponding to each label, and the teaching stage index to which each teaching behavior belongs.

5. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 4, characterized in that, The specific methods for obtaining the behavioral state transition probability matrix include: Collect multimodal classroom data from multiple teachers, and label the boundaries of teaching stages and the specific teaching behavior sequences within each teaching stage in the classroom data. The frequency of transitions between adjacent teaching behavior pairs within the same teaching stage is statistically analyzed to construct a teaching behavior transition frequency matrix; The teaching behavior transition frequency matrix is ​​normalized row by row so that the sum of the elements in each row is 1, thus obtaining the behavior state transition probability matrix.

6. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 1, characterized in that, Based on the comprehensive analysis results of teachers' behaviors, and according to the preset multi-dimensional evaluation indicators, a structured teaching behavior evaluation report is generated, specifically including: Based on the teaching stage category labels and their start and end timestamps in the teacher comprehensive behavior analysis results, the teacher comprehensive behavior analysis results for the whole lesson are divided into teaching stages to obtain the behavioral data subsets corresponding to each teaching stage. For each teaching stage, according to the corresponding preset assessment focus, a subset of assessment indicators activated in that stage is selected from the preset multi-dimensional assessment indicators, and behavioral instances related to the activated assessment indicator subset are extracted from the behavioral data subset of that stage for indicator calculation to obtain the stage assessment score for each teaching stage. The stage assessment scores of each teaching stage are weighted and combined with the global assessment scores calculated over the entire classroom time window to generate a structured teaching behavior assessment report.

7. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 6, characterized in that, Extracting behavioral instances from the subset of behavioral data for this stage that are related to the activated subset of evaluation indicators, and calculating the indicators to obtain the stage evaluation scores for each teaching stage, specifically including: Obtain the original calculated value of each indicator within the current teaching stage from the subset of assessment indicators activated in the current teaching stage; Obtain the preset stage standard value of the corresponding indicator. The preset stage standard value is a reference value obtained by statistical analysis based on the calculation results of the corresponding indicators of multiple teachers in the same teaching stage. Calculate the degree of deviation between the original calculated value and the preset stage standard value within the teaching stage, map the degree of deviation to the preset scoring range, and obtain the stage evaluation score of the corresponding indicator in the current teaching stage.

8. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 6, characterized in that, The weighted summation of the stage assessment scores for each teaching phase with the overall assessment score calculated over the entire class time window includes: The proportion of the duration of each teaching stage to the total class time is used as the duration weighting coefficient for each stage. Obtain the pre-configured importance weight coefficients for each teaching stage, wherein the importance weight coefficients are set according to the degree of emphasis on each teaching stage in the teaching evaluation criteria; For each evaluation indicator, the global evaluation score calculated over the entire class time window and the stage evaluation scores for each teaching stage are weighted and summed in three layers according to the global preset weight, the duration weight coefficient, and the importance weight coefficient to obtain the weighted comprehensive score of the indicator. The weighted scores of each indicator are summed to generate the total evaluation score in the teaching behavior evaluation report.

9. The intelligent analysis and evaluation method for teacher teaching behavior based on multimodal data as described in claim 1, characterized in that, The teaching behavior evaluation report includes: the total evaluation score for the whole class, the global evaluation score for each multi-dimensional evaluation indicator, the stage evaluation score for each teaching stage, and the sub-item score for each indicator.

10. A teacher teaching behavior intelligent analysis and evaluation system based on multimodal data, used to execute the teacher teaching behavior intelligent analysis and evaluation method based on multimodal data as described in any one of claims 1 to 9, characterized in that, The intelligent analysis and evaluation system includes: The data acquisition module is used to simultaneously collect classroom audio streams, classroom video streams, and teaching courseware image streams in the teaching scenario, forming a multimodal data sequence; The feature fusion module is used to extract video behavior features, speech behavior features, and courseware behavior features from the multimodal data sequence, respectively, and perform cross-modal feature fusion to generate a fused feature sequence. The behavior analysis module is used to perform time-series behavior detection based on the fused feature sequence and output the comprehensive behavior analysis results of teachers. The behavior assessment module is used to generate a structured teaching behavior assessment report based on the comprehensive analysis results of the teachers' behavior and according to preset multi-dimensional assessment indicators.

Images