Intelligent teaching evaluation and diagnosis system and method based on multi-modal audio and video analysis

Abstract

The present application relates to the technical field of teaching evaluation diagnosis management, in particular to an intelligent teaching evaluation and diagnosis system and method based on multi-modal audio and video analysis, wherein the audio and video dynamic segmentation module combines the teaching state feature data in the historical data to perform classroom teaching event identification on the teaching state feature set extracted at the current time; and combines the MTES technology to perform time sequence model-based classroom teaching event evolution process analysis, and based on the obtained classroom teaching event evolution process analysis result, the classroom audio and video are dynamically segmented. The comprehensive quantification mode of the index set bound to each classroom teaching event based on the MTES technology can effectively solve the modal fragmentation phenomenon between the multi-modal collected data in the teaching evaluation and diagnosis process; the dynamic segmentation points of the classroom audio and video are screened, locked and calibrated from multiple angles, so that the accuracy of the classroom audio and video segmentation segments is ensured.

Description

Technical Field

[0001] This invention relates to the field of teaching evaluation, diagnosis and management technology, specifically to an intelligent teaching evaluation and diagnosis system and method based on multimodal audio and video analysis. Background Technology

[0002] Teaching evaluation is a core component for improving education quality, promoting teacher professional development, and ensuring student learning outcomes. Traditional teaching evaluation mainly relies on experts or peers to conduct evaluations through on-site classroom observations and manual completion of evaluation questionnaires.

[0003] With the development of computer technology and smart education, classroom teaching evaluation technology has shifted from manual observation to automated analysis. Existing technologies, such as analyzing student focus using a single camera or assessing teacher instruction quality based on speech recognition, have some effectiveness but also suffer from several bottlenecks. For example, existing technologies often process video, audio, and screen content independently, failing to capture cross-modal correlations such as students looking down at courseware while the teacher asks a question, resulting in significant fragmentation between data collected from different modalities. Furthermore, current technologies often use fixed time slices (e.g., 5 minutes) for teaching evaluation and diagnosis, which can lead to incorrect segmentation of transitional stages like "questioning → discussion," significantly interfering with subsequent evaluation and diagnostic results. Therefore, to address these technological bottlenecks, a new intelligent teaching evaluation and diagnostic system and method based on multimodal audio and video analysis is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent teaching evaluation and diagnosis system and method based on multimodal audio and video analysis to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: an intelligent teaching evaluation and diagnosis method based on multimodal audio and video analysis, the method comprising:

[0006] S1. By deploying camera arrays, directional microphone groups and screen capture devices in the classroom, synchronously collect time-stamped video streams of teacher and student behavior, audio streams and teaching screen content streams to construct multimodal audio and video data;

[0007] S2. Perform content parsing on the constructed multimodal audio and video data, and extract the corresponding teaching status feature set based on the content parsing results;

[0008] S3. Combining teaching status feature data from historical data, identify classroom teaching events from the teaching status feature set extracted at the current time; and combine MTES technology to analyze the evolution process of classroom teaching events based on time series models. Based on the analysis results of the classroom teaching event evolution process, dynamically segment the classroom audio and video.

[0009] S4. Based on the dynamic segmentation results of classroom audio and video, generate a diagnostic assessment report corresponding to each segmented segment of classroom teaching audio and video.

[0010] This invention extracts classroom camera, microphone, and screen recording information to construct multimodal audio and video data (including teacher and student behavior, expressions, voice, teaching content, etc.), performs spatiotemporal alignment and feature extraction, and integrates the features of each modality to identify teaching events. Furthermore, it combines MTES technology to analyze the evolution process of classroom teaching events based on a time-series model (quantifying teaching events and the evolution trend of quantified indicators), dynamically segments classroom audio and video, and generates diagnostic evaluation reports (each segment is bound to a diagnostic evaluation report, each segment consists of one or more consecutive classroom teaching events, and the diagnostic evaluation report is a comprehensive analysis result of each classroom teaching event corresponding to each segment).

[0011] Furthermore, the camera array includes a panoramic camera covering the teacher's overall dynamics, a teacher tracking camera capturing the teacher's body language and blackboard writing, and a student expression camera collecting student facial response data.

[0012] This invention effectively solves the problem of spatiotemporal mismatch in data acquisition across modalities by using a multimodal data synchronous acquisition method. Furthermore, this invention achieves cross-modal correlation analysis between behavior, voice, and content through a camera array, directional microphone group, and screen capture device. The hierarchical deployment of the camera array, which achieves combined coverage of panoramic and close-up views, is intended to improve the detection accuracy of teacher movement trajectory and the recognition accuracy of student facial expressions.

[0013] Furthermore, the teaching status feature set in S2 includes teacher and student behavior features based on video stream parsing, voice interaction features based on audio stream parsing, and teaching content structure features based on screen content parsing.

[0014] The teacher-student behavioral characteristics are composed of teacher movement trajectory density and student head-up rate. The teacher movement trajectory density at time t is denoted as Dt.

[0015]

[0016] Where Nt represents the number of times the teacher's position is updated within a preset unit time interval before time t in the acquired video stream; T represents the preset sampling time window length; and A represents the classroom area.

[0017] The student head-up rate at time t is equal to the ratio of the number of students in the captured video stream who are looking at the teacher / PPT playback screen at time t to the total number of students.

[0018] This invention uses quantitative formulas corresponding to various parameters in the process of acquiring teacher and student behavioral characteristics to effectively transform subjective teaching behaviors into objective indicator characteristics, which facilitates comprehensive analysis of various teaching state characteristics in subsequent steps.

[0019] In the process of acquiring speech interaction features based on audio stream parsing, the frame length of the acquired audio stream is obtained, where the audio frame length represents the number of audio sampling points contained in the corresponding audio frame. The audio frame corresponding to any acquisition time and the midpoint of each sampling point within the corresponding audio frame are acquired. The short-time energy of the speech corresponding to the corresponding audio frame is obtained, serving as the speech interaction feature of the acquisition time corresponding to the midpoint of each sampling point within the corresponding audio frame. The short-time energy of the speech corresponding to the nth frame is denoted as En.

[0020]

[0021] Where K represents the number of preset audio sampling points contained in the audio frame; This represents the square of the amplitude (instantaneous power) corresponding to the k-th sampling point within the n-th frame.

[0022] In the process of obtaining the structured features of teaching content based on screen content parsing, the screen recording text is recognized using OCR technology. Preset keyword types corresponding to the teaching content at collection time t are extracted, along with the ratio of the frequency of each preset keyword to the total frequency of extracted preset keywords. The teaching themes corresponding to the preset keyword types extracted from the database's preset forms are queried. Based on the ratio of the frequency of each preset keyword to the total frequency of extracted preset keywords, a teaching theme bias coefficient at collection time t is calculated. This teaching theme bias coefficient at collection time t is equal to the ratio of the frequency of each preset keyword to the total frequency of extracted preset keywords multiplied by the sum of the weight factors of the corresponding preset keyword types based on the corresponding teaching themes in the database's preset forms. The obtained teaching theme bias coefficient and the corresponding teaching theme are used as the first and second parameters in the structured features of the teaching content, respectively.

[0023] Furthermore, in the process of identifying classroom teaching events using the teaching state feature set extracted at the current time in step S3, a preset teaching event sequence is obtained, and the event probability of each element in the preset teaching event sequence based on the teaching state feature data at collection time t is calculated. The element with the highest event probability based on the teaching state feature data at collection time t in the preset teaching event sequence is taken as the classroom teaching event identification result at collection time t. The calculation formulas involved are as follows:

[0024]

[0025]

[0026] Where FGt represents the teaching state characteristic data collected at time t; P(E|FGt) represents the event probability of any element E in the preset teaching event sequence based on the teaching state characteristic data FGt collected at time t; g m Both gh and gh represent state feature functions, which are functions with base e and exponents calculated by subtracting the square of the difference between the average of all values ​​of the same teaching state feature type corresponding to the first parameter and the teaching state feature data corresponding to the second parameter in historical data. H represents a transition feature function. If the event state data pair formed by the first and second parameters in the transition feature function belongs to a preset effective transition set of event states, then the function value of the corresponding transition feature function is determined to be 1; otherwise, the function value of the corresponding transition feature function is determined to be 0. Each element in the effective transition set of event states corresponds to an event state data pair. The summary set of the first parameters in teacher movement trajectory density, student head-up rate, voice interaction features, and teaching content structured features in FGt is obtained and denoted as the first analysis set. m E represents the m-th parameter value in the first analysis set. t-1 The most recently identified teaching event is represented before the acquisition time t; W(FGt, m, E) represents the event adaptation evaluation coefficient of any element E in the preset teaching event sequence based on the teaching state feature data FGt at acquisition time t.

[0027] Z(FGt) represents the sum of each W(FGt, m, E) corresponding to each element in the preset teaching event sequence; This represents the preset state feature weight corresponding to the m-th parameter in the first analysis set; The preset state feature weights corresponding to the teaching state evaluation coefficients are represented by μ; μ represents the transition feature weights, where the value of μ is equal to the ratio of the frequency of the event state data pair formed by the first and second parameters in the corresponding transition feature function appearing in the historical data to the sum of the frequencies of all event state data pairs in the historical data that have the same first parameter and the first parameter of the corresponding transition feature function; max{} represents the maximum value function.

[0028]

[0029] Where Ft represents the teaching status evaluation coefficient at acquisition time t; Lt represents the student head-up rate at acquisition time t; E(t) represents the short-time energy of the corresponding audio frame at acquisition time t; Xt represents the first parameter in the structured features of the teaching content at acquisition time t; r1, r2 and r3 are the preset first evaluation weight, second evaluation weight and third evaluation weight, respectively, and r1+r2+r3=1.

[0030] Furthermore, in the process of analyzing the evolution of classroom teaching events based on a time-series model using MTES technology, an array consisting of the identification results of each classroom teaching event is obtained sequentially according to time, denoted as the classroom teaching event evolution analysis array. In this array, before a new classroom teaching event is identified, the original classroom teaching identification results remain unchanged, and each time point corresponds to a unique classroom teaching event identification result. Based on MTES technology, the indicator set bound to each classroom teaching event in the classroom teaching time evolution analysis array is calculated. The indicator set includes interaction density and cognitive load index. The calculation method for interaction density is as follows:

[0031]

[0032] Where QN represents the number of valid questions asked by the teacher within the time period corresponding to the corresponding classroom teaching event identification result; TS represents the interval length of the time period corresponding to the corresponding classroom teaching event identification result; NSUM represents the sum of the number of students who participated in each teacher questioning interaction within the time period corresponding to the corresponding classroom teaching event identification result; and NA represents the total number of students in the classroom.

[0033] The cognitive load index is calculated as follows:

[0034]

[0035] Where CH represents the cognitive load index corresponding to the identification results of the corresponding classroom teaching event; YP tb tb represents the average facial confusion level of each student at any time point tb within the time period corresponding to the corresponding classroom teaching event recognition result; the facial confusion level of a student is equal to the quotient of the number of wrinkles between the eyebrows obtained through image recognition divided by the area of ​​the area between the eyebrows; tb1 represents the minimum time point within the time period corresponding to the corresponding classroom teaching event recognition result; VS tb The VS represents the speech rate change coefficient at any time point tb within the time period corresponding to the classroom teaching event recognition result. tb The value is equal to the difference between the most recent monitored speech rate of the audio stream after time point tb in the historical data and the most recent monitored speech rate of the audio stream before time point tb in the historical data, divided by the quotient of the interval between two adjacent speech rate monitoring.

[0036] This invention uses MTES technology to calculate the interaction density and cognitive load index of each classroom teaching event in the classroom teaching time evolution analysis array. Combined with the constructed multimodal audio and video data, it can effectively solve the modal fragmentation phenomenon between the data collected from different modalities in the teaching evaluation and diagnosis process.

[0037] Furthermore, based on the analysis results of the evolution of classroom teaching events, during the dynamic segmentation of classroom audio and video, when the absolute value of the difference between the cognitive load indices corresponding to two adjacent classroom teaching events is greater than or equal to a preset value, and the time interval between the time boundary point between two adjacent classroom teaching events and the previous time segmentation point is greater than or equal to a preset duration, then the time boundary point between the corresponding two adjacent classroom teaching events is determined as a dynamic segmentation reserve point for the corresponding classroom audio and video; otherwise, it is determined that there is no dynamic segmentation point for the corresponding classroom audio and video between the corresponding two adjacent classroom teaching events.

[0038] Based on the teaching themes corresponding to the second parameter in the structured features of teaching content at different times, each dynamic segmentation reserve point is calibrated to obtain classroom audio-visual segments between two adjacent dynamic segmentation reserve points, which are recorded as segments to be analyzed. The minimum absolute value of the difference in cognitive load index between each classroom teaching event within the previous preset duration in the segment to be analyzed and the previous classroom teaching event of the first dynamic segmentation storage point in the two adjacent dynamic segmentation reserve points is extracted and recorded as the cognitive load index deviation reference value. If the teaching themes of the two classroom teaching events corresponding to the cognitive load index deviation reference value, as well as the teaching themes of the two classroom teaching events corresponding to the first dynamic segmentation storage point in the two adjacent dynamic segmentation reserve points, are the same, then the first dynamic segmentation storage point in the two adjacent dynamic segmentation reserve points is deleted. Each calibrated dynamic segmentation reserve point is used as a dynamic segmentation point of the classroom audio-visual video to obtain each segment of the classroom audio-visual video.

[0039] The diagnostic assessment report for each segment of classroom teaching audio and video includes each segment and its corresponding score.

[0040] The calculation formulas involved in scoring the corresponding classroom audio and video segment are as follows:

[0041]

[0042] Wherein, GS represents the score of the corresponding classroom audio-visual segment; DCP represents the average interaction density of each classroom teaching event within the corresponding classroom audio-visual segment; CHP represents the average cognitive load index of each classroom teaching event within the corresponding classroom audio-visual segment; and φ1 represents the preset scoring factor.

[0043] This invention comprehensively filters dynamic segmentation reserve points for classroom audio and video based on the cognitive load index and the time interval between the time boundary point of two adjacent classroom teaching events and the previous time segmentation point; and based on the teaching theme corresponding to the second parameter in the structured features of teaching content, it realizes secondary calibration of each dynamic segmentation reserve point of the acquired classroom audio and video (taking into account the synergy of teaching themes), effectively eliminating useless dynamic segmentation reserve points at the teaching theme level, ensuring the integrity of classroom audio and video segmentation segments (reducing the frequency of the same teaching theme being divided into different segmentation segments and corresponding diagnostic assessment reports showing large differences).

[0044] An intelligent teaching evaluation and diagnosis system based on multimodal audio and video analysis, the system comprising:

[0045] A multimodal audio and video data construction module, which constructs multimodal audio and video data by simultaneously collecting time-stamp-aligned video streams of teacher and student behavior, audio streams of speech, and content streams of teaching screens through a camera array, directional microphone group, and screen capture device deployed in the classroom;

[0046] The teaching status feature extraction module performs content parsing on the constructed multimodal audio and video data and extracts the corresponding teaching status feature set based on the content parsing results.

[0047] The audio and video dynamic segmentation module combines teaching status feature data from historical data to identify classroom teaching events based on the teaching status feature set extracted at the current time; and combines MTES technology to analyze the evolution process of classroom teaching events based on a time-series model. Based on the analysis results of the classroom teaching event evolution process, the module dynamically segments the classroom audio and video.

[0048] The teaching evaluation and diagnosis management module generates a diagnostic evaluation report for each segment of classroom teaching audio and video based on the dynamic segmentation results of the classroom audio and video.

[0049] Furthermore, the audio and video dynamic segmentation module includes a teaching event recognition unit, an event evolution process analysis unit, and a segmentation point dynamic management unit;

[0050] The teaching event identification unit combines teaching status feature data from historical data to identify classroom teaching events based on the teaching status feature set extracted at the current time.

[0051] The event evolution process analysis unit combines MTES technology to analyze the evolution process of classroom teaching events based on a time-series model;

[0052] The segmentation point dynamic management unit dynamically segments classroom audio and video based on the analysis results of the evolution process of classroom teaching events.

[0053] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0054] (1) The present invention can effectively solve the problem of spatiotemporal mismatch of corresponding data acquisition when cross-modal data is acquired through multimodal data synchronous acquisition;

[0055] (2) The present invention is based on the comprehensive quantitative method of the indicator set bound to each classroom teaching event in the classroom teaching event evolution analysis array using MTES technology. Combined with the constructed multimodal audio and video data, it can effectively solve the modal fragmentation phenomenon between the modal data collected in the teaching evaluation and diagnosis process.

[0056] (3) This invention filters and locks the dynamic segmentation points of classroom audio and video from multiple angles, and performs secondary calibration of the filtered and locked dynamic segmentation points through teaching themes to ensure the accuracy of classroom audio and video segmentation. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a schematic diagram of the intelligent teaching evaluation and diagnosis system based on multimodal audio and video analysis of the present invention;

[0059] Figure 2 This is a flowchart illustrating the intelligent teaching evaluation and diagnosis method based on multimodal audio and video analysis of this invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Please see Figures 1-2 The present invention provides a technical solution: such as Figure 1 As shown, this embodiment provides an intelligent teaching evaluation and diagnosis system based on multimodal audio and video analysis. The system includes:

[0062] A multimodal audio and video data construction module, which constructs multimodal audio and video data by simultaneously collecting time-stamp-aligned video streams of teacher and student behavior, audio streams of speech, and content streams of teaching screens through a camera array, directional microphone group, and screen capture device deployed in the classroom;

[0063] The teaching status feature extraction module performs content parsing on the constructed multimodal audio and video data and extracts the corresponding teaching status feature set based on the content parsing results.

[0064] The audio and video dynamic segmentation module includes a teaching event recognition unit, an event evolution process analysis unit, and a segmentation point dynamic management unit.

[0065] The teaching event identification unit combines teaching status feature data from historical data to identify classroom teaching events based on the teaching status feature set extracted at the current time.

[0066] The event evolution process analysis unit combines MTES technology to analyze the evolution process of classroom teaching events based on a time-series model;

[0067] The segmentation point dynamic management unit dynamically segments classroom audio and video based on the analysis results of the evolution process of classroom teaching events.

[0068] The teaching evaluation and diagnosis management module generates a diagnostic evaluation report for each segment of classroom teaching audio and video based on the dynamic segmentation results of the classroom audio and video.

[0069] like Figure 2 As shown in the example, this paper provides an intelligent teaching evaluation and diagnosis method based on multimodal audio and video analysis. The method includes:

[0070] S1. By deploying camera arrays, directional microphone groups and screen capture devices in the classroom, synchronously collect time-stamped video streams of teacher and student behavior, audio streams and teaching screen content streams to construct multimodal audio and video data;

[0071] The camera array includes a panoramic camera covering the teacher's overall movements, a teacher tracking camera capturing the teacher's body language and blackboard writing, and a student expression camera collecting student facial response data.

[0072] This example uses a physics classroom in the second year of high school. The classroom is 60 square meters and there are 45 students. Two panoramic cameras, model DS-2CD3T86WD, are deployed diagonally. The teacher tracking camera is a PTZ camera (30x optical zoom). Six Logitech C930e cameras are used to capture student expressions. Each camera is placed in each row of student seats in the classroom.

[0073] The directional microphone array uses Shure MXA910 (ceiling array, 8 nodes);

[0074] The screen capture device is a computer with screen recording software installed;

[0075] In this embodiment, the IEEE 1588 PTP protocol is used to realize the device clock synchronization, and the monitored synchronization error is less than or equal to ±0.1ms (the measured standard deviation σ=0.03ms).

[0076] S2. Perform content parsing on the constructed multimodal audio and video data, and extract the corresponding teaching status feature set based on the content parsing results;

[0077] The teaching status feature set in S2 includes teacher and student behavior features based on video stream parsing, voice interaction features based on audio stream parsing, and teaching content structure features based on screen content parsing.

[0078] The teacher-student behavioral characteristics are composed of teacher movement trajectory density and student head-up rate. The teacher movement trajectory density at time t is denoted as Dt.

[0079]

[0080] Where Nt represents the number of times the teacher's position is updated within a preset unit time interval before time t in the acquired video stream; T represents the preset sampling time window length; and A represents the classroom area.

[0081] The student head-up rate at time t is equal to the ratio of the number of students in the captured video stream who are looking at the teacher / PPT playback screen at time t to the total number of students.

[0082] In the process of acquiring speech interaction features based on audio stream parsing, the frame length of the acquired audio stream is obtained, where the audio frame length represents the number of audio sampling points contained in the corresponding audio frame. The audio frame corresponding to any acquisition time and the midpoint of each sampling point within the corresponding audio frame are acquired. The short-time energy of the speech corresponding to the corresponding audio frame is obtained, serving as the speech interaction feature of the acquisition time corresponding to the midpoint of each sampling point within the corresponding audio frame. The short-time energy of the speech corresponding to the nth frame is denoted as En.

[0083]

[0084] Where K represents the number of preset audio sampling points contained in the audio frame; This represents the square of the amplitude (instantaneous power) corresponding to the k-th sampling point within the n-th frame.

[0085] In the process of obtaining the structured features of teaching content based on screen content parsing, the screen recording text is recognized using OCR technology. Preset keyword types corresponding to the teaching content at collection time t are extracted, along with the ratio of the frequency of each preset keyword to the total frequency of extracted preset keywords. The teaching themes corresponding to the preset keyword types extracted from the database's preset forms are queried. Based on the ratio of the frequency of each preset keyword to the total frequency of extracted preset keywords, a teaching theme bias coefficient at collection time t is calculated. This teaching theme bias coefficient at collection time t is equal to the ratio of the frequency of each preset keyword to the total frequency of extracted preset keywords multiplied by the sum of the weight factors of the corresponding preset keyword types based on the corresponding teaching themes in the database's preset forms. The obtained teaching theme bias coefficient and the corresponding teaching theme are used as the first and second parameters in the structured features of the teaching content, respectively.

[0086] S3. Combining teaching status feature data from historical data, identify classroom teaching events from the teaching status feature set extracted at the current time; and combine MTES technology to analyze the evolution process of classroom teaching events based on time series models. Based on the analysis results of the classroom teaching event evolution process, dynamically segment the classroom audio and video.

[0087] In this embodiment, MTES stands for Multimodal Teaching Evaluation System. This system combines multiple modalities (such as text, audio, video, and images) to comprehensively evaluate and analyze the teaching process. Utilizing advanced technologies such as artificial intelligence, machine learning, and big data analytics, it captures, records, and evaluates various elements of teaching activities to provide more comprehensive, objective, and detailed teaching feedback. It can generate targeted classroom teaching evaluation reports for teachers of different levels, helping them improve their classroom teaching and promoting the transformation from knowledge-based classrooms to competency-based classrooms.

[0088] In the process of S3 identifying classroom teaching events from the teaching state feature set extracted at the current time, a preset teaching event sequence is obtained, and the event probability of each element in the preset teaching event sequence based on the teaching state feature data at collection time t is calculated. The element with the highest event probability based on the teaching state feature data at collection time t in the preset teaching event sequence is taken as the classroom teaching event identification result at collection time t. The calculation formulas involved are as follows:

[0089]

[0090]

[0091] Where FGt represents the teaching state characteristic data collected at time t; P(E|FGt) represents the event probability of any element E in the preset teaching event sequence based on the teaching state characteristic data FGt collected at time t; g m Both gh and gh represent state feature functions, which are functions with base e and exponents calculated by subtracting the square of the difference between the average of all values ​​of the same teaching state feature type corresponding to the first parameter and the teaching state feature data corresponding to the second parameter in historical data. H represents a transition feature function. If the event state data pair formed by the first and second parameters in the transition feature function belongs to a preset effective transition set of event states, then the function value of the corresponding transition feature function is determined to be 1; otherwise, the function value of the corresponding transition feature function is determined to be 0. Each element in the effective transition set of event states corresponds to an event state data pair. The summary set of the first parameters in teacher movement trajectory density, student head-up rate, voice interaction features, and teaching content structured features in FGt is obtained and denoted as the first analysis set. m E represents the m-th parameter value in the first analysis set. t-1 The most recently identified teaching event is represented before the acquisition time t; W(FGt, m, E) represents the event adaptation evaluation coefficient of any element E in the preset teaching event sequence based on the teaching state feature data FGt at acquisition time t.

[0092] Z(FGt) represents the sum of each W(FGt, m, E) corresponding to each element in the preset teaching event sequence; This represents the preset state feature weight corresponding to the m-th parameter in the first analysis set; The preset state feature weights corresponding to the teaching state evaluation coefficients are represented by μ; μ represents the transition feature weights, where the value of μ is equal to the ratio of the frequency of the event state data pair formed by the first and second parameters in the corresponding transition feature function appearing in the historical data to the sum of the frequencies of all event state data pairs in the historical data that have the same first parameter and the first parameter of the corresponding transition feature function; max{} represents the maximum value function.

[0093]

[0094] Where Ft represents the teaching status evaluation coefficient at acquisition time t; Lt represents the student head-up rate at acquisition time t; E(t) represents the short-time energy of the corresponding audio frame at acquisition time t; Xt represents the first parameter in the structured features of the teaching content at acquisition time t; r1, r2 and r3 are the preset first evaluation weight, second evaluation weight and third evaluation weight, respectively, and r1+r2+r3=1.

[0095] In the process of analyzing the evolution of classroom teaching events based on a time-series model using MTES technology, an array consisting of the identification results of each classroom teaching event is obtained sequentially according to time, denoted as the classroom teaching event evolution analysis array. In this array, before a new classroom teaching event is identified, the original classroom teaching identification results remain unchanged, and each time point corresponds to a unique classroom teaching event identification result. Based on MTES technology, a set of indicators bound to each classroom teaching event in the classroom teaching time evolution analysis array is calculated. This set of indicators includes interaction density and cognitive load index. The calculation method for interaction density is as follows:

[0096]

[0097] Where QN represents the number of valid questions asked by the teacher within the time period corresponding to the corresponding classroom teaching event identification result; TS represents the interval length of the time period corresponding to the corresponding classroom teaching event identification result; NSUM represents the sum of the number of students who participated in each teacher questioning interaction within the time period corresponding to the corresponding classroom teaching event identification result; and NA represents the total number of students in the classroom.

[0098] The cognitive load index is calculated as follows:

[0099]

[0100] Where CH represents the cognitive load index corresponding to the identification results of the corresponding classroom teaching event; YP tb tb represents the average facial confusion level of each student at any time point tb within the time period corresponding to the corresponding classroom teaching event recognition result; the facial confusion level of a student is equal to the quotient of the number of wrinkles between the eyebrows obtained through image recognition divided by the area of ​​the area between the eyebrows; tb1 represents the minimum time point within the time period corresponding to the corresponding classroom teaching event recognition result; VS tb The VS represents the speech rate change coefficient at any time point tb within the time period corresponding to the classroom teaching event recognition result. tb The value is equal to the difference between the most recent monitored speech rate of the audio stream after time point tb in the historical data and the most recent monitored speech rate of the audio stream before time point tb in the historical data, divided by the quotient of the interval between two adjacent speech rate monitoring.

[0101] Based on the analysis results of the evolution of classroom teaching events, during the dynamic segmentation of classroom audio and video, when the absolute value of the difference between the cognitive load indices corresponding to two adjacent classroom teaching events is greater than or equal to a preset value, and the time interval between the time boundary point between two adjacent classroom teaching events and the previous time segmentation point is greater than or equal to a preset time, then the time boundary point between the corresponding two adjacent classroom teaching events is determined as a dynamic segmentation reserve point for the corresponding classroom audio and video; otherwise, it is determined that there is no dynamic segmentation point for the corresponding classroom audio and video between the corresponding two adjacent classroom teaching events.

[0102] Based on the teaching themes corresponding to the second parameter in the structured features of teaching content at different times, each dynamic segmentation reserve point is calibrated to obtain classroom audio-visual segments between two adjacent dynamic segmentation reserve points, which are recorded as segments to be analyzed. The minimum absolute value of the difference in cognitive load index between each classroom teaching event within the previous preset duration in the segment to be analyzed and the previous classroom teaching event of the first dynamic segmentation storage point in the two adjacent dynamic segmentation reserve points is extracted and recorded as the cognitive load index deviation reference value. If the teaching themes of the two classroom teaching events corresponding to the cognitive load index deviation reference value, as well as the teaching themes of the two classroom teaching events corresponding to the first dynamic segmentation storage point in the two adjacent dynamic segmentation reserve points, are the same, then the first dynamic segmentation storage point in the two adjacent dynamic segmentation reserve points is deleted. Each calibrated dynamic segmentation reserve point is used as a dynamic segmentation point of the classroom audio-visual video to obtain each segment of the classroom audio-visual video.

[0103] S4. Based on the dynamic segmentation results of classroom audio and video, generate a diagnostic assessment report corresponding to each segmented segment of classroom teaching audio and video.

[0104] The diagnostic assessment report for each segment of classroom teaching audio and video includes each segment and its corresponding score.

[0105] The calculation formulas involved in scoring the corresponding classroom audio and video segment are as follows:

[0106]

[0107] Wherein, GS represents the score of the corresponding classroom audio-visual segment; DCP represents the average interaction density of each classroom teaching event within the corresponding classroom audio-visual segment; CHP represents the average cognitive load index of each classroom teaching event within the corresponding classroom audio-visual segment; and φ1 represents the preset scoring factor.

[0108] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0109] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent teaching evaluation and diagnosis based on multi-modal audio-video analysis, characterized in that, The method comprises: S1, synchronously collecting timestamp-aligned teacher-student behavior video stream, voice audio stream and teaching screen content stream by deploying camera array, directional microphone group and screen capture device in the classroom, and constructing multi-modal audio-video data; S2, performing content analysis on the constructed multi-modal audio-video data, and extracting corresponding teaching state feature set based on the content analysis result; S3, combining the teaching state feature data in the historical data, performing classroom teaching event identification on the teaching state feature set extracted at the current time, and performing classroom teaching event evolution process analysis based on the time sequence model combined with the MTES technology, and based on the obtained classroom teaching event evolution process analysis result, performing dynamic segmentation on the classroom audio-video; In the process of performing classroom teaching event evolution process analysis based on the time sequence model combined with the MTES technology, an array formed by the classroom teaching event identification results in time sequence is obtained in turn, which is recorded as a classroom teaching event evolution analysis array; the original classroom teaching identification result remains unchanged in the classroom teaching event evolution analysis array before a new classroom teaching event is identified, and the unique classroom teaching event identification result corresponding to each time point; based on the MTES technology, the index set bound to each classroom teaching event in the classroom teaching time evolution analysis array is calculated, and the index set includes interaction density and cognitive load index; the calculation method of the interaction density is as follows: Wherein, QN represents the effective number of questions of the teacher in the time period corresponding to the corresponding classroom teaching event identification result; TS represents the interval length of the time period corresponding to the corresponding classroom teaching event identification result; NSUM represents the sum of the number of students participating in the teacher's question interaction in the time period corresponding to the corresponding classroom teaching event identification result; NA represents the total number of students in the classroom; The calculation method of the cognitive load index is as follows: wherein CH represents the cognitive load index corresponding to the recognition result of the corresponding classroom teaching event; YP tb represents the average value of the facial confusion degree of each student at any time point tb in the time period corresponding to the recognition result of the corresponding classroom teaching event; the facial confusion degree corresponding to the student is equal to the quotient of the number of frown lines of the student obtained through image recognition divided by the area of the glabella region; tb1 represents the minimum time point in the time period corresponding to the recognition result of the corresponding classroom teaching event; VS tb represents the speech speed mutation coefficient at any time point tb in the time period corresponding to the recognition result of the corresponding classroom teaching event; the value of the VS tb is equal to the quotient of the difference between the speech speed of the audio stream monitored last time after the time point tb in the historical data and the speech speed of the audio stream monitored last time before the time point tb in the historical data divided by the interval length of adjacent two speech speed monitoring. S4, generating a diagnostic evaluation report corresponding to each classroom teaching audio-video segmentation segment according to the dynamic segmentation result of the classroom audio-video. 2.The intelligent teaching evaluation and diagnosis method based on multi-modal audio-video analysis according to claim 1, characterized in that: The camera array comprises a panoramic camera covering the global dynamics of the teacher, a teacher tracking camera capturing the body language and board writing of the teacher, and a student expression camera collecting student facial response data. 3.The intelligent teaching evaluation and diagnosis method based on multi-modal audio-video analysis of claim 1, wherein: The teaching state feature set in S2 includes teacher-student behavior features based on video stream analysis, voice interaction features based on audio stream analysis, and teaching content structured features based on screen content analysis; The teacher-student behavior features are composed of teacher movement trajectory density and student head-up rate, and the teacher movement trajectory density at time t is recorded as Dt, Wherein, Nt represents the number of teacher position updates in the preset unit time before time t in the collected video stream; T represents the length of the preset sampling time window; A represents the area of the classroom; The student head-up rate at time t is equal to the ratio of the number of students watching the teacher / PPT display screen at time t in the collected video stream to the total number of students; In the process of obtaining the voice interaction feature based on audio stream analysis, the frame length in the collected audio stream is obtained, the audio frame length representing the number of audio sampling points contained in the corresponding audio frame, and the audio frame corresponding to any collection time and the midpoint of each sampling point in the corresponding audio frame are obtained; the voice short-time energy corresponding to the corresponding audio frame is obtained as the voice interaction feature corresponding to the collection time of the midpoint of each sampling point in the corresponding audio frame, and the voice short-time energy corresponding to the nth frame is denoted as En, K represents the preset number of audio sampling points contained in the audio frame; represents the square of the amplitude corresponding to the kth sampling point in the nth frame. In the process of obtaining the teaching content structured feature based on screen content analysis, the recording screen text is identified by the OCR technology, the types of preset keywords corresponding to the teaching content at the collection time t and the ratio of the frequency of occurrence of each preset keyword to the total frequency of the extracted preset keywords are extracted; the teaching theme corresponding to the extracted preset keyword types in the database preset form is queried, and based on the ratio of the frequency of occurrence of each preset keyword to the total frequency of the extracted preset keywords, the teaching theme bias coefficient at the collection time t is calculated, which is equal to the ratio of the frequency of occurrence of each preset keyword to the total frequency of the extracted preset keywords multiplied by the cumulative sum of the weight factors of the corresponding preset keyword types based on the corresponding teaching theme in the database preset form, and the obtained teaching theme bias coefficient and the corresponding teaching theme are respectively taken as the first parameter and the second parameter in the teaching content structured feature. 4.The intelligent teaching evaluation and diagnosis method based on multi-modal audio-video analysis of claim 3, characterized in that: In the process of S3 classroom teaching event identification of the teaching state feature set extracted at the current time, the preset teaching event sequence is obtained, and the event probability of each element in the preset teaching event sequence based on the teaching state feature data at the collection time t is calculated, and the element with the maximum event probability of the teaching state feature data at the collection time t in the preset teaching event sequence is taken as the classroom teaching event identification result at the collection time t, and the calculation formula is as follows: Wherein, FGt represents the teaching state feature data at the collection time t; P(E|FGt) represents the event probability of any one element E in the preset teaching event sequence based on the teaching state feature data FGt at the collection time t; g m Both gh represent state feature functions, which represent functions with base e and with the square of the difference between the average value of each value in the same teaching state feature type as the second parameter corresponding teaching state feature data in the first parameter corresponding teaching event in the historical data and the second parameter corresponding teaching state feature data as the index; H represents a transition feature function, if the event state data pair formed by the first parameter and the second parameter in the transition feature function belongs to the preset event state effective transition set, it is determined that the function value corresponding to the transition feature function is 1, otherwise, it is determined that the function value corresponding to the transition feature function is 0; Each element in the event state effective transition set corresponds to an event state data pair; The first parameter of the teacher moving track density, the student head-up rate, the voice interaction feature and the teaching content structured feature in FGt is obtained, and the collection set is recorded as the first analysis set; FGt m represents the mth parameter value in the first analysis set; E t-1 represents the teaching event recognized last time before the collection time t; W(FGt,m,E) represents the event adaptation evaluation coefficient of any one element E in the preset teaching event sequence based on the teaching state feature data FGt at the collection time t; Z(FGt) represents the sum of each W(FGt, m, E) corresponding to each element E in the preset teaching event sequence; represents the preset state feature weight corresponding to the mth parameter in the first analysis set; represents the preset state feature weight corresponding to the teaching state evaluation coefficient; μ represents the transition feature weight, and a value of the μ is equal to a ratio of a frequency of an event state data pair composed of a first parameter and a second parameter in a corresponding transition feature function appearing in historical data to a sum of frequencies of each event state data pair corresponding to the first parameter and the same as the first parameter in the corresponding transition feature function appearing in the historical data; max{} represents a maximum value function; Wherein, Ft represents the teaching state evaluation coefficient at the collection time t; Lt represents the student head-up rate at the collection time t; E(t) represents the voice short-time energy of the corresponding audio frame at the collection time t; Xt represents the first parameter in the teaching content structured feature at the collection time t; r1, r2 and r3 are respectively the first evaluation weight, the second evaluation weight and the third evaluation weight, and r1+r2+r3=1. 5.The intelligent teaching evaluation and diagnosis method based on multi-modal audio-video analysis according to claim 1, characterized in that: Based on the obtained classroom teaching event evolution process analysis result, in the process of dynamically segmenting the classroom audio and video, when the absolute value of the difference between the cognitive load indexes corresponding to the adjacent two classroom teaching events is greater than or equal to the preset value, and the interval length of the time junction point between the adjacent two classroom teaching events from the last time segmentation point is greater than or equal to the preset length, it is determined that the time junction point between the corresponding adjacent two classroom teaching events is a dynamic segmentation reserve point of the corresponding classroom audio and video; On the contrary, it is determined that there is no dynamic segmentation point of the corresponding classroom audio and video between the corresponding adjacent two classroom teaching events. Based on the second parameter corresponding to the teaching theme in the structural features of the teaching content at different times, the obtained dynamic segmentation reserve points are calibrated to obtain the classroom audio and video segments between adjacent two dynamic segmentation reserve points, denoted as to-be-analyzed segments; the minimum value of the absolute value of the cognitive load index difference between each classroom teaching event in the to-be-analyzed segment within the preset time and the previous classroom teaching event of the first dynamic segmentation storage point in the adjacent two dynamic segmentation storage points is extracted, denoted as a cognitive load index deviation reference value; If the teaching themes of the two classroom teaching events corresponding to the cognitive load index deviation reference value and the teaching themes of the two classroom teaching events corresponding to the previous and subsequent two classroom teaching events of the first dynamic segmentation storage point in the adjacent two dynamic segmentation storage points are all the same, the first dynamic segmentation storage point in the adjacent two dynamic segmentation storage points is deleted; each calibrated dynamic segmentation reserve point is taken as a dynamic segmentation point of the classroom audio and video to obtain each segmentation segment of the classroom audio and video; The diagnostic evaluation report corresponding to each classroom teaching audio and video segmentation segment includes each classroom audio and video segmentation segment and the score of the corresponding classroom audio and video segmentation segment, The calculation formula involved in calculating the score of the corresponding classroom audio and video segmentation segment is as follows: Wherein, GS represents the score of the corresponding classroom audio and video segmentation segment; DCP represents the average value of the interaction density corresponding to each classroom teaching event in the corresponding classroom audio and video segmentation segment; CHP represents the average value of the cognitive load index corresponding to each classroom teaching event in the corresponding classroom audio and video segmentation segment; φ1 represents a preset score factor.

6. The intelligent teaching evaluation and diagnosis system based on multi-modal audio-video analysis, applying the intelligent teaching evaluation and diagnosis method based on multi-modal audio-video analysis of any one of claims 1-5, characterized in that, The system comprises: A multi-modal audio and video data construction module, which synchronously collects time stamp aligned teacher and student behavior video stream, voice audio stream and teaching screen content stream through a camera array, a directional microphone group and a screen capture device arranged in a classroom, and constructs multi-modal audio and video data; A teaching state feature extraction module, which performs content analysis on the constructed multi-modal audio and video data, and extracts a corresponding teaching state feature set based on the content analysis result; An audio and video dynamic segmentation module, which combines the teaching state feature data in the historical data to perform classroom teaching event identification on the teaching state feature set extracted at the current time, and performs classroom teaching event evolution process analysis based on a time sequence model by combining the MTES technology, and performs dynamic segmentation on the classroom audio and video based on the obtained classroom teaching event evolution process analysis result; A teaching evaluation diagnosis management module, which generates a diagnostic evaluation report corresponding to each classroom teaching audio and video segmentation segment according to the dynamic segmentation result of the classroom audio and video. 7.The intelligent teaching evaluation and diagnosis system based on multi-modal audio-video analysis of claim 6, wherein: The audio and video dynamic segmentation module comprises a teaching event identification unit, an event evolution process analysis unit and a segmentation point dynamic management unit; The teaching event identification unit combines the teaching state feature data in the historical data to perform classroom teaching event identification on the teaching state feature set extracted at the current time; The event evolution process analysis unit analyzes the classroom teaching event evolution process based on a timing model in combination with the MTES technology. The dynamic segmentation point management unit dynamically segments the classroom audio and video based on the obtained classroom teaching event evolution process analysis result.

Images