A teacher ability diagnosis and evaluation method and system based on multi-modal data fusion

Abstract

This invention discloses a method and system for diagnosing and assessing teacher competence based on multimodal data fusion, applied in the field of data analysis. The method includes acquiring teaching video data of the target teacher and its corresponding lesson plan text data; then adaptively segmenting the lesson plan text into multiple text blocks, with the segmentation boundaries determined based on semantic density distribution; fusing corresponding sentiment enhancement vectors into each text block and extracting audio track prosodic features from the teaching video; establishing a precise correspondence between each text block and audio track segment using spatiotemporal alignment technology, and then calculating the first and second matching degrees; adjusting the contribution weight of semantic density to each matching degree to generate the final diagnostic assessment result of teacher teaching competence. The teacher competence diagnostic assessment method based on multimodal data fusion provided by this invention effectively improves the objectivity, granularity, and educational guidance value of teacher teaching competence assessment.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a method and system for diagnosing and assessing teacher competence based on multimodal data fusion. Background Technology

[0002] With the deep integration of educational informatization and artificial intelligence technologies, the use of multimodal data analysis technology to automat and intelligently diagnose and evaluate teachers' classroom teaching abilities has become an important development direction in the field of educational evaluation.

[0003] Existing assessment methods mostly rely on scoring the completeness of lesson plan texts or expert observation and scoring, which makes it difficult to objectively reflect the consistency between teachers' language expression and the logic of teaching content during actual teaching. Although some automated assessment methods have introduced teaching video or audio analysis, they usually treat lesson plan texts and classroom audio as independent modalities, lacking in-depth modeling of the temporal evolution and semantic structure between the two, which makes it difficult to meet the actual needs of intelligent and precise teaching diagnosis. Summary of the Invention

[0004] This invention provides a teacher competence diagnosis and assessment method and system based on multimodal data fusion to solve the technical problem of lack of fine-grained, temporally aligned quantitative correlation between lesson plan text and teacher's speech and prosody in the teacher competence assessment process, so as to achieve automated assessment of teachers' teaching competence in terms of content expression consistency and emotional transmission suitability.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a teacher competence diagnostic and assessment method based on multimodal data fusion, comprising: Obtain the data to be evaluated, which includes at least teaching video data and corresponding lesson plan text data; The lesson plan text data is segmented to obtain various text blocks, wherein the dependency relationships between the various text blocks reflect the semantic density distribution of the lesson plan text data. The corresponding sentiment enhancement vectors are fused into each of the aforementioned text blocks; At least the prosodic features of each audio track are extracted from the teaching video data; Based on spatiotemporal alignment technology, the correspondence between each of the text blocks and the teaching video data is determined. Based on the correspondence, the matching degree between all the text blocks and all the audio track prosodic features is obtained. The matching degree includes the first matching degree between the sentiment enhancement vector of the current text block and the corresponding first audio track prosodic feature, and the second matching degree between the semantic function of the current text block and the intonation fluctuation information. The intonation fluctuation information is determined by the second audio track prosodic features corresponding to other text blocks adjacent to the current text block. The contribution weight of the matching degree is adjusted using the semantic density distribution, and the ability diagnostic assessment result of the target teacher corresponding to the data to be assessed is determined based at least on the adjusted matching degree.

[0006] As one preferred embodiment, the segmentation of the lesson plan text data to obtain various text blocks includes: The lesson plan text data is segmented to obtain multiple candidate text units; Based on the semantic association strength between the candidate text units, an initial semantic network graph is constructed; Calculate the semantic density value of each local subgraph in the initial semantic network graph to characterize the semantic density distribution in different regions of the lesson plan text data; Based on the semantic density distribution, the candidate text units are processed to obtain each text block.

[0007] As one preferred embodiment, the process of processing the candidate text units based on the semantic density distribution to obtain each text block includes: Based on the semantic density distribution, determine the global average semantic density; Candidate text units with semantic density values ​​higher than the global average semantic density are segmented, and candidate text units with semantic density values ​​lower than the global average semantic density are merged. The results of the segmentation and merging processes are output as the text blocks.

[0008] As one preferred embodiment, the segmentation process for candidate text units whose semantic density values ​​are higher than the global average semantic density includes: Identify teaching action verbs in the candidate text units, and determine the corresponding teaching action type based on the teaching action verbs; If the candidate text unit includes at least two different teaching behavior types, then the text is segmented at adjacent and different teaching behavior types to obtain multiple text blocks; The merging process for candidate text units whose semantic density value is lower than the global average semantic density includes: If adjacent candidate text units have the same teaching behavior type and their semantic density values ​​are all lower than the global average semantic density, the candidate text units are merged to obtain the text block.

[0009] As one preferred embodiment, the extraction of prosodic features from at least the teaching video data for each audio track includes: The teaching audio track data is separated from the teaching video data, and the teaching audio track data is processed with speech enhancement and speech activity detection to retain valid speech segments; The effective speech segments are subjected to overlapping frame segmentation to obtain a time-series audio sequence; Extract acoustic features related to the prosodic performance of the target teacher during the lesson from each audio frame; The acoustic features of adjacent audio frames are aggregated according to a preset time window to generate the audio track prosodic features.

[0010] As one preferred embodiment, the step of extracting acoustic features related to the prosodic performance of the target teacher during instruction from each audio frame includes: Perform time-domain autocorrelation analysis and frequency-domain cepstral analysis on the audio frame, and determine the fundamental frequency F0 based on the analysis results; Calculate the signal energy of the audio frame, and obtain the logarithmic energy based on the signal energy; Based on the results of the speech activity detection and processing, speech rate-sensitive features related to the speech temporal distribution are generated.

[0011] As one preferred embodiment, the calculation of the matching degree includes: Based on the sentiment enhancement vector of the current text segment, determine the corresponding expected sentiment representation; The first audio track's prosodic features are compared with the expected emotional representation to determine the first degree of matching; Based on the type of semantic function of the current text block, determine the expected prosodic transition relationship between the other text blocks adjacent to the current text block; Based on the second audio track prosodic features corresponding to the current text block and the adjacent other text blocks, verify whether the expected prosodic transition relationship is valid, so as to determine the second matching degree.

[0012] As one preferred embodiment, the step of comparing the prosodic features of the first audio track with the expected emotional representation to determine the first matching degree includes: Based on the emotional intensity level mapped by the emotional enhancement vector, the corresponding multidimensional matching rule is invoked; The first audio track prosodic features are compared with the multidimensional matching rules, and the first matching degree is determined based on the comparison results.

[0013] As one preferred embodiment, the step of verifying whether the expected prosodic transition relationship holds true based on the second audio track prosodic features corresponding to the current text block and the adjacent other text blocks, in order to determine the second matching degree, includes: Based on the type of semantic function of the current text segment, the corresponding prosodic transition rule is invoked; The second audio track prosodic features are compared with the prosodic transition rules, and the second matching degree is determined based on the comparison results.

[0014] Another embodiment of the present invention provides a teacher competence diagnostic and assessment system based on multimodal data fusion, comprising: The data acquisition module is used to acquire the data to be evaluated, which includes at least teaching video data and corresponding lesson plan text data. The data segmentation module is used to segment the lesson plan text data to obtain various text blocks, wherein the dependency relationships between the various text blocks reflect the semantic density distribution of the lesson plan text data. The vector enhancement module is used to fuse the corresponding sentiment enhancement vector into each of the text blocks. The feature extraction module is used to extract prosodic features of each audio track from the teaching video data at least; The alignment module is used to determine the correspondence between each text block and the teaching video data based on spatiotemporal alignment technology; The feature matching module is used to obtain the matching degree between all the text blocks and all the audio track prosodic features based on the correspondence. The matching degree includes a first matching degree between the sentiment enhancement vector of the current text block and the corresponding first audio track prosodic feature, and a second matching degree between the semantic function of the current text block and the intonation fluctuation information. The intonation fluctuation information is determined by the second audio track prosodic features corresponding to other text blocks adjacent to the current text block. The diagnostic assessment module is used to adjust the contribution weight of the matching degree using the semantic density distribution, and to determine the diagnostic assessment result of the target teacher's ability corresponding to the data to be assessed, based at least on the adjusted matching degree.

[0015] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following: (1) This invention segments lesson plan text data into text blocks with dependency relationships based on semantic density and integrates sentiment enhancement vectors; extracts the audio track prosodic features from teaching videos, and uses spatiotemporal stacking alignment to establish a correspondence between the text blocks integrated with sentiment enhancement vectors and the audio track prosodic features. Then, it calculates the matching degree from two dimensions: sentiment consistency and semantic coordination. Finally, it dynamically adjusts the weights based on semantic density to generate a comprehensive diagnostic evaluation result. This invention achieves refined quantification of the collaborative quality of content organization, emotional expression, and prosodic performance during teacher instruction, thereby significantly improving the objectivity, fine granularity, and interpretability of teacher competence diagnosis, and providing data-driven, precise feedback for teacher professional development.

[0016] (2) Compared with the prior art, the present invention can significantly improve the objectivity and adaptability of teacher competence assessment. On the one hand, by introducing the semantic density distribution of lesson plan text as the basis for weight adjustment, the assessment results are more focused on the presentation quality of key teaching content. On the other hand, it creatively constructs a dual-channel mechanism to measure the consistency between the teacher's emotional expression and actual speech rhythm, and to examine whether its semantic function matches the context and intonation fluctuations, effectively identifying typical teaching problems such as "reading from the book" and "monotonous intonation". Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a teacher competence diagnosis and assessment method based on multimodal data fusion in one embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the alignment of text blocks with the time interval of a teaching video in one embodiment of the present invention; Figure 3 This is a schematic diagram of a teacher competence diagnosis and assessment system based on multimodal data fusion in one embodiment of the present invention.

[0018] Figure label: The module includes: 11. Data acquisition module; 12. Data segmentation module; 13. Vector enhancement module; 14. Feature extraction module; 15. Alignment module; 16. Feature matching module; 17. Diagnostic evaluation module; and 21. Time axis. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] One embodiment of the present invention provides a teacher competence diagnosis and assessment method based on multimodal data fusion. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 The diagram shown is a flowchart of a teacher competence diagnosis and assessment method based on multimodal data fusion according to one embodiment of the present invention, which includes steps S1 to S7: S1: Obtain the data to be evaluated, which includes at least teaching video data and corresponding lesson plan text data; S2: Segment the lesson plan text data to obtain various text blocks. The dependency relationships between the text blocks reflect the semantic density distribution of the lesson plan text data. S3: Fuse the corresponding sentiment enhancement vectors into each text block; S4: Extract prosodic features from at least the teaching video data for each audio track; S5: Based on spatiotemporal alignment technology, determine the correspondence between each text block and the teaching video data; S6: Based on the correspondence, obtain the matching degree between all text blocks and all audio track prosodic features. The matching degree includes the first matching degree between the sentiment enhancement vector of the current text block and the corresponding first audio track prosodic feature, and the second matching degree between the semantic function of the current text block and the intonation fluctuation information. The intonation fluctuation information is determined by the second audio track prosodic features corresponding to other text blocks adjacent to the current text block. S7: Adjust the contribution weight of the matching degree using the semantic density distribution, and determine the ability diagnostic assessment results of the target teacher corresponding to the data to be assessed, based at least on the adjusted matching degree.

[0022] Specifically, the first step is to acquire the data to be evaluated, including a 45-minute teaching video and its corresponding lesson plan text. The teaching video data is captured synchronously using a high-definition camera and directional microphone installed on the ceiling of the recording studio, with a video frame rate of 30fps and an audio sampling rate of 16kHz. The lesson plan text data is a Word or PDF document containing structured content such as the lesson title, learning objectives, explanations of key knowledge points, classroom questions, and a summary, prepared by the teacher before class. Since the lesson plan represents the teacher's pre-designed instructional plan, comparing it with the actual audio during the lesson provides a more accurate reflection of the teacher's content organization skills and consistency with their on-the-spot delivery.

[0023] Preferably, in one embodiment of the present invention, the lesson plan text data is segmented to obtain various text blocks, including: The lesson plan text data is segmented to obtain multiple candidate text units; An initial semantic network graph is constructed based on the semantic association strength between candidate text units; Calculate the semantic density value of each local subgraph in the initial semantic network graph to characterize the semantic density distribution in different regions of the lesson plan text data; Based on the semantic density distribution, candidate text units are processed to obtain various text blocks.

[0024] This embodiment constructs a semantic network graph from lesson plan text data and calculates the semantic density of local subgraphs. Combined with teaching behavior type recognition, it differentiates and processes candidate text units, thereby achieving intelligent text segmentation oriented towards the teaching logic structure. This method can effectively distinguish between semantically dense core teaching areas and semantically sparse transitional management areas in lesson plans, generating text segments adapted to the needs of teacher competency diagnosis, and providing high-quality data input for subsequent multimodal alignment.

[0025] Specifically, the lesson plan text data first needs to be preprocessed to remove non-teaching content such as headers, footers, and numbering, retaining only the plain text data. Then, the lesson plan text data is initially segmented using natural sentences as the basic unit, resulting in multiple candidate text units. Each candidate text unit corresponds to a complete semantic sentence, serving as the basic node for constructing the initial semantic network graph. Natural sentences are chosen as candidate text units because teaching language often contains multiple teaching actions within a paragraph (such as asking questions + explaining + summarizing). Directly using paragraphs as units would mask internal semantic changes, hindering precise diagnosis.

[0026] Then, each candidate text unit is encoded using a pre-trained language model (e.g., BERT-base) to obtain its context-aware embedding vector. By calculating the semantic association strength between any two adjacent candidate text units, a directed acyclic graph is constructed, in which nodes represent candidate text units and edges represent strong semantic coherence relationships, which is the initial semantic network graph.

[0027] Specifically, the semantic association strength is obtained by calculating the cosine similarity between any two adjacent candidate text units, as shown in the following formula: ; In the formula, It represents the semantic association strength between the i-th candidate text unit and the (i+1)-th candidate text unit. It is a dimensionless numerical value with a range of [0,1]. and These represent the context-aware embedding vectors obtained after encoding the i-th and (i+1)-th candidate text units by a pre-trained language model, respectively. and Representing vectors respectively and The Euclidean norm is used to normalize the dot product result, so that the final result only reflects directional similarity and is not affected by the absolute size of the vector.

[0028] If the semantic association strength is greater than the preset threshold, it means that the adjacent candidate text units are highly semantically coherent and remain connected in the initial semantic network graph; if the semantic association strength is less than the preset threshold, it means that there are semantic breakpoints between the adjacent candidate text units, and the edges need to be broken in the initial semantic network graph as potential block boundaries.

[0029] After obtaining the initial semantic network graph, the semantic density of each maximally connected subgraph in the graph needs to be calculated. The calculation formula is as follows: ; In the formula, The number of units contained in the k-th subgraph; This represents the semantic density value of the k-th maximal connected subgraph. A higher semantic density value indicates that the semantics within that region are more tightly packed and the information is more concentrated.

[0030] Furthermore, the global average semantic density is calculated using the following formula: ; In the formula, K represents the total number of connected subgraphs; This represents the global average semantic density of the lesson plan text. Semantic density is not simply a statistical count of word frequency or keyword count, but rather a quantification of the "concentration of teaching information per unit length" through semantic coherence, which is more in line with the laws of educational cognition.

[0031] Preferably, in one embodiment of the present invention, candidate text units are processed based on semantic density distribution to obtain various text blocks, including: Determine the global average semantic density based on the semantic density distribution; Candidate text units with semantic density values ​​higher than the global average semantic density are segmented, while candidate text units with semantic density values ​​lower than the global average semantic density are merged. The output of the segmented and merged results is used as text blocks.

[0032] Specifically, based on the calculated global average semantic density, each candidate text unit is processed differently: for subgraphs with semantic density values ​​higher than the global average semantic density, further analysis is conducted to determine whether there are multiple teaching behavior types within them, and then they are segmented based on the analysis results; for subgraphs with semantic density values ​​lower than the global average semantic density, it is checked whether they are composed of adjacent units with the same teaching behavior type, and they are merged based on the results.

[0033] Preferably, in one embodiment of the present invention, segmentation processing is performed on candidate text units whose semantic density values ​​are higher than the global average semantic density, including: Identify teaching action verbs in candidate text units and determine the corresponding teaching action type based on the teaching action verbs; If a candidate text unit includes at least two different types of teaching behavior, then it is segmented at adjacent and different types of teaching behavior to obtain multiple text blocks; Candidate text units with semantic density values ​​lower than the global average semantic density are merged, including: If adjacent candidate text units have the same teaching behavior type and their semantic density values ​​are all lower than the global average semantic density, the candidate text units are merged to obtain text blocks.

[0034] Specifically, the first step is to construct a dictionary of teaching behavior verbs, as shown in Table 1 below: Table 1. Dictionary of Verbs Classified by Teaching Behavior Explanation Imparting knowledge and clarifying concepts Explanation, elaboration, derivation, introduction, etc. Question type To stimulate students' thinking or interaction Asking questions, thinking, discussing, and guessing, etc. Instruction class Guide students in performing specific classroom tasks Please look, pay attention, complete, record, etc. Summary Summarize key points and strengthen core theories In conclusion, therefore, so, summarizing, etc. For each candidate text unit in the subgraph, the teaching action verbs are identified using a BiLSTM-CRF model with rule matching or fine-tuning, and assigned a main action label b∈{explanation, questioning, instruction, summary}. If different action labels are detected between adjacent units in a continuous unit sequence, and the location is in a high-density region, a block boundary is inserted at that location. For example, the unit sequence: “Who can answer this question? (questioning) → The correct answer is… (explanation) → Please write it down (instruction)”. Although the three are semantically coherent, the frequent switching of action types indicates that the teacher has completed a complete teaching cycle of “inspiration-instruction-reinforcement” in this high-density segment, and it should be divided into three independent text blocks.

[0035] If adjacent candidate text units have the same behavior label, these units are merged into a single text chunk. For example, the unit sequence: "Please open your textbook to page 35. (Instruction) → Find example 2. (Instruction) → Try doing it yourself first. (Instruction)". Although each sentence is independent, they are all "instruction" behaviors with low semantic density, so merging them into a single chunk better reflects the overall teaching intent.

[0036] Finally, the output is a sequence of text blocks after the above segmentation and merging operations. Each text block has internal consistency and external distinguishability in terms of semantics and pedagogical function. The teaching text processing method in this embodiment avoids the "semantic fragmentation" or "functional mixing" caused by mechanical segmentation, providing a high-quality foundation for subsequent accurate spatiotemporal alignment and matching degree calculation with the prosodic features of the audio track, and significantly improving the fine-grainedness and accuracy of teacher ability diagnosis.

[0037] The corresponding sentiment enhancement vector is fused into each text segment. This sentiment enhancement vector is a numerical vector representing the emotional context information associated with the current teaching text segment. Its function is to inject non-semantic emotional cues into the semantic representation of the text to reflect the emotional expression pattern expected in the instructional design. In this embodiment, the sentiment enhancement vector is generated based on the teaching segment type in the lesson plan text. Specifically, for each text segment, the system first identifies its teaching segment type. This type usually exists in a structured form in the lesson plan, for example, identified by square brackets (such as "[Question]", "[Summary]"), XML tags, or preset keywords (such as "Please think about this", "Let's summarize now"). Common teaching segment types include "Introduction", "Explanation", "Question", "Discussion", "Demonstration", and "Summary".

[0038] Different teaching segments correspond to different expected emotional representations. For example, in the "questioning" segment, teachers are expected to use an inquisitive and open-ended tone to stimulate student thinking; in the "summary" segment, a steady and definite tone is expected to convey the knowledge loop; and in the "introduction" or "discussion" segments, a relaxed and interactive atmosphere is preferred. To quantify these expectations, the system has a built-in teaching segment-emotion mapping table. This table is pre-constructed by analyzing expert-annotated data from 30 typical demonstration lessons, mapping each teaching segment type to a standardized emotional intensity index with a value ranging from 0 to 1. Finally, this normalized emotional intensity index is expanded into a 768-dimensional constant vector, where the value of each dimension in the vector is equal to the normalized emotional intensity index, thus obtaining the final emotional enhancement vector.

[0039] After obtaining the sentiment enhancement vector, it is fused with the semantic representation of the corresponding text block. Specifically, each text block consists of candidate text units. The system averages the embedding vectors of the candidate text units obtained in the previous steps to obtain the semantic embedding of the text block. Then, the obtained semantic vector is added to the corresponding sentiment enhancement vector along its dimension to form the final fused representation. In this embodiment, the final text block not only retains the original teaching semantic content but also incorporates sentiment expectation information corresponding to its teaching segment type, providing a more comprehensive semantic-sentiment joint representation for subsequent matching calculations with the prosodic features of the audio track.

[0040] The fused text chunk representation already contains both instructional semantic content and emotional expectation information. To achieve multimodal consistency assessment, it is also necessary to obtain the speech expression features of the target teacher during actual instruction from the teaching video and establish their correspondence with the text chunks. For details, see [link to relevant documentation]. Figure 2 , Figure 2 The diagram illustrates the alignment of text blocks with the time intervals of a teaching video in one embodiment of the present invention. Preferably, in one embodiment of the present invention, at least the prosodic features of each audio track are extracted from the teaching video data, including: The teaching audio track data is separated from the teaching video data, and the teaching audio track data is processed by speech enhancement and speech activity detection to retain effective speech segments; The effective speech segments are subjected to overlapping frame segmentation to obtain a time-series audio sequence. Extract acoustic features from each audio frame that are related to the prosodic performance of the target teacher during the lesson; The acoustic features of adjacent audio frames are aggregated according to a preset time window to generate audio track prosodic features.

[0041] The audio track prosodic features refer to a set of acoustic quantitative indicators extracted from the teacher's lecturing voice contained in the teaching video, which can characterize the rhythm, intonation changes, and emotional intensity of the teacher's speech expression. Specifically, the teaching audio track data is first separated from the teaching video data. The teacher's voice channel in the mixed audio is extracted using a sound source separation model, and interference components such as student discussions and environmental noise are suppressed. The sound source separation model can adopt deep learning-based methods (such as Conv-TasNet, a convolutional temporal network that separates the target speaker's voice from the mixed signal through end-to-end training) or traditional blind source separation methods (such as Independent Component Analysis (ICA)). Then, the separated teaching audio track data is subjected to speech enhancement processing. The open-source tool RNNoise (a real-time noise suppression algorithm based on recurrent neural networks) is used to suppress background noise and improve speech clarity. Finally, Voice Activity Detection (VAD) is used to identify the time periods containing valid human voices. This step can employ WebRTC VAD (a widely deployed lightweight detector that uses a Gaussian mixture model to determine whether a frame contains speech), or it can be replaced with a deep learning-based VAD (such as SincNet-VAD) to adapt to different signal-to-noise ratio environments. In this embodiment, long periods of silence or non-speech segments are removed through speech enhancement and speech activity detection, retaining only continuous and valid speech segments for subsequent analysis.

[0042] Next, the obtained valid speech segments are subjected to overlapping framing processing. Specifically, the valid speech segments are divided into short frames of 25 milliseconds in length, with a 10-millisecond overlap between adjacent frames, thus forming a continuous temporal audio frame sequence. This processing ensures that the speech characteristics within each frame are relatively stable, while avoiding the loss of prosodic information caused by inter-frame transitions.

[0043] Preferably, in one embodiment of the present invention, extracting acoustic features related to the prosodic performance of the target teacher during instruction from each audio frame includes: Perform time-domain autocorrelation analysis and frequency-domain cepstral analysis on the audio frames, and determine the fundamental frequency F0 based on the analysis results; Calculate the signal energy of the audio frame and obtain the logarithmic energy based on the signal energy; Based on the results of speech activity detection processing, speech rate-sensitive features related to speech temporal distribution are generated.

[0044] The acoustic features specifically include: Fundamental frequency F0: The fundamental frequency reflects the pitch variation of a human voice and is a core indicator for judging intonation fluctuations (e.g., rising and falling pitches). In this embodiment, the fundamental frequency F0 is obtained by simultaneously performing time-domain autocorrelation analysis and frequency-domain cepstral analysis on each audio frame. The time-domain autocorrelation analysis can preliminarily estimate F0 by calculating the signal periodicity, for example, using the AMDF or ACF algorithm; the frequency-domain method corrects for possible octave errors by using the cepstral peak positions, i.e., cases where F0 is misclassified as an integer or fractional multiple of it. The two results are fused through a weighted average or confidence selection mechanism to output a robust fundamental frequency value.

[0045] Logarithmic energy: Used to characterize the intensity or emphasis of a teacher's voice, determining whether the teacher raises their volume to instruct students on key points. Specifically, it is first calculated as the sum of the squares of all samples within each audio frame as the signal energy, then the logarithm is taken to base 10, with a small constant added to prevent taking the logarithm of zero, resulting in the logarithmic energy value.

[0046] Speech rate sensitivity feature: This feature is used to capture the pace and pause habits of teachers' lectures, effectively reflecting whether teachers slow down their speech at key concepts to facilitate student comprehension. This feature can be obtained by combining the results of speech activity detection. Specifically, it calculates the proportion of frames with detected "active speech" per unit of time. A higher proportion indicates a faster speech rate and fewer pauses during lectures, while a lower proportion indicates a slower speech rate, allowing students more time to think.

[0047] The acoustic features extracted from multiple adjacent audio frames are aggregated according to a preset time window to generate structured audio track prosodic features. In this embodiment, the time window can be set to 0.5 seconds, that is, the mean of the fundamental frequency, log energy, and speech rate sensitivity features of all frames within 0.5 seconds is calculated to form a three-dimensional vector. This three-dimensional vector is the audio track prosodic feature corresponding to the time period. In this embodiment, the audio track prosodic features obtained in this way not only retain the dynamic trend of prosody, but also reduce the data dimensionality, which is convenient for subsequent spatiotemporal alignment and matching calculations with text blocks.

[0048] After obtaining the structured prosodic features of the audio tracks, the correspondence between each text segment and the teaching video data is determined based on spatiotemporal alignment technology. In this embodiment, for each text segment, all original lesson plan sentences contained therein need to be extracted, and these sentences are concatenated in order and input into a Text-to-Speech (TTS) synthesis engine, such as Google Cloud Text-to-Speech, to generate a synthesized speech segment corresponding to the content of that text segment. This synthesized speech segment contains the original semantic coherence and temporal structure of the text segment. Subsequently, the synthesized speech segment is aligned with the teaching audio track data after speech enhancement and speech activity detection processing using Dynamic Time Warping (DTW). The DTW algorithm calculates the local distance between two audio segments in the Mel spectral domain and finds the path with the minimum cumulative distortion, outputting the time interval in the teaching audio track data where the synthesized speech segment is most likely to match.

[0049] For example, for a text segment with the content "Review the content of the last lesson. Please think about: How does the direction of force affect acceleration? This is a key question.", the synthesized speech length is 5.2 seconds. After DTW alignment, the system determines that the start and end times of this text segment in the teaching video are [140.2 seconds, 147.8 seconds]. This time interval is the correspondence between the text segment and the teaching video data, and all subsequent extraction and matching of audio track prosodic features are based on this time interval.

[0050] Based on this correspondence, the matching degree between text segments and audio track prosodic features is further calculated. This matching degree includes two independent but complementary dimensions, which are used to evaluate the teacher's performance in terms of emotional expression consistency and intonation rhythm coordination. Preferably, in one embodiment of the present invention, the calculation of the matching degree includes: Based on the sentiment enhancement vector of the current text segment, determine the corresponding expected sentiment representation; The prosodic features of the first audio track are compared with the expected emotional representation to determine the first degree of matching; Based on the semantic function type of the current text block, determine the expected prosodic transition relationship between other text blocks adjacent to the current text block; Based on the second audio track prosodic features corresponding to the current text block and other adjacent text blocks, verify whether the expected prosodic transition relationship holds true in order to determine the second matching degree.

[0051] The first audio track prosody refers to the prosodic features of the teacher's speech that is time-aligned with the current text block, used to characterize the teacher's actual speech performance when delivering the current teaching content; the first matching degree is a score that measures whether the emotional enhancement vector of the current text block (representing the "expected emotional representation" of the lesson plan design) is consistent with the first audio track prosodic features, used to determine whether the teacher expressed the appropriate emotional intensity in the teaching segment; the second audio track prosodic features refer to the audio track prosodic features corresponding to other text blocks adjacent to the current text block, used to analyze how the intonation transitions from one teaching segment to another, thereby determining whether the teacher has a reasonable "intonation rhythm"; the second matching degree is a score that measures whether the semantic function type of the current text block (such as "asking a question" or "summarizing") and the intonation change pattern reflected by the second audio track prosodic features of adjacent blocks conform to teaching norms, used to assess whether the teacher used a reasonable intonation rhythm when switching teaching segments, reflecting the fluency and professionalism of the teaching.

[0052] Preferably, in one embodiment of the present invention, comparing the prosodic features of the first audio track with the expected emotional representation to determine a first degree of matching includes: Based on the sentiment intensity level mapped by the sentiment enhancement vector, the corresponding multidimensional matching rules are invoked; The prosodic features of the first audio track are compared with the multidimensional matching rules, and the degree of first matching is determined based on the comparison results.

[0053] Specifically, for the current text segment, the system needs to extract a normalized sentiment intensity index from its sentiment enhancement vector. This index reflects the expected sentiment intensity level during lesson plan design. This index is a scalar value, and its magnitude directly corresponds to a preset sentiment intensity level (e.g., low, medium, and high). In this embodiment, the sentiment enhancement vector is generated from the teaching segment type in the lesson plan: the system has a built-in teaching segment type – sentiment intensity level mapping table. For example, “emphasizing key points” is mapped to “high intensity” (sentiment intensity index ≥ 0.7), “asking questions” or “summarizing” is mapped to “medium intensity” (0.3 ≤ index < 0.7), and “smooth statement” or “transition” is mapped to “low intensity” (index < 0.3). It should be noted that this mapping table can be pre-configured by educational experts based on pedagogical theories. The system calls the multi-dimensional matching rules corresponding to the level of the extracted sentiment intensity index. Each level of emotional intensity is associated with a specific set of acoustic threshold conditions, which are used to constrain the teacher's voice to perform reasonably in three dimensions: absolute value of fundamental frequency change (reflecting intonation fluctuations), logarithmic energy mean (reflecting volume strength), and speech rate sensitivity features (reflecting rhythm speed).

[0054] Then, the system extracts actual observations from the prosodic features of the first audio track in three dimensions: absolute value of fundamental frequency change rate, mean log energy, and speech rate sensitivity features. Then, it sets multi-dimensional matching rules according to the type of teaching segment.

[0055] For example, when the teaching segment type of the text block is "emphasizing key points," the system requires that its first audio track's prosodic features simultaneously meet the following conditions: absolute value of fundamental frequency change ≥ 0.25 Hz / sec (indicating intonation fluctuations), logarithmic energy ≥ mean + 1.0 dB (indicating increased volume for emphasis), and speech rate sensitivity ∈ [0.6, 0.8] (indicating moderate rhythm, neither hurried nor dragging). Alternatively, when the teaching segment type of the text block is "smooth narration," it requires a fundamental frequency change < 0.15 Hz / sec, logarithmic energy deviation from the global mean < 0.5 dB, and speech rate sensitivity > 0.75 (indicating coherent narration without pauses).

[0056] Specifically, the complete mapping relationship of the multidimensional matching rules is shown in Table 2 below: Table 2 Mapping Relationship of Multidimensional Matching Rules Question ≥ 0.3 Within ±0.5 dB ≤ 0.6 Completely satisfied Emphasize key points ≥ 0.25 ≥ +1.0 dB 0.6–0.8 Completely satisfied Summarize ≤ 0.15 Fluctuation standard deviation < 0.5 dB ≥ 0.7 Completely satisfied explain Unrestricted Unrestricted Unrestricted Automatic matching The multidimensional matching rules in Table 2 were determined through a small-scale teaching experiment. Specifically, the system collected 30 demonstration lesson videos and corresponding lesson plans presented by provincial-level or higher renowned teachers. Three senior teaching and research staff manually annotated each teaching segment of each lesson, including: the type of teaching segment (such as "questioning" or "summarizing"); and whether the segment was "expressed appropriately" (i.e., whether the pronunciation conformed to teaching standards).

[0057] Simultaneously, prosodic features (fundamental frequency change rate, logarithmic energy, and speech rate sensitivity features) of the audio track are extracted for each lesson. The prosodic features of the first audio track corresponding to the current text block are compared item by item with the threshold conditions in the invoked multidimensional matching rules. If all conditions are met, the speech performance is determined to be consistent with the emotional expectation, and the first matching degree is recorded as 1; otherwise, it is recorded as 0. In this embodiment, the emotional enhancement vector, as a digital carrier of the lesson plan intent, drives the dynamic selection of matching rules, while the comparison process ensures that the evaluation results are based on objective acoustic evidence, thereby achieving an accurate diagnosis of the teacher's "appropriateness of emotional expression".

[0058] Preferably, in one embodiment of the present invention, verifying whether the expected prosodic transition relationship holds true based on the second audio track prosodic features corresponding to the current text block and other adjacent text blocks, in order to determine the second matching degree, includes: Based on the semantic function type of the current text block, invoke the corresponding prosodic transition rule; The prosodic features of the second audio track are compared with the prosodic transition rules, and the second matching degree is determined based on the comparison results.

[0059] In this embodiment, the second matching degree is used to measure the coordination of the teacher's speech rhythm during the switching between different teaching stages. It quantifies the expected relationship between the speech function type of a text block and its adjacent speech segments in terms of acoustic performance by introducing prosodic transition rules. The prosodic transition rules and the multi-dimensional matching rules both belong to the teaching speech consistency rule system, constraining the teacher's speech behavior from the perspectives of local performance and cross-segment coordination, respectively. Based on the semantic function type of the current text block (such as questioning, emphasis, summarizing, etc.), the prosodic transition rules predetermine the expected prosodic transition relationship between the current text block and its adjacent text blocks in terms of speech rhythm. Specifically, this is reflected in the joint constraint relationship of several acoustic parameters, typically including: whether there should be a non-speech pause of a certain duration at the end of the current block; whether the speech rate sensitivity features of the current block and its adjacent blocks should show an upward or downward trend; and whether the mean change in the fundamental frequency of both should be controlled within a specific range.

[0060] For example, question-based questions aim to stimulate student thinking, thus requiring teachers to provide "cognitive blanks" after the questions. Technically, this is manifested by: slowing down the speaking rate (current speaking rate sensitivity feature ≤ next speaking rate − 0.15), inserting pauses (non-speech segments ≥ 0.8 seconds), or maintaining a steady tone (|ΔF0| ≤ 3 Hz); emphasis-based questions aim to highlight core information, thus requiring "salience" in speech performance, technically manifested by: significantly increased fundamental frequency (ΔF0 ≥ +2 Hz), increased energy (logarithmic energy difference ≥ +1.0 dB), and slightly decreased speaking rate (current speaking rate ≤ previous speaking rate − 0.1); summarizing-based questions aim to consolidate the knowledge structure, thus requiring speech to tend towards "convergence and stability," technically manifested by: continuously decreasing speaking rate (current speaking rate ≤ previous speaking rate − 0.15), reduced fundamental frequency fluctuations (|ΔF0| ≤ 2 Hz), and stable energy.

[0061] Although the various prosodic transition rules have different parameter combinations, they all originate from the same design logic: transforming the cognitive purpose of teaching behavior into quantifiable cross-block acoustic constraints. During system operation, the system automatically invokes the corresponding rule based on the identified semantic function type and substitutes the extracted second-track prosodic features for verification. If the conditions specified in the rule are met, the transition is deemed harmonious, and the second matching degree is recorded as 1; otherwise, it is recorded as 0. All rules can be calibrated through a combination of educational expert summarization and statistical analysis of high-quality classroom corpora.

[0062] After obtaining the matching degree of each text block, the system further uses the semantic density distribution to weight and adjust the contribution of the matching degree of each text block to ensure that the evaluation structure is more focused on the core area of ​​the teaching content. Specifically, in this embodiment, for each text block, the system has obtained its first matching degree and second matching degree. First, it takes the arithmetic mean of the first matching degree and the second matching degree to obtain the comprehensive matching score of the block; then, it multiplies the comprehensive matching score by its corresponding semantic density value to obtain the weighted contribution value of the block; then, it sums the weighted contribution values ​​of all text blocks and divides them by the sum of the semantic density values ​​of all text blocks to finally obtain a normalized comprehensive evaluation score. The score ranges from 0 to 1. The higher the value, the more consistent the teacher's speech performance is with the lesson plan design intent in high-information-density teaching content.

[0063] After obtaining the comprehensive assessment score, the system converts the score into a percentage score from 0 to 100, which serves as the diagnostic assessment result for the target teacher's competence. For example, if the normalized score is 0.87, the output will be 87. Because the score automatically reduces the weight of transitional statements with low semantic density (such as "Let's look at...") during the calculation process, it can more accurately reflect the teacher's professional competence in teaching core knowledge points.

[0064] In another embodiment of the invention, the emotion enhancement vector can also be obtained through real-time analysis of students' facial expressions in teaching video data collected by a camera in front of the recording classroom, thereby reflecting the students' genuine emotional feedback during the current teaching session. This embodiment is applicable to scenarios where teaching evaluation needs to be centered on student emotional feedback.

[0065] Specifically, the system separates the video stream containing the student area from the teaching video data. Since teachers and students are located in different physical areas during actual lessons, the system extracts the image region containing all students using pre-defined spatial coordinate ranges or region segmentation methods based on human pose estimation, such as OpenPose or HRNet. Then, it employs face detection algorithms, such as MTCNN or YOLOv5-face, to locate the bounding boxes of all visible student faces in each video frame and performs expression recognition on each valid face. The expression recognition can utilize a deep learning expression classification model, such as a ResNet or VGG architecture pre-trained on standard datasets like AffectNet and FER2013, fine-tuned for the classroom scenario before deployment. This model outputs the probability distribution of each face across seven basic emotions: happiness, sadness, surprise, anger, fear, disgust, and neutrality.

[0066] To obtain the overall emotional state of the group, the system aggregates all detected student faces within the same time window (e.g., 0.5 seconds, aligned with the temporal granularity of the audio track's rhythm features): first, low-quality detection results with a confidence level below 0.6 are removed; then, the average probability of each emotion is calculated; finally, the emotion category with the highest probability is selected as the dominant emotion for that time window. Furthermore, the obtained dominant emotion is mapped to a connected emotional intensity index to construct an emotional enhancement vector. The specific mapping rules are as follows: if the dominant emotion is "happy" or "surprised," it indicates active student engagement, and the emotional intensity index is set to 0.8; if it is "neutral," it indicates stable attention, and is set to 0.5; if it is a negative emotion such as "sadness" or "anger," it is considered low engagement or confusion, and is set to 0.3.

[0067] The aforementioned emotion mapping rules are based on research in educational psychology regarding students' classroom emotions and cognitive engagement, and are derived from annotation using small-scale empirical teaching data. Specifically, referencing the Control-Value Theory of Academic Emotions proposed by Pekrun et al., this theory states that positive high-arousal emotions (such as happiness and surprise) are generally positively correlated with high attentional engagement and deep cognitive processing; neutral emotions correspond to normal information reception; while negative emotions (such as sadness and anger) are often accompanied by cognitive resource inhibition or decreased learning motivation. Based on this theory, the system initially set three emotion intensity ranges: high (0.7–0.9), medium (0.4–0.6), and low (0.2–0.4). Subsequently, by collecting 20 junior high school mathematics class videos, simultaneously recording students' facial expressions, teachers' teaching content, and immediate post-class test scores, the average accuracy rate of students' answers under different dominant emotions was analyzed. According to the statistical results, the average accuracy rate of students' answers was 82% during the "happy" or "surprised" period; 65% during the "neutral" period; and 48% during the "sad" or "angry" period. Based on the statistical results, the accuracy rate was mapped to the interval [0.3, 0.8] using a linear normalization method, and the above sentiment mapping rule was finally determined.

[0068] Furthermore, after obtaining a continuous emotional intensity index based on the aforementioned emotional mapping rules, the time interval corresponding to each text block in the teaching video is determined using the spatiotemporal alignment results. For all overlapping 0.5-second video windows within this time interval (whose temporal granularity is consistent with the audio track's prosodic features), the emotional intensity index corresponding to the dominant emotion of the student group calculated for each window is extracted. Then, these emotional intensity indices are arithmetically averaged to obtain a comprehensive emotional intensity value representing the overall student emotional feedback of the current text block. Finally, this comprehensive emotional intensity value is expanded into a 768-dimensional constant vector to obtain the emotional enhancement vector of the text block.

[0069] In this embodiment, the emotional enhancement vector generated based on students' real-time emotional feedback is still used as the input basis for calculating the matching degree. The comprehensive emotional intensity value corresponding to the emotional enhancement vector of the current text block is compared with the prosodic features of the first audio track aligned with the block to determine whether the teacher's actual voice performance is consistent with the emotional state of the student group during the teaching period.

[0070] For example, if the student's overall emotional intensity value for the current text segment is 0.8 (indicating high student engagement), but the teacher's fundamental frequency variation rate during this period is below 0.15 Hz / second and their speech rate sensitivity feature is above 0.8 (indicating flat, rapid narration), then the first matching degree will be determined as 0.0, reflecting the teacher's failure to adjust their expression based on the student's positive feedback. Conversely, if the teacher simultaneously increases their intonation and vocal intensity, the matching degree will be 1.0, demonstrating good classroom awareness and responsiveness. Subsequently, the system combines the semantic density value of the text segment to weight the matching degree and aggregates the weighted results of all text segments to generate a normalized comprehensive evaluation score, ultimately converting it into a percentage-based ability diagnostic evaluation result from 0 to 100.

[0071] In this embodiment, students' genuine emotional feedback serves as the basis for emotional enhancement, shifting the teacher competence assessment from a lesson plan-centered approach to a student-centered one. The assessment results not only measure the teacher's execution of the pre-designed lesson plan but also effectively reflect the teacher's professional ability to perceive students' states in real classroom scenarios and dynamically adjust the teaching pace. Choosing the emotional signals carried by students' facial expressions as the feedback source is crucial. From a teaching practice perspective, students' expressions are the most direct window into their internal cognitive state: positive and enthusiastic emotions are usually accompanied by deep processing and high participation, while persistent negative emotions indicate learning obstacles or a disconnect from teaching. Therefore, incorporating students' emotions into the assessment system essentially transforms the genuine feedback of "how well they learned" into a diagnostic basis for "how well they taught." Simultaneously, using conventional cameras deployed in front of the recording classroom, the system can capture the emotional dynamics of the entire class in real time without interfering with normal teaching and without requiring additional equipment, through mature face detection and expression classification methods. This non-intrusive perception method respects the original teaching state of the classroom while providing another perspective for teacher competence assessment.

[0072] In another embodiment of the present invention, the emotion enhancement vector is constructed from two parts of information: one part comes from the teaching design intent reflected in the pre-set teaching segment tags in the lesson plan, and the other part comes from the real-time emotional feedback reflected in the analysis of students' facial expressions in the teaching video. By weighted fusion of the two, a hybrid emotion enhancement vector that takes into account both "teaching" and "learning" is generated, which is suitable for scenarios of high-level diagnosis of teachers' comprehensive teaching abilities.

[0073] Specifically, for each text segment, two sentiment intensity indicators need to be obtained separately: (1) The emotional intensity index obtained by looking up the teaching link labels based on the lesson plan is called the lesson plan-driven emotional intensity index. (2) The emotional intensity index obtained by analyzing and calculating students’ facial expressions is called the student feedback-driven emotional intensity index.

[0074] A fused emotional intensity index is obtained by weighted averaging of the two emotional intensity indicators. Specifically, this weighted averaging process introduces a configurable weight coefficient to adjust the relative importance of lesson plan intent and student feedback. In the default configuration, the two indicators are assigned equal weights, meaning the fused emotional intensity index equals the arithmetic mean of the lesson plan-driven indicator and the student feedback-driven indicator. In other configurations, if the evaluation objective focuses more on the teacher's execution of the lesson plan, the weight of the lesson plan-driven indicator is increased; if it focuses more on the teacher's responsiveness to student states, the weight of the student feedback-driven indicator is increased. The fused emotional intensity index is then expanded into a 768-dimensional constant vector, serving as the hybrid emotional enhancement vector for the current text segment. Based on this, the subsequent calculation of the matching degree still follows multi-dimensional matching rules: according to the intensity range (e.g., high, medium, low) of the fused emotional intensity index, the threshold requirements of the corresponding fundamental frequency change rate, logarithmic energy, and speech rate sensitivity features are invoked to determine whether the teacher's actual speech performance is consistent with the fused expectation.

[0075] For example, a text segment might correspond to a high level of lesson plan-driven emotional intensity, but student facial expression analysis indicates a "neutral" group emotion, corresponding to a medium level of student feedback-driven emotional intensity. After equal-weighted fusion, the overall emotional intensity of this segment falls into the medium emphasis range. In this case, if the teacher's voice exhibits moderate intonation and a slightly higher volume, the match will be successful. However, if the teacher still uses a high-intensity expression style (such as high volume and rapid speech), the match may fail because it is inconsistent with the expected fusion result, reflecting a failure to dynamically adjust the teaching pace according to the students' actual comprehension.

[0076] Finally, the system also combines the semantic density value of the text block to weight the matching degree, and aggregates the weighted results of all text blocks to generate a normalized comprehensive evaluation score, which is ultimately converted into a capability diagnostic evaluation result on a scale of 0 to 100. In this embodiment, the final capability diagnostic evaluation result is no longer limited to a single-dimensional judgment, but integrates the dual signals of "instructional design intent" and "real classroom learning situation," comprehensively reflecting the teacher's two core professional competencies: on the one hand, it reflects their systematic grasp of course objectives and knowledge structure; on the other hand, it demonstrates their ability to perceive and dynamically adjust to students' states. Compared with a single feedback mode, this hybrid strategy can avoid the mechanical execution bias caused by lesson plan-centered teaching, such as teachers reading from the textbook while ignoring students' states, and it can also prevent misjudgments that may be caused by student-centered teaching, such as students being excited because the learning content is simple, but the teaching depth is insufficient. Through a configurable weighting mechanism, the system can also adapt to different assessment objectives in a zero-sum manner. For example, in new teacher training, the focus is on cultivating responsiveness, and the weight is tilted towards student feedback; in the evaluation of high-quality courses, the emphasis is on the rigor of instructional design, and the weight of lesson plan-driven assessment is increased.

[0077] Another embodiment of the present invention provides a teacher competence diagnostic and assessment system based on multimodal data fusion. For details, please refer to [link to relevant documentation]. Figure 3 , Figure 3 The system shown is a teacher competence diagnostic and assessment system based on multimodal data fusion, according to one embodiment of the present invention. The system includes: Data acquisition module 11 is used to acquire the data to be evaluated, which includes at least teaching video data and corresponding lesson plan text data. The data segmentation module 12 is used to segment the lesson plan text data to obtain various text blocks. The dependency relationships between the various text blocks reflect the semantic density distribution of the lesson plan text data. Vector enhancement module 13 is used to fuse the corresponding sentiment enhancement vector into each text block; Feature extraction module 14 is used to extract prosodic features of each audio track from the teaching video data at least; Alignment module 15 is used to determine the correspondence between each text block and the teaching video data based on spatiotemporal alignment technology; The feature matching module 16 is used to obtain the matching degree between all text blocks and all audio track prosodic features based on the correspondence relationship. The matching degree includes the first matching degree between the sentiment enhancement vector of the current text block and the corresponding first audio track prosodic feature, and the second matching degree between the semantic function of the current text block and the intonation fluctuation information. The intonation fluctuation information is determined by the second audio track prosodic features corresponding to other text blocks adjacent to the current text block. The diagnostic assessment module 17 is used to adjust the contribution weight of the matching degree using the semantic density distribution, and to determine the diagnostic assessment result of the target teacher's ability corresponding to the data to be assessed, based at least on the adjusted matching degree.

[0078] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following: (1) Existing technologies typically evaluate teachers based solely on isolated acoustic features of their speech (such as speech rate and volume), which are insufficient to reflect the actual intent of their teaching behavior. This invention introduces the semantic structure of the lesson plan and emotion enhancement vectors to associate speech features with specific teaching steps, enabling the evaluation results to determine whether the teacher's speech performance matches their teaching intent, thereby improving the educational relevance and practicality of the evaluation results.

[0079] (2) In actual classrooms, lesson plans often contain a large number of transitional statements (such as "Let's look at..."), which have low information density. If they are treated the same as the core knowledge points, it is easy to lead to assessment bias. This invention analyzes the semantic density distribution and weights the matching degree of different text blocks, automatically reducing the weight of low-density areas, so that the assessment focuses more on the teaching quality of key teaching content.

[0080] (3) Traditional methods only focus on the speech intensity of a single teaching segment, while ignoring the connection between teaching segments. This invention further analyzes whether the rhythmic transition of adjacent text blocks (such as "question" followed by "explanation") conforms to teaching routines in terms of intonation, speech rate and pauses, thereby capturing the teacher's performance in classroom rhythm control and interactive guidance, and making up for the shortcomings of the existing assessment system in cross-segment coordination.

[0081] (4) This invention supports multi-source construction of emotion enhancement vectors: it can be based on the teaching segment tags preset in the lesson plan, or it can integrate the real-time analysis results of students' facial expressions in the recorded video. Therefore, the system can not only evaluate the teacher's implementation of the instructional design, but also reflect the teacher's ability to dynamically adjust teaching strategies based on student feedback, and is suitable for various scenarios such as formative assessment, teaching research reflection, and teaching assistance.

[0082] (5) This invention only requires conventional recorded video lessons (including audio and video) and electronic lesson plans as input, without the need for additional sensors, wearable devices, or manual annotation, to complete fine-grained multi-dimensional ability diagnosis. This design lowers the system deployment threshold and is suitable for promotion and application in real teaching environments such as daily teaching and research in schools, teacher training for students, and online teacher training.

[0083] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A teacher competence diagnosis and assessment method based on multimodal data fusion, characterized in that, include: Obtain the data to be evaluated, which includes teaching video data and corresponding lesson plan text data; The lesson plan text data is segmented to obtain various text blocks, wherein the dependency relationships between the various text blocks reflect the semantic density distribution of the lesson plan text data. The corresponding sentiment enhancement vectors are fused into each of the aforementioned text blocks; Extract the prosodic features of each audio track from the teaching video data; Based on spatiotemporal alignment technology, the correspondence between each of the text blocks and the teaching video data is determined. Based on the correspondence, the matching degree between all the text blocks and all the audio track prosodic features is obtained. The matching degree includes the first matching degree between the sentiment enhancement vector of the current text block and the corresponding first audio track prosodic feature, and the second matching degree between the semantic function of the current text block and the intonation fluctuation information. The intonation fluctuation information is determined by the second audio track prosodic features corresponding to other text blocks adjacent to the current text block. The contribution weight of the matching degree is adjusted using the semantic density distribution, and the ability diagnostic assessment result of the target teacher corresponding to the data to be assessed is determined based on the adjusted matching degree.

2. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 1, characterized in that, The process of segmenting the lesson plan text data to obtain various text blocks includes: The lesson plan text data is segmented to obtain multiple candidate text units; Based on the semantic association strength between the candidate text units, an initial semantic network graph is constructed; Calculate the semantic density value of each local subgraph in the initial semantic network graph to characterize the semantic density distribution in different regions of the lesson plan text data; Based on the semantic density distribution, the candidate text units are processed to obtain each text block.

3. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 2, characterized in that, The process of processing the candidate text units based on the semantic density distribution to obtain each text block includes: Based on the semantic density distribution, determine the global average semantic density; Candidate text units with semantic density values ​​higher than the global average semantic density are segmented, and candidate text units with semantic density values ​​lower than the global average semantic density are merged. The results of the segmentation and merging processes are output as the text blocks.

4. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 3, characterized in that, The segmentation process for candidate text units whose semantic density values ​​are higher than the global average semantic density includes: Identify teaching action verbs in the candidate text units, and determine the corresponding teaching action type based on the teaching action verbs; If the candidate text unit includes at least two different teaching behavior types, then the text is segmented at adjacent and different teaching behavior types to obtain multiple text blocks; The merging process for candidate text units whose semantic density value is lower than the global average semantic density includes: If adjacent candidate text units have the same teaching behavior type and their semantic density values ​​are all lower than the global average semantic density, the candidate text units are merged to obtain the text block.

5. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 1, characterized in that, The extraction of prosodic features from the teaching video data for each audio track includes: The teaching audio track data is separated from the teaching video data, and the teaching audio track data is processed by speech enhancement and speech activity detection to retain valid speech segments; The effective speech segments are subjected to overlapping frame segmentation to obtain a time-series audio sequence; Extract acoustic features related to the prosodic performance of the target teacher during the lesson from each audio frame; The acoustic features of adjacent audio frames are aggregated according to a preset time window to generate the audio track prosodic features.

6. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 5, characterized in that, The extraction of acoustic features related to the prosodic performance of the target teacher during instruction from each audio frame includes: Perform time-domain autocorrelation analysis and frequency-domain cepstral analysis on the audio frame, and determine the fundamental frequency F0 based on the analysis results; Calculate the signal energy of the audio frame, and obtain the logarithmic energy based on the signal energy; Based on the results of the speech activity detection and processing, speech rate-sensitive features related to the speech temporal distribution are generated.

7. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 1, characterized in that, The calculation of the matching degree includes: Based on the sentiment enhancement vector of the current text segment, determine the corresponding expected sentiment representation; The first audio track's prosodic features are compared with the expected emotional representation to determine the first degree of matching; Based on the type of semantic function of the current text block, determine the expected prosodic transition relationship between the other text blocks adjacent to the current text block; Based on the second audio track prosodic features corresponding to the current text block and the adjacent other text blocks, verify whether the expected prosodic transition relationship is valid, so as to determine the second matching degree.

8. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 7, characterized in that, The step of comparing the prosodic features of the first audio track with the expected emotional representation to determine the first matching degree includes: Based on the emotional intensity level mapped by the emotional enhancement vector, the corresponding multidimensional matching rule is invoked; The first audio track prosodic features are compared with the multidimensional matching rules, and the first matching degree is determined based on the comparison results.

9. The teacher competence diagnosis and assessment method based on multimodal data fusion as described in claim 7, characterized in that, The step of verifying whether the expected prosodic transition relationship holds true based on the second audio track prosodic features corresponding to the current text block and the adjacent other text blocks, in order to determine the second matching degree, includes: Based on the type of semantic function of the current text segment, the corresponding prosodic transition rule is invoked; The second audio track prosodic features are compared with the prosodic transition rules, and the second matching degree is determined based on the comparison results.

10. A teacher competence diagnostic and assessment system based on multimodal data fusion, characterized in that, include: The data acquisition module is used to acquire the data to be evaluated, which includes teaching video data and corresponding lesson plan text data. The data segmentation module is used to segment the lesson plan text data to obtain various text blocks, wherein the dependency relationships between the various text blocks reflect the semantic density distribution of the lesson plan text data. The vector enhancement module is used to fuse the corresponding sentiment enhancement vector into each of the text blocks. The feature extraction module is used to extract the prosodic features of each audio track from the teaching video data; The alignment module is used to determine the correspondence between each text block and the teaching video data based on spatiotemporal alignment technology; The feature matching module is used to obtain the matching degree between all the text blocks and all the audio track prosodic features based on the correspondence. The matching degree includes a first matching degree between the sentiment enhancement vector of the current text block and the corresponding first audio track prosodic feature, and a second matching degree between the semantic function of the current text block and the intonation fluctuation information. The intonation fluctuation information is determined by the second audio track prosodic features corresponding to other text blocks adjacent to the current text block. The diagnostic assessment module is used to adjust the contribution weight of the matching degree using the semantic density distribution, and based on the adjusted matching degree, to determine the diagnostic assessment result of the target teacher's ability corresponding to the data to be assessed.

Images