Microteaching video and voice intelligent evaluation method and system

Abstract

The invention relates to the field of computer video and voice recognition and analysis, and discloses a micro teaching video and voice intelligent evaluation method and system, and the method specifically comprises the following steps: collecting video and audio data in a micro teaching process; performing voice recognition, rhythm recognition and emotion recognition on the audio data, and extracting audio features; performing finger gesture recognition, facial expression recognition, posture recognition and head gesture recognition on the video data, and extracting video features; constructing a micro teaching evaluation model based on a pre-normalized modal feature fusion neural network; and based on a decision tree improved algorithm, analyzing the video features and the audio features by using the micro teaching evaluation model, and generating a teaching evaluation result. According to the method, the problem that automatic evaluation cannot be carried out by comprehensively considering multi-modal characteristics in the prior art is solved, and the method has the characteristic that micro teaching evaluation can be standardized and systematized.

Description

Technical Field

[0001] The present invention relates to the field of computer video and speech recognition and analysis, and more specifically, to a method and system for evaluating video and speech intelligence in micro-teaching. Background Art

[0002] In recent years, microteaching competitions and microteaching applications have become widespread, but research on systematic evaluation indicators for microteaching has yet to be seen. Furthermore, the quality of existing microteaching programs varies greatly. Effective microteaching evaluation plays a crucial role in monitoring the development of microteaching. Establishing a scientific and rational microteaching evaluation system can provide a reliable reference standard for improving the quality of microteaching resources.

[0003] For decades, the implementation model of microteaching has remained a combination of theory, practice, and teacher evaluation. This evaluation process is time-consuming, labor-intensive, and highly subjective. Furthermore, due to the highly abstract and rigorous logic inherent in mathematics, microteaching evaluation in mathematics requires a higher level of professionalism. Therefore, in-depth research on how technology can empower educational evaluation and promote the construction of a new intelligent educational evaluation system is urgently needed. Artificial intelligence, as a key technology in the digital transformation of education, provides new tools and approaches for the development of teacher education programs in universities. With the development of artificial intelligence, how to use it to reform and upgrade microteaching evaluation systems has become a key research direction for universities in promoting teacher training.

[0004] The existing technology has an intelligent assessment and portrait visualization solution for the basic teaching skills of normal school students. This solution is based on image recognition, body detection, language recognition, text mining, emotion recognition and other technologies, and uses data visualization to display the characteristics and level of teachers' abilities in multiple dimensions.

[0005] However, existing technologies have limited evaluation capabilities and are unable to comprehensively consider multimodal features for automatic evaluation. Therefore, how to invent an evaluation method for video and voice intelligence suitable for micro-teaching is a technical problem that urgently needs to be solved in this technical field. Summary of the Invention

[0006] In order to solve the problem that the existing technology cannot comprehensively consider multimodal features for automatic evaluation, the present invention provides a video and voice intelligent evaluation method and system for micro-teaching, which has the characteristics of being able to standardize and systematize micro-teaching evaluation.

[0007] In order to achieve the above-mentioned purpose of the present invention, the technical solutions adopted are as follows:

[0008] A method for evaluating video and voice intelligence in micro-teaching, comprising the following steps:

[0009] Collect video and audio data during micro-teaching;

[0010] Perform speech recognition, rhythm recognition, and emotion recognition on audio data, and extract audio features;

[0011] Perform finger gesture recognition, facial expression recognition, posture recognition, and head posture recognition on video data to extract video features;

[0012] Construct a microteaching evaluation model based on pre-normalized modal feature fusion neural network;

[0013] Based on the decision tree improved algorithm, the micro-teaching evaluation model is used to analyze the video features and audio features to generate teaching evaluation results.

[0014] Preferably, the video and audio data of the micro-teaching process are collected, and the specific steps are: synchronously collecting the teaching video and audio data through the recording and broadcasting equipment, performing key frame extraction and posture annotation on the video data, and performing noise reduction and time domain segmentation processing on the audio data.

[0015] Furthermore, the audio features specifically include rhythm, loudness, speaking speed, positivity, and voice standardization.

[0016] Furthermore, we can perform speech recognition, rhythm recognition, and emotion recognition on the audio data and extract audio features. The specific steps are as follows:

[0017] Extract low-level acoustic descriptors, set window setting parameters, and calculate speech rate, pause interval, and syllable density features. Use piecewise statistical functions to cluster rhythm patterns of set speech segments, build a classroom rhythm classification model based on a decision tree, and output rhythm scores and silence duration ratios.

[0018] The human ear simulation algorithm is used to calculate the loudness mean and standard deviation, and the improved pitch detection algorithm is used to extract the fundamental frequency mean, standard deviation and global average value of the dynamic range; the cepstrum analysis technique is used to calculate the kurtosis prominence of the periodic signal energy in the cepstrum domain;

[0019] A dual-stream deep learning model was constructed. The first branch inputs several orders of Mel spectrograms, and the second branch inputs manually designed rhythmic feature vectors. The relational knowledge distillation method was used to fuse the deep features and manual features, and finally output the emotional neutrality, positivity, and negativity.

[0020] An acoustic model is constructed based on the existing classroom hot word library, and an improved DNN-HMM hybrid architecture is used for speech recognition and decoding; the pronunciation standardization index is calculated using the phoneme-level Goodness of Pronunciation algorithm.

[0021] Furthermore, the speech recognition model, emotion recognition model, and rhythm recognition model are all recognition models without reference text. They directly process the audio of teaching scenes through speech recognition technology with hot words without relying on classroom text transcription files.

[0022] Furthermore, the video features specifically include gesture ratio, proportion of positive, negative and neutral expressions, head movement, and eye direction.

[0023] Furthermore, we can perform finger gesture recognition, facial expression recognition, posture recognition, and head posture recognition on the video data to extract video features. The specific steps are as follows:

[0024] A gesture classifier is used to detect the hand area, and the gesture type is identified through edge detection and finger key point algorithm. The symbolic gesture ratio, digital gesture ratio and thumb extension ratio are calculated.

[0025] The facial landmark detector is used to locate the key points of the eyes, nose, and mouth. The ratio of the distance between the left eye and the right eye and the distance between the left eye and the nose is calculated as the eye width ratio. The ratio of the distance between the mouth and the nose and the distance between the left eye and the nose is calculated as the mouth height ratio. The expression types corresponding to different ratios are defined. The proportion of positive, negative, and neutral expressions is calculated.

[0026] Obtain skeleton key points, calculate the angles of each limb using the vector cosine formula, count the open posture ratio, closed posture ratio, and contradictory posture ratio, and calculate the time series variance related to posture stability;

[0027] Based on the facial key point tracking algorithm, the head movement trajectory is analyzed through continuous frames, the number of nods and shakes per unit time is counted, and the maximum, minimum and average movement amplitudes are calculated by combining the three-axis acceleration data;

[0028] The improved PnP algorithm combined with the FSA-Net lightweight model is used to extract the Pitch / Yaw / Roll Euler angles from continuous frames. The maximum, mean and range characteristics of their time series are calculated, and the line of sight focus area is estimated using the iris positioning algorithm.

[0029] Furthermore, the microteaching evaluation model specifically includes two single-modality enhancement modules and one multi-modality combination module;

[0030] The two unimodal enhancement modules are used to enhance video features and audio features respectively; the unimodal enhancement module includes a cascaded first-layer normalization block, an attention mechanism block, a second-layer normalization block, an activation block, a third-layer normalization block, a feedforward network block, and a fourth-layer normalization block; the first-layer normalization block is also connected to its unimodal input residual; the activation block is also connected to the output residual of the attention mechanism block; the feedforward network block is also connected to the output residual of the fourth-layer normalization block;

[0031] The multimodal combination module is used to perform feature fusion on the output vectors of the single-modal modules and assign weights; the multimodal combination module includes a first-layer normalization block, a second-layer normalization block, a cross-attention block, a third-layer normalization block, an activation block, a fourth-layer normalization block, a feedforward network block, and a fifth-layer normalization block; the outputs of the fourth-layer normalization blocks of the two single-modal enhancement modules are respectively connected to the first-layer normalization block and the second-layer normalization block of the multimodal combination module by residual connection; the outputs of the first-layer normalization block and the second-layer normalization block are respectively connected to the cross-attention block; the cross-attention block, the third-layer normalization block, the activation block, the fourth-layer normalization block, the feedforward network block, and the fifth-layer normalization block are cascaded; the activation block is also connected to the output residual of the cross-attention block; and the feedforward network block is also connected to the output residual of the fifth-layer normalization block.

[0032] Furthermore, based on the decision tree improved algorithm, the micro-teaching evaluation model is used to analyze the video features and audio features to generate teaching evaluation results. The specific steps are as follows:

[0033] The existing micro-teaching dataset is used as the training set. The attributes with the highest information gain ratio are selected from the training set as split nodes to construct the decision tree trunk. The splitting is stopped when the purity of the node sample reaches the preset threshold or the partitioning ability is lower than the truth index.

[0034] A microteaching evaluation model is constructed at the leaf nodes of the decision tree: at each leaf node, a pre-normalized attention mechanism is used to input the corresponding video and audio features into the corresponding single-modal enhancement module for enhancement processing and modality alignment;

[0035] At each leaf node, the enhanced two single-modal features are input into the multimodal combination module for interaction. A joint representation vector is formed through feature multiplication and weighted concatenation to generate the local evaluation of each leaf node.

[0036] The local evaluation of each leaf node neural network is integrated to output the overall teaching evaluation results.

[0037] An evaluation system for video and voice intelligence of micro-teaching, including a data acquisition module, a voice recognition module, a video recognition module, and a comprehensive evaluation module;

[0038] The data acquisition module is used to collect video and audio data during the micro-teaching process;

[0039] The speech recognition module is used to perform speech recognition, rhythm recognition, emotion recognition on audio data and extract audio features;

[0040] The video recognition module is used to perform finger gesture recognition, facial expression recognition, posture recognition and head posture recognition on video data and extract video features;

[0041] The comprehensive evaluation module is used to construct a micro-teaching evaluation model based on a pre-normalized modal feature fusion neural network, and based on a decision tree improved algorithm, uses the micro-teaching evaluation model to analyze the video features and audio features to generate teaching evaluation results.

[0042] The beneficial effects of the present invention are as follows:

[0043] This invention discloses a method for evaluating video and voice intelligence in microteaching. It integrates audio and video recognition, and uses the identified features to output evaluation results through a microteaching evaluation model based on a pre-normalized modal feature fusion neural network. This system constructs a standardized and systematic microteaching evaluation system. Through comparative analysis, it identifies the characteristics of high-quality microteaching videos and provides effective recommendations to teacher trainees. Teacher trainees can quickly identify areas for optimization based on their own video and spoken language performance, thereby promoting their professional development. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a method for evaluating video and voice intelligence in micro-teaching according to the present invention.

[0045] Figure 2 This is a schematic diagram of a specific process of a method for evaluating video and voice intelligence in micro-teaching according to Example 2 of the present invention.

[0046] Figure 3 It is a schematic diagram of the unimodal feature enhancement module.

[0047] Figure 4 It is a schematic diagram of the multimodal feature interaction module. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] Example 1

[0050] like Figure 1 As shown, a method for evaluating video and voice intelligence in micro-teaching includes the following steps:

[0051] Collect video and audio data during micro-teaching;

[0052] Perform speech recognition, rhythm recognition, and emotion recognition on audio data, and extract audio features;

[0053] Perform finger gesture recognition, facial expression recognition, posture recognition, and head posture recognition on video data to extract video features;

[0054] Construct a microteaching evaluation model based on pre-normalized modal feature fusion neural network;

[0055] Based on the decision tree improved algorithm, the micro-teaching evaluation model is used to analyze the video features and audio features to generate teaching evaluation results.

[0056] Example 2

[0057] like Figure 2 As shown, in a specific embodiment, video and audio data are collected during micro-teaching, and the specific steps are: synchronously collect teaching video and audio data through recording and broadcasting equipment, extract key frames and annotate postures of video data, and perform noise reduction and time domain segmentation processing on audio data.

[0058] In a specific embodiment, the audio features include rhythm, loudness, speaking speed, positivity, and voice standardization.

[0059] In a specific embodiment, as shown in Table 1, speech recognition, rhythm recognition, and emotion recognition are performed on audio data to extract audio features. The specific steps are as follows:

[0060] Based on the OpenSMILE toolkit, we extracted acoustic low-level descriptors (LLDs) and calculated speech rate, pause intervals, and syllable density features using a 25ms frame length and 10ms frame shift window. We clustered rhythmic patterns in 100ms speech segments using a piecewise statistical function and constructed a classroom rhythm classification model based on a decision tree. The output rhythm score includes the proportion of rhythmic audio segments, the proportion of silence duration, and the proportion of effective silence duration.

[0061] The ITU-R BS.1770-4 recommended algorithm is used to calculate the loudness mean and standard deviation. The improved pitch detection algorithm implemented in the PRAAT toolkit is used to extract the fundamental frequency mean, standard deviation, and global average of the dynamic range. Cepstrum analysis is used to calculate the kurtosis prominence (CPP) mean of the periodic signal energy in the cepstrum domain. The improved YIN algorithm is used to extract the fundamental frequency feature to eliminate the frequency multiplication error.

[0062] A two-stream deep learning model based on the VGGish network was constructed. The first branch inputs a 128-order Mel spectrogram generated by Hamming window framing and short-time Fourier transform. The second branch inputs a manually designed prosodic feature vector containing fundamental frequency standard deviation, speech rate variation coefficient, etc. The deep features are fused with the manual features using the relational knowledge distillation (RKD) method. The final output is the emotional neutrality, positivity, and negativity, corresponding to the duration of discrete emotional audio clips.

[0063] An acoustic model was constructed based on the popular vocabulary database of Chinese mathematics classroom, and an improved DNN-HMM hybrid architecture was used for speech recognition and decoding. The goodness of pronunciation index was calculated using the phoneme-level Goodness of Pronunciation algorithm: the phoneme posterior probability P(q|O)=P(O|q) / Σq'P(O|q') was defined, where O is the acoustic feature vector and q is the phoneme state sequence. The final calculated standard mean is the weighted average of the GoP-NN scores of all phonemes.

[0064] Table 1

[0065]

[0066] In a specific embodiment, the speech recognition model, emotion recognition model, and rhythm recognition model are all recognition models without reference text, which directly process the teaching scene audio through speech recognition technology with hot words without relying on classroom text transcription files.

[0067] In a specific embodiment, the video features include gesture ratio, proportion of positive, negative and neutral expressions, head movement, and eye direction.

[0068] In a specific embodiment, as shown in Table 2, finger gesture recognition, facial expression recognition, posture recognition, and head gesture recognition are performed on video data to extract video features. The specific steps are as follows:

[0069] A Haar cascade classifier is used to detect the hand area. The gesture type is identified through Canny edge detection and finger keypoint algorithm. The symbolic gesture rate (rate_hand), digital gesture rate (rate_hand_number), and thumb extension rate (rate_hand_thumb) are calculated, and a feature histogram is constructed based on the video frame sequence.

[0070] The facial landmark detector of the dlib library is used to locate the key points of the eyes, nose, and mouth. The ratio of the distance between the left eye and the right eye / the distance between the left eye and the nose is calculated as the eye width ratio. The ratio of the distance between the mouth and the nose / the distance between the left eye and the nose is calculated as the mouth height ratio. The proportion of positive, negative, and neutral expressions is counted to form the positive_rate, negative_rate, and neutral_rate features.

[0071] Use the OpenPose algorithm to obtain skeleton key points, calculate the angles of each limb using the vector cosine formula, calculate the open posture ratio pose_open_rate, the closed posture ratio pose_close_rate, and the contradictory posture ratio pose_neutral_rate, and calculate the time series variance var_pitch / var_yaw / var_roll related to posture stability;

[0072] Based on the facial key point tracking algorithm, the head movement trajectory is analyzed through continuous frames, the number of nods and shakes per unit time is counted, and the maximum, minimum and average movement amplitudes are calculated based on the three-axis acceleration data;

[0073] The improved PnP algorithm combined with the FSA-Net lightweight model is used to extract the Pitch / Yaw / Roll Euler angles from continuous frames. The maximum value (max_pitch / max_yaw / max_roll), mean (mean_pitch / mean_yaw / mean_roll) and range features of the time series are calculated. The line of sight focus area is estimated using the iris positioning algorithm.

[0074] Table 2

[0075]

[0076]

[0077] In a specific embodiment, Figure 3 、 Figure 4 As shown in FIG, the microteaching evaluation model specifically includes two single-modal enhancement modules and one multi-modal combination module;

[0078] The two unimodal enhancement modules are used to enhance video features and audio features respectively; the unimodal enhancement module includes a cascaded first-layer normalization block, an attention mechanism block, a second-layer normalization block, an activation block, a third-layer normalization block, a feedforward network block, and a fourth-layer normalization block; the first-layer normalization block is also connected to its unimodal input residual; the activation block is also connected to the output residual of the attention mechanism block; the feedforward network block is also connected to the output residual of the fourth-layer normalization block;

[0079] The multimodal combination module is used to perform feature fusion on the output vectors of the single-modal modules and assign weights; the multimodal combination module includes a first-layer normalization block, a second-layer normalization block, a cross-attention block, a third-layer normalization block, an activation block, a fourth-layer normalization block, a feedforward network block, and a fifth-layer normalization block; the outputs of the fourth-layer normalization blocks of the two single-modal enhancement modules are respectively connected to the first-layer normalization block and the second-layer normalization block of the multimodal combination module by residual connection; the outputs of the first-layer normalization block and the second-layer normalization block are respectively connected to the cross-attention block; the cross-attention block, the third-layer normalization block, the activation block, the fourth-layer normalization block, the feedforward network block, and the fifth-layer normalization block are cascaded; the activation block is also connected to the output residual of the cross-attention block; and the feedforward network block is also connected to the output residual of the fifth-layer normalization block.

[0080] In a specific embodiment, based on the decision tree improved algorithm, the micro-teaching evaluation model is used to analyze the video features and audio features to generate teaching evaluation results. The specific steps are as follows:

[0081] The existing micro-teaching dataset is used as the training set. The attributes with the highest information gain ratio are selected from the training set as split nodes to construct the decision tree trunk. The splitting is stopped when the purity of the node sample reaches the preset threshold or the partitioning ability is lower than the truth index.

[0082] A microteaching evaluation model is constructed at the leaf nodes of the decision tree: at each leaf node, a pre-normalized attention mechanism is used to input the corresponding video and audio features into the corresponding single-modal enhancement module for enhancement processing and modality alignment;

[0083] At each leaf node, the enhanced two single-modal features are input into the multimodal combination module for interaction. A joint representation vector is formed through feature multiplication and weighted concatenation to generate the local evaluation of each leaf node.

[0084] The local evaluation of each leaf node neural network is integrated and the overall teaching evaluation result is output after custom weighting.

[0085] In this embodiment, audio and video feature extraction is performed based on a micro-teaching scenario. The present invention constructs an audio and video evaluation scale based on the teaching scenario and maps it to computer-calculated features. The present invention implements a pronunciation standard evaluation scheme for a text without reference, and constructs a rhythm and emotion recognition model adapted to the micro-teaching scenario based on fine-tuning of the model. 13 feature-rich audio language classification features are obtained by supplementing the traditional rhythmic features such as speech speed and loudness. 24 videos are obtained by constructing three major feature extraction models, and classification modeling is performed based on a decision tree. Finally, the micro-teaching evaluation model has an accuracy rate of 0.78 for single audio modality evaluation, an accuracy rate of 0.88 for single video modality evaluation, and an accuracy rate of 0.91 for audio and video feature-level fusion feature evaluation. The extracted features are of practical significance, have good classification capabilities, and have certain practical application value.

[0086] A pre-normalized neural network decision tree model based on audio and video fusion. This paper uses the decision tree model as the base model, retains the model construction rules as the guidance rules for normal students, and trains the audio and video fusion neural network model through subset training on leaf nodes, improving the decision tree classification accuracy and achieving accurate teaching evaluation.

[0087] Based on the needs of actual scenarios, the present invention trains an acoustic model for Chinese scenarios and a rhythm recognition model for classroom teaching scenarios, and adopts a speech recognition model with hot words based on mathematics teaching scenarios, making the entire system more suitable for Chinese mathematics micro-teaching scenarios and having better performance.

[0088] Before the implementation of the technology, the present invention first constructed evaluation indicators for micro-teaching videos and language skills based on literature and guidance from professional teachers. Based on this, the indicators required for automatic evaluation are richer and more practical, and can better reflect the level of teaching ability.

[0089] Example 3

[0090] An evaluation system for video and voice intelligence of micro-teaching, including a data acquisition module, a voice recognition module, a video recognition module, and a comprehensive evaluation module;

[0091] The data acquisition module is used to collect video and audio data during the micro-teaching process;

[0092] The speech recognition module is used to perform speech recognition, rhythm recognition, emotion recognition on audio data and extract audio features;

[0093] The video recognition module is used to perform finger gesture recognition, facial expression recognition, posture recognition and head posture recognition on video data and extract video features;

[0094] The comprehensive evaluation module is used to construct a micro-teaching evaluation model based on a pre-normalized modal feature fusion neural network, and based on a decision tree improved algorithm, uses the micro-teaching evaluation model to analyze the video features and audio features to generate teaching evaluation results.

[0095] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for evaluating video and voice intelligence in micro-teaching, characterized by: The following steps are involved: Collect video and audio data during micro-teaching; Perform speech recognition, rhythm recognition, and emotion recognition on audio data, and extract audio features; Perform finger gesture recognition, facial expression recognition, posture recognition, and head posture recognition on video data to extract video features; Construct a microteaching evaluation model based on pre-normalized modal feature fusion neural network; Based on the decision tree improved algorithm, the micro-teaching evaluation model is used to analyze the video features and audio features to generate teaching evaluation results.

2. The method for evaluating video and voice intelligence in microteaching according to claim 1, characterized in that: The video and audio data of the micro-teaching process are collected. The specific steps are as follows: synchronously collect teaching video and audio data through recording and broadcasting equipment, extract key frames and annotate postures of video data, and perform noise reduction and time domain segmentation on audio data.

3. The method for evaluating video and voice intelligence in microteaching according to claim 1, characterized in that: The audio features specifically include rhythm, loudness, speaking speed, positivity, and voice standardization.

4. The method for evaluating video and voice intelligence in microteaching according to claim 3, characterized in that: Perform speech recognition, rhythm recognition, and emotion recognition on audio data to extract audio features. The specific steps are as follows: Extract low-level acoustic descriptors, set window setting parameters, and calculate speech rate, pause interval, and syllable density features. Use piecewise statistical functions to cluster rhythm patterns of set speech segments, build a classroom rhythm classification model based on a decision tree, and output rhythm scores and silence duration ratios. The human ear simulation algorithm is used to calculate the loudness mean and standard deviation, and the improved pitch detection algorithm is used to extract the fundamental frequency mean, standard deviation and global average value of the dynamic range; the cepstrum analysis technique is used to calculate the kurtosis prominence of the periodic signal energy in the cepstrum domain; A dual-stream deep learning model was constructed. The first branch inputs several orders of Mel spectrograms, and the second branch inputs manually designed rhythmic feature vectors. The relational knowledge distillation method was used to fuse the deep features and manual features, and finally output the emotional neutrality, positivity, and negativity. An acoustic model is constructed based on the existing classroom hot word library, and an improved DNN-HMM hybrid architecture is used for speech recognition and decoding; the pronunciation standardization index is calculated using the phoneme-level Goodness of Pronunciation algorithm.

5. The method for evaluating video and voice intelligence in micro-teaching according to claim 4, characterized in that: The speech recognition model, emotion recognition model, and rhythm recognition model are all recognition models without reference text. They directly process the audio of teaching scenes through speech recognition technology with hot words without relying on classroom text transcription files.

6. The method for evaluating video and voice intelligence in microteaching according to claim 1, characterized in that: The video features specifically include gesture ratio, proportion of positive, negative and neutral expressions, head movement, and eye direction.

7. The method for evaluating video and voice intelligence in microteaching according to claim 6, characterized in that: Perform finger gesture recognition, facial expression recognition, posture recognition, and head posture recognition on video data to extract video features. The specific steps are as follows: A gesture classifier is used to detect the hand area, and the gesture type is identified through edge detection and finger key point algorithm. The symbolic gesture ratio, digital gesture ratio and thumb extension ratio are calculated. The key points of the eyes, nose and mouth are located by using a facial landmark detector, and the ratio of the distance between the left eye and the right eye and the distance between the left eye and the nose is calculated as the eye width ratio. The ratio of the distance between the mouth and the nose and the distance between the left eye and the nose is calculated as the mouth height ratio; Define the types of expressions corresponding to different proportions; calculate the proportions of positive, negative, and neutral expressions; Obtain skeleton key points, calculate the angles of each limb using the vector cosine formula, count the open posture ratio, closed posture ratio, and contradictory posture ratio, and calculate the time series variance related to posture stability; Based on the facial key point tracking algorithm, the head movement trajectory is analyzed through continuous frames, the number of nods and shakes per unit time is counted, and the maximum, minimum and average movement amplitudes are calculated by combining the three-axis acceleration data; The improved PnP algorithm combined with the FSA-Net lightweight model is used to extract the Pitch / Yaw / Roll Euler angles from continuous frames. The maximum, mean and range characteristics of their time series are calculated, and the line of sight focus area is estimated using the iris positioning algorithm.

8. The method for evaluating video and voice intelligence in microteaching according to claim 1, characterized in that: The microteaching evaluation model specifically includes two single-modal enhancement modules and one multi-modal combination module; The two unimodal enhancement modules are used to enhance video features and audio features respectively; the unimodal enhancement module includes a cascaded first-layer normalization block, an attention mechanism block, a second-layer normalization block, an activation block, a third-layer normalization block, a feedforward network block, and a fourth-layer normalization block; the first-layer normalization block is also connected to its unimodal input residual; the activation block is also connected to the output residual of the attention mechanism block; the feedforward network block is also connected to the output residual of the fourth-layer normalization block; The multimodal combination module is used to perform feature fusion on the output vectors of the single-modal modules and assign weights; the multimodal combination module includes a first-layer normalization block, a second-layer normalization block, a cross-attention block, a third-layer normalization block, an activation block, a fourth-layer normalization block, a feedforward network block, and a fifth-layer normalization block; the outputs of the fourth-layer normalization blocks of the two single-modal enhancement modules are respectively connected to the first-layer normalization block and the second-layer normalization block of the multimodal combination module by residual connection; the outputs of the first-layer normalization block and the second-layer normalization block are respectively connected to the cross-attention block; the cross-attention block, the third-layer normalization block, the activation block, the fourth-layer normalization block, the feedforward network block, and the fifth-layer normalization block are cascaded; the activation block is also connected to the output residual of the cross-attention block; and the feedforward network block is also connected to the output residual of the fifth-layer normalization block.

9. The method for evaluating video and voice intelligence in micro-teaching according to claim 8, characterized in that: Based on the improved decision tree algorithm, the micro-teaching evaluation model is used to analyze the video features and audio features to generate teaching evaluation results. The specific steps are as follows: The existing micro-teaching dataset is used as the training set. The attributes with the highest information gain ratio are selected from the training set as split nodes to construct the decision tree trunk. The splitting is stopped when the purity of the node sample reaches the preset threshold or the partitioning ability is lower than the truth index. A microteaching evaluation model is constructed at the leaf nodes of the decision tree: at each leaf node, a pre-normalized attention mechanism is used to input the corresponding video and audio features into the corresponding single-modal enhancement module for enhancement processing and modality alignment; At each leaf node, the enhanced two single-modal features are input into the multimodal combination module for interaction. A joint representation vector is formed through feature multiplication and weighted concatenation to generate the local evaluation of each leaf node. The local evaluation of each leaf node neural network is integrated to output the overall teaching evaluation results.

10. A video and voice intelligence evaluation system for micro-teaching, characterized by: Including data acquisition module, speech recognition module, video recognition module, and comprehensive evaluation module; The data acquisition module is used to collect video and audio data during the micro-teaching process; The speech recognition module is used to perform speech recognition, rhythm recognition, emotion recognition on audio data and extract audio features; The video recognition module is used to perform finger gesture recognition, facial expression recognition, posture recognition and head posture recognition on video data and extract video features; The comprehensive evaluation module is used to construct a micro-teaching evaluation model based on a pre-normalized modal feature fusion neural network, and based on a decision tree improved algorithm, uses the micro-teaching evaluation model to analyze the video features and audio features to generate teaching evaluation results.

Images