Classroom teacher teaching performance description method, model and system based on multi-modal data fusion and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a classroom teacher teaching performance description method, model and system based on multi-modal data fusion and a storage medium. By introducing a multi-modal data fusion strategy, cooperative processing of classroom teacher visual information, audio information and text information is realized, and the accuracy and time sequence continuity of teacher target perception are significantly improved. Compared with a traditional single-mode method, the method not only can extract the posture, expression and action characteristics of the teacher from the visual mode, but also can recognize the voice emotion and the side language signal from the audio mode, and achieves the full-dimensional description of the teaching behavior of the teacher in combination with the text semantics. The objective of the invention is to solve the problem of how to perform multi-dimensional evaluation on teaching performance of a classroom teacher based on data of multiple modalities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a classroom teacher teaching performance portrayal method, model, system and storage medium based on multi-modal data fusion. BACKGROUND

[0002] The classroom teacher teaching performance portrayal task aims to comprehensively describe the teaching activity performance of teachers in the classroom teaching process, covering the ability and characteristics of their speech and non-verbal behavior in teaching interaction, knowledge transmission, classroom regulation, etc. This task has important significance in intelligent education, teaching evaluation and teacher development, etc. Application scenarios can provide key support for precise teaching feedback, teacher ability diagnosis and intelligent classroom auxiliary decision-making.

[0003] In related technical solutions, the information processing method of the classroom teacher teaching performance portrayal task is mainly the processing of single modal information, for example, through a single visual modality to mainly focus on the actions, postures and spatial trajectories of teachers, or through a single audio modality to analyze the speech content, tone changes and emotional states of teachers.

[0004] However, this single modal information processing method has the problem of being difficult to accurately portray the classroom performance of college teachers. In view of this, the present application proposes a classroom teacher teaching performance portrayal method based on multi-modal data fusion, aiming to perform multi-dimensional evaluation of the teaching performance of classroom teachers based on data of multiple modalities. SUMMARY

[0005] The main purpose of the present application is to provide a classroom teacher teaching performance portrayal method based on multi-modal data fusion, aiming to solve the problem of how to perform multi-dimensional evaluation of the teaching performance of classroom teachers based on data of multiple modalities.

[0006] To achieve the above-mentioned purpose, the present application provides a classroom teacher teaching performance portrayal method based on multi-modal data fusion, which comprises the following steps: S10, determining the image frame data and audio stream data associated with the classroom teacher in the classroom recording and broadcasting video; S20, extracting the posture features, facial features and action features of the classroom teacher in the image frame data, and determining the transcription text, target text classification and target speech emotion classification of the classroom teacher in the audio stream data; S30, extracting first visual information and first auditory information representing an emotional support dimension from the posture feature, the facial feature, the motion feature, the transcription text, the target text classification and the target speech emotion classification according to a first preset extraction rule, and determining the first teaching performance result of the classroom teacher in the emotional support dimension according to the first visual information and the first auditory information; and S40, extracting second visual information and second auditory information representing a classroom organization dimension from the posture feature, the motion feature, the transcription text and the target text classification according to a second preset extraction rule, and determining the second teaching performance result of the classroom teacher in the second classroom organization dimension according to the second visual information and the second auditory information; and S50, extracting third visual information and third auditory information representing a teaching support dimension from the posture feature, the motion feature, the transcription text and the target text classification according to a third preset extraction rule, and determining the third teaching performance result of the classroom teacher in the teaching support dimension according to the third visual information and the third auditory information; S60, evaluating the teaching performance characterization result of the classroom teacher based on the first teaching performance result, the second teaching performance result and the third teaching performance result.

[0007] Optionally, the first visual information includes body orientation, facial emotional state, facial orientation and body posture, the first auditory information includes speech feedback, speech habit, speech emotional state and paralanguage feature, and the step S30 includes: S31, determining the target body orientation classification, the target facial emotional state classification, the target facial orientation classification and the target body posture classification corresponding to the classroom teacher in the preset classification rule according to at least one of the posture feature, the facial feature and the motion feature; and determining the target speech feedback classification, the target speech habit classification, the target speech emotional state classification and the target paralanguage feature classification corresponding to the classroom teacher in the preset classification rule according to the transcription text, the target text classification and the target speech emotion classification; S32, taking the target body orientation classification, the target facial emotional state classification, the target facial orientation classification and the target body posture classification as the first visual target classification; and taking the target speech feedback classification, the target speech habit classification, the target speech emotional state classification and the target paralanguage feature classification as the first auditory target classification; S33, evaluating the first teaching performance result based on the first visual target classification and the first auditory target classification.

[0008] Optionally, the first visual target classification comprises at least one of: The body orientation classification comprises facing the student, sideways to the student, and facing away from the student. The facial emotional state classification comprises positive emotion, neutral emotion, and negative emotion. The facial orientation classification comprises facing the student and facing other objects. The body posture classification comprises standing upright, leaning forward, walking, sitting, and leaning against standing. And the first auditory target classification comprises at least one of: The verbal feedback classification comprises encouragement, praise, affirmation, negation, criticism, and sarcasm. The verbal habit classification comprises soft, collective subject expression, and casual conversation. The verbal emotional state classification comprises positive emotion, neutral emotion, and negative emotion. The paralinguistic feature classification comprises first-level volume, second-level volume, third-level volume, first-level speech rate, second-level speech rate, and third-level speech rate.

[0009] Optionally, the second visual information comprises body movement, the second auditory information comprises instructional language and inquiry language, and the step S40 comprises: S41, according to the posture feature and the movement feature, determining the target body movement classification of the teacher in the pre-set classification rule; And, according to the transcription text and the target text classification, determining the target instructional language classification and the target inquiry language classification of the teacher in the respective classification rule; S42, taking the obtained target body movement classification as the second visual target classification; And, taking the obtained target instructional language classification and the target inquiry language classification as the second auditory target classification; S43, based on the second visual target classification and the second auditory target classification, evaluating the second teaching performance result.

[0010] Optionally, the second visual target classification comprises at least one of: The body movement classification comprises knocking the blackboard, knocking the desk, starting the classroom activity, ending the classroom activity, asking the student to answer, and gesture action. And the second auditory target classification comprises at least one of: The instructional language classification comprises starting and ending the class, starting and ending the teaching activity, practice, discussion, random question, designated question, silence, and concentration. The inquiry language classification comprises homework opinion, deadline opinion, discussion topic opinion, and examination opinion.

[0011] Optionally, the third visual information includes adaptive behavior, indicative behavior, regulatory behavior, instrumental behavior, illustrative behavior, evaluative behavior and distancing behavior, the third auditory information includes question type, and the step S50 includes: S51, determining target adaptive behavior, target indicative behavior, target regulatory behavior, target instrumental behavior, target illustrative behavior, target evaluative behavior and target distancing behavior satisfied by the classroom teacher according to the posture feature and the action feature; and determining target question type satisfied by the classroom teacher according to the transcription text and the target text classification; S52, taking the target adaptive behavior, the target indicative behavior, the target regulatory behavior, the target instrumental behavior, the target illustrative behavior, the target evaluative behavior and the target distancing behavior as the third visual target classification of the classroom teacher; and taking the target question type as the third auditory target classification of the classroom teacher; S53, evaluating the third teaching performance result of the classroom teacher in the teaching support dimension according to the third visual target classification and the third auditory target classification.

[0012] Optionally, the adaptive behavior includes hands behind the back, crossed arms, hands on the waist, hands in the pocket, arms supporting the desk, standing with crossed legs, hands in front of the abdomen, and hands in front of the body. The indicative behavior includes pointing to the blackboard, pointing to the projection, and pointing to the experimental instrument. The regulatory behavior includes self-habitual behavior, adjusting behavior, arranging clothes, and playing with irrelevant items. The instrumental behavior includes writing on the board, erasing the blackboard, operating experimental equipment, operating teaching models, operating multimedia equipment, holding teaching materials to teach, and referring to multimedia equipment. The illustrative behavior includes gestures, object demonstration, and movement demonstration. The evaluative behavior includes thumbs up, clapping, nodding, and shaking. The distancing behavior includes walking onto the podium, walking off the podium, walking on the podium, walking in the front row of the classroom, and walking in the aisle of the classroom. The question type includes what kind of question, why kind of question, how kind of question, if kind of question, and from kind of question.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a classroom teacher teaching performance portrayal model, which comprises: A multi-modal data classification module is configured to determine image frame data and audio stream data associated with a classroom teacher in a classroom recording and broadcasting video. a multi-modal data extraction module configured to extract posture features, facial features and action features of the teacher in the image frame data, and determine transcription text, target text classification and target speech emotion classification of the teacher in the audio stream data; a support emotion dimension characterization module configured to extract first visual information and first auditory information representing a support emotion characterization dimension from the posture features, the facial features, the action features, the transcription text, the target text classification and the target speech emotion classification according to a first preset extraction rule, and determine a first teaching performance result of the teacher in the support emotion dimension according to the first visual information and the first auditory information; a classroom organization dimension characterization module configured to extract second visual information and second auditory information representing a classroom organization characterization dimension from the posture features, the action features, the transcription text and the target text classification according to a second preset extraction rule, and determine a second teaching performance result of the teacher in the second classroom organization dimension according to the second visual information and the second auditory information; a teaching support characterization module configured to extract third visual information and third auditory information representing a teaching support characterization dimension from the posture features, the action features, the transcription text and the target text classification according to a third preset extraction rule, and determine a third teaching performance result of the teacher in the teaching support dimension according to the third visual information and the third auditory information; a teaching performance characterization module configured to evaluate a teaching performance characterization result of the teacher based on the first teaching performance result, the second teaching performance result and the third teaching performance result.

[0014] In addition, to achieve the above object, the present application also provides a computer system, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the multi-modal data fusion based teaching performance characterization method of a teacher in a classroom according to any one of the above.

[0015] In addition, to achieve the above object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-modal data fusion based teaching performance characterization method of a teacher in a classroom according to any one of the above.

[0016] The present application has at least the following beneficial effects: By introducing a multi-modal data fusion strategy, the collaborative processing of classroom teacher visual information, audio information and text information is realized, and the accuracy and time sequence continuity of teacher target perception are significantly improved. Compared with the traditional single modal method, the present application can not only extract the posture, expression and action features of the teacher from the visual modal, but also identify the speech emotion and paralanguage signals from the audio modal, and combine the text semantics to realize the full-dimensional characterization of the teacher's teaching behavior. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a classroom teacher teaching performance characterization method based on multi-modal data fusion related to the embodiments of the present application; Figure 2 A recognition flowchart of a classroom teacher posture estimation task related to the embodiments of the present application; Figure 3 A flowchart of a classroom teacher action recognition task related to the embodiments of the present application; Figure 4 A recognition flowchart of a classroom teacher expression recognition task related to the embodiments of the present application; Figure 5 A recognition flowchart of a classroom teacher speech text transcription task related to the embodiments of the present application; Figure 6 A flowchart of a classroom teacher teaching transcription text classification task related to the embodiments of the present application; Figure 7 A task flowchart of a classroom teacher speech emotion recognition related to the embodiments of the present application; Figure 8 A multi-modal data fusion classroom teacher teaching performance characterization framework related to the embodiments of the present application; Figure 9 An architecture diagram of a hardware running environment of a computer system related to the embodiments of the present application.

[0018] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] In order to better understand the above technical solutions, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to convey the scope of the present disclosure to those skilled in the art.

[0020] First embodiment Reference Figure 1The embodiment provides a classroom teacher teaching performance characterization method based on multi-modal data fusion. S10, image frame data and audio stream data associated with a classroom teacher in a classroom recording and broadcasting video are determined. In the embodiment, the input of the intelligent characterization framework of the college teacher classroom teaching performance is video data of the college classroom teaching process, and the video content contains rich visual information and audio information. Based on the behavior index system defined by the college teacher classroom teaching performance characterization model, the image frame data as the visual mode are mainly used for identifying the non-verbal behavior of the teacher, and the audio stream data as the audio mode are used for identifying the verbal behavior of the teacher.

[0021] In order to effectively utilize the multi-modal information, the system first performs basic processing on the visual information and the audio information. The visual information is extracted by a target tracking and pose estimation algorithm to obtain the spatial position and key point motion trajectory of the teacher, and the audio information is obtained by means of speaker recognition and speech recognition technology to obtain the speaking period and speech content of the teacher.

[0022] S20, extracting the posture feature, face feature and action feature of the classroom teacher in the image frame data, and determining the transcription text, target text classification and target speech emotion classification of the classroom teacher in the audio stream data. After the basic feature information extraction is completed, the specific teaching behavior of the teacher in the classroom is identified based on action recognition, pose recognition, facial expression recognition, text classification and speech emotion recognition.

[0023] For the analysis of the image frame data, in the embodiment, four tasks including classroom teacher target tracking, classroom teacher pose estimation, classroom teacher action recognition and classroom teacher expression recognition are realized by comprehensively using various advanced visual recognition technologies.

[0024] In the embodiment, in order to realize the single target tracking of the classroom teacher in the classroom scene, the PyTracking visual tracking algorithm is used for design in the embodiment, and the algorithm is not the focus of the embodiment, and is not described here.

[0025] For classroom teacher posture estimation, the classroom teacher posture estimation is to detect the positions of each key point of the human body from an image or a video, and construct the posture of the human body according to the relative positional relationship between the key points. Based on the RTMPose model in the MMPose tool box, CSPNeXt is used as the backbone network, combined with the SimCC key point detection algorithm, the RTMPose model is trained using the constructed college teacher classroom recording and broadcasting video, and the posture estimation of the teacher can achieve a high recognition accuracy, and the model can realize lower delay and better robustness. The posture information of the teacher is one of the input features of the teacher action recognition model, so the recognition effect of the teacher posture directly affects the effect of the teacher action recognition.

[0026] Exemplarily, the recognition process of the classroom teacher posture estimation task is as shown in Figure 2 The input of the whole process mainly includes two parts, which are the classroom teaching video and the pre-generated teacher object video sequence target box of the target tracking task. Both of these two types of inputs need to be uniformly cropped to avoid the large jitter that may be caused by directly cropping according to the sequence target box, thereby improving the stability of the visual input. After cropping, the processed image frames will be input into the teacher posture estimation model, the model will recognize these frames and finally output the time sequence posture information of the teacher. The recognition process of the classroom teacher posture estimation task can be represented by the following formula:

[0027]

[0028] Among them, the input of the recognition process includes video input and video sequence target box , represents the uniform cropping operation of the image frame, represents the image frame sequence.

[0029] For classroom teacher action recognition, it refers to understanding and analyzing a video content, and identifying what actions the person in the video is performing. In a classroom scenario, identifying the actions of a teacher includes actions such as writing on the board, erasing the blackboard, operating multimedia tools, and other actions with teaching significance. To implement the classroom teacher action recognition task, the embodiment in the present application develops a classroom teacher action recognition model based on spatio-temporal dual-branch feature fusion based on the MMAction2 toolkit. MMAction2 is an open-source toolkit based on PyTorch, which supports a large number of video understanding models, implements the most advanced behavior recognition, positioning, detection, and other models, including behavior recognition, skeleton-based behavior recognition, spatio-temporal behavior detection, and temporal action positioning. The classroom teacher action recognition model based on spatio-temporal dual-branch feature fusion considers the time sequence information and spatial information of the teacher's action, inputs the posture information and RGB image information of two modalities into the model, respectively passes through the 2D and 3D backbone networks, and finally fuses the information of the two modalities to better understand and recognize the teaching actions of the teacher in the classroom teaching process.

[0030] The flow of the classroom teacher action recognition task is shown in Figure 3 The input of the whole task consists of three parts, which are the classroom teaching video, the teacher object video sequence bounding box, and the teacher's time sequence posture information, wherein the latter two are the results generated by the preposed target tracking and posture estimation tasks. The teacher object video sequence bounding box is used to uniformly crop the image frames in the classroom teaching video to obtain single-frame teacher target images, ensuring the stability and consistency of the image information. At the same time, the generation of the teacher posture information heat map is based on the results of the preposed posture estimation task. The input of the model includes two parts, one part is the single-frame teacher target image, and the other part is the teacher's time sequence posture heat map. Through the fusion of spatio-temporal dual-branch features, the algorithm model can more effectively capture the action features of the teacher in the complex classroom environment. By using the cooperative processing of spatio-temporal information, the model improves the accuracy of action recognition and enhances the robustness in the variable teaching scene, so that the recognition of classroom teacher action realizes efficient and accurate judgment on the basis of multi-dimensional feature fusion. The input of the classroom teacher action recognition is shown in the following formula:

[0031] In the formula, represents the generation of the posture information heat map, represents the final output of the teacher action model .

[0032] In this embodiment, the teacher's facial expressions include positive emotions, negative emotions, and neutral emotions. Positive emotions include happiness and surprise, while negative emotions include anger, disgust, fear, and sadness. The teacher's nonverbal emotional state is determined by recognizing the teacher's facial expressions.

[0033] As an alternative implementation, the DeepFace framework can be used to implement teacher facial expression recognition tasks. DeepFace can achieve a recognition accuracy of over 95% on general datasets. This study fine-tunes the facial expression recognition algorithm on a college teacher classroom teaching dataset constructed in this paper.

[0034] The recognition process for the classroom teacher facial expression recognition task is as follows: Figure 4 As shown. Although existing facial expression recognition methods are relatively mature, further optimization is still needed when applied to the intelligent characterization architecture of classroom teachers' performance. Typically, facial expression recognition tasks require first detecting the face region, and then inputting the detected facial images into the facial expression recognition model for analysis. However, the visual analysis-based intelligent characterization architecture for classroom teachers' teaching performance constructed in this study already includes the results of multiple pre-tasks, thus eliminating the need for a dedicated face detection step. Through this visual analysis-based framework, information about the teacher's target object region can be extracted from the teacher's target video sequence. The results of temporal pose estimation are used to accurately predict the teacher's head position, thereby extracting the facial region and providing accurate input to the facial expression recognition model. Subsequently, facial expression recognition is performed on the extracted facial region to achieve intelligent analysis of the teacher's expressions. This optimized method reduces the step of separately detecting the face region, improving the efficiency and accuracy of the overall recognition process. The entire task is shown in the following equation:

[0035]

[0036] This indicates the extraction of the teacher's facial region. This represents a teacher facial expression recognition model. Indicates the teacher's facial area. This indicates the teacher's chronological expression category.

[0037] The main purpose of analyzing audio stream data is to identify the speaking time information of teachers in the classroom and to perceive the teacher's subject, so as to provide a data foundation for subsequent tasks such as classroom speech-to-text transcription, classroom transcript classification, and classroom teacher speech emotion recognition.

[0038] As an optional implementation, this embodiment uses the SOND model in the FunASR framework to implement the classroom speaker log task.

[0039] For transcribed text, i.e., speech text transcription, i.e. speech recognition.

[0040] As an optional implementation, this embodiment uses the ParaFormer model in the FunASR framework to implement the classroom speech-to-text transcription task.

[0041] The recognition process for classroom teacher speech-to-text transcription tasks is as follows: Figure 5 As shown, the core task is to accurately identify audio information using a speech recognition model during classroom teaching. The entire process begins with the input of classroom teaching audio, accurately obtaining the teacher's speaking time information using a pre-built speaker log model. First, the classroom teaching audio is trimmed according to the timestamps provided by the speaker log, extracting audio segments containing only the teacher's speech. Then, speech recognition is performed on these segments to generate corresponding text content. Finally, this process successfully transcribes the classroom teacher's speech into text, providing reliable text data support for subsequent analysis. The simplified process is shown in the following equation:

[0042] In the formula, This represents a speech recognition model. This indicates audio trimming. This indicates the time the teacher speaks in class. This refers to audio recordings of classroom teaching. This represents the teacher's spoken text in class.

[0043] For target text classification, text classification is an important task in the field of natural language processing. It involves assigning a given text to a predefined category or label. Text classification is of great significance for organizing and understanding large amounts of text data.

[0044] As an alternative implementation, this embodiment utilizes the Qwen 2.5 open-source pre-trained large language model to implement the classroom transcription text classification task, using a parameter size of 7B.

[0045] The process of classroom teacher-led transcription text classification tasks is as follows: Figure 6 As shown, relying on a prior speech recognition task, the system obtains transcribed text of a teacher's lecture. The first step in the process is to filter the original text, removing irrelevant or noisy words to ensure a clearer and more concise transcript of the teacher's speech. The processed text is then input into a pre-trained text classification model. This model can classify and analyze the teacher's speech behaviors, identifying the specific categories of speech behaviors in the classroom. The task flow can be simplified as follows:

[0046] In the formula, Represents a text classification model. Represents the text filtering process. This represents the teacher's spoken text in class. This refers to the verbal behavior of teachers in the classroom.

[0047] Target speech emotion classification refers to identifying the emotional state expressed by a speaker by analyzing the sound features in the speech signal.

[0048] As an optional implementation, this embodiment uses the TIM-Net speech emotion recognition model to complete the task of recognizing the emotion in a teacher's speech during class. The teacher speech emotion recognition task is essentially a type of audio classification, its core being the analysis and processing of the teacher's speech signal to identify its emotional features. The model takes speech as input, extracts features from the audio signal, models them, and combines relevant algorithms to classify the emotional state of the speech into specific category labels. The TIM-Net model can also be used for other audio classification tasks; simply changing the dataset labels used during training allows it to adapt to different classification needs. In this embodiment, when constructing the model to characterize the classroom teaching performance of university teachers, the TIM-Net model architecture is used for all parts involving audio classification methods. Since the model's performance in audio classification tasks has been widely verified for its adaptability and mature effectiveness, this embodiment will not provide further detailed explanation.

[0049] The task flow of classroom teacher voice emotion recognition is as follows: Figure 7 As shown in the diagram. The task requires input including audio data of classroom teaching and teacher speaking time information. The speaking time information is used to trim the audio and extract audio segments of the teacher's speech. The trimmed speech segments are then input into the speech emotion recognition model. Through analysis and processing of speech features, the speech emotion classification task is finally completed. The entire simplified process is shown in the following formula:

[0050] In the formula, This represents a speech emotion recognition model. For audio trimming, the input includes and The output of the task is .

[0051] S30, according to the first preset extraction rule, extract first visual information and first auditory information representing the emotional support characterization dimension from the posture features, facial features, action features, transcribed text, target text classification, and target speech emotion classification; and determine the first teaching performance result of the classroom teacher in the emotional support dimension based on the first visual information and the first auditory information; and, S40, according to the second preset extraction rule, extract second visual information and second auditory information representing the classroom organization dimension from the posture features, the action features, the transcribed text, and the target text classification; and determine the second teaching performance result of the classroom teacher in the second classroom organization dimension based on the second visual information and the second auditory information; and, S50, according to the third preset extraction rule, extract the third visual information and the third auditory information that characterize the teaching support dimension from the posture features, the action features, the transcribed text and the target text classification, and determine the third teaching performance result of the classroom teacher in the teaching support dimension based on the third visual information and the third auditory information. This embodiment proposes three dimensions to evaluate the teaching performance of classroom teachers: emotional support, classroom organization, and teaching support.

[0052] The theoretical foundation of emotional support primarily stems from attachment theory and self-determination theory. Attachment theory emphasizes the crucial role of individuals in providing emotional support and creating a predictable, consistent, and safe environment. When students experience this emotional support from adults, they become more independent and demonstrate greater confidence and courage in exploring the world. Students subconsciously feel that someone will be there to support them when they need help. Self-determination theory revolves around three basic psychological needs: autonomy, competence, and relatedness. It posits that fulfilling these three basic psychological needs is a key factor in promoting intrinsic motivation and mental health. When these needs are hindered, intrinsic motivation and mental health are negatively impacted. Human behavior, thoughts, consciousness, and identity are all formed through interactions with others and within social structures. No individual can exist independently of their social environment, and school life constructs the most important social relationships for students during their learning years. Positive interactions enhance students' social skills, boost self-confidence, and promote academic progress. Teachers in the classroom should not only focus on knowledge transmission but also emphasize emotional support to comprehensively promote student development.

[0053] The theoretical foundation of classroom organization primarily stems from research on student self-regulation theory and executive function skills. Students' self-regulation and executive function significantly impact learning and academic achievement. Clear and consistent behavior and time management in the classroom often help students develop these important skills. Teachers need to create an orderly and well-managed classroom environment to promote the effective development of students' self-regulation and executive function skills. Secondly, preventative management strategies are crucial. By establishing clear rules and procedures, teachers can address potential problems before they occur, thereby maintaining classroom order.

[0054] The Instructional Support (IPS) domain primarily focuses on how teachers can facilitate students' cognitive and language development through teaching methods and interactions. Its theoretical foundation stems from research on student cognitive and language development, emphasizing that students not only need to learn facts but also construct "useful knowledge"—that is, understand the interrelationships and organization between facts. Scaffolding theory posits that students' cognitive and language development depends on opportunities provided by teachers that enable them to express existing skills and build upon them to develop more complex ones. Overall learning is most effective when teachers connect new information with students' background knowledge and real-world examples. Therefore, in the classroom, IPS helps students understand and apply complex concepts through questioning and guided discussions, provides timely, specific, and constructive feedback, and designs challenging tasks and problems, thereby effectively cultivating students' critical thinking and problem-solving abilities.

[0055] The computer implementation of the above three teaching performance results will be described in subsequent embodiments, and will not be repeated in this embodiment.

[0056] S60, based on the first teaching performance result, the second teaching performance result, and the third teaching performance result, evaluate the teaching performance characterization result of the classroom teacher.

[0057] Finally, the teaching performance profiles of classroom teachers were evaluated based on the first teaching performance outcome representing emotional support, the second teaching performance outcome representing classroom organization, and the third teaching performance outcome representing teaching support.

[0058] It should be noted that the focus of this embodiment is to provide an intelligent characterization scheme for the classroom teaching performance of university teachers, emphasizing the identification and feature description of various teaching behaviors of teachers, without involving specific quantitative scoring or value judgment.

[0059] In the technical solution provided in this embodiment, the theoretical framework of classroom assessment and scoring is applied to the evaluation of the classroom teaching performance of college teachers. It reflects the classroom teaching performance of college teachers from three core dimensions: emotional support, classroom organization, and teaching support, thereby effectively depicting the teaching quality and teaching effect of teachers in higher education.

[0060] Second Embodiment Based on the first embodiment, this embodiment provides a model building method for the emotional support dimension. This dimension primarily involves a detailed description of teachers' emotional expressions in the classroom. Teachers' emotional support not only needs to convey emotions through verbal behavior but should also encompass a rich array of nonverbal behaviors. In terms of verbal behavior, this can be described through teachers' verbal feedback, speech habits, verbal emotional state, and paralinguistic features; in terms of nonverbal behavior, it can be demonstrated through teachers' body orientation, facial expressions, facial posture, and other physical characteristics.

[0061] In this embodiment, the first auditory information includes speech feedback, speech habits, speech emotional state, and paralinguistic features. The selection of these indicators is explained below: Verbal feedback from teachers is a crucial step in the teaching process. It affirms students' answers and encourages specific behaviors, generally conveying a positive and encouraging attitude. In the classroom, teachers' evaluations and feedback are timely and typically offer guidance and constructive suggestions, helping students understand their learning progress and reflect on areas for improvement. This research suggests that verbal feedback promotes student participation and comprehension, stimulates learning interest and positive emotions, and enhances self-confidence. It employs Hattie and Timperley's classification of verbal feedback: task feedback, process feedback, learner feedback, and metacognitive feedback. Different scholars categorize verbal feedback differently based on their research needs. In this embodiment, verbal feedback is divided into positive and negative feedback. Positive feedback represents affirmation and agreement, while negative feedback represents negation and disagreement.

[0062] In the interview study, most interviewees mentioned the importance of verbal feedback in classroom teaching. Respondent R1 stated, "Feedback is essential for motivating students in the classroom. When a student gives you feedback, you should also give them corresponding feedback. If they do well, you should encourage them; if they don't do well, you shouldn't criticize or negate them." Respondent R3 stated, "When teachers ask questions in class and students answer, teachers should provide feedback. This not only helps students confirm their understanding but also enhances their learning confidence. Positive feedback, such as 'You answered very well' or 'You understand very well,' can encourage students to continue their efforts. In some cases, appropriate negative feedback is also important, such as pointing out errors or suggesting improvements, helping students recognize problems and correct them. This feedback mechanism, combining positive and negative feedback, can more effectively improve students' learning outcomes." During the research process, by integrating literature, relevant lesson examples, and interview data, the verbal feedback behavioral indicator was determined to include six secondary behavioral indicators: encouraging students, praising students, affirming students, negating students, criticizing students, and mocking students. The preliminary verbal feedback behavioral indicators are shown in Table 1 below: Table 1. Verbal Feedback Behavioral Indicators

[0063] In the behavioral indicator of verbal habits, it is necessary to reflect how teachers create a harmonious atmosphere in the classroom through verbal expression. Teachers' verbal expression habits have a significant impact on the achievement of teaching objectives and the classroom atmosphere. When students answer questions incorrectly, teachers using tolerant, understanding, and gentle language can reduce students' psychological burden of fearing making mistakes. Before formally introducing the relevant teaching content, teachers usually engage in a period of casual conversation. This casual conversation often revolves around topics not directly related to the lesson content, such as discussing recent campus events, interesting anecdotes, or lighthearted topics related to students' personal experiences. Regarding teachers' "verbal expression habits," interviewee R2 stated, "If you ask a question and the students don't respond, shouldn't you guide them, telling them not to be afraid to answer questions, or that it's okay if they answer incorrectly, we're still having a normal lesson, everyone is learning?" Interviewee R5 stated, "Sometimes when you pose a question and the students don't respond, as a teacher, you certainly can't just leave them alone. You need to guide them, for example, by encouraging them to say, 'Don't be afraid to answer incorrectly, class is a practice process, it's okay to answer incorrectly, the important thing is that everyone participates and learns together.'" Teachers using expressions including "we" when sharing their personal experiences and insights can effectively increase intimacy with students. Conversely, if teachers frequently show impatience and a bad attitude when responding to students' questions, it can make students feel looked down upon, dampening their enthusiasm for learning. By reflecting on their verbal expression habits, teachers can adjust their teaching strategies to better meet students' needs in the classroom, breaking down students' stereotypes of teachers and thus establishing a positive image in their minds. Watching recorded classroom teaching videos reveals that some teachers engage in casual conversation during class to adjust the classroom atmosphere. Based on comprehensive analysis, this implementation plan establishes three secondary behavioral indicators under the behavioral indicator of speech habits: tolerance and understanding, collective subject expression, and casual conversation. The preliminary draft of the speech habit behavioral indicators is shown in Table 2: Table 2. Verbal Habitual Behavior Indicators

[0064] Speech is the most common and direct form of human communication, an important tool for expressing thoughts and transmitting information, and a powerful carrier of human emotions and psychological activities. A speaker's speech clearly expresses their views, thoughts, and information, profoundly reflecting their emotional state and psychological activities. For the listener, speech is not only a channel for receiving information but also a crucial way to capture the speaker's emotional information. This transmission and reception of emotions plays a vital role in interpersonal interactions. In classroom teaching, the teacher's verbal behavior is not only a carrier of knowledge but also a manifestation of the teacher's emotional expression. Through the use of language, teachers guide students cognitively and subtly influence their learning attitudes and emotional experiences emotionally. When teachers demonstrate positive emotional attitudes, such as speaking with enthusiasm, affirmation, and joy, students are more willing to actively participate in classroom discussions and interactive activities. This kind of emotional resonance can help students understand knowledge. When encouraging students to express their opinions, teachers' positive verbal expressions, such as "Your idea is very creative" or "I really like your approach," can make students express their opinions more confidently and gradually cultivate their independent thinking ability and innovative awareness. Teachers' verbal behavior, to some extent, subtly guides students' psychological development. On the other hand, teachers maintaining a positive verbal and emotional state in the classroom can create a warm and inclusive learning environment. Conveying encouragement and support through engaging verbal expression makes classroom teaching more effective.

[0065] Teachers' verbal and emotional states in the classroom significantly influence teaching. If teachers exhibit negative emotional states, such as impatience, indifference, or irritability, it can negatively impact the classroom atmosphere. A teacher's impatience or dismissive attitude when students ask questions may cause students to doubt themselves, thereby reducing their willingness to participate in class. Based on this analysis, this implementation plan establishes three secondary behavioral indicators under the behavioral indicator of verbal and emotional states: positive emotions, neutral emotions, and negative emotions, as shown in Table 3. Table 3. Verbal and Affective State Behavioral Indicators

[0066] Paralinguistic features of instruction refer to the various vocal characteristics that accompany a teacher's verbal communication during instruction, such as volume and speaking speed. Maintaining a strong, rhythmic voice during instruction can enhance learners' learning experience and increase satisfaction. It also helps learners focus their attention on the learning content, leading to a deeper understanding and ultimately improving learning outcomes. Whether the volume is muffled or loud, resonant and varied or flat and weak, significantly impacts student listening comprehension and the effectiveness of instruction. In a large classroom environment, experienced teachers often increase their volume and tone to ensure students at the back can hear the content, or to attract their attention. Appropriate use of paralinguistic features can make the content more engaging and interesting. Conversely, inappropriate use of paralinguistic features, such as a flat, monotone tone, can drowsiness and significantly reduce the effectiveness of instruction. Excellent teachers observe student reactions and use feedback to adjust their paralinguistic expressions accordingly.

[0067] In this embodiment, the preliminary proposed paralinguistic features include a teacher's first-level volume, second-level volume, and third-level volume, as well as a first-level speech rate, second-level speech rate, and third-level speech rate. Specifically, first-level volume indicates a teacher's voice is low, second-level volume indicates a teacher's voice is moderately loud, and third-level volume indicates a teacher's voice is relatively loud; first-level speech rate indicates a teacher's speech rate is slow, second-level speech rate indicates a teacher's speech rate is moderate, and third-level speech rate indicates a teacher's speech rate is fast.

[0068] For example, the range of volume level 1 can be set to volume < 60dB; the range of volume level 2 can be set to volume 60dB ≤ volume < 70dB; and the range of volume level 3 can be set to volume 70dB ≤ volume.

[0069] For example, the range of speech speed for Level 1 can be set to speech speed < 120 words / minute; the range of speech speed for Level 2 can be set to speech speed ≤ 120 words / minute ≤ 220 words / minute; and the range of speech speed for Level 3 can be set to speech speed > 220 words / minute.

[0070] As shown in Table 4.

[0071] Table 4. Preliminary draft of paralinguistic characteristic behavior indicators

[0072] In this embodiment, the first visual information includes body orientation, facial emotional state, facial orientation, and body posture.

[0073] Facial orientation is an important nonverbal behavior for teachers to convey emotions and attitudes in the classroom. It refers to the direction in which a teacher's face is turned, closely related to body orientation and facial expressions. For example, when a teacher turns their body towards the student being asked a question, it more effectively conveys a positive and encouraging attitude. This, coupled with a smiling face and a nod, further promotes positive interaction between teachers and students and is an important means of motivating student engagement in class. Especially when students are answering questions or sharing ideas, a teacher's attentive listening and positive responses significantly enhance students' willingness to express themselves. If, during the lesson, a teacher is only focused on their own teaching materials and indifferent to students' answers, or even turns their face away from other students or lowers their head, students may feel ignored, gradually lose confidence in expressing themselves, and become unwilling to participate actively, affecting the teaching effect and the achievement of teaching objectives. Teachers should be aware of the importance of facial orientation in the classroom and provide emotional support to students through positive body language and facial expressions, thereby creating a positive and highly interactive learning atmosphere. The preliminary facial orientation behavior indicators proposed based on the analysis are shown in Table 5 below.

[0074] Table 5. Facial Orientation Behavioral Indicators

[0075] Body posture refers to a teacher's standing, leaning, or sitting posture during the teaching process. Different body postures in the classroom can convey different teaching meanings. An open and relaxed body posture conveys a friendly and kind attitude and signal. For example, an upright and relaxed teaching posture expresses the teacher's positive subjective feelings, can ease the teacher's tension in class, and promote students' active participation. Conversely, a closed teaching posture is often used for defense, aggression, and avoidance. For example, a teacher who frequently sits on the podium during a lecture conveys negative emotions towards students, further affecting students' learning emotions, impacting the classroom atmosphere, and making students unwilling to approach the teacher. A teacher's standing, leaning, or sitting posture, i.e., body posture, can be used as a measure of a teacher's nonverbal affinity. From the perspective of physical education, body language is divided into facial expressions; demonstration actions; sign language and gestures; and interpersonal distance between the teacher and students. Based on the theoretical basis of body language elements, spatial expression system, and paralinguistic expression system, the primary goals of body language are divided into facial expressions, posture, spatial language, demeanor, and vocal language.

[0076] In lesson examples such as C3 and C4, it was observed that teachers maintained a seated posture for extended periods during lectures, with some even leaning against the podium or classroom tables. This limited movement can sometimes affect the dynamism of the classroom, potentially leading to decreased student attention or reduced interaction and activity. This embodiment, drawing on relevant research findings and considering the teaching characteristics of university teachers, categorizes body posture into five secondary behavioral indicators: upright standing, leaning forward, walking, sitting, and leaning while standing. The preliminary body posture behavioral indicators are shown in Table 6.

[0077] Table 6. Body Posture Behavior Indicators

[0078] The summary shows that step S30 includes: S31, based on at least one of the posture features, facial features, and action features, determine the target body orientation classification, target facial emotion state classification, target facial orientation, and target body posture classification of the classroom teacher in the preset classification rules; Furthermore, based on the transcribed text, the target text classification, and the target speech emotion classification, the target speech feedback classification, target speech habit classification, target speech emotion state classification, and target paralinguistic feature classification of the classroom teacher in the preset classification rules are determined; S32, the obtained target body orientation classification, target facial emotion state classification, target facial orientation classification and target body posture classification are used as the first visual target classification; Furthermore, the obtained target speech feedback classification, target speech habit classification, target speech emotional state classification, and target paralinguistic feature classification are used as the first auditory target classification; S33, based on the first visual target classification and the first auditory target classification, evaluate the first teaching performance result.

[0079] Third Embodiment Based on any of the above embodiments, this embodiment provides a model building method in the dimension of classroom organization. In this dimension, the focus is on a deep depiction of the teacher's organizational behavior in the classroom. The teacher's classroom organization performance needs to convey instructions and ask questions through verbal behavior, encompassing the supporting role of nonverbal behavior, to better present their overall performance in classroom management. In terms of verbal behavior, it can be described using two primary behavioral indicators: instructional language and questioning language; while in terms of nonverbal behavior, the teacher's supporting role in the classroom organization process can be demonstrated through auxiliary behaviors.

[0080] In this embodiment, the second auditory information includes commands and inquiries.

[0081] Directives refer to the language used to issue clear instructions or directives. They also include warnings issued by teachers to manage classroom order and prevent student misbehavior. The use of directives may be for timely feedback on undesirable classroom behavior, or it may be preventative, reminding students to concentrate, stop whispering, and not be distracted. Directives also include verbal actions taken by teachers to guide student participation in teaching activities and improve student engagement. This type of language is generally expressed positively, providing students with suggestive instructions. Teachers might guide students to follow the pace of the lesson with phrases like, "Please turn to…", "I hope you will concentrate in this lesson…", or "Please be quiet!" The preliminary directives drafted after comprehensive analysis are shown in Table 7.

[0082] Table 7. Indicators of Command Language Behavior

[0083] In addition to the aforementioned directives, inquiries made by teachers in the classroom to understand students' learning progress or opinions also fall under the category of classroom organizational speech behavior. Inquiries are questions posed by teachers to understand students' learning status or to gain a deeper understanding of their actual situation. Inquiries serve as a bridge for communication and emotional connection between teachers and students. Through effective questioning, teachers can gain a more comprehensive understanding of students' status and needs, creating a more harmonious classroom atmosphere. This helps improve teaching effectiveness and, to some extent, enhances students' self-confidence and sense of belonging in the classroom.

[0084] By using questioning techniques, teachers can assess students' understanding of knowledge points during the teaching process, thereby adjusting teaching content or methods to ensure the effectiveness of knowledge transmission. This embodiment will combine relevant literature analysis on questioning techniques to summarize and derive preliminary behavioral indicators for questioning techniques as shown in Table 8.

[0085] Table 8. Indicators of Inquiry Behavior

[0086] On the other hand, second visual information includes body movements.

[0087] In classroom teaching, teachers' various body language not only supplements verbal language but also serves as an important means to enhance the classroom atmosphere and teaching effectiveness. By using gestures, facial expressions, and postures, teachers can effectively attract students' attention, deepen their understanding of the teaching content, and encourage students to actively participate in classroom interaction. Teachers can actively guide students through body language, such as interactive questioning and gesturing to indicate key points. However, non-teaching behaviors need to be standardized. Teaching behaviors are defined as lecturing, asking questions, writing on the blackboard, and indicating key points, while non-teaching behaviors are defined as constantly bowing one's head and sitting down. This example argues that teachers' gestures are particularly crucial when expressing key content in the classroom. Pointing to keywords on the blackboard can guide students to focus on important information; expanding or waving upwards can imbue the content with emotion, enhancing the vividness and appeal of the knowledge. These body language techniques can effectively convey information and help students understand complex concepts, especially for students with weaker language comprehension abilities, where body language can compensate for the limitations of verbal expression.

[0088] Teachers' auxiliary behaviors in classroom teaching can supplement and enhance verbal expression, helping to better organize classroom activities. By emphasizing key points through gestures or body movements, teachers can control the pace of the entire class. Many respondents indicated that teachers need to use various forms of body language to support verbal expression during lectures. "You have to move around; you can't just stand there and talk. This is how you guide students' attention to the lesson..." "During class, teachers themselves need to be more active; you can't just stand there motionless lecturing. You have to move around and interact to engage students' attention, making it easier for them to focus on the lesson." Based on this information, preliminary auxiliary behavior indicators are shown in Table 9.

[0089] Table 9. Supporting Behavioral Indicators

[0090] After processing, step S40 includes: S41, Based on the posture features and the action features, determine the target limb action category corresponding to the classroom teacher in the preset classification rules; Furthermore, based on the transcribed text and the target text classification, the target instruction language classification and target inquiry language classification corresponding to the classroom teacher in their respective classification rules are determined; S42, classify the obtained target limb movements as the second visual target classification; Furthermore, the obtained target instruction categories and target inquiry categories are used as the second auditory target categories; S43, based on the second visual target classification and the second auditory target classification, evaluate the second teaching performance results.

[0091] Fourth embodiment Based on any of the above embodiments, this embodiment provides a model building method for the teaching support dimension. The teaching support dimension provides an in-depth description of teachers' behaviors in providing teaching assistance in the classroom. In terms of verbal behavior, it mainly describes the types of questions asked. In terms of nonverbal behavior, it demonstrates how teachers adapt to and guide students' learning process in various ways through multiple details such as adaptive behavior, directive behavior, regulatory behavior, instrumental behavior, explanatory behavior, evaluative behavior, and distancing behavior, thereby better supporting the effective conduct of classroom teaching.

[0092] In this embodiment, the third auditory information includes the question type.

[0093] Teacher questioning is the most common interactive method in teaching activities. It is a teaching behavior used to promptly grasp students' learning outcomes and gains, and is an indispensable part of teaching activities. Through questioning, teachers can intuitively understand students' mastery of knowledge points, depth of understanding, and existing problems and confusions, thereby providing targeted guidance and assistance. Questioning can also stimulate students' thinking, encourage them to actively explore and express themselves, effectively improve their initiative and participation in learning, and make the classroom a place for knowledge exploration and intellectual exchange. Based on the nature and purpose of the questions teachers ask during interactions with students, teacher questioning types can be divided into the following five categories: "What is it?", "Why?", "How?", "If?", and "From what reason?". "What is it?" questions are mainly used to confirm students' accuracy in understanding basic concepts, facts, or phenomena; "Why?" questions focus on exploring causes and explaining the principles or laws behind phenomena; "How?" questions guide students to master methods, steps, or skills; "If?" questions encourage students to engage in creative thinking through hypothetical scenarios; and "From what reason?" questions focus on the origin, background, or influencing factors of things. The preliminary behavioral indicators for question types, based on comprehensive analysis, are shown in Table 10.

[0094] Table 10. Question Type Behavioral Indicators

[0095] In this embodiment, the third visual information includes adaptive behavior, indicative behavior, regulatory behavior, instrumental behavior, explanatory behavior, evaluative behavior, and distance behavior.

[0096] Adaptive behavior refers to the physical actions or behaviors that teachers naturally adopt during classroom teaching to meet or adapt to their own physiological, psychological, or objective needs. These behaviors are often gradually formed through long-term teaching practice, better adapting to the classroom environment. From the perspective of their manifestations, teachers' adaptive behavior encompasses a wide range of physical actions. These actions not only reflect the teacher's personal style and teaching habits but are also closely related to the teacher's psychological state, emotional investment, and keen insight into students' learning status. Teachers' adaptive behaviors do not exist in isolation; they often interact with various factors such as teaching content, student responses, and classroom atmosphere, collectively influencing teaching effectiveness. Based on the integration and summarization of relevant research, preliminary adaptive behavior indicators are proposed as shown in Table 11.

[0097] Table 11. Indicators of Adaptive Behavior

[0098] Directive behavior refers to the process by which teachers convey teaching information or guide students' behavior through specific physical actions. Teachers using gestures that have a directive or substitutive function can not only save time and improve efficiency, but also make teaching more intuitive and vivid. These behaviors are characterized by their intuitiveness, immediacy, and contextuality, quickly attracting students' attention, guiding them to focus on key teaching points, or providing clear instructions and guidance in practical activities such as experiments and operations. These behaviors help students understand and perform specific tasks, activities, or behaviors, thereby promoting learning and maintaining classroom order. The core of directive behavior lies in its clarity and directness. In complex and ever-changing teaching environments, teachers need to quickly and accurately convey their teaching intentions and guide students towards predetermined goals. Directive behavior can be one of the most powerful tools for teachers, such as pointing to the blackboard, the projector, or experimental equipment. In actual teaching, directive behavior and instructions often complement and reinforce each other, together constituting an effective strategy for teacher classroom communication. Preliminary indicators for directive behavior are shown in Table 12.

[0099] Table 12. Indicator Behavioral Indicators

[0100] Regulatory behaviors refer to physical actions performed by teachers during the teaching process that are not directly related to the teaching content. These actions are usually actions performed by one part of the body on other parts of the body or external objects, generally short in duration and occurring sporadically. They include habitual or self-adjusting actions, such as touching the nose, scratching the ears, grooming hair, rubbing the eyes, and rubbing the hands. Teachers also habitually manipulate or adjust objects unrelated to teaching, such as chalk, erasers, or teaching aids. During interviews, all respondents mentioned this, and most hoped that these habitual actions could be observed to facilitate subsequent improvement. A preliminary list of regulatory behaviors, based on comprehensive analysis, is shown in Table 13.

[0101] Table 13. Preliminary draft of regulatory behavior indicators

[0102] Instrumental behaviors refer to the various teaching tools teachers use to support and promote student learning, enhance student participation, and further strengthen students' understanding of the teaching content in order to complement verbal instruction. Instrumental behaviors of teachers in the classroom are categorized as follows: writing on the blackboard, erasing the blackboard, operating experimental equipment, operating teaching models, operating multimedia equipment, lecturing with handheld teaching materials, and referring to multimedia equipment. The preliminary instrumental behavior indicators, after summarizing these, are shown in Table 14.

[0103] Table 14. Preliminary Drafting of Instrumental Behavioral Indicators

[0104] Explanatory behavior refers to a teaching strategy in which teachers use body movements to describe, explain, or interpret the postures, actions, or attributes of objects during classroom teaching, thereby helping students better understand and recognize the content being learned. These behaviors go beyond simple verbal descriptions; through a dual encoding of visual and verbal information, students can understand abstract concepts or complex phenomena in a more intuitive and vivid way, thus forming a clearer and more accurate knowledge structure. The core characteristics of explanatory behavior lie in its intuitiveness and explanatoryness. Intuitiveness is reflected in the teacher's use of body movements to simulate or demonstrate the object or process being explained, allowing students to directly observe these actions and establish a visual connection. Explanatoryness lies in the fact that these actions are not merely simple imitations but include the teacher's explanation and description, helping students understand the principles or attributes behind the actions. When implementing explanatory behavior, teachers typically use a series of gestures and actions to assist in the explanation. Commonly used gestures include gestures indicating the height and weight of people, gestures indicating the length, width, and shape of objects, and gestures indicating the movement of people or objects. In summary, the preliminary indicators for explanatory behavior are shown in Table 15.

[0105] Table 15. Preliminary Drafting of Explanatory Behavioral Indicators

[0106] Evaluative behavior refers to the actions of teachers in evaluating and providing feedback on students' learning performance and participation. It reflects the teacher's recognition or correction of the students' learning process and carries the function of monitoring and guiding students' learning progress. After receiving evaluation and feedback from teachers, students can clearly understand their learning status and make necessary adjustments and improvements in subsequent learning. Evaluative behavior can be implemented in various ways, conveying teachers' evaluations and expectations through different forms to stimulate students' intrinsic learning motivation. Four common types of evaluative behavior include giving a thumbs up, clapping, nodding, and shaking one's head. Based on comprehensive analysis, preliminary evaluative behavior indicators are shown in Table 16.

[0107] Table 16. Preliminary Drafting of Evaluative Behavioral Indicators

[0108] Distance-based behavior refers to teachers consciously adjusting the physical distance between themselves and students during the teaching process to influence and control the classroom atmosphere, stimulate students' learning enthusiasm, and maintain good classroom discipline to a certain extent. The characteristic of distance-based behavior lies in its flexibility, mainly manifested in the teacher's ability to flexibly adjust the distance to students according to changes in the classroom situation, thereby adapting to different teaching needs. This behavior can manifest as approaching (reducing) or moving away from (expanding) the distance to students, depending on the teaching objectives and needs. The implementation methods of distance-based behavior are diverse, mainly including walking onto the podium, walking off the podium, walking in the aisle of the classroom, and walking back and forth in the front row of the classroom. By adjusting the distance to students, teachers can convey different information, thereby influencing students' psychological state and behavioral performance. Distance-based behavior helps teachers better observe students' classroom performance and promptly identify and address problems in the classroom. This embodiment, referencing Zhou Pengsheng's classification of distance-based actions into two categories—expanding and reducing distance—considers any movement of the teacher onto the podium as expanding the distance, and any physical movement of the teacher off the podium as reducing the distance. After comprehensive analysis, preliminary distance-based behavior indicators are proposed as shown in Table 17.

[0109] Table 17. Preliminary Drafting of Distance-Related Behavioral Indicators

[0110] Furthermore, this embodiment also proposes a classroom teacher teaching performance characterization model, which includes: The multimodal data classification module is used to identify the image frame data and audio stream data associated with the classroom teacher in the recorded classroom video. The multimodal data extraction module is used to extract the posture features, facial features, and action features of the classroom teacher in the image frame data, and to determine the transcribed text, target text classification, and target speech emotion classification of the classroom teacher in the audio stream data. The emotional support dimension characterization module is used to extract first visual information and first auditory information representing the emotional support characterization dimension from the posture features, facial features, action features, transcribed text, target text classification and target speech emotion classification according to the first preset extraction rules, and to determine the first teaching performance result of the classroom teacher in the emotional support dimension based on the first visual information and the first auditory information. The classroom organization dimension characterization module is used to extract second visual information and second auditory information representing the classroom organization dimension from the posture features, the action features, the transcribed text and the target text classification according to the second preset extraction rules, and to determine the second teaching performance result of the classroom teacher in the second classroom organization dimension based on the second visual information and the second auditory information. The instructional support characterization module is used to extract third visual information and third auditory information representing the instructional support characterization dimension from the posture features, the action features, the transcribed text and the target text classification according to the third preset extraction rules, and to determine the third teaching performance result of the classroom teacher in the instructional support dimension based on the third visual information and the third auditory information. The teaching performance characterization module is used to evaluate the teaching performance characterization results of the classroom teacher based on the first teaching performance result, the second teaching performance result, and the third teaching performance result.

[0111] Furthermore, as another implementation scheme, this embodiment also proposes a method such as... Figure 8 The multimodal data fusion framework shown describes the classroom teaching performance of teachers. In this framework, the image frame data and audio stream data associated with classroom teachers are extracted separately and analyzed from several dimensions such as action recognition, posture estimation, facial expression recognition, speech emotion recognition, paralinguistic feature recognition and text classification. Non-verbal behavior and verbal behavior are divided and aligned in terms of time dimension and behavior dimension.

[0112] Using classroom teaching videos as input, the system extracts image frames and classroom audio information through a teacher target intelligent perception module, which then enters the image modality and audio modality processing paths respectively. The image modality includes teacher posture estimation and facial expression recognition to obtain nonverbal behavior features; the audio modality extracts teacher verbal behavior information through speech recognition, speech emotion recognition, paralinguistic feature recognition, and text classification. All modal features are processed uniformly in the "multimodal feature fusion" module, ultimately aligning and fusing the extracted nonverbal and verbal behaviors across temporal and behavioral dimensions to achieve a comprehensive and multidimensional portrayal of the teacher's classroom teaching behavior.

[0113] As one implementation scheme, Figure 9 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0114] like Figure 9 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0115] Those skilled in the art will understand that Figure 9 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0116] like Figure 9 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0117] exist Figure 9In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0118] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein: When processor 1001 calls a computer program stored in memory 1005, it performs the following operations: S10, determine the image frame data and audio stream data associated with the classroom teacher in the recorded classroom video; S20, extract the posture features, facial features, and motion features of the classroom teacher from the image frame data, and determine the transcribed text, target text classification, and target speech emotion classification of the classroom teacher from the audio stream data; S30, according to the first preset extraction rule, extract first visual information and first auditory information representing the emotional support characterization dimension from the posture features, facial features, action features, transcribed text, target text classification, and target speech emotion classification; and determine the first teaching performance result of the classroom teacher in the emotional support dimension based on the first visual information and the first auditory information; and, S40, according to the second preset extraction rule, extract second visual information and second auditory information representing the classroom organization dimension from the posture features, the action features, the transcribed text, and the target text classification; and determine the second teaching performance result of the classroom teacher in the second classroom organization dimension based on the second visual information and the second auditory information; and, S50, according to the third preset extraction rule, extract the third visual information and the third auditory information that characterize the teaching support dimension from the posture features, the action features, the transcribed text and the target text classification, and determine the third teaching performance result of the classroom teacher in the teaching support dimension based on the third visual information and the third auditory information. S60, based on the first teaching performance result, the second teaching performance result, and the third teaching performance result, evaluate the teaching performance characterization result of the classroom teacher.

[0119] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0120] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the classroom teacher teaching performance characterization method based on multimodal data fusion as described in the above embodiments.

[0121] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0122] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0123] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0127] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0128] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0129] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for characterizing classroom teacher teaching performance based on multimodal data fusion, characterized in that, The method includes the following steps: S10, determine the image frame data and audio stream data associated with the classroom teacher in the recorded classroom video; S20, extract the posture features, facial features, and motion features of the classroom teacher from the image frame data, and determine the transcribed text, target text classification, and target speech emotion classification of the classroom teacher from the audio stream data; S30, according to the first preset extraction rule, extract first visual information and first auditory information representing the emotional support characterization dimension from the posture features, facial features, action features, transcribed text, target text classification, and target speech emotion classification; and determine the first teaching performance result of the classroom teacher in the emotional support dimension based on the first visual information and the first auditory information; and, S40, according to the second preset extraction rule, extract second visual information and second auditory information representing the classroom organization dimension from the posture features, the action features, the transcribed text, and the target text classification; and determine the second teaching performance result of the classroom teacher in the second classroom organization dimension based on the second visual information and the second auditory information; and, S50, according to the third preset extraction rule, extract the third visual information and the third auditory information that characterize the teaching support dimension from the posture features, the action features, the transcribed text and the target text classification, and determine the third teaching performance result of the classroom teacher in the teaching support dimension based on the third visual information and the third auditory information. S60, based on the first teaching performance result, the second teaching performance result, and the third teaching performance result, evaluate the teaching performance characterization result of the classroom teacher.

2. The method as described in claim 1, characterized in that, The first visual information includes body orientation, facial emotional state, facial orientation, and body posture; the first auditory information includes speech feedback, speech habits, speech emotional state, and paralinguistic features; step S30 includes: S31, based on at least one of the posture features, facial features, and action features, determine the target body orientation classification, target facial emotion state classification, target facial orientation, and target body posture classification of the classroom teacher in the preset classification rules; Furthermore, based on the transcribed text, the target text classification, and the target speech emotion classification, the target speech feedback classification, target speech habit classification, target speech emotion state classification, and target paralinguistic feature classification of the classroom teacher in the preset classification rules are determined; S32, the obtained target body orientation classification, target facial emotion state classification, target facial orientation classification and target body posture classification are used as the first visual target classification; Furthermore, the obtained target speech feedback classification, target speech habit classification, target speech emotional state classification, and target paralinguistic feature classification are used as the first auditory target classification; S33, based on the first visual target classification and the first auditory target classification, evaluate the first teaching performance result.

3. The method as described in claim 2, characterized in that, The first visual target classification includes at least one of the following: Body orientation can be categorized as facing the student, sideways to the student, or with one's back to the student. Facial emotional states can be categorized into positive emotions, neutral emotions, and negative emotions. Facial orientation can be categorized as facing students or facing other objects; Body postures can be categorized into standing upright, leaning forward, walking, sitting, and leaning. Furthermore, the first auditory target classification includes at least one of the following: Verbal feedback can be categorized into encouragement, praise, affirmation, negation, criticism, and sarcasm. Speech habits can be categorized into mild, collective subject expressions, and small talk; Verbal affective states can be categorized into positive affect, neutral affect, and negative affect. The paralinguistic features are classified into Level 1 volume, Level 2 volume, Level 3 volume, Level 1 speech rate, Level 2 speech rate, and Level 3 speech rate.

4. The method as described in claim 1, characterized in that, The second visual information includes body movements, the second auditory information includes commands and inquiries, and step S40 includes: S41, Based on the posture features and the action features, determine the target limb action category corresponding to the classroom teacher in the preset classification rules; Furthermore, based on the transcribed text and the target text classification, the target instruction language classification and target inquiry language classification corresponding to the classroom teacher in their respective classification rules are determined; S42, classify the obtained target limb movements as the second visual target classification; Furthermore, the obtained target instruction categories and target inquiry categories are used as the second auditory target categories; S43, based on the second visual target classification and the second auditory target classification, evaluate the second teaching performance results.

5. The method as described in claim 4, characterized in that, The second visual target classification includes at least one of the following: Body language can be categorized into tapping the blackboard, tapping the lectern, starting a class activity, ending a class activity, asking students to answer questions, and hand gestures. Furthermore, the second auditory target classification includes at least one of the following: Instructions can be categorized as follows: start and end of class, start and end of teaching activities, practice, discussion, random questioning, designated questioning, quiet, and focus attention. The categories of inquiries include comments on assignments, deadlines, discussion topics, and exams.

6. The method as described in claim 1, characterized in that, The third visual information includes adaptive behavior, indicative behavior, regulatory behavior, instrumental behavior, explanatory behavior, evaluative behavior, and distance-related behavior; the third auditory information includes question type; and step S50 includes: S51, based on the posture characteristics and the action characteristics, determine the target adaptive behavior, target indicative behavior, target regulatory behavior, target instrumental behavior, target explanatory behavior, target evaluative behavior, and target distance behavior that the classroom teacher satisfies; Furthermore, based on the transcribed text and the target text classification, the target question types satisfied by the classroom teacher are determined; S52, the target adaptive behavior, the target indicative behavior, the target regulatory behavior, the target instrumental behavior, the target explanatory behavior, the target evaluative behavior, and the target distance behavior are used as the third visual target classification for the classroom teacher; Furthermore, the target question type is used as the third auditory target classification for the classroom teacher; S53, based on the third visual target classification and the third auditory target classification, evaluate the classroom teacher's third teaching performance results in the teaching support dimension.

7. The method as described in claim 6, characterized in that, Adaptive behaviors include placing hands behind the back, crossing arms, hands on hips, hands in pockets, supporting oneself on a lectern, standing with legs crossed, placing hands in front of the abdomen, and placing hands in front of the body; Directive behaviors include pointing to the blackboard, pointing to the projector, and pointing to experimental instruments; Regulatory behaviors include habitual behaviors, adjustment behaviors, dressing, and fiddling with unrelated objects; Instrumental behaviors include writing on the blackboard, erasing the blackboard, operating experimental equipment, operating teaching models, operating multimedia equipment, lecturing with handheld teaching materials, and viewing multimedia equipment; Explanatory behaviors include hand gestures, object gestures, and movement gestures; Evaluative behaviors include giving a thumbs-up, clapping, nodding, and shaking one's head; Distance behaviors include walking up to the podium, walking down from the podium, walking on the podium, walking in the front row of the classroom, and walking in the aisle of the classroom; Question types include what type of question, why type of question, how type of question, if type of question, and by what type of question.

8. A model for depicting classroom teacher performance, characterized in that, The classroom teacher performance characterization model includes: The multimodal data classification module is used to identify the image frame data and audio stream data associated with the classroom teacher in the recorded classroom video. The multimodal data extraction module is used to extract the posture features, facial features, and action features of the classroom teacher in the image frame data, and to determine the transcribed text, target text classification, and target speech emotion classification of the classroom teacher in the audio stream data. The emotional support dimension characterization module is used to extract first visual information and first auditory information representing the emotional support characterization dimension from the posture features, facial features, action features, transcribed text, target text classification and target speech emotion classification according to the first preset extraction rules, and to determine the first teaching performance result of the classroom teacher in the emotional support dimension based on the first visual information and the first auditory information. The classroom organization dimension characterization module is used to extract second visual information and second auditory information representing the classroom organization dimension from the posture features, the action features, the transcribed text and the target text classification according to the second preset extraction rules, and to determine the second teaching performance result of the classroom teacher in the second classroom organization dimension based on the second visual information and the second auditory information. The instructional support characterization module is used to extract third visual information and third auditory information representing the instructional support characterization dimension from the posture features, the action features, the transcribed text and the target text classification according to the third preset extraction rules, and to determine the third teaching performance result of the classroom teacher in the instructional support dimension based on the third visual information and the third auditory information. The teaching performance characterization module is used to evaluate the teaching performance characterization results of the classroom teacher based on the first teaching performance result, the second teaching performance result, and the third teaching performance result.

9. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the classroom teacher teaching performance characterization method based on multimodal data fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the classroom teacher teaching performance characterization method based on multimodal data fusion as described in any one of claims 1 to 7.

Images