Intelligent classroom instant feedback system and method based on multi-modal interaction

Abstract

The invention discloses an intelligent classroom instant feedback system and method based on multi-modal interaction, and the system comprises a multi-modal data collection module which is used for collecting the facial expression, voice and intonation, limb movement, eye movement track and classroom interaction text of a student; the data processing and analyzing module is used for carrying out preprocessing and feature extraction on the collected data; the state recognition and modeling module is used for fusing the multi-modal features and recognizing the learning state of the student; the instant feedback module is used for feeding back the recognition result to the teacher in a visual or voice form; the teaching intervention suggestion generation module is used for generating teaching strategy suggestions according to the recognition result; the data storage and management module is used for storing historical data and supporting subsequent analysis; according to the invention, fusion analysis of multi-modal data is realized, and the accuracy and robustness of student classroom state identification are improved; and a real-time feedback mechanism is provided, so that the teacher can timely master the learning condition of the student, and the teaching quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of educational informatization technology, and specifically relates to an intelligent classroom instant feedback system and method based on multimodal interaction. Background Art

[0002] With the rapid development of educational informatization, classroom teaching is gradually evolving towards intelligent, personalized, and interactive learning. In traditional teaching models, teachers rely primarily on classroom questions, homework assignments, and periodic tests to understand students' learning status. These methods lack effective means of perceiving students' cognitive states and emotional responses during real-time classroom learning. This results in delayed feedback, making it difficult to adjust teaching strategies in a timely manner, and impacting teaching effectiveness.

[0003] In recent years, the continuous maturity of technologies such as artificial intelligence, computer vision, speech recognition, and natural language processing has made it possible to build more intelligent teaching environments. In the field of human-computer interaction, in particular, multimodal data fusion (such as facial expressions, voice intonation, body movements, and physiological signals) has been widely applied in scenarios such as affective computing and user experience analysis, significantly improving the system's perception capabilities and interactive experience.

[0004] In educational settings, some research has attempted to use single-modality data (such as speech or images) for detecting student attention, identifying emotions, or assessing student engagement. However, these approaches are often limited by the incompleteness of a single information source and are susceptible to factors such as environmental noise and individual differences, resulting in inaccurate and incomplete feedback. Furthermore, existing systems mostly focus on data analysis and post-event feedback, lacking mechanisms for immediate response to classroom instruction, and thus unable to achieve dynamic intervention and optimization during the teaching process.

[0005] Therefore, there is an urgent need for an intelligent classroom instant feedback system and method that can integrate multiple interaction modes, collect and analyze students' classroom behavior data in real time, and quickly provide feedback to teachers to support their teaching decisions. Summary of the Invention

[0006] (1) Technical problems solved In response to the shortcomings of the existing technology, the present invention aims to provide an intelligent classroom instant feedback system and method based on multimodal interaction, which can collect students' various behavioral data during classroom teaching in real time, integrate multimodal information such as vision, hearing, and text, analyze students' cognitive state, emotional changes and participation, and provide instant feedback to teachers through visual interfaces or voice prompts, thereby realizing dynamic optimization of teaching strategies and personalized guidance.

[0007] (2) Technical solution To achieve the above objectives, the present invention provides the following technical solutions: a smart classroom instant feedback system and method based on multimodal interaction, comprising: Multimodal data collection module, used to collect students' facial expressions, voice intonation, body movements, eye movement trajectories and classroom interaction texts; Data processing and analysis module, used for preprocessing and feature extraction of collected data; State recognition and modeling module, used to fuse multimodal features and identify students' learning states; Instant feedback module, used to provide recognition results to teachers in the form of visualization or voice; The teaching intervention suggestion generation module is used to generate teaching strategy suggestions based on the recognition results; Data storage and management module, used to save historical data and support subsequent analysis.

[0008] Preferably, the multimodal data acquisition module includes one or more combinations of a camera, a microphone, an eye tracker, an electronic answerer and an infrared sensor.

[0009] Preferably, the state recognition and modeling module uses a Transformer-based multimodal fusion model to predict student states.

[0010] Preferably, the instant feedback module includes a graphical user interface and a voice broadcast device, which supports real-time display of students' attention index, emotional tendency and participation level.

[0011] Preferably, the teaching intervention suggestion generation module outputs teaching strategy suggestions based on the teaching knowledge base and expert rules.

[0012] Preferably, the system supports two operating modes: local deployment and cloud service, and is suitable for face-to-face classroom, distance learning and hybrid teaching scenarios.

[0013] An instant feedback method for smart classroom based on multimodal interaction includes the following steps: S1: Start the system and enter classroom monitoring mode; S2: Collect students’ multimodal data; S3: Preprocessing and feature extraction of collected data; S4: Fusion of multimodal features to identify students’ learning status; S5: Feedback the recognition results to the teacher; S6: Generate instructional intervention suggestions; S7: Record the entire process data for subsequent analysis.

[0014] (3) Beneficial effects Compared with the prior art, the present invention provides an intelligent classroom instant feedback system and method based on multimodal interaction, which has the following beneficial effects: The present invention realizes the fusion analysis of multimodal data, improves the accuracy and robustness of the recognition of students' classroom status; provides a real-time feedback mechanism, which helps teachers to grasp students' learning situation in a timely manner and improve teaching quality; supports intelligent recommendation of teaching strategies, and enhances the pertinence and effectiveness of teaching; the system has good scalability and compatibility, and can adapt to different teaching scenarios and hardware equipment; it can serve as an important part of the smart education platform to promote the development of educational informatization. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 This is a structural block diagram of the intelligent classroom instant feedback system based on multimodal interaction according to the present invention; Figure 2 It is a flow chart of the method described in the present invention. DETAILED DESCRIPTION

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0017] Specific examples are given below.

[0018] Example 1: Real-time feedback application in primary and secondary school classroom teaching scenarios Background: In traditional primary and secondary school classrooms, teachers mainly understand students' learning status through questioning and observation. However, due to time and energy constraints, it is difficult to make a comprehensive judgment on each student's learning situation.

[0019] Implementation process: Deploy cameras, microphone arrays, and electronic answerers in the classroom; the system automatically collects data such as students' facial expressions, voice intonation, and answering behaviors; the data processing module denoises the images and extracts micro-expression features; performs emotion recognition on voice signals; and counts the accuracy of answers; the multimodal fusion model comprehensively analyzes the above information and outputs each student's attention level (high / medium / low) and emotional state (positive / neutral / negative); the instant feedback interface displays a heat map of the overall status of the student group, allowing teachers to quickly identify students with decreased attention or abnormal emotions; the teaching suggestion generation module prompts based on the overall trend: "The current class's attention is generally low, and it is recommended to increase interactive sessions"; teachers adjust the teaching rhythm accordingly, insert group discussions or questions, and improve classroom activity.

[0020] This embodiment significantly improves teachers' ability to perceive students' learning status, helps them adjust teaching strategies in a timely manner, and improves classroom efficiency and teaching quality.

[0021] Example 2: Real-time Participation Evaluation in University Remote Live Classrooms Background: Online education is becoming increasingly popular, but during remote teaching, teachers cannot directly observe students' performance and lack an immediate feedback mechanism.

[0022] Implementation process: Students access the course through the video conferencing platform, and the system enables camera and microphone permissions; students' facial expressions, head posture, eye direction and voice reactions are collected in real time; eye tracking algorithms analyze whether students' gaze is focused on the lecture content; students' classroom participation is evaluated by combining voice activity and response speed; the system generates a participation score for each student and displays it to the teacher in the form of a dashboard; when a student shows low participation for consecutive times, the system automatically reminds the teacher to pay attention to the student; after class, the teacher can view each student's participation trend through the data management module for personalized tutoring.

[0023] This embodiment solves the problem of teachers being insensitive to students' status in distance learning, and improves the teaching interactivity and pertinence of online classes.

[0024] Example 3: Emotion Recognition and Intervention in Special Education Contexts Background: In special education settings, such as classes for children with autism, students' emotional expressions are often unclear or difficult to understand, making conventional teaching feedback ineffective.

[0025] Implementation process: Install non-contact sensors (such as infrared cameras and physiological signal acquisition equipment) in the classroom; the system collects multi-dimensional data such as students' heart rate, skin electrode response, and facial expression changes; uses deep neural networks to model emotions based on physiological signals and identify students' emotional fluctuations; combines visual data analysis to determine whether students are in a state of anxiety, irritability, or excitement; the system prompts the teaching assistant through voice broadcast: "Student A's current heart rate is elevated and may be in an anxious state. Please comfort him / her"; teachers or teaching assistants can take appropriate intervention measures based on the prompts, such as adjusting teaching content or providing psychological support; all data are uploaded to the cloud synchronously for the expert team to conduct long-term behavioral pattern analysis.

[0026] This embodiment provides a scientific emotion recognition tool for special education, helping teachers to more accurately understand and respond to students' psychological needs, thereby improving the quality of education and care.

[0027] Example 4: Integrated application of smart education platform Background: With the advancement of smart campus construction, various teaching systems are gradually integrated, and a unified data analysis platform is urgently needed to assist teaching decision-making.

[0028] Implementation process: Integrate the system of the present invention into the school's smart education platform; the platform connects to the existing teaching system (such as the academic affairs system and learning management system LMS); the system continuously collects multimodal data in the classroom and combines it with students' historical grades, attendance records, etc.; after each class, the system automatically generates a classroom feedback report, including: students' overall attention curve; emotional change trends; participation distribution chart; teaching suggestion summary; the report is automatically pushed to the teacher's side and the academic affairs management system for teaching evaluation; teaching and research personnel can use historical data to analyze the effects of different teaching methods and optimize teaching design; the system supports API interfaces, which facilitates linkage with other AI teaching assistants and virtual teacher systems.

[0029] This embodiment realizes closed-loop management from data collection to teaching optimization, promotes the digital transformation of education, and improves the level of intelligent teaching.

[0030] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

Claims

1. An intelligent classroom instant feedback system based on multimodal interaction, characterized by: include: Multimodal data collection module, used to collect students' facial expressions, voice intonation, body movements, eye movement trajectories and classroom interaction texts; Data processing and analysis module, used for preprocessing and feature extraction of collected data; State recognition and modeling module, used to fuse multimodal features and identify students' learning states; Instant feedback module, used to provide recognition results to teachers in the form of visualization or voice; The teaching intervention suggestion generation module is used to generate teaching strategy suggestions based on the recognition results; Data storage and management module, used to save historical data and support subsequent analysis.

2. The system according to claim 1, wherein: The multimodal data acquisition module includes one or more combinations of a camera, a microphone, an eye tracker, an electronic answerer and an infrared sensor.

3. The system according to claim 1, wherein: The state recognition and modeling module uses a Transformer-based multimodal fusion model to predict student states.

4. The system according to claim 1, wherein: The instant feedback module includes a graphical user interface and a voice broadcast device, which supports real-time display of students' attention index, emotional tendency and participation level.

5. The system according to claim 1, wherein: The teaching intervention suggestion generation module outputs teaching strategy suggestions based on the teaching knowledge base and expert rules.

6. The system according to claim 1, wherein: The system supports two operating modes: local deployment and cloud service, and is suitable for face-to-face classroom, distance learning and hybrid teaching scenarios.

7. A smart classroom instant feedback method based on multimodal interaction, characterized in that: The steps include: S1: Start the system and enter classroom monitoring mode; S2: Collect students’ multimodal data; S3: Preprocessing and feature extraction of collected data; S4: Fusion of multimodal features to identify students’ learning status; S5: Feedback the recognition results to the teacher; S6: Generate instructional intervention suggestions; S7: Record the entire process data for subsequent analysis.

Images