Evaluation method of student participation degree automatic evaluation system based on deep learning

By combining SAMSA-Net and YOLO models into a deep learning system, the problems of neglecting spatiotemporal data and insufficient robustness in student engagement assessment in existing technologies are solved. This enables multi-dimensional engagement assessment and real-time feedback, thereby improving the teaching effectiveness and learning quality of online education.

CN118918634APending Publication Date: 2024-11-08ANHUI NORMAL UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410814214.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-24
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing online education systems neglect spatiotemporal data analysis when assessing student engagement, relying on a single model or data source, which lacks robustness and real-time performance, making it difficult to accurately capture changes in students' emotions and behaviors.

Method used

An automatic student engagement assessment system based on deep learning is adopted, which combines the SAMSA-Net sentiment analysis model and the YOLO behavior recognition model. It analyzes students' emotional and behavioral engagement from multiple dimensions, generates real-time feedback, and provides personalized records.

Benefits of technology

It enables comprehensive and accurate assessment of student engagement, provides real-time feedback and personalized analysis, and helps teachers and students adjust their teaching and learning strategies in a timely manner to improve learning efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918634A_ABST
    Figure CN118918634A_ABST
Patent Text Reader

Abstract

The invention discloses an evaluation method of a student participation degree automatic evaluation system based on deep learning, and the method comprises the following steps: S1, enabling the system to collect video stream data, and carrying out the preprocessing of the video stream data, and generating a plurality of sub-video sequences; s2, inputting the sub-video sequences into an emotion analysis model for emotion participation degree evaluation; a YOLO model is adopted to identify the behavior mode of the student, and the behavior participation degree of the student is evaluated; s3, based on the analysis results of the emotion participation degree and the behavior participation degree, the classification weights of emotion analysis and behavior recognition are integrated, an overall participation degree evaluation result is calculated, and feedback information is generated; s4, visually displaying an overall participation degree evaluation result and feedback information in multiple forms; and S5, the system performs personalized recording of the detection result for storing and managing the detection result. By means of the information, an educator can adjust teaching strategies and teaching material contents more flexibly and accurately, so that the teaching effect is practically improved, the learning experience of students is optimized, and the learning efficiency of the students is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to an evaluation method for an automatic student engagement evaluation system based on deep learning. Background Art

[0002] The evolution of online education from an emergency plan to a core component of the education sector highlights its importance as a bridge, connecting students in resource-poor areas with quality learning opportunities. Online education is not only a key strategy to cope with future uncertainties, but also an effective way to promote global education equity and universalization.

[0003] As the digitalization process continues to accelerate, users' interactive behaviors in online education, e-commerce and other fields are increasing. Especially in the field of education, accurate evaluation of user engagement is crucial to improving the quality of interaction, user experience and human-computer interaction efficiency. Teachers need to understand students' participation in real time, which has a direct impact on teaching effectiveness and students' learning motivation.

[0004] Systems that automatically detect student engagement can not only improve learning quality, but also help teachers provide more personalized teaching resources, thereby achieving personalized and precise education. Through these systems, teachers can promptly identify students' needs and difficulties, adjust teaching methods, and enhance students' learning enthusiasm and effectiveness. The application of this technology will further promote the development of online education and improve the overall level of global education.

[0005] There is no unified definition of learning engagement in existing technologies. The commonly used definition was proposed by Fredricks et al., which regards engagement as a comprehensive attribute that includes three dimensions: behavior, emotion, and cognition. Among them, behavioral engagement focuses on participation behaviors, such as attention and questioning; emotional engagement involves students' emotional reactions such as interest or boredom; and cognitive engagement includes internal cognitive processes such as learning and self-regulation.

[0006] There are currently a variety of automated methods for measuring engagement, including those based on behavior, sensors, and computer vision. Among these methods, computer vision methods based on facial regions have attracted much attention. The face contains rich emotional information, so it is of great significance to use facial features to predict engagement. Face-based engagement prediction methods can be divided into traditional computer vision techniques and deep learning-based methods. The former usually designs action units (AUs) or hand-crafted patterns to obtain facial features and uses machine learning to estimate engagement; the latter uses deep learning to capture fine-grained facial movements, such as eye gaze and head posture, and predict engagement. These automated methods have the advantages of fast speed and high temporal resolution, and are gradually gaining attention and becoming a research hotspot. Their applications in classroom behavior analysis and engagement estimation are gradually increasing.

[0007] However, the existing methods have the following disadvantages:

[0008] 1. Ignoring the problem of spatiotemporal data analysis: Many existing systems rely on a single image or video frame to evaluate student engagement, ignoring the fact that engagement is a data problem that varies over time and space. Student engagement varies over time and space, and analysis based solely on a single time point may not fully grasp the true state of student engagement.

[0009] 2. Limitations of existing systems: Existing systems usually rely on a single mode or a single data source for analysis, for example, using only facial expression recognition (single mode) or only image data (single data source) for student engagement assessment. However, learning engagement is a multi-dimensional composite attribute that requires comprehensive assessment from multiple perspectives. Analysis of single mode and single data source may lead to an incomplete understanding of student engagement.

[0010] 3. Robustness and real-time accuracy challenges: Existing methods lack robustness in the wild and have difficulty accurately predicting students’ engagement levels. Video-based behavior and facial expression analysis faces challenges in real-time and accuracy, especially when identifying students’ complex emotional states and subtle behavioral changes. Current methods may not be able to capture students’ emotional and behavioral changes instantly and accurately, thus affecting accurate assessment and timely feedback of student engagement. Summary of the invention

[0011] In order to solve the existing problems, the present invention provides an evaluation method of a student engagement automatic evaluation system based on deep learning. The specific scheme is as follows:

[0012] An evaluation method for a student engagement automatic evaluation system based on deep learning, comprising the following steps:

[0013] S1, the system collects video stream data and preprocesses it; wherein the preprocessing method is: using the MTCNN algorithm to extract face images from the original frame and generate multiple sub-video sequences;

[0014] S2, input the generated sub-video sequence into the sentiment analysis model to evaluate the students’ emotional engagement; at the same time, input the preprocessed full image into the YOLO model to identify the students’ behavior patterns to evaluate their behavioral engagement;

[0015] S3, based on the above analysis results of emotional engagement and behavioral engagement, the classification weights of emotional analysis and behavioral recognition are integrated to calculate the overall engagement evaluation result and generate feedback information;

[0016] S4, uses various forms to visualize the overall engagement evaluation results and feedback information, and provides an easy-to-understand user interface;

[0017] S5, the system makes personalized records of the test results for storage and management, facilitating long-term tracking and analysis.

[0018] Preferably, the specific steps of constructing multiple sub-videos include:

[0019] S11, performing video downsampling by selecting a frame interval to reduce the number of video frames;

[0020] S12, dividing the downsampled video into multiple sliding windows;

[0021] S13, select representative frames in each sliding window to construct multiple sub-videos.

[0022] Preferably, in step S2, the sentiment analysis model is a deep learning model using SAMSA-Net.

[0023] Preferably, the visualized content in step S4 includes the detection box, the classification label and the final engagement score, and the visualization form includes but is not limited to heat map, chart and text annotation.

[0024] Preferably, a system for executing any of the above-mentioned evaluation methods comprises:

[0025] Video acquisition module, used for acquiring video stream data;

[0026] A face detection module is used to perform face detection from the original frames of the video stream data using the MTCNN algorithm, extract face images, and generate multiple sub-video sequences;

[0027] The sentiment analysis module is used to input the sub-video sequence into the sentiment analysis model to evaluate the students' emotional engagement;

[0028] The behavior recognition module is used to identify students’ behavior patterns using YOLO technology, thereby evaluating their behavioral engagement;

[0029] The final engagement result generation and feedback module is used to calculate the overall engagement evaluation results and generate feedback information based on the analysis results of emotional engagement and behavioral engagement;

[0030] The human-computer interaction module uses various forms to visualize the overall engagement evaluation results and feedback information, and provides an easy-to-understand user interface;

[0031] Data logging module, used to store and manage personalized records of test results for long-term tracking and analysis.

[0032] Preferably, the video acquisition module, facial detection module, emotion analysis module, behavior recognition module, final engagement result generation and feedback module, and human-computer interaction module communicate with each other through an API, so that the system can be flexibly deployed and expanded on different platforms.

[0033] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. After the computer program is run, any of the above methods is executed.

[0034] The present invention also discloses a computer system, including a processor and a storage medium, wherein a computer program is stored on the storage medium, and the processor reads and runs the computer program from the storage medium to execute any of the methods described above.

[0035] The beneficial effects of the present invention are:

[0036] (1) Assessment of emotional engagement: This paper uses a deep learning model called SAMSA-Net. The system can perceive fine-grained engagement status and improve the accuracy of emotional engagement assessment.

[0037] (2) Assessment of behavioral engagement: The YOLO model-based method ensures high speed and accuracy of behavioral pattern recognition, providing reliable support for assessing students’ behavioral engagement.

[0038] (3) Multi-dimensional engagement assessment: This system establishes a comprehensive engagement assessment framework by integrating the two dimensions of engagement analysis: emotion and behavior. This comprehensive assessment not only provides an in-depth understanding of students’ engagement from multiple perspectives, but also provides comprehensive and in-depth personalized analysis of the test records.

[0039] (4) Diverse visualization and feedback: Feedback is provided in various forms such as heat maps and text reminders, so that teachers and students can understand the level of participation intuitively and timely and make corresponding adjustments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 Flowchart of the system for assessing student engagement;

[0042] Figure 2 Figure 2 is the SAMSA-Net emotion recognition model diagram. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The present invention provides an evaluation method for an automatic student engagement evaluation system based on deep learning, the core of which includes the SAMSA-Net emotion recognition model, which can process and analyze fine-grained emotional changes in facial expressions to evaluate emotional engagement. At the same time, the system integrates behavioral analysis functions and uses the YOLO algorithm to identify and evaluate students' behavioral patterns. By combining multi-dimensional analysis of emotions and behaviors, the system can comprehensively evaluate students' engagement and provide real-time visual feedback and detailed engagement evaluation reports. Such a design is intended to provide educators with instant information to help them adjust teaching strategies and content according to students' engagement levels to improve students' learning efficiency.

[0045] Specifically, Figure 1 and Figure 2 , an evaluation method of a student engagement automatic evaluation system based on deep learning, comprising the following steps:

[0046] S1, the system collects video stream data and preprocesses it, wherein the preprocessing method is: using the MTCNN algorithm to extract face images from the original frames and generate multiple sub-video sequences.

[0047] The specific steps of constructing multiple sub-videos include:

[0048] S11, performing video downsampling by selecting a frame interval 'frame_interval' to reduce the number of video frames;

[0049] S12, divide the downsampled video into 'num_windows' sliding windows;

[0050] S13, select representative frames in each sliding window to construct multiple sub-videos.

[0051] S2, input the generated sub-video sequence into the sentiment analysis model to evaluate the students' emotional engagement. The sentiment analysis model is a deep learning model using SAMSA-Net, which is used to analyze the spatiotemporal characteristics of the video sequence.

[0052] In a digital learning environment, automatically detecting student engagement is the key to improving teaching quality and achieving personalized learning experience. The engagement of students in online classes is not only a display of emotional behavior across time and space, but also a complex spatiotemporal data analysis problem. In order to accurately extract student engagement from video data, this study not only analyzes the student's behavioral state in each video frame, but also carefully examines the dynamic changes of the state between consecutive frames, thereby achieving real-time and accurate evaluation of student engagement.

[0053] Using the SAMSA-Net deep learning model, the system is able to perceive fine-grained engagement states and improve the accuracy of emotional engagement assessment.

[0054] like Figure 2 As shown, SAMSA-Net is introduced as follows:

[0055] (1) Processing of spatially dependent data:

[0056] Multi-scale convolutional feature extraction: We use convolution kernels of different sizes C(V) = {c1(V), c2(V), ..., cM(V)} to capture multi-scale spatial features, where M represents the number of convolutional layers.

[0057] SE attention mechanism: After each convolutional layer output, the SE module SE(cm(V)) is applied to rescale the features, emphasizing important information and suppressing unimportant information.

[0058] MLP branch processing: The parallel MLP branch MLP(V) processes the flattened features to capture global information that may be missed by the convolutional layer.

[0059] Gating unit: The gating unit G(C(V),MLP(V)) controlled by the sigmoid function improves the flexibility and accuracy of the model in integrating features.

[0060] Feature fusion: The final feature fusion is expressed as F = G(C(V), MLP(V)), which integrates multi-scale features and global features to enhance the understanding of complex time series patterns.

[0061] (2) Processing of time-dependent data:

[0062] The model uses the gated recurrent unit GRU and the self-attention mechanism to handle temporal dependencies and effectively capture the long-term and short-term dependencies of engagement. The temporal feature extraction function T(G(C(V),MLP(V))) is defined, combining the outputs of GRU and the self-attention mechanism.

[0063] SAMSA-Net enhances the feature extraction capability by integrating the multi-scale channel attention fusion module MSCAFM, and combines the gated recurrent unit GRU and the self-attention mechanism to effectively process the time dimension of the data, optimizing the ability to capture the changes in students' emotional state over time. Using this series of extracted facial feature sequences, the model can accurately evaluate students' emotional engagement and divide students' emotional engagement into four levels from low to high: very low engagement (0), low engagement (1), high engagement (2), and very high engagement (3).

[0064] This model is trained based on two representative field participation datasets: the DAiSEE dataset and the EmotiW-EP dataset, ensuring the accuracy and reliability of its evaluation.

[0065] Specifically, the DAiSEE dataset consists of 9068 video clips produced by 112 students in a natural learning environment. Each clip is 112 seconds long, with a resolution of 1920×1080 pixels and a frame rate of 30 frames per second. These videos record students' reactions to educational content in detail. According to the degree of engagement, the videos are annotated into four levels of engagement: very low engagement (0), low engagement (1), high engagement (2), and very high engagement (3). In order to enhance the generalization ability of the model, we introduce the EmotiW-EP dataset. This dataset focuses on emotional expression and collects the reactions of 78 adult subjects in a natural environment, including 147 training videos and 115 test and validation videos. The video resolution is 640x480 pixels. These videos are annotated into different levels of engagement from complete disengagement to high engagement.

[0066] The annotation of both datasets relies on group assessments and is closely tied to the gold standard developed by a team of psychology experts, ensuring objectivity and accuracy in annotations. Labels are based not only on observed emotional responses, but also on facial muscle movements and other non-verbal behavioral factors, providing a comprehensive and detailed perspective on student engagement assessment. The combination of these two datasets not only provides rich real-world scenario data, but also enhances the model’s ability to identify and predict student engagement. By applying and evaluating the methods of the present invention on these datasets, it is possible to significantly improve recognition accuracy and set new research standards in the field of automatic detection of online learning engagement.

[0067] At the same time, YOLO technology is used to monitor and identify students' behavior patterns to evaluate their behavioral engagement. The YOLO algorithm can preferably use YOLO v7, YOLO v8 or YOLO v9. These behavior patterns cover a variety of actions such as using mobile phones, raising hands, sleeping, reading, taking notes, thinking, etc. The system accurately identifies these behaviors by training the self-collected data sets. By analyzing the frequency and patterns of these behaviors, the system can evaluate students' behavioral engagement. The method based on the YOLO model ensures the high speed and accuracy of behavioral pattern recognition, providing reliable support for evaluating students' behavioral engagement.

[0068] S3, based on the above analysis results of emotional engagement and behavioral engagement, the classification weights of emotional analysis and behavioral recognition are integrated to calculate the overall engagement evaluation results and generate feedback information.

[0069] S4 uses a variety of forms to visually display the overall engagement evaluation results and feedback information, and provides an easy-to-understand user interface.

[0070] The tool for visualization is the real-time video window of the human-computer interaction terminal. The content displayed in the real-time video window of the human-computer interaction terminal includes detection boxes, classification labels and final participation scores. The forms of visualization include but are not limited to heat maps, charts and text annotations. The above-mentioned diversified visualization and feedback mechanisms enable teachers and students to understand the participation status intuitively and timely, and make corresponding adjustments.

[0071] For teachers, visual displays can help them adjust their questioning strategies, change their teaching methods, and develop personalized teaching plans. For example, teachers can interact more with students with low participation, or adjust teaching content and methods to improve overall class participation based on their participation.

[0072] At the same time, the system provides students with instant text reminders based on real-time analysis results. When it detects that a student's participation is low, the system will display prompts such as "Please pay attention" or "Please actively participate in the discussion" to help students adjust their status in time and improve their participation.

[0073] This diverse visual display and feedback mechanism can not only help teachers understand and adjust teaching strategies in real time, but also help students recognize their own participation and make improvements, thereby improving overall teaching effectiveness and learning efficiency.

[0074] S5, the system makes personalized records of the test results for storage and management, facilitating long-term tracking and analysis.

[0075] Specifically, the system records and analyzes the long-term participation trends of each student, generates detailed participation reports, and regularly provides them to teachers and students to help them reflect on themselves and improve their learning habits. This provides educators with in-depth insights to help them optimize teaching strategies and improve teaching effectiveness.

[0076] The present invention also discloses a deep learning-based automatic student engagement assessment system, which includes a video acquisition module, a facial detection module, a sentiment analysis module, a behavior recognition module, a final engagement result generation and feedback module, and a human-computer interaction module that communicate through an API.

[0077] Among them, the video acquisition module is used to collect video stream data. The face detection module is used to use the MTCNN algorithm to perform face detection from the original frame of the video stream data, extract the face image, and generate multiple sub-video sequences. The sentiment analysis module is used to input the sub-video sequence into the sentiment analysis model to evaluate the students' emotional engagement. The behavior recognition module is used to use the YOLO technology to identify the students' behavior patterns, so as to evaluate their behavioral engagement. The final engagement result generation and feedback module is used to calculate the overall engagement evaluation result and generate feedback information based on the analysis results of emotional engagement and behavioral engagement. The human-computer interaction module uses multiple forms to visualize the overall engagement evaluation results and feedback information, and provides an easy-to-understand user interface. The data recording module is used to store and manage personalized records of the detection results for long-term tracking and analysis.

[0078] The above modules communicate with each other through API, so that the system can be flexibly deployed and expanded on different platforms.

[0079] Specifically, the human-computer interaction module can preferably be a mobile phone, a tablet computer or a laptop computer. The system is developed using a cross-platform development framework, such as React Native, Flutter, etc., to ensure consistency and compatibility on different device platforms.

[0080] In summary, the present invention establishes a comprehensive engagement assessment framework by integrating the engagement analysis of the two dimensions of emotion and behavior. This comprehensive assessment not only deeply understands the students' engagement from multiple perspectives, provides real-time feedback, and helps students and teachers make quick adjustments and changes, but also provides comprehensive and profound personalized analysis and detection records.

[0081] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. After the computer program is run, any of the above methods is executed.

[0082] The present invention also discloses a computer system, including a processor and a storage medium, wherein a computer program is stored on the storage medium, and the processor reads and runs the computer program from the storage medium to execute any of the methods described above.

[0083] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0084] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0085] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0086] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0087] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

[0088] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An evaluation method for an automatic student engagement evaluation system based on deep learning, characterized in that: The following steps are involved: S1, the system collects video stream data and preprocesses it; wherein the preprocessing method is: using the MTCNN algorithm to extract face images from the original frame and generate multiple sub-video sequences; S2, input the generated sub-video sequence into the sentiment analysis model to evaluate the students’ emotional engagement; at the same time, input the preprocessed full image into the YOLO model to identify the students’ behavior patterns to evaluate their behavioral engagement; S3, based on the above analysis results of emotional engagement and behavioral engagement, the classification weights of emotional analysis and behavioral recognition are integrated to calculate the overall engagement evaluation result and generate feedback information; S4, uses various forms to visualize the overall engagement evaluation results and feedback information, and provides an easy-to-understand user interface; S5, the system makes personalized records of the test results for storage and management, facilitating long-term tracking and analysis.

2. The method according to claim 1, characterized in that The specific steps of constructing multiple sub-videos include: S11, performing video downsampling by selecting a frame interval to reduce the number of video frames; S12, dividing the downsampled video into multiple sliding windows; S13, select representative frames in each sliding window to construct multiple sub-videos.

3. The method according to claim 1, characterized in that: In step S2, the sentiment analysis model is a deep learning model using SAMSA-Net.

4. The method according to claim 1, characterized in that: The visualization content in step S4 includes the detection box, the classification label and the final engagement score, and the visualization form includes but is not limited to heat map, chart and text annotation.

5. A system for executing the evaluation method according to any one of claims 1 to 4, characterized in that: include: Video acquisition module, used for acquiring video stream data; A face detection module is used to perform face detection from the original frames of the video stream data using the MTCNN algorithm, extract face images, and generate multiple sub-video sequences; The sentiment analysis module is used to input the sub-video sequence into the sentiment analysis model to evaluate the students' emotional engagement; The behavior recognition module is used to identify students’ behavior patterns using YOLO technology, thereby evaluating their behavioral engagement; The final engagement result generation and feedback module is used to calculate the overall engagement evaluation results and generate feedback information based on the analysis results of emotional engagement and behavioral engagement; The human-computer interaction module uses various forms to visualize the overall engagement evaluation results and feedback information, and provides an easy-to-understand user interface; Data logging module, used to store and manage personalized records of test results for long-term tracking and analysis.

6. The system according to claim 5, characterized in that: The video acquisition module, facial detection module, emotion analysis module, behavior recognition module, final engagement result generation and feedback module, and human-computer interaction module communicate with each other through an API, so that the system can be flexibly deployed and expanded on different platforms.

7. A computer-readable storage medium, characterized in that: A computer program is stored on the medium, and after the computer program is run, the method according to any one of claims 1 to 4 is executed.

8. A computer system, characterized in that: The method comprises a processor and a storage medium, wherein a computer program is stored in the storage medium, and the processor reads and runs the computer program from the storage medium to execute the method as claimed in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Intelligent analysis method and system for student classroom learning interest

    CN108399376A

  • Student classroom participation degree analysis system based on classroom videos

    CN111046823A

  • Learning participation degree determination method and device and computer equipment

    CN112085392A

  • Student behavior identification method, teaching state evaluation method and related equipment

    CN115909130A

  • Teaching information processing system and method based on artificial intelligence

    CN118197367A