Real-time Visual Analysis Method for Student Engagement in Online Courses

By obtaining and analyzing student video data in real time, establishing analysis models in attention, emotion and fatigue, the problem of real-time assessment of student participation in online courses is solved, efficient visual analysis of participation is achieved, and teaching quality and teachers and students are improved. interactive.

CN114973126BActive Publication Date: 2024-11-22CENT SOUTH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210540837.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-17
Publication Date
2024-11-22
Estimated Expiration
2042-05-17

AI Technical Summary

Technical Problem

It is difficult for existing technology to evaluate and visualize student participation in online courses in real time and accurately, especially when the data is large, complex and requires high real-time requirements, it is difficult for teachers to analyze the reasons behind student participation and conduct real-time analysis.

Method used

By obtaining student video data in real time, facial features are extracted and analysis models of attention, emotion, fatigue degree and cognitive state are established. Multi-task cascaded convolutional neural network and deep learning models such as ResNet-50 are used for preprocessing and analysis. Combining gaze direction and head attitude estimation, abstracts, monitoring, distribution and personal views were constructed for visual analysis.

Benefits of technology

Real-time visual analysis with high reliability, accuracy and real-time student participation is achieved, helping teachers quickly understand students' participation and make teaching decisions, and improving the teaching quality and learning effect of online classrooms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114973126B_ABST
    Figure CN114973126B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time visualization analysis method for student engagement in online courses, including real-time acquisition of videos of students in online courses; pre-processing video data to extract students' facial features; establishing a student engagement analysis model from four aspects: attention, emotion, fatigue level, and cognitive state; and visualizing the analysis results of the student engagement analysis model to complete the real-time visualization analysis of student engagement in online courses. The present invention uses computer vision methods to extract student facial features, then establishes a student engagement analysis model, and then uses visual analysis methods to help teachers analyze student engagement from multiple angles; compared to the traditional linear method of viewing student videos, the method of the present invention can automatically extract student engagement information and visualize it, with high reliability, good accuracy, high real-time performance, and is convenient and fast.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image data processing, and in particular relates to a real-time visual analysis method for student participation in online courses. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, people have become more active in learning new knowledge and new skills. Therefore, more and more online courses have appeared on the Internet, and people are more and more willing to participate in these online courses.

[0003] Engagement is an important evaluation indicator for students in online courses, and the degree of engagement directly affects the quality of the course and the quality of student learning. Therefore, real-time evaluation and visualization of student engagement in courses has always been one of the research focuses of online course platforms.

[0004] However, there are often various problems in the current real-time evaluation of students' engagement in online courses. Some researchers obtain students' engagement-related features through various detection methods, such as heart rate sensors, eye trackers, computer vision methods, etc., and then calculate students' engagement through automated methods. However, automated methods still have difficulty solving the above problems. First, automated classification methods usually directly give student engagement categories, but because teachers do not understand the logic behind the model, it is difficult to analyze the reasons behind student engagement; secondly, due to the large number of students, real-time data, and the inclusion of contextual information, the data is large and complex, making it difficult for teachers to perform real-time analysis; finally, most of the current student engagement analysis work is to analyze offline historical data, which is difficult to meet the real-time analysis needs of online classes in terms of analysis speed and perceived efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a method for real-time visualization analysis of student participation in online courses that is highly reliable, accurate, real-time, and convenient.

[0006] The real-time visual analysis method of student participation in online courses provided by the present invention comprises the following steps:

[0007] S1. Real-time acquisition of students’ videos in online courses;

[0008] S2. Preprocess the video data obtained in step S1 to extract the facial features of the students;

[0009] S3. Establish a student engagement analysis model based on four aspects: attention, emotion, fatigue level and cognitive state;

[0010] S4. Visualize the analysis results of the student engagement analysis model established in step S3, thereby completing the real-time visualization analysis of student engagement in the online course.

[0011] The real-time acquisition of the video of the student in the online course described in step S1 specifically includes the following steps:

[0012] Use cameras to capture real-time video of students in online courses;

[0013] According to the blinking frequency of human eyes, set the number of frames per second to collect images;

[0014] Before the video collection begins, collect images of students looking directly at the center of the screen and observing the four vertices around the screen from the center of the screen {I c ,I lt ,I lb ,I rt ,I rb}, used to check the head direction and sight direction; where I c For students to face the image in the center of the screen, lt This is the image when the student observes the top left corner of the screen from the center of the screen. lb This is the image when the student observes the vertex in the lower left corner of the screen from the center of the screen. rt This is the image when the student observes the vertex in the upper right corner of the screen from the center of the screen. rb This is the image when students observe the vertex in the lower right corner of the screen in the center of the screen.

[0015] Step S2 of preprocessing the video data obtained in step S1 to extract the facial features of the students specifically includes the following steps:

[0016] Perform preprocessing on the client where the student is located; the preprocessing includes face recognition preprocessing, face alignment preprocessing, emotion recognition preprocessing, head posture estimation preprocessing and sight estimation preprocessing;

[0017] Among them, face recognition preprocessing is to use a deep learning model to detect whether there is face information in the collected image; face alignment preprocessing is to use a neural network model to extract the face coordinate position information in the collected image; emotion recognition preprocessing is to use a neural network model to identify the students' real-time emotions; the emotions include anger, naturalness, sadness, surprise, happiness, disgust and fear; head posture estimation preprocessing is to use head posture technology to extract the head posture angle of the person from the collected image, and judge the relationship between the student's head and the screen based on the head posture angle; line of sight estimation preprocessing is to use a full-face-based line of sight estimation algorithm to extract the student's line of sight direction, and judge the relationship between the student's line of sight and the screen based on the line of sight direction.

[0018] The face recognition preprocessing specifically includes the following steps:

[0019] A multi-task cascade convolutional neural network deep learning model is used to detect whether an image contains facial information; the multi-task cascade convolutional neural network deep learning model detects the position (x, y, w, h) of the face matrix in the image, where (x, y) represents the coordinates of the upper left corner of the face matrix when the upper left corner of the image is the coordinate origin, w represents the width of the rectangular area corresponding to the detected face matrix, and h is the height of the rectangular area corresponding to the detected face matrix; in specific implementation, if the face matrix is ​​detected, it means that the face information is recognized.

[0020] The face alignment preprocessing specifically includes the following steps:

[0021] The cascade convolutional neural network model is used to extract the coordinates of key points of the face. When extracting the coordinates, the upper left corner of the image is taken as the origin, and the extracted coordinates of the key points of the face are [x i ,y i ].

[0022] The emotion recognition preprocessing specifically includes the following steps:

[0023] ResNet-50 was used as the convolutional neural network model, and the FER2013 facial expression recognition public dataset was used for training to obtain the final preprocessing model; the emotions output by the preprocessing model included anger, naturalness, sadness, surprise, happiness, disgust, and fear.

[0024] The head posture estimation preprocessing specifically includes the following steps:

[0025] The head posture estimation algorithm is used to obtain the head posture of the person from the image; the head posture is represented by three Euler angles, namely pitch angle, yaw angle and roll angle, which are used to represent nodding, shaking head and turning head respectively;

[0026] The head posture estimation algorithm is based on the key point 2D projection to 3D method to extract the head posture; the head posture is represented by a three-dimensional vector [Pitch, Yaw, Roll], where Pitch is the pitch angle, Yaw is the yaw angle, and Roll is the roll angle;

[0027] Before preprocessing the head posture estimation, a preliminary check is performed; during the check, the head posture angle of the student facing the center of the screen is extracted, as well as the head posture angles in four directions when the student's head is at the center of the screen and observes the surroundings of the screen; during the check, the student's head posture is proportionally converted so that the Pitch value and Yaw value when the student is looking at the center of the screen are both 0, and the absolute value of the pitch angle of the head when the student is looking at the upper and lower edges of the screen and the yaw angle when the student is looking at the left and right edges is 0.5;

[0028] During calibration, the following formula is used to calculate the pitch angle Pitch2 after calibration: Where Pitch1 is the pitch angle detected, Pitch c Pitch is the pitch angle when students look directly at the center of the screen. t The pitch angle for students to see the border on the screen;

[0029] After verification, if the deflection angle value is greater than 1, it will be directly corrected to the maximum value 1;

[0030] Finally, when the deflection angle is between -0.5 and 0.5, it is determined that the student's head is facing the screen.

[0031] The sight line estimation preprocessing specifically comprises the following steps:

[0032] The gaze estimation algorithm based on the whole face is used to extract the gaze direction of the students, and the gaze direction of the students is obtained as [X, Y], where X is the rotation angle of the gaze in the horizontal direction and Y is the rotation angle of the gaze in the vertical direction;

[0033] Before the preprocessing of line of sight estimation, a preliminary verification is performed; during the verification, the line of sight angles of the student looking at the center of the screen and the surroundings of the screen when the student's head is at the center of the screen are extracted; then according to the verification results, the line of sight data is converted so that the line of sight when the student looks at the center of the screen is [0,0], and the absolute value of the angle corresponding to the student's line of sight looking at the surroundings of the screen is 0.5;

[0034] During calibration, the angle value Y2 of the sight direction after calibration is calculated using the following formula: Where Y1 is the angle value of the line of sight direction detected, Y c Y is the angle of the student's line of sight when looking directly at the center of the screen. t It is the angle value of the direction of the students’ line of sight when they look at the border on the screen;

[0035] After verification, if the angle value of the sight direction is greater than 1, it will be directly corrected to the maximum value of 1;

[0036] Finally, when the angle value of the line of sight direction is between -0.5 and 0.5, it is determined that the student is looking at the screen.

[0037] Step S3 establishes a student engagement analysis model from four aspects: attention, emotion, fatigue level, and cognitive state, and specifically includes the following steps:

[0038] Attention:

[0039] The following formula is used to calculate the student viewing screen state variable S: S = "X≤0.5" & "Y≤0.5"; where [X, Y] is the pre-processed student line of sight direction; the value rule of "a" is: if a is true, then "a" = 1, if a is false, then "a" = 0; & is an AND operation; when the student viewing screen state variable S is 1, it indicates that the student is viewing the screen; when the student viewing screen state variable S is 0, it indicates that the student is not viewing the screen;

[0040] A sliding window with a time span of 30 seconds is used to receive the student's line of sight [X, Y] and the student's viewing screen state variable S, and N = [X1, X2, ..., X n ]、M=[Y1,Y2,...,Y n ] and W=[S1,S2,...,S n ], where X i is the horizontal sight direction of the student in the i-th frame, Y i is the vertical sight direction of the student in the i-th frame, S i is the state variable of the student watching the screen in the i-th frame; the student daze state variable D is calculated as D = "(max(N)-min(N))≤T"&"(max(M)-min(M))≤T", where max(N) is the maximum value of the elements in N, min(N) is the minimum value of the elements in N, D is 1 for the student not dazed, and D is 0 for the student dazed; finally, the student's attention variable A is calculated using the following formula:

[0041] Fatigue level:

[0042] (1) For students, the average eye aspect ratio B2, blink frequency P2, average blink duration D2 and yawn frequency M2 were selected as indicators to model the students' fatigue level;

[0043] Extract the eye key points and mouth key points of the face: the eye key points include 6 key points, the eye key point E1 at the outermost corner of the eye, the eye key point E4 at the innermost corner of the eye, the eye key point E2 at the outer 1 / 4 of the upper eye contour, the eye key point E4 at the inner 1 / 4 of the upper eye contour, the eye key point E6 at the outer 1 / 4 of the lower eye contour, and the eye key point E5 at the inner 1 / 4 of the lower eye contour; the mouth key point information includes 4 key points, the leftmost key point M1 of the mouth contour, the rightmost key point M3 of the mouth contour, the highest key point M2 of the upper lip, and the lowest key point M4 of the lower lip; each key point is a two-dimensional coordinate;

[0044] The following formula is used to calculate the student's eye aspect ratio α: Where E2-E6 represents the straight-line distance between key point E2 and key point E6; E3-E5 represents the straight-line distance between key point E3 and key point E5; E1-E4 represents the straight-line distance between key point E1 and key point E4;

[0045] The following formula is used to calculate the length-to-width ratio of the student's mouth: Where M3-M1 represents the straight-line distance between key point M3 and key point M1, and M2-M4 represents the straight-line distance between key point M2 and key point M4;

[0046] The eye length-to-width ratio and mouth length-to-width ratio of each student are obtained in advance and normalized so that the eye length-to-width ratio α is 0 when the eyes are fully closed and 1 when the eyes are fully open; the mouth length-to-width ratio β is 0 when the mouth is fully closed and 1 when the mouth is fully open;

[0047] When it is detected that the aspect ratio α of the student's eyes is less than the set threshold, it is considered that the student has closed eyes;

[0048] (2) Using a sliding window of 30 seconds, the video image is intercepted and F = [B1, B2, ..., B n ], where B i is the eye length-to-width ratio of the i-th frame image in the sliding window; the average eye length-to-width ratio B of the students is calculated using the following formula:

[0049] The blinking frequency P is defined as the number of blinking actions of students in 30 seconds;

[0050] A sliding window with a duration of 30 seconds is used to capture the video image, and G = [D1, D2, ..., D m ], where m blinks are detected in the sliding window, D i is the duration of the i-th blink; the average blink duration D of the students is calculated using the following formula:

[0051] When the mouth length-to-width ratio is greater than the set threshold and the duration is greater than the set threshold, the student is deemed to have yawned; the student's yawn frequency M is the number of yawns within 30 seconds;

[0052] The dynamic weight fuzzy comprehensive evaluation method was used to analyze the students' fatigue level: the initial average eye aspect ratio B weight, blink frequency P weight, average blink duration D weight and yawn frequency M weight were 0.42, 0.13, 0.32 and 0.13 respectively; when B < 0.2, the average eye aspect ratio B weight, blink frequency P weight, average blink duration D weight and yawn frequency M weight were adjusted to 0.8, 0.1, 0.1 and 0 respectively; each indicator corresponds to a corresponding value V at different values; when B = 1, the value is 0, when B = 0.8, the value is 0.2, when B = 0.6, the value is 0.4, when B = 0.4, the value is 0.6, when B = 0.2, the value is 0.8, when B = 0, the value is 1; when P is in the range of 6-8, the value is 0, and when P is 5 When P is 9, the value is 0.2; when P is 4 or in the range of 10-12, the value is 0.4; when P is 3 or in the range of 12-14, the value is 0.6; when P is 2 or in the range of 15-16, the value is 0.8; when P≤1 or P>16, the value is 1; when D is in the range of 0.2-0.4, the value is 0; when D is in the range of 0.5-0.6, the value is 0.2; when D is in the range of 0.7-0.8, the value is 0.4; when D is in the range of 0.8-1.5, the value is 0.6; when D is in the range of 1.5-3, the value is 0.8; when D is above 3, the value is 1; when M is 0, the value is 0, when M is 1, the value is 0.4, when M is 2, the value is 0.6, when M is 3, the value is 0.8; when M≥4, the value is 1; finally, the fatigue degree F is calculated as: F=V B *W B +V P *W P +V D *W D +V M *W M , where W B is the average eye aspect ratio B weight, W P is the blink frequency P weight, W D is the average blink duration D weight, W M is the yawning frequency M weight, V B is the value of the eye aspect ratio B, V P is the value of blink frequency P, V D is the value of the average blink duration D, V M is the value of yawning frequency M.

[0053] Emotional Aspects:

[0054] Using the sentiment classification model, the sentiments are classified into anger, natural, sad, surprised, happy, disgusted and afraid;

[0055] Cognitive status:

[0056] It is stipulated that students nod when they understand the course content and shake their heads when they do not understand the course content;

[0057] Detect the student's head movement: extract the yaw angle and pitch angle of the student's head, and then establish two sliding windows with a time span of 3s to receive the yaw angle and pitch angle respectively;

[0058] Use the following steps to detect students' nodding behavior:

[0059] It is stipulated that if the head direction continuously decreases and exceeds the set threshold, it is judged that the head is lowered; if the head direction continuously increases and exceeds the set threshold, it is judged that the head is raised; if both lowering and raising behaviors are detected within the observation interval, it is judged that nodding behavior occurs.

[0060] Step S4 of visualizing the analysis results of the student engagement analysis model established in step S3 specifically includes the following steps:

[0061] (1) Construct summary view, monitoring view, distribution view and personal view to visualize the analysis results; the summary view is used to present the participation of all students in real time; the monitoring view is used to present the participation of a specified student group in real time; the distribution view is used to present the distribution of students' participation in a specified time period; and the personal view is used to display the relevant characteristics of individual students' participation;

[0062] (2) Summary view: A summary view is constructed based on a stacked bar chart to simultaneously present four indicators: student emotion, fatigue, attention, and student status; it presents student engagement from three perspectives: attention, fatigue, and emotion;

[0063] A view that provides an overview of student engagement from the perspective of attention and fatigue: the Y-axis represents different levels of attention intervals, which are divided into four groups from low to high; the X-axis represents the number of students in each group; each column consists of several small rectangles, each rectangle represents a student, and the color of the rectangle corresponds to the student's emotion; an embedded dark matrix is ​​used in each small rectangle to represent the degree of fatigue; the height of the embedded matrix represents the value of the degree of fatigue; the students are sorted in descending order of fatigue to avoid visual confusion; circular icons of different colors are used under each rectangle to present the student's cognitive state and daze state; through a view that provides an overview of student engagement from the perspective of attention and fatigue, teachers can quickly get an overview of student engagement;

[0064] A view that provides an overview of student engagement from an emotional perspective: the Y axis represents the student's emotional category, and the length of the rectangular column corresponds to the number of students in a certain category of emotion; each column consists of several rectangles, each representing a student, with two dark rectangles embedded in the rectangle, the height of the upper rectangle corresponds to the student's attention, and the height of the lower rectangle corresponds to the student's fatigue level; sorting by attention; teachers can switch to different angles according to the teaching scenario to overview the student's engagement; when the teacher hovers the mouse over the view, the view will stop updating in real time to facilitate the teacher to view or select students; when the mouse hovers over a student, the student's detailed information will be displayed; click to view the student's information; teachers can select the student group of interest in the view and conduct further analysis;

[0065] (3) Monitoring view: used to help teachers keep an eye on the participation of students of interest; construct a face model; each face represents a student, and the color of the hair corresponds to the student's emotion, and the emotion color is consistent with the summary view; the relative position of the eyeball in the eye corresponds to the student's line of sight; if the student looks directly at the screen, the eyeball is in the center of the eye; the relative position of the hair, eyes, nose and mouth in the face is used to encode the direction of head rotation; the height of the eyelids is used to encode the student's fatigue level; through the monitoring view, teachers can perceive students' abnormal participation behavior;

[0066] (4) Distribution view: used to help teachers understand the distribution of students' participation in a certain period of time. The distribution view is a matrix, in which each grid represents a student. Each student is represented by a glyph design, and the pie chart in the middle represents the distribution of students' emotions in a specified period of time. The rings next to the pie chart represent attention. The angle of the dark ring corresponds to the average attention in the period of time, and the angle of the top of the light ring corresponds to the upper quartile of attention. The outermost ring represents fatigue. The angle of the dark ring corresponds to the average fatigue, and a full circle indicates complete fatigue. The angle of the top of the light ring represents the upper quartile of fatigue. The distribution view can be sorted in a variety of ways. After the sorting is completed, the student position will remain fixed and updated in real time. When the teacher selects some students from the summary view, the distribution view will default to sorting these student groups from high to low in terms of attention. There is a time bar slider in the distribution view, and the teacher can use the slider to select a time period and view the corresponding information. The teacher can view the detailed information of the students by hovering the mouse, and add the students to the monitoring view and personal view by clicking.

[0067] (5) Personal view: used to provide teachers with relevant characteristic information of student participation and the original image of the student; the time span presented in the personal view is consistent with the time slider in the distribution view; the personal view contains the gaze direction, head direction, eye aspect ratio, mouth aspect ratio, the original image of the student and the student's action information; the X-axis in the personal view represents time; there is a set of dotted lines at the top and bottom of the personal view, which are used to represent the gaze direction of the student when looking at the upper and lower edges of the screen; there is a flow graph in the dotted line, the width of the flow graph is equal to the width between the two dotted lines, and the position represents the direction of the head; the upper flow graph represents the direction of the head rotating around the Y axis, and the lower flow graph represents the direction of the head rotating around the X axis; the color-coded solid line is used to represent the gaze direction; if the line is between two color lines, it means that the student is looking at the screen, and if the line is outside the color line, it means that the student's gaze is off the screen. The color of the line encodes the student's emotion category; the color of the emotion is consistent with the summary view; there are two Gantt charts in the middle of the two river charts, the first Gantt chart represents blinking information, the blank area in the Gantt chart represents eye closure, and the transparency of the matrix in the Gantt chart encodes the average eye length-to-width ratio between two blinks; the second Gantt chart presents mouth opening information, if a specific color rectangle appears, it means the mouth is open, and if not, it means closed mouth; below the first Gantt chart, color-coded circular icons represent students' actions, the first color represents nodding, and the second color represents shaking head; when the teacher clicks on the flow chart at a certain moment in the figure, the original image of the student at that moment will be displayed to help the teacher verify his or her analysis results.

[0068] The real-time visual analysis method for student engagement in online courses provided by the present invention uses a computer vision method to extract student facial features, then establishes a student engagement analysis model, and then uses a visual analysis method to help teachers analyze student engagement from multiple angles; compared with the traditional linear method of viewing student videos, the method of the present invention can automatically extract student engagement information and display it visually, with high reliability, good accuracy, high real-time performance, and convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0070] Figure 2 Schematic diagram of 68 key coordinate points of a face in the present invention.

[0071] Figure 3 Schematic diagram of the coordinates of the key points of the eyes and mouth of the present invention.

[0072] Figure 4 It is a schematic diagram of the summary view of the present invention observed from the perspective of attention.

[0073] Figure 5 A schematic diagram showing a summary view of the present invention from an emotional perspective.

[0074] Figure 6 It is a monitoring view schematic diagram of the present invention.

[0075] Figure 7 This is a schematic diagram of the distribution view of the present invention.

[0076] Figure 8 It is a schematic diagram of glyph design in the distribution view of the present invention.

[0077] Fig. 9 It is a schematic diagram of a personal view of the present invention.

[0078] Fig.10 This is a schematic diagram of an attention distribution view according to the first embodiment of the present invention.

[0079] Fig.11 This is a schematic diagram of the distribution of students' cognitive states according to the first embodiment of the present invention.

[0080] Fig.12 It is a schematic diagram of a distribution view of the first embodiment of the present invention.

[0081] Fig.13 This is a schematic diagram of a monitoring view according to the second embodiment of the present invention.

[0082] Fig.14 It is a schematic diagram of a personal view according to the second embodiment of the present invention.

[0083] Fig.15 This is a schematic diagram of a view of the student 200048 sleeping in class according to the second embodiment of the present invention.

[0084] Fig.16 This is a schematic diagram of a monitoring view for comparative analysis of participation of different students according to the third embodiment of the present invention.

[0085] Fig.17 This is a schematic diagram of the distributed view and personal view of the third embodiment. DETAILED DESCRIPTION

[0086] like Figure 1 The method flow chart of the present invention is shown as follows: The real-time visual analysis method of student participation in online courses provided by the present invention comprises the following steps:

[0087] S1. Real-time acquisition of students' videos in online courses; specifically includes the following steps:

[0088] Use cameras to capture real-time video of students in online courses;

[0089] According to the blinking frequency of human eyes, set the number of frames per second to collect images;

[0090] Before the video collection begins, collect images of students looking directly at the center of the screen and observing the four vertices around the screen from the center of the screen {I c ,I lt ,I lb ,I rt ,I rb}, used to check the head direction and sight direction; where I c For students to face the image in the center of the screen, lt This is the image when the student observes the top left corner of the screen from the center of the screen. lb This is the image when the student observes the vertex in the lower left corner of the screen from the center of the screen. rt This is the image when the student observes the vertex in the upper right corner of the screen from the center of the screen. rb This is the image when students observe the vertex in the lower right corner of the screen from the center of the screen;

[0091] S2. Preprocessing the video data obtained in step S1 to extract the facial features of the students; specifically comprising the following steps:

[0092] Perform preprocessing on the client where the student is located; the preprocessing includes face recognition preprocessing, face alignment preprocessing, emotion recognition preprocessing, head posture estimation preprocessing and sight estimation preprocessing;

[0093] Among them, face recognition preprocessing is to use a deep learning model to detect whether there is face information in the collected image; face alignment preprocessing is to use a neural network model to extract the face coordinate position information in the collected image; emotion recognition preprocessing is to use a neural network model to identify the student's real-time emotions; the emotions include anger, nature, sadness, surprise, happiness, disgust and fear; head posture estimation preprocessing is to use head posture technology to extract the head posture angle of the person from the collected image, and judge the relationship between the student's head and the screen based on the head posture angle; sight line estimation preprocessing is to use a full-face-based sight line estimation algorithm to extract the student's sight line direction, and judge the relationship between the student's sight line and the screen based on the sight line direction;

[0094] In specific implementation, face recognition preprocessing specifically includes the following steps:

[0095] A multi-task cascade convolutional neural network deep learning model is used to detect whether an image contains facial information; the multi-task cascade convolutional neural network deep learning model detects the position (x, y, w, h) of the face matrix in the image, where (x, y) represents the coordinates of the upper left corner of the face matrix when the upper left corner of the image is the coordinate origin, w represents the width of the rectangular area corresponding to the detected face matrix, and h represents the height of the rectangular area corresponding to the detected face matrix; in specific implementation, if the face matrix is ​​detected, it means that the face information is recognized;

[0096] Face alignment preprocessing specifically includes the following steps:

[0097] The cascade convolutional neural network model is used to extract the coordinates of key points of the face. When extracting the coordinates, the upper left corner of the image is taken as the origin, and the extracted coordinates of the key points of the face are [x i ,y i ];

[0098] Emotion recognition preprocessing specifically includes the following steps:

[0099] ResNet-50 was used as the convolutional neural network model, and the FER2013 facial expression recognition public dataset was used for training to obtain the final preprocessing model; the emotions output by the preprocessing model included anger, naturalness, sadness, surprise, happiness, disgust, and fear;

[0100] The head posture estimation preprocessing specifically includes the following steps:

[0101] The head posture estimation algorithm is used to obtain the head posture of the person from the image; the head posture is represented by three Euler angles, namely pitch angle, yaw angle and roll angle, which are used to represent nodding, shaking head and turning head respectively;

[0102] The head posture estimation algorithm is based on the key point 2D projection to 3D method to extract the head posture; the head posture is represented by a three-dimensional vector [Pitch, Yaw, Roll], where Pitch is the pitch angle, Yaw is the yaw angle, and Roll is the roll angle;

[0103] Before preprocessing the head posture estimation, a preliminary check is performed; during the check, the head posture angle of the student facing the center of the screen is extracted, as well as the head posture angles in four directions when the student's head is at the center of the screen and observes the surroundings of the screen; during the check, the student's head posture is proportionally converted, so that the pitch value and Yaw value when the student is looking at the center of the screen are both 0, and the absolute value of the pitch angle of the head when the student is looking at the upper and lower edges of the screen and the yaw angle when the student is looking at the left and right edges is 0.5;

[0104] During calibration, the following formula is used to calculate the pitch angle Pitch2 after calibration: Where Pitch1 is the pitch angle detected, Pitch c Pitch is the pitch angle when students look directly at the center of the screen. t The pitch angle for students to see the border on the screen;

[0105] After verification, if the deflection angle value is greater than 1, it will be directly corrected to the maximum value 1;

[0106] Finally, when the deflection angle is between -0.5 and 0.5, it is determined that the student's head direction is facing the screen;

[0107] The line of sight estimation preprocessing specifically includes the following steps:

[0108] The gaze estimation algorithm based on the whole face is used to extract the gaze direction of the students, and the gaze direction of the students is obtained as [X, Y], where X is the rotation angle of the gaze in the horizontal direction and Y is the rotation angle of the gaze in the vertical direction;

[0109] Before the preprocessing of line of sight estimation, a preliminary verification is performed; during the verification, the line of sight angles of the student looking at the center of the screen and the surroundings of the screen when the student's head is at the center of the screen are extracted; then according to the verification results, the line of sight data is converted so that the line of sight when the student looks at the center of the screen is [0,0], and the absolute value of the angle corresponding to the student's line of sight looking at the surroundings of the screen is 0.5;

[0110] During calibration, the angle value Y2 of the sight direction after calibration is calculated using the following formula: Where Y1 is the angle value of the line of sight direction detected, Y c Y is the angle of the student's line of sight when looking directly at the center of the screen. t It is the angle value of the direction of the students’ line of sight when they look at the border on the screen;

[0111] After verification, if the angle value of the sight direction is greater than 1, it will be directly corrected to the maximum value of 1;

[0112] Finally, when the angle value of the sight direction is between -0.5 and 0.5, it is determined that the student is looking at the screen;

[0113] S3. Establish a student engagement analysis model based on four aspects: attention, emotion, fatigue level and cognitive state. The specific steps include:

[0114] Attention:

[0115] The following formula is used to calculate the student viewing screen state variable S: S = "X≤0.5" & "Y≤0.5"; where [X, Y] is the pre-processed student line of sight direction; the value rule of "a" is: if a is true, then "a" = 1, if a is false, then "a" = 0; & is an AND operation; when the student viewing screen state variable S is 1, it indicates that the student is viewing the screen; when the student viewing screen state variable S is 0, it indicates that the student is not viewing the screen;

[0116] A sliding window with a time span of 30 seconds is used to receive the student's line of sight [X, Y] and the student's viewing screen state variable S, and N = [X1, X2, ..., X n ]、M=[Y1,Y2,...,Y n ] and W=[S1,S2,...,S n ], where X iis the horizontal sight direction of the student in the i-th frame, Y i is the vertical sight direction of the student in the i-th frame, S i is the screen viewing state variable of the i-th middle school student; the student daze state variable D is calculated as D = "(max(N)-min(N))≤T"&"(max(M)-min(M))≤T", where max(N) is the maximum value of the elements in N, min(N) is the minimum value of the elements in N, D is 1 for the student not dazed, and D is 0 for the student dazed; finally, the student's attention variable A is calculated using the following formula:

[0117] Fatigue level:

[0118] (1) For students, the average eye aspect ratio B2, blink frequency P2, average blink duration D2 and yawn frequency M2 were selected as indicators to model the students' fatigue level;

[0119] Extract the eye key points and mouth key points of the face: the eye key points include 6 key points, the eye key point E1 at the outermost corner of the eye, the eye key point E4 at the innermost corner of the eye, the eye key point E2 at the outer 1 / 4 of the upper eye contour, the eye key point E4 at the inner 1 / 4 of the upper eye contour, the eye key point E6 at the outer 1 / 4 of the lower eye contour, and the eye key point E5 at the inner 1 / 4 of the lower eye contour; the mouth key point information includes 4 key points, the leftmost key point M1 of the mouth contour, the rightmost key point M3 of the mouth contour, the highest key point M2 of the upper lip, and the lowest key point M4 of the lower lip; each key point is a two-dimensional coordinate;

[0120] The following formula is used to calculate the student's eye aspect ratio α: Where E2-E6 represents the straight-line distance between key point E2 and key point E6; E3-E5 represents the straight-line distance between key point E3 and key point E5; E1-E4 represents the straight-line distance between key point E1 and key point E4;

[0121] The following formula is used to calculate the length-to-width ratio of the student's mouth: Where M3-M1 represents the straight-line distance between key point M3 and key point M1, and M2-M4 represents the straight-line distance between key point M2 and key point M4;

[0122] The eye length-to-width ratio and mouth length-to-width ratio of each student are obtained in advance and normalized so that the eye length-to-width ratio α is 0 when the eyes are fully closed and 1 when the eyes are fully open; the mouth length-to-width ratio β is 0 when the mouth is fully closed and 1 when the mouth is fully open;

[0123] When it is detected that the aspect ratio α of the student's eyes is less than the set threshold, it is considered that the student has closed eyes;

[0124] (2) Using a sliding window of 30 seconds, the video image is intercepted and F = [B1, B2, ..., B n ], where B i is the eye length-to-width ratio of the i-th frame image in the sliding window; the average eye length-to-width ratio B of the students is calculated using the following formula:

[0125] The blinking frequency P is defined as the number of blinking actions of students in 30 seconds;

[0126] A sliding window with a duration of 30 seconds is used to capture the video image, and G = [D1, D2, ..., D m ], where m blinks are detected in the sliding window, D i is the duration of the i-th blink; the average blink duration D of the students is calculated using the following formula:

[0127] When the mouth length-to-width ratio is greater than the set threshold and the duration is greater than the set threshold, the student is deemed to have yawned; the student's yawn frequency M is the number of yawns within 30 seconds;

[0128] The dynamic weight fuzzy comprehensive evaluation method was used to analyze the students' fatigue level: the initial average eye aspect ratio B weight, blink frequency P weight, average blink duration D weight and yawn frequency M weight were 0.42, 0.13, 0.32 and 0.13 respectively; when B < 0.2, the average eye aspect ratio B weight, blink frequency P weight, average blink duration D weight and yawn frequency M weight were adjusted to 0.8, 0.1, 0.1 and 0 respectively; each indicator corresponds to a corresponding value V at different values; when B = 1, the value is 0, when B = 0.8, the value is 0.2, when B = 0.6, the value is 0.4, when B = 0.4, the value is 0.6, when B = 0.2, the value is 0.8, when B = 0, the value is 1; when P is in the range of 6-8, the value is 0, and when P is 5 When P is 9, the value is 0.2; when P is 4 or in the range of 10-12, the value is 0.4; when P is 3 or in the range of 12-14, the value is 0.6; when P is 2 or in the range of 15-16, the value is 0.8; when P≤1 or P>16, the value is 1; when D is in the range of 0.2-0.4, the value is 0; when D is in the range of 0.5-0.6, the value is 0.2; when D is in the range of 0.7-0.8, the value is 0.4; when D is in the range of 0.8-1.5, the value is 0.6; when D is in the range of 1.5-3, the value is 0.8; when D is above 3, the value is 1; when M is 0, the value is 0, when M is 1, the value is 0.4, when M is 2, the value is 0.6, when M is 3, the value is 0.8; when M≥4, the value is 1; finally, the fatigue degree F is calculated as: F=V B *W B +V P *W P +V D *W D +V M *W M , where W B is the average eye aspect ratio B weight, W P is the blink frequency P weight, W D is the average blink duration D weight, W M is the yawning frequency M weight, V B is the value of the eye aspect ratio B, V P is the value of blink frequency P, V D is the value of the average blink duration D, V M is the value of yawn frequency M;

[0129] Emotional Aspects:

[0130] Using the sentiment classification model, the sentiments are classified into anger, natural, sad, surprised, happy, disgusted and afraid;

[0131] Cognitive status:

[0132] It is stipulated that students nod when they understand the course content and shake their heads when they do not understand the course content;

[0133] Detect the student's head movement: extract the yaw angle and pitch angle of the student's head, and then establish two sliding windows with a time span of 3s to receive the yaw angle and pitch angle respectively;

[0134] Use the following steps to detect students' nodding and shaking behaviors:

[0135] It is stipulated that if the head direction continuously drops and exceeds the set threshold value Tnod, it is considered that the head-down behavior occurs; conversely, if the head direction continuously rises and exceeds the set threshold value Tnod, it is considered that the head-up behavior occurs; within the observation interval, if both the head-down and head-up behaviors are detected at the same time, it is determined that the head-nodding behavior occurs; after a large number of experimental results, the value of the set threshold value Tnod is preferably 0.25;

[0136] S4. Visualize the analysis results of the student engagement analysis model established in step S3, thereby completing the real-time visualization analysis of student engagement in online courses; specifically, the steps include:

[0137] (1) Construct summary view, monitoring view, distribution view and personal view to visualize the analysis results; the summary view is used to present the participation of all students in real time; the monitoring view is used to present the participation of a specified student group in real time; the distribution view is used to present the distribution of students' participation in a specified time period; and the personal view is used to display the relevant characteristics of individual students' participation;

[0138] (2) Summary view: A summary view is constructed based on a stacked bar chart to simultaneously present four indicators: student emotion, fatigue, attention, and student status; it presents student engagement from three perspectives: attention, fatigue, and emotion;

[0139] Overview of student engagement from an attention and fatigue perspective (e.g. Figure 4 As shown in the figure): the Y-axis represents the attention intervals of different degrees, which are divided into four groups from low to high; the X-axis represents the number of students in each group; each column is composed of several small rectangles, each rectangle represents a student, and the color of the rectangle corresponds to the student's emotion; an embedded dark matrix is ​​used in each small rectangle to represent the fatigue level; the height of the embedded matrix represents the value of the fatigue level; the fatigue levels are sorted in descending order to avoid visual confusion; circular icons of different colors are used under each rectangle to present the cognitive state and daze state of the students; through the view that provides an overview of student participation from the perspective of attention and fatigue, teachers can quickly get an overview of student participation;

[0140] Overview of student engagement from an affective perspective (e.g. Figure 5As shown in the figure): the Y axis represents the emotion category of students, and the length of the rectangular column corresponds to the number of students with a certain emotion; each column is composed of several rectangles, each rectangle represents a student, and two dark rectangles are embedded in the rectangle. The height of the upper rectangle corresponds to the student's attention, and the height of the lower rectangle corresponds to the student's fatigue level; sorting is based on attention; teachers can switch different angles according to the teaching scene to get an overview of the students' participation; when the teacher hovers the mouse over the view, the view will stop updating in real time to facilitate the teacher to view or select students; when the mouse hovers over a student, the student's detailed information will be displayed; click to view the student's information; teachers can select the student group of interest in the view and conduct further analysis;

[0141] (3) Monitoring view: used to help teachers keep an eye on the participation of students of interest; build face models (such as Figure 6 Each face represents a student, and the color of the hair corresponds to the student's emotion. The color of the emotion is consistent with the summary view. The relative position of the eyeball in the eye corresponds to the student's line of sight. If the student looks directly at the screen, the eyeball is in the center of the eye. The relative position of the hair, eyes, nose and mouth in the face is used to encode the direction of head rotation. The height of the eyelid is used to encode the student's fatigue level. Through the monitoring view, teachers can perceive students' abnormal participation behavior.

[0142] (4) Distribution view: used to help teachers understand the distribution of students' participation in a certain period of time (such as Figure 7 The distribution view is a matrix, where each grid represents a student. Each student is represented by a glyph design, with the pie chart in the middle (such as Figure 8 The diagram shows the emotional distribution of students in a specified time period. The circles next to the pie chart represent attention. The angle of the dark circle corresponds to the average attention in the time period, and the angle of the top of the light circle corresponds to the upper quartile of attention. The outermost circle represents fatigue. The angle of the dark circle corresponds to the average fatigue, and a full circle represents complete fatigue. The angle of the top of the light circle represents the upper quartile of fatigue. The distribution view can be sorted in a variety of ways. After the sorting is completed, the student position will remain fixed and updated in real time. When the teacher selects some students from the summary view, the distribution view will default to sorting these student groups from high to low in terms of attention. There is a time bar slider in the distribution view, and the teacher can use the slider to select a time period and view the corresponding information. The teacher can view the detailed information of the students by hovering the mouse, and add the students to the monitoring view and personal view by clicking.

[0143] (5) Personal view: used to provide teachers with relevant feature information of student participation and the original image of the student; the time span presented in the personal view is consistent with the time slider in the distribution view; the personal view contains the gaze direction, head direction, eye aspect ratio, mouth aspect ratio, the original image of the student and the student's action information; the personal view (such as Fig. 9 The X-axis in the figure represents time; there is a set of dotted lines at the top and bottom of the personal view, which are used to represent the direction of the students' sight when they look at the upper and lower edges of the screen; there is a flow graph in the dotted lines, the width of the flow graph is equal to the width between the two dotted lines, and the position represents the direction of the head; the upper flow graph represents the direction of rotation of the head around the Y-axis, and the lower flow graph represents the direction of rotation of the head around the X-axis; color-coded solid lines are used to represent the direction of sight; if the line is between two color lines, it means that the student is looking at the screen, and if the line is outside the color lines, it means that the student's sight is off the screen. The color of the line encodes the student's emotion category; the color of the emotion is consistent with the summary view; there are two Gantt charts in the middle of the two river charts, the first Gantt chart represents blinking information, the blank area in the Gantt chart represents eye closure, and the transparency of the matrix in the Gantt chart encodes the average eye length-to-width ratio between two blinks; the second Gantt chart presents mouth opening information, if a specific color rectangle appears, it means the mouth is open, and if not, it means closed mouth; below the first Gantt chart, color-coded circular icons represent students' actions, the first color represents nodding, and the second color represents shaking head; when the teacher clicks on the flow chart at a certain moment in the figure, the original image of the student at that moment will be displayed to help the teacher verify his or her analysis results.

[0144] The following is combined with an experiment to prove the practicability and effectiveness of the present invention.

[0145] Experimental process: In order to verify the effectiveness and usability of the system, one of the cooperating teachers was invited to conduct an online classroom teaching experiment using the visual analysis method of student engagement proposed in this invention. In this experiment, the teacher taught the "Machine Learning" course to 53 students. First, students were required to log in to the data acquisition and preprocessing system on the browser, turn on the camera through the browser, complete the sight and head direction verification before class, and place the browser in the corner of the desktop. Then we clearly told each student that the teacher could use the image detection method to perceive whether the student was in front of the screen or whether he was attending class seriously. In addition, students were also informed that they could convey their knowledge of knowledge to the teacher at any time by nodding or shaking their heads. Finally, the teacher was introduced to the use of the visual analysis method of student engagement. An additional display screen was provided for the teacher to display the visual analysis system. During the experiment, three cases were used to prove the effectiveness and usability of the system.

[0146] Case Study 1: A Quick Overview of Student Engagement

[0147] In the offline teaching process, teachers will check the participation of all students from time to time, and then adopt various teaching methods according to the status of students, so as to enhance the learning atmosphere in the classroom and improve the teaching quality. However, in online classes, because it is difficult for teachers to quickly perceive the participation of students, teachers usually teach on their own and rarely interact with students. In this experiment, the teacher used the proposed visual analysis method of participation to quickly perceive the participation of all students and make a series of positive teaching decisions. First, he chose to sort by attention in the summary view. At the beginning of the course, the students' attention was relatively focused, but after a period of time, the teacher found that most students' attention shifted to the 0.25-0.75 range, and some students were sad and their fatigue level was higher than before, such as Fig.10 As shown in the figure, the teacher realized that his class might have bored the students. So the teacher paused and said to everyone, "Please pay attention, I will talk about the key points next." He raised the volume appropriately in the following class and emphasized the tone when talking about important knowledge points. After the teacher took a series of teaching measures, the students' attention returned to the 0.75-1 range.

[0148] Because the present invention provides cognitive status feedback function, teachers will frequently interact with students during class. For example, after explaining a knowledge point, the teacher asks students a question: "Do you all understand?" After asking the question, the teacher checks the students' nodding and shaking heads in the summary view. He finds that most students have red circles below, but a few students have green circles below. Fig.11 As shown in the figure, most students understood, but there were still a few students who were confused. So he stopped teaching and asked these students to type their questions into the chat interface. After answering the students' questions, he continued the course.

[0149] In addition, the teacher is also very concerned about the emotional state of students. The teacher said that he would try his best to keep students in a natural and happy emotional state during class. However, during class, the teacher found that many negative emotions such as sadness, anger, and frustration were added to the summary view. So he chose to sort by emotion in the summary view, and then swiped these students with negative emotions, and checked the emotional distribution of students in the past half minute in the distribution view. Fig.12 As shown in the figure, he found that the students' negative emotions had lasted for a while. Therefore, the teacher temporarily interrupted the class and told the students a little joke to liven up the classroom atmosphere. Then the number of happy students in the summary view increased significantly. The teacher said that he received timely feedback, which significantly improved his teaching confidence. In the following teaching process, he slowed down the pace of the class and tried to explain every knowledge point as clearly as possible.

[0150] The above cases show that the present invention can effectively convey student participation information to teachers, helping teachers make appropriate teaching decisions. In addition, the present invention can effectively promote interaction between teachers and students and improve the teaching quality and learning effect of online classes.

[0151] Case Study 2: Identifying Students with Abnormal Engagement

[0152] In online classes, there is a serious problem of time and space isolation between teachers and students. It is difficult for teachers to find students who are out of class or have abnormal participation, so they cannot correct these students in time, resulting in poor learning results. The present invention can help teachers quickly find students with abnormal participation. For example, during the experiment, the teacher found that there was always a student in the attention interval of 0-0.25, and the color was black from time to time, which meant that the student was occasionally not detected. In order to analyze the cause, the teacher checked the student's personal information in the summary view, and added the student to the monitoring view and personal view. The teacher found that in the monitoring view, the student always lowered his head and his line of sight was also facing downward, such as Fig.13 The teacher then sets the time slider to the last 90 seconds and looks at the student's personal view, as shown in Fig.14 As shown. He learned that the student started to lower his head at around 380s and kept doing so for a long time. Therefore, the teacher suspected that the student was playing with his phone. Next, the teacher clicked on the most recent time point to view the student's original image. By viewing the student's original image, the teacher confirmed his judgment. So he gave the student a verbal warning.

[0153] In addition, during class, the teacher adjusts the time span of the distribution view to the last 30 seconds to view the student participation in the recent period. Through the distribution view, the teacher can easily detect the abnormal participation behavior of students. For example, the teacher finds that the fatigue level of student 200048 has increased to 1. Fig.15 As shown in (a), the teacher adds it to the monitoring view and observes it for a period of time, and finds that its eyelids are always fully closed. Fig.15 (b) As shown. Finally, the teacher views the student's detailed information in the personal view, such as Fig.15 As shown in (c), he found that the student's eye length-to-width ratio was less than the eye-closing threshold in the recent period, so the teacher judged that the student was asleep. The teacher criticized the student by name but the student did not respond, so the teacher asked other students to call and wake up the student.

[0154] Through the distribution view, the teacher also found that the pie chart in the middle of a student was almost entirely purple, which meant that the student was feeling disgusted with the course. When the teacher found this phenomenon, he asked the student questions and brought the student back to the class.

[0155] The teacher said that the system can effectively help him find students with abnormal participation and understand the reasons behind the abnormal participation, and then adopt effective teaching intervention measures, which is difficult to do in normal online teaching.

[0156] Case Study 3: Comparing Different Students’ Engagement

[0157] It is very important for teachers to compare the participation of different students. By comparing between students, teachers can find out which students are active and which are not in class, which provides favorable support for teachers to grade classroom performance, and can also help teachers find students of interest. In this experiment, the teacher found that more students were in a negative state, so he sorted them by emotion in the summary view. Then he selected students with sad and disgusted emotions and observed them in the monitoring view. In order to liven up the atmosphere, the teacher told a joke, and most of the students in the monitoring view changed their emotions to happy or natural. However, through comparison, the teacher found that student 100015 was still in a sad state, such as Fig.16 As shown in the figure, the teacher judged that the student needed some help. So he called the student by name, connected the microphone with him, and asked if he had any difficulties.

[0158] After the course is complete, the teacher sets the time slider in the distribution view to the entire course length and compares the participation of different students. Fig.17 As shown in the left view of , the teacher finds students with high and low participation through the distribution view. Finally, he compares the participation details of different students in the individual view. Fig.17 As shown in the right view of the video, he found that students with high participation had relatively fewer head movements, smaller eye gaze fluctuations, and relatively positive emotions, while students with low participation had more head movements, larger eye gaze fluctuations, and more negative emotions.

Claims

1. A real-time visual analysis method for student engagement in an online course, comprising the following steps: S1. Real-time acquisition of students’ videos in online courses; S2. Preprocess the video data obtained in step S1 to extract the facial features of the students; S3. Establish a student engagement analysis model based on four aspects: attention, emotion, fatigue level and cognitive state. The specific steps include: Attention: The following formula is used to calculate the student viewing screen state variable S: S = "|X|≤0.5" & "|Y|≤0.5"; where [X, Y] is the pre-processed student line of sight direction; the value rule of "a" is: if a is true, "a" = 1, if a is false, "a" = 0; & is an AND operation; when the student viewing screen state variable S is 1, it indicates that the student is viewing the screen; when the student viewing screen state variable S is 0, it indicates that the student is not viewing the screen; A sliding window with a time span of 30 seconds is used to receive the student's line of sight [X, Y] and the student's viewing screen state variable S, and N = [X1, X2, ..., X n ]、M=[Y1,Y2,...,Y n ] and W=[S1,S2,...,S n ], where X i is the horizontal sight direction of the student in the i-th frame, Y i is the vertical sight direction of the student in the i-th frame, S i is the state variable of the student watching the screen in the i-th frame; the student daze state variable D is calculated as D = "(max(N)-min(N))≤T"&"(max(M)-min(M))≤T", where max(N) is the maximum value of the elements in N, min(N) is the minimum value of the elements in N, D is 1 for the student not dazed, and D is 0 for the student dazed; finally, the student's attention variable A is calculated using the following formula: Fatigue level: (1) For students, the average eye aspect ratio B2, blink frequency P2, average blink duration D2 and yawn frequency M2 were selected as indicators to model the students' fatigue level; Extract the eye key points and mouth key points of the face: the eye key points include 6 key points, the eye key point E1 at the outermost corner of the eye, the eye key point E4 at the innermost corner of the eye, the eye key point E2 at the outer 1 / 4 of the upper eye contour, the eye key point E4 at the inner 1 / 4 of the upper eye contour, the eye key point E6 at the outer 1 / 4 of the lower eye contour, and the eye key point E5 at the inner 1 / 4 of the lower eye contour; the mouth key point information includes 4 key points, the leftmost key point M1 of the mouth contour, the rightmost key point M3 of the mouth contour, the highest key point M2 of the upper lip, and the lowest key point M4 of the lower lip; each key point is a two-dimensional coordinate; The following formula is used to calculate the student's eye aspect ratio α: Where |E2-E6| represents the straight-line distance between key point E2 and key point E6; |E3-E5| represents the straight-line distance between key point E3 and key point E5; |E1-E4| represents the straight-line distance between key point E1 and key point E4; The following formula is used to calculate the length-to-width ratio of the student's mouth: Where M3-M1| represents the straight-line distance between key point M3 and key point M1, and |M2-M4| represents the straight-line distance between key point M2 and key point M4; The eye length-to-width ratio and mouth length-to-width ratio of each student are obtained in advance and normalized so that the eye length-to-width ratio α is 0 when the eyes are fully closed and 1 when the eyes are fully open; the mouth length-to-width ratio β is 0 when the mouth is fully closed and 1 when the mouth is fully open; When it is detected that the aspect ratio α of the student's eyes is less than the set threshold, it is considered that the student has closed eyes; (2) Using a sliding window of 30 seconds, the video image is intercepted and F = [B1, B2, ..., B n ], where B i is the eye length-to-width ratio of the i-th frame image in the sliding window; the average eye length-to-width ratio B of the students is calculated using the following formula: The blinking frequency P is defined as the number of blinking actions of students in 30 seconds; A sliding window with a duration of 30 seconds is used to capture the video image, and G = [D1, D2, ..., D m ], where m blinks are detected in the sliding window, D i is the duration of the i-th blink; the average blink duration D of the students is calculated using the following formula: When the mouth length-to-width ratio is greater than the set threshold and the duration is greater than the set threshold, the student is deemed to have yawned; the student's yawn frequency M is the number of yawns within 30 seconds; The dynamic weight fuzzy comprehensive evaluation method was used to analyze the students' fatigue level: the initial average eye aspect ratio B weight, blink frequency P weight, average blink duration D weight and yawn frequency M weight were 0.42, 0.13, 0.32 and 0.13 respectively; when B < 0.2, the average eye aspect ratio B weight, blink frequency P weight, average blink duration D weight and yawn frequency M weight were adjusted to 0.8, 0.1, 0.1 and 0 respectively; each indicator corresponds to a corresponding value V at different values; when B = 1, the value is 0, when B = 0.8, the value is 0.2, when B = 0.6, the value is 0.4, when B = 0.4, the value is 0.6, and when B = 0 .2, the value is 0.8, and the value is 1 when B=0; the value is 0 when P is between 6 and 8, the value is 0.2 when P is 5 or 9, the value is 0.4 when P is equal to 4 or in the range of 10 to 12, the value is 0.6 when P is equal to 3 or in the range of 12 to 14, the value is 0.8 when P is equal to 2 or in the range of 15 to 16, and the value is 1 when P≤1 or P>16; the value is 0 when D is between 0.2 and 0.4, the value is 0.2 when D is between 0.5 and 0.6, the value is 0.4 when D is between 0.7 and 0.8, the value is 0.6 when D is between 0.8 and 1.5, the value is 0.8 when D is between 1.5 and 3, and the value is 1 when D is above 3; When M is equal to 0, the value is 0; when M is equal to 1, the value is 0.4; when M is equal to 2, the value is 0.6; when M is equal to 3, the value is 0.8; when M≥4, the value is 1; finally, the fatigue degree F is calculated as: F=V B *W B +V P *W P +V D *W D +V M *W M , where W B is the average eye aspect ratio B weight, W P is the blink frequency P weight, W D is the average blink duration D weight, W M is the yawning frequency M weight, V B is the value of the eye aspect ratio B, V P is the value of blink frequency P, V D is the value of the average blink duration D, V M is the value of yawn frequency M; Emotional Aspects: Using the sentiment classification model, the sentiments are classified into anger, natural, sad, surprised, happy, disgusted and afraid; Cognitive status: It is stipulated that students nod when they understand the course content and shake their heads when they do not understand the course content; Detect the student's head movement: extract the yaw angle and pitch angle of the student's head, and then establish two sliding windows with a time span of 3s to receive the yaw angle and pitch angle respectively; Use the following steps to detect students' nodding and shaking behaviors: It is stipulated that if the head direction continuously drops and exceeds the set threshold, it is determined that the head is lowered; if the head direction continuously rises and exceeds the set threshold, it is determined that the head is raised; if both lowering and raising behaviors are detected within the observation interval, it is determined that the head is nodded; S4. Visualize the analysis results of the student engagement analysis model established in step S3, thereby completing the real-time visualization analysis of student engagement in the online course.

2. The real-time visual analysis method for student participation in online courses according to claim 1 is characterized in that The real-time acquisition of the video of the student in the online course described in step S1 specifically includes the following steps: Use cameras to capture real-time video of students in online courses; According to the blinking frequency of human eyes, set the number of frames per second to collect images; Before the video collection begins, collect images of students looking directly at the center of the screen and observing the four vertices around the screen from the center of the screen {I c ,I lt ,I lb ,I rt ,I rb }, used to check the head direction and sight direction; where I c For students to face the image in the center of the screen, lt This is the image when the student observes the top left corner of the screen from the center of the screen. lb This is the image when the student observes the vertex in the lower left corner of the screen from the center of the screen. rt This is the image when the student observes the vertex in the upper right corner of the screen from the center of the screen. rb This is the image when students observe the vertex in the lower right corner of the screen in the center of the screen.

3. The real-time visual analysis method for student participation in online courses according to claim 2 is characterized in that Step S2 of preprocessing the video data obtained in step S1 to extract the facial features of the student specifically includes the following steps: Perform preprocessing on the client where the student is located; the preprocessing includes face recognition preprocessing, face alignment preprocessing, emotion recognition preprocessing, head posture estimation preprocessing and sight estimation preprocessing; Among them, face recognition preprocessing is to use a deep learning model to detect whether there is face information in the collected image; face alignment preprocessing is to use a neural network model to extract the face coordinate position information in the collected image; emotion recognition preprocessing is to use a neural network model to identify the students' real-time emotions; the emotions include anger, naturalness, sadness, surprise, happiness, disgust and fear; head posture estimation preprocessing is to use head posture technology to extract the head posture angle of the person from the collected image, and judge the relationship between the student's head and the screen based on the head posture angle; line of sight estimation preprocessing is to use a full-face-based line of sight estimation algorithm to extract the student's line of sight direction, and judge the relationship between the student's line of sight and the screen based on the line of sight direction.

4. The real-time visual analysis method for student participation in online courses according to claim 3 is characterized in that The face recognition preprocessing specifically includes the following steps: A multi-task cascade convolutional neural network deep learning model is used to detect whether an image contains facial information; the multi-task cascade convolutional neural network deep learning model detects the position (x, y, w, h) of the face matrix in the image, where (x, y) represents the coordinates of the upper left corner of the face matrix when the upper left corner of the image is the coordinate origin, w represents the width of the rectangular area corresponding to the detected face matrix, and h is the height of the rectangular area corresponding to the detected face matrix; in specific implementation, if the face matrix is ​​detected, it means that the face information is recognized.

5. The real-time visual analysis method for student participation in online courses according to claim 4 is characterized in that The face alignment preprocessing specifically includes the following steps: The cascade convolutional neural network model is used to extract the coordinates of key points of the face. When extracting the coordinates, the upper left corner of the image is taken as the origin, and the extracted coordinates of the key points of the face are [x i ,y i ].

6. The real-time visual analysis method for student participation in online courses according to claim 5 is characterized in that The emotion recognition preprocessing specifically includes the following steps: ResNet-50 was used as the convolutional neural network model, and the FER2013 facial expression recognition public dataset was used for training to obtain the final preprocessing model; the emotions output by the preprocessing model included anger, naturalness, sadness, surprise, happiness, disgust, and fear.

7. The real-time visual analysis method for student participation in online courses according to claim 6 is characterized in that The head posture estimation preprocessing specifically includes the following steps: The head posture estimation algorithm is used to obtain the head posture of the person from the image; the head posture is represented by three Euler angles, namely pitch angle, yaw angle and roll angle, which are used to represent nodding, shaking head and turning head respectively; The head posture estimation algorithm is based on the key point 2D projection to 3D method to extract the head posture; the head posture is represented by a three-dimensional vector [Pitch, Yaw, Roll], where Pitch is the pitch angle, Yaw is the yaw angle, and Roll is the roll angle; Before preprocessing the head posture estimation, a preliminary check is performed; during the check, the head posture angle of the student facing the center of the screen is extracted, as well as the head posture angles in four directions when the student's head is at the center of the screen and observes the surroundings of the screen; during the check, the student's head posture is proportionally converted so that the Pitch value and Yaw value when the student is looking at the center of the screen are both 0, and the absolute value of the pitch angle of the head when the student is looking at the upper and lower edges of the screen and the yaw angle when the student is looking at the left and right edges is 0.5; During calibration, the following formula is used to calculate the pitch angle Pitch2 after calibration: Where Pitch1 is the pitch angle detected, Pitch c Pitch is the pitch angle when students look directly at the center of the screen. t The pitch angle for students to see the border on the screen; After verification, if the deflection angle value is greater than 1, it will be directly corrected to the maximum value 1; Finally, when the deflection angle is between -0.5 and 0.5, it is determined that the student's head is facing the screen.

8. The real-time visual analysis method for student participation in online courses according to claim 7 is characterized in that The sight line estimation preprocessing specifically comprises the following steps: The gaze estimation algorithm based on the whole face is used to extract the gaze direction of the students, and the gaze direction of the students is obtained as [X, Y], where X is the rotation angle of the gaze in the horizontal direction and Y is the rotation angle of the gaze in the vertical direction; Before the preprocessing of line of sight estimation, a preliminary verification is performed; during the verification, the line of sight angles of the student looking at the center of the screen and the surroundings of the screen when the student's head is at the center of the screen are extracted; then according to the verification results, the line of sight data is converted so that the line of sight when the student looks at the center of the screen is [0,0], and the absolute value of the angle corresponding to the student's line of sight looking at the surroundings of the screen is 0.5; During calibration, the angle value Y2 of the sight direction after calibration is calculated using the following formula: Where Y1 is the angle value of the line of sight direction detected, Y c Y is the angle of the student's line of sight when looking directly at the center of the screen. t It is the angle value of the direction of the students’ line of sight when they look at the border on the screen; After verification, if the angle value of the sight direction is greater than 1, it will be directly corrected to the maximum value of 1; Finally, when the angle value of the line of sight direction is between -0.5 and 0.5, it is determined that the student is looking at the screen.

9. The real-time visual analysis method for student participation in online courses according to claim 8 is characterized in that Step S4 of visualizing the analysis results of the student engagement analysis model established in step S3 specifically includes the following steps: (1) Construct summary view, monitoring view, distribution view and personal view to visualize the analysis results; the summary view is used to present the participation of all students in real time; the monitoring view is used to present the participation of a specified student group in real time; the distribution view is used to present the distribution of students' participation in a specified time period; and the personal view is used to display the relevant characteristics of individual students' participation; (2) Summary view: A summary view is constructed based on a stacked bar chart to simultaneously present four indicators: student emotion, fatigue, attention, and student status; it presents student engagement from three perspectives: attention, fatigue, and emotion; A view that provides an overview of student engagement from the perspective of attention and fatigue: the Y-axis represents different levels of attention intervals, which are divided into four groups from low to high; the X-axis represents the number of students in each group; each column consists of several small rectangles, each rectangle represents a student, and the color of the rectangle corresponds to the student's emotion; an embedded dark matrix is ​​used in each small rectangle to represent the degree of fatigue; the height of the embedded matrix represents the value of the degree of fatigue; the students are sorted in descending order of fatigue to avoid visual confusion; circular icons of different colors are used under each rectangle to present the student's cognitive state and daze state; through a view that provides an overview of student engagement from the perspective of attention and fatigue, teachers can quickly get an overview of student engagement; A view that provides an overview of student engagement from an emotional perspective: the Y axis represents the student's emotional category, and the length of the rectangular column corresponds to the number of students in a certain category of emotion; each column consists of several rectangles, each representing a student, with two dark rectangles embedded in the rectangle, the height of the upper rectangle corresponds to the student's attention, and the height of the lower rectangle corresponds to the student's fatigue level; sorting by attention; teachers can switch to different angles according to the teaching scenario to overview the student's engagement; when the teacher hovers the mouse over the view, the view will stop updating in real time to facilitate the teacher to view or select students; when the mouse hovers over a student, the student's detailed information will be displayed; click to view the student's information; teachers can select the student group of interest in the view and conduct further analysis; (3) Monitoring view: used to help teachers keep an eye on the participation of students of interest; construct a face model; each face represents a student, and the color of the hair corresponds to the student's emotion, and the emotion color is consistent with the summary view; the relative position of the eyeball in the eye corresponds to the student's line of sight; if the student looks directly at the screen, the eyeball is in the center of the eye; the relative position of the hair, eyes, nose and mouth in the face is used to encode the direction of head rotation; the height of the eyelids is used to encode the student's fatigue level; through the monitoring view, teachers can perceive students' abnormal participation behavior; (4) Distribution view: used to help teachers understand the distribution of students' participation in a certain period of time. The distribution view is a matrix, in which each grid represents a student. Each student is represented by a glyph design, and the pie chart in the middle represents the distribution of students' emotions in a specified period of time. The rings next to the pie chart represent attention. The angle of the dark ring corresponds to the average attention in the period of time, and the angle of the top of the light ring corresponds to the upper quartile of attention. The outermost ring represents fatigue. The angle of the dark ring corresponds to the average fatigue, and a full circle indicates complete fatigue. The angle of the top of the light ring represents the upper quartile of fatigue. The distribution view can be sorted in a variety of ways. After the sorting is completed, the student position will remain fixed and updated in real time. When the teacher selects some students from the summary view, the distribution view will default to sorting these student groups from high to low in terms of attention. There is a time bar slider in the distribution view, and the teacher can use the slider to select a time period and view the corresponding information. The teacher can view the detailed information of the students by hovering the mouse, and add the students to the monitoring view and personal view by clicking. (5) Personal view: used to provide teachers with relevant characteristic information of student participation and the original image of the student; the time span presented in the personal view is consistent with the time slider in the distribution view; the personal view contains the gaze direction, head direction, eye aspect ratio, mouth aspect ratio, the original image of the student and the student's action information; the X-axis in the personal view represents time; there is a set of dotted lines at the top and bottom of the personal view, which are used to represent the gaze direction of the student when looking at the upper and lower edges of the screen; there is a flow graph in the dotted line, the width of the flow graph is equal to the width between the two dotted lines, and the position represents the direction of the head; the upper flow graph represents the direction of the head rotating around the Y axis, and the lower flow graph represents the direction of the head rotating around the X axis; the color-coded solid line is used to represent the gaze direction; if the line is between the two The middle of the color line indicates watching the screen, and if the line is outside the color line, it means that the student's eyes are away from the screen; the color of the line encodes the student's emotion category; the color of the emotion is consistent with the summary view; there are two Gantt charts in the middle of the two river charts, the first Gantt chart shows blinking information, the blank area in the Gantt chart indicates eye closure, and the transparency of the matrix in the Gantt chart encodes the average eye length-to-width ratio between two blinks; the second Gantt chart presents mouth opening information, if a specific color rectangle appears, it means the mouth is open, and if not, it means closed mouth; color-coded circular icons below the first Gantt chart represent students' actions, the first color indicates nodding, and the second color indicates shaking head; when the teacher clicks on the flow chart at a certain moment in the figure, the original image of the student at that moment will be displayed to help the teacher verify his or her analysis results.