Online learning attention monitoring method based on real-time multi-modal feature fusion

Abstract

The invention relates to the field of online learning monitoring, and discloses an online learning attention monitoring method based on real-time multi-modal feature fusion, and the method comprises the following steps: S1, starting a camera to collect a facial image of a student; s2, calculating the distance from the face of the student to the screen; s3, resolving a horizontal eyeball staring vector, a longitudinal eyeball staring vector and an eye opening and closing degree; s4, resolving to obtain a final horizontal eyeball staring vector and a final longitudinal eyeball staring vector; s5, obtaining a lecture attending state index based on a resolving result; and S6, according to the class attending state index, giving a prompt and a praise, and recording. According to the invention, the eye movement tracking and face orientation technology is combined, the eyeball staring vector and the face orientation vector are calculated, and the attention state of the student is accurately monitored. The attention change is objectively evaluated in real time, compared with traditional questionnaires or subjective evaluation, the problem of lack of objectivity and real-time performance is solved, the attention of students is timely fed back, and the learning effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of online learning monitoring, and specifically to an online learning attention monitoring method based on real-time multi-modal feature fusion. Background Art

[0002] With the popularization of online education, more and more students are learning through online courses. Although this flexible learning method is convenient for students to arrange time according to their personal pace, it also brings a series of challenges in the learning process. Especially in the online environment lacking real-time supervision, students are prone to distraction, resulting in a decline in learning effects. The flexibility of online education weakens students' self-discipline ability to a certain extent, and students may find it difficult to stay focused and complete the course content.

[0003] Most traditional methods for evaluating students' learning status rely on self-report or teachers' subjective evaluation. These methods have certain limitations, lacking objectivity and real-time nature, and it is difficult to comprehensively reflect students' true learning status. In addition, existing learning status evaluation tools often rely on questionnaires or observations of students' behaviors. These methods usually cannot effectively capture students' attention changes, especially in the context of online courses, and it is difficult to conduct real-time and accurate status monitoring.

[0004] Eye movement tracking technology, as a non-invasive and efficient means for monitoring students' status, has gradually become a research hotspot. Through eye movement tracking technology, students' eye movement data can be obtained in real time, and then their attention concentration can be inferred. However, existing eye movement tracking technologies often rely on expensive professional equipment such as eye trackers, which makes it face a high cost threshold in the process of popularization and application and is difficult to adapt to low-requirement hardware environments such as home learning. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the present invention provides an online learning attention monitoring method based on real-time multi-modal feature fusion, which solves the problem of how to reasonably and effectively capture students' attention changes in online education.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: An online learning attention monitoring method based on real-time multi-modal feature fusion, including the following steps:

[0007] S1. After the course starts, start the camera to collect students' facial images, extract facial key points through Mediapipe. The facial images include the actual width of the eye socket, the degree of eyelid closure, and the diameter of the eyeball. The key points include eye socket key points and pupil center key points;

[0008] S2. Calculate the distance from the students' face to the screen based on the camera focal length and the actual width of the eye socket;

[0009] S3. Based on the facial image and the distance from the student's face to the screen, adopt a step-by-step recognition strategy to calculate the horizontal eye gaze vector, the vertical eye gaze vector, and the degree of eye opening and closing respectively;

[0010] S4. Calculate the student's face orientation vector and add it to the horizontal eye gaze vector and the vertical eye gaze vector respectively to obtain the final horizontal eye gaze vector and the final vertical eye gaze vector. During this period, use the Kalman filter and the median filter to exclude interference factors;

[0011] S5. Calculate the class listening state index based on the modulus of the final horizontal eye gaze vector, the modulus of the final vertical eye gaze vector, and the degree of eyelid closure;

[0012] S6. Give reminders and praises according to the class listening state index and record them.

[0013] Preferably, the S1 includes the following steps:

[0014] S1.1. Read the user's camera and capture the student's head photo;

[0015] S1.2. Use Mediapipe to identify the eye frame key points and the pupil center key points for the student's head photo obtained in step S1.1, and calculate the relative two-dimensional key point coordinates according to the camera screen size.

[0016] Preferably, the step S2 includes the following steps:

[0017] S2.1. Use a measuring tool to measure the camera focal length f and the actual width w of the eye socket e ;

[0018] S2.2. Calculate the distance d from the student's face to the screen according to the formula. The specific calculation formula is:

[0019]

[0020] where w0 is the reference width, which is the width of the eyes under standard or calibrated settings.

[0021] Preferably, the step S3 includes the following steps:

[0022] S3.1. Calculate the binocular horizontal eye gaze vector according to the eye key points The specific calculation formula is:

[0023]

[0024] where is the difference vector between the pupil center and the eye center; r is the eye radius; is the distance d from the student's face to the screen; h real is the absolute offset;

[0025] S3.2. Obtain the deformation degrees of the upper and lower eyelids of both eyes through the eye key points, and calculate the longitudinal gaze vectors of both eyes according to the different degrees of deformation of the upper and lower eyelids generated when the human eyes move vertically. , and the specific calculation formula is:

[0026]

[0027] Among them, both γ and α are correction factors, and δ is a deformation index;

[0028] S3.3. Obtain the vertical height of both eyes through the eye key points, that is, the eye opening degree e lid .

[0029] Preferably, in the step S3.1, the absolute offset h real is the abscissa length value of the offset between the pupil center and the eye center in centimeters in real life, and the calculation formula is:

[0030]

[0031] Among them, w0 is the proportion of the orbital width in the camera screen; d is the distance from the student's face to the screen; f is the focal length of the camera.

[0032] Preferably, in the step S3.2, considering the non-uniformity of the deformation of the upper and lower eyelids and the influence of the change in the distance from the screen on the deformation index δ, the calculation formula is:

[0033] δ = σ + ηd - β;

[0034] Among them, σ is the relative deformation difference, η is the distance factor, β is the offset, and d is the distance from the student's face to the screen;

[0035] Among them, the calculation method of the relative deformation difference σ is:

[0036]

[0037] Among them, N is the total number of samples, and L represents the set of the vertical coordinates of the key points; is the mean value of the set of the vertical coordinates of the key points; L i is the vertical coordinate value of the i-th key point.

[0038] Preferably, the S4 includes the following steps:

[0039] S4.1. Use the solvePnP method of OpenCV to calculate the face orientation vector m;

[0040] S4.2. Use a filter to perform temporal smoothing processing on the following data:

[0041] (1) The Kalman filter filters the inter-frame mutation value of the binocular longitudinal eye gaze vector in step S3.2 generated by blinking.

[0042] (2) The Kalman filter filters the high-frequency jitter of the binocular horizontal eye gaze vector in step S3.1 caused by small errors.

[0043] (3) The median filter filters the inter-frame mutation value of the eye opening and closing degree e in step S3.3 caused by blinking. lid

[0044] S4.3 Utilizes the average vector of the binocular eye gaze vectors obtained in step S3 .

[0045] S4.4 Utilizes the face orientation vector m and adds it to the binocular eye gaze vectors obtained in step S4.2 and the binocular longitudinal eye gaze vectors obtained in step S3.2 to obtain the combined eye gaze vector .

[0046] Preferably, the said step S5 includes the following steps:

[0047] S5.1: Calculate the horizontal attention index S h :

[0048] S h = 2Score(-0.05, max(0, |g x | - 60), 0);

[0049] S5.2: Calculate the longitudinal attention index S v :

[0050] S v = 2Score(-0.065, max(0, |g y | - 40), 0);

[0051] Wherein, g x is the horizontal gaze offset; g y is the vertical gaze offset;

[0052] S5.3: Since the eyelid closure degree can reflect the fatigue degree and concentration degree, it is also necessary to calculate the eyelid closure index S e :

[0053] S e = Score(9, e lid , 1.16);

[0054] Wherein, elid is the degree of eye opening and closing; Score(.) is an exponential decay scoring function;

[0055] S5.4 The final class listening state index S consists of 30% of the horizontal attention index S h , 45% of the longitudinal attention index S v , and 25% of the eyelid closure index S e .

[0056] Preferably, the exponential decay scoring function is defined as follows:

[0057]

[0058] where μ is the curve steepness parameter; E is the input variable; E0 is the activation threshold; e is the base of the natural logarithm.

[0059] Preferably, the step S6 includes the following steps:

[0060] S6.1 When the class listening state index S obtained in step S5.4 is greater than the maximum threshold for a long time, praise the student;

[0061] S6.2 When the class listening state index S obtained in step S5.4 is less than the minimum threshold for a long time, remind the student;

[0062] S6.3 Record the class listening state index S obtained in step S5.4 until the end of the course.

[0063] The present invention provides an online learning attention monitoring method based on real-time multimodal feature fusion. It has the following beneficial effects:

[0064] 1. By adopting the technical solution of combining eye movement tracking and facial orientation, the present invention accurately monitors the attention state of students by calculating the eye gaze vector and facial orientation vector of students. It achieves the technical effect of objectively and real-time evaluating the change of students' attention. Compared with the traditional technical solution relying on questionnaire surveys or teachers' subjective evaluations, it solves the deficiencies of lack of objectivity and real-time. The attention state of students can be instantaneously feedback, which helps to improve the learning effect.

[0065] 2. By only tracking eye movement through a camera, the present invention breaks through the cost bottleneck of expensive traditional eye tracking devices and achieves the effect of reducing technical costs. Compared with the existing technical solutions relying on expensive eye tracking devices, the present invention can benefit more students and educational platforms, especially suitable for scenarios with low hardware requirements such as home learning, and improves the popularity and operability of the technology.

[0066] 3. The present invention smooths the eye movement data by using a Kalman filter and a median filter, avoiding data fluctuations caused by blinking and minor errors. It achieves the effect of improving the stability and accuracy of eye movement data. Compared with the eye movement tracking solutions without filtering in the prior art, the present invention can reduce error interference in practical applications, making the learning state assessment more accurate and reliable.

[0067] 4. By combining the student's listening state index with an intelligent feedback mechanism, the present invention can give real-time reminders and praises according to the student's learning state. This dynamically adjusted feedback scheme achieves the effect of improving the student's learning concentration. Compared with the traditional static evaluation method, the present invention can more effectively help students maintain concentration, timely correct distracted behaviors, and thus improve the efficiency of online learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the specification of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0070] For a better understanding of the present invention, the above content will be described in detail below in conjunction with specific embodiments.

[0071] Please refer to the attached Figure 1 , the embodiment of the present invention provides an online learning attention monitoring method based on real-time multimodal feature fusion, including the following steps:

[0072] S1. After the course starts, start the camera to collect the student's facial image, and extract facial key points through Mediapipe. The facial image includes the actual width of the eye socket, the degree of eyelid closure, and the diameter of the eyeball. The key points include eye socket key points and pupil center key points;

[0073] In this embodiment, step S1 is the starting stage of online learning attention monitoring, aiming to collect the student's facial image through the camera and extract relevant facial features, so as to realize the real-time monitoring of the student's attention state. This step uses computer vision technology and combines facial recognition algorithms (such as Mediapipe) to obtain the key point information of the student's eyes, laying a foundation for the subsequent calculation of the eye gaze vector and the evaluation of the attention state.

[0074] Specifically, step S1 includes two main operations: First, start the camera to collect facial images of students; Second, process the captured head images through the Mediapipe framework to extract key points of the eye sockets and the center of the pupils. In addition, parameters such as the calculated eye width, eyelid closure degree, and eyeball diameter are also key information of concern in this step. This information provides basic data for the calculation of the eyeball gaze vector and attention state in subsequent steps.

[0075] Specifically, the specific implementation method of step S1 is as follows:

[0076] Collect facial images of students

[0077] First, the system starts the facial image collection process by reading the camera signal of the user. Usually, the resolution and image quality of the camera will have a certain impact on subsequent processing, so it is necessary to ensure the image clarity to accurately extract facial key points. In general implementation, the captured facial images will include the entire facial area of the student, especially the area where the eyes and pupils are located. During this process, the configuration of the camera needs to be adjusted according to the actual application to ensure that the captured facial images can clearly present the details of the eyes.

[0078] Extract facial key points

[0079] In this embodiment, the Mediapipe framework is used to process the captured head images of students. Mediapipe is a commonly used computer vision technology framework that can accurately identify and extract facial features including key points such as eye frames and pupil centers. Specifically, Mediapipe, through its facial landmark model, identifies a set of predefined facial key points, including the center positions of the eyes, eye sockets, and pupils. These key points play an important role in eyeball gaze calculation.

[0080] For the calculation of the eye socket width, it is first necessary to determine the actual width of the eyes based on the key points of the eye frames, and then calculate the eyeball gaze vector. The key point of the pupil center provides data support for the subsequent calculation of the gaze direction of the pupil. To enable the system to adapt to different camera perspectives, Mediapipe calculates the relatively two-dimensional key point coordinates by calibrating the camera screen size, which provides the necessary conditions for subsequent distance calculation and accurate solution of the eyeball gaze vector.

[0081] Calculate relevant parameters of the eyes

[0082] Through the extraction of the above key points, the system can obtain key information such as the actual width of the student's eye sockets, the degree of eyelid closure, and the eyeball diameter. Specifically:

[0083] Actual orbital width: The actual orbital width is obtained by calculating the distance between the key points of the left and right orbits.

[0084] Degree of eyelid closure: Based on the changes in the key points of the eyes, especially the up and down deformation degree of the eyelids, the degree of eyelid closure is evaluated, which plays an important role in subsequent judgments of the student's attention concentration and fatigue level.

[0085] Eye diameter: Through the relative positions of the key point of the pupil center and other key points of the eyes, the diameter of the eyeball is further calculated to help evaluate the eye movement range of the student.

[0086] Generally, the accuracy of key point extraction is a key factor affecting subsequent calculation results. Therefore, the system needs to preprocess the collected images to reduce noise interference and improve the accuracy of key point extraction. In addition, the calibration of the camera, the lighting conditions, and the student's facial posture will all affect the accuracy of key point extraction. Therefore, it is necessary to combine the coordinated optimization of hardware and software to ensure more accurate extraction of facial key points.

[0087] Therefore, by combining the Mediapipe framework and computer vision technology, the facial key point information of students can be obtained in real time and accurately. This information provides a solid foundation for subsequent calculations of the eye gaze vector and evaluation of the student's attention state. In this process, the calculations of parameters such as orbital width, degree of eyelid closure, and eye diameter all provide important reference values for subsequent attention monitoring. This technical solution can effectively support the attention monitoring of students in the online learning environment and provide a basis for personalized teaching and intelligent feedback.

[0088] S2. Calculate the distance from the student's face to the screen based on the camera focal length and the actual orbital width;

[0089] In this embodiment, the goal of step S2 is to calculate the distance between the student's face and the screen through the camera focal length and the actual orbital width. This step is crucial for subsequent calculations of the eye gaze vector and evaluation of the student's attention state. In step S1, the system has successfully collected the student's facial image and extracted relevant facial key point information, including the actual width of the eye frame and the position of the pupil center. Based on this information, step S2 further uses physical principles and geometric relationships, combined with the focal length data of the camera, to calculate the actual distance between the student and the screen. This process provides basic data for subsequent calculations of the eye gaze vector.

[0090] The specific implementation method of step S2 is as follows:

[0091] Use of measuring tools

[0092] First, to ensure the accuracy of the calculation, the system needs to use specialized measurement tools to obtain the focal length of the camera and the actual width of the eye socket. Generally, the focal length of the camera and the width of the eye socket are known parameters, which can be obtained through manual measurement or device parameter reading. The focal length is a basic physical property of the camera lens, while the actual width of the eye socket is obtained by analyzing the key points of the eye socket in the student's facial image.

[0093] In this way, the system can accurately capture the focal length of the camera and the actual width of the eye socket, providing essential input data for subsequent calculations.

[0094] Calculation of the distance from the face to the screen

[0095] In this embodiment, the system calculates the distance between the student's face and the screen based on the focal length and the width of the eye socket, using geometric relationships. Specifically, the distance between the student's face and the screen can be calculated by the following formula:

[0096]

[0097] where f is the measured focal length of the camera, w0 is the reference width, which is the width of the eyes under standard or calibrated settings; w e is the actual width of the eye socket; d is the distance from the student's face to the screen;

[0098] Accuracy and optimization

[0099] To ensure the accuracy of this calculation step, it is usually necessary to appropriately calibrate the focal length and eye width data. The calibration process usually includes:

[0100] Testing with a standard reference object of known distance (such as a card or calibration board of known size).

[0101] Calibrating different camera perspectives and focal length settings to eliminate errors caused by device differences.

[0102] To improve the adaptability of the system, the measurement of the camera focal length and the eye socket width can be adjusted according to the specific application environment. For example, in different online education platforms or remote learning environments, the installation position of the camera may be different, so appropriate adjustments and optimizations need to be made during specific implementation.

[0103] In addition, the influence of environmental factors, such as lighting conditions and the student's facial posture, can be taken into account in some embodiments to further improve the accuracy of the calculation results.

[0104] Therefore, by using the focal length of the camera and the orbital width data, the actual distance from the student's face to the screen is effectively calculated using geometric calculation methods. The implementation of this technical solution provides accurate basic data for the calculation of the eye gaze vector in the subsequent steps. Through precise calibration and optimization measures, the accuracy of the distance calculation between the face and the screen is ensured, thus providing reliable data support for the entire online learning attention monitoring system.

[0105] S3. Based on the facial image and the distance from the student's face to the screen, adopt a step-by-step recognition strategy to calculate the horizontal eye gaze vector, vertical eye gaze vector, and eye opening degree respectively;

[0106] In this embodiment, the main task of step S3 is to accurately calculate the student's eye gaze vector and eye opening degree based on the facial image data and the distance from the student's face to the screen obtained in the previous steps through a step-by-step recognition strategy. Specifically, first, through the key points of the eyes, the horizontal eye gaze vector, vertical eye gaze vector, and eye opening degree of both eyes are calculated respectively. These calculations provide key data support for evaluating the student's attention state and gaze behavior. Combining the distance information between the student and the screen, this embodiment can accurately evaluate the eye position and eyelid deformation of the student, thereby achieving accurate tracking of the student's visual focus.

[0107] The specific implementation method of step S3 is as follows:

[0108] S3.1. Calculate the horizontal eye gaze vector of both eyes according to the key points of the eyes The specific calculation formula is:

[0109]

[0110] Where, is the difference vector between the pupil center and the eye center; r is the radius of the eyeball; is the distance d from the student's face to the screen; h real is the absolute offset;

[0111] And the absolute offset h real is the abscissa length value of the offset between the pupil center and the eye center in centimeters in real life, and the calculation formula is:

[0112]

[0113] Where, w0 is the proportion of the orbital width in the camera screen; d is the distance from the student's face to the screen; f is the focal length of the camera.

[0114] Subsequently, through the key points of the eyes, the deformation degree of the upper and lower eyelids of both eyes is obtained, and according to the different degrees of deformation of the upper and lower eyelids generated when the human eye moves longitudinally, the vertical eye gaze vector of both eyes is calculated The specific calculation formula is as follows:

[0115]

[0116] Among them, both γ and α are correction factors, and δ is the deformation index;

[0117] Considering the non-uniformity of the deformation of the upper and lower eyelids and the influence of the change in the distance from the screen on the deformation index δ, its calculation formula is:

[0118] δ = σ + ηd - β;

[0119] Among them, σ is the relative deformation difference, η is the distance factor, β is the offset, and d is the distance from the student's face to the screen;

[0120] Among them, the calculation method of the relative deformation difference σ is:

[0121]

[0122] Among them, N is the total number of samples, and L represents the set of the vertical coordinates of the key points; is the mean value of the set of the vertical coordinates of the key points; L i is the vertical coordinate value of the i-th key point.

[0123] Therefore, by adopting the step-by-step recognition strategy, this embodiment can accurately calculate the horizontal and vertical eye gaze vectors of the student, as well as the degree of eye opening and closing. This technical solution effectively improves the calculation accuracy of the eye gaze vector by combining the facial key point data, camera focal length, and the distance information from the student's face to the screen obtained in the foregoing steps, providing reliable data support for subsequent student attention assessment. This solution can monitor the changes in the student's visual behavior in real time, provide personalized feedback for the online education platform, and thus optimize the learning effect

[0124] S4. Solve the student's face orientation vector and add it to the horizontal eye gaze vector and the vertical eye gaze vector respectively to obtain the final horizontal eye gaze vector and the final vertical eye gaze vector. During this process, the Kalman filter and the median filter are used to exclude interference factors;

[0125] S5. Calculate the listening state index based on the modulus of the final horizontal eye gaze vector, the modulus of the final vertical eye gaze vector, and the degree of eyelid closure;

[0126] In this embodiment, steps S4 and S5 involve the calculation of the student's face orientation, eye gaze vector, and degree of eyelid closure to comprehensively evaluate the student's listening state. By solving the face orientation vector, combining the relevant data of the eye gaze vector, and using the filter for time series smoothing processing to remove the interference of factors such as blinking, the horizontal and vertical attention indices, as well as the eyelid closure index, are finally calculated, and the final listening state index is synthesized.

[0127] Specifically, the core task of step S4 is to calculate the student's facial orientation vector and add this vector to the horizontal and vertical eye gaze vectors to obtain the final eye gaze vector. The specific implementation steps are as follows: First, use the solvePnP method in OpenCV to calculate the student's facial orientation vector through the known facial feature points and camera internal parameters. This method solves the Perspective-n-Point problem to obtain a 3D orientation vector representing the spatial direction of the student's face.

[0128] Data time series smoothing processing

[0129] To improve the stability of the eye gaze vector and avoid mutations caused by blinking or minor errors, a Kalman filter and a median filter are used to smooth the key data. The specific steps include:

[0130] Kalman filter processing of the vertical eye gaze vector: The vertical eye gaze vector obtained in step S3.2 is easily affected by blinking and generates frame-to-frame mutations. Smoothing it with a Kalman filter can effectively remove these mutation values and improve the stability of the data.

[0131] Kalman filter processing of the horizontal eye gaze vector: Similarly, the horizontal eye gaze vector obtained in step S3.1 may exhibit high-frequency jitter due to minor errors. The Kalman filter can effectively reduce these errors and make the data smoother.

[0132] Median filter processing of the eyelid opening and closing degree: In step S3.3, the eyelid opening and closing degree may be affected by blinking and generate obvious frame-to-frame mutations. The median filter effectively removes the outliers caused by blinking by calculating the median of the local window.

[0133] Calculating the average vector of the eye gaze vector

[0134] According to the binocular gaze vectors obtained in step S3, take their average value as the final eye gaze vector. This operation can, to a certain extent, remove the errors caused by unilateral eye movement and obtain a more accurate gaze vector.

[0135] Synthesis of the facial orientation vector and the eye gaze vector

[0136] Finally, synthesize the facial orientation vector with the horizontal and vertical eye gaze vectors to obtain the final eye gaze vector. Specifically, the facial orientation vector will be combined with the eye gaze vector smoothed in step S4.2 to obtain a more realistic and stable gaze vector. This synthesis operation takes into account the influence of the student's facial orientation on the eye gaze direction, thereby improving the accuracy and stability of the gaze vector.

[0137] Calculate the listening state index

[0138] The purpose of step S5 is to calculate the attention state of the student based on the final eyeball gaze vector and eyelid opening and closing degree. This process includes the following sub-steps:

[0139] Calculate the horizontal attention index

[0140] The calculation of the horizontal attention index is based on the magnitude of the final horizontal eyeball gaze vector, that is, to calculate the offset of the student's horizontal gaze. Generally, a smaller horizontal gaze offset indicates that the student is concentrated, while a larger offset indicates that the student may not be focused on the screen.

[0141] The formula is expressed as:

[0142] S h =2Score(-0.05,max(0,|g x |-60),0);

[0143] Calculate the vertical attention index

[0144] The calculation of the vertical attention index is similar to that of the horizontal attention index, but the vertical gaze offset is considered. A smaller vertical offset indicates that the student maintains better attention in the vertical direction.

[0145] The formula is expressed as:

[0146] S v =2Score(-0.065,max(0,|g y |-40),0);

[0147] where g x is the horizontal gaze offset; g y is the vertical gaze offset;

[0148] Calculate the eyelid closure index

[0149] Since the eyelid closure degree reflects the fatigue and concentration of the student, it is also necessary to calculate the eyelid closure index. This index is mainly based on the opening and closing degree of the eyelids and is weighted by an exponential decay scoring function. Specifically:

[0150] S e =Score(9,e lid ,1.16);

[0151] where e lid is the eye opening and closing degree; Score(.) is the exponential decay scoring function;

[0152] And the definition of the exponential decay scoring function is as follows:

[0153]

[0154] Among them, μ is the curve steepness parameter; E is the input variable; E0 is the activation threshold; e is the base of the natural logarithm.

[0155] Calculate the final class attendance status index

[0156] The final class attendance status index is obtained through weighted calculation, considering three aspects: horizontal attention, vertical attention, and eyelid closure. The specific weighting ratios are as follows:

[0157] 30% of the horizontal attention index S h ;

[0158] 45% of the vertical attention index S v ;

[0159] 25% of the eyelid closure index S e which is composed of;

[0160] Therefore, by combining multi-modal information of face orientation, eye gaze vector, and eyelid closure degree, a method for monitoring students' attention in online learning is designed. By resolving the face orientation vector, smoothing the eye gaze vector, and calculating the attention index and eyelid closure index, the class attendance status index is finally formed, providing a more accurate attention evaluation means for the online learning platform.

[0161] S6. Give reminders and praises according to the class attendance status index, and record.

[0162] In this embodiment, the purpose of step S6 is to provide feedback to students based on the calculated class attendance status index and record the change in their attention state. According to the students' class attendance status, the system can intelligently give praises or reminders to help students maintain or improve their learning attention. At the same time, the system will also record the students' class attendance status index, providing data support for subsequent analysis and learning behavior optimization.

[0163] Step S6 performs the state feedback of students based on the previously calculated class attendance status index, mainly including links such as praise, reminder, and status recording. The specific operations are as follows:

[0164] Give praise to students

[0165] In some embodiments, when the class attendance status index obtained in step S5.4 is greater than the maximum threshold for a long time, it indicates that the student has maintained a high level of concentration during the learning process. Generally, if this index is greater than the maximum threshold (for example, 90%) and lasts for more than a certain duration (such as 5 minutes), the system will determine that the student is in a good learning state.

[0166] As an option, in this case, the system will send praise feedback to the student, such as popping up a praise message or playing a voice prompt, to encourage the student to continue to maintain a high level of attention. For example, the system can display a message "Very good, keep it up!" to the student, or send an encouraging message like "You are listening to the class very attentively, keep it up!" through the voice assistant.

[0167] Giving a reminder to the student

[0168] In another implementation, when the listening state index obtained in step S5.4 is less than the minimum threshold for a long time, it indicates that the student's learning state may be relatively scattered or inattentive. Generally, if this index is lower than the minimum threshold (for example, 30%) and lasts for more than a certain duration (such as 5 minutes), the system will determine that the student is distracted.

[0169] Specifically, the system can send a reminder message to the student, suggesting that the student adjust their learning state. The reminder method can be a visual prompt or a voice prompt. For example, the system can pop up a message "Please concentrate, you seem to be a little distracted", or prompt through the voice assistant "Please refocus and avoid wandering".

[0170] Recording the listening state index

[0171] In some embodiments, in order to further optimize the learning process, the system will record the listening state index calculated in step S5.4 until the end of the course. Specifically, the system will store the listening state index at each moment in a database or a cloud platform to ensure that the changes in the listening state of the student can be comprehensively recorded throughout the learning process.

[0172] These records will include the listening state index within each time period, as well as whether praise or reminder has been given. For example, the system can store the listening state index at each time point together with the corresponding feedback (such as praise or reminder). Through these data, the platform can analyze the student's learning habits, attention fluctuations, etc., and provide data support for personalized recommendations for subsequent courses and optimization of learning behaviors.

[0173] Generally, the recorded data can help teachers or educational platforms analyze the student's learning progress and participation, and provide a basis for adjusting teaching strategies. In addition, these data can also be used in the student's own learning report to help the student understand their learning state, and then improve learning methods and time management.

[0174] Therefore, through the intelligent feedback mechanism, praise or reminder is automatically carried out according to the student's listening state index to ensure that the student's attention remains at the best level during the online learning process. The feedback mechanism of the system can not only improve the learning enthusiasm of students, but also help students adjust their learning state in time to avoid distraction. By recording the student's listening state index, the system provides valuable data support for teachers, students and educational platforms, promoting the continuous optimization of the learning process.

[0175] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An online learning attention monitoring method based on real-time multimodal feature fusion, characterized in that It includes the following steps: S1. After the course starts, activate the camera to collect students' facial images, and extract facial key points through Mediapipe. The facial images include the actual width of the eye sockets, the degree of eyelid closure, and the diameter of the eyeballs. The key points include eye socket key points and pupil center key points; S2. Calculate the distance from the students' faces to the screen based on the camera focal length and the actual width of the eye sockets; S3. Based on the facial images and the distance from the students' faces to the screen, adopt a step-by-step recognition strategy to calculate the horizontal eyeball gaze vector, the vertical eyeball gaze vector, and the degree of eye opening and closing respectively; S4. Calculate the students' facial orientation vector and add it to the horizontal eyeball gaze vector and the vertical eyeball gaze vector respectively to obtain the final horizontal eyeball gaze vector and the final vertical eyeball gaze vector. During this period, use Kalman filters and median filters to exclude interference factors; S5. Calculate the class listening status index based on the magnitudes of the final horizontal eyeball gaze vector, the final vertical eyeball gaze vector, and the degree of eyelid closure; S6. Give reminders and praises according to the class listening status index and record them.

2. The online learning attention monitoring method based on real-time multimodal feature fusion according to claim 1, wherein The S1 includes the following steps: S1.

1. Read the user's camera and collect photos of the students' heads; S1.

2. Use Mediapipe to identify the eye socket key points and pupil center key points for the students' head photos obtained in step S1.1, and calculate the relative two-dimensional key point coordinates based on the camera screen size.

3. The online learning attention monitoring method based on real-time multi-modal feature fusion according to claim 1, characterized in that The step S2 includes the following steps: S2.1 Use a measuring tool to measure the focal length f of the camera and the actual width w of the eye socket e ; S2.2 Calculate the distance d from the students' faces to the screen according to the formula. The specific calculation formula is: where w0 is the reference width, which is the width of the eyes under standard or calibrated settings.

4. The online learning attention monitoring method based on real-time multi-modal feature fusion according to claim 1, wherein The step S3 includes the following steps: S3.

1. Calculate the binocular horizontal eye gaze vector based on the eye key points The specific calculation formula is as follows: Among them, is the difference vector between the pupil center and the eye center; r is the radius of the eyeball; is the distance from the student's face to the screen; h real is the absolute offset; S3.

2. Obtain the deformation degrees of the upper and lower eyelids of both eyes through the eye key points, and calculate the vertical eye gaze vectors of both eyes according to the different degrees of deformation of the upper and lower eyelids generated when the human eyes move vertically. The specific calculation formula is as follows: where γ and α are both correction factors, and δ is the deformation index; S3.

3. Obtain the vertical height of both eyes through the eye key points, that is, the eye opening degree e lid .

5. The online learning attention monitoring method based on real-time multimodal feature fusion according to claim 4, characterized in that In the step S3.1, the absolute offset h real is the abscissa length value of the offset between the pupil center and the eye center in centimeters in real life, and the calculation formula is: where w0 is the proportion of the eye socket width in the camera screen; d is the distance from the students' faces to the screen; f is the camera focal length.

6. The online learning attention monitoring method based on real-time multi-modal feature fusion according to claim 4, characterized in that In step S3.2, considering the non-uniformity of the deformation of the upper and lower eyelids and the influence of the change in the distance to the screen on the deformation index δ, its calculation formula is: δ = σ + ηd - β; where σ is the relative deformation difference, η is the distance factor, β is the offset, and d is the distance from the students' faces to the screen; where the calculation method of the relative deformation difference σ is: Where N is the total number of samples, and L represents the set of the vertical coordinates of the key points; is the mean value of the set of the vertical coordinates of the key points; L i is the vertical coordinate value of the i-th key point.

7. The online learning attention monitoring method based on real-time multi-modal feature fusion according to claim 1, wherein The S4 includes the following steps: S4.1 Use the solvePnP method in OpenCV to calculate the facial orientation vector m; S4.2 Use filters to perform temporal smoothing processing on the following data: (1) The Kalman filter filters the inter-frame mutation values of the binocular longitudinal eye gaze vectors in step S3.2 generated by blinking ; (2) The Kalman filter filters the high-frequency jitter of the binocular horizontal eye gaze vector in step S3.1 caused by tiny errors ; (3) Median filter to filter the inter-frame mutation value of the eye opening and closing degree e in step S3.3 caused by blinking lid ; S4.3 Utilize the average vector of the binocular eye gaze vectors obtained in step S3 S4.4 Utilize the facial orientation vector m and add it to the binocular eye gaze vector obtained in step S4.2 and the binocular longitudinal eye gaze vector obtained in step S3.2 to obtain the combined eye gaze vector 8. The online learning attention monitoring method based on real-time multimodal feature fusion according to claim 1, wherein The step S5 includes the following steps: S5.1: Calculate the horizontal attention index S h : S h = 2Score(-0.05, max(0, |g x | - 60), 0); S5.2: Calculate the longitudinal attention index S v : S v = 2Score(-0.065, max(0, |g y | - 40), 0); where g x is the horizontal gaze offset; g y is the vertical gaze offset; S5.3: Since the degree of eyelid closure can reflect the degree of fatigue and concentration, it is necessary to additionally calculate the eyelid closure index S e : S e = Score(9, e lid , 1.16); where e lid is the degree of eye opening and closing; Score(.) is the exponential decay scoring function; S5.4 The final listening state index S consists of 30% of the horizontal attention index S h , 45% of the vertical attention index S v , and 25% of the eyelid closure index S e .

9. The online learning attention monitoring method based on real-time multi-modal feature fusion according to claim 8, wherein, The definition of the exponential decay scoring function is as follows: where μ is the curve steepness parameter; E is the input variable; E0 is the activation threshold; e is the base of the natural logarithm.

10. The online learning attention monitoring method based on real-time multi-modal feature fusion according to claim 8, wherein, The step S6 includes the following steps: S6.1 When the class listening status index S obtained in step S5.4 is greater than the maximum threshold for a long time, give praise to the students; S6.2 When the class listening status index S obtained in step S5.4 is less than the minimum threshold for a long time, give reminders to the students; S6.3 Record the class listening status index S obtained in step S5.4 until the end of the course.

Images