An online teaching platform capable of detecting student concentration

Abstract

The application relates to an online teaching platform capable of realizing student concentration detection, which mainly comprises a teacher terminal platform and a student terminal platform. The main function of the student terminal platform is to receive screen content shared by the teacher terminal and track student eye movement tracks in real time. The student eye movement tracks are tracked by collecting image information through a camera on the student terminal device, and then the eye movement tracks are tracked and sent to the server of the system. The private cloud server of the system analyzes, calculates and processes the received student attention data, and then judges the concentration of the students in class. The main function of the teacher terminal platform is to share the screen content of the teacher terminal to the devices of various student terminals and receive the student concentration data processed in the private cloud server. The teacher can realize the concentration of each student in class in real time, and timely remind the students when the students are absent-minded. Compared with the traditional teaching platform, the online teaching platform can help the teacher realize the concentration of the students in real time, and is beneficial to the improvement of the teaching quality.

Description

Technical Field

[0001] This invention relates to computer vision technology, and more particularly to an online teaching platform that can detect student attention levels. Background Technology

[0002] Currently, existing online teaching technologies simply share the teacher's computer screen and audio messages with each student, allowing the teacher to observe the students' learning progress through their webcams.

[0003] However, existing online teaching technologies can only share the teacher's screen and voice with each student, and share the content captured by each student's camera with the teacher. While this allows teachers to see if each student is in class, it doesn't allow them to assess the students' specific status, resulting in low-quality online education and failing to achieve high-efficiency teaching. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention discloses an online teaching platform capable of detecting student attention levels. This platform can not only acquire students' camera information but also their eye movement data during class. By analyzing this eye movement data, the platform can determine students' level of focus and promptly remind them when they lose focus, thereby improving the efficiency of online teaching.

[0005] To achieve the above objectives, the present invention discloses the following technical solution:

[0006] Firstly, this invention discloses an online teaching platform capable of detecting student attention span. The platform comprises three parts: a student-side platform, a teacher-side platform, and a private cloud server terminal. The student-side platform receives screen content from the teacher, displays the teacher's video and audio on the platform, and captures students' eye movement trajectories during class using a camera. This captured data is then sent to the platform's private cloud server terminal, where the server analyzes and processes the data. The teacher-side platform shares its screen content and audio with each student, ensuring that each student receives the teacher's lessons in real time. The teacher-side platform also receives the processed eye movement trajectory data for each student from the server. The private cloud server terminal of the teaching platform not only establishes communication between the student and teacher sides but also analyzes and processes the eye movement trajectories captured by the student side, sending the processed data to the teacher-side platform according to its requirements.

[0007] Secondly, the student-side platform of the teaching platform provided by this invention includes: receiving video and audio messages from the teacher's end transmitted from the teaching platform server terminal and collecting the eye movement trajectory of students during class. The main steps for collecting the eye movement trajectory include: first, determining whether the student's face is directly facing the screen on the student's end and whether the camera can capture the student's face; second, using a multi-task cascaded convolutional network for MTCNN face detection to select the eye region of the face; then, performing image preprocessing on the grayscale information of the eye region using a novel contrast stretching method, calculating the pupil center based on the discrete features of the pixel grayscale values ​​around the pupil center using a feature search algorithm, and accurately locating the pupil using a CV extension model; finally, considering that students cannot accurately align their heads with the position of the smart camera, establishing a pupil-screen conversion model to eliminate the offset caused by head deviation that would affect the position of the pupil mapped to the screen. The collected eye movement trajectory of students during class also needs to be uploaded to the server terminal for analysis and processing.

[0008] Thirdly, the teacher's end of the teaching platform provided by this invention includes: a platform for sharing screen content and audio information to each student's end, and a server terminal for receiving processed student eye movement trajectory data. Specifically, the processed student eye movement trajectory data received by the server terminal mainly includes: the teacher can select a specific area of ​​the screen for a specified time period on the teacher's end, and the server terminal sends the specific information about students' attention to that area during that time period to the teacher's end. The content sent by the server terminal includes: the average time students pay attention to that area, the longest and shortest attention times, how many students pay attention to that area for longer than the average time, and whether any students do not pay attention to that area at all.

[0009] Fourthly, the private cloud server terminal of the teaching platform provided by the present invention includes: a central hub for the teacher's end and the student's end, processing data from the teacher's end and the student's end, sending information data from the teacher's end to the student's end of each student, and processing the eye movement trajectory of the student's end and sending it to the teacher's end.

[0010] This application provides an online teaching platform capable of detecting student attention levels. The biggest advantage of this platform compared to traditional teaching platforms is that traditional platforms only allow information interaction through screen, audio, and video, failing to capture the student's true state during class, significantly impacting teaching quality. This application adds eye-tracking technology to the traditional online teaching platform, enabling real-time monitoring of student attention during class, thus greatly improving the quality of online teaching.

[0011] These or other aspects of this application will become clearer and more readily understood from the following description of embodiments. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art are briefly introduced below.

[0013] Figure 1 This is a schematic diagram of the structure of this application;

[0014] Figure 2 This is a schematic diagram of the eye-tracking algorithm. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0016] Traditional educational platforms can only exchange information through screen, audio, and video, failing to capture the students' true state during class. This greatly affects the quality of teaching, making online teaching far inferior to offline teaching.

[0017] To address the aforementioned technical issues, the inventors, through long-term research, discovered and proposed an online teaching platform capable of detecting student attention levels. By adding eye-tracking technology to a traditional online teaching platform, this platform allows for real-time monitoring of student attention during class, significantly improving the quality of online teaching. Teachers can use the data from students' eye movements to determine their current learning status and provide reminders accordingly.

[0018] Figure 1 A schematic diagram of the online teaching platform of this application is shown, as follows: Figure 1 As shown, the online teaching platform includes: a student-side platform, a teacher-side platform, and a private cloud server.

[0019] In this invention, the student-side platform is built around eye-tracking technology. The student-side platform receives video and audio information shared by the teacher and collects video streams of students during class through a smart camera. The eye-tracking algorithm is used to obtain the trajectory of the student's eye movements during class from the collected video streams.

[0020] The teacher-side platform uses the system by uploading their screen information and audio of their speech to a private cloud server, which then sends this information to each student-side platform. The teacher-side platform can also specify a time period and a designated area, and the private cloud server will calculate the specific details of student attention to that area during that time period, including: the average time students pay attention to that area, the longest and shortest time they pay attention to that area, how many students pay attention to that area for a longer than average time, and whether any students do not pay attention to that area at all.

[0021] The private cloud server is a powerful computing hub that facilitates information exchange between the teacher's learning platform and the student's user platform. It also receives eye movement data from the student's device, analyzes and processes this data, and sends the required eye movement information to the teacher's platform according to the teacher's needs.

[0022] Based on the aforementioned eye movement trajectory tracking technology for student terminals, this invention proposes an eye movement trajectory tracking algorithm. The following is a detailed description of the eye tracking algorithm provided by this invention, combined with… Figure 2 Please provide an explanation.

[0023] Figure 2 This is a schematic diagram of the eye trajectory tracking algorithm of the present invention. Figure 2 As shown, the algorithm includes:

[0024] S1. Capturing the student's facial area: The first step in eye movement tracking is capturing the student's facial area. In other words, the smart camera can only collect the student's facial information when the student is in a correct posture and facing the student-side teaching platform. Only after collecting complete facial information can the algorithm continue to track the student's eye movement trajectory.

[0025] S2, the eye region detection algorithm, adopts a joint detection and alignment framework of cascaded CNNs and designs a lightweight CNN architecture to achieve real-time performance. It is used to detect faces and locate five landmarks, and finally select the eye region.

[0026] First, a fully convolutional P-Net (proposal network) is used to obtain the facial window and its bounding box regression vector. Candidates are calibrated based on the estimated bounding box regression vectors, and then highly overlapping candidates are merged using non-maximum suppression (NMS). Next, all candidates are fed into an R-Net (refine network) CNN, which rejects a large number of erroneous candidates, performs bounding box regression for calibration, and again performs NMS. Finally, the facial region is identified, and five facial landmark regions are selected.

[0027] The CNN detector was trained using three tasks: face detection, bounding box regression, and face landmark localization.

[0028] The learning objective for face classification is defined as a binary classification problem, where for each sample x... i Cross-entropy is used as the loss:

[0029]

[0030] Where p i It is the probability of the network being generated, representing x. i The probability that it's the same face. This indicates a live feed label.

[0031] Bounding box regression: For each candidate bounding box window, we predict its offset from the nearest value, which is expressed as a regression problem, and the regression equation is:

[0032]

[0033] in, The regression target is obtained from the network. These are the actual coordinates.

[0034] Facial landmark localization: Similar to bounding box regression, facial landmark detection is also a regression problem. The goal is to minimize the Euclidean loss.

[0035]

[0036] in, These are facial landmark coordinates obtained from the internet. These are the ground truth coordinates of the i-th sample. There are five ground coordinates in total, including: left eye, right eye, nose, left corner of mouth, and right corner of mouth.

[0037] By performing the above operations, you can obtain the coordinates of important positions on the face, and then use these coordinates to select the eye area.

[0038] S3. Pupil localization: While eye region detection algorithms can accurately locate the eyes in a face, simply locating the eyes is insufficient; the pupil's location also needs to be determined. Precise pupil localization mainly includes: image preprocessing, initial pupil localization, and precise pupil localization.

[0039] Image preprocessing involves performing grayscale processing on the eye region selected by bounding box S2. Based on the grayscale information of the eye region, a novel contrast stretching function is used to preprocess the image. The stretching function is as follows:

[0040]

[0041] Where S represents the output gray level of the image, r represents the input gray level of the image, c is a positive number that can be adjusted as needed, and γ is a positive number, usually taken as 1.

[0042] The initial pupil localization is mainly to locate the pupil center and pupil region. The pupil center is calculated and located based on discrete features such as the standard deviation and mean of the gray values ​​around the pupil center.

[0043] The original image is non-linearly stretched to obtain the processed image f. Point A(x0, y0) is located in image f, where x0 represents the x-coordinate of point A and y0 represents the y-coordinate of point A. Set I i It is composed of pixel grayscale values ​​centered at A and with radius i:

[0044] I i ={I(x,y)|i-1<[(x-x0)} 2 +(y-y0) 2 ] 1 / 2 ≤i}

[0045] I = [I1, I2, LL, I n i = 1, 2, 3, ..., n

[0046] M=[mean(I1),mean(I2),L,mean(I n )]

[0047] Where I(x,y) represents the pixel grayscale value, and I represents Ix. i The set of i, i = 1, 2, 3, ..., n. M is the set of i. i The set of averages, mean(I i ) represents I i The average value of i = 1, 2, 3, ..., n.

[0048]

[0049]

[0050] The function J(x,y) is composed of the standard deviation of M and the mean of I, where σ M The standard deviation of M is μ. I It is the average value of I. The maximum value of J(x,y) is calculated at point (x...). * ,y * ), maximum value (x) * ,y * The center of the pupil is (x). After locating the pupil center, the pupil radius can be calculated using the function G. Ultimately, the initial pupil outline curve is determined to be a circle centered at (x). * ,y * A circle with radius m.

[0051] The above steps have allowed for the preliminary localization of the pupil's center and radius. However, this method only yields good results when there is no interference. For pupil region images with interference from eyebrows, eyelids, eyelashes, lighting, and spots, the results are unsatisfactory. A CV extended model with convexity constraints and a combination of standard and mean constraints can be used.

[0052] The CV (Contour Contour) model is a region-based active contour model that uses the grayscale information of an image as energy to construct an energy function, and then performs object segmentation by minimizing this energy function. Assume the image is segmented into two parts: background and target. The energy function is as follows:

[0053] E(C)=μgLength(C)+νgArea[inside(C)]

[0054] +λ1∫ inside(C) |I(x,y)-c0| 2 dxdy

[0055] +λ2∫ outside(C) |I(x,y)-c b | 2 dxdy

[0056] Where C represents a closed curve in the image, inside(C) is the region inside the curve, outside(C) is the region outside the curve, E(C) is the total energy of curve C, I(x,y) is the pixel gray value, c0 is the average gray value of inside(C), and c b λ1 is the average gray value outside(C), μ is the length coefficient, ν is the area coefficient, and λ1 and λ2 are usually 1.

[0057] CV models also produce many errors when processing images with high levels of interference; therefore, the constraint σ is used. C -μ CThe extended CV model is defined with Converxity(C) as a constraint. The energy function of the extended CV model is as follows:

[0058] E(C)=μgLength(C)+νgArea[inside(C)]+λ1∫ inside(C) |I(x,y)-c0| 2 dxdy+λ2∫ outside(C) |I(x,y)-c b | 2 dxdy+R(C)+Converxity(C)

[0059] Where, σ C It is the standard deviation of the grayscale values ​​of the pixels adjacent to curve C, μ C R(C) is the average grayscale value of the pixels adjacent to curve C, and Converxity(C) is the convexity value of curve C. R(C) enables the curve to be positioned near the edge of the pupil, and Converxity(C) keeps the curve as convex as possible.

[0060] The R(C) function is as follows:

[0061]

[0062] The first term of R(C) can suppress the excessive extension of the curve, and the second term can prevent the curve from excessive contraction.

[0063] Since the edge curve of the pupil is always convex, while the calculated curve is not convex due to certain disturbances, the extension term involving the graph is called Converxity(C).

[0064] The energy function of the extended CV model is minimized to drive the initial curve given by feature search to the pupil edge and accurately segment the pupil, thereby achieving precise pupil localization.

[0065] S4. Establish a pupil-screen conversion model and map the pupil coordinates to the screen. The above three steps can achieve accurate pupil positioning. However, since the human head will shift and rotate, it is impossible to ensure that every student is facing the student terminal when using the platform during class, resulting in a certain offset of the student's pupil being mapped to the screen.

[0066] This invention uses the Eye-ORB algorithm to match the positional relationship between several points of the head in previous and subsequent frames. Through weighted processing, a reference point that best reflects the degree of head offset is obtained. Based on this reference point, frame offset is calculated to determine the appropriate amount of head offset. An improved RBF algorithm is introduced to fit the transformation matrix from pupil coordinates to screen coordinates after removing the head offset, thus completing the gaze tracking of the human eye.

[0067] The Eye-ORB algorithm described above is an optimization based on the ORB algorithm. Specific improvements include: optimizing the traditional ORB algorithm by utilizing the scale invariance of the SURF algorithm and the image enhancement algorithm based on the Laplacian operator; and introducing an image enhancement algorithm based on the Laplacian operator to enhance the locally prepared image for subsequent feature point detection.

[0068] The ORB algorithm is a fast feature point extraction and description algorithm. The biggest feature of the ORB algorithm is its fast computation speed.

[0069] The improved RBF algorithm is a linear combination of a set of weighted radial basis functions, typically consisting of three layers: an input layer, hidden layers with activation functions, and an output layer. This algorithm represents a nonlinear mapping that transforms an input vector into an output scalar through a set of basis functions. When applied to regression problems, the unknown function to be fitted can be directly approximated using RBF. f(x) is shown in the formula:

[0070]

[0071] In the formula, x is the input vector, and w i It is the weight, where It is a basis function, c i is the center vector, and N is the number of center vectors.

[0072] The above four steps demonstrate the core idea of ​​the eye-tracking algorithm. Through these four steps, the eye movement trajectories of students during class can be collected. By uploading each student's eye movement trajectory to a private cloud server, the teacher can specify specific requirements, and the private cloud server can send the data to the teacher's end according to those requirements.

Claims

1. An online teaching platform capable of detecting student concentration, characterized in that, The application relates to a teaching platform and a method thereof. The teaching platform mainly comprises three parts: a student terminal platform, a teacher terminal platform and a private cloud server. The student terminal platform receives screen content shared by the teacher terminal platform and captures eye movement tracks of students during class, and the student terminal platform comprises the following steps: The student terminal platform receives screen content shared by the teacher terminal platform, and realizes real-time screen sharing by receiving video stream information of the screen of the teacher terminal platform and voice stream information of the teacher. The camera captures eye movement tracks of students during class, mainly including an eye region detection algorithm, a pupil positioning algorithm and a pupil-screen conversion model. The eye movement tracks of all students are transmitted to the private cloud server of the teaching platform, and the private cloud server analyzes and calculates the incoming data by using an attention tracking algorithm. The teacher terminal platform shares its own screen content on the devices of the student terminal platforms and receives processed student concentration data in the private cloud server, so that the teacher can realize real-time understanding of the concentration of each student during class. The student terminal platform uses the eye region detection algorithm, and the student terminal platform comprises the following steps: It is judged whether the face of the student is directly facing the screen of the student terminal, and whether the camera can capture the face of the student. If the face region of the student is captured, a joint detection and alignment framework of a cascade CNN is used, and a lightweight CNN architecture is designed to realize real-time performance, so as to detect a human face and five landmark positions, and finally frame the eye region. The student terminal platform uses the pupil positioning algorithm, and the student terminal platform comprises the following steps: First, the framed eye region is subjected to gray scale processing, and a novel contrast stretching function is used to pretreat the image according to the gray scale information of the eye region. Second, a feature search algorithm is used to preliminarily position the pupil center and the pupil region, the pupil center is calculated according to the discrete features of the pixel gray scale values around the pupil center, and the pupil region can also be determined through gradient information of the pupil. Finally, an improved CV scalable model with convexity constraint and standard deviation and mean combination constraint is added to the CV model, and the CV scalable model can effectively and accurately position the pupil. The student terminal platform uses the pupil-screen conversion model, and the student terminal platform comprises the following steps: The eye region detection algorithm and the pupil positioning algorithm can accurately position the position of the pupil, but the head of the student can be turned during class, and the head cannot be accurately aligned with the intelligent camera, so that the pupil mapping position on the screen also appears a certain offset. An Eye-ORB algorithm is used to match the position relationship of a plurality of points of the head in front and back frames, a reference point which can best reflect the head offset degree is obtained through weighted processing, frame offset calculation is carried out, and the head offset amount is obtained. An improved RBF algorithm is introduced to fit a transformation matrix of pupil coordinates to screen coordinates after removing the head offset amount, and the line of sight tracking of the human eye is completed.

2. The online teaching platform for enabling student concentration detection as claimed in claim 1 wherein, The student terminal platform receives the screen shared by the teacher terminal platform, and the student terminal platform comprises the following steps: The student terminal uses the platform to establish a communication connection with the private cloud server of the teaching platform, the teacher terminal uses the platform and the private cloud server of the teaching platform; The teacher terminal uses the platform to send the input of the computer screen content and audio content to the private cloud server of the teaching platform after processing; The private cloud server of the teaching platform sends the video and audio information of the teacher terminal to all student terminals, so that the student terminal receives the shared screen of the teacher terminal.

3. The online teaching platform of claim 1, wherein, The eye movement of the student is transmitted to the private cloud server of the teaching platform, and the incoming data is analyzed and calculated by the private cloud server, including: The collected eye movement trajectory of the student during the class is a set of irregular data, which is large and inconvenient to read. The collected eye movement trajectory of each student can be analyzed and calculated; The collected eye movement trajectory of each student is uploaded to the private cloud server of the teaching platform, and the private cloud server analyzes and calculates the eye movement trajectory of each student to determine the concentration of the student during the class; The teaching platform sends the data calculated by the private cloud server to the teacher terminal, so that the teacher terminal can easily observe the concentration of each student during the class.

4. The online teaching platform of claim 1, wherein, The teacher terminal mainly shares the audio to the student terminal and receives the teaching platform, including: The teacher terminal sends the computer screen and voice information to the private cloud server of the teaching platform, and the private cloud server shares the audio information to each student terminal; The teacher terminal receives the data calculated by the private cloud server, which is displayed in the form of a chart on the teacher terminal, so that the teacher can check the concentration of the student during the class.

5. The online teaching platform for enabling student concentration detection as claimed in claim 1 wherein, The teacher terminal displays the eye movement trajectory of the student during the class in the form of a chart, including: The data calculated by the private cloud server of the teaching platform is sent to the teacher terminal, which is inconvenient to view. The teacher terminal visualizes the received data and displays it in the form of a chart on the teacher terminal; The teacher can select a specified area of the screen in a specified time period on the teacher terminal, and the private cloud server sends the specific situation of the students focusing on the area in the time period to the teacher terminal; The content sent by the private cloud server includes: the average time of the students focusing on the area, the longest and shortest time of focusing on the area, the number of people focusing on the area for more than the average time, whether any student has never focused on the area, etc.

Images