Student fixation point estimation method based on double cameras in classroom scene
Through dual camera images acquisition and combining YuNet and SFace models, the problem of low accuracy of gaze point estimation in classroom scenes is solved, effective fusion of multi-view data and precise gaze point coordinate calculation, and the accuracy of gaze point estimation is improved.
Patent Information
- Application Number
- CN202411208525.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-22
AI Technical Summary
The existing gaze point estimation method in classroom scenarios is not very accurate in dense scenarios and lacks multi-view data fusion.
The dual camera is used to synchronize multi-view images, face detection and identity identification are performed through YuNet and SFace models, and head pose estimation model is trained in combination with a homemade data set to calculate the individual's gaze coordinates. Specific steps include dual camera calibration, image processing, construction and training of head pose estimation models, real-time image shooting to estimate head poses, calculating nose tip coordinates and rotation matrix, and ultimately transforming them into gaze coordinates under the world coordinate system.
It improves the accuracy and accuracy of students' gaze point estimation, and can effectively integrate multi-view data in classroom scenarios to provide more accurate individual gaze point coordinates.
Smart Images

Figure CN119007269A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a method for estimating a student's gaze point based on dual cameras in a classroom scene. Background Art
[0002] With the popularization of the Internet and the rapid development of artificial intelligence technology, artificial intelligence is deeply integrated with various industries and fields. In the context of the new era, the deep integration of artificial intelligence and education has become an urgent development need, aiming to promote the transformation and innovation of education in the intelligent era. The classroom is the main place for students to learn and contains a wealth of information about students' learning status. By using computer vision technology to automatically obtain students' learning status (such as attention status), teachers can fully grasp the classroom situation and adjust teaching strategies accordingly to promote students' academic progress. Today's classrooms are gradually developing towards smart education, and classrooms are usually equipped with two or more cameras. Therefore, it is particularly important to make full use of the image information obtained by multiple cameras and combine it with a deep learning network to analyze students' classroom situation from multiple perspectives.
[0003] There are currently some methods for estimating gaze points in classroom scenarios. For example, Chen et al. mentioned in the document "Visual analysis of learning attention based on single image PnP head pose estimation" that the gaze point is estimated by using head pose and similar triangle geometry derivation; Liu Honghai et al. proposed a method based on uncertainty in viewpoint estimation in the Chinese patent with publication number CN202311061679.0, which integrates multiple features and extracts the network to predict the position of the viewpoint; Xia Chen et al. proposed a method using convolutional neural network and probabilistic embedding technology in the Chinese patent with publication number CN202310216242.3, which processes input and label uncertainties. Qualitatively, a method for more accurate human eye gaze estimation is achieved; Li Tianjun et al. proposed a method based on monocular images, combined with head posture information and eye three-dimensional coordinate model, to achieve high-precision gaze point estimation and line of sight tracking in Chinese patent publication number CN202211007463.1; Zhong Fanghong et al. proposed a student viewpoint estimation method based on deep learning in Chinese patent publication number CN202110868818.5, which improves the accuracy of student viewpoint estimation in complex classroom environments through video frame acquisition, face detection, HopeNet head posture estimation and other steps.
[0004] Existing gaze point estimation methods generally have the problem of low accuracy in estimating students' gaze points in densely populated classroom scenes and lack of multi-view data fusion. Summary of the invention
[0005] The purpose of the present invention is to provide a student gaze point estimation method based on dual cameras in a classroom scenario, which uses dual cameras to synchronously capture multi-view images, performs face detection and identity authentication through YuNet and SFace models, trains a head posture estimation model in combination with a self-made data set, and calculates the individual gaze point coordinates.
[0006] To achieve the above object, the present invention provides a method for estimating a student's gaze point based on dual cameras in a classroom scene, comprising the following steps:
[0007] Step 1: Calibrate based on dual cameras and obtain the coordinate system pose relationship;
[0008] Step 2: Collect and process the images to obtain a self-made data set;
[0009] Step 3: Build a head posture estimation model based on ResNet50 and HopeNet methods and train it;
[0010] Step 4: Obtain the image pairs captured by the dual cameras in the classroom in real time and estimate the head posture;
[0011] Step 5: Calculate the rotation matrix of the individual's nose tip coordinates and head posture;
[0012] Step 6: Transform to obtain the individual's gaze point coordinates in the world coordinate system.
[0013] Optionally, during the execution of step 1, the dual cameras are calibrated using a chessboard, and the rotation matrix R of the two cameras in the camera coordinate system is obtained by combining the intrinsic parameters of the dual cameras and the universal stereo calibration method. c and the translation vector t c ; Manually mark the chessboard and obtain the rotation matrix R of the camera coordinate system relative to the auxiliary coordinate system through pose estimation cTos and the translation vector t cTos ; Manually measure the rotation matrix R of the auxiliary coordinate system relative to the world coordinate system sTow and the translation vector t sTow .
[0014] Optionally, during the execution of step 2, the YuNet model is used to perform face detection on the captured image and obtain the pixel coordinates of the individual's nose tip; then the SFace model is used to find individuals with matching identities in the image to form image pairs; the obtained image pairs are made into a self-made dataset D1.
[0015] Optionally, the same characteristics of individual postures in the image pairs of the self-made dataset D1 in the world coordinate system are used to perform consistent self-supervised learning on the model. The total loss function formula used in the head posture estimation model training is:
[0016]
[0017] Among them, L reg and L cls Corresponding to the supervised training loss, L r_con and L p_con Corresponding to the self-supervised loss, the superscript i is used to indicate the loss of the i-th sample, and D2 is the 300W_LP dataset;
[0018] L reg and L p_con is the mean square error loss function, L cls is the cross entropy loss function, L r_con is the geodesic distance loss, and the corresponding expression is as follows:
[0019]
[0020] L p_con =(x1-x2) 2 +(y1-y2) 2 +(z1-z2) 2
[0021] Among them, θ, φ, and θ', φ', Represents the actual and predicted yaw angle, pitch angle, and roll angle, y j is the one-hot encoding form of the true label, is the probability of being predicted as class j.
[0022] Optionally, the execution process of step 4 is to use YuNet to perform face detection on the captured image and obtain the pixel coordinates of the nose tip of the individual in the image; use the SFace model to obtain the individual with the corresponding identity in the image pair. Input the found individual into the trained head posture estimation model to obtain the Euler angle corresponding to the individual head posture; convert the obtained Euler angle into the rotation matrix R in the order of ZYX t .
[0023] Optionally, the rotation matrix of the individual's nose tip coordinates and head posture in step 5 are both data in the world coordinate system, which are obtained by transforming the data in the camera coordinate system.
[0024] The present invention provides a method for estimating the student gaze point based on dual cameras in a classroom scene. First, two cameras are deployed in the classroom scene to simultaneously capture image data of students in class. Secondly, the YuNet model is used to perform face detection on the captured images, and the pixel coordinates of the individual nose tip are obtained. Next, the SFace model is used to identify the individual images captured by the left and right cameras, pair them into image pairs, and make the obtained image pairs into a self-made data set. A head posture estimation model with ResNet50 as the backbone network is used to perform supervised training on the 300W_LP data set, and the characteristics of the same head posture of individuals in the image pairs of the self-made data set in the world coordinate system are used to perform self-supervised training on the model, thereby obtaining a head posture estimation model suitable for classroom scenes. Next, the trained model is used to detect the head posture of the individual, and the Euler angle of the individual head posture in the camera coordinate system is obtained, and it is converted into a rotation matrix. Using the pixel coordinates of the individual nose tip in the image pair, the three-dimensional coordinates of the individual nose tip in the camera coordinate system are obtained by using the triangulation method. In order to associate the camera coordinate system with the world coordinate system, an auxiliary coordinate system is introduced. According to the pose relationship between the camera coordinate system and the auxiliary coordinate system, and the pose relationship between the auxiliary coordinate system and the world coordinate system, the rotation matrix of the individual head posture and the three-dimensional coordinates of the nose tip in the world coordinate system are calculated. Finally, through the derivation of geometric relationships, the three-dimensional coordinates of the student's gaze point in the world coordinate system are estimated, that is, the gaze position on the blackboard. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1 It is a schematic diagram of the steps of a method for estimating a student's gaze point based on dual cameras in a classroom scenario of the present invention.
[0027] Figure 2 It is a schematic diagram of the position and posture of each coordinate system in a specific embodiment of the present invention.
[0028] Figure 3 It is a schematic diagram of the production process of a self-made data set according to a specific embodiment of the present invention.
[0029] Figure 4 It is a schematic diagram of the self-supervised learning process of the head posture estimation model according to a specific embodiment of the present invention.
[0030] Figure 5It is a schematic diagram of a head posture estimation model according to a specific embodiment of the present invention.
[0031] Figure 6 It is a schematic diagram of the student's gaze point according to a specific embodiment of the present invention. DETAILED DESCRIPTION
[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0033] See also Figure 1 The present invention provides a method for estimating a student's gaze point based on dual cameras in a classroom scene, comprising the following steps:
[0034] S1: Calibrate based on dual cameras and obtain the coordinate system pose relationship;
[0035] S2: collect images and process them to obtain a self-made data set;
[0036] S3: Build a head pose estimation model based on ResNet50 and HopeNet methods and perform training;
[0037] S4: Obtain the image pairs captured by the dual cameras in the classroom in real time and estimate the head posture;
[0038] S5: Calculate the individual's nose tip coordinates and the rotation matrix of the head posture;
[0039] S6: Transform to obtain the individual's gaze point coordinates in the world coordinate system.
[0040] The following is further described in conjunction with specific embodiments and execution steps:
[0041] See also Figures 2 to 6 ,Deploy dual cameras in the classroom to obtain image pairs of students in class. From the observer’s perspective, the camera on the left side of the blackboard is defined as the left camera, and the camera on the right side of the blackboard is defined as the right camera.
[0042] The process of step S1 is to calibrate the camera and obtain the coordinate system pose relationship: calibrate the dual cameras to obtain the intrinsic parameters of the dual cameras. Combined with the intrinsic parameters of the dual cameras, the universal stereo calibration method is used to obtain the relative pose information of the two cameras in the camera coordinate system: the rotation matrix R c and the translation vector t c The coordinates of the four points on the chessboard are manually marked, and the rotation matrix R of the camera coordinate system relative to the auxiliary coordinate system is obtained by estimating the position and posture of the four feature points. cTosand the translation vector t cTos By manual measurement, the rotation matrix R of the auxiliary coordinate system relative to the world coordinate system is obtained. sTow and the translation vector t sTow The pose relationship between each coordinate system, such as Figure 2 As shown, orange is the camera coordinate system, red is the world coordinate system, and blue is the auxiliary coordinate system. In the entire process of calibrating and obtaining the relationship between the coordinate systems, OpenCV tools are used to implement it.
[0043] Step S2 is the process of collecting data sets: Based on the dual cameras deployed in the classroom, image pairs of students in class are collected. First, the YuNet model is used to perform face detection on the captured images and obtain the pixel coordinates of the nose tip of each student. Second, the SFace model is used to find individuals with matching identities in the image pairs to form image pairs. The process is as follows: Figure 3 As shown in the figure, the red dashed box represents each image pair. The acquired image pairs are made into a self-made dataset D1.
[0044] Step S3 is the process of training the head posture estimation model: ResNet50 is selected as the backbone network and the classification and regression task processing methods in HopeNet are used to build a head posture estimation model. The 300W_LP dataset D2 is used as the training set to train the model. Using the self-made dataset D1 in step S2, the model is self-supervised by aligning the head posture rotation matrix and gaze point coordinates of the individual in the image pair in the world coordinate system, thereby obtaining a head posture model that is adaptive to the classroom scene. The process is as follows: Figure 4 and Figure 5 The total loss function used in training is shown in formula (1).
[0045]
[0046] Where L reg and L cls Corresponding to the supervised training loss, L r_con and L p_con Corresponding to the self-supervised loss, the superscript i is used to indicate the loss of the i-th sample. reg and L p_con is the mean square error loss function, L cls is the cross entropy loss function, L r_con is the geodesic distance loss. The specific expression of each loss is shown in formulas (2)-(5). For simplicity, i is omitted.
[0047]
[0048]
[0049] L p_con =(x1-x2) 2 +(y1-y2) 2 +(z1-z2) 2 (5)
[0050] In the above formula (2), θ, φ, and θ', φ', They represent the actual and predicted yaw angle, pitch angle, and roll angle respectively. In formula (3), y j is the one-hot encoding form of the true label, Is the probability of predicting the jth class. In this embodiment, the angle range of the head posture Euler angle (yaw angle, pitch angle, roll angle) is discretized into 66 classes from -99 degrees to 99 degrees, expressed as the probability distribution of the angle range. Then, the probability distribution of the model output is converted into a normalized probability distribution through the Softmax function. Finally, the cross entropy loss between the unique hot encoding vector of the target label and the normalized probability distribution is calculated. In formula (4), R1 and R2 represent the head posture rotation matrix of the image pair (individuals captured by the left camera and the right camera) in the world coordinate system. In formula (5), (x1, y1, z1) and (x2, y2, z2) represent the coordinates of the individual gaze point in the image pair in the world coordinate system.
[0051] Step S4 processes the image pair and estimates the head posture: obtain the image pair captured by the dual cameras in the classroom. Use YuNet to perform face detection on the captured images and obtain the coordinates of the nose tip of the individual in the image. Use the SFace model to obtain the individual with the corresponding identity in the image pair. Input the found individual into the trained model to obtain the Euler angle corresponding to the individual's head posture. Convert the obtained Euler angle into the rotation matrix R in the order of ZYX t .
[0052] Step S5: Calculate the rotation matrix of the individual's nose tip coordinates and head posture.
[0053] Specifically, the coordinates of the nose tip of the individual using the image obtained in step S4 and the R in step S1 c and t c , through the triangulation method, the three-dimensional coordinates P of the individual nose tip in the camera coordinate system are obtained t .P t Combined with R in step S1 cTos and t cTos, and R sTow and t sTow , through formula (6), we can get the three-dimensional coordinates P of the individual nose tip in the world coordinate system w , Pw is expressed as formula (7). R in step S4 t Combined with R in step S1 cTos and R sTow , through formula (8), we can get the rotation matrix R of the individual head posture in the world coordinate system w , R w It is expressed as formula (9).
[0054] P w =R sTow ·R cTos ·P t +R sTow ·t cTos +t sTow (6)
[0055]
[0056] R w =R sTow ·R cTos ·R t (8)
[0057]
[0058] Step S6: Calculate the coordinates of the individual's gaze point in the world coordinate system: w and R w , through formula (10), we can get the coordinates of the individual gaze point in the world coordinate system (x w ,y w ,z w ), the effect is as follows Figure 6 As shown, the two small pictures on the left are visualization pictures of the head posture (left camera and right camera) of the individual; the picture on the right is a visualization picture of the coordinates of the gaze point of the individual in the world coordinate system calculated by the method of the present invention, the small dots are the estimated gaze points, and the large circles represent the possible range of the individual's gaze. In this embodiment, the world coordinate system is in millimeters.
[0059]
[0060] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. A method for estimating student gaze points based on dual cameras in a classroom scene, characterized in that: The following steps are involved: Step 1: Calibrate based on dual cameras and obtain the coordinate system pose relationship; Step 2: Collect and process the images to obtain a self-made data set; Step 3: Build a head posture estimation model based on ResNet50 and HopeNet methods and train it; Step 4: Obtain the image pairs captured by the dual cameras in the classroom in real time and estimate the head posture; Step 5: Calculate the rotation matrix of the individual's nose tip coordinates and head posture; Step 6: Transform to obtain the individual's gaze point coordinates in the world coordinate system.
2. The method for estimating student gaze points based on dual cameras in a classroom scene as claimed in claim 1, characterized in that: During the execution of step 1, the dual cameras are calibrated using a chessboard. Combining the dual camera intrinsic parameters and the universal stereo calibration method, the rotation matrix R of the two cameras in the camera coordinate system is obtained. c and the translation vector t c ; Manually mark the chessboard and obtain the rotation matrix R of the camera coordinate system relative to the auxiliary coordinate system through pose estimation cTos and the translation vector t cTos ; Manually measure the rotation matrix R of the auxiliary coordinate system relative to the world coordinate system sTow and the translation vector t sTow .
3. The method for estimating student gaze points based on dual cameras in a classroom scene as described in claim 2, characterized in that: During the execution of step 2, the YuNet model is used to perform face detection on the captured image and obtain the pixel coordinates of the individual's nose tip; then the SFace model is used to find individuals with matching identities in the image to form image pairs; the obtained image pairs are made into a self-made dataset D1.
4. The method for estimating student gaze points based on dual cameras in a classroom scene as claimed in claim 3, characterized in that: Using the same characteristics of individual postures in the world coordinate system in the image pairs of the self-made dataset D1, the model is self-supervised for consistency learning. The total loss function formula used in the head posture estimation model training is: Among them, L reg and L cls Corresponding to the supervised training loss, L r_con and L p_con Corresponding to the self-supervised loss, the superscript i is used to indicate the loss of the i-th sample, and D2 is the 300W_LP dataset; L reg and L p_con is the mean square error loss function, L cls is the cross entropy loss function, L r_con is the geodesic distance loss, and the corresponding expression is as follows: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> p_con <h2 style=";text-align:left;direction:ltr"> (x1-x2)<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +(y1-y2)<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +(z1-z2)<h2 style=";text-align:left;direction:ltr"> 2 Among them, θ, φ, and θ', φ', Represents the actual and predicted yaw angle, pitch angle, and roll angle, y j is the one-hot encoding form of the true label, is the probability of being predicted as class j.
5. The method for estimating student gaze points based on dual cameras in a classroom scene as claimed in claim 4, characterized in that: The execution process of step 4 is to use YuNet to perform face detection on the captured image and obtain the pixel coordinates of the nose tip of the individual in the image; use the SFace model to obtain the individual with the corresponding identity in the image pair; input the found individual into the trained head posture estimation model to obtain the Euler angle corresponding to the individual head posture; convert the obtained Euler angle into the rotation matrix R in the order of ZYX t .
6. The method for estimating student gaze points based on dual cameras in a classroom scene as claimed in claim 5, characterized in that: In step 5, the rotation matrix of the individual's nose tip coordinates and head posture are both data in the world coordinate system, which are obtained by transforming the data in the camera coordinate system.
Citation Information
Cited By
Cross-view-angle sight line estimation method based on feature decoupling and attention mechanism
CN120088839A
A cross-view gaze estimation method based on feature decoupling and attention mechanism
CN120088839B
