Online learning attention tracking method based on gaze estimation and its application

Abstract

The present application discloses an online learning attention tracking method based on line of sight estimation and its application, the method comprising: obtaining a facial image, an eye image and a binocular infrared image of an object to be detected, and a scene image of a learning environment in which the object to be detected is located; the scene image contains an image of a learning device that interacts with the object to be detected; the facial image, the eye image and the binocular infrared image are input into a line of sight estimation recognition model to obtain a three-dimensional gaze direction of both eyes of the object to be detected in a camera coordinate system; the three-dimensional gaze direction in the camera coordinate system is converted into a two-dimensional gaze point in a screen coordinate system in which the learning device is located; according to the positional relationship between the two-dimensional gaze point and the learning area in the scene image, a current attention detection result of the object to be detected is generated; the present invention realizes the complementarity of head posture and two eye features, improves the accuracy of line of sight estimation under complex backgrounds, and provides objective support data for improving students' online learning attention.

Description

Technical Field

[0001] The present application relates to the field of intelligent human-computer interaction technology, and more specifically, to an online learning attention tracking method based on line of sight estimation, a computer device, and a readable medium. Background Art

[0002] With the rapid development of educational informatization, artificial intelligence technology has been increasingly used in the process of education and teaching. In recent years, the outbreak of the new coronavirus pneumonia has broken the traditional offline teaching model, and the online distance teaching method has been widely adopted for a time. However, in the absence of close supervision by teachers, how to achieve remote supervision or self-supervision to ensure the learning state and learning efficiency of learners' independent learning has become an urgent problem to be solved. Therefore, adopting an online learning attention tracking method to assist learners in online learning is of great significance to improving the learning efficiency of learners' online learning.

[0003] The detection and tracking of learning attention can be started from the perspective of estimating the learner's gaze direction during the learning process; in a broad sense, gaze estimation refers to the research related to eyeballs, eye movements, and gaze. Generally speaking, gaze estimation methods can be divided into two categories: model-based methods and appearance-based methods. The basic idea of ​​the model-based method is to estimate the gaze direction based on features such as corneal reflection of the eye, combined with 3D eyeball prior knowledge. The appearance-based method directly extracts the visual features of the eye, trains a regression model, and learns a model that maps the appearance to the gaze direction, thereby estimating the gaze. Through comparative analysis, the model-based method obtains higher accuracy, but the image quality and resolution requirements are also high. In order to achieve this goal, it is usually required to use specialized hardware, and the mobility of the user's head posture is greatly restricted; while the appearance-based method performs better for low-resolution and high-noise images, but the training of its model requires a large amount of data and is prone to overfitting. With the development of deep learning and the disclosure of a large number of data sets, the appearance-based method has received more and more attention.

[0004] Zhang first tried to use neural networks for gaze estimation in his paper "Appearance-based gaze estimation in the wild". It used monocular images as input, used LeNet's shallow network for learning, and spliced ​​the head posture information with the extracted eye features for gaze estimation, and finally obtained an error of 6.3° on the MPIIGAZE dataset. Later, Zhang proposed a full-face gaze estimation method based on the attention mechanism in his paper "It's Written All Over Your Face: Full-FaceAppearance-Based Gaze Estimation". This method uses the attention mechanism to increase the weight of the eye area and suppress the weight of other areas not related to gaze. It uses an end-to-end learning strategy to directly obtain the gaze direction in the camera coordinate system. Park et al. proposed a gaze estimation method based on eye graphic representation in their paper "Deep pictorialgaze estimation". They abstracted the eyes into an eyeball graphic representation through a deep network to improve the accuracy of gaze estimation. The eyeball graphic representation, a supervisory signal, is generated by the true value of the gaze through geometric inversion. In the same year, Yu et al. proposed a gaze estimation method based on a constraint model based on the idea of ​​multi-task learning in their paper “Deep multitask gaze estimation with a constrained landmark-gaze model”. This method detects the position of the key points of the eyes while estimating the gaze. The two tasks are learned simultaneously, and the information is complementary, which can be improved together to a certain extent. Experiments have shown that the model has achieved more accurate results than the direct regression method.

[0005] At present, although great progress has been made in the research on gaze estimation technology, the accuracy of gaze estimation obtained based on general models and single eye features is limited. In addition, the movement amplitude of the user's head will have a great impact on the measurement results, reducing the recognition accuracy, and thus limiting the accuracy of online learning attention tracking detection. Summary of the invention

[0006] In response to at least one defect or improvement need in the prior art, the present invention provides an online learning attention tracking method based on line of sight estimation and its application, which can achieve information complementarity of different modalities, with the aim of improving the accuracy of line of sight estimation in complex backgrounds, thereby improving the reliability of online learning attention judgment.

[0007] To achieve the above object, according to a first aspect of the present invention, an online learning attention tracking method based on line of sight estimation is provided, which comprises:

[0008] Acquire a facial image, an eye image, and a binocular infrared image of the object to be detected, as well as a scene image of the learning environment in which the object to be detected is located; the scene image is acquired by a scene camera set on the head of the object to be detected, and the scene image includes an image of a learning device that interacts with the object to be detected;

[0009] Inputting the facial image, eye image and binocular infrared image into a trained sight line estimation recognition model to obtain the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system;

[0010] Convert the three-dimensional gaze direction in the camera coordinate system to a two-dimensional gaze point in the screen coordinate system of the learning device;

[0011] According to the positional relationship between the two-dimensional gaze point and the pre-divided learning area in the scene image, a current attention detection result of the object to be detected is generated.

[0012] Preferably, in the above-mentioned online learning attention tracking method, the step of inputting the facial image, the eye image and the binocular infrared image into a trained sight line estimation and recognition model to obtain the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system specifically includes:

[0013] Extract features from facial images and generate corresponding head posture feature vectors;

[0014] Extract features from the eye image and generate corresponding eye feature vectors;

[0015] Extract the features of the eyes in the binocular infrared image and generate the corresponding binocular infrared feature vectors;

[0016] The head posture feature vector, the eye feature vector and the binocular infrared feature vector are spliced ​​to obtain a fused feature vector, and the three-dimensional gaze direction of the binoculars of the object to be detected in the camera coordinate system is generated according to the fused feature vector.

[0017] Preferably, in the above-mentioned online learning attention tracking method, converting the three-dimensional gaze direction in the camera coordinate system into a two-dimensional gaze point in the screen coordinate system where the learning device is located includes:

[0018] The three-dimensional target gaze point of both eyes of the object to be detected in the camera coordinate system is calculated according to the three-dimensional gaze direction and the three-dimensional gaze origin; the three-dimensional gaze origin is the face center or the eyeball center of the object to be detected; the calculation formula is as follows:

[0019]

[0020] Where g represents the three-dimensional gaze direction in the camera coordinate system, g = (g x ,gy ,g z ), g x , g y , g z They represent the coordinates of the eyeball center in three-dimensional space; o represents the three-dimensional gaze origin; t represents the three-dimensional target gaze point in the camera coordinate system;

[0021] Obtain the space conversion matrix between the camera coordinate system and the screen coordinate system, and generate the two-dimensional gaze point of both eyes of the object to be detected in the screen coordinate system according to the space conversion matrix and the three-dimensional target gaze point; the calculation formula is as follows:

[0022] t=Rs[u,v,0] T +Ts

[0023] Where {Rs, Ts} represents the spatial transformation matrix, Rs is the rotation matrix, and Ts is the translation matrix; (u, v) represents the two-dimensional gaze point in the screen coordinate system; [·] T Represents the transpose operation of a matrix.

[0024] Preferably, in the above-mentioned online learning attention tracking method, the training process of the gaze estimation and recognition model includes:

[0025] Acquire a training sample, wherein the training sample has a true three-dimensional gaze direction label;

[0026] Inputting the training samples and their corresponding three-dimensional gaze direction labels into a sight line estimation and recognition model, and outputting the predicted sight line direction corresponding to the training samples through the sight line estimation and recognition model to be trained;

[0027] A loss function is calculated according to the predicted line of sight direction and the three-dimensional gaze direction label, and the model parameters of the line of sight estimation and recognition model to be trained are adjusted in reverse until the loss function is minimized to obtain a trained line of sight estimation and recognition model.

[0028] Preferably, in the above online learning attention tracking method, the loss function is:

[0029]

[0030] Where L represents the loss function; g gt (I) represents the three-dimensional gaze direction label; g p (I) represents the predicted sight direction; I represents the training sample; D represents the training data set, and |·| is the cardinality operator.

[0031] Preferably, the above-mentioned online learning attention tracking method further includes:

[0032] During the training process of the line of sight estimation and recognition model, the cosine annealing restart learning rate mechanism is used to dynamically adjust the learning rate of the model.

[0033] According to a second aspect of the present invention, there is also provided an online learning attention tracking system based on line of sight estimation, which comprises:

[0034] An image acquisition unit, used to acquire a facial image, an eye image, and a binocular infrared image of the object to be detected, as well as a scene image of the learning environment in which the object to be detected is located; wherein the scene image is acquired by a scene camera arranged on the head of the object to be detected, and the scene image includes an image of a learning device interacting with the object to be detected;

[0035] A detection unit is used to obtain a facial image, an eye image and a binocular infrared image of the object to be detected and input them into a trained sight line estimation and recognition model to obtain a three-dimensional gaze direction of the eyes of the object to be detected in a camera coordinate system;

[0036] A conversion unit, used for converting the three-dimensional gaze direction in the camera coordinate system into a two-dimensional gaze point in the screen coordinate system where the learning device is located;

[0037] The result output unit is used to generate the current attention detection result of the object to be detected according to the positional relationship between the two-dimensional gaze point and the pre-divided learning area in the scene image.

[0038] Preferably, in the above-mentioned online learning attention tracking system, the image acquisition unit further comprises an infrared micro camera and a foreground camera;

[0039] The infrared micro camera is used to collect binocular infrared images of the object to be detected during the learning process, including a left-eye infrared camera and a right-eye infrared camera;

[0040] The left-eye infrared camera, the right-eye infrared camera and the scene camera are integrated on the head-mounted device, and the left-eye infrared camera and the right-eye infrared camera are distributed on both sides of the scene camera;

[0041] The foreground camera is used to collect facial images of the object to be detected during the learning process.

[0042] According to the third aspect of the present invention, there is also provided a computer device, which includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs any step of the above-mentioned online learning attention tracking method.

[0043] According to a fourth aspect of the present invention, there is also provided a computer-readable medium storing a computer program executable by a computer device, wherein when the computer program runs on the computer device, the computer device executes any of the steps of the online learning attention tracking method described above.

[0044] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0045] (1) The online learning attention tracking method based on line of sight estimation provided by the present invention collects the facial image, eye image and binocular infrared image of the object to be detected and performs feature extraction to obtain the head posture feature, eye feature and binocular infrared feature vector, fuses the features of the three different modes and calculates the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system based on the fused multi-modal features. After the information of the three modes is connected and fused, they assist each other and greatly improve the accuracy of line of sight estimation. The line of sight estimation result is applied to the detection of online learning attention after the coordinate system is converted, which is of great significance to improving the learning quality and assisting teaching.

[0046] (2) In the present invention, the left-eye infrared camera, the right-eye infrared camera and the scene camera are integrated on the head-mounted device, and the left-eye infrared camera and the right-eye infrared camera are distributed on both sides of the scene camera; based on the head-mounted device, the eye image and binocular infrared image of the object to be detected and the scene image are collected, and more eye details can be obtained without being affected by lighting changes, relative positions of humans and machines, interference and occlusion, etc. during the eye image acquisition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of an online learning attention tracking method based on line of sight estimation provided in this embodiment;

[0049] Figure 2 A schematic diagram of the network structure of the sight line estimation and recognition model provided in this embodiment;

[0050] Figure 3 A schematic diagram of the coordinate system conversion process of the sight line direction provided in this embodiment;

[0051] Figure 4 A schematic diagram of the composition of the image acquisition unit provided in this embodiment;

[0052] Figure 5 A schematic diagram of the composition of the computer device provided in this embodiment. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0054] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0055] Figure 1 is a flowchart of an online learning attention tracking method based on line of sight estimation provided in this embodiment, see Figure 1 As shown, the method mainly includes the following steps:

[0056] S1 acquires a facial image, an eye image, and a binocular infrared image of the object to be detected, as well as a scene image of a learning environment in which the object to be detected is located; the scene image is acquired by a scene camera disposed on the head of the object to be detected, and the scene image includes an image of a learning device that interacts with the object to be detected;

[0057] In this embodiment, in the online learning scenario, firstly, the RGB facial image, RGB eye image, and left and right eye infrared images of the object to be detected during the learning process are collected; wherein, the RGB eye image can be obtained from the RGB facial image, and of course, the eye region of the object to be detected can also be collected separately. Further, the RGB facial image, RGB eye image, and left and right eye infrared images are preprocessed by image scaling, for example, the RGB facial image is converted into a grayscale image of 224×224 size, and the RGB eye image and left and right eye infrared images are all converted into grayscale images of 36×60 size.

[0058] The scene camera is mainly used to collect learning scenes of the object to be detected. In this embodiment, the scene camera is set on the head of the object to be detected to ensure that the collected scene image contains the image of the learning device that interacts with the object to be detected.

[0059] S2 inputs the facial image, eye image and binocular infrared image into the trained sight line estimation recognition model to obtain the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system;

[0060] Figure 2 is a schematic diagram of the network structure of the sight line estimation and recognition model provided in this embodiment. Figure 2 As shown, the line of sight estimation and recognition model includes a first network branch, a second network branch, a third network branch and a multi-layer perception feature fusion network;

[0061] The first network branch is a head posture information extraction network, which is used to extract features from RGB facial images and generate corresponding head posture feature vectors;

[0062] The RGB facial image is used to extract three feature points, namely the nose and the eyes. The three feature points form a triangular area, and the image features extracted from the area are used to form a head feature vector representing the head posture.

[0063] The second network branch is a left and right eye feature extraction network, which is used to extract features from the RGB eye image and generate corresponding eye feature vectors;

[0064] The third network branch is a left-right eye infrared feature extraction network, which is used to extract features of the eyes in binocular infrared images and generate corresponding binocular infrared feature vectors;

[0065] Eye images are divided into infrared and visible light images. The extracted eye feature vector focuses on the features of the pupil and the corner of the eye. After extraction by the neural network, a feature vector that fuses the pupil and the corner of the eye is formed.

[0066] The multi-layer perception feature fusion network is used to splice the information of the three modalities of the head posture feature vector, eye feature vector and binocular infrared feature vector to obtain a fused feature vector, and generate the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system according to the fused feature vector.

[0067] Before concatenating the head posture feature vector, eye feature vector, and binocular infrared feature vector, normalize the feature vectors to make their feature values ​​in a uniform limited range, that is, between 0 and 1. Then, based on the Euclidean distance, continuously optimize the distance between the network predicted value and the true value to minimize the distance. In this way, the predicted value of the network can predict the gaze direction to a certain extent, and the three-dimensional gaze direction is transformed into a two-dimensional coordinate vector on the screen.

[0068] In a specific example, the first network branch, the second network branch, and the third network branch all include a 3D graph convolutional neural network and a fully connected layer; the 3D graph convolutional neural network mainly includes a convolutional layer and a pooling layer, wherein the convolutional layer mainly extracts image features through convolution operations, and each convolutional layer is composed of multiple two-dimensional planes, which are convolved with the previous layer through different convolution kernels to form a feature map, thereby realizing the layer-by-layer propagation of input features. The pooling layer is also called a downsampling layer, which is usually set after the convolutional layer and is used to downsample the local area of ​​the feature response map. Cluster analysis is performed by taking the average or maximum value of the features at different positions to achieve the purpose of reducing the dimension of the input image, thereby achieving the reduction of the amount of calculation while avoiding overfitting. The fully connected layer connects the feature maps extracted by the convolutional layer and the pooling layer, so that each input and output neuron are connected, thereby maximally representing the global features, and then the extracted feature representation is targetedly mapped to the sample label space for regression tasks.

[0069] See also Figure 2 , the RGB facial image, RGB eye image, and binocular infrared image are input into the sight estimation and recognition model, and enter the head posture information extraction network, the left and right eye feature extraction network, and the left and right eye infrared feature extraction network respectively. The left and right eye infrared feature extraction network is used to extract the features of the binocular infrared image. As a specific example, the left and right eye infrared feature extraction network has a 50-layer structure, and the input is a 3×36×60 eye image I. After 50 layers of convolution blocks, the output is a multidimensional array of size 2048×7×7, and then after a fully connected layer, the output is a 1×64 binocular infrared feature vector. Similarly, the head posture feature information extraction network and the left and right eye RGB image feature extraction network adopt the same architecture, and output a 1×128 head posture feature vector and a 1×64 eye feature vector respectively. The head posture feature vector, eye feature vector, and binocular infrared feature vector are spliced ​​through the concat function in the Pytorch framework, and the output is a 1×256 fusion feature vector. Then, the fusion feature vector enters two consecutive fully connected layers, and a three-dimensional sight direction is obtained by training a linear regression.

[0070] In this embodiment, the training process of the sight line estimation recognition model includes:

[0071] Obtain a training set, which includes multiple training samples, and each training sample has a true three-dimensional gaze direction label;

[0072] Input the training samples and their corresponding three-dimensional gaze direction labels into the gaze estimation and recognition model, and output the predicted gaze direction corresponding to the training samples through the gaze estimation and recognition model to be trained;

[0073] A loss function is calculated according to the predicted line of sight direction and the three-dimensional gaze direction label, and the model parameters of the line of sight estimation and recognition model to be trained are reversely adjusted until the loss function is minimized to obtain a trained line of sight estimation and recognition model.

[0074] As a specific example, the loss function is expressed as:

[0075]

[0076] Where L represents the loss function; g gt (I) represents the three-dimensional gaze direction label; g p (I) represents the predicted sight direction; I represents the training sample; D represents the training data set, and |·| is the cardinality operator.

[0077] As a preferred example, during the training of the line of sight estimation recognition model, the cosine annealing restart learning rate mechanism is used to dynamically adjust the learning rate of the model. Specifically, the SGD optimizer is used for gradient optimization, the initial learning rate is set to 0.01, the batch is set to 128, and the cosine annealing restart learning rate mechanism is used to make the network training more stable. The specific steps are as follows: After setting cawb_steps = 50, the initial learning rate will be multiplied by a proportional factor step_scale each time it is restarted. By adjusting parameters such as step_scale and epoch_scale, you can realize whether the learning rate jumps up or down. You can also adjust the intermediate iteration steps without going through an annealing process, maintain a high learning rate, and implement a more complex learning rate change mechanism.

[0078] Furthermore, after the line of sight estimation recognition model is trained using the training set, the line of sight estimation model is fine-tuned and learned using the verification sample set, and then the line of sight estimation model is tested for effectiveness using the test sample set. In a specific example, the learning rate in the above-mentioned fine-tuning and effectiveness testing process can be set to e-6.

[0079] S3 converts the three-dimensional gaze direction in the camera coordinate system into a two-dimensional gaze point in the screen coordinate system where the learning device is located;

[0080] In this embodiment, after obtaining the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system output by the model, it is necessary to project the three-dimensional gaze direction in the camera coordinate system to the screen coordinate system to obtain the two-dimensional gaze point of the three-dimensional gaze direction on the learning device screen; specifically:

[0081] See also Figure 3First, the three-dimensional target gaze point of both eyes of the object to be detected in the camera coordinate system is calculated according to the three-dimensional gaze direction and the three-dimensional gaze origin; the three-dimensional gaze origin can be the face center or the eyeball center of the object to be detected; the calculation formula is as follows:

[0082]

[0083] Where g represents the three-dimensional gaze direction in the camera coordinate system, g = (g x ,g y ,g z ), g x , g y , g z They represent the coordinates of the eye center in three-dimensional space; t represents the three-dimensional target gaze point in the camera coordinate system; o represents the three-dimensional gaze origin, which can be estimated by key point detection algorithm or stereo measurement method.

[0084] Then, the space conversion matrix between the camera coordinate system and the screen coordinate system is obtained, and the two-dimensional gaze points of the eyes of the object to be detected in the screen coordinate system are generated according to the space conversion matrix and the three-dimensional target gaze point; the calculation formula is as follows:

[0085] t=Rs[u,v,0] T +Ts

[0086] Where {Rs, Ts} represents the spatial transformation matrix, Rs is the rotation matrix, and Ts is the translation matrix; (u, v) represents the two-dimensional gaze point in the screen coordinate system; [·] T Represents the transpose operation of a matrix.

[0087] S4 generates a current attention detection result of the object to be detected according to the positional relationship between the two-dimensional gaze point and the pre-divided learning area in the scene image.

[0088] In this embodiment, it is determined whether the two-dimensional gaze points of the eyes of the object to be detected fall within a learning area pre-divided in the scene image. Generally speaking, the learning area is the screen area of ​​the learning device. If so, it indicates that the object to be detected is currently paying attention. Otherwise, it indicates that the object to be detected is currently not paying attention. Corresponding intervention measures can be provided, such as issuing voice warnings, to further focus the learner's concentration and improve the learner's autonomous ability in online learning.

[0089] Furthermore, based on the above steps, the focus areas of the eyes of the object to be detected at different times (i.e., two-dimensional gaze points) are obtained, and an attention data log is generated. Using data visualization analysis technology, the attention changes of online learners are analyzed and displayed in real time.

[0090] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above-mentioned embodiments, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0091] This embodiment also provides an online learning attention tracking system based on line of sight estimation, the system comprising an image acquisition unit, a detection unit, a conversion unit and a result output unit;

[0092] The image acquisition unit is mainly used to acquire the facial image, eye image and binocular infrared image of the object to be detected, as well as the scene image of the learning environment where the object to be detected is located;

[0093] Figure 4 is a schematic diagram of the composition of the image acquisition unit provided in this embodiment, see Figure 4 ,In this embodiment, the image acquisition unit includes an infrared micro camera, a scene camera and a foreground camera;

[0094] Among them, the infrared micro camera is used to collect binocular infrared images of the object to be detected during the learning process, including a left-eye infrared camera and a right-eye infrared camera;

[0095] The scene camera is set on the head of the object to be detected, and is used to collect the learning environment of the object to be detected. The scene image includes the image of the learning device that interacts with the object to be detected;

[0096] As a preferred example, the left-eye infrared camera, the right-eye infrared camera and the scene camera are integrated on a head-mounted device, and the left-eye infrared camera and the right-eye infrared camera are distributed on both sides of the scene camera; further preferably, the scene camera is arranged on the center line of the left-eye infrared camera and the right-eye infrared camera, and the positional relationship between the two-dimensional gaze point of both eyes of the object to be detected and the scene image captured by the scene camera at this position is calculated, which has better accuracy and reliability.

[0097] The foreground camera is used to collect facial images of the object to be detected during the learning process. The foreground camera can be a camera built into the learning device such as a tablet, computer, mobile phone, etc., or an independent camera device can be used and mounted on the learning device, or placed in other areas where the facial image of the object to be detected can be collected.

[0098] The detection unit is used to obtain the facial image, eye image and binocular infrared image of the object to be detected and input them into the trained sight estimation recognition model to obtain the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system;

[0099] The conversion unit is used to convert the three-dimensional gaze direction in the camera coordinate system into a two-dimensional gaze point in the screen coordinate system where the learning device is located;

[0100] The result output unit is used to generate the current attention detection result of the object to be detected according to the positional relationship between the two-dimensional gaze point and the pre-divided learning area in the scene image.

[0101] The online learning attention tracking system provided in this embodiment, the network structure of the gaze estimation and recognition model and its training process, as well as the specific functional implementation of the conversion unit and the result output unit can be found in the online learning attention tracking method section and will not be repeated here.

[0102] In the online learning attention tracking system, in addition to the image acquisition unit, the detection unit, the conversion unit and the result output unit can be implemented in software and / or hardware and can be integrated on a computer device.

[0103] This embodiment also provides a computer device, see Figure 5 , which includes at least one processor and at least one memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the above-mentioned online learning attention tracking method. The specific steps are shown in the previous embodiment and will not be repeated here. In this embodiment, the types of processors and memories are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be a volatile memory, a non-volatile memory, or a combination thereof.

[0104] In addition, the computer device may also communicate with one or more external devices (such as a keyboard, a pointing terminal, a display, etc.), may also communicate with one or more terminals that enable a user to interact with the computer device, and / or communicate with any terminal that enables the computer device to communicate with one or more other computing terminals (such as a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the computer device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter.

[0105] It should be noted that the computer device can be an external monitoring device independent of the learning device, or it can refer to the learning device. In this case, a computer program that can execute the above-mentioned online learning attention tracking method needs to be embedded in the learning device. The computer device preferably has a voice broadcast function, which can provide voice prompts when it detects that the learner is not concentrating.

[0106] This embodiment also provides a computer-readable medium, which stores a computer program executable by a computer device, and when the computer program is run on the computer device, the computer device executes the steps of the above-mentioned online learning attention tracking method. The types of computer-readable media include but are not limited to storage media such as SD cards, USB flash drives, fixed hard disks, and mobile hard disks.

[0107] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An online learning attention tracking method based on gaze estimation, characterized in that: include: Acquire a facial image, an eye image, and a binocular infrared image of the object to be detected, as well as a scene image of a learning environment in which the object to be detected is located; The scene image is acquired by a scene camera arranged on the head of the object to be detected, and the scene image includes an image of a learning device interacting with the object to be detected; The facial image, the eye image and the binocular infrared image are input into a trained line of sight estimation and recognition model to obtain the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system, specifically including: extracting features from the facial image to generate a corresponding head posture feature vector; extracting features from the eye image to generate a corresponding eye feature vector; extracting features from the eyes of the binocular infrared image to generate a corresponding binocular infrared feature vector; splicing the head posture feature vector, the eye feature vector and the binocular infrared feature vector to obtain a fused feature vector, and generating the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system according to the fused feature vector; The three-dimensional gaze direction in the camera coordinate system is converted into a two-dimensional gaze point in the screen coordinate system where the learning device is located, including: calculating the three-dimensional target gaze point of both eyes of the object to be detected in the camera coordinate system according to the three-dimensional gaze direction and the three-dimensional gaze origin; the three-dimensional gaze origin is the center of the eyeball of the object to be detected; the calculation formula is as follows: ； in, represents the 3D gaze direction in the camera coordinate system, , , , Respectively represent the coordinates of the eyeball center in three-dimensional space; represents the three-dimensional gaze origin; Represents the three-dimensional target gaze point in the camera coordinate system; Obtain the space conversion matrix between the camera coordinate system and the screen coordinate system, and generate the two-dimensional gaze point of both eyes of the object to be detected in the screen coordinate system according to the space conversion matrix and the three-dimensional target gaze point; the calculation formula is as follows: in, represents the space transformation matrix, is the rotation matrix, is the translation matrix; Represents the two-dimensional gaze point in the screen coordinate system; Represents the transpose operation of a matrix; According to the positional relationship between the two-dimensional gaze point and the pre-divided learning area in the scene image, a current attention detection result of the object to be detected is generated.

2. The online learning attention tracking method according to claim 1, characterized in that: The training process of the sight line estimation recognition model includes: Acquire a training sample, wherein the training sample has a true three-dimensional gaze direction label; Inputting the training samples and their corresponding three-dimensional gaze direction labels into a sight line estimation and recognition model, and outputting the predicted sight line direction corresponding to the training samples through the sight line estimation and recognition model to be trained; A loss function is calculated according to the predicted line of sight direction and the three-dimensional gaze direction label, and the model parameters of the line of sight estimation and recognition model to be trained are adjusted in reverse until the loss function is minimized to obtain a trained line of sight estimation and recognition model.

3. The online learning attention tracking method as claimed in claim 2, characterized in that: The loss function is: in, represents the loss function; Represents the three-dimensional gaze direction label; Indicates the predicted sight direction; represents training samples; represents the training data set, is the cardinality operator.

4. The online learning attention tracking method as claimed in claim 2, characterized in that: Also includes: During the training process of the line of sight estimation and recognition model, the cosine annealing restart learning rate mechanism is used to dynamically adjust the learning rate of the model.

5. An online learning attention tracking system based on gaze estimation, characterized in that: include: An image acquisition unit, used to acquire a facial image, an eye image, and a binocular infrared image of the object to be detected, as well as a scene image of the learning environment in which the object to be detected is located; wherein the scene image is acquired by a scene camera arranged on the head of the object to be detected, and the scene image includes an image of a learning device interacting with the object to be detected; The detection unit is used to obtain the facial image, eye image and binocular infrared image of the object to be detected and input them into the trained sight estimation recognition model to obtain the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system, specifically including: extracting features from the facial image to generate a corresponding head posture feature vector; extracting features from the eye image to generate a corresponding eye feature vector; extracting features from the eyes of the binocular infrared image to generate a corresponding binocular infrared feature vector; splicing the head posture feature vector, the eye feature vector and the binocular infrared feature vector to obtain a fused feature vector, and generating the three-dimensional gaze direction of the eyes of the object to be detected in the camera coordinate system according to the fused feature vector; A conversion unit, used to convert the three-dimensional gaze direction in the camera coordinate system into a two-dimensional gaze point in the screen coordinate system where the learning device is located, including: calculating the three-dimensional target gaze point of both eyes of the object to be detected in the camera coordinate system according to the three-dimensional gaze direction and the three-dimensional gaze origin; the three-dimensional gaze origin is the eyeball center of the object to be detected; the calculation formula is as follows: ； in, represents the 3D gaze direction in the camera coordinate system, , , , Respectively represent the coordinates of the eyeball center in three-dimensional space; represents the three-dimensional gaze origin; Represents the three-dimensional target gaze point in the camera coordinate system; Obtain the space conversion matrix between the camera coordinate system and the screen coordinate system, and generate the two-dimensional gaze point of both eyes of the object to be detected in the screen coordinate system according to the space conversion matrix and the three-dimensional target gaze point; the calculation formula is as follows: in, represents the space transformation matrix, is the rotation matrix, is the translation matrix; Represents the two-dimensional gaze point in the screen coordinate system; Represents the transpose operation of a matrix; The result output unit is used to generate the current attention detection result of the object to be detected according to the positional relationship between the two-dimensional gaze point and the pre-divided learning area in the scene image.

6. The online learning attention tracking system as claimed in claim 5, characterized in that: The image acquisition unit also includes an infrared micro camera and a foreground camera; The infrared micro camera is used to collect binocular infrared images of the object to be detected during the learning process, including a left-eye infrared camera and a right-eye infrared camera; The left-eye infrared camera, the right-eye infrared camera and the scene camera are integrated on the head-mounted device, and the left-eye infrared camera and the right-eye infrared camera are distributed on both sides of the scene camera. The scene camera is set on the center line of the left-eye infrared camera and the right-eye infrared camera, and the position relationship between the two-dimensional gaze point of the eyes of the object to be detected is calculated based on the scene image collected by the scene camera at this position; The foreground camera is used to collect facial images of the object to be detected during the learning process.

7. A computer device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 4.

8. A computer-readable medium, characterized in that It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 4.

Images