Intelligent human-machine interaction method based on viewpoint tracking

Abstract

The application discloses a kind of intelligent man-machine interaction methods based on viewpoint tracking, belong to the man-machine interaction technical field based on viewpoint tracking.The application includes: image is collected by the camera of computer self, and face detection is carried out to the image collected, and face image is extracted based on face detection frame;Face feature point detection is carried out to face image, and the face feature point of face image is obtained;Head posture estimation is carried out to face feature point and obtains eye image, and viewpoint tracking processing based on deep learning, man-machine interaction processing based on viewpoint tracking.The application realizes a kind of only through computer self camera, without the aid of any other hardware equipment, in the condition without any constraint, utilize deep learning algorithm real-time to track user viewpoint, and monitor user facial posture change, utilize viewpoint and facial posture change to realize man-machine interaction based on viewpoint tracking.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology based on viewpoint tracking, and specifically relates to an intelligent human-computer interaction method based on viewpoint tracking. Background Technology

[0002] Human research on viewpoint tracking has evolved from early methods that relied on external hardware to determine eye movements to today's viewpoint tracking technologies that use cameras or sensors to estimate the user's viewpoint position. As viewpoint tracking technology has progressed, humans have also sought more efficient human-computer interaction methods, leading to the development of viewpoint tracking-based human-computer interaction technologies.

[0003] Viewpoint tracking technology is a technique that uses specialized equipment to collect and analyze a user's eye information to determine their visual focus. It has wide applications in numerous fields. For example, in education, it can assess student learning quality based on their gaze point and area of ​​focus; in psychology, it can detect mental state based on changes in gaze, potentially aiding in the treatment of mental illnesses; in security, it can verify identity based on pre-set viewpoint trajectories; and in transportation, it can identify blind spots and assist driving. In the future, due to its compatibility, viewpoint tracking technology will be integrated with even more fields, showing great promise for future applications.

[0004] Human-computer interaction (HCI) technology refers to the exchange of information between humans and computers to achieve desired tasks. HCI research primarily focuses on the design, gaze, and evaluation of novel interfaces that improve HCI interaction, thereby achieving a more intuitive and robust HCI approach. Therefore, HCI naturally involves both computer science and the humanities. Computer science includes image processing, computer vision, and programming languages, while the humanities include artificial intelligence and cognitive psychology. Viewpoint-tracking-based intelligent HCI technology organically combines computer science and the humanities, forming a more intuitive and robust HCI approach.

[0005] Viewpoint-tracking-based intelligent human-computer interaction technology offers users a more convenient and efficient operating experience through a novel approach to human-computer interaction. Furthermore, this technology can eliminate the need for certain hardware devices such as keyboards and mice, helping people with disabilities and illnesses operate computers, and thus has broad research and application value. Summary of the Invention

[0006] This invention provides an intelligent human-computer interaction method based on viewpoint tracking, which aims to achieve real-time tracking of the user's viewpoint and monitoring of changes in the user's facial posture using deep learning algorithms, without relying on any other hardware devices and without any constraints, using viewpoint and facial posture changes to realize human-computer interaction based on viewpoint tracking.

[0007] The technical solution adopted in this invention is:

[0008] A viewpoint tracking-based intelligent human-computer interaction method, comprising the following steps:

[0009] Step 1: Capture images using the computer's built-in camera, perform face detection on the captured images, and extract face images based on the face detection bounding boxes;

[0010] Facial feature point detection is performed on the face image to obtain the facial feature points of the face image;

[0011] Step 2, Head pose estimation:

[0012] Step 201: Define a 3D face model that includes 6 key points. The 6 key points of the 3D face model are: each eye includes two key points at the left and right corners of the eyes, and the mouth includes two key points at the left and right corners of the mouth.

[0013] Step 202: Among the facial feature points obtained in step 1, find the feature points corresponding to the positions of the 6 key points of the 3D face model to obtain 2D key points; and extract the eye image from the face image based on the key points of the two eyes in the 2D key points.

[0014] Step 203: Based on the 6 key points of the 3D face model and the corresponding 2D key points, the EPNP algorithm is used to obtain the head rotation vector r, and the rotation matrix R′ and the corresponding Euler angles are calculated based on the head rotation vector r. The Euler angles determine the rotation angle of the head, i.e. the 2D head angle.

[0015] Step 204, Normalization:

[0016] Define a head coordinate system, which includes x, y and z axes. The x-axis is the midpoint connecting the two eyes and runs from right to left. The y-axis starts from the midpoint of the right eye and points towards the mouth, perpendicular to the x-axis. The z-axis starts from the midpoint of the right eye and points towards the back of the face, perpendicular to both the x-axis and y-axis.

[0017] Calculate the translation vector t between the head coordinate system and the camera coordinate system. r ;

[0018] According to formula e r =t r +eh Calculate the positions e of the left and right eyes in the camera coordinate system respectively. r , where e h The midpoint between the two key points at the outer corners of the eyes;

[0019] Calculate the transformation matrix T = SR′, where the scaling matrix S = diag(1, 1, d) n / ‖e r ‖), d n Indicates the origin of the camera coordinate system;

[0020] Based on the transformation matrix T, the key point positions of the eyes in the 2D key points are transformed, and the 2D head angle is transformed to obtain the normalized 2D eye key point positions and 2D head angle.

[0021] Step 3, deep learning-based viewpoint tracking processing;

[0022] The VGG16 convolutional neural network architecture is used as the network model for viewpoint tracking. The eye image is used as the input of the model, and the normalized 2D head angle is input into the first fully connected layer of VGG16. The pooling stride of the first and second pooling layers of VGG16 is modified to 1. The third fully connected layer of VGG16 is used to predict the spatial gaze vector of the eye in the current input image.

[0023] The predicted spatial gaze vector is spatially mapped to obtain the screen coordinates corresponding to the current spatial gaze vector:

[0024] Calculate the distance t from the eye to the screen: Among them, z e This represents the z-axis coordinate of the eye center in the head coordinate system. g Represents the z-axis coordinate of the spatial gaze vector in the head coordinate system;

[0025] Calculate the coordinates (x, y) of the spatial gaze vector on the camera plane: x = x e +t×x g y = y e +t×y g ;where x e y e Let x and y be the coordinates of the eye center in the head coordinate system. g y g The x and y coordinates of the spatial gaze vector in the head coordinate system;

[0026] Convert the coordinates (x, y) of the camera plane to screen pixels to obtain the screen-mapped point (P) of the spatial gaze vector. x P y ):

[0027]

[0028]

[0029] Among them, W x H x W represents the number of pixels on the screen. m H m Indicates the screen size.

[0030] Step 4, Human-computer interaction processing based on viewpoint tracking:

[0031] Based on the pre-defined mapping between user facial pose change information and input device operation methods, the input device operation method corresponding to the current user facial pose change information is extracted. The operation position of this operation method on the screen is the screen mapping point (P) of the spatial gaze vector. x P y ).

[0032] Furthermore, in step 1, a face detection algorithm based on directional gradient histogram is used to perform face detection on the acquired images.

[0033] Furthermore, in step 1, a face feature point detection algorithm based on cascaded regression trees is used to detect face feature points in the face image.

[0034] Furthermore, in step 4, mouse operation is controlled based on changes in eye posture.

[0035] Furthermore, in step 4, the operation of the mouse and keyboard is controlled based on the changes in the mouth's posture and the position of the nose.

[0036] The technical solution provided by this invention brings at least the following beneficial effects:

[0037] (1) It can use only the ordinary monocular network camera that comes with the computer as the image acquisition tool for viewpoint tracking, which is highly applicable.

[0038] (2) A VGG16-based convolutional neural network was used as a deep learning algorithm to track the user’s viewpoint. Small adaptive adjustments were made to the input eye image for the VGG16 framework. At the same time, the head pose information was input into the fully connected layer, and the VGG16 framework was extended into a multimodal model, which improved the accuracy of viewpoint tracking.

[0039] (3) By using the human-computer interaction method of user facial changes, real-time, convenient and efficient human-computer interaction is achieved. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of key points of a 2D human face in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of the normalization operation in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the head coordinate system in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the VGG16 convolutional neural network structure;

[0045] Figure 5 This is a schematic diagram of the viewpoint tracking process in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the processing using a VGG16-based convolutional neural network in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0048] The intelligent human-computer interaction method based on viewpoint tracking provided in this invention mainly includes: facial feature point detection, head pose estimation, deep learning-based viewpoint tracking, and human-computer interaction based on viewpoint tracking.

[0049] 1. Facial feature point detection.

[0050] Face and feature point detection are fundamental to intelligent human-computer interaction based on viewpoint tracking. Viewpoint tracking research can only proceed after accurate detection of faces and feature points. Face detection determines whether a face is in front of the camera, while facial feature point detection, by extracting facial features such as the eyes, provides valuable information for head pose estimation and viewpoint tracking.

[0051] Histogram of Oriented Gradients (HOG) is a feature description method used for object detection in computer vision and image processing. This method is applicable to frontal and slightly non-frontal faces, and is characterized by its speed and resistance to occlusion, effectively addressing the core problems faced in face detection. Face detection based on Histogram of Oriented Gradients includes the following steps:

[0052] (1) Image normalization:

[0053] To reduce the impact of lighting conditions, the RGB image needs to be converted to grayscale and then standardized in the Gamma space. The Gamma compression formula is:

[0054] I′(x, y)=I(x, y) γ (1)

[0055] Where γ is the standardization coefficient, usually taken as 0.2, I(x,y) represents the pixel value at point (x,y), and I′(x,y) represents the standardized pixel value.

[0056] (2) Calculate the image gradient:

[0057] An image is composed of pixels and can be viewed as a two-dimensional discrete function f(x, y). The gradient G(x, y) of the image function f(x, y) at a point (x, y) is a vector with magnitude and direction. The formula for the image gradient algorithm is:

[0058] G x (x,y)=I(x+1,y)-I(x-1,y) (2)

[0059] Among them, G x (x, y) is the rate of change of pixel value in the horizontal direction.

[0060] G y (x,y)=I(x,y+1)-I(x,y-1) (3)

[0061] Among them, G y (x, y) is the rate of change of pixel value in the vertical direction of the pixel.

[0062] The pixel value gradient at a point can be obtained from the changes in pixel values ​​in both the horizontal and vertical directions. Its magnitude and direction are:

[0063]

[0064]

[0065] (3) Calculate the weighted projection:

[0066] Taking a 128×64 detection bounding box as an example, the weight projection calculation process is as follows: Divide the 128×64 detection bounding box into 8×8 structural units. For the gradient of each pixel in the structural unit, based on the relationship between the gradient direction and the angle of the histogram, distribute the gradient magnitude proportionally to the group to which the histogram angle belongs. The distribution formula is:

[0067]

[0068] Where, value left It is the value assigned to the left side of the gradient direction angle in the histogram, bin right It is the angle to the right of the gradient direction angle.

[0069]

[0070] Where, value right It is the value assigned to the left side of the gradient direction angle in the histogram, bin left It is the angle to the left of the gradient direction angle.

[0071] (4) Block normalization:

[0072] The structural units of the detection window are grouped into blocks of 2×2. Structural units with a step size of 1 are selected in both the row and column directions of each block, resulting in a total of 15×7=105 blocks. The values ​​of each group of histograms of the structural units within each block are concatenated into a vector, giving each block a 4×9=36-dimensional feature vector. The concatenated vector is then normalized using the L2 norm, as shown in the formula:

[0073]

[0074] Where, ω i These are the elements in the vector to be normalized, where n represents the number of elements in the vector to be normalized, and ω′ i This is the normalized value.

[0075] (5) Calculate HOG features:

[0076] The feature vectors of all blocks in the detection window are concatenated to form the HOG feature vector of that detection window. This HOG feature vector is then input into a classifier to determine whether a face exists within the detection window.

[0077] The core of the face feature point detection method based on cascaded regression trees is the use of two layers of regression to establish a mathematical model. The iterative formula for the first layer of regression is:

[0078]

[0079] Here, the superscript t represents the iteration number, and S is a shape vector containing 68 facial key points. The current shape vector and the image are input into the regressor γ of the first layer. t Then, the updated keypoint positions are output. This way, the cascaded regressor updates the keypoint positions. The second layer of regression is γ. t In the internal regression process, this method uses the GTBA (gradient tree boosting algorithm) method to obtain the column regression tree, thereby completing the second level of regression.

[0080] Face detection algorithms based on histogram of oriented gradients and face landmark detection algorithms based on cascaded regression trees can be implemented using the Dlib library. The Dlib library is a cross-platform open-source software library written in both C++ and Python, containing various algorithms for image processing, deep learning, and more, and is widely used in various fields.

[0081] After face detection is completed, facial landmark detection can be performed on the obtained face image. In this embodiment of the invention, facial landmark detection yields a face image containing 51 internal feature points and 17 contour feature points. The left and right eyes each contain 6 feature points: outer corner P1, inner corner P4, and four feature points on the upper and lower eyelids: P2, P3, P5, and P6. P2 and P3 are located on the upper eyelid, and P5 and P6 are located on the lower eyelid. The relationship between the coordinates of these points differs when the eyes are open and closed; that is, the aspect ratio of the eyes is the same when they are open and closed.

[0082] Therefore, the open / closed state of the eyes can be determined by calculating the aspect ratio of the eyes. The formula for the aspect ratio of the eyes is:

[0083]

[0084] Statistical data shows that the aspect ratio of the eyes is basically consistent when they are open, around 0.25. However, when the eyes are closed, the aspect ratio immediately drops below 0.1. This study uses 0.2 as a threshold; when the aspect ratio is greater than this threshold, the user's eyes are considered open. When the aspect ratio is less than this threshold, the user's eyes are considered closed.

[0085] The algorithm for mouth opening detection is the same as that for blinking detection, which determines whether the mouth is open or closed based on the length-to-width ratio of the mouth.

[0086] 2. Head pose estimation.

[0087] Because head posture estimation and viewpoint estimation are intrinsically linked, head posture provides a rough direction for the viewpoint. Physiological studies have shown that the prediction of a person's viewpoint comes from a combination of head posture and eye direction.

[0088] In this embodiment of the invention, the EPNP (efficient perspective-n-point) head pose estimation algorithm is used, which mainly includes:

[0089] (1) Define a 3D face model with 6 key points. The 6 key points defined in this invention are determined based on the average facial shape of participants in the MPIIGaze dataset, namely the left and right corners of the eyes and the corners of the mouth. Based on facial feature point detection, the locations of the defined key points in the user's 2D image are found, such as... Figure 1 As shown.

[0090] (2) The head rotation vector is obtained by using the EPNP algorithm based on the key points of the 3D face model and the key points of the 2D face.

[0091] (3) Convert the head rotation vector to Euler angles, specifically:

[0092] (1): Find four control points in the world coordinate system, denoted as

[0093] Set 3D key point set Select the centroid of the 3D key points as the first control point:

[0094]

[0095] Where n represents the number of 3D key points.

[0096] This leads to matrix A:

[0097]

[0098] Let A be the name of the person in question. T The eigenvalue of A is γ c，i For i = 1, 2, 3, the corresponding eigenvector V c，i Since i = 1, 2, 3, the remaining three control points can be calculated using the following formula:

[0099]

[0100] (2) The coordinates of the feature points are represented by a weighted sum of the coordinates of the control points:

[0101]

[0102] Where, α ij Indicates the weight.

[0103] (3) Calculate the coordinates of the control points in the camera coordinate system:

[0104] Let K be the intrinsic parameter matrix of the camera, which can be obtained through calibration. {μ i} i＝1，…，n The key point {P i} i＝1，…，n The 2D projection, i.e., the 2D key points, then:

[0105]

[0106] use Substitute into the above equation and express K as the focal length f. u f v and photocenter (u c v c (in the form of)

[0107]

[0108] From the above equation, we can obtain two linear equations:

[0109]

[0110]

[0111] Connecting the n points in series yields a system of linear equations:

[0112] Mx = 0 (19)

[0113] in, This refers to the calculated coordinates of the control point in the camera coordinate system.

[0114]

[0115] Among them, v i These are the eigenvectors corresponding to the N zero eigenvalues ​​of matrix M. Therefore, the i-th control point c i for:

[0116]

[0117] in It is the feature vector v k The i-th 3×1 vector. Next, we can calculate M. T The eigenvectors of M yield v i Calculate {β} using the following formula. i} i＝1，…，N

[0118]

[0119] (4) Calculate the coordinates of the 3D key points in the camera coordinate system:

[0120]

[0121] (5) Calculation center of gravity Sum matrix A:

[0122]

[0123]

[0124] (6) Calculation center of gravity Sum matrix B:

[0125]

[0126]

[0127] (7) Calculate matrix H:

[0128] H = B T A (28)

[0129] (8) Calculate the SVD decomposition of matrix H:

[0130] H=U∑V T (29)

[0131] (9) Calculate the rotation vector r in the pose:

[0132] r = UV T (30)

[0133] (10) Calculate the rotation matrix R:

[0134] R = cosθI + (1 - cosθ)nn T +sinθn^ (31)

[0135] Where I is the identity matrix, n is the unit vector of the rotation vector r, and θ is the magnitude of the rotation vector.

[0136] (11) Calculate Euler angles:

[0137] R = R z (α)R y (β)R x (γ) (32)

[0138] Where α, β, and γ represent Euler angles, and R z (α), R y (β), R x(β) represents the rotation matrix obtained by rotating the object based on the z, y, and x axes by angles α, β, and γ, respectively.

[0139] Normalization refers to the process of normalizing the input image, which involves converting images of different head poses, varying distances between the person and the camera, and other common conditions into a predefined, invariant normalized image before viewpoint tracking is performed. Normalization increases the user's freedom, freeing them from fixed poses, positions, and distances, and also further improves the accuracy of convolutional neural networks.

[0140] In this embodiment of the invention, the normalized diagram is as follows: Figure 2 As shown, R r Let C represent the rotation vector. r C represents the position of the origin of the camera coordinate system. n This indicates the position of the origin of the camera coordinate system after the scaling matrix transformation.

[0141] In this embodiment of the invention, the normalization operation steps are as follows: Figure 2 As shown, it includes:

[0142] (1) Define the head coordinate system.

[0143] The head coordinate system is determined by three center points: the eyes and the mouth. Figure 3 As shown. The x-axis is the midpoint connecting the two eyes, running from right to left; the y-axis starts from the midpoint of the right eye and points towards the mouth, perpendicular to the x-axis; the z-axis starts from the midpoint of the right eye and points towards the back of the face, perpendicular to both the x-axis and y-axis.

[0144] (2) Calculate the translation vector t between the head coordinate system and the camera coordinate system. r And the rotation matrix R.

[0145] (3) Calculate the position of the human eye in the camera coordinate system (calculate the left eye and the right eye separately).

[0146] e r =t r +e h (33)

[0147] Among them, t r e is the translation vector h It is the midpoint between the two key points at the corners of the eyes.

[0148] (4) Calculate the rotation matrix R′ so that the eye position is on the z-axis of the camera coordinate system:

[0149] R′=R -1 (34)

[0150] (5) Calculate the scaling matrix S so that the eye position is located at a distance d from the origin of the camera coordinate system. s Location:

[0151] S = diag(1, 1, d) s / ||e r ||) (35)

[0152] (6) Calculate the transformation matrix T and complete the normalization:

[0153] T=SR′ (36)

[0154] 3. Viewpoint tracking based on deep learning.

[0155] Convolutional neural networks (CNNs) for image processing typically include convolutional layers, pooling layers, and fully connected layers. In the input layer of a CNN, the form of the image input differs from that of a regular neural network. CNNs allow the input image to retain its original structure. For example, in a 28×28 black and white image, the input to a CNN is a 28×28 two-dimensional neuron. In a 28×28 RGB image, the input is a 3×28×28 three-dimensional neuron (each color channel within RGB has a 28×28 matrix).

[0156] In convolutional neural networks, the convolutional layer is a key structure that extracts features from an image. Convolution is an operation that locally weights the input signal. Therefore, the characteristics of the input signal depend on the selected set of weights. Thus, choosing appropriate weights is crucial for feature extraction. There are two important concepts in convolutional layers: (1) receptive field and (2) shared weights. Suppose a 28×28 two-dimensional neuron is input, and a 5×5 neuronal region is extracted from it and connected to a neuron in the hidden layer. This region is the receptive field. The receptive field can be understood as: in the hidden layer, the neuron can perceive features within a fixed-size region of the previous layer. In a fully connected neural network, the receptive field of the neuron in the hidden layer is the entire region of the previous layer, so it can perceive all the features of the previous layer. However, in a convolutional neural network, the receptive field of the neuron in the hidden layer is very narrow, and it can only perceive the local features of the previous layer. The remaining features of the previous layer are obtained by translating the receptive field. The translation interval of the receptive field is called the stride. When the stride is large, translating the receptive field may exceed the dimension of the neuron, so the value of the neuron needs to be set according to the requirements to supplement the dimension. Assuming the translation step size is 1: translate from left to right, translating 1 unit at a time. After completing the row translation, translate down 1 unit, and then translate from left to right again.

[0157] Neurons in a convolutional layer also have weights w. The matrix of neuron weights in the receptive field is called the convolution kernel, and the size of the convolution kernel is user-defined. To ensure the bias of the convolution kernel, an offset b is added to it. This offset can be randomly generated and can be changed according to training. For neurons in the next layer, they all perceive the features of neurons in the previous layer from different parts. The matrix of neurons in the next layer generated by the receptive field is called a feature map. Each convolution kernel generates a feature map. Therefore, when there are three different convolution kernels, three different feature maps can be obtained. In this way, the number of parameters required for training in the convolutional neural network is greatly reduced to (5×5+1)×3=78. When the input data is an RGB image, the size of the convolution kernel is not only measured by its length and width, but also by its depth. At this time, the receptive field also needs to have depth. The depth of the convolution kernel and the receptive field is the same, depending on the input data, but their length, width, and number can be defined by the user, and each convolution kernel corresponds to a feature map.

[0158] The operation of a convolutional layer involves linearly transforming the input data using a convolutional kernel. Therefore, it's necessary to convert this linear operation into a non-linear one to fit the desired function. This requires an activation function to perform a non-linear mapping on the data from the convolutional layer.

[0159] The task of the pooling layer is to reduce the dimensionality of the feature map. Although the convolutional layer has already reduced the dimensionality of the input data through its receptive field, its receptive field and stride are relatively small, making it unable to obtain a low-dimensional feature map. Therefore, the pooling layer is needed to further map the feature map through its pooling field. There are two mapping methods for the pooling field:

[0160] (1) Max pooling: Only the maximum value in the pooling view matrix is ​​taken as the feature.

[0161] (2) Average pooling: Calculate the average value of the pooling view matrix as a feature.

[0162] When the pooling field of view is 2×2 and the stride is 2, three 24×24 feature maps can be pooled into three 12×12 feature maps.

[0163] Fully connected layers and output layers: A fully connected layer is a linear feature map that maps multi-dimensional feature inputs to two-dimensional feature outputs, thereby obtaining the desired result. The output layer is mainly responsible for outputting the prediction result.

[0164] Preferably, a VGG16 network can be used for viewpoint tracking. The VGG16 network consists of 16 weight layers: 13 convolutional layers and 3 fully connected layers. Five pooling layers are interspersed between these 16 weight layers. The input to the VGG16 network is the first convolutional layer, followed by three stacked fully connected layers. The first pooling layer follows the second convolutional layer, the second pooling layer follows the fourth convolutional layer, the third pooling layer follows the seventh convolutional layer, the fourth pooling layer follows the tenth convolutional layer, and the fifth pooling layer follows the fifteenth convolutional layer. The final fully connected layer includes a softmax function. Figure 4 As shown, the process begins by locating facial landmarks in images acquired by a rectified computer RGB camera using a face detection method based on histogram of oriented gradients (HARQ) and a face landmark detection method based on cascaded regression trees. Next, a 3D face shape model is created using the average facial shape of all participants in the MPIIGAZE dataset to estimate the 3D pose of the detected faces. Normalization techniques are then used to convert eye images into standard training data. Then, a VGG16-based convolutional neural network is used to learn the mapping relationship between eye images, head pose, and gaze direction. Finally, based on the linear relationship, the spatial mapping is restored to the screen to achieve viewpoint tracking, as shown below. Figure 5 As shown.

[0165] The dataset used is the MPIIGAZE dataset, which contains 213,659 images collected over several months from the daily lives of 15 laptop users. Collection employed an experience sampling method: participants were periodically triggered to view random locations on a laptop screen. This method allows MPIIGAZE to provide realism in terms of eye appearance and lighting variations. All images in the dataset include the gaze target and preprocessed six key points for the eyes and mouth.

[0166] In this embodiment of the invention, the task of the VGG16-based convolutional neural network is to learn the mapping from input features: 2D head angle h and eye image e to spatial gaze angle g. First, the pixel size of the eye image input is set to 60×30. Then, the stride of the first and second pooling layers of the VGG16 convolutional neural network is modified to 1 to accommodate the smaller pixel size of the eye image. Finally, the 2D head angle h is added to the first fully connected layer, and the network structure is as follows: Figure 6 As shown.

[0167] The loss function used is the L2 loss function, which minimizes the squared error function, i.e., minimizes the true value y. i and predicted value f(y) i The difference D between ) L1 The sum of squares. The L2 loss function formula is as follows:

[0168]

[0169] Among them, D L2 For the mean absolute error, the L2 loss function is minD. L2 .

[0170] Because the L2 loss function squares the error, it has better robustness. At the same time, the L2 loss function facilitates gradient descent, which is beneficial for function convergence. Therefore, the L2 loss function is used to measure the distance between the predicted spatial gaze angle g and the true gaze angle g'.

[0171] The output of the view tracking algorithm based on the VGG16 convolutional neural network is the spatial gaze vector of the eye (i.e., the spatial gaze angle g). Therefore, it is necessary to map the spatial gaze vector to screen coordinates. The spatial mapping steps are as follows:

[0172] (1) Taking the center of the eye as the starting point of the spatial gaze vector, calculate the distance t from the eye to the screen:

[0173]

[0174] Among them, z e It is the z-axis coordinate of the center of the eye, z g It is the z-axis coordinate of the spatial gaze vector, and the two are in opposite directions.

[0175] (2) Calculate the coordinates of the spatial gaze vector on the camera plane based on the linear relationship:

[0176] x = x e +t×x g (39)

[0177] y = y e +t×y g (40)

[0178] Where, x e y e Let x and y be the coordinates of the center of the eye. g y g Let x and y be the coordinates of the spatial gaze vector.

[0179] (3) Convert the coordinates of the camera plane to screen pixels:

[0180]

[0181]

[0182] Among them, W x H x It refers to the screen's pixels, W. m Hm It refers to the screen size.

[0183] 4. Human-computer interaction based on viewpoint tracking.

[0184] This means using the user's viewpoint coordinates (screen coordinates mapped from the spatial gaze vector) and facial information to control computer input devices (mouse, keyboard).

[0185] Based on a pre-set mapping table of user facial information (such as eye status, open, closed, blinking; mouth status, etc.) and input device operation methods, the input device operation method corresponding to the current user facial information is extracted, and the screen position of the operation method is the user's viewpoint coordinates.

[0186] As one possible implementation, the intelligent human-computer interaction method based on viewpoint tracking provided in this embodiment of the invention includes the following steps:

[0187] Step 1, Face and Feature Point Detection:

[0188] The captured video images are read frame by frame. Distortion correction is performed on each frame to ensure image accuracy.

[0189] A face detection algorithm based on histogram of oriented gradients is used to detect faces in images, and the face locations are marked with rectangular detection boxes. When there are more than one face in the image, the face with the largest detection box is selected; when no face is found in the image, no further processing is performed.

[0190] A face detection algorithm based on cascaded regression trees was used to detect the selected faces and mark 68 feature points of the faces.

[0191] Step 2, Head pose estimation:

[0192] Based on the MPIIGAZE dataset, an average 3D face model is calculated as a baseline for head pose estimation.

[0193] The EPNP algorithm is used to select six key points, including the corners of the eyes and mouth, detected by facial feature points to solve for the head pose, i.e., Euler angles.

[0194] Draw a coordinate system for head posture with the center of the nose as the origin.

[0195] Step 3, Viewpoint Tracking:

[0196] The VGG16-based convolutional neural network was trained using the MPIIGAZE dataset. Training was stopped when the preset training termination conditions were met (either the number of training iterations or the prediction accuracy reached a specified condition).

[0197] The detected eyes from facial feature points are normalized and cropped into individual eye images. These cropped eye images, along with head pose information, are then input into a VGG16-based convolutional neural network to obtain the spatial gaze angle of the eyes.

[0198] Finally, based on the linear relationship, the spatial mapping is restored to the screen, and the coordinates of the viewpoint, i.e., the screen coordinates (P), are calculated. x P y ).

[0199] Step 4, Human-Computer Interaction:

[0200] (1) Mouse function design:

[0201] Primarily relying on viewpoint tracking and eye posture, with the nose and mouth playing a secondary role, the eye replaces the mouse in performing functions such as cursor movement, clicking, double-clicking, and scrolling. The eye-mouse functions are shown in Table 1.

[0202] Table 1 Eye-Mouse Functions

[0203] Function Eye posture Left click The left eye blinks and remains closed for more than 1 second. Right-click The right eye blinks and remains closed for more than 1 second. Left-click double-click The left eye blinks and remains closed for more than 2.5 seconds. scroll Close your eyes for more than 2 seconds

[0204] (1) Mouse control method:

[0205] Viewpoint tracking based on convolutional neural networks is coarse. Directly setting the viewpoint position predicted by viewpoint tracking as the mouse position can cause mouse jitter and make it difficult to accurately locate small targets. Therefore, using the mouth and nose to assist in mouse control solves this problem.

[0206] For example, opening the mouth for 0.5 seconds activates viewpoint tracking: In the first second, no viewpoint estimation is performed; the user determines the desired mouse position within this second. In the second second, the user's average viewpoint position within this second is calculated, and the mouse is moved to that position within 0.1 seconds, then viewpoint tracking is deactivated. Next, the user uses their nose to move up, down, left, and right to control the mouse movement in the same direction at a speed of 0.5 cm / s, precisely modifying the mouse position. Then, the corresponding mouse functions are implemented according to the eye-mouse function table. Finally, opening the mouth for 0.5 seconds activates the next viewpoint tracking cycle.

[0207] (3) Keyboard control method:

[0208] Viewpoint-tracking-based keyboard control can be based on the operating system's built-in touch keyboard, and input can be completed by clicking the touch keyboard using viewpoint-tracking-based mouse control.

[0209] In this embodiment of the invention, human-computer interaction is achieved by detecting the state of the eyes (left eye, right eye) and mouth, and controlling the mouse and keyboard based on the PyAutoGUI library.

[0210] Taking reading news online as an example, first, open your mouth for 0.5 seconds to activate view tracking. Look at the touchscreen keyboard; after 2 seconds, move the mouse near the keyboard and then precisely move the mouse to the keyboard using your nose. Close your left eye for 1 second and left-click the keyboard. Open your mouth for 0.5 seconds to activate view tracking. Look at the search box; after 2 seconds, move the mouse near the search box and then precisely move the mouse to the search box using your nose. Close your left eye for 1 second and left-click the search box. Repeat the mouse movement and left-clicking to type "news". Left-click "Baidu Search" to complete the search. Close both eyes for 2 seconds to activate scroll wheel control; use your nose to scroll the wheel up and down to select the content you want to read. After selecting the content, open the webpage to read it using the mouse control method.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0212] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for intelligent human-machine interaction based on viewpoint tracking, characterized in that, Comprise the following steps: Step 1, the image is collected by the camera of the computer, and the collected image is subjected to face detection, and the face image is extracted based on the face detection frame; Face feature point detection is performed on the face image to obtain face feature points of the face image; Step 2, head pose estimation: Step 201, define a 3D face model including 6 key points, wherein the 6 key points of the 3D face model are: each eye includes two key points of left and right corners, and the mouth includes two key points of left and right corners; Step 202, in the face feature points obtained in step 1, find the feature points corresponding to the positions of the 6 key points of the 3D face model to obtain 2D key points; and based on the key points of the two eyes in the 2D key points, extract eye images from the face image; Step 203, based on the 6 key points of the 3D face model and the corresponding 2D key points, the EPNP (efficient perspective-n-point) algorithm is used to obtain the head rotation vector r, and the rotation matrix R' and the corresponding Euler angle are calculated based on the head rotation vector r, wherein the Euler angle determines the rotation angle of the head, i.e. 2D head angle; Step 204, normalization processing: Define a head coordinate system, which includes x, y and z axes, wherein the x axis is the midpoint connecting the two eyes, and the direction is from right to left; the y axis is from the right eye midpoint to the mouth direction, and is perpendicular to the x axis; the z axis is from the right eye midpoint to the back of the face and perpendicular to the x axis and the y axis; calculating a translation vector t between the head coordinate system and the camera coordinate system r ; According to formula e r =t r +e h Calculate the positions e of the left and right eyes in the camera coordinate system respectively. r , where e h The midpoint between the two key points at the outer corners of the eyes; Compute the transformation matrix T = SR' where the scaling matrix S = diag(l, l, d n / ‖e r ‖), d n denotes the origin of the camera coordinate system; Based on the conversion matrix T, the key point positions of the eyes in the 2D key points are converted, and the 2D head angle is converted to obtain the normalized 2D eye key point positions and 2D head angle; Step 3, view point tracking processing based on deep learning; The convolutional neural network VGG16 architecture is used as the network model of the view point tracking, the eye image is used as the input of the model, and the normalized 2D head angle is input into the first fully connected layer of VGG16; and the pooling step of the first and second pooling layers of VGG16 is modified to 1; the third fully connected layer of VGG16 is used to predict the spatial gaze vector of the current input image; The predicted spatial gaze vector is subjected to spatial mapping processing to obtain the screen coordinates corresponding to the current spatial gaze vector: Compute the eye-to-screen distance t: where z e represents the z-axis coordinate of the eye center in the head coordinate system, z g represents the z-axis coordinate of the spatial gaze vector in the head coordinate system; The spatial gaze vector at the coordinate point (x, y) of the camera plane: x = x e + t x g , y = y e + t y g ; wherein x e , y e are the x, y axis coordinates of the eye center in the head coordinate system, and x g , y g are the x, y axis coordinates of the spatial gaze vector in the head coordinate system; Transform the coordinate point (x, y) of the camera plane into a screen pixel point to obtain the screen mapping point (P x ,P y ) of the spatial gaze vector. where W x , H x represent the size of the screen; and where W m , H m represent the size of the screen. Step 4, human-computer interaction processing based on view point tracking: Based on the preset mapping of user facial posture change information and operation mode of the input device, an input device operation mode corresponding to the current user facial posture change information is extracted, and an operation position of the operation mode on the screen is a screen mapping point (P x ,P y ) of the spatial gaze vector.

2. The method of claim 1, wherein, In step 1, the face detection algorithm based on direction gradient histogram is used to detect the face in the collected image.

3. The method of claim 1, wherein, In step 1, the face feature point detection algorithm based on hierarchical regression tree is used to detect the face feature points of the face image.

4. The method of claim 1, wherein, In step 4, the operation of the mouse is controlled based on the posture change of the eyes.

5. The method of claim 1, wherein, In step 4, the operation of the mouse and keyboard is controlled based on the posture change of the mouth and the position of the nose.

Images