Human-computer interaction real-time sight line estimation method based on deep learning

Abstract

The invention relates to the technical field of human-computer interaction sight line estimation, in particular to a human-computer interaction real-time sight line estimation method based on deep learning, which comprises the following steps: continuously capturing a face image of a user, and extracting a standardized eye sub-region image and a head posture parameter after processing; the eye appearance, the eye geometric structure and the head posture features are extracted in parallel through a multi-stream feature extraction network and are dynamically fused into comprehensive feature representation. The features are input into a recurrent neural network to model a time sequence dynamic state, and a preliminary sight line direction is obtained; and carrying out space mapping and calibration in combination with the three-dimensional environment model, and outputting a final sight line drop point coordinate under a world coordinate system. According to the method, through heterogeneous feature dynamic fusion and time sequence modeling, the estimation precision and the anti-interference capability are improved, and precise driving of a specific interaction scene is realized through three-dimensional space mapping.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction gaze estimation, and in particular to a human-computer interaction real-time gaze estimation method based on deep learning. BACKGROUND

[0002] Real-time gaze estimation is a key technology in the field of human-computer interaction, aiming to determine the gaze direction of the user by analyzing the user image. The existing mainstream technology usually relies on extracting features from a single frame of face image for estimation. One class of methods focuses on learning appearance features from cropped eye region images, and another class of methods attempts to combine simple head pose information. These methods generally treat the eye as a whole or only use the head pose angle as auxiliary information to shallowly splice with the eye features. In terms of time sequence processing, most schemes independently process each frame of image, considering gaze estimation as a static image task, ignoring the time sequence characteristics of continuous and smooth changes of gaze in real scenes.

[0003] The current technical solution has defects: the single feature flow or simple feature fusion method is difficult to fully express the complex mechanism of gaze formation. The eye appearance is greatly affected by light and occlusion, while the eye geometry is relatively stable. The existing methods fail to explicitly distinguish and effectively fuse these heterogeneous features, resulting in limited model generalization ability and decreased accuracy when the user's posture changes greatly. The frame-by-frame independent estimation method introduces result jitter and cannot utilize the time sequence continuity prior of gaze motion, resulting in unstable estimated trajectory. In addition, most methods stop at outputting an abstract gaze direction vector, lack a general processing link to accurately map it to a specific three-dimensional interactive scene, and are difficult to directly drive practical applications. SUMMARY

[0004] The purpose of the present application is to solve the problems in the prior art, and a human-computer interaction real-time gaze estimation method based on deep learning is proposed.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a human-computer interaction real-time gaze estimation method based on deep learning, comprising: capturing multiple continuous images containing the face region of the user by an image acquisition device to form an original image sequence; performing face key point positioning and alignment processing on each frame of image in the original image sequence to generate a standardized face region image; extracting an eye sub-region image and a head pose parameter from the standardized face region image; inputting the eye sub-region image and the head pose parameter into a multi-stream feature extraction network to extract eye appearance features, eye geometry features and head pose features in parallel; The eye appearance feature, the eye geometry feature and the head pose feature are dynamically fused to generate a comprehensive gaze feature identifier; The comprehensive gaze feature identifier is input into a recurrent neural network, the recurrent neural network is used to capture gaze dynamic change information between adjacent frames, and an initial gaze direction vector is calculated according to an output of the recurrent neural network; The initial gaze direction vector is spatially mapped and calibrated in combination with a three-dimensional environment model of a current scene to obtain a final gaze landing point coordinate of a user in a world coordinate system; The final gaze landing point coordinate is transmitted to a human-computer interaction interface in real time and is used to drive an interaction response.

[0006] As a further scheme of the application, the face key point positioning and alignment processing on each frame of image in the original image sequence to generate a standardized face region image comprises: A pre-trained face key point detection model is called to process each frame of the original image sequence, and a two-dimensional coordinate point set of a face contour, eyes, a nose tip and a mouth corner is identified; A rotation angle, a scaling ratio and a translation amount of the face in an image plane are calculated according to the two-dimensional coordinate point set; An affine transformation is performed on the original image according to the calculated rotation angle, scaling ratio and translation amount, and a face region is corrected to a predefined standard pose and standard size; The face region in the corrected image is cropped and uniformly scaled to a fixed resolution to generate the standardized face region image.

[0007] As a further scheme of the application, the eye sub-region image and the head pose parameter are extracted from the standardized face region image, comprising: In the standardized face region image, a bounding box of the eyes is determined according to the positions of the face key points; Pixel regions in the bounding box corresponding to the left and right eyes are respectively intercepted to obtain a left eye original image and a right eye original image; The left eye original image and the right eye original image are subjected to grayscale, contrast enhancement and size normalization processing to generate the eye sub-region image; Three-dimensional face shape model parameters output by the face key point detection model are matched with the two-dimensional coordinate point set, the head pose parameter is estimated by solving a perspective projection equation, and the head pose parameter comprises a yaw angle, a pitch angle and a roll angle of the head relative to a camera coordinate system.

[0008] As a further aspect of the present invention, the step of inputting the eye sub-region image and the head pose parameters into a multi-stream feature extraction network to extract eye appearance features, eye geometric structure features, and head pose features in parallel includes: The original images of the left eye and the right eye are respectively input into a convolutional neural network branch with shared weights. Through multiple convolution and pooling operations, the eye appearance features are extracted. Based on the coordinates of the corners of the eyes in the facial key points, the center of the eyeball, the distance between the inner and outer eye points, and the geometric quantity of eyelid curvature are calculated. The center of the eyeball, the distance between the inner and outer eye points, and the geometric quantity of eyelid curvature are then used to form a feature vector, which is encoded through a fully connected network to obtain the geometric structural features of the eye. The head pose parameter vector is input into another independent fully connected network for nonlinear transformation and feature dimensionality upscaling to obtain higher-level head pose features.

[0009] As a further aspect of the present invention, the dynamic fusion of the eye appearance features, the eye geometric structure features, and the head posture features to generate a comprehensive gaze feature identifier includes: The eye appearance features, the eye geometric structure features, and the head posture features are mapped to the same feature dimension space; An attention mechanism is used to calculate the cross-correlation weights among the eye appearance features, the eye geometric structure features, and the head posture features; Based on the cross-correlation weights, the eye appearance features, the eye geometric structure features, and the head posture features are weighted and summed to obtain a preliminary fusion feature; The preliminary fused features are compressed and deredundanted through a bottleneck layer to output a compact and information-rich integrated gaze feature identifier.

[0010] As a further aspect of the present invention, the step of inputting the integrated gaze feature identifier into a recurrent neural network and using the recurrent neural network to capture gaze dynamic change information between adjacent frames includes: The integrated gaze feature identifier of the current frame is concatenated with the state vector of the previous frame hidden by the recurrent neural network. The concatenated vector is input into the internal unit of the recurrent neural network to update the current state vector and output vector of the recurrent neural network; The output vector represents the current gaze feature that incorporates historical time-series information, and the current gaze feature includes information on the speed and direction of gaze movement.

[0011] As a further aspect of the present invention, the step of calculating the initial gaze direction vector based on the output of the recurrent neural network includes: The current gaze features output by the recurrent neural network are input into a fully connected regression layer; The fully connected regression layer outputs a three-dimensional vector, which represents the offset angle of the line of sight relative to the front of the head in the head coordinate system. Based on the offset angle and the known definition of the head coordinate system, the initial gaze direction vector in the head coordinate system is calculated.

[0012] As a further aspect of the present invention, the step of combining the current scene's three-dimensional environment model to spatially map and calibrate the initial gaze direction vector to obtain the user's final gaze point coordinates in the world coordinate system includes: Using the head pose parameters, a rotation transformation matrix from the head coordinate system to the camera coordinate system is constructed; Using the camera intrinsic parameter matrix, construct the projection transformation relationship from the camera coordinate system to the image coordinate system; The initial gaze direction vector in the head coordinate system is transformed to the camera coordinate system using the rotation transformation matrix to obtain the gaze direction vector in the camera coordinate system. By back-projecting the line-of-sight vector in the camera coordinate system, a three-dimensional spatial ray is obtained that originates from the optical center of the camera and passes through the corresponding pixel on the image plane. The intersection of the three-dimensional spatial ray with the scene geometry surface in the three-dimensional environment model is calculated, and the intersection point is the coordinate of the user's final line of sight in the world coordinate system. The steps for constructing the three-dimensional environment model include: Color and depth images of the interactive scene are acquired using a depth camera array; The color image and depth image are registered and fused together to generate dense 3D point cloud data of the scene. Plane detection and surface reconstruction are performed on the dense 3D point cloud data to identify the 3D geometric surfaces of the main objects in the scene; The identified 3D geometric surfaces are meshed and assigned unique object identifiers to construct the 3D environment model containing scene geometric information.

[0013] As a further aspect of the present invention, when the three-dimensional spatial ray has no intersection with the three-dimensional environment model, the method further includes: Along the direction of the three-dimensional spatial ray, find the nearest virtual interaction plane in the three-dimensional environment model; Calculate the coordinates of the intersection point between the three-dimensional spatial ray and the virtual interactive plane, and use the intersection point coordinates as the alternative final line-of-sight landing point coordinates; Record the uncertainty markers of the final gaze point coordinates of the candidate points for subsequent human-computer interaction logic to perform differentiated processing.

[0014] As a further aspect of the present invention, the step of employing an attention mechanism to calculate the correlation weights among the eye appearance features, the eye geometric structure features, and the head posture features includes: The eye appearance features, eye geometric structure features, and head posture features are linearly projected into the query space, key space, and value space, respectively, to generate query vector, key vector, and value vector; Calculate the first dot product between the query vector corresponding to the eye appearance feature and the key vector corresponding to the eye geometric structure feature, and scale the first dot product. Calculate the second dot product between the query vector corresponding to the eye appearance feature and the key vector corresponding to the head pose feature, and then scale the second dot product. The scaled first dot product and the second dot product are input into the Softmax function for normalization to generate the first attention weight between the eye appearance features and the eye geometric structure features, and the second attention weight between the eye appearance features and the head posture features. The value vector corresponding to the eye geometric structure feature is weighted according to the first attention weight, and the value vector corresponding to the head pose feature is weighted according to the second attention weight. The two weighted results are added to the value vector corresponding to the eye appearance feature to generate a weighted context feature vector. The weighted context feature vector is then input again into the linear projection layer and the Softmax function to calculate the cross-correlation weights between the eye appearance features, the eye geometric structure features, and the head pose features.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: A multi-stream feature extraction network processes eye appearance images, eye geometric parameters, and head pose parameters in parallel, generating a comprehensive gaze feature identifier through a dynamic fusion mechanism. This scheme decouples and collaboratively models eye appearance (representing texture information), eye geometry (representing morphological constraints), and head pose (representing spatial reference). The dynamic fusion mechanism adaptively adjusts the contribution weights of different feature streams based on the actual input data. This allows the system to flexibly rely on more reliable feature sources when dealing with changes in illumination, partial occlusion, or individual user differences, thereby improving the accuracy and environmental robustness of the basic gaze direction estimation.

[0016] The aforementioned comprehensive features are input into a recurrent neural network (RNN) to capture dynamic changes in gaze across multiple consecutive frames, performing temporal smoothing and contextual optimization on the initial estimation sequence. The RNN implicitly learns prior knowledge of the continuity and smoothness of gaze motion, effectively filtering out random noise and abnormal jitter introduced by single-frame estimation, outputting a stable gaze trajectory that conforms to physiological movement patterns. Based on this, the temporally optimized gaze direction vector is spatially mapped and calibrated with a pre-defined 3D environment model. This step achieves a precise transformation from the user's relative coordinate system to the world's absolute coordinate system, directly outputting the physical coordinates in the specific interaction scenario, completing the closed loop from perceived signals to directly driving interactive command information. Attached Figure Description

[0017] Figure 1 This is a flowchart of the real-time gaze estimation method for human-computer interaction based on deep learning described in this invention; Figure 2 A flowchart for extracting eye sub-region images and head pose parameters; Figure 3 A heatmap showing the multi-index performance comparison of different fusion methods in the feature fusion stage; Figure 4 The curve showing the dynamic change of the activation of the hidden state in a multilayer LSTM network; Figure 5 This is a heatmap of the Pearson correlation coefficient between the coordinates of the points where the line of sight falls. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] See Figure 1A deep learning-based real-time gaze estimation method for human-computer interaction continuously captures multiple consecutive images containing the user's facial region using an image acquisition device, forming an original image sequence. Facial keypoint localization and alignment are performed on each frame of the original image sequence to generate standardized facial region images. Eye sub-region images and head pose parameters are extracted from the standardized facial region images. These images and parameters are then input into a multi-stream feature extraction network to extract eye appearance features, eye geometric features, and head pose features in parallel. The eye appearance features, eye geometric features, and head pose features are dynamically fused to generate a comprehensive gaze feature identifier. This identifier is then input into a recurrent neural network (RNN), which captures dynamic changes in gaze between adjacent frames. Based on the RNN's output, an initial gaze direction vector is calculated. Combined with a 3D environment model of the current scene, the initial gaze direction vector is spatially mapped and calibrated to obtain the user's final gaze point coordinates in the world coordinate system. These final gaze point coordinates are then transmitted in real-time to the human-computer interaction interface to drive the interaction response.

[0021] In one embodiment of the present invention, a pre-trained facial keypoint detection model is invoked to process each frame of the original image sequence, identifying a set of two-dimensional coordinate points for the facial contours, eyes, nose tip, and corners of the mouth. The rotation angle, scaling factor, and translation amount of the face within the image plane are calculated based on the two-dimensional coordinate point set. Based on the calculated rotation angle, scaling factor, and translation amount, an affine transformation is performed on the original image to correct the facial region to a predefined standard pose and size. The facial region in the corrected image is cropped and uniformly scaled to a fixed resolution to generate a standardized facial region image.

[0022] In practical implementation, when performing facial keypoint localization and alignment, a pre-trained facial keypoint detection model needs to be called to process each frame of the original image sequence. This facial keypoint detection model can be a convolutional neural network-based architecture that can identify a set of two-dimensional coordinate points for facial contours, eyes, nose tip, and corners of the mouth. This set of two-dimensional coordinate points typically contains dozens to hundreds of predefined facial feature point coordinates. The rotation angle, scaling ratio, and translation amount of the face in the image plane are calculated based on the two-dimensional coordinate point set. This calculation process involves aligning the detected two-dimensional coordinate point set with the coordinates of a standard frontal facial template. The standard frontal facial template defines the ideal positions of facial keypoints in a standard pose. The optimal transformation parameters are solved by minimizing the error between the two sets of points. The mathematical expression for calculating the rotation angle, scaling ratio, and translation amount can be accomplished by solving an optimization problem, where the coordinates of the standard frontal facial template are represented as a matrix. The detected set of two-dimensional coordinate points is represented as a matrix. The goal is to find a similarity transformation such that the transformed... as close as possible This transformation can be described by the following relation: in, It is The matrix represents the detected... Two-dimensional coordinates of facial key points It is The matrix represents the coordinates of the corresponding standard frontal facial template. It is a scalar scaling factor. It is The rotation matrix, It is The translation vector. This equation requires that every corresponding element of the matrices on both sides of the equation be equal, thus solving for the transformation parameters. , and .

[0023] Optionally, based on the calculated rotation angle, scaling factor, and translation amount, an affine transformation is performed on the original image. This affine transformation uses the calculated rotation matrix, scaling factor, and translation vector to construct a 2x3 transformation matrix, mapping the coordinates of each pixel in the original image to a new image plane. This corrects the facial region to a predefined standard pose and size, typically referring to a face facing forward with eyes at a horizontal position. The corrected facial region is then cropped and uniformly scaled to a fixed resolution to generate a standardized facial region image. This cropping operation is based on a fixed rectangular region defined by a standard frontal facial template, covering the entire facial area from the hairline to the jawline. The cropped image is then scaled to a uniform pixel size, such as 224x224 pixels, using bilinear interpolation to generate the standardized facial region image.

[0024] In one embodiment of the present invention, see [reference] Figure 2In a standardized facial region image, the bounding boxes for both eyes are determined based on the positions of facial key points. Pixel regions within the bounding boxes corresponding to the left and right eyes are extracted to obtain the original images for the left and right eyes. The original left and right eye images are then processed for grayscale conversion, contrast enhancement, and size normalization to generate sub-region images of the eyes. The 3D facial morphology model parameters output by the facial key point detection model are matched with a set of 2D coordinate points. By solving the perspective projection equation, head pose parameters are estimated, including the yaw, pitch, and roll angles of the head relative to the camera coordinate system. The original left and right eye images are input into a convolutional neural network branch with shared weights. Through multiple convolution and pooling operations, eye appearance features are extracted. Based on the coordinates of the corners of the eyes in the facial key points, the distance between the eyeball center, the distance between the inner and outer eye points, and the eyelid curvature geometry are calculated. These parameters are then used to construct a feature vector, which is encoded through a fully connected network to obtain the geometric structural features of the eyes. The head pose parameter vector is input into another independent fully connected network for nonlinear transformation and feature upsizing to obtain higher-level head pose features. In practice, when extracting eye sub-region images from standardized facial region images, the bounding boxes for both eyes need to be determined based on the positions of facial keypoints in the standardized facial region images. The positions of the facial keypoints are provided by a pre-trained facial keypoint detection model. The bounding boxes for both eyes are typically expanded based on the coordinates of the keypoints at the corners of the left and right eyes to fully encompass the eyelids, irises, and part of the sclera. Pixel regions within the corresponding bounding boxes of the left and right eyes are cropped to obtain the original images for the left and right eyes. These original images are color or grayscale image patches, and the cropping operation ensures that the image patches contain complete structural information of the eyes.

[0025] In some embodiments, the original images of the left and right eyes are subjected to grayscale conversion, contrast enhancement, and size normalization to generate an eye sub-region image. Grayscale conversion converts the color image into a single-channel grayscale image to reduce computational complexity. Contrast enhancement improves the visibility of details in the eye region of the image through histogram equalization. Size normalization scales the image blocks to a uniform size, such as 36 pixels multiplied by 60 pixels, to meet the input requirements of the subsequent convolutional neural network. It can be understood that the 3D facial morphology model parameters output by the facial keypoint detection model are matched with the set of 2D coordinate points. By solving the perspective projection equation, the head pose parameters are estimated. The head pose parameters include the yaw angle, pitch angle, and roll angle of the head relative to the camera coordinate system. The 3D facial morphology model parameters represent the standard coordinates of facial keypoints in 3D space, and the set of 2D coordinate points is the projected coordinates of these keypoints on the image plane. The perspective projection equation describes the mapping relationship from 3D points to 2D points, and its form can be expressed as: in: Represents the coordinates of key points in a two-dimensional image coordinate system. This represents the coordinates of the corresponding facial key points in the three-dimensional head coordinate system. It is a known camera intrinsic parameter matrix. It is a 3x3 rotation matrix that encodes the yaw, pitch, and roll angles of the nose. It is a 3x1 translation vector. (Symbol) This indicates equal proportions and is used to handle homogeneous coordinates. The rotation matrix can be solved by minimizing the reprojection error, i.e., calculating the difference between the coordinates of the 3D facial morphology model parameter points projected onto the image plane and the detected set of 2D coordinate points. Translation vector From this, the yaw angle, pitch angle, and roll angle can be further decomposed.

[0026] Optionally, the original images of the left and right eyes are input into a convolutional neural network branch with shared weights. Through multiple convolutional and pooling operations, eye appearance features are extracted. The shared weights mean that the left and right eye images are processed using the same network structure and parameters, ensuring consistency in feature extraction. The convolutional layers use small-sized filters to extract local texture features, and the pooling layers reduce the spatial dimension of the feature maps and enhance feature invariance. In some embodiments, based on the coordinates of the corners of the eyes in facial keypoints, the eyeball center, the distance between the inner and outer corners of the eyes, and the eyelid curvature geometry are calculated. These coordinates are then used to construct a feature vector, which is encoded through a fully connected network to obtain the eye geometric structure features. The eyeball center coordinates can be estimated by the average of the inner and outer corner coordinates. The distance between the inner and outer corners of the eyes is the Euclidean distance between the inner and outer corners of the eyes. The eyelid curvature can be represented by the quadratic curve parameters obtained by fitting the eyelid keypoints. The head pose parameter vector is input into another independent fully connected network for nonlinear transformation and feature upsizing to obtain higher-level head pose features. The head pose parameter vector is a three-dimensional vector containing yaw, pitch, and roll angles. The independent fully connected network consists of multiple linear layers and nonlinear activation functions, mapping the low-dimensional head pose parameters to a high-dimensional feature space to capture the complex influence of head pose on the gaze direction. Optionally, when calculating eye geometry features, the feature vector can also include the offset of the pupil center relative to the corner of the eye. The pupil center offset is obtained through additional image processing steps or an appearance-based regression network to enrich the expressive power of the geometric features.

[0027] In one embodiment of the present invention, eye appearance features, eye geometric features, and head pose features are mapped to the same feature dimension space. The mapping operation is performed through three independent linear projection layers, each corresponding to a feature type and outputting a feature vector of the same dimension. An attention mechanism is used to calculate the correlation weights among eye appearance features, eye geometric features, and head pose features. The core of the attention mechanism is to evaluate the importance of different features to the current gaze estimation task.

[0028] In some embodiments, eye appearance features, eye geometric features, and head pose features are linearly projected onto the query space, key space, and value space, respectively, to generate query vectors, key vectors, and value vectors. Linear projection is achieved by multiplying by a learnable weight matrix, with each feature type having an independent weight matrix for generating the query vector, key vector, and value vector. A first dot product is calculated between the query vector corresponding to the eye appearance features and the key vector corresponding to the eye geometric features, and this first dot product is scaled by dividing by a factor related to the vector dimension. A second dot product is calculated between the query vector corresponding to the eye appearance features and the key vector corresponding to the head pose features, and this second dot product is scaled using the same scaling factor as the first dot product. The calculation of the cross-correlation weights can be described by the following relationship: in, Indicates the first A query vector for each feature. Indicates the first The key vector of each feature. Indicates the first Key vectors of each feature It is a fixed dimension of the key vector. Indicates the first The first feature and the second The final weights after processing the unnormalized correlation scores among the features using the Softmax function. It is the total number of features involved in the calculation, in this scenario. Dot product operation corresponds to eye appearance features, eye geometric structure features, and head posture features, respectively. Represents the vector dot product. It is an exponential function.

[0029] The scaled first and second dot products are input into a Softmax function for normalization, generating a first attention weight between eye appearance features and eye geometric features, and a second attention weight between eye appearance features and head pose features. The Softmax function ensures that the sum of all weights is 1. The value vector corresponding to the eye geometric features is weighted according to the first attention weight, and the value vector corresponding to the head pose features is weighted according to the second attention weight. The two weighted results are then added to the value vector corresponding to the eye appearance features to generate a weighted context feature vector. The weighting operation is an element-wise multiplication of scalar weights with the value vectors. This weighted context feature vector is then input again into a linear projection layer and the Softmax function to calculate the cross-correlation weights between eye appearance features, eye geometric features, and head pose features. This linear projection layer transforms the context feature vector to the same dimension as the number of features. The Softmax function outputs a probability distribution as the final cross-correlation weights. Based on the cross-correlation weights, the eye appearance features, eye geometric structure features, and head pose features are weighted and summed to obtain a preliminary fused feature. The weighted sum is obtained by multiplying each feature vector by its corresponding cross-correlation weight and then adding them together. The preliminary fused feature is then passed through a bottleneck layer for feature compression and redundancy removal, outputting a compact and information-rich comprehensive gaze feature identifier. The bottleneck layer is a fully connected layer with fewer neurons than the input dimension. When generating the query vector, key vector, and value vector, the bias term in the linear projection layer can be omitted to simplify computation. In some embodiments, the factor used for scaling can be the square root of the key vector dimension. The attention mechanism can process all feature pairs in parallel to improve computational efficiency. A dropout layer can be connected after the bottleneck layer to randomly disable some neurons during the training phase to prevent overfitting.

[0030] See Figure 3This study compares the performance of four fusion methods in three core metrics: estimation accuracy, inference time, and model parameter count. Estimation accuracy (%): The simple concatenation method achieves 82.50%, the weighted summation method improves to 86.80%, the attention fusion method further improves to 94.20%, and the "attention + bottleneck layer" method reaches the highest value of 96.70%. This indicates that the attention mechanism can effectively capture the cross-correlation between multiple features, while the feature compression and redundancy removal of the bottleneck layer further optimizes the effectiveness of feature representation. Inference time (ms): The inference time of the four methods ranges from 0.12 to 0.16 ms. The simple concatenation method has the shortest time (0.12 ms), the attention fusion method has the longest time (0.16 ms), and the "attention + bottleneck layer" method has a time of 0.14 ms, reflecting the balancing effect of the bottleneck layer on computational efficiency while maintaining high accuracy. Model parameter count (in ten thousands): The simple concatenation and weighted summation methods both have 704,000 parameters, the attention fusion method has 712,000 parameters, while the "attention + bottleneck layer" method has a sharp drop in parameter count to 384,000. This is because the bottleneck layer achieves model lightweighting through dimensionality reduction, effectively controlling the growth of parameter count.

[0031] In one embodiment of the invention, the comprehensive gaze feature identifier of the current frame is concatenated with the hidden state vector of the previous frame from the recurrent neural network. The concatenated vector is then input into the internal unit of the recurrent neural network to update the current state vector and output vector of the recurrent neural network. The output vector represents the current gaze feature incorporating historical temporal information, which includes information on the speed and direction of gaze movement. The current gaze feature output by the recurrent neural network is then input into a fully connected regression layer. The fully connected regression layer outputs a three-dimensional vector, which represents the offset angle of the gaze direction relative to the front of the head in the head coordinate system. Based on the offset angle and the known definition of the head coordinate system, the initial gaze direction vector in the head coordinate system is calculated.

[0032] In practice, the comprehensive gaze feature identifier of the current frame is concatenated with the hidden state vector of the previous frame in the recurrent neural network (RNN). The comprehensive gaze feature identifier is a fixed-dimensional feature vector, while the hidden state vector of the previous frame is the network's internal memory retained after processing the previous frame. The concatenation operation connects the two vectors along the feature dimension to form an extended feature vector. The concatenated vector is then input into the internal units of the RNN to update the current state vector and output vector. The internal units of the RNN can be long short-term memory units or gated recurrent units, which control the forgetting and updating of information through gating mechanisms. The output vector represents the current gaze feature, which incorporates historical temporal information and includes information on the speed and direction of gaze movement.

[0033] In some embodiments, the internal units of the recurrent neural network employ a long short-term memory structure. Its computation process involves input gates, forget gates, output gates, and cell states, and the update process follows a specific sequence relationship. It can be understood that the gaze movement speed and direction trend information contained in the current gaze features can be indirectly reflected by analyzing the changes in the internal state vectors of the recurrent neural network over multiple consecutive frames. This dynamic information helps smooth noise in single-frame estimations and predict short-term gaze movements. Optionally, the recurrent neural network can be designed as a multi-layer structure to capture more complex temporal dependencies. A multi-layer structure means that multiple recurrent neural network units are stacked, with the output of the lower layer serving as the input of the upper layer. The process of calculating the gaze angular velocity based on the current gaze features and the relevant state of the previous frame can be described by the following relationship: in, Indicates a point in time The instantaneous line-of-sight direction angle vector decoded from the current line-of-sight features includes pitch and yaw components. Indicates a point in time The corresponding direction angle vector, It is a fixed time interval between two consecutive frames. The calculated line-of-sight angular velocity vector reflects the rate of change of the line-of-sight direction in the pitch and yaw dimensions. This information is encoded in the updated recurrent neural network state vector.

[0034] The current gaze characteristics output from the recurrent neural network are input into a fully connected regression layer. This fully connected regression layer consists of one or more linear transformation layers, ultimately outputting a three-dimensional vector. This three-dimensional vector represents the offset angle of the gaze direction relative to the front of the head in the head coordinate system. The offset angle typically includes two components: pitch offset and yaw offset. The third component can be set to a fixed zero value or used to represent the confidence level of the estimate. Based on the offset angle and the known definition of the head coordinate system, the initial gaze direction vector in the head coordinate system is calculated. The head coordinate system is typically defined with the center of the head as the origin, the positive direction as the nose tip, the right side as the positive X-axis, the top as the positive Y-axis, and the front as the positive Z-axis. For the parameter configuration of the fully connected regression layer, refer to Table 1. Table 1: Parameters of Fully Connected Regression Layer Network Structure In some embodiments, when calculating the initial line-of-sight direction vector from the offset angle, a transformation from a spherical coordinate system to a three-dimensional Cartesian coordinate system is used. Assuming the line-of-sight vector directly in front of the head is (0,0,1), rotation is performed based on pitch and yaw offsets. It can be understood that the fully connected regression layer uses a mean squared error loss function for optimization during the training phase. This loss function calculates the difference between the offset angle predicted by the network and the offset angle actually labeled. Optionally, the recurrent neural network requires an initial hidden state vector at the beginning of the sequence. This initial hidden state vector can be set to an all-zero vector or initialized using a learnable parameter.

[0035] See Figure 4 In the process of processing gaze temporal features in a multi-layer LSTM network, the dynamic changes in the activation of hidden states reveal the network's encoding pattern of gaze information in different frames. Specifically, the activation of LSTM layer 1 (blue curve) generally shows an upward trend followed by a downward trend. Within the frame number range of 0–40, it gradually increases from 0.55 to 0.91, reflecting the network's feature extraction and memory enhancement of gaze movement in the initial stage. Within the range of 40–100, it gradually decreases from 0.91 to 0.30, indicating that this layer compresses and removes redundancy from the gaze features in the subsequent stable stage. The activation of LSTM layer 2 (red curve) shows a dynamic trend opposite to that of layer 1. Within the range of 0–40, it decreases from 0.98 to 0.20, reflecting the filtering and abstraction of the initial complex gaze dynamics by the upper layer network. Within the range of 40–100, it rises again from 0.20 to 0.92, reflecting its reconstruction of higher-order features of gaze movement trends in the later stages. The two trend fitting curves (dashed lines) further validate this reverse complementary activation mode, indicating that the multi-layer LSTM achieves multi-scale capture of gaze temporal features through a hierarchical memory mechanism—the lower layer network is responsible for encoding short-term dynamic details, while the upper layer network is responsible for abstracting and predicting long-term trends. This structure helps to improve the smoothness and robustness of gaze estimation.

[0036] In one embodiment of the present invention, a rotation transformation matrix from the head coordinate system to the camera coordinate system is constructed using head pose parameters. A projection transformation relationship from the camera coordinate system to the image coordinate system is constructed using the camera intrinsic parameter matrix. The initial gaze direction vector in the head coordinate system is transformed to the camera coordinate system using the rotation transformation matrix, resulting in the gaze direction vector in the camera coordinate system. The gaze direction vector in the camera coordinate system is back-projected to obtain a three-dimensional spatial ray originating from the camera optical center and passing through the corresponding pixel on the image plane. The intersection of this three-dimensional spatial ray with the scene geometry surface in the three-dimensional environment model is calculated, and the intersection point is the final gaze point coordinate of the user in the world coordinate system. Color and depth images of the interactive scene are acquired using a depth camera array. Registration and point cloud fusion processing are performed on the color and depth images to generate dense three-dimensional point cloud data of the scene. Plane detection and surface reconstruction are performed on the dense three-dimensional point cloud data to identify the three-dimensional geometric surfaces of the main objects in the scene. The identified three-dimensional geometric surfaces are meshed and assigned unique object identifiers to construct a three-dimensional environment model containing scene geometric information. When the 3D spatial ray has no intersection with the 3D environment model, the nearest virtual interaction plane is found in the 3D environment model along the direction of the 3D spatial ray. The coordinates of the intersection point between the 3D spatial ray and the virtual interaction plane are calculated, and these coordinates are used as candidate final viewpoint coordinates. The uncertainty markers of the candidate final viewpoint coordinates are recorded for subsequent differential processing by the human-computer interaction logic.

[0037] In practice, a rotation transformation matrix from the head coordinate system to the camera coordinate system is constructed using head pose parameters. These head pose parameters include the yaw, pitch, and roll angles of the head relative to the camera coordinate system. These three Euler angles are converted into a 3x3 rotation matrix using a standard rotation sequence. The projection transformation relationship from the camera coordinate system to the image coordinate system is constructed using the camera intrinsic parameter matrix. It is a 3x3 upper triangular matrix containing the camera's focal length and principal point coordinates. The initial gaze direction vector in the head coordinate system is transformed to the camera coordinate system using a rotation transformation matrix, resulting in the gaze direction vector in the camera coordinate system. This transformation can be expressed as transforming the vector in the head coordinate system... Left multiplication by rotation matrix ,Right now ,in This represents the unit direction vector in the camera coordinate system. Back-projecting the line-of-sight vector in the camera coordinate system yields a 3D spatial ray originating from the camera's optical center and passing through the corresponding pixel on the image plane. The back-projection operation depends on the camera intrinsic parameter matrix. The inverse transformation. In specific implementations, the spatial ray obtained by back projection can be represented in the following parametric form: in, This represents the origin of the camera coordinate system, i.e., the position of the camera's optical center, which is usually located at coordinates (0,0,0). Represents the unit view direction vector in the camera coordinate system. , It is a non-negative real scalar parameter representing the distance from the optical center along the ray direction to a point in space. Represents the corresponding parameters on the ray. The three-dimensional coordinates of the point.

[0038] The process involves calculating the intersection of a 3D spatial ray with the geometric surfaces of the scene in the 3D environment model. The intersection point represents the coordinates of the user's final viewpoint in the world coordinate system. This intersection calculation traverses all geometric primitives in the 3D environment model, calculating the intersection of the ray with the plane containing each primitive and determining whether the intersection point lies within the primitive's boundary. In some embodiments, the 3D environment model construction steps include acquiring color and depth images of the interactive scene using a depth camera array. The depth camera array consists of multiple paired color cameras and depth sensors, simultaneously capturing data from different perspectives. The color and depth images are then registered and fused to generate dense 3D point cloud data of the scene. Registration unifies the depth images from different perspectives to the same world coordinate system, while fusion averages the point clouds in overlapping areas to reduce noise. Optionally, the dense 3D point cloud data undergoes plane detection and surface reconstruction to identify the 3D geometric surfaces of major objects in the scene. Plane detection can use a random sampling consistency algorithm to fit planar regions in the point cloud, and surface reconstruction can use Poisson reconstruction or triangulation methods to generate continuous mesh surfaces from the point cloud. The identified 3D geometric surfaces are meshed and assigned unique object identifiers to construct a 3D environment model containing scene geometry information. Meshing converts curved surfaces into a mesh composed of triangular or quadrilateral patches. Object identifiers are used to distinguish different objects in the interaction logic. When a 3D spatial ray does not intersect with the 3D environment model, the nearest virtual interaction plane is searched within the 3D environment model along the direction of the 3D spatial ray. The virtual interaction plane is a pre-defined abstract plane used for interaction, such as the plane of the display screen or the desktop plane. The coordinates of the intersection point between the 3D spatial ray and the virtual interaction plane are calculated and used as candidate final viewpoint coordinates. The virtual interaction plane is defined by plane equations, and calculating the intersection point involves solving a system of equations combining the ray parametric equations and the plane equations. The uncertainty marker of the candidate final viewpoint coordinates is recorded for subsequent differentiation processing in the human-computer interaction logic. The uncertainty marker can be a Boolean flag or a floating-point number representing the estimated confidence level.

[0039] In some embodiments, finding the nearest virtual interaction plane can be accomplished by calculating the distance from the ray origin (camera optical center) to each preset virtual plane, and then selecting the plane with the smallest distance for intersection calculation. It is understood that the virtual interaction plane can be manually defined by the user during the 3D environment model construction phase or set through automatic analysis of the scene structure. Optionally, the intersection calculation with the virtual interaction plane can ignore the determination of whether the intersection point lies within the finite boundaries of the plane; the default plane extends infinitely.

[0040] See Figure 5 The Pearson correlation coefficients among the three-dimensional coordinates (X, Y, Z) of the gaze point are presented. Specifically, the correlation coefficient between each coordinate component and itself is 1.000, showing a perfect positive correlation, which is consistent with the basic properties of a correlation coefficient matrix. The correlation coefficient between the X and Y coordinates is 0.072, showing a very weak positive correlation, indicating that the linear correlation between the two is extremely low; the correlation coefficients between the X and Z coordinates are -0.088, and the correlation coefficients between the Y and Z coordinates are -0.185, both showing very weak negative correlations, indicating that there is no significant linear dependence among the three. This weak correlation characteristic reflects that the distribution of the gaze point in three-dimensional space has good independence, and also verifies that the outputs of the coordinate components of this gaze estimation method do not show obvious collinearity problems during the three-dimensional space mapping process.

[0041] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A real-time gaze estimation method for human-computer interaction based on deep learning, characterized in that, The method includes: The image acquisition device continuously captures multiple consecutive images containing the user's facial area to form the original image sequence; Facial key point localization and alignment are performed on each frame of the original image sequence to generate a standardized facial region image; Extract the eye sub-region image and head pose parameters from the standardized facial region image; The eye sub-region image and the head pose parameters are input into a multi-stream feature extraction network to extract eye appearance features, eye geometric structure features and head pose features in parallel. The eye appearance features, eye geometric features, and head posture features are dynamically fused to generate a comprehensive gaze feature identifier; The integrated gaze feature identifier is input into a recurrent neural network, which captures the dynamic change information of gaze between adjacent frames. Based on the output of the recurrent neural network, the initial gaze direction vector is calculated. Based on the current 3D environment model, the initial gaze direction vector is spatially mapped and calibrated to obtain the final gaze point coordinates of the user in the world coordinate system. The coordinates of the final line of sight are transmitted to the human-computer interaction interface in real time to drive the interaction response.

2. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 1, characterized in that, The step of performing facial key point localization and alignment processing on each frame of the original image sequence to generate a standardized facial region image includes: The pre-trained facial key point detection model is invoked to process each frame of the original image sequence to identify the set of two-dimensional coordinate points of facial contours, eyes, nose tip and corner of mouth. Calculate the rotation angle, scaling ratio, and translation amount of the face in the image plane based on the set of two-dimensional coordinate points; Based on the calculated rotation angle, scaling ratio, and translation amount, an affine transformation is performed on the original image to correct the facial region to a predefined standard pose and standard size. The facial region in the corrected image is cropped out and uniformly scaled to a fixed resolution to generate the standardized facial region image.

3. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 2, characterized in that, The step of extracting the eye sub-region image and head pose parameters from the standardized facial region image includes: In the standardized facial region image, the bounding boxes of the eyes are determined based on the positions of facial key points; The pixel regions within the bounding boxes corresponding to the left and right eyes are respectively cropped to obtain the original image of the left eye and the original image of the right eye. The original images of the left eye and the right eye are subjected to grayscale conversion, contrast enhancement, and size normalization to generate the eye sub-region image; The three-dimensional facial morphology model parameters output by the facial key point detection model are matched with the two-dimensional coordinate point set. By solving the perspective projection equation, the head posture parameters are estimated. The head posture parameters include the yaw angle, pitch angle and roll angle of the head relative to the camera coordinate system.

4. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 3, characterized in that, The step of inputting the eye sub-region image and the head pose parameters into a multi-stream feature extraction network to extract eye appearance features, eye geometric structure features, and head pose features in parallel includes: The original images of the left eye and the right eye are respectively input into a convolutional neural network branch with shared weights. Through multiple convolution and pooling operations, the eye appearance features are extracted. Based on the coordinates of the corners of the eyes in the facial key points, the center of the eyeball, the distance between the inner and outer eye points, and the geometric quantity of eyelid curvature are calculated. The center of the eyeball, the distance between the inner and outer eye points, and the geometric quantity of eyelid curvature are then used to form a feature vector, which is encoded through a fully connected network to obtain the geometric structural features of the eye. The head pose parameter vector is input into another independent fully connected network for nonlinear transformation and feature dimensionality upscaling to obtain higher-level head pose features.

5. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 4, characterized in that, The step of dynamically fusing the eye appearance features, the eye geometric structure features, and the head posture features to generate a comprehensive gaze feature identifier includes: The eye appearance features, the eye geometric structure features, and the head posture features are mapped to the same feature dimension space; An attention mechanism is used to calculate the cross-correlation weights among the eye appearance features, the eye geometric structure features, and the head posture features; Based on the cross-correlation weights, the eye appearance features, the eye geometric structure features, and the head posture features are weighted and summed to obtain a preliminary fusion feature; The preliminary fused features are compressed and deredundanted through a bottleneck layer to output a compact and information-rich integrated gaze feature identifier.

6. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 5, characterized in that, The step of inputting the integrated gaze feature identifier into a recurrent neural network and using the recurrent neural network to capture gaze dynamic change information between adjacent frames includes: The integrated gaze feature identifier of the current frame is concatenated with the state vector of the previous frame hidden by the recurrent neural network. The concatenated vector is input into the internal unit of the recurrent neural network to update the current state vector and output vector of the recurrent neural network; The output vector represents the current gaze feature that incorporates historical time-series information, and the current gaze feature includes information on the speed and direction of gaze movement.

7. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 6, characterized in that, The step of calculating the initial gaze direction vector based on the output of the recurrent neural network includes: The current gaze features output by the recurrent neural network are input into a fully connected regression layer; The fully connected regression layer outputs a three-dimensional vector, which represents the offset angle of the line of sight relative to the front of the head in the head coordinate system. Based on the offset angle and the known definition of the head coordinate system, the initial gaze direction vector in the head coordinate system is calculated.

8. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 7, characterized in that, The process of combining the current scene's 3D environment model with spatial mapping and calibration of the initial gaze direction vector yields the user's final gaze point coordinates in the world coordinate system, including: Using the head pose parameters, a rotation transformation matrix from the head coordinate system to the camera coordinate system is constructed; Using the camera intrinsic parameter matrix, construct the projection transformation relationship from the camera coordinate system to the image coordinate system; The initial gaze direction vector in the head coordinate system is transformed to the camera coordinate system using the rotation transformation matrix to obtain the gaze direction vector in the camera coordinate system. By back-projecting the line-of-sight vector in the camera coordinate system, a three-dimensional spatial ray is obtained that originates from the optical center of the camera and passes through the corresponding pixel on the image plane. The intersection of the three-dimensional spatial ray with the scene geometry surface in the three-dimensional environment model is calculated, and the intersection point is the coordinate of the user's final line of sight in the world coordinate system. The steps for constructing the three-dimensional environment model include: Color and depth images of the interactive scene are acquired using a depth camera array; The color image and depth image are registered and fused together to generate dense 3D point cloud data of the scene. Plane detection and surface reconstruction are performed on the dense 3D point cloud data to identify the 3D geometric surfaces of the main objects in the scene; The identified 3D geometric surfaces are meshed and assigned unique object identifiers to construct the 3D environment model containing scene geometric information.

9. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 8, characterized in that, When the three-dimensional spatial ray has no intersection with the three-dimensional environment model, the method further includes: Along the direction of the three-dimensional spatial ray, find the nearest virtual interaction plane in the three-dimensional environment model; Calculate the coordinates of the intersection point between the three-dimensional spatial ray and the virtual interactive plane, and use the intersection point coordinates as the alternative final line-of-sight landing point coordinates; Record the uncertainty markers of the final gaze point coordinates of the candidate points for subsequent human-computer interaction logic to perform differentiated processing.

10. The real-time gaze estimation method for human-computer interaction based on deep learning according to claim 5, characterized in that, The step of calculating the cross-correlation weights among the eye appearance features, the eye geometric structure features, and the head posture features using an attention mechanism includes: The eye appearance features, eye geometric structure features, and head posture features are linearly projected into the query space, key space, and value space, respectively, to generate query vector, key vector, and value vector; Calculate the first dot product between the query vector corresponding to the eye appearance feature and the key vector corresponding to the eye geometric structure feature, and scale the first dot product. Calculate the second dot product between the query vector corresponding to the eye appearance feature and the key vector corresponding to the head pose feature, and then scale the second dot product. The scaled first dot product and the second dot product are input into the Softmax function for normalization to generate the first attention weight between the eye appearance features and the eye geometric structure features, and the second attention weight between the eye appearance features and the head posture features. The value vector corresponding to the eye geometric structure feature is weighted according to the first attention weight, and the value vector corresponding to the head pose feature is weighted according to the second attention weight. The two weighted results are added to the value vector corresponding to the eye appearance feature to generate a weighted context feature vector. The weighted context feature vector is then input again into the linear projection layer and the Softmax function to calculate the cross-correlation weights between the eye appearance features, the eye geometric structure features, and the head pose features.

Images