Gaze tracking method and system based on fusion of dual-view geometry and deep learning

Abstract

A gaze tracking method and system based on the fusion of dual-view geometry and deep learning. This invention relates to the field of driver gaze tracking technology, specifically a gaze tracking method and system based on the fusion of dual-view geometry and deep learning. The invention innovatively introduces a geometric consistency constraint mechanism, effectively reducing the dependence on high-precision ground truth data and significantly reducing system construction costs. Simultaneously, through robust optimization strategies, it greatly improves the algorithm's adaptability to complex lighting changes and severely occluded scenarios. To further enhance stability under dynamic conditions, this invention integrates head pose calculation and eye gaze modeling technologies, achieving accurate tracking of the gaze target. Furthermore, the system employs a dual-camera complementary perspective design and is equipped with an intelligent degradation processing strategy, achieving an optimal balance between complexity and performance while ensuring reliability, demonstrating outstanding engineering practical value.

Description

Technical Field

[0001] This invention relates to the field of driver gaze tracking technology, specifically to a gaze tracking method and system based on the fusion of dual-view geometry and deep learning. Background Technology

[0002] With the rapid development of intelligent driving technology, the demand for real-time perception of driver status has significantly increased. Especially in the field of driving safety, driver distraction or lack of attention is widely considered a major factor contributing to traffic risks. Therefore, regulations and industry standards in various countries are continuously promoting the integration of driver status monitoring capabilities into vehicles. "Driver's gaze or direction of gaze" is a crucial indicator for measuring driver attention, distracted behavior, and takeover ability. Determining whether the driver is continuously looking at the road ahead, rearview mirror, instrument panel, or distracting targets inside the vehicle provides key input for vehicle human-machine interaction and safety strategies. In real-world cockpit environments, driver gaze tracking systems often need to operate stably for extended periods under complex lighting conditions, including strong daylight, low nightlight, backlighting, and reflections. They must also adapt to challenges such as significant head movements, glare or obstruction from glasses, blinking, and image blurring caused by vehicle vibrations. Therefore, gaze tracking solutions for real-world driving require a balance between robustness, calibrability, deployability, and generalization across different user groups and environments.

[0003] In the field of driver status monitoring technology, existing driver detection systems often employ monocular camera solutions at the hardware level, with cameras typically positioned on the left A-pillar, steering wheel / instrument panel, or rearview mirror area. When the driver's head posture exceeds the effective detection range of the camera, the confidence level of the gaze tracking system's output will significantly decrease or even fail. At the algorithm level, current methods for achieving gaze tracking can be mainly categorized as follows: one is the traditional geometric method based on pupil-corneal reflection. This type of method mainly extracts the pupil center and one or more corneal reflection points, using the relative geometric relationship between the pupil and the reflection points to estimate the gaze or fixation point; another type of method is the end-to-end deep learning regression method based on appearance. This type of method treats eye movement estimation as a visual regression task, inputting face / eye data and directly regressing the gaze from the face / eye region. To achieve real-time gaze tracking, some studies have introduced pupil-corneal reflection geometry in deep learning models to achieve a balance between speed and accuracy. These methods typically introduce three-dimensional geometric priors, such as estimating head pose first and then regressing eye / gaze in head coordinates, or regressing denser intermediate representations to improve model robustness.

[0004] However, in driver eye tracking applications aimed at "usability in real driving scenarios", the existing data recognition-based technical solutions generally have the following shortcomings: existing deep learning-based eye tracking methods mostly rely on large-scale labeled data for training, while high-precision eye tracking ground truth usually requires the use of professional eye trackers or complex calibration processes, resulting in high data acquisition costs and complex processes. At the same time, it is difficult to continuously obtain high-quality eye tracking ground truth in practical applications, which limits the training and optimization of the model in real driving scenarios. This problem mainly exists in eye estimation methods based on neural network pattern recognition. Existing methods, such as those relying on pupil and corneal reflection features, have high requirements for data quality. Under conditions of strong light, weak light, eyeglass reflection, or partial occlusion, key features are prone to failure, leading to jumps or interruptions in gaze estimation. This problem mainly exists in geometric solution methods based on eye feature extraction. In real driving, the driver's head posture changes frequently, and vehicle vibration can cause data jitter and blurring. Existing methods often fail to effectively decouple the effects of head movement and eye rotation, resulting in significant noise in gaze output as posture changes occur, affecting continuity and stability. Summary of the Invention

[0005] To address the shortcomings of existing technologies, such as high data annotation costs, easy feature failure under complex lighting conditions, insufficient head-eye motion decoupling, and limited single-view observation range, which make it difficult to meet the high-precision and high-stability monitoring requirements of real-world driving scenarios, this invention aims to propose a gaze tracking method and system based on the fusion of dual-view geometry and deep learning. This invention reduces the cost of ground truth acquisition through geometric consistency constraints, enhances robustness under complex lighting and occlusion conditions, improves dynamic stability by combining head pose calculation and eye gaze modeling, and balances system complexity and reliability by employing complementary perspectives from dual cameras and degradation processing strategies.

[0006] The method includes the following steps: S1. Cameras are installed on the left and right A-pillars of the vehicle to acquire real-time images from the left and right perspectives. S2. Synchronize the left-view image and the right-view image in time to construct a dual-view image pair; S3. Extract key point coordinates from the dual-view image pair to obtain the face region, facial key point coordinates, and eye key point coordinates. The face region is preprocessed based on the coordinates of the facial key points and the eye key points to obtain dual-view facial images and dual-view eye images. S4. Establish a correspondence between the facial key point coordinates and the preset three-dimensional face model, and solve the head pose parameters through the PnP algorithm; S5. Construct a dual-branch feature extraction network to extract dual-view features based on dual-view facial images and dual-view eye images. , , For view index value, Left-hand perspective From the right perspective; S6. The head posture parameters are processed by the posture encoder and then compared with... The images are stitched together to obtain the stitched dual-view features. ; S7 After processing by a deep learning-based gaze regression head, dual-view gaze vectors are obtained. Dual-view geometric line-of-sight vector Dual-view gating coefficient and confidence of dual-viewpoint gaze vector ; S8, according to , and Calculate the final dual-view gaze vector ; S9. Constructing Quality Scores Based on the quality score, the system will perform dual-view fusion output or single-view downgraded output, suppressed output, and degraded output.

[0007] Furthermore, the process of constructing the dual-view image pair is as follows: setting a time synchronization threshold. Select timestamp interpolation less than A dual-view image pair is constructed from the left-view and right-view images.

[0008] Furthermore, the key point coordinate extraction process is as follows: the dual-view image pairs are processed sequentially by the face detection algorithm, the key point tracking algorithm, and the Kalman filter to obtain the face region, facial key point coordinates, and eye key point coordinates. The preprocessing includes: cropping, rotating, translating, and scaling; The head attitude parameters include: yaw angle, pitch angle, and roll angle.

[0009] Furthermore, the dual-branch feature extraction network includes: a left-view feature extraction branch and a right-view feature extraction branch; The left-view feature extraction branch and the right-view feature extraction branch share the same feature extraction network, which passes through the following layers from input to output: feature extraction convolutional layer, feature extraction batch normalization layer and feature extraction nonlinear activation layer; Parameters are shared between dual-view facial images and dual-view eye images.

[0010] Furthermore, the attitude encoder passes through the following layers sequentially from input to output: the first fully connected layer of attitude encoding, the batch normalization layer of attitude encoding, the nonlinear activation layer of attitude encoding, and the second fully connected layer of attitude encoding.

[0011] Furthermore, the gaze regression head includes a direct regression branch, a geometric branch, a gated fusion branch, and a confidence estimation branch; The direct regression branch is used to output the dual-view gaze vector, and the calculation formula for the direct regression branch is: ,in, This represents the weight matrix of the direct regression branch. This represents the bias vector of the direct regression branch. Represents the normalization function; The geometric branch is used to output the dual-view geometric line-of-sight vector, and the formula for calculating the geometric branch is: ,in, Indicates the first From the camera's optical center, passing through the central pixel coordinates of the iris, in each perspective... The direction vector of the back-projected ray; Indicates the first The three-dimensional coordinates of the eyeball center in the camera coordinate system from each perspective; The gated fusion branch is used to output the dual-view gating coefficients, and the calculation formula for the gated fusion branch is: , This represents the sigmoid activation function. This represents the transpose of the gating fusion branch weight vector. The bias scalar represents the gating fusion branch; The confidence estimation branch is used to output the confidence score of the dual-view gaze vector. The calculation formula for the confidence estimation branch is: ,in, This represents the weight parameters of the confidence estimation branch. This represents the bias parameter of the confidence estimation branch.

[0012] Furthermore, the final dual-viewpoint line-of-sight vector The formula for calculation is: ; quality score The formula for calculation is: ,in, Indicates the first Image quality metrics from various perspectives Indicates the first Key quality indicators from each perspective Indicates the first Aspect-based usability metrics for pose. An index representing the consistency between the left and right perspectives. Indicates the first Confidence index of gaze estimation results from each perspective , This represents the weighting coefficient of the corresponding indicator. , This represents the index value of the indicator. ; The specific steps for determining whether to perform dual-view fusion output or single-view degradation output, suppress output, or degraded output based on the quality score are as follows: S81, when , ,and When the view is in the correct position, execute the dual-view fusion output; otherwise, execute step S82, where... Indicates the preset quality threshold, This indicates the preset viewpoint consistency threshold. S82, when ,and or ,and When the output is in a single-view downgraded state, execute the output; otherwise, execute step S83, where... Indicates the single-view output threshold; S83, when , ,and , When the output is suppressed, step S84 is executed; otherwise, step S84 is executed. Indicates the inhibition threshold; S84, when and At that time, degenerate output is executed.

[0013] Furthermore, dual-view fusion output or single-view degraded output, suppressed output, and degraded output; The formula for calculating the dual-view fusion output is: ; Indicates the first The fusion weight of each perspective Indicates dual-view output; The formula for calculating the single-view downgraded output is: , The index value represents a single viewpoint. The value is or , Indicates single-view output; The suppressed output means: no output is output in the current frame, and the stable result of the previous frame is maintained; The degenerate output indicates that, based on head pose parameters, the head is oriented approximately towards the line of sight.

[0014] A gaze tracking system based on the fusion of dual-view geometry and deep learning is used to implement the above method. The system includes: an image acquisition module, an image pair construction module, a key point extraction module, a head pose parameter solving module, a dual-view feature extraction module, a feature stitching module, a gaze regression head module, a gaze vector calculation module, and an output module. The image acquisition module has cameras installed at the left and right A-pillars of the vehicle to acquire images from the left and right perspectives in real time. The image pair construction module synchronizes the left-view and right-view images in time to construct dual-view image pairs. The key point extraction module extracts and preprocesses the key point coordinates of the dual-view image pair in sequence to obtain facial key point coordinates, dual-view facial images, and dual-view eye images. The head pose parameter solving module solves the head pose parameters based on the coordinates of the facial key points. The dual-view feature extraction module constructs a dual-branch feature extraction network to extract dual-view features based on dual-view facial images and dual-view eye images; The feature stitching module stitches together the head pose parameters and dual-view features to obtain the stitched dual-view features. The gaze regression head module generates dual-view gaze vectors based on the stitched dual-view features. Dual-view geometric line-of-sight vector Dual-view gating coefficient and confidence of dual-viewpoint gaze vector ; The line-of-sight vector calculation module, based on , and Calculate the line-of-sight vector ; Output module, constructs quality score Based on the quality score, perform dual-view fusion output or single-view downgraded output, suppressed output, and degraded output; The system is trained and calibrated using a progressive strategy, which includes pre-training during stationary periods, fine-tuning during moving periods, and initiation of short calibrations.

[0015] Furthermore, the pre-training of the static segment specifically includes: S10. When the vehicle is stationary, multiple marked targets are displayed in a preset order via the vehicle's display device. And record each marked target Display time and spatial location information; S11. Acquire dual-view images of the driver and determine the corresponding marker targets for each view image based on timestamp matching. ,in, This indicates the three-dimensional coordinates of the target in the camera coordinate system; S12. Obtain the training dataset based on the driver's dual-view images. ; S14, according to The system is trained using the point-to-line-of-sight loss function as a geometric constraint to obtain the first-stage model parameters. ; The point-to-line-of-sight loss function is calculated as follows: ,in, This indicates the coordinates of the driver's eye center in the camera's coordinate system. Represents the square of the norm; The fine-tuning of the motion segment specifically involves: simultaneously acquiring the driver's dual-view images, target display information, and vehicle or camera attitude change information while the vehicle is in motion; and... Based on this, the coordinate transformation relationship corresponding to changes in vehicle or camera posture is introduced, and the system is fine-tuned and trained to obtain the second-stage model parameters. ; The initiation of short calibration specifically involves: setting a predetermined duration of target calibration process before each driving session begins, acquiring the driver's dual-view images and corresponding target information; and based on... Only for , and Update to obtain individualized eye-tracking parameters for the current driver. .

[0016] The beneficial effects of the method described in this invention are as follows: (1) In view of the problem that existing monocular camera solutions are prone to failure in line-of-sight estimation when the driver veers or is obstructed, the present invention arranges near-infrared cameras on the left and right A-pillars of the vehicle to form a dual-view observation structure. Compared with the single-camera solution, it can still obtain at least one side of the face and eye information when the driver's head veers, is partially obstructed or is reflected on one side, thereby reducing the risk of line-of-sight tracking interruption caused by single-view failure, improving the continuity of line-of-sight estimation under complex driving postures, and improving the overall robustness without significantly increasing the system hardware complexity.

[0017] (2) In view of the problem that existing methods fail to effectively decouple the influence of head movement and eye rotation, resulting in significant noise in the gaze output as the posture changes, the method of the present invention calculates the head posture based on facial key points and a preset face model in the gaze estimation process, and performs gaze direction modeling on this basis, so that the gaze estimation is no longer directly affected by the overall head movement, thereby reducing the impact of head posture changes on the gaze regression accuracy, improving the stability of gaze estimation under driving conditions with large posture changes, and reducing the temporal noise introduced by vehicle vibration or the driver's natural movements.

[0018] (3) In view of the problems that existing deep learning eye tracking methods rely on large-scale labeled data, and the high cost and complex process of obtaining high-precision eye tracking ground truth, the system described in this invention constructs geometric consistency constraints between eye tracking rays and targets by displaying marked targets with known spatial positions during the training phase. This constraint is used for training and parameter adjustment of the eye tracking model without relying on dedicated eye-tracking measurement equipment. This simplifies the process of obtaining eye tracking ground truth, reduces system deployment and maintenance costs, improves the repeatability and engineering feasibility of supervision signals, and makes model training and calibration closer to the real driving environment.

[0019] (4) In response to the problem that vehicle vibration causes data jitter and blurring, affecting the continuity and stability of gaze estimation, the system described in this invention adopts a three-stage progressive training strategy during training for the working conditions of "the scene is static and the camera moves with the platform or vehicle body" in driving simulation or real vehicle, so as to improve the robustness of the gaze model under vehicle vibration and dynamic working conditions, reduce the impact of dynamic posture changes on gaze estimation accuracy, and make the gaze tracking model more in line with the usage conditions of real driving scenarios.

[0020] (5) In view of the problem that multi-camera solutions lack effective fusion and degradation processing mechanisms and significantly increase system complexity, this invention evaluates image quality, availability of key features and consistency of multiple perspectives during eye tracking, and triggers corresponding degradation processing strategies when the observation quality deteriorates, so as to avoid sudden changes or failures in eye output, and avoid excessive complexity caused by multi-camera systems while ensuring system stability. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method described in this invention; Figure 2 This is a schematic diagram of the gaze tracking process that integrates dual-view geometry and deep learning as described in this invention. Figure 3 This is a schematic diagram of the system structure described in this invention; Figure 4 This is a schematic diagram of the progressive strategy described in this invention. Detailed Implementation

[0022] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 This embodiment provides a gaze tracking method based on the fusion of dual-view geometry and deep learning. The implementation process of this embodiment is shown in the scenario of "driving simulator + 270° circular display screen + six-degree-of-freedom platform (only the seat / camera posture is changed, the screen is fixed)".

[0024] The implementation environment and hardware setup are as follows: Display device: 270° circular display screen, composed of 210 LED screens, with an overall shape approximately cylindrical; the display screen is fixed in place. Motion platform: The six-degree-of-freedom platform supports the seat and dual near-infrared cameras, allowing the seat or cameras to change their relative posture as the platform moves; Camera placement: A near-infrared monochrome camera is installed on each of the left and right A-pillars of the vehicle, with near-infrared supplementary lighting; both video streams record timestamps and are time-synchronized and paired. Computational Unit: The vehicle-mounted / industrial control computer performs face detection, key point extraction, head pose PnP calculation, gaze regression, quality assessment and degradation processing, and completes training and fine-tuning.

[0025] The method is executed in real time during vehicle movement or simulated driving, continuously outputting the driver's gaze direction or gaze area from dual-view near-infrared images, and maintaining output stability under complex conditions such as occlusion, reflection, lighting changes, and vehicle vibration. A schematic diagram of the gaze tracking process fusion using dual-view geometry and deep learning is shown below. Figure 2 As shown.

[0026] like Figure 1 As shown, the method includes, but is not limited to, the following steps: S1. Near-infrared cameras are installed on the left and right A-pillars of the vehicle to acquire images from the left and right perspectives in real time. The relevant operations in step S1 will be introduced with specific examples: During runtime, the imaging parameters of the camera (in this embodiment, a near-infrared camera is used) and the time synchronization reference of the two cameras are initialized first; the camera's intrinsic and extrinsic parameters, coordinate system parameters, and the 3D model of the face and the corresponding point set are read.

[0027] Both cameras capture data with timestamp information; the frame rate and resolution are set according to the real-time requirements of the vehicle.

[0028] S2. Synchronize the left-view image and the right-view image in time to construct a dual-view image pair; The relevant operations in step S2 will be introduced with specific examples: Image frames from the left and right cameras are synchronized in time to construct dual-view image pairs under the same time reference, and cache management is performed. Left and right image frames are matched according to timestamps; when the time difference between two frames is less than a preset synchronization threshold, an image pair is formed; frames that cannot be matched or whose time difference exceeds the threshold are discarded or delayed to avoid multi-view misalignment that may cause subsequent calculation errors.

[0029] Specifically, this involves setting a time synchronization threshold. Select timestamp interpolation less than In this embodiment, a dual-view image pair is constructed from the left-view image and the right-view image. Take 0.05s.

[0030] S3. Extract key point coordinates from the dual-view image pair to obtain the face region, facial key point coordinates, and eye key point coordinates. The face region is preprocessed based on the coordinates of the facial key points and the eye key points to obtain dual-view facial images and dual-view eye images. The relevant operations in step S3 will be introduced with specific examples: For images in a dual-view image pair, a deep learning-based face detection algorithm is used to perform face detection separately to obtain the driver's face region. A "detection-tracking combination" strategy is adopted, performing face detection at low frequency and face region tracking at high frequency to reduce the system's computational load and maintain the continuous stability of the face region.

[0031] A predetermined number of facial key points are extracted within the face region to describe the facial geometry. Simultaneously, eye key points or eye region boundaries are extracted to determine the positions of the left and right eye regions. The eye region is then cropped based on the eye key points, and preprocessing steps such as scale normalization, rotation correction, and brightness normalization are performed to reduce the impact of camera viewing angle differences and lighting variations.

[0032] Specifically, a low-frequency detection-high-frequency tracking strategy is adopted. Every N frames (N is a preset number of frames; in this embodiment, N is 3), a face detection algorithm is used to detect the left-view and right-view images to obtain the face region, facial keypoint coordinates, and eye keypoint coordinates (the face region refers to the area determined by the facial keypoint coordinates and eye keypoint coordinates). The face region is then cropped and normalized (rotated, translated, and scaled) to obtain dual-view (left-view and right-view) face images and dual-view eye images. Between adjacent non-detection frames, a keypoint tracking algorithm is used to perform inter-frame tracking of keypoints in the previous detection or tracking frame to update the facial keypoint coordinates and eye keypoint coordinates.

[0033] In this embodiment, the face detection algorithm is the RetinaFace face detection algorithm; the key point tracking algorithm is a key point tracking algorithm based on "Lucas-Kanade optical flow", combined with Kalman filtering for temporal smoothing; The normalization process for eye keypoint coordinates is the same as that for facial keypoint coordinates. Taking the processing of eye keypoint coordinates as an example, we will now introduce the normalization process (rotation, translation, and scaling): The similarity transformation matrix is ​​calculated based on the coordinates of key eye points and a preset target eye coordinate template. The similarity transformation matrix is ​​then used to rotate, translate, and scale the eye image to obtain a dual-view eye image (normalized eye image). The construction of the similarity transformation matrix first involves pre-setting a target eye coordinate template. , This represents the index value of the eye coordinates. And obtain the current eye coordinates. Construct a similarity transformation matrix T between the current eye keypoint coordinates and the target eye coordinates, such that... The similarity transformation matrix includes rotation, translation, and scaling, and can be specifically represented as:

[0034] Where s is the scaling factor, For rotation amount, and This is the translation amount; The optimal similarity transformation matrix is ​​obtained by minimizing the reprojection error between the current eye key points and the target eye coordinate template. Specifically:

[0035] In this embodiment, the least squares method or Procrustes analysis method is used to solve the problem. Based on the solution of the optimal similarity transformation matrix, the actual rotation, translation, and scaling amounts are derived in reverse. This represents the square of the L2 norm, which is the square of the Euclidean distance. This represents the set of all independent variables for which the objective function achieves its minimum value, i.e., through... The operation can find the independent variable that minimizes the objective function within the parentheses.

[0036] S4. Establish a correspondence between the facial key point coordinates and the preset three-dimensional face model, and solve the head pose parameters through the PnP algorithm; The relevant operations in step S4 will be introduced with specific examples: A correspondence is established between 2D facial key points and 3D face model points. The 3D face model is a predefined sparse 3D structure model in a face coordinate system. This model contains multiple 3D feature points with the same semantic meaning as the 2D detected key points. These 3D feature points include at least one or more of the following: inner and outer corners of the eyes, tip of the nose, alar of the nose, corner of the mouth, chin, and iris boundary. The coordinates of the corresponding 2D facial key points are extracted from the near-infrared image using a facial key point detection algorithm. These 2D facial key points are then matched with the corresponding 3D feature points in the 3D face model according to their semantic positions, forming a 2D-3D correspondence point set. Based on this correspondence point set and camera intrinsic parameters, the PnP algorithm is used to solve for the spatial pose parameters of the face relative to the camera coordinate system. These pose parameters include rotation and translation parameters. Further, the rotation parameters can be decomposed into one or more of yaw, pitch, and roll angles. To reduce image noise, vehicle vibration, and pose jitter caused by instantaneous detection errors, a short-term smoothing filter is applied to the head pose parameters.

[0037] S5. Construct a dual-branch feature extraction network to extract dual-view features based on dual-view facial images and dual-view eye images. , , For view index value, Left-hand perspective From the right perspective; The relevant operations in step S5 will be introduced with specific examples: The deep learning component extracts gaze-related features from eye images to directly regress the gaze vector and key points on the eyeball; the geometric component incorporates head pose information to achieve head-eye movement decoupling or coordinate correction. The model output can be a 3D unit gaze vector or an equivalent representation in the form of yaw / pitch angles. In dual-view scenarios, two gaze estimation results can be obtained from the left and right cameras respectively, and the two results along with their corresponding confidence scores are fed into the subsequent fusion module.

[0038] The dual-branch feature extraction network includes: a left-view feature extraction branch and a right-view feature extraction branch; The left-view feature extraction branch and the right-view feature extraction branch share the same feature extraction network, and the corresponding eye feature extraction sub-network, face feature extraction sub-network, and pose encoder share parameters. The feature extraction network includes at least one convolutional block consisting of a convolutional layer (Conv), a batch normalization layer (BN layer), and a ReLU activation layer, used for parallel feature extraction of eye images from different viewpoints, outputting left-view features respectively. and right-angle features .

[0039] S6. The head posture parameters are processed by the posture encoder and then compared with... The images are stitched together to obtain the stitched dual-view features. ; The relevant operations in step S6 will be introduced with specific examples: For any perspective The feature extraction branch simultaneously receives eye images, facial images, and head pose parameters from that viewpoint. Specifically, the eye image is input into the eye feature extraction subnetwork to obtain eye features, the facial image is input into the facial feature extraction subnetwork to obtain facial context features, and the head pose parameters are input into the pose encoder to obtain head pose features. Subsequently, the eye features, facial context features, and head pose features are concatenated or weighted and fused to form the fused features for that viewpoint. .

[0040] The attitude encoder is used for feature mapping and encoding of head attitude parameters. The attitude encoder receives a head attitude parameter vector as input, which includes at least one or more of yaw, pitch, and roll angles. From input to output, the attitude encoder sequentially includes a first fully connected attitude encoding layer, a batch normalization attitude encoding layer, a nonlinear activation attitude encoding layer, and a second fully connected attitude encoding layer. The nonlinear activation attitude encoding layer uses the ReLU activation function. The attitude encoder encodes the head attitude parameters into a high-dimensional attitude representation relevant to the line-of-sight estimation task, thereby enhancing the system's robustness under conditions of large yaw angles, local occlusion, and varying illumination.

[0041] In this embodiment, the attitude encoder also includes translation parameters of the head in the camera coordinate system. The attitude encoder is a lightweight attitude coding network built on a multi-layer perceptron (MLP). S7 After processing by a deep learning-based gaze regression head, dual-view gaze vectors are obtained. Dual-view geometric line-of-sight vector Dual-view gating coefficient and confidence of dual-viewpoint gaze vector ; The relevant operations in step S7 will be introduced with specific examples: The gaze regression head includes a direct regression branch, a geometric branch, a gated fusion branch, and a confidence estimation branch; The direct regression branch is used to output the dual-view gaze vector. The formula for calculating the direct regression branch is: ,in, This represents the weight matrix of the direct regression branch. This represents the bias vector of the direct regression branch. and The model is learned through joint learning of backpropagation and gradient optimization during the model training phase, and is used consistently during the inference phase. The normalization function is represented by the direct regression branch, which is used to learn the gaze direction estimation result directly from image features without explicitly relying on precise geometric modeling, thereby compensating for the error of the geometric branch under conditions of occlusion, reflection, or keypoint instability.

[0042] The geometric branch is used to output the dual-view geometric line-of-sight vector. The formula for calculating the geometric branch is: ,in, Indicates the first From the camera's optical center, passing through the central pixel coordinates of the iris, in each perspective... The direction vector of the back-projected ray; Indicates the first The three-dimensional coordinates of the eyeball center in the camera coordinate system from each perspective; The acquisition process is as follows: based on the facial key points and the preset 3D face model, pose registration is performed to obtain the head pose transformation matrix. Then, the eye center reference coordinates in the preset 3D face model are used. By mapping the head pose transformation to the camera coordinate system, we obtain:

[0043] in, This is a rotation matrix used to characterize the head's orientation relative to the camera. This is a translation vector used to represent the position of the head relative to the camera; ,in, Indicates the first The inverse matrix of the intrinsic parameter matrix of each view camera; The gated fusion branch is used to fuse the dual-viewpoint gaze vector and the dual-viewpoint geometric gaze vector to obtain the gating coefficients for the corresponding viewpoints. The calculation formula for the gated fusion branch is as follows: , This represents the sigmoid activation function. This represents the transpose of the gating fusion branch weight vector. This represents the bias scalar of the gated fusion branch. and It is obtained through joint learning of backpropagation and gradient optimization during the model training phase and is used indefinitely during the inference phase. Indicates the first The gating coefficients for each perspective characterize the distribution of confidence levels between the geometric branch results and the direct regression branch results from the current perspective. ; The confidence estimation branch is used to output the confidence of the dual-view gaze vector. The formula for calculating the confidence estimation branch is: ,in, This represents the weight parameters of the confidence estimation branch. This represents the bias parameter of the confidence estimation branch. and It is obtained through joint learning of backpropagation and gradient optimization during the model training phase and is used indefinitely during the inference phase.

[0044] S8, according to , and Calculate the final dual-view gaze vector ; The relevant operations in step S8 will be introduced with specific examples: For any perspective Its final dual-view line of sight vector Dual-view gaze vectors output by the direct regression branch Dual-view geometric line-of-sight vectors output by the geometric branch Obtained through gated weighted fusion, specifically as follows: .

[0045] S9. Constructing Quality Scores Based on the quality score, the system will perform dual-view fusion output or single-view downgraded output, suppressed output, and degraded output.

[0046] The relevant operations in step S9 will be introduced with specific examples: The system assesses the sharpness, brightness, contrast, and motion blur of the input image at the current moment; it also evaluates whether the difference between the left and right gaze estimation results exceeds a preset consistency threshold and whether the rate of change between the current gaze result and historical results is abnormal. A weighted output is then used to synthesize the overall observation quality.

[0047] quality score The formula for calculation is: ,in, Indicates the first The image quality index for each viewpoint is calculated based on the sharpness of the eye image, the degree of brightness deviation, and the proportion of overexposure or underexposure, and is normalized and mapped to the [0,1] interval; where the larger the value, the higher the image quality of that viewpoint. Indicates the first The key point quality index from each perspective is calculated based on the key point detection confidence and the rationality of the key point geometric structure, and then normalized and mapped to the [0,1] interval; where the larger the value, the more reliable the key point monitoring results are. Indicates the first The attitude availability index for each viewpoint is calculated based on whether the head yaw angle, pitch angle, and roll angle are within a preset effective range and their temporal stability, and is normalized and mapped to the [0,1] interval; where the larger the value, the more suitable the viewpoint is for line-of-sight estimation under the current attitude condition. The consistency index between the left and right viewpoints is calculated based on the angle difference between the left and right viewpoint line-of-sight estimation results, and then normalized and mapped to the [0,1] interval; where the more consistent the left and right viewpoint estimation results are, the higher the consistency index. The larger; Indicates the first The confidence index of the gaze estimation result for each viewpoint is calculated based on the confidence of the gaze regression branch output, the fusion gating coefficient, or the output temporal stability. The larger the value, the more reliable the gaze estimation result for that viewpoint. , This represents the weighting coefficient of the corresponding indicator. , This represents the index value of the indicator. In this embodiment, arrive The values ​​are 0.1, 0.1, 0.2, 0.3, and 0.3 respectively. The specific steps for determining whether to perform dual-view fusion output or single-view degradation output, suppress output, or degraded output based on the quality score are as follows: S81, when , ,and When the view is in the correct position, execute the dual-view fusion output; otherwise, execute step S82, where... Indicates the preset quality threshold, This represents a preset viewpoint consistency threshold; in this embodiment... The value is 0.60. The value is 0.60.

[0048] S82, when ,and or ,and When (the quality score of only one viewpoint is not lower than the single-viewpoint output threshold) If the output is a single-view downgrade, then execute step S83; otherwise, execute step S83. This represents the single-view output threshold; in this embodiment, The value is 0.40. ; S83, when , ,and , If the consistency of the line-of-sight results from the two perspectives does not meet the requirements, output suppression is performed; otherwise, step S84 is executed. Indicates the suppression threshold; in this embodiment, The value is 0.20.

[0049] S84, when and If the eye keypoints are unavailable, perform a degenerate output.

[0050] The formula for calculating the dual-view fusion output is: ; Indicates the first The fusion weight of each perspective Indicates dual-view output; The formula for calculating the single-view downgraded output is: , The index value represents a single viewpoint. The value is or , Indicates single-view output; The suppressed output is to maintain the stable line-of-sight output of the previous moment or to output a null value; The degenerate output is an approximate gaze based on head pose estimation; The above steps are used to perform fusion or degradation processing on the gaze estimation results to obtain a stable and continuous final gaze output. The output format can be a 3D gaze unit vector, gaze region identifier, confidence score, or low-confidence status code. When both viewpoints are available, the left and right gaze results are weighted and fused, with the weights determined by the confidence score and head pose. When a single viewpoint is available, the gaze result of the available viewpoint is output. When eye information degrades, an approximate gaze direction based on head pose is output, or the previous stable gaze result is maintained. When a sudden change in gaze result is detected and the quality is low, the output can be smoothed, limited, or temporarily frozen to avoid abnormal frames causing subsequent system misjudgments.

[0051] Example 2 This embodiment further defines Embodiment 1. This embodiment provides a driver gaze tracking system based on dual near-infrared cameras. The system is used to implement the method described in Embodiment 1, such as... Figure 3 As shown, the system includes: an image acquisition module, an image pair construction module, a key point extraction module, a head pose parameter solving module, a dual-view feature extraction module, a feature stitching module, a gaze regression head module, a gaze vector calculation module, and an output module; The image acquisition module includes near-infrared cameras positioned at the left and right A-pillars of the vehicle to acquire real-time images from both the left and right perspectives. These near-infrared cameras are monochrome imaging cameras operating in the near-infrared band. They work in conjunction with a near-infrared supplementary light source to improve imaging stability in low-light or nighttime conditions. This dual-view acquisition method ensures that at least one camera acquires usable image data for subsequent processing, even when the driver's head is significantly turned or partially obstructed.

[0052] The image pair construction module synchronizes the left and right view images in time to create dual-view image pairs. By synchronizing and caching images captured by the two cameras, it ensures that images from different perspectives are on a unified time standard. This module receives each frame and its timestamp from the left and right cameras and aligns the images based on the timestamps to form image pairs with a unified time standard. Furthermore, this module balances the computational latency between different modules by setting up a cache queue.

[0053] The keypoint extraction module sequentially extracts and preprocesses keypoint coordinates from the dual-view image pairs to obtain facial keypoint coordinates, dual-view facial images, and dual-view eye images. It locates the driver's face region in the synchronized near-infrared image and extracts facial and eye keypoints for geometric calculations. This module performs face detection on the input image, determines the driver's face position and size, and extracts a preset number of facial keypoints within the face region to describe the face's geometric structure. Simultaneously, it extracts the eye region and eye keypoints for subsequent input to the estimation model.

[0054] The head pose parameter solving module calculates head pose parameters based on the coordinates of the facial key points. This module calculates the spatial pose of the driver's head relative to the camera based on the facial key points and a preset 3D face model. The module establishes a mapping relationship between the 2D facial key points obtained by the face detection and key point extraction module and the corresponding key points in the 3D face model. It uses the Perspective-n-Point (PnP) algorithm to calculate head pose parameters and outputs information such as head yaw angle, pitch angle, roll angle, and head coordinates.

[0055] The dual-view feature extraction module constructs a dual-branch feature extraction network to extract dual-view features based on dual-view facial images and dual-view eye images; The feature stitching module stitches together the head pose parameters and dual-view features to obtain the stitched dual-view features. The gaze regression head module generates dual-view gaze vectors based on the stitched dual-view features. Dual-view geometric line-of-sight vector Dual-view gating coefficient and confidence of dual-viewpoint gaze vector ; The line-of-sight vector calculation module, based on , and Calculate the final dual-view gaze vector ; Output module, constructs quality score Based on the quality score, the module determines whether to perform dual-view fusion output, single-view downgraded output, suppressed output, or degraded output. This module evaluates the observation quality based on image sharpness, keypoint detection confidence, and dual-view consistency. When the evaluation result falls below a preset threshold, a degradation processing strategy is triggered. The degradation processing strategy includes weighted fusion of the dual-camera line-of-sight results, selective output, or maintaining the previous stable line-of-sight result. This module prevents sudden changes in line-of-sight output under conditions of occlusion, reflection, or transient failure. like Figure 4As shown, the system is trained and calibrated using a progressive strategy, which includes pre-training in the stationary phase, fine-tuning in the moving phase, and short calibration at the start.

[0056] The pre-training of the static segment specifically refers to: S10. When the vehicle is stationary, multiple marked targets are displayed in a preset order via the vehicle's display device. And record each marked target Display time and spatial location information; The marked targets are displayed in circles, and the target positions are recorded with the center position of the circle. S11. Acquire dual-view images of the driver and determine the corresponding marker targets for each view image based on timestamp matching. ,in, This indicates the three-dimensional coordinates of the target in the camera coordinate system; S12. Obtain the training dataset based on the driver's dual-view images. ; The training dataset includes: dual-view facial images of the driver, facial key points, eye key points, head pose parameters, and the three-dimensional coordinates of the corresponding labeled targets in the camera coordinate system. ; S14, according to Using the driver's line of sight pointing towards the marked target as a geometric constraint, the system is trained to obtain the first-stage model parameters. ; The geometric constraints include: a point-ray loss function, which is a distance loss function between the marked target point and the predicted line-of-sight ray; and a loss function for the consistency of the three-dimensional coordinates and angles of the marked target in the camera coordinate system, used to constrain the angle between the predicted line-of-sight direction and the target line-of-sight direction.

[0057] The point-to-line-of-sight loss function is calculated as follows: ,in, This represents the three-dimensional coordinates of the driver's eye center in the camera coordinate system. It is obtained by: performing pose fitting based on facial key points and a 3D face model to obtain the head pose transformation matrix; and then mapping the eye center reference point from the standard face model to the camera coordinate system. Represents the square of the norm; The formula for calculating the angle consistency loss function is: The first stage model parameters The method for obtaining it is: using the static segment pre-training dataset The eye-tracking model is trained under supervised supervision, with the training objective being to minimize the total loss function. The model parameters are iteratively optimized to obtain the first-stage model parameters. .

[0058] ; ; in, This represents the model parameters to be optimized, which are obtained through joint learning via backpropagation and gradient optimization during the model training phase and are fixed for use during the inference phase. N represents the number of pre-training samples in the static segment. and The loss weight coefficients are a set of manually set hyperparameters. In this embodiment, Take 1, Take 0.6.

[0059] The fine-tuning of the motion segment specifically involves: simultaneously acquiring the driver's dual-view images, target display information, and vehicle or camera attitude change information while the vehicle is in motion; and... Based on this, the coordinate transformation relationship corresponding to changes in vehicle or camera posture is introduced, and the system is fine-tuned and trained to obtain the second-stage model parameters. ; The initiation of short calibration specifically involves: setting a predetermined duration of target calibration process before each driving session begins, acquiring the driver's dual-view images and corresponding target information; and based on... Only for , and (One or more) are updated to obtain individualized eye-tracking parameters for the current driver. .

[0060] Example 3 This embodiment further defines embodiments 1 and 2, providing an alternative dual-camera installation solution: while maintaining the overall system structure and training process, the cameras can be placed on the left and right sides of the dashboard, or on the left A-pillar above the central control screen, etc. The cameras can use fixed brackets or adjustable brackets (adjustable pitch / yaw / roll) to adapt to different vehicle cabin structures.

[0061] Example 4 This embodiment further defines embodiments 1 and 2, providing an alternative solution for face detection and keypoint extraction: the face detection module can be implemented using any face detection method based on traditional image processing or deep learning, without being limited to a specific model structure or open-source project; and a "detection + tracking" combination strategy can be adopted to reduce computational load. The number of facial keypoints can be 5, 68, 106, or a facial mesh; eye localization can be achieved using keypoint regression, eye region segmentation, or iris / pupil detection, etc. Keypoint extraction can be performed monocularly or by cross-validation with binoculars to enhance stability.

[0062] Example 5 This embodiment further defines embodiments 1 and 2, providing an alternative solution for head pose estimation (PnP solution) and pose source: the input can use only binocular images, or a combination of monocular images + facial images, or binocular images + facial images + head pose parameters. Multiple frames of input with short time windows can be added to form a temporal model to improve stability.

[0063] The feature extraction network can be a convolutional neural network, a Transformer network, or a combination of both; dual-view fusion can be achieved through methods such as concatenation, weighted averaging, gating fusion, and attention fusion; the model can output gaze vectors, gaze angle pairs, or gaze point coordinates, etc.

[0064] The model can directly regress the gaze vector or regress intermediate geometric quantities such as the iris / pupil center and eyeball center, and then calculate the gaze direction, or it can simultaneously have direct regression and geometric branching, and fuse them through a gating mechanism.

[0065] Example 6 This embodiment further defines embodiments 1 and 2, providing an alternative to coordinate modeling and "truth / supervision" methods: the marked target is not limited to pixels, but can be a cross, ring, concentric circle, high-contrast graphic, flashing target, or region block, and the target area can be set to adapt to driver visibility and gaze stability. Training supervision can be implemented using one or more equivalent forms, including but not limited to point-to-ray shortest distance constraints, angular consistency constraints between the line of sight and the "eye center pointing to the target point," regional consistency constraints between the line of sight and the target area (tolerance radius), and dual-view consistency regularization (consistent left and right outputs or consistent gaze points), etc.

[0066] Example 7 This embodiment further defines embodiments 1 and 2, providing an alternative to the phased progressive training and calibration process: in addition to the three stages of "static pre-training—motion fine-tuning—initiating a short calibration segment," static pre-training + initiating a short calibration; or static pre-training + mild motion fine-tuning + strong motion fine-tuning + initiating a short calibration. Calibration can be performed by updating only the final layer bias or a small number of modulation parameters or estimating the line-of-sight bias vector for output correction, or a driver's individual profile can be established and retrieved and fine-tuned each time it is used.

Claims

1. A gaze tracking method based on the fusion of dual-view geometry and deep learning, characterized in that, The method includes the following steps: S1. Cameras are installed on the left and right A-pillars of the vehicle to acquire real-time images from the left and right perspectives. S2. Synchronize the left-view image and the right-view image in time to construct a dual-view image pair; S3. Extract key point coordinates from the dual-view image pair to obtain the face region, facial key point coordinates, and eye key point coordinates. The face region is preprocessed based on the coordinates of the facial key points and the eye key points to obtain dual-view facial images and dual-view eye images. S4. Establish a correspondence between the facial key point coordinates and the preset three-dimensional face model, and solve the head pose parameters through the PnP algorithm; S5. Construct a dual-branch feature extraction network to extract dual-view features based on dual-view facial images and dual-view eye images. , , For view index value, Left-hand perspective From the right perspective; S6. The head posture parameters are processed by the posture encoder and then compared with... The images are stitched together to obtain the stitched dual-view features. ; S7 After processing by a deep learning-based gaze regression head, dual-view gaze vectors are obtained. Dual-view geometric line-of-sight vector Dual-view gating coefficient and confidence of dual-viewpoint gaze vector ; S8, according to , and Calculate the final dual-view gaze vector ; S9. Constructing Quality Scores Based on the quality score, the system will perform dual-view fusion output or single-view downgraded output, suppressed output, and degraded output.

2. The gaze tracking method based on the fusion of dual-view geometry and deep learning according to claim 1, characterized in that, The process of constructing dual-view image pairs is as follows: setting a time synchronization threshold. Select timestamp interpolation less than A dual-view image pair is constructed from the left-view and right-view images.

3. The gaze tracking method based on the fusion of dual-view geometry and deep learning according to claim 2, characterized in that, The process of extracting the key point coordinates is as follows: the dual-view image pairs are processed by the face detection algorithm and the key point tracking algorithm in sequence, and then Kalman filtering is performed to obtain the face region, the coordinates of the facial key points and the coordinates of the eye key points. The preprocessing includes: cropping, rotating, translating, and scaling; The head attitude parameters include: yaw angle, pitch angle, and roll angle.

4. The gaze tracking method based on the fusion of dual-view geometry and deep learning according to claim 3, characterized in that, The dual-branch feature extraction network includes: a left-view feature extraction branch and a right-view feature extraction branch; The left-view feature extraction branch and the right-view feature extraction branch share the same feature extraction network, which passes through the following layers from input to output: feature extraction convolutional layer, feature extraction batch normalization layer and feature extraction nonlinear activation layer; Parameters are shared between dual-view facial images and dual-view eye images.

5. The gaze tracking method based on the fusion of dual-view geometry and deep learning according to claim 4, characterized in that, The attitude encoder passes through the following layers sequentially from input to output: the first fully connected layer of attitude encoding, the batch normalization layer of attitude encoding, the nonlinear activation layer of attitude encoding, and the second fully connected layer of attitude encoding.

6. The gaze tracking method based on the fusion of dual-view geometry and deep learning according to claim 5, characterized in that, The gaze regression head includes a direct regression branch, a geometric branch, a gated fusion branch, and a confidence estimation branch; The direct regression branch is used to output the dual-view gaze vector, and the calculation formula for the direct regression branch is: ,in, This represents the weight matrix of the direct regression branch. This represents the bias vector of the direct regression branch. Represents the normalization function; The geometric branch is used to output the dual-view geometric line-of-sight vector, and the formula for calculating the geometric branch is: ,in, Indicates the first From the camera's optical center, passing through the central pixel coordinates of the iris, in each perspective... The direction vector of the back-projected ray; Indicates the first The three-dimensional coordinates of the eyeball center in the camera coordinate system from each perspective; The gated fusion branch is used to output the dual-view gating coefficients, and the calculation formula for the gated fusion branch is: , This represents the sigmoid activation function. This represents the transpose of the gating fusion branch weight vector. The bias scalar represents the gating fusion branch; The confidence estimation branch is used to output the confidence score of the dual-view gaze vector. The calculation formula for the confidence estimation branch is: ,in, This represents the weight parameters of the confidence estimation branch. This represents the bias parameter of the confidence estimation branch.

7. The gaze tracking method based on the fusion of dual-view geometry and deep learning according to claim 6, characterized in that, Final dual-view line of sight vector The formula for calculation is: ; quality score The formula for calculation is: ,in, Indicates the first Image quality metrics from various perspectives Indicates the first Key quality indicators from each perspective Indicates the first Aspect-based usability metric for pose. An index representing the consistency between the left and right perspectives. Indicates the first Confidence index of gaze estimation results from each perspective , This represents the weighting coefficient of the corresponding indicator. , This represents the index value of the indicator. ; The specific steps for determining whether to perform dual-view fusion output or single-view degradation output, suppress output, or degraded output based on the quality score are as follows: S81, when , ,and When the view is in the correct position, execute the dual-view fusion output; otherwise, execute step S82, where... Indicates the preset quality threshold, This indicates the preset viewpoint consistency threshold. S82, when and or and When the output is in a single-view downgraded state, execute the output; otherwise, execute step S83, where... Indicates the single-view output threshold; S83, when , ,and , When the output is suppressed, step S84 is executed; otherwise, step S84 is executed. Indicates the inhibition threshold; S84, when and At that time, degenerate output is executed.

8. The gaze tracking method based on the fusion of dual-view geometry and deep learning according to claim 7, characterized in that, Dual-view fusion output or single-view degraded output, suppressed output, and degradation output are specifically as follows: The formula for calculating the dual-view fusion output is: ; Indicates the first The fusion weight of each perspective Indicates dual-view output; The formula for calculating the single-view downgraded output is: , The index value represents a single viewpoint. The value is or , Indicates single-view output; The suppressed output means: no output is output in the current frame, and the stable result of the previous frame is maintained; The degenerate output indicates that, based on head pose parameters, the head is oriented approximately towards the line of sight.

9. A gaze tracking system based on the fusion of dual-view geometry and deep learning, characterized in that, The system is used to implement the method according to any one of claims 1 to 8, and the system includes: an image acquisition module, an image pair construction module, a key point extraction module, a head pose parameter solving module, a dual-view feature extraction module, a feature stitching module, a gaze regression head module, a gaze vector calculation module, and an output module; The image acquisition module has cameras installed at the left and right A-pillars of the vehicle to acquire images from the left and right perspectives in real time. The image pair construction module synchronizes the left-view and right-view images in time to construct dual-view image pairs. The key point extraction module extracts and preprocesses the key point coordinates of the dual-view image pair in sequence to obtain facial key point coordinates, dual-view facial images, and dual-view eye images. The head pose parameter solving module solves the head pose parameters based on the coordinates of the facial key points. The dual-view feature extraction module constructs a dual-branch feature extraction network to extract dual-view features based on dual-view facial images and dual-view eye images; The feature stitching module stitches together the head pose parameters and dual-view features to obtain the stitched dual-view features. The gaze regression head module generates dual-view gaze vectors based on the stitched dual-view features. Dual-view geometric line-of-sight vector Dual-view gating coefficient and confidence of dual-viewpoint gaze vector ; The line-of-sight vector calculation module, based on , and Calculate the line-of-sight vector ; Output module, constructs quality score Based on the quality score, perform dual-view fusion output or single-view downgraded output, suppressed output, and degraded output; The system is trained and calibrated using a progressive strategy, which includes pre-training during stationary periods, fine-tuning during moving periods, and initiation of short calibrations.

10. The gaze tracking system based on the fusion of dual-view geometry and deep learning according to claim 9, characterized in that, The pre-training of the static segment specifically refers to: S10. When the vehicle is stationary, multiple marked targets are displayed in a preset order via the vehicle's display device. And record each marked target Display time and spatial location information; S11. Acquire dual-view images of the driver and determine the corresponding marker targets for each view image based on timestamp matching. ,in, This indicates the three-dimensional coordinates of the target in the camera coordinate system; S12. Obtain the training dataset based on the driver's dual-view images. ; S14, according to The system is trained using the point-line loss function as a geometric constraint to obtain the first-stage model parameters. ; The point-to-line-of-sight loss function is calculated as follows: ,in, This indicates the coordinates of the driver's eye center in the camera's coordinate system. Represents the square of the norm; The fine-tuning of the motion segment specifically involves: simultaneously acquiring the driver's dual-view images, target display information, and vehicle or camera attitude change information while the vehicle is in motion; and... Based on this, the coordinate transformation relationship corresponding to changes in vehicle or camera posture is introduced, and the system is fine-tuned and trained to obtain the second-stage model parameters. ; The initiation of short calibration specifically involves: setting a predetermined duration of target calibration process before each driving session begins, acquiring the driver's dual-view images and corresponding target information; and based on... Only for , and Update to obtain individualized eye-tracking parameters for the current driver. .

Images