Pilot eye tracking method, system, device and storage medium based on multi-camera fusion

Abstract

The present invention belongs to the field of eye movement data tracking, and specifically relates to a pilot eye tracking method, system, device, and storage medium based on multi-camera fusion. This method aims to address the existing issues of traditional eye trackers' limited viewing angles and line of sight estimation errors caused by head movement. The present invention comprises: constructing a multi-perspective visual perception network by deploying multiple cameras and eye trackers in the cockpit. Spatial coordinate system mapping and time synchronization are used to achieve spatiotemporal alignment of multi-source data, and head motion errors are compensated for by combining dynamic cropping of eye regions of interest and three-dimensional head pose estimation. The geometric path generates physical drive vectors based on iris fitting and eyeball models, and the data path regresses vectors using a deep learning model; after fusion, the vectors are converted to the world coordinate system using head pose. High-precision eye tracker data is preferentially used in the central area, while peripheral areas switch to the system's extended field of view data. The present invention significantly improves field of view coverage and tracking accuracy under motion interference.

Description

Technical Field

[0001] The present invention belongs to the field of eye movement data tracking, and in particular relates to a pilot eye movement tracking method, system, device and storage medium based on multi-camera fusion. Background Art

[0002] Traditional eye trackers typically rely on a single or a small number of cameras, resulting in a limited field of view (FOV), typically less than 120 degrees. However, the human visual system has a much wider field of view, approaching 200 degrees horizontally and approximately 135 degrees vertically. This difference in FOV makes it difficult for traditional eye trackers to capture the wide-ranging visual search behaviors of pilots in the cockpit, such as lateral glances. These behaviors are crucial for pilots to detect potential threats, monitor their surroundings, and maintain comprehensive situational awareness.

[0003] Pilots are not static when operating in the cockpit, and head movement is normal. However, head movement can significantly affect the accuracy of eye tracking. If the eye tracking system fails to effectively compensate for head movement, it will lead to significant errors in line of sight estimation, with an average offset of up to 8-12 degrees. The error in line of sight calculation will affect the assessment of attention allocation during pilot training. For example, during landing, it is necessary to pay attention to whether the pilot allocates his attention to areas such as the runway centerline, PAPI lights, speed, altitude, descent rate, and attitude. These areas are relatively close in position in the visual field or instruments, which may make it impossible to accurately detect problems with the pilot's attention allocation during training.

[0004] Based on this, the present invention proposes a pilot eye tracking method, system, device and storage medium based on multi-camera fusion. Summary of the Invention

[0005] In order to solve the above-mentioned problems in the prior art, namely, the limited viewing angle of traditional eye trackers in the prior art, and the problem that head movement causes line of sight estimation errors, which affects the attention allocation assessment in pilot training, the present invention provides a pilot eye tracking method, system, device and storage medium based on multi-camera fusion.

[0006] In a first aspect, the present invention proposes a pilot eye tracking method based on multi-camera fusion, the method comprising:

[0007] Multiple cameras are deployed in a preset area within the cockpit. The image data collected by each camera is mapped into a unified three-dimensional spatial coordinate system to obtain multi-view images based on spatiotemporal alignment. An eye tracker is also deployed to collect eye movement data in the central area of ​​the cockpit.

[0008] Extracting facial region images from the multi-view images and dynamically cropping a region of interest including the eyes, adaptively adjusting a window position and size of the region of interest according to a head deflection angle, and calculating three-dimensional head posture data of the pilot based on feature points of the facial region images;

[0009] The gaze direction vector is calculated in parallel through the geometric path and the data path, the dual-path calculation results are integrated, and the gaze direction vector is converted to the world coordinate system using the 3D head posture data to compensate for head motion errors.

[0010] The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model;

[0011] The error-compensated gaze direction vector is spatially and temporally aligned with the eye movement data collected by the eye tracker, and the weights are dynamically assigned to obtain the final gaze direction vector:

[0012] When the line of sight is in the center area of ​​the cockpit, the eye movement data collected by the eye tracker is used first. When the line of sight exceeds the center area, it switches to the final line of sight direction vector.

[0013] Furthermore, the image data collected by each camera is mapped to a unified three-dimensional space coordinate system by:

[0014] Calibrate parameters of each camera to determine internal and external parameters;

[0015] The internal parameters include focal length, principal point coordinates and lens distortion parameters, and the internal parameter matrix of the camera is constructed; the external parameters include the rotation matrix and translation vector of the camera in the world coordinate system;

[0016] The world coordinates of any 3D space point are first converted to the camera coordinate system through the rotation matrix and translation vector, and then the 3D coordinates in the camera coordinate system are projected onto the 2D image plane according to the intrinsic parameter matrix;

[0017] Establish a transformation relationship between different camera coordinate systems. Through the rotation matrix and translation vector from one camera to another, the three-dimensional point coordinates in one camera coordinate system are converted into the three-dimensional coordinates in the other camera coordinate system. Finally, the three-dimensional data collected by all cameras are mapped to the same global three-dimensional coordinate system to achieve cross-view data fusion.

[0018] Furthermore, the facial region image in the multi-view image is extracted by:

[0019] A hybrid deep network model is used to extract facial area images from the multi-view images, and the hybrid deep network model is jointly constructed based on a ResNet50 network model and a ViT network model.

[0020] Furthermore, the region of interest including the eyes is dynamically cropped by:

[0021] Set the initial reference width and height of the eye's region of interest. The initial reference width and height are pre-configured based on the pilot's eye anatomy and camera resolution.

[0022] The width and height of the region of interest are proportionally expanded or contracted according to the horizontal deflection angle and vertical pitch angle of the pilot's head;

[0023] Calculate the eye center position based on the detected eye corner key point coordinates;

[0024] Generate a rotation matrix based on the head deflection angle, perform a rotation transformation on the original offset between the preset eye center and the center of the region of interest, and obtain the dynamic offset after the head deflection;

[0025] The dynamic offset is superimposed on the eye center coordinates to determine the final center position of the region of interest window;

[0026] The rotation direction of the ROI window is synchronously adjusted according to the head deflection angle, so that the cropping frame is always aligned with the actual deflection direction of the eyeball, ensuring complete coverage of the eye area under extreme head postures.

[0027] Furthermore, the width and height of the region of interest are proportionally expanded or contracted according to the horizontal deflection angle and vertical pitch angle of the pilot's head, specifically:

[0028] According to the horizontal deflection angle of the pilot's head, based on the reference width, the width of the region of interest is proportionally expanded or contracted. The expansion amplitude is proportional to the absolute value of the sine of the deflection angle, and the scaling factor controls the maximum adjustment amplitude.

[0029] According to the vertical pitch angle of the head, the height of the region of interest is proportionally expanded or contracted based on the reference height. The expansion amplitude is proportional to the absolute value of the sine of the pitch angle, and the scaling factor is independent of the horizontal direction.

[0030] Furthermore, the three-dimensional head posture data of the pilot is calculated based on the feature points of the facial region image, and the method is as follows:

[0031] Predefining a three-dimensional facial model including key points of the corners of the eyes, the tip of the nose, and the corners of the mouth, wherein each key point in the model has known three-dimensional spatial coordinates; detecting two-dimensional facial key points corresponding to the three-dimensional model from the facial region image, and establishing a matching relationship between the three-dimensional model points and the two-dimensional image points;

[0032] Through perspective projection geometry, the 3D model points are projected onto the theoretical positions of the 2D image plane, and the positional deviation between the theoretical positions and the actual detection points is calculated;

[0033] The rotation matrix and translation vector are used as optimization variables, and their values ​​are iteratively adjusted to minimize the sum of squared projection deviations of all matching key points, thereby solving the optimal head posture parameters.

[0034] The pilot's head yaw angle, pitch angle, and roll angle are extracted from the optimized rotation matrix as the pilot's three-dimensional head posture data.

[0035] Furthermore, the gaze direction vector is generated based on the iris fitting and eyeball model, and the method is as follows:

[0036] Locate the pupil center within the region of interest of the eye and fit an ellipse geometry feature based on the iris edge;

[0037] According to the predefined eyeball geometric model, a three-dimensional spatial relationship between the pupil center and the eyeball center is established;

[0038] A normalized sight direction vector is calculated based on the three-dimensional spatial relationship.

[0039] Another aspect of the present invention provides a pilot eye tracking system based on multi-camera fusion, and a pilot eye tracking method based on multi-camera fusion. The system includes:

[0040] A multi-view image and eye movement data acquisition module is configured to deploy multiple cameras in a preset area within the cockpit, map the image data collected by each camera into a unified three-dimensional spatial coordinate system to obtain multi-view images based on spatiotemporal alignment, and deploy an eye tracker to collect eye movement data in the central area of ​​the cockpit;

[0041] a region of interest and three-dimensional head pose data calculation module, configured to extract facial region images from the multi-view images, dynamically crop a region of interest including the eyes, adaptively adjust the window position and size of the region of interest according to the head deflection angle, and calculate the pilot's three-dimensional head pose data based on feature points of the facial region images;

[0042] a gaze direction vector calculation and error compensation module, configured to calculate the gaze direction vector in parallel through a geometric path and a data path, fuse the dual-path calculation results, and convert the gaze direction vector into a world coordinate system using three-dimensional head posture data to compensate for head motion errors;

[0043] The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model;

[0044] The allocation module is configured to align the error-compensated gaze direction vector with the eye movement data collected by the eye tracker in time and space, and dynamically assign weights to obtain the final gaze direction vector:

[0045] When the line of sight is in the center area of ​​the cockpit, the eye movement data collected by the eye tracker is used first. When the line of sight exceeds the center area, it switches to the final line of sight direction vector.

[0046] A third aspect of the present invention provides an electronic device, comprising:

[0047] at least one processor; and

[0048] a memory communicatively connected to at least one of the processors; wherein,

[0049] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned pilot eye tracking method based on multi-camera fusion.

[0050] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned pilot eye tracking method based on multi-camera fusion.

[0051] Beneficial effects of the present invention:

[0052] Expanding the field of view of eye tracking: Through multi-camera collaboration and 3D spatial mapping, the field of view limitations of a single camera are overcome, enabling complete capture of the pilot's wide-area scanning behavior, effectively covering visual activities in the sides and edge areas of the cockpit, and ensuring the pilot's wide-area environmental monitoring capabilities.

[0053] Improved tracking accuracy in dynamic environments: Combining real-time head posture compensation with a dual-path gaze calculation mechanism significantly reduces gaze offset errors caused by head movement, ensuring that pilots maintain stable gaze positioning accuracy during maneuvers such as turning their heads and pitching.

[0054] Enhance the synergy of multi-source data: Through spatiotemporal alignment algorithms and dynamic weight allocation strategies, seamless data fusion between the new system and traditional eye trackers is achieved, maintaining high-precision acquisition in the central area while expanding the effective tracking capability in the peripheral areas.

[0055] Optimizing adaptability to complex scenarios: Dynamically adjusted eye regions of interest and adaptive computational models can cope with varying lighting conditions, head deflection angles, and cockpit layouts, ensuring system robustness in real-world flight environments.

[0056] Reduce the cognitive load of human-computer interaction: Through high-precision line of sight positioning and error compensation mechanism, the pilot is prevented from making additional operational corrections due to system misjudgment, maintaining a natural human-computer interaction mode and continuity of situational awareness.

[0057] Supports accurate flight training assessment: Providing a complete visual behavior data chain, accurately recording the pilot's attention distribution and scanning patterns across the entire cockpit, providing a reliable basis for training quality assessment and operational optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0059] Figure 1 This is an overall flow chart of a pilot eye tracking method based on multi-camera fusion according to the present invention. DETAILED DESCRIPTION

[0060] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0061] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] The present invention provides a pilot eye tracking method based on multi-camera fusion, the method comprising:

[0063] Step S10: deploy multiple cameras in a preset area within the cockpit, map the image data collected by each camera into a unified three-dimensional spatial coordinate system to obtain a multi-view image based on spatiotemporal alignment, and deploy an eye tracker to collect eye movement data in the central area of ​​the cockpit;

[0064] Step S20, extracting facial region images from the multi-view images, and dynamically cropping a region of interest (ROI) including the eyes, adaptively adjusting the window position and size of the ROI according to the head deflection angle, and calculating three-dimensional head posture data of the pilot based on feature points of the facial region images;

[0065] Step S30, calculating the gaze direction vector in parallel through the geometric path and the data path, fusing the dual-path calculation results, and converting the gaze direction vector to the world coordinate system using the three-dimensional head posture data to compensate for head motion errors;

[0066] The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model;

[0067] Step S40: align the error-compensated gaze direction vector with the eye movement data collected by the eye tracker in time and space, and dynamically assign weights to obtain the final gaze direction vector:

[0068] When the line of sight is in the center area of ​​the cockpit, the eye movement data collected by the eye tracker is used first. When the line of sight exceeds the center area, it switches to the final line of sight direction vector.

[0069] In order to more clearly illustrate the pilot eye tracking method based on multi-camera fusion of the present invention, the following is combined with Figure 1 Each step in the embodiment of the present invention is described in detail, including step S10 to step S40, and each step is described in detail as follows:

[0070] Step S10: deploy multiple cameras in a preset area within the cockpit, map the image data collected by each camera into a unified three-dimensional spatial coordinate system to obtain a multi-view image based on spatiotemporal alignment, and deploy an eye tracker to collect eye movement data in the central area of ​​the cockpit;

[0071] In this embodiment, multiple sensors are deployed within the cockpit to comprehensively capture the pilot's visual information. First, a non-contact eye tracker is deployed below the sun visor or PFD / ND instrument to collect eye movement data directly in front of the pilot. This deployment leverages the high accuracy of existing eye trackers in a central viewing angle.

[0072] Second, four cameras are deployed in key cockpit areas, including the overhead panel, central instrument panel, center console, and side windows. These cameras capture the pilot's face and eyes from different perspectives, aiming to provide a wider field of view, particularly capturing side glances. The specific number and placement of the cameras can be adjusted to suit different cockpit layouts and requirements, with the goal of maximizing coverage of the pilot's visual range.

[0073] In this embodiment, the image data collected by each camera is mapped to a unified three-dimensional space coordinate system by:

[0074] Calibrate parameters of each camera to determine internal and external parameters;

[0075] The internal parameters include focal length, principal point coordinates and lens distortion parameters, and the internal parameter matrix of the camera is constructed; the external parameters include the rotation matrix and translation vector of the camera in the world coordinate system;

[0076] The world coordinates of any 3D space point are first converted to the camera coordinate system through the rotation matrix and translation vector, and then the 3D coordinates in the camera coordinate system are projected onto the 2D image plane according to the intrinsic parameter matrix;

[0077] Establish a transformation relationship between different camera coordinate systems. Through the rotation matrix and translation vector from one camera to another, the three-dimensional point coordinates in one camera coordinate system are converted into the three-dimensional coordinates in the other camera coordinate system. Finally, the three-dimensional data collected by all cameras are mapped to the same global three-dimensional coordinate system to achieve cross-view data fusion.

[0078] Since multiple groups of cameras are distributed in different positions in the cockpit, the image data they collect are in different local coordinate systems. In order to achieve effective fusion and analysis of multi-view image data, the present invention constructs a spatial coordinate system mapping network. The main function of this network is to map image data from different cameras into a unified three-dimensional spatial coordinate system. This usually requires calibration of each camera to determine its internal parameters (such as focal length, principal point) and external parameters (such as position, posture). The calibration process can use standard camera calibration methods, such as methods based on calibration objects such as chessboards. Through the spatial coordinate system mapping network, precise alignment of multi-view images in time and space can be achieved, laying the foundation for subsequent feature extraction and line of sight calculation.

[0079] Camera projection model: any 3D space point The projection in the camera coordinate system can be expressed as:

[0080] ;

[0081] ;

[0082] in, is the three-dimensional point coordinate in the camera coordinate system, 、 is the pixel coordinate after projection.

[0083] in, is the rotation matrix, is the translation vector, is the camera intrinsic parameter matrix:

[0084] ;

[0085] in, is the focal length in the x and y directions (in pixels), which is determined by the physical focal length and the sensor pixel density. is the principal point coordinate (in pixels), is the center of the image, and represents the intersection of the optical axis and the image plane. s is the tilt factor, which describes the non-perpendicularity of the x-axis and y-axis in the image coordinate system.

[0086] Multi-camera coordinate transformation: The relationship between different cameras can be established through the transformation matrix:

[0087] ;

[0088] in, and is the rotation matrix and translation vector from camera 1 to camera 2, is the three-dimensional point coordinate in the camera 1 coordinate system, is the 3D point coordinate in the camera 2 coordinate system after conversion.

[0089] Time synchronization model: For time synchronization, a linear interpolation method is used to align sensor data with different sampling rates:

[0090] ;

[0091] in, , , Represents data, is the interpolation weight factor, are two adjacent raw data sampling time points, t The target time point that needs to be synchronized;

[0092] Step S20, extracting facial region images from the multi-view images, and dynamically cropping a region of interest (ROI) including the eyes, adaptively adjusting the window position and size of the ROI according to the head deflection angle, and calculating three-dimensional head posture data of the pilot based on feature points of the facial region images;

[0093] In this embodiment, the facial region image is extracted from the multi-view image by:

[0094] A hybrid deep network model is used to extract facial area images from the multi-view images, and the hybrid deep network model is jointly constructed based on a ResNet50 network model and a ViT network model.

[0095] In this embodiment, after obtaining synchronized image data from multiple perspectives, the feature extraction layer first needs to accurately locate the pilot's facial area. The present invention uses a ResNet50+ViT hybrid model for facial positioning, with the goal of achieving a positioning accuracy of ±3 pixels. ResNet50 is a classic deep convolutional neural network with powerful feature extraction capabilities, which can effectively capture local texture and structural information in images. The VisionTransformer (ViT) model is based on the self-attention mechanism and is good at capturing global contextual information in images. Combining the ResNet50 and ViT models can fully utilize the advantages of both, improve the robustness and accuracy of facial positioning, and stably detect faces even under complex conditions such as changes in lighting and head posture.

[0096] Hybrid deep network: A hybrid network architecture consisting of ResNet50 and ViT is used, which can be expressed as:

[0097] ;

[0098] ;

[0099] ;

[0100] in, is the region of interest, yes feature, It is the feature enhanced by Transformer, and finally outputs the facial bounding box and key point coordinates. is the pixel coordinate of the center point of the bounding box, are the width and height of the bounding box, is the pixel coordinate set of facial key points (such as eyes, nose, corners of mouth, etc.), is the number of key points, and its specific value is defined by the task.

[0101] After facial localization, a region of interest (ROI) containing the eyes needs to be cropped from the facial image. Because pilots may move their heads during flight, traditional fixed ROI cropping methods may not effectively capture the entire eye region. To address this issue, the present invention designs a dynamic eye ROI cropping algorithm. This algorithm dynamically adjusts the size, position, and angle of the cropping frame based on head pose estimation, effectively capturing the eye region even when the head is tilted ±45 degrees. Dynamic cropping ensures that subsequent eye feature extraction and gaze calculation models receive high-quality eye image data, improving overall system performance.

[0102] In this embodiment, the method for dynamically cropping the region of interest including the eyes is as follows:

[0103] Set the initial reference width and height of the eye's region of interest. The initial reference width and height are pre-configured based on the pilot's eye anatomy and camera resolution.

[0104] The width and height of the region of interest are proportionally expanded or contracted according to the horizontal deflection angle and vertical pitch angle of the pilot's head;

[0105] Calculate the eye center position based on the detected eye corner key point coordinates;

[0106] Generate a rotation matrix based on the head deflection angle, perform a rotation transformation on the original offset between the preset eye center and the center of the region of interest, and obtain the dynamic offset after the head deflection;

[0107] The dynamic offset is superimposed on the eye center coordinates to determine the final center position of the region of interest window;

[0108] The rotation direction of the ROI window is synchronously adjusted according to the head deflection angle, so that the cropping frame is always aligned with the actual deflection direction of the eyeball, ensuring complete coverage of the eye area under extreme head postures.

[0109] The width and height of the region of interest are proportionally expanded or contracted according to the horizontal deflection angle and vertical pitch angle of the pilot's head, specifically:

[0110] According to the horizontal deflection angle of the pilot's head, based on the reference width, the width of the region of interest is proportionally expanded or contracted. The expansion amplitude is proportional to the absolute value of the sine of the deflection angle, and the scaling factor controls the maximum adjustment amplitude.

[0111] According to the vertical pitch angle of the head, the height of the region of interest is proportionally expanded or contracted based on the reference height. The expansion amplitude is proportional to the absolute value of the sine of the pitch angle, and the scaling factor is independent of the horizontal direction.

[0112] Specifically, this embodiment is based on the key points of the corners of the eyes and and head posture angle , dynamically adjust the ROI window size and position:

[0113] Width adjustment: ;

[0114] Height Adjustment: ;

[0115] ;

[0116] in, is the base window size, are the horizontal and vertical scaling factors, respectively. is the rotation matrix, is the initial offset, which indicates the preset offset of the ROI center relative to the eye corner key point. The coordinates of the key points of the corners of the eyes are usually the average of the left and right corners or the coordinates of the corners of one side, which serve as the initial positioning reference of the ROI. is the horizontal position of the ROI center in the image coordinate system, is the vertical position of the ROI center in the image coordinate system.

[0117] In this embodiment, the three-dimensional head posture data of the pilot is calculated based on the feature points of the facial region image, and the method is as follows:

[0118] Predefining a three-dimensional facial model including key points of the corners of the eyes, the tip of the nose, and the corners of the mouth, wherein each key point in the model has known three-dimensional spatial coordinates; detecting two-dimensional facial key points corresponding to the three-dimensional model from the facial region image, and establishing a matching relationship between the three-dimensional model points and the two-dimensional image points;

[0119] Through perspective projection geometry, the 3D model points are projected onto the theoretical positions of the 2D image plane, and the positional deviation between the theoretical positions and the actual detection points is calculated;

[0120] The rotation matrix and translation vector are used as optimization variables, and their values ​​are iteratively adjusted to minimize the sum of squared projection deviations of all matching key points, thereby solving the optimal head posture parameters.

[0121] The pilot's head yaw angle, pitch angle, and roll angle are extracted from the optimized rotation matrix as the pilot's three-dimensional head posture data.

[0122] Specifically, to accurately compensate for the impact of head motion on gaze estimation, this invention introduces a 3D head pose estimation module. This module aims to estimate the pilot's head's three-dimensional pose, including yaw, pitch, and roll, with an error of less than 2 degrees in each direction. Head pose estimation can be achieved using a variety of methods, such as leveraging the 3D positional information of facial key points (such as the corners of the eyes, mouth, and nose tip) combined with model fitting or deep learning techniques. High-precision head pose estimation is crucial for head motion compensation in the subsequent gaze calculation layer.

[0123] Using known 3D facial model points and its 2D image projection The correspondence between them is used to estimate the head posture through the PnP algorithm:

[0124] ;

[0125] in is the projection function, is the camera internal parameter, and are the rotation matrix and translation vector representing the head pose, respectively.

[0126] Euler Angle Conversion: From Rotation Matrix Extract the head pose expressed in Euler angles:

[0127] Yaw angle ;

[0128] Pitch angle ;

[0129] Roll angle ;

[0130] in is the rotation matrix elements.

[0131] Step S30, calculating the gaze direction vector in parallel through the geometric path and the data path, fusing the dual-path calculation results, and converting the gaze direction vector to the world coordinate system using the three-dimensional head posture data to compensate for head motion errors;

[0132] The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model;

[0133] The gaze calculation layer is responsible for calculating the pilot's gaze direction based on extracted eye features and head posture information. This paper proposes a dual-path fusion network structure and introduces the HFA-Net (Head-Fusion Attention Network) to further improve the accuracy and robustness of gaze estimation. This includes both a geometric path and a data path, aiming to leverage the advantages of both model-based and data-based approaches.

[0134] Geometric path:

[0135] The geometric path calculates the gaze vector based on an iris ellipse fit. This method first requires the precise location of the pupil center, with a positioning error of less than 0.5 mm. Then, by fitting the iris ellipse, the eye's rotation angle is estimated. Combined with known eye model parameters, the gaze vector in three-dimensional space is calculated. The geometric path is a direct, physics-based gaze estimation method, and its accuracy relies heavily on the accuracy of pupil center location.

[0136] The gaze direction vector is generated based on iris fitting and eyeball model. The method is:

[0137] Step S31, locating the pupil center within the region of interest of the eye, and fitting elliptical geometric features based on the iris edge;

[0138] Step S32, establishing a three-dimensional spatial relationship between the pupil center and the eye center according to a predefined eyeball geometric model;

[0139] Step S33: Calculate a normalized sight direction vector based on the three-dimensional spatial relationship.

[0140] The method for locating the pupil center within the region of interest of the eye is as follows:

[0141] In step S31, the pupil center coordinates are located within the eye region of interest by using the grayscale centroid method or morphological operations, and the three-dimensional spatial positioning error of the pupil center is required to be less than 0.5 mm;

[0142] The grayscale centroid method determines the pupil center by calculating the intensity-weighted average of pixel coordinates within the pupil area, and the morphological operation extracts the pupil geometric center through binarization and connected domain analysis;

[0143] Locate the pupil center using grayscale centroid method or morphological operations :

[0144] ;

[0145] in, is the pupil area, is the pixel intensity, expressed as the image at coordinates ( x,y ) at the brightness value.

[0146] In step S31, the ellipse geometric features are fitted based on the iris edge, and the method is as follows:

[0147] Perform ellipse fitting on the iris edge and use random sampling consensus algorithm or Hough transform to select the optimal ellipse parameters, including the ellipse center coordinates, major axis length, minor axis length and rotation angle;

[0148] The ellipse parameters are used to characterize the deformation of the iris in the image plane and are combined with the camera intrinsic parameters to infer the three-dimensional spatial orientation of the iris;

[0149] Ellipse iris fitting: Fit the iris ellipse by Hough transform or RANSAC method:

[0150] ;

[0151] in is the center of the ellipse, is the length of the major and minor axes, is the ellipse rotation angle.

[0152] In step S32, a three-dimensional spatial relationship between the pupil center and the eye center is established based on a predefined eyeball geometric model, and the method is as follows:

[0153] Based on a predefined single-sphere or double-sphere eyeball model, the three-dimensional spatial relative position relationship between the pupil center and the eyeball center is established;

[0154] Normalizing and calculating the sight direction vector according to the displacement vector of the pupil center relative to the eye center in the eye model;

[0155] Calculate the gaze vector based on a single-sphere or double-sphere eyeball model :

[0156] ;

[0157] in is the position of the pupil center in 3D space, It is the center of the eyeball.

[0158] In step S33, a normalized sight direction vector is calculated based on the three-dimensional spatial relationship.

[0159] Subtract the coordinates of the pupil center in three-dimensional space from the coordinates of the eyeball center to obtain the unnormalized sight line vector;

[0160] The vector is normalized and the three-dimensional sight direction calculated by the geometric path is output.

[0161] Data Path:

[0162] The data path uses a deep learning regression model. This model is pre-trained on a large dataset of eye images to learn the mapping between eye image features and 3D gaze direction. In practice, the data path directly takes a cropped eye ROI image as input and outputs the corresponding 3D gaze direction. The advantage of the data path is that it can learn complex patterns from large amounts of data and is adaptable to individual differences and environmental changes.

[0163] Using a deep CNN structure, it can be expressed as:

[0164] ;

[0165] ;

[0166] in is the eye ROI image, is the extracted feature vector, is the predicted gaze vector, and are network parameters.

[0167] Loss Function : Train the network with angular error loss:

[0168] ;

[0169] in is the true view vector.

[0170] The 3D head pose data is used to transform the gaze direction vector into the world coordinate system to compensate for head motion errors. Specifically:

[0171] To further improve the accuracy and robustness of gaze estimation, this paper uses HFA-Net (Head-Fusion Attention Network). HFA-Net mainly implements the following two functions: the first is head posture compensation, and the second is the attention mechanism to suppress light interference. Specifically:

[0172] Head pose compensation: HFA-Net uses the 3D head pose information estimated in the feature extraction layer to compensate the gaze vector calculated by the dual-path fusion network. Specifically, it can convert the gaze vector from the camera coordinate system to the world coordinate system, thereby eliminating the gaze estimation error caused by head movement.

[0173] The relative gaze vector is transformed into the world coordinate system using the head pose matrix:

[0174] ;

[0175] in is the sight vector relative to the head coordinate system, is the view vector in the world coordinate system.

[0176] Attention Mechanism Suppresses Illumination Interference: HFA-Net introduces an attention mechanism that learns to assign different weights to different regions of the eye image. Regions susceptible to illumination interference (e.g., high-light reflections) are assigned lower weights, while regions containing valid eye information are assigned higher weights. This effectively suppresses interference with gaze estimation caused by complex lighting environments, improving system robustness.

[0177] Adopt channel attention mechanism to suppress light interference:

[0178] ;

[0179] ;

[0180] in is the feature map, is the attention mask, is element-wise multiplication.

[0181] Fusion strategy: Using learning-based weighted fusion:

[0182] ;

[0183] in and is the fusion weight obtained by learning, satisfying .

[0184] Attention mechanism suppresses light interference:

[0185] Analyze the feature maps of the region of interest through the attention mechanism to identify areas susceptible to illumination interference and areas with effective eye information;

[0186] Assign low weights to highlight reflection areas and high weights to effective eye feature areas to generate channel attention masks;

[0187] Multiplying the channel attention mask by the original feature map element by element to suppress the feature response of the illumination interference area and retain the effective eye features;

[0188] Perform a learning-based weighted fusion of the sight vectors calculated by the geometric path and the sight vectors calculated by the data path, and dynamically adjust the weight coefficients of the two;

[0189] The sum of the constraint weight coefficients is 1. When the illumination interference is significant, the weight of the data path is increased to enhance the robustness of the model. When the eye features are clear, the weight of the geometric path is increased to utilize the accuracy of the physical model.

[0190] The dual-path sight vectors are weighted and summed according to the dynamic weights to output the final sight direction.

[0191] Step S40: align the error-compensated gaze direction vector with the eye movement data collected by the eye tracker in time and space, and dynamically assign weights to obtain the final gaze direction vector:

[0192] When the line of sight is in the center area of ​​the cockpit, the eye movement data collected by the eye tracker is used first. When the line of sight exceeds the center area, it switches to the final line of sight direction vector.

[0193] This paper develops a multi-source data fusion algorithm for spatiotemporally aligning eye movement data collected by our system with data from traditional eye trackers. Because the two systems may have different sampling frequencies and coordinate systems, temporal synchronization and spatial coordinate conversion are necessary. Data fusion leverages the strengths of both systems. For example, while traditional eye trackers may offer higher accuracy within a central viewing angle, our system can provide a wider field of view.

[0194] Through the transformation matrix Convert the eye tracker data to a unified coordinate system:

[0195] ;

[0196] in, is the original data vector in the eye tracker coordinate system, including information such as eye position and sight direction. is the data vector after conversion to the world coordinate system, aligned with the global coordinate system, Represents the transformation matrix from the eye tracker coordinate system (tracker) to the world coordinate system (world).

[0197] Data synchronization is achieved through timestamp matching and interpolation:

[0198] ;

[0199] in, , represents the interpolation weight factor, are the adjacent sampling time points of the eye tracker data, For the time point The raw data vector of the eye tracker collected below, Interpolate is the interpolation function.

[0200] To more effectively utilize multi-source data, the present invention constructs a dynamic weighting model. This model dynamically adjusts the weighting of data from our system and traditional eye trackers based on the pilot's gaze. Specifically, when the pilot's gaze is in the center of the cockpit, the model prioritizes data from traditional eye trackers, as traditional eye trackers typically have higher accuracy in this area. When the gaze moves to the edge of the cockpit, the model switches to data from our system, as our system has more comprehensive coverage in this area. This dynamic weighting strategy achieves optimal eye tracking performance across the entire field of view.

[0201] Calculate weights based on head posture angle and gaze direction angle :

[0202] ;

[0203] in It is an adjustable parameter.

[0204] The final output is synthesized according to the dynamic weights as the final sight direction vector :

[0205] ;

[0206] when When it is close to 0, the eye tracker data is used first; when When it is close to 1, the data of this system is used first, i.e. is the sight direction vector after error compensation.

[0207] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0208] A second embodiment of the present invention provides a pilot eye tracking system based on multi-camera fusion, based on a pilot eye tracking method based on multi-camera fusion, and includes:

[0209] A multi-view image and eye movement data acquisition module is configured to deploy multiple cameras in a preset area within the cockpit, map the image data collected by each camera into a unified three-dimensional spatial coordinate system to obtain multi-view images based on spatiotemporal alignment, and deploy an eye tracker to collect eye movement data in the central area of ​​the cockpit;

[0210] a region of interest and three-dimensional head pose data calculation module, configured to extract facial region images from the multi-view images, dynamically crop a region of interest including the eyes, adaptively adjust the window position and size of the region of interest according to the head deflection angle, and calculate the pilot's three-dimensional head pose data based on feature points of the facial region images;

[0211] a gaze direction vector calculation and error compensation module, configured to calculate the gaze direction vector in parallel through a geometric path and a data path, fuse the dual-path calculation results, and convert the gaze direction vector into a world coordinate system using three-dimensional head posture data to compensate for head motion errors;

[0212] The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model;

[0213] The allocation module is configured to align the error-compensated gaze direction vector with the eye movement data collected by the eye tracker in time and space, and dynamically assign weights to obtain the final gaze direction vector:

[0214] When the line of sight is in the center of the cockpit, the eye movement data collected by the eye tracker is preferentially used. When the line of sight exceeds the center, the final line of sight direction vector is switched. Those skilled in the art will clearly understand that for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be referred to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0215] It should be noted that the pilot eye tracking system based on multi-camera fusion provided in the above embodiment is merely illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into a single module or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are merely for the purpose of distinguishing the modules or steps and are not to be considered as undue limitations of the present invention.

[0216] An electronic device according to a third embodiment of the present invention includes:

[0217] at least one processor; and

[0218] a memory communicatively connected to at least one of the processors; wherein,

[0219] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned pilot eye tracking method based on multi-camera fusion.

[0220] A fourth embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are configured to be executed by the computer to implement the aforementioned pilot eye tracking method based on multi-camera fusion.

[0221] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0222] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0223] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0224] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0225] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A pilot eye tracking method based on multi-camera fusion, characterized in that: The method includes: Multiple cameras are deployed in a preset area within the cockpit. The image data collected by each camera is mapped into a unified three-dimensional spatial coordinate system to obtain multi-view images based on spatiotemporal alignment. An eye tracker is also deployed to collect eye movement data in the central area of ​​the cockpit. Extracting facial region images from the multi-view images and dynamically cropping a region of interest including the eyes, adaptively adjusting a window position and size of the region of interest according to a head deflection angle, and calculating three-dimensional head posture data of the pilot based on feature points of the facial region images; Dynamically crop the region of interest containing the eyes using the following method: Set the initial reference width and height of the eye's region of interest. The initial reference width and height are pre-configured based on the pilot's eye anatomy and camera resolution. The width and height of the region of interest are proportionally expanded or contracted according to the horizontal deflection angle and vertical pitch angle of the pilot's head; Calculate the eye center position based on the detected eye corner key point coordinates; Generate a rotation matrix based on the head deflection angle, perform a rotation transformation on the original offset between the preset eye center and the center of the region of interest, and obtain the dynamic offset after the head deflection; The dynamic offset is superimposed on the eye center coordinates to determine the final center position of the region of interest window; The rotation direction of the ROI window is adjusted synchronously according to the head deflection angle, so that the cropping frame is always aligned with the actual deflection direction of the eyeball, ensuring complete coverage of the eye area under extreme head postures; The gaze direction vector is calculated in parallel through the geometric path and the data path, the dual-path calculation results are integrated, and the gaze direction vector is converted to the world coordinate system using the 3D head posture data to compensate for head motion errors. The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model; The gaze direction vector is generated based on iris fitting and eyeball model. The method is: Locate the pupil center within the region of interest of the eye and fit an ellipse geometry feature based on the iris edge; According to the predefined eyeball geometric model, a three-dimensional spatial relationship between the pupil center and the eyeball center is established; Calculating a normalized sight direction vector based on the three-dimensional spatial relationship; The error-compensated gaze direction vector is spatially and temporally aligned with the eye movement data collected by the eye tracker, and the weights are dynamically assigned to obtain the final gaze direction vector: When the line of sight is in the center area of ​​the cockpit, the eye movement data collected by the eye tracker is used first. When the line of sight exceeds the center area, it switches to the final line of sight direction vector.

2. The pilot eye tracking method based on multi-camera fusion according to claim 1, characterized in that: The image data collected by each camera is mapped to a unified three-dimensional space coordinate system as follows: Calibrate parameters of each camera to determine internal and external parameters; The internal parameters include focal length, principal point coordinates and lens distortion parameters, and the internal parameter matrix of the camera is constructed; the external parameters include the rotation matrix and translation vector of the camera in the world coordinate system; The world coordinates of any 3D space point are first converted to the camera coordinate system through the rotation matrix and translation vector, and then the 3D coordinates in the camera coordinate system are projected onto the 2D image plane according to the intrinsic parameter matrix; Establish a transformation relationship between different camera coordinate systems. Through the rotation matrix and translation vector from one camera to another, the three-dimensional point coordinates in one camera coordinate system are converted into the three-dimensional coordinates in the other camera coordinate system. Finally, the three-dimensional data collected by all cameras are mapped to the same global three-dimensional coordinate system to achieve cross-view data fusion.

3. The pilot eye tracking method based on multi-camera fusion according to claim 1, characterized in that: Extracting facial region images from the multi-view images, the method is as follows: A hybrid deep network model is used to extract facial area images from the multi-view images, and the hybrid deep network model is jointly constructed based on a ResNet50 network model and a ViT network model.

4. The pilot eye tracking method based on multi-camera fusion according to claim 1, characterized in that: The width and height of the region of interest are proportionally expanded or contracted according to the horizontal deflection angle and vertical pitch angle of the pilot's head, specifically: According to the horizontal deflection angle of the pilot's head, based on the reference width, the width of the region of interest is proportionally expanded or contracted. The expansion amplitude is proportional to the absolute value of the sine of the deflection angle, and the scaling factor controls the maximum adjustment amplitude. According to the vertical pitch angle of the head, the height of the region of interest is proportionally expanded or contracted based on the reference height. The expansion amplitude is proportional to the absolute value of the sine of the pitch angle, and the scaling factor is independent of the horizontal direction.

5. The pilot eye tracking method based on multi-camera fusion according to claim 1, characterized in that: The pilot's three-dimensional head posture data is calculated based on the feature points of the facial area image. The method is as follows: Predefining a three-dimensional facial model including key points of the corners of the eyes, the tip of the nose, and the corners of the mouth, wherein each key point in the three-dimensional facial model has known three-dimensional spatial coordinates; detecting two-dimensional facial key points corresponding to the three-dimensional model from the facial region image, and establishing a matching relationship between the three-dimensional model points and the two-dimensional image points; Through perspective projection geometry, the 3D model points are projected onto the theoretical positions of the 2D image plane, and the positional deviation between the theoretical positions and the actual detection points is calculated; The rotation matrix and translation vector are used as optimization variables, and their values ​​are iteratively adjusted to minimize the sum of squared projection deviations of all matching key points, thereby solving the optimal head posture parameters. The pilot's head yaw angle, pitch angle, and roll angle are extracted from the optimized rotation matrix as the pilot's three-dimensional head posture data.

6. A pilot eye tracking system based on multi-camera fusion, used to implement the pilot eye tracking method based on multi-camera fusion according to any one of claims 1 to 5, characterized in that: The system includes: A multi-view image and eye movement data acquisition module is configured to deploy multiple cameras in a preset area within the cockpit, map the image data collected by each camera into a unified three-dimensional spatial coordinate system to obtain multi-view images based on spatiotemporal alignment, and deploy an eye tracker to collect eye movement data in the central area of ​​the cockpit; a region of interest and three-dimensional head pose data calculation module, configured to extract facial region images from the multi-view images, dynamically crop a region of interest including the eyes, adaptively adjust the window position and size of the region of interest according to the head deflection angle, and calculate the pilot's three-dimensional head pose data based on feature points of the facial region images; a gaze direction vector calculation and error compensation module, configured to calculate the gaze direction vector in parallel through a geometric path and a data path, fuse the dual-path calculation results, and convert the gaze direction vector into a world coordinate system using three-dimensional head posture data to compensate for head motion errors; The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model; The allocation module is configured to align the error-compensated gaze direction vector with the eye movement data collected by the eye tracker in time and space, and dynamically assign weights to obtain the final gaze direction vector: When the line of sight is in the center area of ​​the cockpit, the eye movement data collected by the eye tracker is used first. When the line of sight exceeds the center area, it switches to the final line of sight direction vector.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the pilot eye tracking method based on multi-camera fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the pilot eye tracking method based on multi-camera fusion as described in any one of claims 1 to 5.

Images