Pilot eye movement tracking method, system and device based on multi-camera fusion and storage medium

Abstract

The invention belongs to the field of eye movement data tracking, particularly relates to a pilot eye movement tracking method, system and equipment based on multi-camera fusion and a storage medium, and aims to solve the problems that in the prior art, the visual angle of a traditional eye tracker is limited, and sight estimation errors are caused by head movement. The method comprises the following steps of: constructing a multi-view visual perception network by deploying multiple cameras and eye trackers in a cockpit; multi-source data space-time alignment is achieved through space coordinate system mapping and time synchronization, and head motion errors are compensated by combining dynamic cutting of eye interested areas and three-dimensional head posture estimation. The geometric path generates a physical driving vector based on iris fitting and an eyeball model, and the data path regresses the vector through a deep learning model; and converting to a world coordinate system through a head posture after fusion. High-precision data of the eye tracker is preferentially adopted in the central area, and field-of-view data of the system is expanded in the edge area. According to the invention, the tracking precision under field angle coverage and motion interference is obviously improved.

Description

Background Art

[0002] Traditional eye trackers usually rely on a single or a few cameras, and their field of view (FOV) is often limited, with a typical value less than 120 degrees. However, the human visual system has a much broader field of view, approaching 200 degrees horizontally and about 135 degrees vertically. This difference in the field of view makes it difficult for traditional eye trackers to capture large-scale visual search behaviors such as lateral saccades of pilots in the cockpit, and these behaviors are crucial for pilots to timely detect potential threats, monitor the surrounding environment, and maintain comprehensive situational awareness.

[0003] The operations of pilots in the cockpit are not static, and head movement is the norm. However, head movement significantly affects the accuracy of eye movement tracking. If the eye movement tracking system fails to effectively compensate for head movement, it will lead to obvious errors in gaze estimation, with an average deviation of up to 8 - 12 degrees. The gaze calculation error will affect the assessment of attention allocation in pilot training. For example, during landing, it is necessary to pay attention to whether the pilot allocates attention to areas such as the runway center line, PAPI lights, speed, altitude, descent rate, and attitude, and these areas are relatively close in the visual scene or instrument, which may result in the inability to accurately detect problems with the pilot's attention allocation during training.

[0004] Based on this, the present invention proposes a pilot eye movement tracking method, system, device, and storage medium based on multi-camera fusion. Summary of the Invention

[0005] To solve the above problems in the prior art, that is, the limited viewing angle of traditional eye trackers in the prior art and the gaze estimation error caused by head movement, which affects the assessment of attention allocation in pilot training, the present invention provides a pilot eye movement tracking method, system, device, and storage medium based on multi-camera fusion.

[0006] In the first aspect of the present invention, a pilot eye movement tracking method based on multi-camera fusion is proposed. The method includes: Deploy multiple cameras in a preset area in the cockpit, map the image data collected by each camera into a unified three-dimensional space coordinate system to obtain multi-view images based on spatio-temporal alignment, and deploy an eye tracker for collecting eye movement data in the central area of the cockpit; Extract the facial region image from the multi-view images, dynamically crop the region of interest (ROI) containing the eyes, the ROI adaptively adjusts the window position and size according to the head deflection angle, and calculates the three-dimensional head pose data of the pilot based on the feature points of the facial region image; Calculate the line-of-sight direction vector in parallel through geometric and data paths, fuse the calculation results of the two paths, and use the three-dimensional head pose data to transform the line-of-sight direction vector into the world coordinate system to compensate for the head movement error; Among them, the geometric path includes generating a line-of-sight direction vector based on iris fitting and an eyeball model for the region of interest; the data path is to extract the line-of-sight direction vector from the region of interest through a deep learning model; Perform spatio-temporal alignment on the line-of-sight direction vector after error compensation and the eye movement data collected by the eye tracker, and dynamically assign weights to obtain the final line-of-sight direction vector: When the line of sight is in the central area of the cockpit, preferentially use the eye movement data collected by the eye tracker, and switch to the final line-of-sight direction vector when the line of sight exceeds the central area.

[0007] Furthermore, map the image data collected by each camera to a unified three-dimensional space coordinate system, and the method is as follows: Calibrate the parameters of each camera to determine the internal parameters and external parameters; Among them, the internal parameters include the focal length, the coordinates of the principal point, and the lens distortion parameters, and construct the internal parameter matrix of the camera; the external parameters include the rotation matrix and translation vector of the camera in the world coordinate system; First, transform the world coordinates of any three-dimensional space point to the camera coordinate system through the rotation matrix and translation vector, and then project the three-dimensional coordinates in this camera coordinate system to the two-dimensional image plane according to the internal parameter matrix; Establish the conversion relationship between different camera coordinate systems, and convert the three-dimensional point coordinates in one camera coordinate system to the three-dimensional coordinates in another camera coordinate system through the rotation matrix and translation vector from one camera to another, and finally map the three-dimensional data collected by all cameras to the same global three-dimensional coordinate system to achieve cross-view data fusion.

[0008] Furthermore, extract the facial region images in the multi-view images, and the method is as follows: Use a hybrid deep network model to extract the facial region images in the multi-view images, and the hybrid deep network model is jointly constructed based on the ResNet50 network model and the ViT network model.

[0009] Furthermore, dynamically crop the region of interest containing the eyes, and the method is as follows: Set the initial reference width and height of the region of interest of the eyes, and the initial reference width and height are pre-configured according to the pilot's eye anatomical characteristics and the camera resolution; According to the horizontal deflection angle and vertical pitch angle of the pilot's head, expand or contract the width and height of the region of interest proportionally; Calculate the eye center position based on the detected key point coordinates of the two eye corners; Generate a rotation matrix according to the head deflection angle, and perform a rotation transformation on the original offset between the preset eye center and the center of the region of interest to obtain the dynamic offset after head deflection; Superimpose the dynamic offset on the eye center coordinates to determine the final center position of the region of interest window; Synchronously adjust the rotation direction of the region of interest window according to the head deflection angle, so that the cropping frame is always aligned with the actual deflection direction of the eyeball, ensuring complete coverage of the eye region in extreme head postures.

[0010] Further, according to the horizontal deflection angle and vertical pitch angle of the pilot's head, proportionally expand or contract the width and height of the region of interest. Specifically: According to the horizontal deflection angle of the pilot's head, based on the reference width, proportionally expand or contract the width of the region of interest. The expansion amplitude is proportional to the absolute value of the sine of the deflection angle, and the scaling coefficient controls the maximum adjustment amplitude; According to the vertical pitch angle of the head, based on the reference height, proportionally expand or contract the height of the region of interest. The expansion amplitude is proportional to the absolute value of the sine of the pitch angle, and the scaling coefficient is independent of the horizontal direction.

[0011] Further, calculate the three-dimensional head pose data of the pilot based on the feature points of the facial region image. The method is as follows: Pre-define a three-dimensional facial model including key points of the eye corners, nose tip, and mouth corners. Each key point in the model has known three-dimensional spatial coordinates; detect two-dimensional facial key points corresponding to the three-dimensional model from the facial region image, and establish a matching relationship between the three-dimensional model points and the two-dimensional image points; Through the perspective projection geometric relationship, project the three-dimensional model points to the theoretical position on the two-dimensional image plane, and calculate the position deviation between the theoretical position and the actual detected points; Take the rotation matrix and translation vector as optimization variables, and iteratively adjust their values to minimize the sum of the squared projection deviations of all matching key points, and solve the optimal head pose parameters; Extract the yaw angle, pitch angle, and roll angle of the pilot's head from the optimized rotation matrix as the three-dimensional head pose data of the pilot.

[0012] Further, generate a line-of-sight direction vector based on iris fitting and an eyeball model. The method is as follows: Locate the pupil center within the region of interest of the eye, and fit the elliptical geometric features based on the iris edge; According to the pre-defined eyeball geometric model, establish a three-dimensional spatial relationship between the pupil center and the eyeball center; Calculate the normalized line-of-sight direction vector based on the three-dimensional spatial relationship.

[0013] On the other hand, the present invention proposes a pilot eye movement tracking system based on multi-camera fusion, based on a pilot eye movement tracking method based on multi-camera fusion. The system includes: A multi-view image and eye movement data acquisition module, which is configured to deploy multiple cameras in a preset area inside the cockpit, map the image data collected by each camera into a unified three-dimensional space coordinate system to obtain multi-view images based on spatio-temporal alignment, and deploy an eye tracker for eye movement data acquisition in the central area of the cockpit; A region of interest and three-dimensional head pose data calculation module, which is configured to extract the facial region image in the multi-view image, dynamically crop the region of interest containing the eyes, the region of interest adaptively adjusts the window position and size according to the head deflection angle, and calculates the three-dimensional head pose data of the pilot based on the feature points of the facial region image; A line-of-sight direction vector calculation and error compensation module, which is configured to calculate the line-of-sight direction vector in parallel through a geometric path and a data path, fuse the calculation results of the two paths, and use the three-dimensional head pose data to convert the line-of-sight direction vector to the world coordinate system to compensate for the head movement error; Wherein, the geometric path includes generating a line-of-sight direction vector based on iris fitting and an eyeball model for the region of interest; the data path is to extract the line-of-sight direction vector from the region of interest through a deep learning model; An allocation module, which is configured to perform spatio-temporal alignment on the line-of-sight direction vector after error compensation and the eye movement data collected by the eye tracker, and dynamically allocate weights to obtain the final line-of-sight direction vector: When the line of sight is in the central area of the cockpit, the eye movement data collected by the eye tracker is preferentially used, and when the line of sight exceeds the central area, it switches to the final line-of-sight direction vector.

[0014] In the third aspect of the present invention, an electronic device is proposed, including: At least one processor; and A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned pilot eye movement tracking method based on multi-camera fusion.

[0015] In the fourth aspect of the present invention, a computer-readable storage medium is proposed, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned pilot eye movement tracking method based on multi-camera fusion.

[0016] Advantages of the present invention: Expand the field of view coverage of eye tracking: Through the collaborative work of multiple cameras and three-dimensional space mapping, break through the field of view limitation of a single camera, achieve complete capture of the pilot's large-scale saccade behavior, effectively cover the visual activities in the side and edge areas of the cockpit, and ensure the pilot's wide-area environmental monitoring ability.

[0017] Improve the tracking accuracy in a dynamic environment: Combine real-time head pose compensation and dual-channel line-of-sight calculation mechanism to significantly reduce the line-of-sight deviation error caused by head movement, and ensure that the pilot can still maintain stable line-of-sight positioning accuracy during actions such as turning the head and pitching.

[0018] Enhance the collaborative efficiency of multi-source data: Through the spatio-temporal alignment algorithm and dynamic weight allocation strategy, achieve seamless data fusion between the new system and traditional eye trackers, and while maintaining high-precision acquisition in the central area, expand the effective tracking ability in the edge area.

[0019] Optimize the adaptability to complex scenarios: Dynamically adjusted regions of interest in the eyes and adaptive calculation models can cope with different lighting conditions, head deflection angles, and cockpit layout differences, and ensure the robustness of the system in the real flight environment.

[0020] Reduce the cognitive load of human-machine interaction: Through high-precision line-of-sight positioning and error compensation mechanisms, avoid additional operation corrections by the pilot due to system misjudgment, and maintain a natural human-machine interaction mode and the continuity of situation awareness.

[0021] Support accurate flight training evaluation: Provide a complete visual behavior data chain, accurately record the attention distribution and saccade patterns of the pilot throughout the cockpit, and provide a reliable basis for training quality evaluation and operation optimization. Brief Description of the Drawings

[0022] Other features, objectives, and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings: Figure 1 It is the overall flowchart of a method for pilot eye tracking based on multi-camera fusion of the present invention. Detailed Embodiment

[0023] The following further elaborates on the present application in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention and do not limit the invention. Additionally, it should be noted that for the sake of description, only parts related to the relevant invention are shown in the drawings.

[0024] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The following will elaborate on the present application in detail with reference to the drawings and embodiments.

[0025] The present invention provides a pilot eye movement tracking method based on multi-camera fusion, and the method includes: Step S10, deploying a plurality of cameras in a preset area inside the cockpit, mapping the image data collected by each camera into a unified three-dimensional space coordinate system to obtain multi-view images based on spatio-temporal alignment, and deploying an eye tracker for collecting eye movement data in the central area of the cockpit; Step S20, extracting the facial region image in the multi-view images, and dynamically cropping the region of interest (ROI) containing the eyes. The ROI adaptively adjusts the window position and size according to the head deflection angle, and calculates the three-dimensional head pose data of the pilot based on the feature points of the facial region image; Step S30, calculating the line-of-sight direction vector in parallel through a geometric path and a data path, fusing the calculation results of the two paths, and converting the line-of-sight direction vector to the world coordinate system by using the three-dimensional head pose data to compensate for the head movement error; Wherein, the geometric path includes generating a line-of-sight direction vector based on iris fitting and an eyeball model for the ROI; the data path is to extract the line-of-sight direction vector from the ROI through a deep learning model; Step S40, performing spatio-temporal alignment on the line-of-sight direction vector after error compensation and the eye movement data collected by the eye tracker, and dynamically allocating weights to obtain the final line-of-sight direction vector: When the line of sight is in the central area of the cockpit, the eye movement data collected by the eye tracker is preferentially used. When the line of sight exceeds the central area, it switches to the final line-of-sight direction vector.

[0026] For a clearer description of a pilot eye movement tracking method based on multi-camera fusion according to the present invention, the following combines Figure 1 Each step in the embodiment of the present invention is described in detail, including steps S10 - S40, and each step is described in detail as follows: Step S10, deploying a plurality of cameras in a preset area inside the cockpit, mapping the image data collected by each camera into a unified three-dimensional space coordinate system to obtain multi-view images based on spatio-temporal alignment, and deploying an eye tracker for collecting eye movement data in the central area of the cockpit; In this embodiment, in order to comprehensively capture the visual information of the pilot, the present invention deploys multiple groups of sensors inside the cockpit. First, a non-contact eye tracker is deployed below the sun visor or below the PFD / ND instrument for collecting the eye movement data directly in front of the pilot. This deployment method can utilize the characteristic that the existing eye tracker usually has high accuracy in the central view.

[0027] Secondly, four groups of cameras are respectively deployed in the key areas of the cockpit, including the overhead panel, the central instrument area, the central console, and the side windows. These cameras capture the facial and eye images of the pilot from different perspectives, aiming to cover a wider field of view, especially to capture lateral saccade behavior. The specific number and installation positions of the cameras can be adjusted according to different cockpit layouts and requirements, with the goal of maximizing the coverage of the pilot's visual range.

[0028] In this embodiment, the image data collected by each camera is mapped to a unified three-dimensional space coordinate system, and the method is as follows: Calibrate the parameters of each camera to determine the internal parameters and external parameters; Among them, the internal parameters include the focal length, the coordinates of the principal point, and the lens distortion parameters, and an internal parameter matrix of the camera is constructed; the external parameters include the rotation matrix and the translation vector of the camera in the world coordinate system; First, convert the world coordinates of any three-dimensional space point to the camera coordinate system through the rotation matrix and the translation vector, and then project the three-dimensional coordinates in this camera coordinate system onto the two-dimensional image plane according to the internal parameter matrix; Establish the conversion relationship between different camera coordinate systems. Through the rotation matrix and the translation vector from one camera to another, convert the three-dimensional point coordinates in one camera coordinate system to the three-dimensional coordinates in another camera coordinate system, and finally map the three-dimensional data collected by all cameras to the same global three-dimensional coordinate system to achieve cross-perspective data fusion.

[0029] Since multiple groups of cameras are distributed at different positions in the cockpit, the image data collected by them is in different local coordinate systems. To achieve effective fusion and analysis of multi-perspective image data, the present invention constructs a spatial coordinate system mapping network. The main function of this network is to map the image data from different cameras to a unified three-dimensional space coordinate system. This usually requires calibrating each camera to determine its internal parameters (such as focal length, principal point) and external parameters (such as position, attitude). The calibration process can use standard camera calibration methods, such as the method based on calibration objects such as checkerboards. Through the spatial coordinate system mapping network, precise alignment of multi-perspective images in time and space can be achieved, laying a foundation for subsequent feature extraction and line-of-sight calculation.

[0030] Camera projection model: Any three-dimensional space point The projection in the camera coordinate system can be expressed as: ; ; Among them, is the three-dimensional point coordinates in the camera coordinate system, 、 are the pixel coordinates after projection.

[0031] Among them, is the rotation matrix, is the translation vector, is the camera intrinsic matrix: ; Among them, are the focal lengths in the x and y directions (in pixel units), which are jointly determined by the physical focal length and the sensor pixel density, is the principal point coordinate (in pixel units), which is the center of the image and represents the intersection point of the optical axis and the image plane, s is the skew factor, which describes the non-perpendicularity of the x-axis and y-axis in the image coordinate system.

[0032] Multi-camera coordinate transformation: The relationship between different cameras can be established through a transformation matrix: ; Among them, and are the rotation matrix and translation vector from camera 1 to camera 2, is the three-dimensional point coordinate in the coordinate system of camera 1, is the three-dimensional point coordinate in the coordinate system of camera 2 after transformation.

[0033] Time synchronization model: For time synchronization, a linear interpolation method is adopted to align sensor data with different sampling rates: ; Among them, , , represents the data, is the interpolation weight factor, are two adjacent original data sampling time points, t is the target time point to be synchronized; Step S20, extract the facial region image in the multi-view image, and dynamically crop the region of interest containing the eyes. The region of interest adaptively adjusts the window position and size according to the head deflection angle, and calculates the three-dimensional head pose data of the pilot based on the feature points of the facial region image; In this embodiment, the method for extracting the facial region image in the multi-view image is: Adopt a hybrid deep network model to extract the facial region image in the multi-view image. The hybrid deep network model is jointly constructed based on the ResNet50 network model and the ViT network model.

[0034] In this embodiment, after obtaining the synchronized image data of multiple perspectives, the feature extraction layer first needs to accurately locate the pilot's facial area. The present invention uses a ResNet50+ViT hybrid model for facial localization, aiming to achieve a localization accuracy of ±3 pixels. ResNet50 is a classic deep convolutional neural network with strong feature extraction capabilities, capable of effectively capturing local texture and structural information in images. The Vision Transformer (ViT) model is based on the self-attention mechanism and is good at capturing global context information in images. Combining ResNet50 and the ViT model can make full use of the advantages of both, improving the robustness and accuracy of facial localization, and stably detecting the face even under complex conditions such as light changes and head pose changes.

[0035] Hybrid deep network: A hybrid network architecture composed of ResNet50 and ViT can be expressed as: ; ; ; Among them, is the region of interest, is feature, is the Transformer-enhanced feature, and finally outputs the facial bounding box and key point coordinates. is the pixel coordinates of the center point of the bounding box, is the width and height of the bounding box, is the set of pixel coordinates of the facial key points (such as eyes, nose, corners of the mouth, etc.), is the number of key points, and the specific value is defined by the task.

[0036] After facial localization, it is necessary to crop the region of interest (ROI) containing the eyes from the facial image. Since the pilot may deflect the head during operation, the traditional fixed ROI cropping method may not be able to effectively intercept the complete eye region. To solve this problem, the present invention designs an eye ROI dynamic cropping algorithm. This algorithm can dynamically adjust the size, position, and angle of the cropping frame according to the result of head pose estimation, so as to support effectively intercepting the eye region even when the head deflects ±45 degrees. Dynamic cropping can ensure that the subsequent eye feature extraction and gaze calculation model receive high-quality eye image data, improving the performance of the overall system.

[0037] In this embodiment, the method for dynamically cropping the region of interest containing the eyes is: Set the initial reference width and height of the region of interest (ROI) for the eye, which are pre-configured according to the pilot's eye anatomical characteristics and camera resolution; Expand or contract the width and height of the ROI proportionally according to the horizontal deflection angle and vertical pitch angle of the pilot's head; Calculate the eye center position based on the detected coordinates of the key points of the inner and outer corners of both eyes; Generate a rotation matrix according to the head deflection angle, and perform a rotation transformation on the original offset between the preset eye center and the ROI center to obtain the dynamic offset after head deflection; Superimpose the dynamic offset on the eye center coordinates to determine the final center position of the ROI window; Synchronously adjust the rotation direction of the ROI window according to the head deflection angle, so that the cropping frame is always aligned with the actual deflection direction of the eyeball, ensuring complete coverage of the eye area in extreme head postures.

[0038] Expand or contract the width and height of the ROI proportionally according to the horizontal deflection angle and vertical pitch angle of the pilot's head. Specifically: According to the horizontal deflection angle of the pilot's head, based on the reference width, expand or contract the width of the ROI proportionally. The expansion amplitude is proportional to the absolute value of the sine of the deflection angle, and the scaling factor controls the maximum adjustment amplitude; According to the vertical pitch angle of the head, based on the reference height, expand or contract the height of the ROI proportionally. The expansion amplitude is proportional to the absolute value of the sine of the pitch angle, and the scaling factor is independent of the horizontal direction.

[0039] Specifically, this embodiment is based on the key points of the inner and outer corners of the eyes and as well as the head pose angle , and dynamically adjusts the size and position of the ROI window: Width adjustment: ; Height adjustment: ; ; Among them, is the reference window size, are the horizontal and vertical scaling factors respectively, is the rotation matrix, is the initial offset, representing the preset offset of the ROI center relative to the key points of the inner and outer corners of the eyes, are the coordinates of the key points of the inner and outer corners of the eyes, usually the average of the left and right inner and outer corners or the coordinates of a single inner and outer corner, serving as the initial positioning reference for the ROI, is the horizontal position of the ROI center in the image coordinate system, is the vertical position of the ROI center in the image coordinate system.

[0040] In this embodiment, the three-dimensional head pose data of the pilot is calculated based on the feature points of the facial region image, and the method is as follows: Pre-define a three-dimensional facial model including key points at the corners of the eyes, the tip of the nose, and the corners of the mouth. Each key point in the model has known three-dimensional spatial coordinates; detect the two-dimensional facial key points corresponding to the three-dimensional model from the facial region image, and establish the matching relationship between the three-dimensional model points and the two-dimensional image points; Through the perspective projection geometric relationship, project the three-dimensional model points to the theoretical positions on the two-dimensional image plane, and calculate the position deviation between the theoretical positions and the actual detected points; Taking the rotation matrix and the translation vector as optimization variables, iteratively adjust their values to minimize the sum of the squared projection deviations of all matching key points, and solve the optimal head pose parameters; Extract the yaw angle, pitch angle, and roll angle of the pilot's head from the optimized rotation matrix as the three-dimensional head pose data of the pilot.

[0041] Specifically, in order to accurately compensate for the influence of head movement on gaze estimation, the present invention introduces a 3D head pose estimation module. The goal of this module is to estimate the three-dimensional pose of the pilot's head, including Yaw (yaw angle), Pitch (pitch angle), and Roll (roll angle), and requires the error in each direction to be less than 2 degrees. Head pose estimation can be achieved based on various methods, such as using the position information of facial key points (such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc.) in three-dimensional space, and combining techniques such as model fitting or deep learning for estimation. High-precision head pose estimation is the key to subsequent head movement compensation in the gaze calculation layer.

[0042] Using known 3D facial model points and their 2D image projections between the corresponding relationships, estimate the head pose through the PnP algorithm: ; where is the projection function, is the camera internal parameter, and are the rotation matrix and the translation vector representing the head pose, respectively.

[0043] Euler angle conversion: Extract the head pose represented by Euler angles from the rotation matrix : Yaw angle ; Pitch angle ; Roll angle ; Among them are the elements of the rotation matrix .

[0044] Step S30: Calculate the line-of-sight direction vector in parallel through the geometric path and the data path, fuse the calculation results of the two paths, and use the three-dimensional head pose data to convert the line-of-sight direction vector to the world coordinate system to compensate for the head movement error; Among them, the geometric path includes generating a line-of-sight direction vector based on iris fitting and an eyeball model for the region of interest; the data path is to extract the line-of-sight direction vector from the region of interest through a deep learning model; Among them, the line-of-sight calculation layer is responsible for calculating the pilot's line-of-sight direction according to the extracted eye features and head pose information. The present invention proposes a dual-path fusion network structure and introduces HFA-Net (Head-Fusion Attention Network) to further improve the accuracy and robustness of line-of-sight estimation. It includes a geometric path and a data path, aiming to comprehensively utilize the advantages of model-based methods and data-based methods.

[0045] Geometric path: The geometric path calculates the line-of-sight vector based on elliptical iris fitting. This method first requires accurate positioning of the pupil center, with a positioning error less than 0.5 mm. Then, by fitting the elliptical shape of the iris, the rotation angle of the eye can be estimated, and combined with known eyeball model parameters, the line-of-sight vector in three-dimensional space can be calculated. The geometric path is a direct, physical model-based line-of-sight estimation method, and its accuracy depends to a large extent on the accuracy of pupil center positioning.

[0046] Generate a line-of-sight direction vector based on iris fitting and an eyeball model. The method is as follows: Step S31: Locate the pupil center within the region of interest of the eye and fit the elliptical geometric features based on the iris edge; Step S32: Establish the three-dimensional spatial relationship between the pupil center and the eyeball center according to the predefined eyeball geometric model; Step S33: Calculate the normalized line-of-sight direction vector based on the three-dimensional spatial relationship.

[0047] Among them, to locate the pupil center within the region of interest of the eye, the method is as follows: In step S31, locate the pupil center coordinates within the region of interest of the eye through the gray centroid method or morphological operations, with a three-dimensional spatial positioning error of the pupil center less than 0.5 mm; The gray centroid method determines the pupil center by calculating the intensity-weighted average of the pixel coordinates within the pupil region, and the morphological operation extracts the geometric center of the pupil through binarization and connected component analysis; Locate the pupil center by gray centroid method or morphological operation : ; Among them, is the pupil area, is the pixel intensity, expressed as the brightness value of the image at the coordinate ( x, y ).

[0048] In step S31, based on the elliptical geometric features fitted from the iris edge, the method is as follows: Perform elliptical fitting on the iris edge, and use the random sample consensus algorithm or the Hough transform to screen the optimal elliptical parameters, including the elliptical center coordinates, the major axis length, the minor axis length, and the rotation angle; The elliptical parameters are used to characterize the deformation of the iris in the image plane, and combined with the camera internal parameters, the three-dimensional spatial orientation of the iris is inversely deduced; Elliptical iris fitting: Fit the iris ellipse by the Hough transform or the RANSAC method: ; Among them is the elliptical center, is the major and minor axis lengths, is the elliptical rotation angle.

[0049] In step S32, according to the predefined eyeball geometric model, establish the three-dimensional spatial relationship between the pupil center and the eyeball center, and the method is as follows: Based on the predefined single-sphere or double-sphere eyeball model, establish the three-dimensional spatial relative position relationship between the pupil center and the eyeball center; According to the displacement vector of the pupil center relative to the eyeball center in the eyeball model, calculate the normalized line-of-sight direction vector; Calculate the line-of-sight vector based on the single-sphere or double-sphere eyeball model : ; Among them is the position of the pupil center in 3D space, is the position of the eyeball center.

[0050] In step S33, calculate the normalized line-of-sight direction vector based on the three-dimensional spatial relationship.

[0051] Subtract the coordinates of the pupil center in three-dimensional space from the coordinates of the eyeball center to obtain the unnormalized line-of-sight vector; Normalize the vector and output the three-dimensional line-of-sight direction calculated by the geometric path.

[0052] Data path: The data path adopts a deep learning regression model. This model is pre-trained with a large number of eye image datasets to learn the mapping relationship between eye image features and three-dimensional line-of-sight directions. In practical applications, the data path directly uses the cropped eye ROI image as input and outputs the corresponding three-dimensional line-of-sight direction. The advantage of the data path is that it can learn complex patterns from a large amount of data and has a certain adaptability to individual differences and environmental changes.

[0053] Adopt a deep CNN structure, which can be expressed as: ; ; where is the eye ROI image, is the extracted feature vector, is the predicted line-of-sight vector, and are network parameters.

[0054] Loss function : Train the network through angular error loss: ; where is the true line-of-sight vector.

[0055] Use the three-dimensional head pose data to convert the line-of-sight direction vector to the world coordinate system to compensate for the head movement error. Specifically: To further improve the accuracy and robustness of line-of-sight estimation, the present invention adopts HFA-Net (Head-Fusion Attention Network). HFA-Net mainly realizes the following two functions. The first function is head pose compensation, and the second function is the attention mechanism to suppress light interference. Specifically: Head pose compensation: HFA-Net uses the 3D head pose information estimated in the feature extraction layer to compensate the line-of-sight vector calculated by the dual-path fusion network. Specifically, it can convert the line-of-sight vector from the camera coordinate system to the world coordinate system, thereby eliminating the line-of-sight estimation error caused by head movement.

[0056] Convert the relative line-of-sight vector to the world coordinate system through the head pose matrix: ; where is the line-of-sight vector relative to the head coordinate system, is the line-of-sight vector in the world coordinate system.

[0057] Attention mechanism to suppress light interference: HFA-Net introduces an attention mechanism to learn to assign different weights to different regions of the eye image. For regions vulnerable to light interference (e.g., specular reflection regions), lower weights are assigned; for regions containing valid eye information, higher weights are assigned. In this way, the interference of complex light environments on gaze estimation can be effectively suppressed, improving the robustness of the system.

[0058] Using channel attention mechanism to suppress light interference: ; ; where is the feature map, is the attention mask, is the element-wise multiplication.

[0059] Fusion strategy: Using learning-based weighted fusion: ; where and are the learned fusion weights, satisfying .

[0060] Attention mechanism to suppress light interference: Analyze the feature map of the region of interest through the attention mechanism to identify regions vulnerable to light interference and regions with valid eye information; Assign low weights to specular reflection regions and high weights to valid eye feature regions to generate a channel attention mask; Multiply the channel attention mask with the original feature map element-wise to suppress the feature response of the light interference region and retain valid eye features; Perform learning-based weighted fusion on the gaze vectors calculated by the geometric path and the gaze vectors calculated by the data path, dynamically adjusting the weight coefficients of the two; Constrain the sum of the weight coefficients to be 1. When light interference is significant, increase the weight of the data path to enhance the robustness of the model; when eye features are clear, increase the weight of the geometric path to utilize the accuracy of the physical model; Weighted sum the dual-path gaze vectors according to the dynamic weights and output the final gaze direction.

[0061] Step S40: Spatially and temporally align the error-compensated gaze direction vector with the eye movement data collected by the eye tracker, and perform dynamic weight assignment to obtain the final gaze direction vector: Preferentially use the eye movement data collected by the eye tracker when the gaze is in the central region of the cockpit, and switch to the final gaze direction vector when the gaze exceeds the central region.

[0062] This paper develops a multi-source data fusion algorithm for spatiotemporally aligning eye movement data collected by our system with data from traditional eye trackers. Because the two systems may have different sampling frequencies and coordinate systems, temporal synchronization and spatial coordinate conversion are necessary. Data fusion leverages the strengths of both systems. For example, while traditional eye trackers may offer higher accuracy within a central viewing angle, our system can provide a wider field of view.

[0063] Through the transformation matrix Convert the eye tracker data to a unified coordinate system: ; in, is the original data vector in the eye tracker coordinate system, including information such as eye position and sight direction. is the data vector after conversion to the world coordinate system, aligned with the global coordinate system, Represents the transformation matrix from the eye tracker coordinate system (tracker) to the world coordinate system (world).

[0064] Data synchronization is achieved through timestamp matching and interpolation: ; in, , represents the interpolation weight factor, are the adjacent sampling time points of the eye tracker data, For the time point The raw data vector of the eye tracker collected below, Interpolate is the interpolation function.

[0065] To more effectively utilize multi-source data, the present invention constructs a dynamic weighting model. This model dynamically adjusts the weighting of data from our system and traditional eye trackers based on the pilot's gaze. Specifically, when the pilot's gaze is in the center of the cockpit, the model prioritizes data from traditional eye trackers, as traditional eye trackers typically have higher accuracy in this area. When the gaze moves to the edge of the cockpit, the model switches to data from our system, as our system has more comprehensive coverage in this area. This dynamic weighting strategy achieves optimal eye tracking performance across the entire field of view.

[0066] Calculate weights based on head posture angle and gaze direction angle : ; in It is an adjustable parameter.

[0067] The final output is synthesized according to the dynamic weights as the final sight direction vector : ; When is close to 0, the eye tracker data is preferentially used; when is close to 1, the data of the present system is preferentially used, that is is the line-of-sight direction vector after error compensation.

[0068] Although the above embodiments describe the various steps in the above order, those skilled in the art can understand that in order to achieve the effects of this embodiment, different steps do not have to be executed in such an order, and they can be executed simultaneously (in parallel) or in a reversed order, and these simple changes are all within the protection scope of the present invention.

[0069] A pilot eye movement tracking system based on multi-camera fusion according to the second embodiment of the present invention is based on a pilot eye movement tracking method based on multi-camera fusion. The system includes: A multi-view image and eye movement data acquisition module configured to deploy multiple cameras in a preset area in the cockpit, map the image data collected by each camera into a unified three-dimensional space coordinate system to obtain a multi-view image based on spatio-temporal alignment, and deploy an eye tracker for collecting eye movement data in the central area of the cockpit; A region of interest and three-dimensional head pose data calculation module configured to extract the facial region image in the multi-view image, dynamically crop the region of interest including the eyes, the region of interest adaptively adjusts the window position and size according to the head deflection angle, and calculates the three-dimensional head pose data of the pilot based on the feature points of the facial region image; A line-of-sight direction vector calculation and error compensation module configured to calculate the line-of-sight direction vector in parallel through a geometric path and a data path, fuse the calculation results of the two paths, and use the three-dimensional head pose data to convert the line-of-sight direction vector to the world coordinate system to compensate for the head movement error; Wherein, the geometric path includes generating a line-of-sight direction vector based on iris fitting and an eyeball model for the region of interest; the data path is to extract the line-of-sight direction vector from the region of interest through a deep learning model; An allocation module configured to perform spatio-temporal alignment on the line-of-sight direction vector after error compensation and the eye movement data collected by the eye tracker, and dynamically allocate weights to obtain the final line-of-sight direction vector: When the line of sight is in the central area of the cockpit, the eye movement data collected by the eye tracker is preferentially used, and when the line of sight exceeds the central area, it is switched to the final line-of-sight direction vector. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and related descriptions of the above-described system can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0070] It should be noted that for the pilot eye movement tracking system based on multi-camera fusion provided in the above embodiments, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further split into multiple sub-modules to complete all or part of the functions described above. For the names of the modules and steps involved in the embodiments of the present invention, they are only used to distinguish each module or step, and are not regarded as improper limitations of the present invention.

[0071] An electronic device according to the third embodiment of the present invention includes: At least one processor; and A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned pilot eye movement tracking method based on multi-camera fusion.

[0072] A computer-readable storage medium according to the fourth embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned pilot eye movement tracking method based on multi-camera fusion.

[0073] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes and related descriptions of the above-mentioned storage devices and processing devices can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0074] Those skilled in the art should be able to realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in the form of electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0075] The terms "first", "second", etc. are used to distinguish similar objects and are not used to describe or indicate a particular order or sequence.

[0076] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to those processes, methods, articles, or apparatus / devices.

[0077] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims

1. A pilot eye movement tracking method based on multi-camera fusion, characterized in that, The method includes: Multiple cameras are deployed in a preset area within the cockpit. The image data collected by each camera is mapped into a unified three-dimensional spatial coordinate system to obtain multi-view images based on spatiotemporal alignment. An eye tracker is also deployed to collect eye movement data in the central area of the cockpit. Extracting facial region images from the multi-view images and dynamically cropping a region of interest including the eyes, adaptively adjusting a window position and size of the region of interest according to a head deflection angle, and calculating three-dimensional head posture data of the pilot based on feature points of the facial region images; The gaze direction vector is calculated in parallel through the geometric path and the data path, the dual-path calculation results are integrated, and the gaze direction vector is converted to the world coordinate system using the 3D head posture data to compensate for head motion errors. The geometric path includes generating a sight direction vector from the region of interest based on iris fitting and an eyeball model; the data path is extracting the sight direction vector from the region of interest through a deep learning model; The error-compensated gaze direction vector is spatially and temporally aligned with the eye movement data collected by the eye tracker, and the weights are dynamically assigned to obtain the final gaze direction vector: When the line of sight is in the center area of the cockpit, the eye movement data collected by the eye tracker is used first. When the line of sight exceeds the center area, it switches to the final line of sight direction vector.

2. The pilot eye tracking method based on multi-camera fusion according to claim 1, characterized in that: The image data collected by each camera is mapped to a unified three-dimensional space coordinate system as follows: Calibrate parameters of each camera to determine internal and external parameters; The internal parameters include focal length, principal point coordinates and lens distortion parameters, and the internal parameter matrix of the camera is constructed; the external parameters include the rotation matrix and translation vector of the camera in the world coordinate system; The world coordinates of any 3D space point are first converted to the camera coordinate system through the rotation matrix and translation vector, and then the 3D coordinates in the camera coordinate system are projected onto the 2D image plane according to the intrinsic parameter matrix; Establish a transformation relationship between different camera coordinate systems. Through the rotation matrix and translation vector from one camera to another, the three-dimensional point coordinates in one camera coordinate system are converted into the three-dimensional coordinates in the other camera coordinate system. Finally, the three-dimensional data collected by all cameras are mapped to the same global three-dimensional coordinate system to achieve cross-view data fusion.

3. The method for pilot eye movement tracking based on multi-camera fusion according to claim 1, wherein, Extracting facial region images from the multi-view images, the method is as follows: A hybrid deep network model is used to extract facial area images from the multi-view images, and the hybrid deep network model is jointly constructed based on a ResNet50 network model and a ViT network model.

4. The pilot eye tracking method based on multi-camera fusion according to claim 1, characterized in that: Dynamically crop the region of interest containing the eyes using the following method: Set the initial reference width and height of the eye's region of interest. The initial reference width and height are pre-configured based on the pilot's eye anatomy and camera resolution. The width and height of the region of interest are proportionally expanded or contracted according to the horizontal deflection angle and vertical pitch angle of the pilot's head; Calculate the eye center position based on the detected eye corner key point coordinates; Generate a rotation matrix based on the head deflection angle, and perform a rotation transformation on the original offset between the preset eye center and the center of the region of interest to obtain the dynamic offset after head deflection; Superimpose the dynamic offset on the eye center coordinates to determine the final center position of the region of interest window; Synchronously adjust the rotation direction of the region of interest window according to the head deflection angle, so that the cropping frame is always aligned with the actual deflection direction of the eyeball, ensuring that the eye area is completely covered in extreme head postures.

5. A pilot eye movement tracking method based on multi-camera fusion according to claim 4, characterized in that, Expand or contract the width and height of the region of interest proportionally according to the horizontal deflection angle and vertical pitch angle of the pilot's head. Specifically: According to the horizontal deflection angle of the pilot's head, based on the reference width, expand or contract the width of the region of interest proportionally. The expansion amplitude is proportional to the absolute value of the sine of the deflection angle, and the scaling factor controls the maximum adjustment amplitude; According to the vertical pitch angle of the head, based on the reference height, expand or contract the height of the region of interest proportionally. The expansion amplitude is proportional to the absolute value of the sine of the pitch angle, and the scaling factor is independent of the horizontal direction.

6. A method for pilot eye movement tracking based on multi-camera fusion according to claim 1, characterized in that, Calculate the three-dimensional head pose data of the pilot based on the feature points of the facial area image. The method is as follows: Pre-define a three-dimensional facial model that includes key points such as the outer corners of the eyes, the tip of the nose, and the corners of the mouth. Each key point in the three-dimensional facial model has known three-dimensional spatial coordinates; detect the two-dimensional facial key points corresponding to the three-dimensional model from the facial area image, and establish the matching relationship between the three-dimensional model points and the two-dimensional image points; Through the perspective projection geometric relationship, project the three-dimensional model points to the theoretical position on the two-dimensional image plane, and calculate the position deviation between the theoretical position and the actual detected points; Taking the rotation matrix and translation vector as optimization variables, iteratively adjust their values to minimize the sum of the squared projection deviations of all matching key points, and solve the optimal head pose parameters; Extract the yaw angle, pitch angle, and roll angle of the pilot's head from the optimized rotation matrix as the three-dimensional head pose data of the pilot.

7. A pilot eye movement tracking method based on multi-camera fusion according to claim 1, characterized in that Generate a line-of-sight direction vector based on iris fitting and an eyeball model. The method is as follows: Locate the pupil center within the region of interest of the eye, and fit the elliptical geometric features based on the iris edge; According to the pre-defined eyeball geometric model, establish the three-dimensional spatial relationship between the pupil center and the eyeball center; Calculate the normalized line-of-sight direction vector based on the three-dimensional spatial relationship.

8. A pilot eye movement tracking system based on multi-camera fusion, based on the method for pilot eye movement tracking based on multi-camera fusion according to any one of claims 1-7, characterized in that, The system includes: A multi-view image and eye movement data acquisition module, which is configured to deploy multiple cameras in a preset area in the cockpit, map the image data collected by each camera to a unified three-dimensional space coordinate system to obtain multi-view images based on spatio-temporal alignment, and deploy an eye tracker for eye movement data acquisition in the central area of the cockpit; A region of interest and three-dimensional head pose data calculation module, which is configured to extract the facial area image from the multi-view image, dynamically crop the region of interest containing the eyes, the region of interest adaptively adjusts the window position and size according to the head deflection angle, and calculates the three-dimensional head pose data of the pilot based on the feature points of the facial area image; A line-of-sight direction vector calculation and error compensation module, configured to calculate the line-of-sight direction vector in parallel through a geometric path and a data path, fuse the calculation results of the two paths, and use the three-dimensional head pose data to convert the line-of-sight direction vector to the world coordinate system to compensate for the head movement error; Among them, the geometric path includes generating a line-of-sight direction vector based on iris fitting and an eyeball model for the region of interest; the data path is to extract the line-of-sight direction vector from the region of interest through a deep learning model; An allocation module, configured to perform spatio-temporal alignment on the line-of-sight direction vector after error compensation and the eye movement data collected by the eye tracker, and dynamically allocate weights to obtain the final line-of-sight direction vector: When the line of sight is in the central area of the cockpit, the eye movement data collected by the eye tracker is preferentially used, and when the line of sight exceeds the central area, it switches to the final line-of-sight direction vector.

9. An electronic device, characterized in that, Comprising: At least one processor; And A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement a pilot eye movement tracking method based on multi-camera fusion according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement a pilot eye movement tracking method based on multi-camera fusion according to any one of claims 1-7.

Images