A pilot gaze object recognition method and system based on eye movement data

Abstract

This invention relates to the field of eye-tracking data processing technology, and provides a method and system for pilot gaze object recognition based on eye-tracking data. The method includes the following steps: sampling video data into images, constructing a training dataset, training a YOLOv8n neural network to obtain a lightweight YOLO target detection model; performing frame-by-frame recognition of the video data to be identified based on the YOLO lightweight target detection model; checking and correcting the preliminary recognition results according to the relative positional relationships between gaze objects; detecting the pointer positions of instrument-type gaze objects and reconstructing the gaze area of ​​such gaze objects based on the pointer positions; and determining the gaze object at each moment based on the gaze point coordinates in the tabular data and the reconstructed gaze area. This invention can accurately identify the pilot's gaze object at each moment during flight training based on tabular data and video data recorded by an eye tracker.

Description

Technical Field

[0001] This invention belongs to the field of eye-tracking data processing technology, and in particular relates to a method and system for pilot gaze object recognition based on eye-tracking data. Background Technology

[0002] Effective attention allocation is crucial for improving flight training quality and ensuring flight safety. Traditional flight instruction primarily relies on instructor demonstrations and explanations, followed by review and debriefing through questioning, observation, and subjective judgment—a method with significant limitations. The rapid development of eye-tracking technology offers a new approach for the quantitative analysis of attention allocation.

[0003] However, existing technologies are typically designed for general scenarios and lack specific methods for gaze object recognition in flight training. Direct application of these methods in flight training results in low accuracy in gaze object recognition. During flight training, significant differences in lighting conditions inside and outside the cockpit lead to overexposure / underexposure in the image acquisition area, making accurate image content identification difficult. Simultaneously, pilots tend to focus more on the pointer scales at the edges of instruments than on the instrument centers. General AOI segmentation methods, which employ region coverage strategies, result in misclassification of gaze points and incorrect gaze object identification. Furthermore, pilots' gaze objects are often irregularly shaped, making traditional circular or rectangular AOI segmentation methods ineffective in accurately depicting the distribution of pilot attention. These issues mean that while eye-tracking data possesses advantages in high sampling rates and resolution, it is difficult to transform it into effective information for flight training optimization. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for pilot gaze object recognition based on eye movement data, in order to solve the above-mentioned technical problems.

[0005] This invention is implemented as follows: a method for pilot gaze object recognition based on eye-tracking data, comprising the following steps:

[0006] Eye movement data of pilots during flight training is collected; the eye movement data includes tabular data recorded at sampling frequency and synchronously collected video data;

[0007] The video data is sampled into images, a training dataset is constructed, and the YOLOV8n neural network is trained to obtain the YOLO lightweight object detection model.

[0008] Based on the YOLO lightweight object detection model, the video data to be identified is identified frame by frame to obtain the preliminary identification results of the gaze object in each frame image;

[0009] Based on the relative positional relationship between the objects being gazed at, the preliminary recognition results are checked and corrected to obtain the corrected object gaze recognition results.

[0010] Based on the corrected gaze object recognition result, the pointer position of the instrument-type gaze object is detected, and the gaze region of the gaze object is reconstructed according to the pointer position to obtain the reconstructed gaze region;

[0011] Based on the gaze point coordinates and the reconstructed gaze region in the table data, the gaze object at each moment is determined, and the gaze object recognition result is obtained.

[0012] Furthermore, the preliminary identification results include the type of the object being gazed upon, the coordinates of its center point, and its geometric dimensions; the objects being gazed upon include instrument-type objects; the instrument-type objects being gazed upon include attitude tables, heading indicators, and altimeters.

[0013] Furthermore, the step of checking and correcting the preliminary recognition result based on the relative positional relationship between the gaze objects to obtain the corrected gaze object recognition result specifically includes:

[0014] Using the attitude table and heading indicator table as reference objects, calculate the distance, angle and lateral consistency between the object to be inspected and the two reference objects. If all three meet the preset threshold conditions, the object to be inspected is retained; otherwise, it is discarded.

[0015] Based on the center point coordinates of the attitude table and heading indicator table, and combined with the preset statistical parameters of the positional relationship between gaze objects, the unidentified gaze objects are estimated. If the estimated gaze object is located within the image boundary, it is added to the preliminary identification result to obtain the corrected gaze object identification result.

[0016] Furthermore, using the attitude table and heading indicator table as reference objects, the distance, angle, and lateral consistency between the object to be inspected and the two reference objects are calculated. If all three meet the preset threshold conditions, the object to be inspected is retained; otherwise, it is discarded. This step specifically includes:

[0017] Define the average distance from the gaze object C to the pose table as: The average distance from the object being observed, C, to the heading indicator is The average angle between the object being observed (C) and the heading indicator at the center point of the attitude indicator is: The center point of the object to be examined The coordinates are ;

[0018] First, determine whether the distance between the object to be inspected and points A and B meets the requirements:

[0019] Calculate the center point of the object to be examined The actual distances to the center point A of the attitude indicator and the center point B of the heading indicator, respectively. and :

[0020] ;

[0021] in, Here are the coordinates of the center point A in the attitude table; The coordinates of the center point B of the heading indicator;

[0022] Calculate the distance error separately and :

[0023] ;

[0024] like and , If the distance is within the preset threshold, the distance is determined to meet the requirements; otherwise, the distance is determined to not meet the requirements, and the object to be inspected is removed.

[0025] Secondly, determine whether the angle meets the requirements:

[0026] Calculate the center point of the object to be examined The actual angle formed by the heading indicator and the attitude indicator at the center point of the attitude indicator. Size;

[0027] Calculate vectors and :

[0028] ;

[0029] ;

[0030] Calculate vectors and dot product and modulus:

[0031] ;

[0032] ;

[0033] ;

[0034] Calculate vectors and The included angle is... :

[0035] ;

[0036] Average angle error :

[0037] ;

[0038] like , If the angle threshold is set, the angle is determined to meet the requirements; otherwise, the angle is determined to not meet the requirements, and the object to be inspected is removed.

[0039] Then, determine whether the lateral consistency meets the requirements:

[0040] Calculate vectors and cross product :

[0041] ;

[0042] If the object of focus C should be to the left of the line AB connecting the center point A of the attitude indicator and the center point B of the heading indicator, then when If the lateral consistency is within the acceptable range, it is determined that the lateral consistency meets the requirements; otherwise, it is determined that the lateral consistency does not meet the requirements.

[0043] If the object of focus C should be to the right of the line AB connecting the center point A of the attitude indicator and the center point B of the heading indicator, then when If the lateral consistency is within the acceptable range, it is determined that the lateral consistency meets the requirements; otherwise, it is determined that the lateral consistency does not meet the requirements.

[0044] Furthermore, based on the center point coordinates of the attitude table and heading indicator table, and combined with preset statistical parameters of the positional relationship between gaze objects, the steps for estimating unidentified gaze objects, and if the estimated gaze object is located within the image boundary, then adding it to the preliminary identification result to obtain the corrected gaze object identification result, specifically include:

[0045] Using the attitude indicator and heading indicator as reference objects, calculate the distance between the center point A of the attitude indicator and the center point B of the heading indicator. :

[0046] ;

[0047] Calculate the heading angles at attitude table center point A and heading table center point B using the arctangent function. :

[0048] ;

[0049] Determine the orientation angles of the center point A of the posture table and the gaze object C based on the lateral orientation of the gaze object. If the object being gazed upon, C, is to the left of AB, then:

[0050] ;

[0051] Conversely:

[0052] ;

[0053] Use distance and direction angle Estimate the initial coordinates of the object being gazed upon, C:

[0054] ;

[0055] Obtain the initial estimated point ;

[0056] Use distance For the initial estimated point The coordinates are verified and adjusted, and the initial estimated points are calculated. Actual distance to the center point B of the heading indicator :

[0057] ;

[0058] like , Accepted at a preset threshold As an estimate, the estimated coordinates of the object being gazed at are obtained; otherwise, the adjustment step is initiated. The adjustment uses a ray-based optimization method to adjust the position of point C of the object being gazed at on the initial ray starting from the center point A of the attitude table, thereby obtaining the estimated coordinates of the object being gazed at.

[0059] If the estimated coordinates of the gaze object are within the image boundary, the gaze object is added to the preliminary recognition result to obtain the corrected gaze object recognition result; otherwise, it is not added.

[0060] Furthermore, the adjustment steps specifically include:

[0061] Define the parametric equation of the ray, where the ray originates from point A and has a direction angle of θ. The distance parameter is t:

[0062] ;

[0063] Define error function :

[0064] ;

[0065] in, , and These are the weighting coefficients. ;

[0066] exist Within the range, search for the optimal distance parameter that minimizes the error. ,use The estimated coordinates of the object being gazed upon are calculated:

[0067] .

[0068] Furthermore, the method for detecting the pointer position of the gaze object in the instrument category is the Hough transform method.

[0069] Furthermore, the method for reconstructing the gaze region of this type of gaze object based on the pointer position is as follows:

[0070] For a gaze object with a pointer, draw a circle with the pointer's length as the radius and one-third of the distance from the pointer's endpoint as the center. This circle is the reconstructed gaze region.

[0071] For a gaze object without a pointer, the reconstructed gaze area is a circle with its center point as the center and the larger of the length and width of the recognition box as the radius.

[0072] Furthermore, the method for determining the object of gaze at each moment is as follows:

[0073] Iterate through all objects being watched and determine whether each one is the current object being watched.

[0074] For a circular gaze object, calculate the distance from the gaze point to the center of the circle and compare it with the radius of the reconstructed gaze region; if the distance from the gaze point to the center of the circle is less than the radius of the reconstructed gaze region, then it is determined to be the current gaze object.

[0075] For non-circular gaze objects, the cross product of the gaze point and the lower edge line of the reconstructed gaze region is used to determine whether the gaze point is located within the reconstructed gaze region, and the determination is made in conjunction with the left and right boundary conditions.

[0076] Another objective of this invention is to provide a pilot gaze object recognition system based on eye-tracking data, for implementing the aforementioned pilot gaze object recognition method based on eye-tracking data, specifically including:

[0077] An eye-tracking data acquisition module is used to collect eye-tracking data of pilots during flight training; the eye-tracking data includes tabular data recorded according to the sampling frequency and synchronously acquired video data;

[0078] The model training module is used to sample the video data into images, construct a training dataset, train the YOLOV8n neural network, and obtain the YOLO lightweight object detection model.

[0079] The preliminary object identification module is used to identify the video data to be identified frame by frame based on the YOLO lightweight object detection model, and obtain the preliminary identification result of the object being identified in each frame image;

[0080] The recognition result checking and correction module is used to check and correct the preliminary recognition result based on the relative positional relationship between the gaze objects, so as to obtain the corrected gaze object recognition result;

[0081] The gaze region reconstruction module is used to detect the pointer position of an instrument-type gaze object based on the corrected gaze object recognition result, and reconstruct the gaze region of the gaze object according to the pointer position to obtain the reconstructed gaze region;

[0082] The gaze object determination module is used to determine the gaze object at each moment based on the gaze point coordinates in the table data and the reconstructed gaze area, and obtain the gaze object recognition result.

[0083] The pilot gaze object recognition method based on eye-tracking data provided by this invention can accurately identify the gaze object of a pilot at every moment during flight training based on tabular and video data recorded by an eye tracker. Specifically, this invention checks and corrects the initial recognition results based on prior information that the relative positions of gaze objects are fixed during flight training, effectively solving the problem of missed and false detections caused by differences in lighting inside and outside the cockpit. Subsequently, for instrument-type gaze objects, the gaze area is reconstructed through pointer detection, overcoming the problem of misclassification of gaze point assignment caused by traditional fixed area division. In addition, this invention matches the reconstructed gaze area with the coordinates of the gaze point, realizing accurate identification of gaze objects at each moment in complex flight training environments. This effectively transforms high-sampling-rate eye-tracking data into structured information that can be used for quantitative analysis of pilot attention allocation, significantly improving the accuracy and robustness of gaze object recognition. Attached Figure Description

[0084] Figure 1 This is a flowchart illustrating the pilot gaze object recognition method based on eye-tracking data provided in an embodiment of the present invention.

[0085] Figure 2 This is a diagram illustrating the relative positions of the objects being gazed at.

[0086] Figure 3 A flowchart illustrating the process of reconstructing the gaze region.

[0087] Figure 4 A schematic diagram for determining the vertical relationship of the fixation point. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0089] like Figure 1As shown, in one embodiment of the present invention, a method for pilot gaze object recognition based on eye-tracking data is provided, comprising the following steps:

[0090] S1. Collect eye movement data of pilots during flight training; eye movement data includes tabular data recorded according to sampling frequency and synchronously collected video data;

[0091] In practical applications, eye trackers are used to collect eye movement data from pilots during flight training, such as commercially available glasses-type eye trackers. The eye movement data is divided into two parts: tabular data in CSV format and video data in MP4 format. The tabular data records eye movement information such as the fixation position (fixation point coordinates), pupil size, and blink frequency of the left and right eyes at each sampling frequency, while the video data records the scene directly in front of the pilot.

[0092] S2. Sample the video data into images, construct a training dataset, train the YOLOV8n neural network, and obtain the YOLO lightweight object detection model;

[0093] Specifically, the collected video data is labeled, sampled into images, and labeled with instruments such as the attitude indicator, heading indicator, and altimeter inside the cabin, as well as the external environment, to obtain a training dataset. For the object detection model, the lightest YOLOv8n network structure in YOLOv8 is selected, and the YOLOv8n neural network is trained until convergence.

[0094] S3. Based on the YOLO lightweight target detection model, perform frame-by-frame recognition on the video data to be identified to obtain preliminary recognition results of the gaze object in each frame image;

[0095] Specifically, the YOLO lightweight target detection model is used for preliminary identification: video data collected during pilot training by eye trackers is extracted frame by frame into images, and a trained YOLOV8n neural network is used to identify the specific types of objects in the images, as well as the object's center point coordinates, width, length, and other geometric dimensions. Among them, gaze objects include instrument gaze objects and other non-instrument gaze objects; instrument gaze objects include attitude indicators, heading indicators, altimeters, etc.

[0096] S4. Based on the relative positional relationship between the objects being gazed upon, the preliminary recognition result is checked and corrected to obtain the corrected object gaze recognition result.

[0097] S5. Based on the corrected gaze object recognition result, detect the pointer position of the instrument-type gaze object, and reconstruct the gaze region of the gaze object according to the pointer position to obtain the reconstructed gaze region.

[0098] S6. Based on the gaze point coordinates and the reconstructed gaze area in the table data, determine the gaze object at each moment and obtain the gaze object recognition result.

[0099] In a preferred embodiment of the present invention, step S4 specifically includes:

[0100] S41. Using the attitude table and heading indicator table as reference objects, calculate the distance, angle and lateral consistency between the object to be checked and the two reference objects. If all three meet the preset threshold conditions, the object to be checked is retained; otherwise, it is discarded.

[0101] S42. Based on the center point coordinates of the attitude table and heading indicator table, and combined with the preset statistical parameters of the positional relationship between gaze objects, estimate the unidentified gaze objects. If the estimated gaze objects are located within the image boundary, add them to the preliminary recognition results to obtain the corrected gaze object recognition results.

[0102] Because the trainee's body remains relatively stationary during training, the observed positional relationships of the gaze objects are also relatively fixed. Although the trainee's head may sway left and right and move forward and backward during training, the distances and angles between the various instruments in the images captured by the eye tracker are relatively fixed. According to statistical results, these values ​​fluctuate around a standard value. Therefore, the positional relationships of the gaze objects can be used to check whether the identified gaze objects are correct and whether there are any missed gaze objects. In practical applications, aircraft instruments generally follow a T-shaped layout; the attitude indicator, located in the center of the cockpit, and the heading indicator below it, have the most obvious image features and the highest recognition accuracy. Moreover, they are always within the pilot's line of sight when observing the instruments. Therefore, this embodiment of the invention selects the attitude indicator and the heading indicator as the benchmark objects to check the recognition results of other gaze objects, eliminate gaze objects incorrectly identified by the YOLO lightweight target detection model, and supplement missed gaze objects.

[0103] like Figure 2 As shown, point A represents the coordinates of the identified attitude indicator center, point B represents the coordinates of the identified heading indicator center, and point C represents the preliminary identification result of the gaze object to be checked; step S41 specifically includes:

[0104] Based on statistical calculations of the labeled data, the average distance from the gaze object C to the pose table is defined as follows: The average distance from the object being observed, C, to the heading indicator is The average angle between the object being observed (C) and the heading indicator at the center point of the attitude indicator is: (Converted to radians), center point of the object to be viewed to be checked. The coordinates are ;

[0105] First, determine whether the distance between the object to be inspected and points A and B meets the requirements:

[0106] Calculate the center point of the object to be examined The actual distances to the center point A of the attitude indicator and the center point B of the heading indicator, respectively. and :

[0107] ;

[0108] in, Here are the coordinates of the center point A in the attitude table; The coordinates of the center point B of the heading indicator;

[0109] Calculate the distance error separately and :

[0110] ;

[0111] like and , If the distance threshold is set to a preset value (e.g., 0.2), the distance is deemed to meet the requirements; otherwise, the distance is deemed not to meet the requirements, and the object to be inspected is removed.

[0112] Secondly, determine whether the angle meets the requirements:

[0113] Calculate the center point of the object to be examined The actual angle formed by the heading indicator and the attitude indicator at the center point of the attitude indicator. ( The size of )

[0114] Calculate vectors and :

[0115] ;

[0116] ;

[0117] Calculate vectors and dot product and modulus:

[0118] ;

[0119] ;

[0120] ;

[0121] Calculate vectors and The included angle is... :

[0122] ;

[0123] Average angle error :

[0124] ;

[0125] like , If the angle threshold is set to a preset value (e.g., 0.1), the angle is deemed to meet the requirements; otherwise, the angle is deemed not to meet the requirements, and the object to be inspected is removed.

[0126] Next, it is determined whether the lateral consistency meets the requirements. Some instruments are located in the left area of ​​the attitude chart, while others are located in the right area. If the initially identified gaze target is not on a specific side of AB, it can be determined that the identification is incorrect. This can be determined through calculation. and The cross product can be used to directly determine the outcome.

[0127] Specifically, calculate vectors and cross product :

[0128] ;

[0129] If the object of focus C should be to the left of the line AB connecting the center point A of the attitude indicator and the center point B of the heading indicator, then when If the lateral consistency is within the acceptable range, it is determined that the lateral consistency meets the requirements; otherwise, it is determined that the lateral consistency does not meet the requirements.

[0130] If the object of focus C should be to the right of the line AB connecting the center point A of the attitude indicator and the center point B of the heading indicator, then when If the lateral consistency is within the acceptable range, it is determined that the lateral consistency meets the requirements; otherwise, it is determined that the lateral consistency does not meet the requirements.

[0131] In summary, if the initially identified object of gaze meets the requirements in terms of distance, angle, and lateral consistency, then the initially identified object of gaze can be considered the correct target.

[0132] In a preferred embodiment of the present invention, due to the significant difference in lighting inside and outside the cockpit, the image acquisition area is overexposed / underexposed, making it difficult to accurately identify the image content, especially the instrument recognition accuracy at the cockpit edge is low. Furthermore, because some preliminary identification results were incorrect and removed during the preceding inspection process, the list of initially identified objects of gaze may be incomplete, requiring correction of the preliminary identification results. In this embodiment of the present invention, the basic idea for correcting the preliminary identification results of objects of gaze is to estimate the position and size of objects of gaze not appearing in the preliminary identification result list based on statistical analysis of the positional relationships between a large amount of labeled data instruments. If the estimated object of gaze is within the image boundary range, it is added to the preliminary identification result of objects of gaze; otherwise, it is not included. Specifically, step S42 includes:

[0133] Still selecting the attitude table and heading indicator as the reference objects, the coordinates of the initially identified center point of the attitude table are calculated as A. The coordinates of the center point of the heading indicator are B. Let C be the coordinates of the center point of the object to be gazed upon. The average distance between object C and the center point of the attitude table is... The average distance from the center point of the heading indicator is The average angle formed by the lines connecting the center points of the attitude table is (Radian value); Knowing in advance the lateral orientation of the object being observed along the line connecting the centers of the attitude and heading indicators, estimate the coordinates of point C based on the above conditions. The main process of estimation is to first, based on... and Calculate the coordinates of point C, then calculate the distance to point B, and... Compare the coordinates and correct the coordinates of point C based on the magnitude of the error.

[0134] Specifically, first calculate the distance between the center point A of the attitude indicator and the center point B of the heading indicator. :

[0135] ;

[0136] Calculate the heading angles at attitude table center point A and heading table center point B using the arctangent function. :

[0137] ;

[0138] in, The range is , represents the angle from the positive x-axis to vector AB;

[0139] Next, the orientation angles of the attitude table center point A and the gaze object C are determined based on the lateral orientation of the gaze object. If the object being gazed upon, C, is to the left of AB (counterclockwise), then:

[0140] ;

[0141] Conversely:

[0142] ;

[0143] To avoid If the value is outside the range, it needs to be normalized (modulo). );

[0144] Use distance and direction angle Estimate the initial coordinates of the object being gazed upon, C:

[0145] ;

[0146] Obtain the initial estimated point ;

[0147] Then, use distance For the initial estimated point The coordinates are verified and adjusted, and the initial estimated points are calculated. Actual distance to the center point B of the heading indicator :

[0148] ;

[0149] like , Accepted at a preset threshold As an estimate, the estimated coordinates of the object being gazed at are obtained; otherwise, the adjustment step is initiated. The adjustment uses a ray-based optimization method to adjust the position of point C of the object being gazed at on the initial ray starting from the center point A of the attitude table, thereby obtaining the estimated coordinates of the object being gazed at.

[0150] The adjustment steps specifically include:

[0151] Define the parametric equation of the ray, where the ray originates from point A and has a direction angle of θ. The distance parameter is t:

[0152] ;

[0153] Define error function :

[0154] ;

[0155] in, , and These are the weighting coefficients. ;

[0156] exist Within the range, search for the optimal distance parameter that minimizes the error. ,use The estimated coordinates of the object being gazed upon are calculated:

[0157] .

[0158] Finally, the following judgment is made: if the estimated coordinates of the gaze object are within the image boundary range, then the gaze object is added to the preliminary recognition result to obtain the corrected gaze object recognition result; otherwise, it is not added.

[0159] like Figure 3 As shown, in a preferred embodiment of the present invention, when a pilot observes the instruments, they focus on the values ​​displayed on the instruments. Therefore, their gaze rests near the pointer's position, rather than at the center of the instrument. This makes it easier to identify the pointer positions of each instrument and more conducive to assessing the trainee's attention allocation. Specifically, in this embodiment of the invention, the Hough transform method is used to detect the pointer positions of instrument-type gaze objects. Based on gaze object recognition, the Hough transform is used to detect instrument pointers, which requires the following steps:

[0160] 1. Image preprocessing: Convert the image to grayscale to reduce computational complexity, use Gaussian blur for noise reduction, and use the Canny operator for edge detection;

[0161] 2. Determine the circular area where the instrument is located, and determine the radius of the circle based on the center point coordinates and length and width values ​​(w, h) of the gaze object in the corrected gaze object recognition results;

[0162] 3. Cropping the circular region yields the ROI, and binarization and morphological operations enhance pointer features;

[0163] 4. Use the Hough transform to detect straight lines;

[0164] 5. Based on conditions such as length and whether it passes through the center of the circle, select the most likely straight line;

[0165] 6. Based on the starting point of the line, determine the area where the pointer is located.

[0166] Furthermore, when pilots observe instruments, they focus on the numerical values ​​displayed, thus their gaze lingers near the pointer's position. The gaze area can be reconstructed based on the pointer's location. Specifically, the method for reconstructing the gaze area for this type of object based on the pointer's position is as follows:

[0167] For a gaze object with a pointer, draw a circle with the pointer's length as the radius and one-third of the distance from the pointer's endpoint as the center. This circle is the reconstructed gaze region.

[0168] For gaze objects without pointers, such as those in the pose table, the above operation is not performed. Instead, a circle is drawn with its center point as the center and the larger of the length and width of the recognition box as the radius. This circle is the reconstructed gaze area.

[0169] In a preferred embodiment of the present invention, the table data in the eye tracker records the coordinates of the left and right eye fixations. The average of the two coordinates is taken as the coordinates of the fixation point at the current moment. Based on the reconstructed fixation area and the fixation point coordinates, the current object of fixation and the distance between the fixation coordinates and the center point of the object of fixation can be determined. Specifically, the method for determining the object of fixation at each moment includes the following steps:

[0170] Iterate through all objects being watched and determine whether each one is the current object being watched.

[0171] For circular objects to be gazed upon (such as cockpit instruments), calculate the gaze point. To the center The distance from the gaze point to the center of the circle is compared with the radius r of the reconstructed gaze region; if the distance from the gaze point to the center of the circle is less than the radius r of the reconstructed gaze region, then it is determined to be the current gaze object; specifically, if:

[0172] ;

[0173] This indicates that the object being watched is the current target; if:

[0174] ;

[0175] This indicates that the object being gazed at is not the current gaze target.

[0176] Furthermore, for rectangular, trapezoidal, and triangular regions outside the cabin, theoretically, the number of intersections between any ray originating from the gaze coordinate point and the region can be used to determine whether it is within the region, but this is computationally complex. In actual calculations, the upper edge of the gaze region outside the cabin usually reaches the top of the image, so only the relationship between the gaze point coordinates and the lower edge of the gaze region needs to be considered; that is, for non-circular gaze objects (such as regions outside the cabin), the cross product of the gaze point and the lower edge line of the reconstructed gaze region can be used to determine whether the gaze point is located within the reconstructed gaze region, and this can be combined with the left and right boundary conditions for determination.

[0177] Specifically, such as Figure 4 As shown, if the lower edge of the gaze region is represented by two vertices F and G, where F is the left point and G is the right point, and the gaze point is represented by H; vectors can be used to... and The cross product is used to calculate the position of the gaze point H relative to the line FG. The cross product is calculated as follows:

[0178] ;

[0179] like If the fixation point H is above line FG, then it indicates that the fixation point H is above line FG; otherwise, it indicates that the fixation point H is below line FG. If the fixation point H is above line FG, it is necessary to determine again whether the fixation point H is within the left and right boundary range, i.e., whether it satisfies the following conditions:

[0180] ;

[0181] If the conditions are met, the current external area can be determined as the target to be observed.

[0182] Using the above method, the pilot's gaze at each moment can be accurately obtained, which can be used to further analyze the pilot's visual attention allocation.

[0183] In another embodiment of the present invention, a pilot gaze object recognition system based on eye-tracking data is also provided to implement the above-described pilot gaze object recognition method based on eye-tracking data, specifically including:

[0184] An eye-tracking data acquisition module is used to collect eye-tracking data of pilots during flight training; the eye-tracking data includes tabular data recorded according to the sampling frequency and synchronously acquired video data;

[0185] The model training module is used to sample the video data into images, construct a training dataset, train the YOLOV8n neural network, and obtain the YOLO lightweight object detection model.

[0186] The preliminary object identification module is used to identify the video data to be identified frame by frame based on the YOLO lightweight object detection model, and obtain the preliminary identification result of the object being identified in each frame image;

[0187] The recognition result checking and correction module is used to check and correct the preliminary recognition result based on the relative positional relationship between the gaze objects, so as to obtain the corrected gaze object recognition result;

[0188] The gaze region reconstruction module is used to detect the pointer position of an instrument-type gaze object based on the corrected gaze object recognition result, and reconstruct the gaze region of the gaze object according to the pointer position to obtain the reconstructed gaze region;

[0189] The gaze object determination module is used to determine the gaze object at each moment based on the gaze point coordinates in the table data and the reconstructed gaze area, and obtain the gaze object recognition result.

[0190] It should be noted that each of the above modules can be implemented as a computer program, which can run on a computer device. The computer device's memory can store the computer program that makes up each module, enabling the processor to execute each step of the above method.

[0191] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0192] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0193] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method for pilot gaze object recognition based on eye-tracking data, characterized in that, Includes the following steps: Eye movement data of pilots during flight training is collected; the eye movement data includes tabular data recorded at sampling frequency and synchronously collected video data; The video data is sampled into images, a training dataset is constructed, and the YOLOV8n neural network is trained to obtain the YOLO lightweight object detection model. Based on the YOLO lightweight object detection model, the video data to be identified is identified frame by frame to obtain the preliminary identification results of the gaze object in each frame image; Based on the relative positional relationship between the objects being gazed at, the preliminary recognition results are checked and corrected to obtain the corrected object gaze recognition results. Based on the corrected gaze object recognition result, the pointer position of the instrument-type gaze object is detected, and the gaze region of the gaze object is reconstructed according to the pointer position to obtain the reconstructed gaze region; Based on the gaze point coordinates and the reconstructed gaze region in the table data, the gaze object at each moment is determined, and the gaze object recognition result is obtained.

2. The pilot gaze object recognition method based on eye-tracking data according to claim 1, characterized in that, The preliminary identification results include the type of the object being gazed upon, the coordinates of its center point, and its geometric dimensions; the objects being gazed upon include instrument-type objects; the instrument-type objects being gazed upon include attitude tables, heading indicators, and altimeters.

3. The pilot gaze object recognition method based on eye-tracking data according to claim 2, characterized in that, The steps of checking and correcting the preliminary recognition results based on the relative positional relationship between the gaze objects to obtain the corrected gaze object recognition results specifically include: Using the attitude table and heading indicator table as reference objects, calculate the distance, angle and lateral consistency between the object to be inspected and the two reference objects. If all three meet the preset threshold conditions, the object to be inspected is retained; otherwise, it is discarded. Based on the center point coordinates of the attitude table and heading indicator table, and combined with the preset statistical parameters of the positional relationship between gaze objects, the unidentified gaze objects are estimated. If the estimated gaze object is located within the image boundary, it is added to the preliminary identification result to obtain the corrected gaze object identification result.

4. The pilot gaze object recognition method based on eye-tracking data according to claim 3, characterized in that, Using the attitude table and heading indicator table as reference objects, the distance, angle, and lateral consistency between the object to be inspected and the two reference objects are calculated. If all three meet the preset threshold conditions, the object to be inspected is retained; otherwise, it is discarded. The specific steps include: Define the average distance from the gaze object C to the pose table as: The average distance from the object being observed, C, to the heading indicator is The average angle between the object being observed (C) and the heading indicator at the center point of the attitude indicator is: The center point of the object to be examined The coordinates are ; First, determine whether the distance between the object to be inspected and points A and B meets the requirements: Calculate the center point of the object to be examined The actual distances to the center point A of the attitude indicator and the center point B of the heading indicator, respectively. and : ； in, Here are the coordinates of the center point A in the attitude table; The coordinates of the center point B of the heading indicator; Calculate the distance error separately and : ； like and , If the distance is within the preset threshold, the distance is determined to meet the requirements; otherwise, the distance is determined to not meet the requirements, and the object to be inspected is removed. Secondly, determine whether the angle meets the requirements: Calculate the center point of the object to be examined The actual angle formed by the heading indicator and the attitude indicator at the center point of the attitude indicator. Size; Calculate vectors and : ； ； Calculate vectors and dot product and modulus: ； ； ； Calculate vectors and The included angle is... : ； Average angle error : ； like , If the angle is within the preset threshold, the angle is determined to meet the requirements; otherwise, the angle is determined to not meet the requirements, and the object to be inspected is removed. Then, determine whether the lateral consistency meets the requirements: Calculate vectors and cross product : ； If the object of focus C should be to the left of the line AB connecting the center point A of the attitude indicator and the center point B of the heading indicator, then when If the lateral consistency is within the acceptable range, it is determined that the lateral consistency meets the requirements; otherwise, it is determined that the lateral consistency does not meet the requirements. If the object of focus C should be to the right of the line AB connecting the center point A of the attitude indicator and the center point B of the heading indicator, then when If the lateral consistency is within the acceptable range, it is determined that the lateral consistency meets the requirements; otherwise, it is determined that the lateral consistency does not meet the requirements.

5. The pilot gaze object recognition method based on eye-tracking data according to claim 4, characterized in that, Based on the center point coordinates of the attitude table and heading indicator table, and combined with preset statistical parameters of the positional relationship between gaze objects, the steps of estimating unidentified gaze objects and adding them to the preliminary recognition results to obtain the corrected gaze object recognition results specifically include: Using the attitude indicator and heading indicator as reference objects, calculate the distance between the center point A of the attitude indicator and the center point B of the heading indicator. : ； Calculate the heading angles at attitude table center point A and heading table center point B using the arctangent function. : ； Determine the orientation angles of the center point A of the posture table and the gaze object C based on the lateral orientation of the gaze object. If the object being gazed upon, C, is to the left of AB, then: ； Conversely: ； Use distance and direction angle Estimate the initial coordinates of the object being gazed upon, C: ； Obtain the initial estimated point ; Use distance For the initial estimated point The coordinates are verified and adjusted, and the initial estimated points are calculated. Actual distance to the center point B of the heading indicator : ； like , Accepted at a preset threshold As an estimate, the estimated coordinates of the object being gazed at are obtained; otherwise, the adjustment step is initiated. The adjustment uses a ray-based optimization method to adjust the position of point C of the object being gazed at on the initial ray starting from the center point A of the attitude table, thereby obtaining the estimated coordinates of the object being gazed at. If the estimated coordinates of the gaze object are within the image boundary, the gaze object is added to the preliminary recognition result to obtain the corrected gaze object recognition result; otherwise, it is not added.

6. The pilot gaze object recognition method based on eye-tracking data according to claim 5, characterized in that, The adjustment steps specifically include: Define the parametric equation of the ray, where the ray originates from point A and has a direction angle of θ. The distance parameter is t: ； Define error function : ； in, , and These are the weighting coefficients. ; exist Within the range, search for the optimal distance parameter that minimizes the error. ,use The estimated coordinates of the object being gazed upon are calculated as follows: 。 7. The pilot gaze object recognition method based on eye-tracking data according to claim 1, characterized in that, The Hough transform method is used to detect the pointer position of an instrument-type gaze object.

8. The pilot gaze object recognition method based on eye-tracking data according to claim 1, characterized in that, The method for reconstructing the gaze region of this type of gaze object based on the pointer position is as follows: For a gaze object with a pointer, draw a circle with the pointer's length as the radius and one-third of the distance from the pointer's endpoint as the center. This circle is the reconstructed gaze region. For a gaze object without a pointer, the reconstructed gaze area is a circle with its center point as the center and the larger of the length and width of the recognition box as the radius.

9. The pilot gaze object recognition method based on eye-tracking data according to claim 1, characterized in that, The method for determining the object of gaze at each moment is as follows: Iterate through all objects being watched and determine whether each one is the current object being watched. For a circular gaze object, calculate the distance from the gaze point to the center of the circle and compare it with the radius of the reconstructed gaze region; if the distance from the gaze point to the center of the circle is less than the radius of the reconstructed gaze region, then it is determined to be the current gaze object. For non-circular gaze objects, the cross product of the gaze point and the lower edge line of the reconstructed gaze region is used to determine whether the gaze point is located within the reconstructed gaze region, and the determination is made in conjunction with the left and right boundary conditions.

10. A pilot gaze object recognition system based on eye-tracking data, used to implement the pilot gaze object recognition method based on eye-tracking data according to any one of claims 1-9, characterized in that, include: An eye-tracking data acquisition module is used to collect eye-tracking data of pilots during flight training; the eye-tracking data includes tabular data recorded according to the sampling frequency and synchronously acquired video data; The model training module is used to sample the video data into images, construct a training dataset, train the YOLOV8n neural network, and obtain the YOLO lightweight object detection model. The preliminary object identification module is used to identify the video data to be identified frame by frame based on the YOLO lightweight object detection model, and obtain the preliminary identification result of the object being identified in each frame image; The recognition result checking and correction module is used to check and correct the preliminary recognition result based on the relative positional relationship between the gaze objects, so as to obtain the corrected gaze object recognition result; The gaze region reconstruction module is used to detect the pointer position of an instrument-type gaze object based on the corrected gaze object recognition result, and reconstruct the gaze region of the gaze object according to the pointer position to obtain the reconstructed gaze region; The gaze object determination module is used to determine the gaze object at each moment based on the gaze point coordinates in the table data and the reconstructed gaze area, and obtain the gaze object recognition result.

Images