A pilot change blind assessment system and method based on deep learning

Abstract

This invention relates to the field of flight vision assessment technology, and discloses a pilot change blindness assessment system and method based on deep learning. The system includes: a data acquisition module, which acquires and aligns cockpit display video sequences and pilot eye movement trajectory sequences to obtain a set of semantically relevant regions; a change anchor point extraction module, which extracts image frames before and after the change and generates a change anchor point mask; a change region determination module, which determines a set of change target regions and a set of change neighborhoods; a gaze sequence construction module, which determines event-level gaze sequences and change contact ratios; a closed-loop feature determination module, which determines change registration closed-loop features; a mismatch determination module, which determines eye movement registration features, image-side change features, and change registration mismatch; and a blindness assessment output module, which outputs a single-event change blindness assessment value, a change blindness level, and an overall change blindness assessment value for the entire flight mission. This invention achieves automated and quantitative assessment of the pilot's change blindness state.

Description

Technical Field

[0001] This invention belongs to the field of flight visual assessment technology, specifically relating to a pilot change blind vision assessment system and method based on deep learning. Background Technology

[0002] With the development of modern avionics display systems, the types and quantities of information displayed in the cockpit are constantly increasing. Pilots need to quickly scan, identify, and process information from multiple interfaces within a limited time. During flight, parameters, symbols, alarm prompts, and status information on the display interface may change dynamically. Under high-load mission conditions, pilots may fail to detect, confirm, or react effectively to critical changes in a timely manner, a phenomenon known as change blindness. This phenomenon not only affects the efficiency of pilots in acquiring interface information but may also adversely impact flight operation safety and mission execution effectiveness.

[0003] Existing methods for assessing pilot visual status often focus on single eye-tracking metrics such as fixation duration, scan rate, and hotspot distribution, or rely on manual video playback and annotation to analyze pilot responses to interface changes. These methods struggle to accurately establish a correlation between interface changes and pilot eye-tracking behavior, and also fail to comprehensively reflect the pilot's process of detecting, dwelling on, revisiting, and confirming changes in the target. Especially in the complex display environment of the cockpit, relying solely on single eye-tracking parameters or simple image differencing results often fails to provide an objective, stable, and interpretable assessment of the degree of blindness to change. Summary of the Invention

[0004] This invention provides a pilot change blindness assessment system and method based on deep learning, which solves the technical problems in related technologies that make it difficult to establish the correspondence between cockpit interface change information and pilot eye movement behavior, and are unable to reliably distinguish between two change blindness states: not seeing the change and the line of sight reaching the change area but not completing effective visual registration.

[0005] This invention provides a deep learning-based pilot change blindness assessment system, comprising: The data acquisition module is used to acquire cockpit display video sequences and pilot eye movement trajectory sequences, perform time alignment, and divide the cockpit display interface into regions to obtain a set of semantically relevant regions. The change anchor point extraction module is used to extract the image frames before and after the change from the cockpit display video sequence in response to change events, and to determine the deep visual features before the change, the deep visual features after the change, and the change anchor point mask. The change region determination module is used to determine the set of change target regions based on the change anchor point mask and the set of semantic interest regions, and to determine the change neighborhood set based on the set of change target regions and the set of semantic interest regions. The gaze sequence construction module is used to determine the event-level gaze sequence based on the pilot's eye movement trajectory sequence and the set of semantic attention regions, and to determine the variable contact ratio based on the event-level gaze sequence and the variable anchor point mask; The closed-loop feature determination module is used to determine the closed-loop features of change registration based on the event-level gaze sequence, the set of changed target regions, the set of changed neighborhoods, the change contact ratio, and the start time of the change event. Among them, the closed-loop features of change registration include: first contact delay, effective registration dwell ratio, number of second follow-up visits, and completeness of closed-loop verification. The mismatch determination module is used to determine eye-tracking registration features, image-side change features, and change registration mismatch based on event-level gaze sequences, change registration closed-loop features, deep visual features before change, deep visual features after change, change anchor mask, set of change target regions, and set of change neighborhoods. The blindness assessment output module is used to determine the blindness assessment value of a single event change, the blindness level of the change, and the overall blindness assessment value of the change for the entire flight mission based on the first contact delay, effective registration dwell ratio, number of second visits, completeness of closed-loop verification, and change registration mismatch.

[0006] This invention also provides a deep learning-based method for pilot change blindness assessment, comprising the following steps: Step 81: Obtain the cockpit display video sequence and the pilot's eye movement trajectory sequence, perform time alignment, and divide the cockpit display interface into regions to obtain a set of semantically relevant regions; Step 82: For the change event, extract the image frames before the change and the image frames after the change from the cockpit display video sequence, and determine the deep visual features before the change, the deep visual features after the change, and the change anchor point mask. Step 83: Based on the change anchor point mask and the set of semantic interest regions, determine the set of change target regions, and based on the set of change target regions and the set of semantic interest regions, determine the set of change neighborhoods; Step 84: Based on the pilot's eye movement trajectory sequence and the set of semantic attention regions, determine the event-level gaze sequence, and determine the variable contact ratio based on the event-level gaze sequence and the variable anchor point mask; Step 85: Based on the event-level gaze sequence, the set of changed target regions, the set of changed neighborhoods, the change contact ratio, and the start time of the change event, determine the closed-loop characteristics of change registration; among which, the closed-loop characteristics of change registration include: first contact delay, effective registration dwell ratio, number of second follow-up visits, and completeness of closed-loop verification; Step 86: Based on the event-level gaze sequence, change registration closed-loop features, deep visual features before change, deep visual features after change, change anchor mask, set of change target regions, and set of change neighborhoods, determine the eye-tracking registration features, image-side change features, and change registration mismatch. Step 87: Based on the initial contact delay, effective registration retention ratio, number of secondary visits, completeness of closed-loop verification, and change registration mismatch, determine the single-event change blindness assessment value, change blindness level, and overall change blindness assessment value for the entire flight mission.

[0007] The beneficial effects of this invention are as follows: By uniformly aligning the cockpit display video sequence with the pilot's eye movement trajectory sequence, and sequentially constructing change anchor points, change target areas, event-level gaze sequences, and change registration closed-loop features around change events, this invention establishes a stable correspondence between actual changes in the interface and the pilot's visual responses. This avoids the crudeness and subjective bias of traditional methods that rely solely on gaze duration, gaze frequency, or manual observation. Furthermore, by extracting eye movement registration features and image-side change features separately and calculating the change registration mismatch between the two, this invention effectively distinguishes between different states such as not seeing changes and the gaze reaching the change area but not completing cognitive registration, improving the accuracy and stability of change blindness recognition. This invention outputs the change blindness assessment value and level result corresponding to a single change event and comprehensively evaluates multiple change events throughout the entire flight mission, forming a hierarchical assessment result from single event to the entire mission, improving the completeness, interpretability, and engineering application value of the assessment results. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of a pilot change blindness assessment system based on deep learning according to the present invention; Figure 2 This is a flowchart of a pilot change blindness assessment method based on deep learning according to the present invention. Detailed Implementation

[0009] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0010] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of the present invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of the present invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0011] like Figures 1-2 As shown, a deep learning-based pilot change blindness assessment system includes: The data acquisition module is used to acquire cockpit display video sequences and pilot eye movement trajectory sequences, perform time alignment, and divide the cockpit display interface into regions to obtain a set of semantically relevant regions. The change anchor point extraction module is used to extract the image frames before and after the change from the cockpit display video sequence in response to change events, and to determine the deep visual features before the change, the deep visual features after the change, and the change anchor point mask. The change region determination module is used to determine the set of change target regions based on the change anchor point mask and the set of semantic interest regions, and to determine the change neighborhood set based on the set of change target regions and the set of semantic interest regions. The gaze sequence construction module is used to determine the event-level gaze sequence based on the pilot's eye movement trajectory sequence and the set of semantic attention regions, and to determine the variable contact ratio based on the event-level gaze sequence and the variable anchor point mask; The closed-loop feature determination module is used to determine the closed-loop features of change registration based on the event-level gaze sequence, the set of changed target regions, the set of changed neighborhoods, the change contact ratio, and the start time of the change event. Among them, the closed-loop features of change registration include: first contact delay, effective registration dwell ratio, number of second follow-up visits, and completeness of closed-loop verification. The mismatch determination module is used to determine eye-tracking registration features, image-side change features, and change registration mismatch based on event-level gaze sequences, change registration closed-loop features, deep visual features before change, deep visual features after change, change anchor mask, set of change target regions, and set of change neighborhoods. The blindness assessment output module is used to determine the blindness assessment value of a single event change, the blindness level of the change, and the overall blindness assessment value of the change for the entire flight mission based on the first contact delay, effective registration dwell ratio, number of second visits, completeness of closed-loop verification, and change registration mismatch.

[0012] In one embodiment of the present invention, a cockpit display video sequence and a pilot eye movement trajectory sequence are first acquired, and time alignment between the two and regional division of the cockpit display interface are completed. The cockpit display video sequence refers to a set of image frames arranged in chronological order obtained by continuously acquiring data from the cockpit display interface during the pilot's flight mission, wherein each image corresponds to a video frame; the pilot eye movement trajectory sequence refers to a set of time-series data formed by continuously recording the pilot's gaze position, sampling time, and optional eye movement state information using an eye movement acquisition device; the eye movement state information may include fixation, saccades, blinking, etc.; time alignment refers to establishing a time correspondence between the two types of heterogeneous time-series data based on the time markers corresponding to the video frames and eye movement sampling points, so that any eye movement trajectory point can correspond to the video frame at which its occurrence time is located or the closest one.

[0013] Specifically, in step 11, the timestamps of each video frame in the cockpit display video sequence are collected, and the timestamps of each eye movement trajectory point in the pilot's eye movement trajectory sequence are simultaneously acquired. Since the video sampling frequency and eye movement sampling frequency are usually different, there is no one-to-one correspondence between video frames and eye movement trajectory points. Therefore, this invention maps each eye movement trajectory point to the corresponding video frame in time according to the timestamp correspondence. During mapping, the eye movement trajectory point can be assigned to the current video frame within the time interval, or it can be assigned to the video frame with the closest time distance, thus obtaining the aligned pilot's eye movement trajectory sequence. After alignment, each video frame can be associated with one or more eye movement trajectory points, and each eye movement trajectory point also has a clear video context, enabling subsequent analysis of the pilot's gaze response to interface changes under a unified temporal reference.

[0014] Step 12: After time alignment, the system divides the cockpit display interface into regions according to the preset region division rules, obtaining a set of semantically relevant regions. The set of semantically relevant regions refers to the set of regions formed by dividing the interface into areas with independent display or operational significance based on the functional layout and task semantics of the cockpit display interface. Examples include the main flight display area, navigation display area, alarm prompt area, parameter indication area, and auxiliary information area. The preset region division rules can be set based on the functional division of different modules in the display interface, the screen's geometric position, the category of display elements, or the monitoring frequency during the flight mission. Based on the task logic of the cockpit human-machine interface, the interface is divided into multiple regions with clear semantic boundaries, so that each region corresponds to a relatively stable display function or information category. Region division can be achieved using manually calibrated templates, interface coordinate configuration files, rule engines, or a combination of visual recognition results. For the same device model or display configuration, the boundaries of the semantically relevant regions can be stored first and then directly retrieved later. For display screens with scaling, offset, or resolution changes, position correction can be performed by combining interface reference points or feature markers to ensure that the region division results are consistent with the actual displayed content.

[0015] By temporally aligning the cockpit display video sequence with the pilot's eye movement trajectory sequence and semantically segmenting the cockpit display interface, this embodiment can establish a correspondence between interface change information and visual gaze behavior under a unified temporal benchmark, improving the accuracy of multi-source data fusion analysis. Furthermore, it can provide structured input with semantic constraints for deep learning models, enhancing their ability to discriminate key display changes, thereby improving the objectivity, automation, and stability of pilot blindness assessment.

[0016] In one embodiment of the present invention, for a change event, image frames before and after the change are extracted from the cockpit display video sequence, and deep visual features before and after the change, as well as a change anchor mask, are determined accordingly. A change event refers to an event in which the same displayed object, content, state, or attribute in the cockpit display interface changes. For example, a change can manifest as the appearance, disappearance, displacement, shape change, color change, brightness change, symbol switching, value refresh, or alarm state change of a target. The image frame before the change refers to an image frame that fully represents the original display state of the changed object before the corresponding change event occurs; the image frame after the change refers to an image frame that fully represents the changed display state of the changed object after the corresponding change event occurs.

[0017] Specifically, in step 21, for the change event to be analyzed, determine the image frames before the change (fully displayed before the change) and after the change (fully displayed after the change) in the cockpit display video sequence, where the image frames before and after the change correspond to the same change event. In practice, key frames corresponding to the events before and after the change can be located in the video sequence based on the change event's time stamp, display state switching time, task log information, or manual annotation results. To ensure the accuracy of subsequent difference analysis, image frames where the changed object is in a stable display state before and after the change are preferentially selected, rather than image frames in the display transition phase, intermediate refresh phase, or partial flicker phase. This method reduces the impact of dynamic interface refresh, short-term flicker, or non-target disturbances on the change analysis results, thereby improving the stability of change event extraction.

[0018] Step 22 involves inputting the unchanged and post-changed image frames obtained in Step 21 into a shared-parameter visual encoding network. Multi-level feature extraction is performed on both images using the same set of network parameters, resulting in the unchanged deep visual features corresponding to the unchanged image frame and the post-changed deep visual features corresponding to the post-changed image frame. The shared-parameter visual encoding network uses a visual feature extraction network with identical structure and parameters for both the unchanged and post-changed image frames. This ensures that both images are encoded within the same feature representation space, avoiding additional biases introduced by inconsistent feature extraction rules. The visual encoding network can employ convolutional neural networks, residual networks, visual Transformer networks, or combinations thereof. Multi-level feature extraction refers to extracting image features step-by-step from shallow to deep layers. Shallow features can retain more edge, texture, and local shape information, while deep features can further represent regional structural relationships and the semantics of displayed content. Encoding the unchanged and post-changed images using a shared-parameter approach helps to characterize the differences in display states under a unified feature benchmark, making subsequent difference localization results more comparable.

[0019] Step 23: Based on the deep visual features before and after the change, establish corresponding positional relationships and calculate the differences between corresponding positions. Use the difference values ​​at each corresponding position as the response values ​​for the corresponding positions in the change response map, thus obtaining the change response map. The corresponding positional relationship refers to the one-to-one correspondence between the deep visual features before and after the change in spatial dimensions. This correspondence can be directly determined by the feature layout of the two images output from the same visual coding network, or it can be established through position correction, feature alignment, or interpolation mapping when there are scale changes, slight shifts, or viewpoint perturbations. The difference calculation refers to measuring the difference between the feature vectors or feature values ​​at corresponding positions. Methods such as absolute difference, squared difference, and distance measurement can be used to reflect the degree of feature change at that position before and after the change. The resulting change response map is essentially a spatial representation of the distribution of differences between the images before and after the change. Positions with higher response values ​​typically correspond to areas where the change is more significant or more likely to be related to the target change, while positions with lower response values ​​typically correspond to stable background areas or areas where no significant changes have occurred.

[0020] Step 24: Based on the change response map, positions with response values ​​not less than a preset threshold are marked as changed positions, and positions with response values ​​less than the preset threshold are marked as unchanged positions, thus obtaining a change anchor mask. The preset threshold is a judgment threshold used to distinguish between regions of significant change and regions of insignificant change. It can be set based on training sample statistical results, empirical rules, adaptive distribution characteristics, or task scenario requirements. After threshold segmentation, positions with a high degree of change in the change response map are retained, forming clear markers for the changed regions. The change anchor mask is a binary marker map, which not only indicates the main occurrence area of ​​change events in the image but also provides positional constraints for subsequent target change region extraction, neighborhood range construction, eye-tracking gaze matching, and change registration mismatch analysis.

[0021] This embodiment extracts image frames before and after the change and performs unified feature representation in a visual coding network with shared parameters. This accurately reflects the changed position of the cockpit display interface at a deep feature level, reducing the problem of noise interference in the original pixel difference. The further generated change response map and change anchor point mask can provide reliable image-based evidence for subsequent target localization and blind change assessment, thereby improving the accuracy and robustness of the assessment results.

[0022] In one embodiment of the present invention, after obtaining the change anchor mask, the target region related to the current change event is determined by further combining it with a set of semantically relevant regions, and a set of change neighborhoods adjacent to the change target region is constructed on this basis. The set of change target regions refers to the set of semantically relevant regions in the set of semantically relevant regions that have a high degree of spatial overlap with the change anchor mask and can reflect the main location of the current change event; the set of change neighborhoods refers to the set of semantically relevant regions in the set of semantically relevant regions that satisfy a preset adjacency relationship with the set of change target regions, but do not belong to the set of change target regions themselves. The preset adjacency relationship set is a set of regional relationships pre-established based on the spatial boundary contact relationships between various semantically relevant regions, used to represent the proximity connection state between different semantically relevant regions.

[0023] Specifically, in step 31, based on the change anchor mask and the set of semantic interest regions, the overlapping area of ​​the change anchor mask within the boundaries of each semantic interest region is calculated, and the ratio of the overlapping area to the total area within the boundary of the corresponding semantic interest region is determined as the overlap coverage rate. The overlapping area refers to the pixel area or equivalent area of ​​the change position marked by the change anchor mask falling within a semantic interest region; the total area within the boundary of the corresponding semantic interest region refers to the complete area covered by the semantic interest region itself. By normalizing the overlapping area to the total area of ​​the semantic interest regions, the influence of differences in size between different semantic interest regions on the judgment result can be eliminated, making the degree of change correlation between different regions comparable. The overlap coverage rate is used to represent the degree to which a semantic interest region is affected by the current change event. The higher the overlap coverage rate, the stronger the spatial correspondence between the region and the change anchor mask, and the more likely the region is to be the direct carrier region of this change event.

[0024] Step 32: Based on the overlap coverage rates and a preset overlap coverage threshold uniformly set for the semantically interested region set, semantically interested regions with an overlap coverage rate not less than the preset overlap coverage threshold are selected from the semantically interested region set to obtain the set of change target regions. The preset overlap coverage threshold is used to distinguish between semantically interested regions with a high degree of correlation to the change event and those with a low degree of correlation. It can be set according to the statistical results of training samples, empirical rules, task scenario requirements, or the display layout characteristics of different device models. Through this selection process, target regions directly related to the current change event can be located from all semantically interested regions, thereby further mapping the change anchor points obtained from pixel-level or feature-level change responses to a region level with clear task semantics.

[0025] Step 33: Based on the set of change target regions, the set of semantically interested regions, and the preset adjacency relationship set corresponding to the set of semantically interested regions, select semantically interested regions from the set of semantically interested regions that satisfy the preset adjacency relationship with the set of change target regions but do not belong to the set of change target regions, thus obtaining the change neighborhood set. The preset adjacency relationship set is established based on the spatial boundary contact relationship between semantically interested regions. Specifically, when two semantically interested regions have boundary contact, boundary adjacency, shared boundary lines, or satisfy a proximity relationship within a preset distance range in the interface layout, they can be considered to have an adjacency relationship. Based on this adjacency relationship, after determining the set of change target regions, it is possible to further identify its surrounding semantically interested regions that are adjacent to it but do not directly carry change events, and these regions are taken as the change neighborhood set. Constructing a change neighborhood set not only focuses on the target area where the change actually occurs, but also preserves the local semantic environment around the target area of ​​the change. This helps to analyze whether the pilot's gaze has shifted to the adjacent area, or whether the pilot has searched around the target area of ​​the change but failed to complete the effective change registration. This helps to more accurately distinguish between different states such as the pilot has noticed the change, is close to noticing the change but has not completed the confirmation, and has not responded effectively to the change at all.

[0026] This embodiment maps change anchor point masks onto a set of semantically relevant regions, enabling the semantic identification of change target regions directly related to change events. Furthermore, it constructs a set of adjacent change neighborhoods, thus mapping image change information to interface functional areas. This approach enhances the semantic clarity of change event localization and provides more complete regional contextual information for subsequent eye-tracking matching analysis and blind change assessment.

[0027] In one embodiment of the present invention, an event-level gaze sequence is determined based on the pilot's eye movement trajectory sequence and a set of semantic attention regions. This sequence is further combined with a change anchor mask to determine the change contact ratio, representing the pilot's actual visual contact with the changing location during a specific change event. The event-level gaze sequence refers to a temporal sequence extracted and organized from the pilot's eye movement trajectory sequence within the event time window corresponding to a specific change event, reflecting the change process of the pilot's gaze region. The change contact ratio refers to the degree to which the pilot's eye movement trajectory point falls within the change location marked by the change anchor mask within the event time window of the corresponding change event, reflecting the direct contact between the pilot's line of sight and the changing region.

[0028] Specifically, in step 41, each eye movement point in the pilot's eye movement trajectory sequence is determined based on its position on the cockpit display interface. It is then determined whether it lies within the boundary of a semantic interest region (SIP) within the set of SIPs. If it does, it is marked as the corresponding SIP; otherwise, it is marked as zero. This yields a sequence of SIP labels for each eye movement point. The SIP label sequence is a sequence of region labels formed by assigning a corresponding region label to each eye movement point according to its temporal order. It is used to represent the distribution and shift of the pilot's gaze across different SIPs. A zero label indicates that the corresponding eye movement point does not fall within any preset SIP. This situation may correspond to a blank edge area of ​​the interface, a position outside the region boundary, or a deviation caused by acquisition errors. For example, when an eye movement point is located inside the main flight display area, it can be marked as the main flight display area; when an eye movement point falls at the interface edge outside two SIPs, it can be marked as zero.

[0029] Step 42: For the event time window of the k-th change event, extract eye movement points located within the event time window from the semantic attention region labeling sequence corresponding to the eye movement points obtained in Step 41. Merge eye movement points with the same labeling and a time interval not exceeding a preset interval within a continuous time period into a gaze unit, forming an event-level gaze sequence corresponding to the k-th change event. Here, the event time window refers to the analysis time interval corresponding to the k-th change event. Its start and end times can be set according to the occurrence time of the change event, the duration of the event, task requirements, or evaluation strategies, used to limit the eye movement analysis range of the current change event. The gaze unit refers to the set of eye movement points where, within a continuous time period, the pilot's gaze remains continuously on the same semantic attention region, and the time interval between adjacent eye movement points does not exceed a preset interval. Setting a preset interval can prevent incorrect segmentation of gaze behavior that should be continuous due to short-term sampling interruptions, instantaneous jitter, or sampling frequency fluctuations. That is, if multiple adjacent eye movement points have the same region labeling, and the time interval between them is within the allowable range, these eye movement points can be considered to constitute the same gaze unit. For example, if, within the event time window of a changing event, multiple consecutive eye movement points of the pilot are located in the navigation display area, and the time interval between adjacent eye movement points does not exceed a preset interval, then these eye movement points can be merged into a single gaze unit corresponding to the navigation display area. If the pilot's gaze subsequently shifts to the warning area and remains there for a period of time, then these eye movement points can form another gaze unit. The sequence of multiple gaze units arranged in chronological order constitutes the event-level gaze sequence corresponding to the changing event. In this way, discrete eye movement points can be integrated into an event-level gaze representation that better conforms to the laws of visual behavior, thereby more clearly reflecting the pilot's area dwell and attention shift during the changing event.

[0030] Step 43: Based on the event-level gaze sequence and change anchor mask obtained in Step 42, determine whether the eye movement trajectory points within each gaze unit in the event-level gaze sequence fall into the change position marked by the change anchor mask. Calculate the ratio of the number of points falling into the change position to the total number of eye movement trajectory points in the event-level gaze sequence, and use this ratio as the change contact ratio for the k-th change event. The change contact ratio reflects the degree of actual spatial contact between the pilot's line of sight and the change position within the analysis time range of the current change event. A high change contact ratio indicates that the pilot has a large number of eye movement trajectory points falling into the change position during the change event, resulting in a high probability of direct contact with the change area. A low change contact ratio indicates that although the pilot may have observed the interface, there is less direct visual contact with the actual change area, suggesting insufficient coverage of the change position. For example, within the event time window corresponding to the k-th change event, if there are 100 eye movement trajectory points in the event-level gaze sequence, and 30 of them fall into the change position marked by the change anchor mask, then the change contact ratio for the k-th change event can be determined to be 0.3.

[0031] This embodiment maps the pilot's eye movement trajectory sequence to a set of semantic attention regions, constructs an event-level gaze sequence corresponding to the changing event, and further combines a change anchor point mask to calculate the change contact ratio. This can characterize the pilot's actual gaze contact with the changing region at the event level, and can simultaneously take into account the temporal range constraints of the changing event and the spatial range constraints of the changing location, thereby more accurately reflecting whether the pilot has formed effective visual coverage of the key changing region.

[0032] In one embodiment of the present invention, based on the event-level gaze sequence, the set of changing target regions, the set of changing neighborhoods, the changing contact ratio, and the start time of the changing event, a change registration closed-loop feature is further determined to characterize the pilot's discovery, stay, departure, review, and closed-loop confirmation process of the changing target after the occurrence of the changing event. The change registration closed-loop feature reflects whether the pilot has formed a relatively complete perception, verification, and confirmation process of the changing target.

[0033] Specifically, in step 51, based on the event-level gaze sequence and the set of changing target regions, gaze units whose gaze regions belong to the set of changing target regions are identified, and the gaze unit that appears first in chronological order is determined as the first contact unit. The first contact unit refers to the gaze unit that appears earliest in time in the event-level gaze sequence corresponding to the current changing event, and whose gaze region belongs to the set of changing target regions. When the gaze region corresponding to a gaze unit falls into the set of changing target regions, it can be considered that the pilot has made direct gaze contact with the changing target region during that time period. Among multiple gaze units that meet the conditions, the one that appears first in chronological order is determined as the first contact unit. For example, if the event-level gaze sequence corresponding to a certain changing event passes through the parameter indication region, the changing target region, the changing neighborhood region, and the changing target region in sequence, then the gaze unit that first falls into the changing target region can be determined as the first contact unit.

[0034] Step 52: Based on the first contact unit and the start time of the change event, the time difference between the start time of the first contact unit and the start time of the change event is determined as the first contact delay. The sum of the durations of gaze units whose gaze area belongs to the set of change target areas is used as the numerator, and the sum of the durations of all gaze units in the event-level gaze sequence is used as the denominator. The ratio of the numerator to the denominator is determined as the effective registration dwell ratio. The start time of the change event refers to the moment when the current change event begins to occur in the cockpit display interface, which can be determined by the change event annotation results, the interface state switching time, or the video analysis results. The first contact delay reflects the reaction time experienced by the pilot from the occurrence of the change to the first gaze to the change target area. The shorter the first contact delay, the earlier the pilot usually contacts the change target area. The effective registration dwell ratio reflects what proportion of gaze time the pilot allocates to the change target area during the current change event analysis. This indicator does not only focus on whether the change target area is briefly scanned, but further measures the pilot's sustained dwell time on the change target area. For example, in an event-level gaze sequence of a certain change event, if the sum of the duration of all gaze units is 10 seconds, and the sum of the duration of gaze units belonging to the set of change target regions is 4 seconds, then the effective registration dwell ratio can be determined to be 0.4.

[0035] Step 53: Based on the initial contact unit and the event-level gaze sequence, count the number of times a pilot re-enters the changing target area set after the initial contact unit, and there are gaze units whose gaze area does not belong to the changing target area set between the pilot and the previously counted gaze units in the changing target area set. This count yields the number of secondary revisits. The number of secondary revisits emphasizes the pilot's behavior of returning to the changing target area after the initial contact, leaving the area, passing through other non-changing target areas, and then re-entering the changing target area. If two gaze units belonging to the changing target area set are temporally adjacent and there are no non-changing target area gaze units in between, they are generally considered as the same continuous target gaze process and are not counted repeatedly. Only when there is at least one gaze unit whose gaze area does not belong to the changing target area set between two entries into the changing target area is it counted as a new revisit. For example, if the event-level gaze sequence is changing target area, changing neighborhood area, auxiliary information area, changing target area, then the subsequent entry into the changing target area can be counted as a secondary revisit; if the event-level gaze sequence is changing target area, changing target area, changing target area, it is usually not counted as multiple revisits. By counting the number of secondary visits, we can reflect whether pilots return to the target area to confirm after initially detecting a changed target, thus helping to distinguish between casual scanning and proactive re-examination.

[0036] Step 54: Based on the event-level gaze sequence, the set of changing target regions, and the set of changing neighborhoods, the number of paths in the event-level gaze sequence that start from the set of changing target regions and enter the set of changing neighborhoods is counted as the number of loop closure candidates. The number of times the path returns to the set of changing target regions is counted as the number of loop closures formed. The ratio of the number of loop closures formed to the number of loop closure candidates is determined as the loop closure verification completeness. The first contact delay, effective registration dwell ratio, number of secondary visits, and loop closure verification completeness are determined as the change registration loop characteristics. The number of loop closure candidates refers to the number of times the pilot starts from the changing target region and moves to the set of changing neighborhoods during the event-level gaze process. The number of loop closures formed is the number of times the pilot returns to the set of changing target regions. That is, only when the pilot's gaze path forms a closed verification process from the changing target region to the set of changing neighborhoods and back to the set of changing target regions is it considered as forming a loop. The loop closure verification completeness is used to reflect the degree to which the pilot returns to the target region to complete the confirmation after verifying the changing target and its surrounding area. For example, if a pilot enters the changed neighborhood set from the changed target area four times during a certain change event, and then returns to the changed target area three times, the number of closed-loop candidates is four, the number of closed-loop formations is three, and the closed-loop verification completeness can be determined as 0.75. This indicator reflects whether the pilot has formed a relatively complete target verification behavior chain, rather than just stopping at a single contact or departure. Furthermore, when combined with the change contact ratio analysis, it can enhance the ability to determine whether the pilot has truly completed the change registration. For example, when the change contact ratio is high and the closed-loop verification completeness is also high, it usually indicates that the pilot not only contacted the changed area but also conducted a relatively thorough verification and confirmation around that changed area; conversely, when the change contact ratio is low or the closed-loop verification completeness is low, it may indicate that although the pilot has passed through the area to some extent, a relatively complete change registration process has not yet been formed.

[0037] This embodiment extracts first-contact delay, effective registration-to-stay ratio, number of second-visit visits, and closed-loop verification completeness to characterize the pilot's registration process for changing targets from multiple dimensions, including discovery, stay, review, and confirmation. Compared to a single gaze indicator, this method is more effective in reflecting whether the pilot has formed a complete change verification behavior chain, thereby improving the accuracy and interpretability of change blindness assessment.

[0038] In one embodiment of the present invention, based on event-level gaze sequences, change registration closed-loop features, pre-change deep visual features, post-change deep visual features, change anchor mask, change target region set, and change neighborhood set, eye-tracking registration features, image-side change features, and change registration mismatch are further determined to jointly reflect the pilot's perception and registration of change events from both the eye-tracking behavior side and the image change side. Specifically, eye-tracking registration features represent the pilot's visual confirmation process of change events, i.e., the visual processing process by which the pilot contacts, lingers on, revisits, and verifies the change target region after the change event occurs; image-side change features represent the change intensity, change location, and change context information of the change event itself in the image space; and change registration mismatch reflects the degree of consistency between the pilot's visual registration process and the actual change event.

[0039] Specifically, in step 61, based on the event-level gaze sequence, the number of region transitions between adjacent gaze units arranged in chronological order is counted to obtain a region transition matrix. The ratio of the number of region transitions from each starting region to each target region to the sum of the number of region transitions from that starting region to all target regions is determined as the region transition probability matrix. The region transition probability matrix can be expressed as: ,in, This represents the region transition probability of moving from region i to region j in the k-th change event. This represents the number of region transitions from region i to region j, where n represents the gaze cell index. This represents the total number of gaze units in the event-level gaze sequence corresponding to the k-th change event. This represents the gaze region marker corresponding to the nth gaze unit in the kth change event. Indicates an indicator function, when and When M is active, the value is 1; otherwise, the value is 0. M represents the total number of semantically relevant regions. Indicates a region index. The constant represents the number of times a pilot moves from one semantic region to another during an event-level gaze. Each element in the matrix represents the number of times a pilot moves from a starting region to a target region. The region transition probability matrix is ​​obtained by normalizing the region transition matrix and reflects the relative probability of a pilot moving from a starting region to each target region. For example, if in a changing event, the pilot moves from the changing target region to the changing neighborhood set twice and to the auxiliary information region once, then the region transition probability from the changing target region to the changing neighborhood set can be determined as 2 / 3, and the region transition probability to the auxiliary information region can be determined as 1 / 3.

[0040] Step 62: Based on the event-level gaze sequence, region transition probability matrix, and change registration closed-loop features, a unified feature extraction is performed on the event-level gaze sequence, region transition probability matrix, and change registration closed-loop features through an eye-tracking registration coding network to obtain eye-tracking registration features. The eye-tracking registration coding network is a feature extraction network used for joint modeling of eye-tracking behavior information. It extracts high-dimensional feature representations that characterize the pilot's change registration status from gaze sequence information, region transition relationship information, and change registration closed-loop behavior information. The unified feature extraction refers to jointly encoding multiple types of eye-tracking related inputs with different temporal and structural attributes through the same eye-tracking registration coding network, enabling the fusion of eye-tracking behavior information from different sources within a unified feature representation framework. The event-level gaze sequence mainly reflects the pilot's gaze dwell process in the time dimension, the region transition probability matrix mainly reflects the transition relationship features between regions, and the change registration closed-loop features further reflect the pilot's discovery, dwelling, revisiting, and verification completeness of the changing target. By jointly encoding the above information, more comprehensive eye-tracking registration features can be obtained. For example, if two change events have similar change contact ratios, but one event has a higher number of secondary follow-up visits and a higher degree of closed-loop verification, while the other event has no obvious follow-up behavior, then after extraction by the eye-tracking registration coding network, the two can exhibit different eye-tracking registration characteristics, which is helpful in distinguishing whether the pilot has truly formed a relatively complete change registration process.

[0041] Step 63: Based on the deep visual features before and after the change, the change anchor mask, the set of change target regions, and the set of change neighborhoods, a unified feature extraction is performed on these elements using an image-side change coding network to obtain image-side change features. The image-side change coding network is a feature extraction network used for joint modeling of image information related to change events. Its input includes not only the deep visual features before and after the change, but also the change anchor mask and the set of change target regions and the set of change neighborhoods at the semantic level. This allows image-side feature extraction to consider both the visual differences of the change itself and its distribution location in the semantic region and its surrounding context. The deep visual features before and after the change mainly characterize the content differences in the deep feature space of the image before and after the change. The change anchor mask mainly provides positional constraints on the changed regions. The set of change target regions characterizes the semantic region directly corresponding to the change event, and the set of change neighborhoods is used to supplement the local contextual information surrounding the changed regions. By uniformly encoding the above information through an image-side change coding network, image-side change features oriented towards the current change event can be obtained, thereby more accurately describing the comprehensive performance of the change event in the image space and semantic space.

[0042] Step 64: Based on eye-tracking registration features and image-side change features, map the eye-tracking registration features and image-side change features to a unified feature space. Take the absolute value of the numerical difference at each corresponding position of the mapped eye-tracking registration features and image-side change features, and sum the absolute difference values ​​at each corresponding position to obtain the change registration mismatch. The unified feature space refers to a common feature representation space that makes the eye-tracking registration features and image-side change features comparable after mapping transformation. Since eye-tracking registration features originate from pilot visual behavior, and image-side change features originate from changes in the cockpit display interface, the two usually differ in their original form, feature dimensions, and semantic expression. Therefore, it is necessary to first transform the two types of features into a comparable unified feature space through mapping. Then, take the absolute value of the numerical difference at each corresponding position of the two types of features, and sum the absolute difference values ​​at all corresponding positions to obtain the change registration mismatch. This mismatch reflects the overall deviation between the pilot's eye-tracking registration process and the actual change event features. For example, when image-side change features show that a change event has a relatively obvious target region and a stable neighborhood structure, but the pilot's eye-tracking registration features show insufficient target region dwell time, infrequent follow-up visits, and low closed-loop verification completeness, the difference between the two types of features in the unified feature space is usually large, resulting in a high change registration mismatch. Conversely, if the pilot's gaze contact, dwell time, follow-up visits, and verification process are consistent with the image-side change features, the change registration mismatch is usually low.

[0043] This embodiment extracts eye-tracking registration features and image-side change features, and further determines the change registration mismatch. It can assess the pilot's registration of change events from the perspective of the consistency between visual behavior and actual change content. This is beneficial for comprehensively utilizing eye-tracking and image-side information to improve the accuracy and robustness of blind change recognition.

[0044] In one embodiment of the present invention, based on the initial contact delay, effective registration dwell ratio, number of secondary visits, closed-loop verification completeness, and change registration mismatch, a single-event change blindness assessment value, a change blindness level, and an overall change blindness assessment value for the entire flight mission are further determined to achieve a quantitative assessment of the pilot's change blindness status in a single change event and the entire flight mission. Change blindness refers to the pilot's failure to promptly detect, confirm, or complete effective visual recording of critical changes in the cockpit display interface. The single-event change blindness assessment value is a comprehensive quantitative result formed based on the pilot's contact, dwell, follow-up, verification, and cross-modal consistency during a specific change event, used to reflect the degree of change blindness under that change event. The change blindness level refers to the level judgment result mapped from the single-event change blindness assessment value, used to classify the degree of change blindness into intervals. The overall change blindness assessment value for the entire flight mission reflects the pilot's overall change blindness level throughout the entire mission.

[0045] Specifically, in step 71, based on the initial contact delay, effective registration retention ratio, number of follow-up visits, completeness of closed-loop verification, and mismatch in change registration, each indicator is normalized according to a unified quantification rule, and then weighted and summed according to preset weights to generate a single-event change blindness assessment value. The unified quantification rule refers to the processing rule that uses a unified standard to convert indicators with different dimensions, value ranges, and directions of change into a comparable quantification range. Since the initial contact delay, effective registration retention ratio, number of follow-up visits, completeness of closed-loop verification, and mismatch in change registration differ in numerical meaning, dimensional form, and direction of numerical change, normalization is required before comprehensive calculation to eliminate the impact of inconsistencies in dimensions between different indicators on the assessment results. For indicators positively correlated with change blindness risk, their numerical direction of change can be maintained; for indicators negatively correlated with change blindness risk, inverse transformation can be performed before normalization to ensure that each indicator has a consistent evaluation direction after normalization. After normalization, higher values ​​for each indicator uniformly indicate a higher risk of pilot blindness due to changes, or a weaker effect of pilot change registration. Subsequently, the normalized indicators are weighted and summed according to preset weights to obtain a single-event change blindness assessment value. These preset weights can be set based on training sample statistics, expert experience rules, task scenario requirements, or model optimization results.

[0046] Step 72: Based on the single-event change blindness assessment value, map the single-event change blindness assessment value to a preset level division interval, thus mapping it to a single-event change blindness level. The preset level division interval is used to map continuously changing single-event change blindness assessment values ​​to discrete level results. The level division interval can be set according to sample statistical distribution, expert annotation standards, risk judgment rules, or airworthiness assessment requirements. After level mapping, the numerical form of the single-event change blindness assessment value can be converted into a more intuitive change blindness level. For example, if the normalized single-event change blindness assessment value ranges from 0 to 1, 0 to 0.25 can be classified as a low-risk level, greater than 0.25 and not greater than 0.5 as a relatively low-risk level, greater than 0.5 and not greater than 0.75 as a relatively high-risk level, and greater than 0.75 and not greater than 1 as a high-risk level.

[0047] Step 73: Based on the blind assessment values ​​of individual events corresponding to all changes, a weighted average is calculated for each individual event blind assessment value to obtain the overall blind assessment value for the entire flight mission. The weighted average of the individual event blind assessment values ​​refers to assigning appropriate weights to the individual event blind assessment values ​​corresponding to different changes based on their importance, mission relevance, change type, alarm level, or training focus within the overall flight mission, and then performing a comprehensive calculation based on these weights. Higher event weights can be assigned to changes with high mission criticality, strong indication of importance, or high safety relevance; lower event weights can be assigned to general, auxiliary, or training reference events. In this way, the overall blind assessment value for the entire flight mission not only reflects the pilot's average performance across all changes but also reflects the differentiated contributions of different changes to the overall mission assessment. For example, if there are three changing events in the entire flight mission, with corresponding blind evaluation values ​​of 0.2, 0.5, and 0.8 for each event, and corresponding weights of 0.2, 0.3, and 0.5, then the weighted average of these events will better highlight the impact of high-weight critical events on the overall result. Compared to a simple averaging method, this approach is more suitable for focusing on evaluating critical changing events in flight mission scenarios.

[0048] This embodiment normalizes and integrates the first contact delay, effective registration dwell ratio, number of secondary visits, completeness of closed-loop verification, and change registration mismatch to obtain a single-event change blindness assessment value, which is further mapped to a change blindness level. Finally, the assessment results of each change event in the entire flight mission are comprehensively summarized, enabling multi-level change blindness assessment from single event to the entire mission level. It can measure the degree of change blindness of pilots in specific change events and reflect the pilots' change perception and registration capabilities in the entire mission, which helps to improve the completeness, interpretability, and application value of the assessment results.

[0049] In one embodiment of the present invention, a deep learning-based method for pilot change blindness assessment is also provided, comprising the following steps: Step 81: Obtain the cockpit display video sequence and the pilot's eye movement trajectory sequence, perform time alignment, and divide the cockpit display interface into regions to obtain a set of semantically relevant regions; Step 82: For the change event, extract the image frames before the change and the image frames after the change from the cockpit display video sequence, and determine the deep visual features before the change, the deep visual features after the change, and the change anchor point mask. Step 83: Based on the change anchor point mask and the set of semantic interest regions, determine the set of change target regions, and based on the set of change target regions and the set of semantic interest regions, determine the set of change neighborhoods; Step 84: Based on the pilot's eye movement trajectory sequence and the set of semantic attention regions, determine the event-level gaze sequence, and determine the variable contact ratio based on the event-level gaze sequence and the variable anchor point mask; Step 85: Based on the event-level gaze sequence, the set of changed target regions, the set of changed neighborhoods, the change contact ratio, and the start time of the change event, determine the closed-loop characteristics of change registration; among which, the closed-loop characteristics of change registration include: first contact delay, effective registration dwell ratio, number of second follow-up visits, and completeness of closed-loop verification; Step 86: Based on the event-level gaze sequence, change registration closed-loop features, deep visual features before change, deep visual features after change, change anchor mask, set of change target regions, and set of change neighborhoods, determine the eye-tracking registration features, image-side change features, and change registration mismatch. Step 87: Based on the initial contact delay, effective registration retention ratio, number of secondary visits, completeness of closed-loop verification, and change registration mismatch, determine the single-event change blindness assessment value, change blindness level, and overall change blindness assessment value for the entire flight mission.

[0050] It should be noted that the interval and threshold sizes are set for ease of comparison. The size of the threshold depends on the amount of sample data and the base number set by those skilled in the art for each set of sample data, as long as it does not affect the proportional relationship between the parameter and the quantized value. Furthermore, the above formulas are all dimensionless calculations, and the formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0051] The embodiments of the present invention have been described above, but the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of the present embodiments, all of which are within the protection scope of the present embodiments.

Claims

1. A pilot change blind evaluation system based on deep learning, characterized in that, include: The data acquisition module is used to acquire cockpit display video sequences and pilot eye movement trajectory sequences, perform time alignment, and divide the cockpit display interface into regions to obtain a set of semantically relevant regions. The change anchor point extraction module is used to extract the image frames before and after the change from the cockpit display video sequence in response to change events, and to determine the deep visual features before the change, the deep visual features after the change, and the change anchor point mask. The change region determination module is used to determine the set of change target regions based on the change anchor point mask and the set of semantic interest regions, and to determine the change neighborhood set based on the set of change target regions and the set of semantic interest regions. The gaze sequence construction module is used to determine the event-level gaze sequence based on the pilot's eye movement trajectory sequence and the set of semantic attention regions, and to determine the variable contact ratio based on the event-level gaze sequence and the variable anchor point mask; The closed-loop feature determination module is used to determine the closed-loop features of change registration based on the event-level gaze sequence, the set of changed target regions, the set of changed neighborhoods, the change contact ratio, and the start time of the change event. Among them, the closed-loop features of change registration include: first contact delay, effective registration dwell ratio, number of second follow-up visits, and completeness of closed-loop verification. The mismatch determination module is used to determine eye-tracking registration features, image-side change features, and change registration mismatch based on event-level gaze sequences, change registration closed-loop features, deep visual features before change, deep visual features after change, change anchor mask, set of change target regions, and set of change neighborhoods. The blindness assessment output module is used to determine the blindness assessment value of a single event change, the blindness level of the change, and the overall blindness assessment value of the change for the entire flight mission based on the first contact delay, effective registration dwell ratio, number of second visits, completeness of closed-loop verification, and change registration mismatch.

2. The pilot change blindness assessment system based on deep learning according to claim 1, characterized in that, The cockpit display video sequence and the pilot's eye movement trajectory sequence are acquired, time-aligned, and the cockpit display interface is divided into regions to obtain a set of semantically relevant regions, including: Step 11: Obtain the timestamps of each video frame in the cockpit display video sequence and the timestamps of each eye movement trajectory point in the pilot's eye movement trajectory sequence. According to the timestamp correspondence, map each eye movement trajectory point to the corresponding video frame to obtain the aligned pilot's eye movement trajectory sequence. Step 12: Divide the cockpit display interface into regions according to the preset region division rules to obtain a set of semantically relevant regions.

3. The pilot change blindness assessment system based on deep learning according to claim 1, characterized in that, For change events, image frames before and after the change are extracted from the cockpit display video sequence to determine the deep visual features before the change, the deep visual features after the change, and the change anchor point mask, including: Step 21: For the change event, determine the image frame before the change and the image frame after the change in the cockpit display video sequence that are in the complete display state before the change. The image frame before the change and the image frame after the change correspond to the same change event. Step 22: Input the image frames before and after the change obtained in step 21 into a visual coding network with shared parameters. Perform multi-layer feature extraction on the image frames before and after the change through the visual coding network with shared parameters to obtain the deep visual features before the change corresponding to the image frames before the change and the deep visual features after the change corresponding to the image frames after the change, respectively. Step 23: Based on the deep visual features before and after the change, establish the corresponding positional relationship, calculate the difference between the corresponding positions in the deep visual features before and after the change, and use the difference value of each corresponding position as the response value of the corresponding position in the change response map to obtain the change response map. Step 24: Based on the change response map, mark the positions in the change response map with response values ​​not less than a preset threshold as change positions, and mark the positions in the change response map with response values ​​less than the preset threshold as non-change positions, thus obtaining the change anchor point mask.

4. The pilot change blindness assessment system based on deep learning according to claim 1, characterized in that, Based on the change anchor mask and the set of semantically interested regions, a set of change target regions is determined, and a set of change neighborhoods is determined based on the set of change target regions and the set of semantically interested regions, including: Step 31: Based on the variable anchor mask and the set of semantic interest regions, calculate the area of ​​the overlapping region of the variable anchor mask within the boundary of each semantic interest region, and determine the overlap coverage rate as the ratio of the area of ​​each overlapping region to the area of ​​all regions within the boundary of the corresponding semantic interest region. Step 32: Based on the overlap coverage rates and the preset overlap coverage rate threshold uniformly set for the set of semantically interested regions, select semantically interested regions from the set of semantically interested regions whose overlap coverage rate is not less than the preset overlap coverage rate threshold to obtain the set of change target regions. Step 33: Based on the set of change target regions, the set of semantic interest regions, and the set of preset adjacency relationships corresponding to the set of semantic interest regions, select semantic interest regions from the set of semantic interest regions that satisfy the preset adjacency relationship with the set of change target regions but do not belong to the set of change target regions to obtain the change neighborhood set. The preset adjacency relationship set is established based on the spatial boundary contact relationship between semantic interest regions.

5. The pilot change blindness assessment system based on deep learning according to claim 1, characterized in that, Based on the pilot's eye movement trajectory sequence and semantic attention region set, an event-level gaze sequence is determined, and the variable contact ratio is determined based on the event-level gaze sequence and the variable anchor point mask, including: Step 41: Determine whether each eye track point in the pilot's eye track sequence is located within the boundary of a semantic interest region in the semantic interest region set based on its position on the cockpit display interface. If it is located, mark it as the corresponding semantic interest region; otherwise, mark it as zero. This will give you the semantic interest region marking sequence corresponding to the eye track point. Step 42: For the event time window of the kth change event, extract the eye movement trajectory points located in the event time window from the semantic attention region labeling sequence corresponding to the eye movement trajectory points obtained in Step 41, and merge the eye movement trajectory points with the same labeling in a continuous time period and with a time interval not exceeding a preset interval into a gaze unit to form an event-level gaze sequence. Step 43: Based on the event-level gaze sequence and change anchor mask obtained in step 42, determine whether the eye movement trajectory point in each gaze unit in the event-level gaze sequence falls into the change position marked by the change anchor mask, calculate the ratio of the number of points falling into the change position to the total number of eye movement trajectory points in the event-level gaze sequence, and use this ratio as the change contact ratio of the k-th change event.

6. The pilot change blindness assessment system based on deep learning according to claim 1, characterized in that, Based on event-level gaze sequences, sets of changed target regions, sets of changed neighborhoods, change contact ratios, and the start time of change events, the closed-loop characteristics of change registration are determined. These closed-loop characteristics include: first contact delay, effective registration dwell rate, number of secondary follow-ups, and completeness of closed-loop verification, including: Step 51: Based on the event-level gaze sequence and the set of changing target regions, identify gaze units whose gaze regions belong to the set of changing target regions, and determine the gaze unit that appears first in chronological order as the first contact unit. Step 52: Based on the first contact unit and the start time of the change event, the time difference between the start time of the first contact unit and the start time of the change event is determined as the first contact delay; the sum of the duration of the gaze units whose gaze region belongs to the set of change target regions is used as the numerator, the sum of the duration of all gaze units in the event-level gaze sequence is used as the denominator, and the ratio of the numerator to the denominator is determined as the effective registration dwell ratio. Step 53: Based on the first contact unit and the event-level gaze sequence, count the number of times a gaze unit re-enters the set of changing target regions after the first contact unit, and there is a gaze unit whose gaze region does not belong to the set of changing target regions with the gaze unit of the previous counted set of changing target regions, and obtain the number of second revisits. Step 54: Based on the event-level gaze sequence, the set of changed target regions, and the set of changed neighborhoods, count the number of paths in the event-level gaze sequence that start from the set of changed target regions and enter the set of changed neighborhoods as the number of loop closure candidates, count the number of times the path returns to the set of changed target regions as the number of loop closures formed, and determine the ratio of the number of loop closures formed to the number of loop closure candidates as the loop closure verification completeness; determine the first contact delay, effective registration dwell ratio, number of second visits, and loop closure verification completeness as the change registration loop closure features.

7. The pilot change blindness assessment system based on deep learning according to claim 1, characterized in that, Based on event-level gaze sequences, change registration closed-loop features, pre-change deep visual features, post-change deep visual features, change anchor mask, set of change target regions, and set of change neighborhoods, eye-tracking registration features, image-side change features, and change registration mismatch are determined, including: Step 61: Based on the event-level gaze sequence, count the number of region transitions between adjacent gaze units arranged in chronological order to obtain the region transition matrix, and determine the region transition probability matrix by the ratio of the number of region transitions from each starting region to each target region to the sum of the number of region transitions from the starting region to all target regions. Step 62: Based on the event-level gaze sequence, region transition probability matrix, and change registration closed-loop features, perform unified feature extraction on the event-level gaze sequence, region transition probability matrix, and change registration closed-loop features through an eye-tracking registration coding network to obtain eye-tracking registration features; Step 63: Based on the deep visual features before the change, the deep visual features after the change, the change anchor mask, the set of changed target regions, and the set of changed neighborhoods, a unified feature extraction is performed on the deep visual features before the change, the deep visual features after the change, the change anchor mask, the set of changed target regions, and the set of changed neighborhoods through an image-side change coding network to obtain the image-side change features. Step 64: Based on the eye-tracking registration features and image-side change features, map the eye-tracking registration features and image-side change features to a unified feature space. Take the absolute value of the numerical difference between the corresponding positions of the mapped eye-tracking registration features and the mapped image-side change features, and sum the absolute difference values ​​of the corresponding positions to obtain the change registration mismatch.

8. A pilot change blindness assessment system based on deep learning according to claim 1, characterized in that, Based on initial contact delay, effective registration-to-stay ratio, number of secondary visits, completeness of closed-loop verification, and mismatch in change registration, the single-event change blindness assessment value, change blindness level, and overall change blindness assessment value for the entire flight mission are determined, including: Step 71: Based on the first contact delay, effective registration retention ratio, number of second follow-up visits, completeness of closed-loop verification, and mismatch of change registration, normalize each indicator according to a unified quantitative rule, and generate a single event change blind assessment value by weighted summation according to preset weights. Step 72: Based on the single-event change blindness assessment value, map the single-event change blindness assessment value to the preset level division interval and map it to the single-event change blindness level. Step 73: Based on the blind assessment values ​​of individual events corresponding to all the changing events, perform a weighted average of the blind assessment values ​​of each individual event to obtain the overall blind assessment value of the changes for the entire flight mission.

9. A deep learning-based method for pilot change blindness assessment, characterized in that, The pilot change blindness assessment system based on deep learning as described in any one of claims 1-8 includes the following steps: Step 81: Obtain the cockpit display video sequence and the pilot's eye movement trajectory sequence, perform time alignment, and divide the cockpit display interface into regions to obtain a set of semantically relevant regions; Step 82: For the change event, extract the image frames before the change and the image frames after the change from the cockpit display video sequence, and determine the deep visual features before the change, the deep visual features after the change, and the change anchor point mask. Step 83: Based on the change anchor point mask and the set of semantic interest regions, determine the set of change target regions, and based on the set of change target regions and the set of semantic interest regions, determine the set of change neighborhoods; Step 84: Based on the pilot's eye movement trajectory sequence and the set of semantic attention regions, determine the event-level gaze sequence, and determine the variable contact ratio based on the event-level gaze sequence and the variable anchor point mask; Step 85: Based on the event-level gaze sequence, the set of changed target regions, the set of changed neighborhoods, the change contact ratio, and the start time of the change event, determine the closed-loop characteristics of change registration; among which, the closed-loop characteristics of change registration include: first contact delay, effective registration dwell ratio, number of second follow-up visits, and completeness of closed-loop verification; Step 86: Based on the event-level gaze sequence, change registration closed-loop features, deep visual features before change, deep visual features after change, change anchor mask, set of change target regions, and set of change neighborhoods, determine the eye-tracking registration features, image-side change features, and change registration mismatch. Step 87: Based on the initial contact delay, effective registration retention ratio, number of secondary visits, completeness of closed-loop verification, and change registration mismatch, determine the single-event change blindness assessment value, change blindness level, and overall change blindness assessment value for the entire flight mission.

Images