A pilot communication capability evaluation method and system based on multimodal data

Abstract

The present invention belongs to the field of data processing technology and relates to a pilot communication capability assessment method and system based on multimodal data, aiming to solve the problem that traditional methods rely only on single behavioral data and have poor scene adaptability. The present invention includes: constructing a feature standard baseline, neural activity feature thresholds, and a scene-indicator-feature correlation matrix; collecting neural layer, speech layer, physiological layer, and operational layer data and aligning them in time series; extracting feature values ​​from the data and marking anomalies; constructing a double-hidden layer Markov model to determine the real-time state distribution and calculate dynamic weights; calculating scores based on weights and feature values ​​and outputting results. The present invention conducts a deep fusion assessment of cognition, behavior, and scenes through dynamic mapping of features and states and scene-adaptive dynamic weights, realizing cross-layer correlation analysis from behavioral representations to neural mechanisms, breaking through the limitations of traditional assessments that rely on single behavioral data and have poor scene adaptability.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a pilot communication capability evaluation method and system based on multimodal data. Background Art

[0002] In the aviation safety system, pilots' communication capabilities are a core element in ensuring flight safety, directly impacting flight operational efficiency and emergency response effectiveness. Current mainstream pilot communication capability assessment methods still have significant technical limitations, making it difficult to accurately assess pilots in complex scenarios.

[0003] Traditional assessment methods often rely on single-dimensional behavioral data (such as voice command accuracy and operation response time), resulting in a focus on behavioral performance over cognitive root causes. For example, analyzing voice standardization through audio recordings or recording operation delays in simulators only reflects the pilot's outward behavioral results, but fails to address deeper cognitive mechanisms such as whether operational delays stem from excessive cognitive load or whether command misjudgments are due to distraction. Consequently, assessment conclusions lack the ability to trace back to the source of weaknesses.

[0004] Although existing multimodal assessment systems attempt to integrate voice and physiological data (such as heart rate and skin electricity), they face three key technical bottlenecks: First, the data synchronization accuracy is insufficient. The timing deviation between the neural layer (such as EEG and near-infrared) and the behavioral layer data often exceeds 50ms, making it difficult to establish a causal relationship between "cognition and behavior"; second, the scenario adaptability is poor. The fixed-weight scoring system is used, and the dynamic differences in the importance of indicators in scenarios such as normal cruising and special situation handling (such as engine failure) are not taken into account, resulting in an assessment error of up to 30% in special situation scenarios; third, the abnormality identification is simplistic, and abnormalities are judged only by behavioral data thresholds (such as response time > 2 seconds), ignoring the coordinated abnormal patterns of neural signals and behavioral performance (such as a sudden increase in blood oxygen change rate accompanied by operational errors), which can easily lead to misjudgment.

[0005] Furthermore, existing technologies lack a deep application mechanism for assessment results, making it impossible to generate targeted training plans based on assessment data. For example, when a pilot's adaptability is found to be insufficient, it is difficult to determine whether it is due to a "lack of strategy library" or "low cognitive switching efficiency." This results in a lack of scientific basis for training improvements and hinders the systematic improvement of pilots' communication capabilities.

[0006] Therefore, there is an urgent need for an assessment method with stronger adaptability and the ability to achieve cognitive-behavioral synergy. Summary of the Invention

[0007] To address the aforementioned problems in the prior art, namely, the problem that traditional methods rely solely on single behavioral data and have poor scenario adaptability, the first aspect of the present invention proposes a pilot communication capability assessment method based on multimodal data, the method comprising the following steps:

[0008] Based on the pre-built scenario-indicator-feature correlation matrix and dual baselines, four-dimensional data is collected in real time and time series aligned;

[0009] The four-dimensional data includes neural layer, speech layer, physiological layer, and operational layer data, wherein the neural layer data includes physiological characteristic parameters reflecting neural activity; the scenario-indicator-feature association matrix and dual baseline are constructed based on historical data and a preset hierarchical evaluation system, which includes primary indicators and secondary indicators, and each primary indicator has multiple secondary indicators under it;

[0010] Based on the aligned real-time data, the characteristic values ​​of each secondary indicator are extracted and abnormal features are marked;

[0011] Construct a double hidden layer Markov model, calculate the matching degree and state transition probability based on the eigenvalues, and then determine the state distribution;

[0012] Determining initial weights of the primary and secondary indicators and secondary dynamic weights based on the state distribution;

[0013] The scores are calculated based on the weights at each level and the characteristic values ​​of the secondary indicators, and the output is the pilot communication capability assessment result, and an evaluation report is generated.

[0014] In some preferred embodiments, in the hierarchical evaluation system, the first-level indicators include communication standardization, situational understanding, adaptability, and load management;

[0015] The secondary indicators under communication standardization are call sign response delay, command repetition accuracy and terminology standardization; the secondary indicators under situational understanding are airspace dynamic correlation, special situation command decoding speed and crew intention matching; the secondary indicators under adaptability are error correction timeliness, special situation communication strategy switching speed and multi-task collaboration efficiency; the secondary indicators under load management are cognitive load peak control, load recovery speed and rationality of cognitive resource allocation.

[0016] In some preferred embodiments, a scenario-indicator-feature correlation matrix and a dual baseline are constructed based on historical data and a preset evaluation system, and the method is as follows:

[0017] Obtain historical data and establish dual baselines for each secondary indicator in the preset hierarchical evaluation system based on the historical data. The dual baselines are the baseline values ​​of the behavioral performance of each indicator and the characteristic thresholds of the corresponding neural layer;

[0018] Based on the set dual baselines, historical data on flight accidents are analyzed to extract the correlation strength between different flight scenarios and primary indicators, secondary indicators, and four-dimensional features;

[0019] Based on the correlation strength, the core first-level indicators and core second-level indicators corresponding to each scenario in the historical data are determined, as well as the key features of the neural layer, speech layer, physiological layer, and operational layer required by the core second-level indicators, and a scenario-indicator-feature correlation matrix is ​​constructed.

[0020] In some preferred embodiments, four-dimensional data is collected in real time based on a pre-built scenario-indicator-feature association matrix and a dual baseline, and the method is as follows:

[0021] Set acquisition parameters based on the scenario-indicator-feature correlation matrix, including setting the sampling rate of neural layer data based on the strong correlation between the scenario and the neural layer features; determine the acquisition duration based on the baseline values ​​of the behavioral performance of each indicator in the dual baseline and the corresponding neural layer feature threshold;

[0022] Obtain the time series characteristics of each secondary indicator, including behavioral characteristics and neural features .

[0023] In some preferred embodiments, timing alignment is performed by:

[0024] Taking the start time of the evaluation scenario as the origin, the time interval is divided according to the scenario stage. Each stage corresponds to a clear time interval, marked as a first-level time scale. ;

[0025] Use key events in the scene as time nodes to record the absolute time when the event occurs , marked as the secondary time scale and associated with the primary time scale, that is, each event node corresponds to a specific position in the scene time scale:

[0026] ;

[0027] in, First-level time scale The start time node; First-level time scale The end time node;

[0028] Based on the clock of the acquisition device, the behavioral data and neural data corresponding to each secondary indicator are time-series aligned:

[0029] Marking behavior data is marked as , the corresponding neural data time scale is ; and determine the neural response lag corresponding to the secondary index , correcting the time scale of the neural data to ; Through the device clock synchronization calibration, Mapping to secondary time scale Corresponding timeline to achieve behavioral characteristics and neural features Under the same event node.

[0030] In some preferred embodiments, the characteristic values ​​of each secondary indicator are extracted and abnormal characteristics are marked as follows:

[0031] Denoising preprocessing based on aligned real-time data;

[0032] Extract the quantitative characteristic values ​​of each secondary indicator of the preprocessed data, and determine whether there is a coordinated abnormality based on the dual baseline slice:

[0033] When the characteristic value of any secondary indicator in the real-time data of the evaluated pilot exceeds the behavioral baseline and the corresponding neural layer feature exceeds the neural layer baseline, an abnormal mark is added to the quantitative feature.

[0034] In some preferred embodiments, the state distribution is determined by:

[0035] The dual-hidden-layer Markov model includes two layers of hidden states, the upper layer is the communication state, including the preparation period, execution period, strain period and correction period, and the lower layers are the cognitive load states, including low, medium and high;

[0036] Based on the correlation matrix, determine the initial state distribution of the upper layer communication state and the lower layer cognitive load state in different scenarios ,in, For upper layer communication status, It is the lower cognitive load state;

[0037] The quantitative feature value is used as the model input to determine the feature matching degree between the feature information and the combination of each upper and lower layer state. and state transition probability :

[0038] ;

[0039] ;

[0040] in, The quantitative characteristics of the k-th secondary indicator at time t, including the neural feature value and behavioral characteristic values , Time-corrected and Alignment, is the behavioral data time stamp, Corrected time scale for neural data; is the upper layer communication state at time t, is the lower cognitive load state at time t, represents a normal distribution, Status The quantitative feature mean and sample standard deviation of the historical data under; is the initial transition probability based on historical data; Target state The optimal behavior characteristic value of the k-th secondary indicator; is the neural layer baseline value corresponding to the k-th secondary indicator; The scene adaptation coefficient of the kth secondary indicator; is the sensitivity coefficient;

[0041] Based on the state transition probability, the transition probability between the combination states is updated, and the probability ratio of each combination state is adjusted according to the feature matching degree to obtain the real-time state distribution.

[0042] In some preferred embodiments, the real-time state distribution is dynamically iterated along with the state transition probability:

[0043] The state transition matrix is ​​constructed with the corrected state transition probability. The initial updated state distribution is obtained by multiplying the matrix with the current real-time state distribution vector. The feature matching degree is then used as the weight to perform a secondary adjustment on the distribution. The algorithm is iterated until the distribution converges to form the final real-time state distribution.

[0044] In some preferred embodiments, the weight of each indicator is dynamically adjusted according to the state distribution, and the method is as follows:

[0045] According to the state distribution, a first-level indicator judgment matrix and a second-level indicator judgment matrix under each first-level indicator are constructed, and a 1-9 scale is used to quantify the relative importance of indicators at the same level;

[0046] The first-level indicator judgment matrix and the second-level indicator judgment matrix are solved by the eigenvalue method to obtain the first-level initial weight vector and the second-level initial weight vector, and the sum of the second-level initial weights under the same level indicator is 1;

[0047] The scenario coefficient is determined based on the strength of the correlation between the scenario and the first-level indicator, and the dynamic weight of the second-level indicator is obtained by multiplying the second-level initial weight by the corresponding scenario coefficient; among which, the sum of the second-level dynamic weights under the same-level indicator is equal to the scenario coefficient.

[0048] In some preferred embodiments, the scores are calculated based on the weights of each level and the characteristic values ​​of the secondary indicators, and the method is as follows:

[0049] Based on the dynamic weights of the secondary indicators, the scores of the corresponding secondary indicators are weighted and summed to obtain the scores of the corresponding primary indicators;

[0050] If the characteristic value of a secondary indicator is marked as abnormal during the score calculation, the score of the secondary indicator is calculated with a preset weight lower than 1, and the preset weight in the special scenario is lower than the preset weight in the normal scenario; otherwise, the score of the secondary indicator is the product of its characteristic value and 10;

[0051] Based on the first-level initial weight vector, the scores of each first-level indicator are weighted and summed to obtain the final score of the pilot's communication ability.

[0052] A second aspect of the present invention provides a pilot communication capability assessment system based on multimodal data, the system comprising:

[0053] a data acquisition module configured to acquire four-dimensional data in real time; the data acquisition module includes a NIRS device, a microphone, a physiological sensor, and an operation simulator, and is used to acquire neural layer data, speech layer data, physiological layer data, and operation layer data;

[0054] The preprocessing module is configured to pre-build the scene-indicator-feature correlation matrix and determine the behavioral baseline and neural layer baseline. After collecting the four-dimensional data, the four-dimensional data is synchronized and aligned in time series through a three-level time-scale synchronization mechanism based on the correlation matrix.

[0055] The scenario-indicator-feature association matrix is ​​constructed based on historical data and a preset hierarchical evaluation system. The preset hierarchical evaluation system includes primary indicators and secondary indicators. Each primary indicator has multiple secondary indicators.

[0056] The feature extraction module is configured to extract the feature values ​​of each secondary indicator based on the aligned real-time data and mark abnormal features;

[0057] The modeling and analysis module is configured to construct a double-hidden-layer Markov model, calculate the matching degree and state transition probability based on the eigenvalues, realize the dynamic mapping between features and states, and then determine the state distribution;

[0058] a weight determination module configured to determine initial weights and secondary dynamic weights of the primary and secondary indicators based on the state distribution;

[0059] The score calculation module is configured to calculate the score based on the weights of each level and the characteristic values ​​of the secondary indicators, output the results of the pilot's communication ability assessment, and generate an evaluation report.

[0060] Beneficial effects of the present invention:

[0061] 1. This invention combines a deep coupling mechanism between neural layers and multimodal data with scenario-adaptive dynamic weights, and performs full-process collaborative anomaly recognition. This enables a deeply integrated assessment of cognition, behavior, and scenario. It can effectively identify shortcomings in pilots' communication capabilities and trace their root causes based on the assessment results. This breaks through the limitations of traditional assessments that rely on single behavioral data and have poor scenario adaptability, provides a scientific basis for developing targeted training programs, and helps systematically improve pilots' communication capabilities, thereby ensuring flight safety.

[0062] 2. Neural layer data is embedded as a cognitive benchmark throughout the entire process. Deep coupling of neural layer and multimodal data creates a dual baseline. This addresses the root cause tracing failure caused by the separation of behavior and cognition in traditional assessments, enables cross-modal anomaly labeling, and realizes cross-layer correlation analysis from behavioral manifestations to neural mechanisms.

[0063] 3. A collaborative verification mechanism based on the dual-dimensional baseline of "behavioral characteristics and neural signals" conducts full-process linkage verification. During the acquisition phase, the data collection scope is determined based on the dual baseline. During the feature acquisition phase, the deviation is calculated and those exceeding the threshold are marked as abnormal. During the modeling process, the abnormality mark is used as a correction factor for the state mapping to ensure that abnormality identification is carried out throughout the entire assessment process. This effectively avoids single-dimensional misjudgment and improves the accuracy of abnormality identification and assessment results.

[0064] 4. A scenario-adaptive dynamic weighting system has been constructed. Based on the correlation matrix, the weights are dynamically adjusted through the scenario coefficients, so that the weight distribution matches the scenario risk level in real time. This solves the problem of assessment distortion caused by fixed weights in complex scenarios, improves the scenario adaptability of the assessment, and can better meet the assessment needs in different scenarios.

[0065] 5. By designing a double hidden layer through a double hidden layer Markov model and combining the observation probability with the corrected state transition probability formula, a precise mapping of multimodal features and dynamic states is achieved, which solves the fuzzy analytical defect of the traditional model for state transitions, improves the accuracy of state recognition, and provides a more reliable state basis for subsequent evaluation and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0067] Figure 1 The present invention is a flowchart of a method for evaluating pilot communication capabilities based on multimodal data in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0069] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0070] The present invention provides a pilot communication capability assessment method based on multimodal data. This method combines dual baselines, scenario-adaptive dynamic weights, a dual-hidden-layer HMM state mapping model, and full-process collaborative anomaly recognition logic to deeply couple neural layer data with speech, physiological, and operational data. This method implements a "cognition-behavior-scenario" deep fusion assessment of pilot communication capability, effectively solving the problem that traditional assessment methods rely solely on single behavioral data and have poor scenario adaptability.

[0071] A pilot communication capability evaluation method based on multimodal data of the present invention comprises the following steps:

[0072] S1. Based on the pre-built scenario-indicator-feature correlation matrix and dual baselines, four-dimensional data is collected in real time and time series aligned.

[0073] The four-dimensional data includes neural layer, speech layer, physiological layer, and operational layer data, wherein the neural layer data includes physiological characteristic parameters reflecting neural activity; the scenario-indicator-feature association matrix and dual baseline are constructed based on historical data and a preset hierarchical evaluation system, which includes primary indicators and secondary indicators, and each primary indicator has multiple secondary indicators under it;

[0074] S2. Based on the aligned real-time data, extract the characteristic values ​​of each secondary indicator and mark abnormal features;

[0075] S3. Construct a double hidden layer Markov model, calculate the matching degree and state transition probability based on the eigenvalues, and then determine the state distribution;

[0076] S4. Determine the initial weights of the primary and secondary indicators and the secondary dynamic weights based on the state distribution;

[0077] S5. Calculate the scores based on the weights at each level and the characteristic values ​​of the secondary indicators, output the results of the pilot's communication capability assessment, and generate an evaluation report.

[0078] In order to more clearly illustrate the pilot communication ability evaluation method based on multimodal data of the present invention, the following is combined with Figure 1 Each step in the embodiment of the present invention is described in detail.

[0079] The pilot communication capability assessment method based on multimodal data according to the first embodiment of the present invention includes steps S1 to S5, each of which is described in detail as follows:

[0080] S1. Based on the pre-built scenario-indicator-feature correlation matrix and dual baselines, four-dimensional data is collected in real time and time series alignment is performed to solve the spatiotemporal consistency problem of multimodal data.

[0081] The four-dimensional data includes neural layer, speech layer, physiological layer and operational layer data, wherein the neural layer data includes physiological characteristic parameters reflecting neural activity; the scenario-indicator-feature association matrix and dual baseline are constructed based on historical data and a preset hierarchical evaluation system, and the preset hierarchical evaluation system includes first-level indicators and second-level indicators, and each first-level indicator has multiple second-level indicators.

[0082] Preferably, the hierarchical evaluation system includes multiple first-level indicators and multiple second-level indicators, specifically:

[0083] The first-level indicator is: communication standardization , situational understanding , adaptability and load management ;

[0084] The secondary indicators are: the secondary indicators under communication standardization are call sign response delay, command repetition accuracy and terminology standardization; the secondary indicators under situational understanding are airspace dynamic correlation, special situation command decoding speed and crew intention matching; the secondary indicators under adaptability are error correction timeliness, special situation communication strategy switching speed and multi-task collaboration efficiency; the secondary indicators under load management are cognitive load peak control, load recovery speed and rationality of cognitive resource allocation.

[0085] Preferably, a scenario-indicator-feature correlation matrix and a dual baseline are constructed based on historical data and a preset evaluation system, and the method is as follows:

[0086] Obtain historical data and establish dual baselines for each secondary indicator in the preset hierarchical evaluation system based on the historical data. The dual baselines are the benchmark values ​​of the behavioral performance of each indicator. And the characteristic threshold of the corresponding neural layer ;

[0087] Based on the set dual baselines, historical data on flight accidents are analyzed to extract the correlation strength between different flight scenarios and primary indicators, secondary indicators, and four-dimensional features;

[0088] Determine the core indicators and first-level indicators corresponding to each scenario S in the historical data based on the correlation strength screening Ii , core secondary indicators , and the neural layers required for the core secondary indicators , voice layer , physiological layer , Operation layer Key features and construct the scenario-indicator-feature association matrix R.

[0089] The data collection method can be performed using conventional means in the prior art.

[0090] In this embodiment, the correlation matrix R is based on the scenario S, the first-level index I i , secondary indicators and four-dimensional characteristics The correlation strength is constructed as follows: ;

[0091] in, is the comprehensive correlation strength value between scenarios, indicators and features, For the scene and i The correlation strength between the first-level indicators is in the range of [-1, 1]. Positive values ​​indicate positive correlation, negative values ​​indicate negative correlation, and the larger the absolute value, the stronger the correlation. Indicates the i The first-level indicators and their k The correlation strength of the secondary indicators, ranging from [0,1]; For the i First-level indicator k Secondary indicators and v The correlation strength of the four-dimensional features is [-1, 1], v = 1...4; n is the number of historical samples in this scenario; It is the sum of all samples in the same scene.

[0092] Preferably, the Pearson coefficient is used to calculate the correlation strength r, the historical sample size n is ≥ 50 (covering pilots of different age groups and flight time), and the data normality needs to be verified by the KS test.

[0093] In this embodiment, For strong correlation, when there is strong correlation, the neural layer data sampling rate is increased to 100Hz (originally 50Hz) to ensure the feature capture accuracy in high-correlation scenarios, and multi-device sampling rate adaptation is achieved through device clock synchronization protocols (such as PTP) to ensure that the timestamp accuracy is ≤1ms.

[0094] In this embodiment, based on the 30 historical data collected, the first level indicator is obtained. k Dual baselines for secondary indicators:

[0095] Behavioral baseline:

[0096] ;

[0097] in, For the k The behavioral baseline of the secondary indicator refers to the normal range threshold of a large number of sample behavioral data in a standard scenario; No. b The first collection k Behavioral data of secondary indicators, such as call sign response delay, command repetition accuracy, and other specific behavioral performance values; is the mean of behavioral data, reflecting the average level of sample behavior; is the standard deviation of the behavioral data, reflecting the degree of dispersion of the sample behavioral data;

[0098] Neural layer baseline:

[0099] ;

[0100] in, is the neural layer baseline of the k-th secondary indicator, which is the threshold of the normal fluctuation range of the neural layer characteristics; The prefrontal blood oxygen change rate, which is collected synchronously for the bth time as the kth secondary indicator, is monitored by the fNIRS device to reflect the baseline level of cognitive activity; is the mean of the neural layer data, representing the average state of the sample neural activity; is the standard deviation of the neural layer data, reflecting the discreteness of the sample neural activity data.

[0101] In this embodiment, the characteristic threshold of the neural layer The threshold value of the normal fluctuation range of the prefrontal blood oxygen change rate is set based on the distribution characteristics of the neural layer activity characteristics, covering the normal fluctuation range of the 95% confidence interval; the prefrontal blood oxygen change rate is the prefrontal oxyhemoglobin concentration change rate based on functional near-infrared spectroscopy monitoring. .

[0102] In this example, the 95% confidence interval statistic is selected, and the sample standard deviation to ensure that the baseline covers the vast majority of normal behavior data.

[0103] Further preferably, in this embodiment, the behavioral baseline is calculated based on the threshold confidence interval method according to the behavioral data corresponding to the secondary indicators in the standardized simulated flight scenarios in history; the neural activity characteristics corresponding to the secondary indicators synchronously recorded in the behavioral data are extracted, and the neural layer baseline is calculated based on the threshold confidence interval method.

[0104] Preferably, based on the pre-built scenario-indicator-feature correlation matrix and dual baselines, four-dimensional data is collected in real time by:

[0105] The acquisition parameters are set according to the scene-indicator-feature association matrix, including setting the sampling rate of the neural layer data according to the strong correlation between the scene and the neural layer features; the acquisition time is determined according to the baseline value of the behavioral performance of each indicator in the dual baseline and the corresponding neural layer feature threshold. All secondary indicator feature values ​​must be bound to the acquisition time t and synchronized with the device time scale to obtain the time of each secondary indicator at time t. t The uncalibrated time series features are used to obtain the quantitative feature values ​​containing the time series :

[0106] ;

[0107] in, Including the behavioral characteristic value of the kth secondary indicator at time t and neural eigenvalues , is the weight of the behavioral data.

[0108] In this embodiment, the technology used for collecting data is existing technology, such as using an fNIRS device (8 channels, 50Hz sampling rate) to detect changes in hemoglobin concentration in the prefrontal cortex to reflect cognitive activity (such as an increase in hemoglobin concentration in the dorsolateral prefrontal cortex when attention is focused) to obtain neural layer data; using a microphone to collect voice to obtain voice layer data; using a wrist sensor to collect heart rate variability to obtain physiological layer data; and collecting stick and rudder operation data to obtain operation layer data.

[0109] Further preferably, in this embodiment, the obtained quantized feature values ​​are obtained by a three-level time-scale anchoring method. To perform timing alignment, the method is:

[0110] First-level time scale (scenario time scale): With the start time of the evaluation scenario as the origin, the time interval is divided according to the scenario stage (preparation, execution, special situation, correction). Each stage corresponds to a clear time interval, which serves as the time frame for the overall evaluation and is marked as (S is the scene stage, S=1,2,...,4);

[0111] Secondary time scale (event time scale): takes key events in the scene (such as call sign sending, command issuance, special situation occurrence, etc.) as time nodes, and records the absolute time when the event occurs is the event number, E=1,2,...,m), and is associated with the first-level time scale, that is, each event node corresponds to a specific position in the scene time scale, ;in, First-level time scale The start time node; First-level time scale The end time node;

[0112] Level 3 time scale (data time scale): For different modal data under each secondary indicator, based on the clock of its acquisition device, the behavioral data time scale is , the corresponding neural data time scale is ; Determine the neural response lag corresponding to different secondary indicators , correct the neural data time scale to ; Through the device clock synchronization calibration, Mapping to secondary time scale On the corresponding time axis, obtain the behavioral features after calibration and alignment at the same event node and neural features , the eigenvalues ​​of different modal data are uniformly mapped to the same time axis and aligned at time t.

[0113] In this embodiment, the neural response delay It can be determined based on the average neural response time of the kth secondary indicator in the historical data. , offsetting physiological response delays and technical acquisition delays.

[0114] Preferably, calculate the inherent deviation between different time scales, such as the deviation between the device clock and the standard clock , event detection delay etc., determine the distribution characteristics of the deviation through historical data statistics, and establish a time-scale deviation model ; When acquiring neural data When the original time scale is calibrated by the time scale deviation model, in the process of mapping the third-level time scale (data time scale) to the second-level time scale (event time scale), based on the time scale deviation model Adjust the three-level time scale;

[0115] Ensure that neural characteristics and behavioral characteristics are synchronized in time, so that the neural correction term in the state transition probability formula can accurately reflect the synergistic relationship between physiology and behavior, and provide a reliable quantitative basis for evaluation.

[0116] In this embodiment, in the three-level anchoring, the error threshold of the event node timestamp is ≤50ms to avoid alignment deviation caused by device delay; the neural layer and the event node are associated through the sliding window analysis of the neural layer data at the event triggering time, with a window length of 2s and a step size of 0.5s.

[0117] S2. Based on the aligned real-time data, the quantitative feature values ​​of each secondary indicator are extracted and anomalies are marked through dual baseline collaborative verification. The method is as follows:

[0118] Denoising preprocessing based on aligned real-time data;

[0119] Extract the quantitative characteristic values ​​of each secondary indicator of the preprocessed data, and determine whether there is a coordinated abnormality based on the dual baseline slice:

[0120] When the characteristic value of any secondary indicator in the real-time data of the evaluated pilot exceeds the behavioral baseline and the corresponding neural layer feature exceeds the neural layer baseline, an abnormal mark is added to the quantitative feature.

[0121] In this embodiment, the denoising preprocessing technology is the existing technology, including removing heartbeat and breathing artifacts from the neural layer data and retaining the 0.5-10 Hz cognitive related frequency band; and removing background noise by spectral subtraction of the speech layer data.

[0122] Preferably, in this embodiment, the quantitative characteristic value (quantitative characteristic value) of each secondary index at time t is obtained. ), including the characteristic values ​​of call sign response delay, command repetition accuracy, terminology standardization, airspace dynamic correlation, special situation command decoding speed, crew intention matching, error correction timeliness, special situation communication strategy switching speed, multi-task coordination efficiency, cognitive load peak control, load recovery speed and rationality of cognitive resource allocation, among which, To quantize the eigenvalue Behavioral data in To quantize the eigenvalue Neural data in .

[0123] Get the behavior data corresponding to the secondary indicators. The specific acquisition method is as follows:

[0124] The characteristic value of terminology standardization = the number of standard terms appearing / the total number of terms × 100%; the standardization of pilots' use of professional terms in communications is analyzed as behavioral data;

[0125] The characteristic value of the command repetition accuracy rate = the number of accurately repeated command words / the total number of original command words × 100%. The pilot's repeated command is compared with the original command, and the percentage of accurate repetitions is calculated as behavioral data.

[0126] The characteristic value of the call sign response delay = the time when the call sign instruction is issued - the time when the first response is made (the absolute value). Behavioral data is obtained by recording the time interval from the pilot receiving the call sign to responding.

[0127] The characteristic value of airspace dynamic relevance is the degree of match between the actual airspace description and radar data. Based on the pilot's mention and association of various dynamic information in the airspace (such as the location of other aircraft and weather changes) in communications, behavioral data is extracted to reflect the depth of their understanding of the airspace situation.

[0128] The characteristic value of the special situation command decoding speed = the time the special situation command is received - the time the command meaning is parsed. The time from when the pilot receives the special situation command to when he understands and converts it into his own action command is recorded as behavioral data.

[0129] The characteristic value of the crew intention matching degree = the consistency of the pilot and co-pilot intention matching degree; behavioral data is obtained by comparing the degree of fit between the pilot's communication content and the overall intention of the crew;

[0130] The characteristic value of error correction timeliness = the time when the communication error is discovered - the time when the correction instruction is issued; the time it takes for the pilot to discover the communication error and complete the correction is recorded as behavioral data;

[0131] The characteristic value of the switching speed of the special situation communication strategy = the time when the special situation level is upgraded - the time when the communication strategy adjustment is completed. When encountering a special situation, the time it takes for pilots to switch from the normal communication strategy to the special situation communication strategy is counted as behavioral data.

[0132] The characteristic value of multi-task coordination efficiency = number of communication tasks completed in parallel / total task duration. Observe the pilots' coordination and cooperation when handling multiple communication tasks simultaneously, such as the order and duration of task completion, and quantify this into behavioral data.

[0133] The characteristic value of cognitive load peak control = 1 - actual cognitive load peak / load threshold. This characteristic is calculated based on neural layer data. Behavioral data is extracted based on pilots' behavioral performance in high-load communication scenarios, such as communication fluency and changes in command processing accuracy.

[0134] The characteristic value of load recovery speed = 1 / (peak load moment - time when the load returns to normal level). When the communication load changes from high to low, the time it takes for the pilot's communication behavior to return to normal is recorded as behavioral data.

[0135] The characteristic value of rational cognitive resource allocation = the proportion of cognitive investment in critical tasks / the proportion of cognitive investment in non-critical tasks (calculated based on neural layer data); when the cognitive investment in non-critical tasks is 0, the value is 1. Analyze the time and energy allocation of pilots on different communication tasks, such as the emphasis on handling critical and non-critical tasks, and quantify it into behavioral data.

[0136] At the same time, fNIRS is used to monitor activity changes in specific areas of the brain to obtain neural data corresponding to secondary indicators.

[0137] Based on the above data obtained, specific values ​​are obtained through existing data processing / analysis methods or mathematical statistical methods, and mapped to the [0,1] interval through normalization processing. Among them, time-related indicators are based on the industry standard thresholds in the existing technology (for example, the call sign response delay threshold is set to 2 seconds, and if it exceeds the threshold, it is counted as 0, and if it responds in advance and within a reasonable range, it is counted as 1, and the excess is linearly attenuated). Ratio-related indicators are directly converted into percentages; the thresholds of neural indicators are determined by adopting the non-parametric Bootstrap method with a 95% confidence interval (sampling times = 1000), eliminating abnormal samples outside of 3 standard deviations, and determining the final threshold; the intention matching degree is obtained based on the matching degree between the command data and the operation data; to ensure that the correlation with multimodal features can be accurately calculated through the Pearson coefficient, adapting to the construction requirements of the scenario-indicator-feature association matrix.

[0138] In this embodiment, the collaborative exception is specifically:

[0139] like , then Anomaly=1; otherwise Anomaly=0, Anomaly is an abnormal mark;

[0140] in, To quantize the eigenvalue Behavioral data in To quantize the eigenvalue Neural data in is the corrected neural data time scale, 、 They are the standard deviation of behavioral data and the standard deviation of neural data calculated based on the latest 30 historical data.

[0141] Further preferably, the method for determining whether there is a collaborative abnormality based on the dual baseline is as follows:

[0142] For the secondary indicators, a one-to-one correspondence between the behavioral baseline and the neural layer baseline is established, and collaborative verification is performed: if a certain behavioral data is within the behavioral baseline range, but the corresponding neural feature exceeds the neural layer baseline, the data is re-verified, and the neural layer data for that period is re-extracted to eliminate device noise; if there are still abnormal characteristics of the neural features after re-verification, it will be included in the assessment report as abnormal data and marked as "potential cognitive risk."

[0143] Preferably, samples marked as potential cognitive risks need to be cross-validated with more than three repeated scenario data to confirm whether they are persistent risks.

[0144] S3. Construct a double-hidden-layer Markov model, calculate the matching degree and state transition probability based on the eigenvalues, construct the dynamic relationship between the double-hidden-layer HMM mapping features and "communication state-cognitive load", and output the state distribution.

[0145] Preferably, the state distribution is determined by:

[0146] The double-hidden-layer HMM model includes two layers of hidden states, the upper layer is the communication state, including the preparation period, execution period, strain period and correction period, and the lower layers are the cognitive load states, including low, medium and high;

[0147] Based on the correlation matrix and the frequency of occurrence of each state combination in the historical data, the initial state distribution of the upper communication state and the lower cognitive load state in different scenarios is determined. ,in, The upper layer communication state (preparation period = S1, execution period = S2, response period = S3, correction period = S4), is the lower cognitive load state (low = L1, medium = L2, high = L3);

[0148] The quantitative feature value is used as the model input to determine the feature matching degree between the feature information and the combination of each upper and lower layer state. and state transition probability :

[0149] ;

[0150] ;

[0151] in, The quantitative characteristics of the k-th secondary indicator at time t, including the neural feature value and behavioral characteristic values , Time-corrected and Alignment, is the behavioral data time stamp, Corrected time scale for neural data; is the upper layer communication state at time t, is the lower cognitive load state at time t, represents a normal distribution, Status The quantitative feature mean and sample standard deviation of the historical data under; is the initial transition probability based on historical data; Target state The optimal behavior characteristic value of the k-th secondary indicator; is the neural layer baseline value corresponding to the k-th secondary indicator; The scene adaptation coefficient of the kth secondary indicator; is the sensitivity coefficient;

[0152] Based on the state transition probability, the transition probability between the combination states is updated, and the probability ratio of each combination state is adjusted according to the feature matching degree to obtain the real-time state distribution.

[0153] At time t, the upper layer communication state is (i=1, 2, 3, 4, corresponding to the preparation period, execution period, response period, and correction period) and the lower cognitive load state is The matching probability of the quantified features is calculated when j = 1, 2, or 3, corresponding to low, medium, and high. Here, time t is an integrated time marker that defines the pilot's state at this moment. It is a comprehensive judgment of behavioral and neural features over a period of time, representing a summary of the state up to that moment. The corresponding time scale is the precise timestamp of data collection, recording when a specific feature occurred. These time scales are microscopic, directly tied to the data, and reflect instantaneous observations.

[0154] In this embodiment, Target state The optimal eigenvalue of the state in the historical data is The 90% quantile of the scene; λ (sensitivity coefficient) ranges from [0.5, 2] (the higher the scene complexity, the larger λ); the scene adaptation coefficient is determined by the scene-indicator association strength Dynamic adjustment, the stronger the correlation The larger the is the correlation strength between scenario S and the first-level indicator I belonging to k second-level indicators); based on historical data statistics, when the neural characteristics are abnormal, a minimum weight of 50% is retained to avoid excessive suppression of reasonable transfer.

[0155] Preferably, the real-time state distribution is dynamically iterated along with the state transition probability:

[0156] The time series feature sequence of the quantized eigenvalues ​​is used as the observation sequence of the HMM, and the state transition matrix is ​​constructed with the modified state transition probability. The dynamic relationship between states at different times is described by the state transition probability matrix. The feature matching degree and state transition probability are calculated to obtain a preliminary updated state distribution. The feature matching degree is then used as a weight to perform a secondary adjustment on the distribution. The iteration is carried out until the distribution converges to form the final real-time state distribution.

[0157] The convergence condition is: when the L2 norm of the difference between the state distribution vectors of two iterations is ≤0.01, the iteration is stopped, and the dimension of the state distribution vector is the number of upper-layer states × the number of lower-layer states.

[0158] S4. Combine the state distribution to generate the initial weights of the first-level and second-level indicators and the second-level dynamic weights. The method is as follows:

[0159] According to the state distribution, a first-level indicator judgment matrix and a second-level indicator judgment matrix under each first-level indicator are constructed, and a 1-9 scale is used to quantify the relative importance of indicators at the same level;

[0160] ;

[0161] Among them, the elements Indicator q Relative indicators p Importance, indicators q and indicators p Both are first-level indicators or second-level indicators under a certain first-level indicator, and the two indicators are at the same level;

[0162] By eigenvalue method , solve the first-level indicator judgment matrix and the second-level indicator judgment matrix respectively, and obtain the first-level initial weight vector and the second-level initial weight vector, and the sum of the second-level initial weights under the same level indicator is 1;

[0163] ;

[0164] in, For the The initial weight vector of the level index, z = 1 or 2; each element The first indicator in the z-level m The initial weight vector of each indicator, m=1...n; when z=1, n=4; when z=2, n=3; To determine the maximum eigenvalue of the matrix A, the initial weights are the basis for subsequent scene adjustments and match the initial distribution of the model state;

[0165] According to the scene and i The correlation strength of the first-level indicators determines the scenario coefficient , multiply the secondary initial weight of the first-level indicator by the corresponding scenario coefficient to obtain the dynamic weight of the secondary indicator ;

[0166] ;

[0167] ;

[0168] Among them, the sum of the secondary dynamic weights under the same level indicator is equal to the scenario coefficient. Take the association strength with a maximum value of 0, Indicates the correlation strength between scenario s and the i-th first-level indicator (value range [-1,1]), It is the sum of all secondary initial weights under the same level indicator.

[0169] The scenario coefficient of irrelevant indicators is 0 to avoid the evaluation logic contradiction caused by negative weights. The positive correlation strength retains the original value, such as the cruise scenario. =1.0, engine failure scenario =1.5; secondary initial weight It reflects the relative importance of the indicators in the standard scenario. The dynamic weight needs to be adjusted in combination with the scenario coefficient on this basis, which not only retains the proportional relationship of the secondary initial weights, but also realizes the scenario adaptation through the scenario coefficient, avoiding the destruction of the inherent importance ratio between the indicators due to direct multiplication of the scenario coefficient.

[0170] Preferably, the obtained weights are subjected to a consistency check:

[0171] ;

[0172] ;

[0173] Among them, CR is the consistency of weights, RI is the average random consistency index, and the standard value is determined according to the order of the judgment matrix. The test results are used to modify the judgment matrix and indirectly optimize the degree of match with the model state;

[0174] If CR≥0.1, by adjusting the judgment matrix The value of (the importance of indicator q to p) is changed (if it deviates from the value ±1), and the weight is recalculated until CR<0.1.

[0175] S5. Calculate the scores based on the weights at each level and the characteristic values ​​of the secondary indicators, output the results of the pilot's communication capability assessment, and generate an evaluation report.

[0176] Preferably, the scores are calculated based on the weights of each level and the characteristic values ​​of the secondary indicators in the following way:

[0177] Based on the dynamic weights of the secondary indicators, the scores of the corresponding secondary indicators are weighted and summed to obtain the scores of the corresponding primary indicators;

[0178] ;

[0179] ;

[0180] in, is the quantitative characteristic value of the kth secondary index and ∈[0,1], is the score of the secondary indicator corresponding to the quantitative characteristic value, is the correction coefficient under different scenarios, and the correction coefficient without abnormal mark is 1; For the first-level indicator score, The dynamic weight of the secondary indicator The kth element of k =1, 2, 3, i =1, 2, 3, 4;

[0181] If the characteristic value of a secondary indicator is marked as abnormal during the score calculation, the score of the secondary indicator is calculated with a preset weight lower than 1, and the preset weight in the special scenario is lower than the preset weight in the normal scenario; otherwise, the score of the secondary indicator is the product of its characteristic value and 10;

[0182] Based on the first-level initial weight vector, the scores of each first-level indicator are weighted and summed to obtain the final score of the pilot's communication ability:

[0183] ;

[0184] in, is the scene weight, which is positively correlated with the scene coefficient and is set based on the scene complexity. For the final scoring, the pilot communication ability assessment results are output, including the scores of indicators at all levels, abnormal marks and ability shortcomings analysis, providing a scientific basis for training.

[0185] In this embodiment, the correction coefficient is θ=0.8 in normal scenario, θ=0.5 in special scenario, and θ=1 when Anomaly=0; the scene weight is =1.0, special scene =1.2, so that the weight distribution matches the scenario risk level in real time.

[0186] Further preferably, when outputting the pilot communication capability assessment results, the abnormal feature is traced to explain the neural layer-behavioral layer data segment corresponding to the abnormal feature (such as "abnormal call sign response delay: the blood oxygen change rate of the neural layer prefrontal cortex exceeds the threshold value of 1.2σ, ​​corresponding to a voice response delay of 2.3 seconds").

[0187] Further preferably, after obtaining the evaluation results, the evaluation deviation is determined. If, in a certain scenario, the abnormality rate of the same secondary indicator in three consecutive evaluations is ≥30%, a dual baseline update is triggered, and the static baseline in the scenario is rechecked. If the deviation between any baseline and the current pilot group characteristics is greater than the benchmark threshold, the baseline is re-determined, and the scenario-indicator association strength of the association matrix is ​​adjusted. The evaluation deviation is the deviation between the actual evaluation result and the true ability level.

[0188] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0189] A second embodiment of the present invention provides a pilot communication capability assessment system based on multimodal data, the system comprising:

[0190] a data acquisition module configured to acquire four-dimensional data in real time; the data acquisition module includes a NIRS device, a microphone, a physiological sensor, and an operation simulator, and is used to acquire neural layer data, speech layer data, physiological layer data, and operation layer data;

[0191] The preprocessing module is configured to pre-build the scene-indicator-feature correlation matrix and determine the behavioral baseline and neural layer baseline. After collecting the four-dimensional data, the four-dimensional data is synchronized and aligned in time series through a three-level time-scale synchronization mechanism based on the correlation matrix.

[0192] The scenario-indicator-feature association matrix is ​​constructed based on historical data and a preset hierarchical evaluation system. The preset hierarchical evaluation system includes primary indicators and secondary indicators. Each primary indicator has multiple secondary indicators.

[0193] The feature extraction module is configured to extract the feature values ​​of each secondary indicator based on the aligned real-time data and mark abnormal features;

[0194] The modeling and analysis module is configured to construct a double-hidden-layer Markov model, calculate the matching degree and state transition probability based on the eigenvalues, realize the dynamic mapping between features and states, and then determine the state distribution;

[0195] a weight determination module configured to determine initial weights and secondary dynamic weights of the primary and secondary indicators based on the state distribution;

[0196] The score calculation module is configured to calculate the score based on the weights of each level and the characteristic values ​​of the secondary indicators, output the results of the pilot's communication ability assessment, and generate an evaluation report.

[0197] It should be noted that the aforementioned embodiment provides a pilot communication capability assessment system based on multimodal data, exemplified by the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed. This means that the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the aforementioned embodiments can be combined into a single module or further divided into multiple submodules to perform all or part of the functions described above. The names of the modules and steps in the embodiments of the present invention are merely for the purpose of distinguishing the modules or steps and are not to be construed as unduly limiting the present invention.

[0198] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0199] An electronic device according to a third embodiment of the present invention includes:

[0200] at least one processor; and

[0201] a memory communicatively connected to at least one of the processors; wherein,

[0202] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned pilot communication capability assessment method based on multimodal data.

[0203] A fourth embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are configured to be executed by a computer to implement the aforementioned method for evaluating pilot communication capabilities based on multimodal data.

[0204] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the electronic device and computer-readable storage medium described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0205] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0206] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0207] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0208] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0209] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0210] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A pilot communication capability assessment method based on multimodal data, characterized in that: The method comprises the following steps: Based on the pre-built scenario-indicator-feature correlation matrix and dual baselines, four-dimensional data is collected in real time and time series aligned; The four-dimensional data includes neural layer, speech layer, physiological layer, and operational layer data, wherein the neural layer data includes physiological characteristic parameters reflecting neural activity; the scenario-indicator-feature association matrix and dual baseline are constructed based on historical data and a preset hierarchical evaluation system, which includes primary indicators and secondary indicators, and each primary indicator has multiple secondary indicators under it; Based on historical data and the preset evaluation system, a scenario-indicator-feature correlation matrix and a dual baseline are constructed. The method is as follows: Obtain historical data and establish dual baselines for each secondary indicator in the preset hierarchical evaluation system based on the historical data. The dual baselines are the baseline values ​​of the behavioral performance of each indicator and the characteristic thresholds of the corresponding neural layer; Based on the set dual baselines, historical data on flight accidents are analyzed to extract the correlation strength between different flight scenarios and primary indicators, secondary indicators, and four-dimensional features; Based on the correlation strength, the core first-level indicators and core second-level indicators corresponding to each scenario in the historical data are screened and determined, as well as the key features of the neural layer, speech layer, physiological layer, and operational layer required by the core second-level indicators, and a scenario-indicator-feature correlation matrix is ​​constructed; Based on the pre-built scenario-indicator-feature correlation matrix and dual baselines, four-dimensional data is collected in real time. The method is as follows: Set acquisition parameters based on the scenario-indicator-feature correlation matrix, including setting the sampling rate of neural layer data based on the strong correlation between the scenario and the neural layer features; determine the acquisition duration based on the baseline values ​​of the behavioral performance of each indicator in the dual baseline and the corresponding neural layer feature threshold; Obtain the time series characteristics of each secondary indicator, including behavioral characteristics and neural features ; Based on the aligned real-time data, the characteristic values ​​of each secondary indicator are extracted and abnormal features are marked; Construct a double hidden layer Markov model, calculate the matching degree and state transition probability based on the eigenvalues, and then determine the state distribution; Determining initial weights of the primary and secondary indicators and secondary dynamic weights based on the state distribution; The scores are calculated based on the weights at each level and the characteristic values ​​of the secondary indicators, and the output is the pilot communication capability assessment result, and an evaluation report is generated.

2. The pilot communication capability evaluation method based on multimodal data according to claim 1, characterized in that: In the hierarchical evaluation system, the first-level indicators include communication standardization, situational understanding, adaptability and load management; The secondary indicators under communication standardization are call sign response delay, command repetition accuracy and terminology standardization; the secondary indicators under situational understanding are airspace dynamic correlation, special situation command decoding speed and crew intention matching; the secondary indicators under adaptability are error correction timeliness, special situation communication strategy switching speed and multi-task collaboration efficiency; the secondary indicators under load management are cognitive load peak control, load recovery speed and rationality of cognitive resource allocation.

3. The pilot communication capability evaluation method based on multimodal data according to claim 1, characterized in that: To perform timing alignment, the method is: Taking the start time of the evaluation scenario as the origin, the time interval is divided according to the scenario stage. Each stage corresponds to a clear time interval, marked as a first-level time scale. ; Use key events in the scene as time nodes to record the absolute time when the event occurs , marked as the secondary time scale and associated with the primary time scale, that is, each event node corresponds to a specific position in the scene time scale: ； in, First-level time scale The start time node; First-level time scale The end time node; Based on the clock of the acquisition device, the behavioral data and neural data corresponding to each secondary indicator are time-series aligned: Marking behavior data is marked as , the corresponding neural data time scale is ; and determine the neural response lag corresponding to the secondary index , correcting the time scale of the neural data to ; Through the device clock synchronization calibration, Mapping to secondary time scale Corresponding timeline to achieve behavioral characteristics and neural features Under the same event node.

4. The pilot communication capability assessment method based on multimodal data according to claim 3, characterized in that: Extract the characteristic values ​​of each secondary indicator and mark abnormal features. The method is as follows: Denoising preprocessing based on aligned real-time data; Extract the quantitative characteristic values ​​of each secondary indicator of the preprocessed data, and determine whether there is a coordinated abnormality based on the dual baseline slice: When the characteristic value of any secondary indicator in the real-time data of the evaluated pilot exceeds the behavioral baseline and the corresponding neural layer feature exceeds the neural layer baseline, an abnormal mark is added to the quantitative feature.

5. The pilot communication capability evaluation method based on multimodal data according to claim 4, characterized in that: Determine the state distribution as follows: The dual-hidden-layer Markov model includes two layers of hidden states, the upper layer is the communication state, including the preparation period, execution period, strain period and correction period, and the lower layers are the cognitive load states, including low, medium and high; Based on the correlation matrix, determine the initial state distribution of the upper layer communication state and the lower layer cognitive load state in different scenarios ,in, For upper layer communication status, It is the lower cognitive load state; Quantize the eigenvalues As the model input, determine the matching degree between the feature information and the combination of upper and lower state and state transition probability : ； ； in, is the quantitative feature of the kth secondary indicator at time t, including the neural feature value and behavioral characteristic values , Time-corrected and Alignment, is the behavioral data time stamp, Corrected time scale for neural data; is the upper layer communication state at time t, is the lower cognitive load state at time t, represents a normal distribution, Status The quantitative feature mean and sample standard deviation of the historical data under; is the initial transition probability based on historical data; Target state The optimal behavior characteristic value of the k-th secondary indicator; is the neural layer baseline value corresponding to the k-th secondary indicator; The scene adaptation coefficient of the kth secondary indicator; is the sensitivity coefficient; Based on the state transition probability, the transition probability between the combination states is updated, and the probability ratio of each combination state is adjusted according to the feature matching degree to obtain the real-time state distribution.

6. The pilot communication capability evaluation method based on multimodal data according to claim 4, characterized in that: The weight of each indicator is dynamically adjusted according to the state distribution, and the method is as follows: According to the state distribution, a first-level indicator judgment matrix and a second-level indicator judgment matrix under each first-level indicator are constructed, and a 1-9 scale is used to quantify the relative importance of indicators at the same level; The first-level indicator judgment matrix and the second-level indicator judgment matrix are solved by the eigenvalue method to obtain the first-level initial weight vector and the second-level initial weight vector, and the sum of the second-level initial weights under the same level indicator is 1; The scenario coefficient is determined based on the strength of the correlation between the scenario and the first-level indicator, and the dynamic weight of the second-level indicator is obtained by multiplying the second-level initial weight by the corresponding scenario coefficient; among which, the sum of the second-level dynamic weights under the same-level indicator is equal to the scenario coefficient.

7. The pilot communication capability evaluation method based on multimodal data according to claim 6, characterized in that: The score is calculated based on the weights of each level and the characteristic values ​​of the secondary indicators. The method is as follows: Based on the dynamic weights of the secondary indicators, the scores of the corresponding secondary indicators are weighted and summed to obtain the scores of the corresponding primary indicators; If the characteristic value of a secondary indicator is marked as abnormal during the score calculation, the score of the secondary indicator is calculated with a preset weight lower than 1, and the preset weight in the special scenario is lower than the preset weight in the normal scenario; otherwise, the score of the secondary indicator is the product of its characteristic value and 10; Based on the first-level initial weight vector, the scores of each first-level indicator are weighted and summed to obtain the final score of the pilot's communication ability.

8. A pilot communication capability evaluation system based on multimodal data, according to a pilot communication capability evaluation method based on multimodal data according to any one of claims 1 to 6, characterized in that: The system includes: a data acquisition module configured to acquire four-dimensional data in real time; the data acquisition module includes a NIRS device, a microphone, a physiological sensor, and an operation simulator, and is used to acquire neural layer data, speech layer data, physiological layer data, and operation layer data; The preprocessing module is configured to pre-build the scene-indicator-feature correlation matrix and determine the behavioral baseline and neural layer baseline. After collecting the four-dimensional data, the four-dimensional data is synchronized and aligned in time series through a three-level time-scale synchronization mechanism based on the correlation matrix. The scenario-indicator-feature association matrix is ​​constructed based on historical data and a preset hierarchical evaluation system. The preset hierarchical evaluation system includes primary indicators and secondary indicators. Each primary indicator has multiple secondary indicators. The feature extraction module is configured to extract the feature values ​​of each secondary indicator based on the aligned real-time data and mark abnormal features; The modeling and analysis module is configured to construct a double-hidden-layer Markov model, calculate the matching degree and state transition probability based on the eigenvalues, realize the dynamic mapping between features and states, and then determine the state distribution; a weight determination module configured to determine initial weights and secondary dynamic weights of the primary and secondary indicators based on the state distribution; The score calculation module is configured to calculate the score based on the weights of each level and the characteristic values ​​of the secondary indicators, output the results of the pilot's communication ability assessment, and generate an evaluation report.

Images