Man-machine fusion pilot intention recognition method based on multi-modal data

Abstract

The invention belongs to the technical field of man-machine ring systems, and discloses a multi-modal data-based man-machine fusion pilot intention recognition method, which comprises the following steps of multi-modal data acquisition, deep learning modeling, expert knowledge modeling and man-machine fusion modeling. The problems of quick intention change, high timeliness requirement, multiple intention types, few data samples and the like of pilot intention recognition in the aviation cockpit are solved, an aviation cockpit intelligent system can accurately recognize the pilot operation intention in real time, display and control information can be pushed according to the pilot operation intention, decision-making operation can be assisted, and the pilot operation intention recognition efficiency is improved. And the man-machine system task performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-machine-environment systems, and specifically relates to a method for identifying the pilot's operating intention in an aircraft cockpit by using multimodal flight status, operating instructions and eye movement behavior data, combined with deep learning and expert knowledge to construct an algorithm model. Background Art

[0002] Intent recognition refers to the process of understanding the intentions of other subjects and inferring their intentions through the observed effects or impacts on the environment. The ability to recognize human intentions is the basis for effective social interaction between people. Intelligent agents with autonomous capabilities can have natural intelligent human-machine interactions with humans, and the ability of intelligent agents to recognize human intentions is an indispensable ability. Enabling the aircraft cockpit intelligent system to realize real-time and accurate recognition of the pilot's operating intentions, so that the intelligent system has the ability to push display and control information and assist decision-making operations according to the pilot's intentions, and promote the transformation of the aircraft cockpit from passive operation to active service, is a feasible way to improve the mission performance of the aircraft cockpit human-machine system.

[0003] The basic process of intent recognition is to infer a specific intent based on a set of intents by observing system data. The system needs to have an intent library, which specifies the relationship between intents, between intents and behavioral variables, or between intents and situational variables. Assuming there is a series of intentions (G) and a series of behaviors (A), the goal of the intent recognition model is to connect the intent and the corresponding behavior, including the path from behavior to intent and the path from intention to behavior. Among them, A is observable, and G is what the model needs to infer. The intent recognition model needs to infer the most likely G based on A.

[0004] There are generally three types of algorithms for existing intent recognition models: expert knowledge method, machine learning method and cognitive modeling method. Most of the early intent recognition algorithms are based on the expert knowledge method. When modeling intent recognition, researchers first define a set of rules that are only applicable to special fields, and then these rules cover relevant field knowledge, enabling the system to infer conclusions through deduction. When solving the "one-to-many" problem between behavioral data and intent, researchers proposed two modeling ideas: abductive reasoning and mixed logic probability reasoning. However, the problems that the expert knowledge method is difficult to overcome include: 1) The expert knowledge model is rigid and has poor performance when dealing with intent switching and noisy data; 2) The system usually assumes that the observed subject is rational and behaves according to expert rules, however, humans often act in irrational ways.

[0005] Machine learning is a method of identifying intent in a data-driven way. In the field of intent recognition, machine learning is often used to identify a known set of intents using input information with a given number of dimensions. Various machine learning algorithms have been applied to intent recognition modeling, including dynamic Bayesian networks, hidden Markov models, linear chain conditional random fields, support vector machines, decision trees, k-nearest neighbors, and artificial neural networks. Machine learning has the following disadvantages: 1) Most algorithms are unexplainable, so people cannot be sure how the model reaches a certain conclusion. 2) When the system is complex and the number of variables is large, a large amount of data is required to train the model, and this data is often difficult to obtain.

[0006] Cognitive modeling is a computational modeling of human cognitive processes based on psychological research, which realizes a relatively general model that can simulate human behavior. Existing cognitive architectures usually simulate cognitive functions such as attention, memory, and learning to achieve an overall simulation of human information processing, such as the Adaptive Control of Thought Rational (ACT-R) cognitive architecture. Overall, these studies are in the initial exploratory stage and are still a long way from actual system application.

[0007] Existing intention recognition is mainly studied in relatively simple, small intention set intention recognition scenarios, and lacks a reliable technical path in complex system fields such as aircraft operation. In particular, the present invention mainly solves the following problems of pilot intention recognition in the aircraft cockpit: 1) The cockpit mission scenario changes quickly, the application time window of the intention recognition result is small, and the output of the model needs to be fast and timely; 2) There are many categories of pilot intentions, but the amount of data samples available for modeling is small, and the recognition accuracy is difficult to improve. Summary of the invention

[0008] The purpose of the present invention is to provide a human-machine fusion pilot intention recognition method based on multimodal data in an aircraft cockpit, so as to solve the problems of rapid intention change, high timeliness requirement, many intention types, and few data samples in the aircraft cockpit pilot intention recognition proposed by the above-mentioned background technology, and realize timely and accurate pilot intention recognition.

[0009] The technical solution of the present invention is achieved in this way:

[0010] A method for identifying pilot intention based on human-machine fusion of multimodal data comprises the following steps:

[0011] Construction of a multimodal database based on simulated air combat human factors experiments;

[0012] Offline construction of data-driven deep learning intent recognition model;

[0013] Offline construction of expert knowledge intention path network probability model;

[0014] Offline construction of human-machine fusion intention recognition model.

[0015] As a further solution of the present invention: it also includes the step of online application of the human-machine fusion intention recognition model.

[0016] As a further solution of the present invention: the multimodal database construction specifically includes: original collection and feature extraction of flight status, operation instructions and eye movement behavior data;

[0017] The deep learning modeling specifically includes: building an intention recognition model based on a deep learning algorithm through the multimodal data and the pilot's intention label drive, and outputting the pilot's operation intention;

[0018] The expert knowledge modeling is specifically as follows: based on the mutual conversion relationship of pilot operation intentions by domain experts and combined with experimental statistical results, a path network probability model of pilot operation intentions is constructed;

[0019] The human-machine fusion modeling is specifically to achieve an organic combination of a deep learning model for intention recognition and an expert knowledge model, and finally output the pilot's operational intention.

[0020] As a further solution of the present invention: the multimodal data is collected through a simulated air combat human factors experimental paradigm, and a retrospective interview method is used to manually annotate the pilot's operation intention labels, such as detection flight control, tracking flight control, and non-threatening active disengagement.

[0021] As a further solution of the present invention: the flight situation data includes flight status data and air-to-air situation data, such as speed, acceleration, altitude, inclination angle, pitch angle, number and position of enemy targets, and number and position of enemy missiles;

[0022] The operation command data includes the pilot's HOTAS commands on the throttle lever, flight stick, and pedals, as well as touch commands on the human-machine interface;

[0023] The eye movement behavior data includes the spatiotemporal trajectory of the pilot's gaze point.

[0024] As a further solution of the present invention: the data-driven intention recognition model is modeled using a long-short-time neural network method; wherein, the data feature extraction adopts a sliding window technology, and its window width and sliding step size are determined based on an offline visual analysis of simulated air combat human factors experimental data, and are assumed to be w seconds and s seconds respectively.

[0025] As a further solution of the present invention: the expert knowledge modeling includes the following steps: first, an air combat mission expert analyzes the pilot's operational intention set and the conversion paths between them, and preliminarily constructs a pilot intention path network; and the preliminary model is modified according to the occurrence frequency of the pilot's intention and the frequency of conversion between intentions in the air combat simulation human factors experiment data, and finally a path network probability model of the pilot's operational intention is formed; the model stipulates the rules and probabilities of mutual conversion between different pilot operational intentions.

[0026] As a further solution of the present invention: the human-machine fusion modeling includes the following steps: using a pilot operation intention path network probability model based on expert knowledge and experimental statistics, correcting the output results of the data-driven deep learning intention recognition model, and outputting the final pilot operation intention label.

[0027] As a further solution of the present invention: the correction method is specifically: given the recognition result output by the data-driven deep learning intention recognition model in the current time window (including the probability value of each pilot's operation intention), combined with the pilot's operation intention output by the model in the previous time window, the constraints of the intention conversion are calculated in the pilot's operation intention path network probability model to obtain the pilot's operation intention with the highest probability in the current time window.

[0028] Beneficial effects of the present invention:

[0029] 1. Based on the sliding window width w seconds determined by offline visual analysis of simulated air combat human factors experimental data, the present invention integrates the multimodal data of flight situation, operation instructions and eye movement behavior, so as to identify the current pilot's operation intention and refresh the pilot's operation intention recognition result within each sliding step s seconds, thereby ensuring that the model can meet the timeliness requirements of air combat tasks when applied online, and can make full use of the original data as input to take into account the accuracy of intention recognition, thereby achieving a certain balance between the real-time performance of air combat tasks and the accuracy of intention recognition.

[0030] 2. The present invention adopts a human-machine fusion modeling approach that combines deep learning models and expert knowledge models to overcome the problems of low recognition accuracy caused by large intent sets and small data samples in existing technologies.

[0031] 3. The present invention enables the aircraft cockpit to provide active services such as display and control information push and auxiliary decision-making operations according to the pilot's intention, thereby improving the mission performance of the human-machine system.

[0032] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1It is a schematic diagram of the pilot intention recognition method based on multi-modal data of human-machine fusion of the present invention;

[0034] Figure 2 This is a schematic diagram of the pilot's operation intention set of the present invention;

[0035] Figure 3 This is a schematic diagram of dividing the focus point interest area of ​​the human-machine interface of the present invention;

[0036] Figure 4 This is a schematic diagram of the pilot's operation intention path network of the present invention;

[0037] Figure 5 This is a schematic diagram of the pilot operation intention path network probability model of the present invention. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] The present invention provides a human-machine fusion pilot intention recognition method based on multimodal data, which includes four parts: multimodal data acquisition, deep learning modeling, expert knowledge modeling and human-machine fusion modeling.

[0040] The multimodal data collection includes the original collection and feature extraction of flight situation, operation instructions and eye movement behavior data. Among them, the flight situation data includes flight status data and air-to-air situation data, such as speed, acceleration, altitude, inclination angle, pitch angle, number and position of enemy targets, number and position of enemy missiles, etc. The operation instruction data includes the pilot's HOTAS instructions on the throttle lever, flight stick, pedals, and touch instructions on the human-computer interface. Eye movement behavior includes the spatiotemporal trajectory of the pilot's gaze point.

[0041] The deep learning modeling is driven by the above multimodal data and the pilot's intention label, and an intention recognition model based on a deep learning algorithm is constructed to output the pilot's operation intention. Among them, the above multimodal data is collected through a simulated air combat human factors experiment paradigm, and the pilot's operation intention label is manually annotated by a retrospective interview method, such as detection flight control, tracking flight control, active disengagement without threat, etc. The data-driven intention recognition model is modeled using a long-short time neural network method. Among them, the data feature extraction adopts a sliding window technology, and its window width and sliding step length are determined based on the offline visual analysis of the simulated air combat human factors experiment data, and are assumed to be w seconds and s seconds respectively.

[0042] The expert knowledge modeling is firstly conducted by the air combat mission experts to analyze the pilot's operation intention set and the conversion paths between them, and initially construct the pilot's intention path network. The initial model is modified according to the occurrence frequency of the pilot's intention and the frequency of conversion between intentions in the air combat simulation human factors experiment data, and finally a path network probability model of the pilot's operation intention is formed. The model stipulates the rules and probabilities of conversion between different pilot operation intentions.

[0043] The human-machine fusion modeling adopts the pilot operation intention path network probability model based on expert knowledge and experimental statistics to correct the output results of the data-driven deep learning intention recognition model and output the final pilot operation intention label. The specific correction method is as follows: given the recognition results output by the data-driven deep learning intention recognition model in the current time window (including the probability values ​​of each pilot operation intention), combined with the pilot operation intention output by the model in the previous time window, the constraint conditions of the intention conversion in the pilot operation intention path network probability model are calculated to obtain the pilot operation intention with the maximum probability in the current time window.

[0044] See attached Figure 1-5 Based on experimental data and expert knowledge, the present invention designs and implements simulated air combat human factors experiments, collects multimodal data related to pilots' operational intentions, constructs a data-driven deep learning intention recognition model and an expert knowledge intention path network probability model, and combines them to form a human-machine fusion intention recognition model to achieve real-time and accurate recognition of pilots' operational intentions. Figure 1 As shown, in the online application stage, the cockpit multimodal data is first collected, processed and features are calculated, the human-machine fusion intention recognition model is called to recognize the pilot's operation intention, and finally the current pilot's operation intention is output.

[0045] To realize such technology, the following five main steps are involved: construction of a multimodal database based on simulated air combat human factors experiments; offline construction of a data-driven deep learning intention recognition model; offline construction of an expert knowledge intention path network probability model; offline construction of a human-machine fusion intention recognition model; and online application of a human-machine fusion intention recognition model.

[0046] Step 1: Construction of a multimodal database based on simulated air combat human factors experiments

[0047] Experimental preparation. The simulated air combat human factors experiment recruits at least 10 subjects, all of whom need to have knowledge of air combat games and reach a certain level of performance. The experiment is conducted in a simulated cockpit, simulating the human-machine interface, operating devices, exterior environment, and air combat missions of a real aircraft cockpit. The simulated cockpit needs to have the following functions:

[0048] 1) Eye tracking function, used to monitor the subject's eye movement data on the human-computer interface;

[0049] 2) Screen recording function, which is used to assist the interview after the experiment, so that the subjects can watch the entire task process and annotate the intentions together with the experimenter;

[0050] 3) Viewpoint-screen recording integration function: when replaying the task process video, the subject's viewpoint coordinates can be synchronously presented on the replay video to assist the subject in reviewing the task process and accurately mark the time when the pilot's operation intention occurred;

[0051] 4) Flight situation and operation instruction data collection function. Each subject completes at least 4 air combat missions to ensure that the database covers typical mission scenarios.

[0052] Experimental process. The implementation process of the simulated air combat human factors experiment is as follows:

[0053] 1) Instructions: Introduce the overall experimental process and experimental tasks to the subjects;

[0054] 2) Eye tracker configuration: Ask the subject to sit in the simulated cockpit and adjust the seat to a comfortable state, and instruct the subject to cooperate with the eye tracker calibration;

[0055] 3) Screen recording settings: start screen recording;

[0056] 4) Air combat simulation software configuration: configure mission scenarios and start flight situation and operation instruction data collection functions;

[0057] 5) Practice trials: The subjects complete a "warm-up" task scenario to familiarize them with the air combat task and experimental environment. The practice data is not used as the model training data set.

[0058] 6) First trial: The subject completed an air combat mission independently;

[0059] 7) Post-event retrospective interview: Play a video recording of the task process containing viewpoint information, and ask the subjects to describe their operational behaviors and intentions at each moment as detailed as possible.

[0060] 8) Subsequent experimental trials: Repeat steps 6)-7) above until the end of the experiment.

[0061] Intent labeling. The intention labeling of the task process includes determining the set of intentions that appear in the task and the time and duration of the intentions in the task process. The intention set is mainly obtained by interviewing the subjects participating in the experiment and analyzing and sorting them out by experts. The pilot operation intention set sorted out based on the air combat task is as follows: Figure 2 According to the intention set, the experimenter needs to label the intention of the task process according to the results of the retrospective interview.

[0062] Step 2: Offline construction of data-driven deep learning intent recognition model

[0063] The sliding window technique is used to process the multimodal data collected in the above experiment to generate training data sets and test data sets for machine learning. The sliding window technique is a computing technique that aims to reduce the use of nested loops and replace them with a single loop, thereby reducing time complexity. That is, this technique calculates the results of the next step by reusing the data of the previous step. The steps for using the sliding window technique are as follows:

[0064] 1) Based on the experimental data in step 1, perform offline visual analysis, refer to the intention label results and the visualization results of flight status, operation execution and eye movement behavior data in different time intervals, and determine the width w seconds and step length s seconds of the sliding time window;

[0065] 2) Calculate the result of the first window;

[0066] 3) Maintain a pointer to the window's starting position;

[0067] 4) Run a loop with a step size of s seconds, sliding the window one step at a time, sliding the pointer one step at a time, and keeping track of the results of each window.

[0068] Intent classification data generation. Within the w second time window, the distribution of intent has the following three cases:

[0069] 1) The intention of the previous time period to continue to this time period;

[0070] 2) the intention to occur within this time period and end within this time period;

[0071] 3) Intent that occurs within this time period and lasts until the next time period. In the first case, the intention start time is set to the time window start time, and in the third case, the intention end time can be set to the time window end time.

[0072] Generally, a time window of w seconds contains at most two intentions. The specific processing methods are as follows:

[0073] 1) When the time window contains only one intent, use this intent to represent the time window;

[0074] 2) In a time window, when the time proportion of one intent is greater than that of another, the intent with the larger proportion is used to represent the time window.

[0075] Model training based on long short-term memory network (LSTM). The overall neural network model includes LSTM layer, Dropout layer and Dense layer. In the LSTM layer, the feature matrix X obtained by preprocessing feature extraction is (f1, f2, ..., f len ) as input, where len is the sliding window length, f i is the trajectory point P i The corresponding trajectory features. The label Y is the user operation intention generated in the marking stage. Each sliding window sequence is input in the form of (X, Y) for training. The output of the LSTM layer H = (h1,h2,…,h len ), where len is the sliding window length and h is the output of the LSTM network. In the Dropout layer, the network randomly loses a portion of the hidden layer output to avoid overfitting of the model. In the final Dense layer, the network uses the processed output H = (h'1,h'2,…,h' len ) is used as input and the softmax function is used to output the predicted user's next operation intention.

[0076] Step 3: Offline construction of expert knowledge intention path network probability model

[0077] Compared with ordinary situations, in highly special situations such as cockpit combat, human behavior will be more affected by the situation. Therefore, it is reasonable to summarize expert knowledge and construct the intention path network probability model based on the cockpit combat situation experiment.

[0078] Construct the pilot operation intention architecture based on the mission process. Domain experts need to determine the mission process in the application scenario of the intention recognition model. In the assumed air combat mission application scenario, the main mission process includes several stages such as detection, combat, defense and reentry. In the mission stage, domain experts need to determine the subordinate relationship between the pilot's operation intention and the mission stage, as well as the path relationship of the pilot's operation intention in a specific stage. This analysis can obtain a preliminary pilot operation intention architecture.

[0079] Intention conversion rules. Based on the pilot's operational intention architecture, domain experts need to determine the rules for conversion between intentions. These rules may include:

[0080] 1) Objective conditions, mainly referring to the situational conditions that lead to the conversion of intentions;

[0081] 2) Subjective conditions, mainly referring to the operation instructions and eye movement behaviors that lead to or refer to the schematic diagram conversion. After adding the intention conversion rules to the pilot operation intention framework, a preliminary pilot operation intention path network is formed, such as Figure 4 shown.

[0082] Correct the path network based on experimental data. Using the database formed in "Step 1: Multimodal database construction based on simulated air combat human factors experiment", by analyzing the conditional probability of conversion between intentions, the conversion relationship between actual pilot operation intentions can be obtained. Due to the limited sample size of experimental data, the intention conversion relationship occurring in the data cannot represent all possibilities. Therefore, the data is only used to verify and correct the intention path network.

[0083] First, all labeled intents are treated as nodes of the graph and numbered 1, 2, …, n.

[0084] Secondly, if there is a potential intention conversion between node i (i = 1, 2, ..., n) and node j (j = 1, 2, ..., n, and j ≠ i), a directed arc is established between nodes i and j, and T ij represents the probability of switching from intention i to intention j.

[0085] Again, based on the labeled experimental data, the frequency of switching from intent i to intent j is counted. Based on the statistical results, the initial estimate of T is obtained after normalization.

[0086] Finally, with the help of the initial estimation of T and the analysis of the expert interview results, the element values ​​of T are determined and normalized. T is the matrix representation of the prior model, forming the pilot operation intention path network probability model, such as Figure 5 shown.

[0087] Step 4: Offline construction of human-machine fusion intention recognition model

[0088] The human-machine fusion intention recognition model mainly uses the expert knowledge intention path network probability model to perform a priori correction on the primary recognition results of user intentions in the data-driven deep learning intention recognition model. The specific correction process method is as follows:

[0089] 1) Sort the output probabilities corresponding to the intents in the primary recognition results of the user intent by the data-driven deep learning intent recognition model in descending order;

[0090] 2) If the difference between the output probabilities corresponding to the first two intentions in the sequence is greater than a given threshold, no prior correction is performed; otherwise, a prior correction is performed;

[0091] 3) Assume that the intention number before this intention prediction is i, and the intention sequence after this intention prediction and sorting by output probability is j1, j2, …, j m , where m = n-1, j k ≠i, k = 1, 2, ..., m. The output probability corresponding to this intention sequence is p j1 ,p j2 ,…,p jm , where pj1 ≥p j2 ≥…p jm , Calculate T i,jk *p jk , (k=1,2,…,m), and all T i,jk *p jk The intention corresponding to the maximum value is the intention result after prior correction.

[0092] Step 5: Online application of human-machine fusion intention recognition model

[0093] When the intention recognition model is applied online, the aircraft cockpit needs to provide the model with real-time flight status, operating instructions and eye movement behavior data to match the data when the model is built offline.

[0094] During the task, the flight status, operation instructions and eye movement behavior data are processed in the same way as the offline stage in units of w seconds and intervals of s seconds to extract features. First, the offline data-driven deep learning intention recognition model is called to perform primary recognition of user intentions, and then the expert knowledge intention path network probability model is called to perform a priori correction on the primary recognition results, and the verified intentions are used as the output results of the human-machine fusion intention recognition model.

[0095] So far, the purpose of the present invention has been achieved.

[0096] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for pilot intention recognition based on multimodal data of human-machine fusion, characterized in that: The steps include: Construction of a multimodal database based on simulated air combat human factors experiments; Offline construction of data-driven deep learning intent recognition model; Offline construction of expert knowledge intention path network probability model; Offline construction of human-machine fusion intention recognition model.

2. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 1 is characterized in that: It also includes the steps of: online application of human-machine fusion intention recognition model.

3. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 1 or 2, characterized in that: The multimodal database construction specifically includes: original collection and feature extraction of flight status, operation instructions and eye movement behavior data; The deep learning modeling specifically includes: building an intention recognition model based on a deep learning algorithm through the multimodal data and the pilot's intention label drive, and outputting the pilot's operation intention; The expert knowledge modeling is specifically as follows: based on the mutual conversion relationship of pilot operation intentions by domain experts and combined with experimental statistical results, a path network probability model of pilot operation intentions is constructed; The human-machine fusion modeling is specifically to achieve an organic combination of a deep learning model for intention recognition and an expert knowledge model, and finally output the pilot's operational intention.

4. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 3 is characterized in that: The multimodal data are collected through a simulated air combat human factors experimental paradigm, and retrospective interviews are used to manually annotate pilot operation intention labels, such as detection flight control, tracking flight control, and non-threatening active disengagement.

5. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 4 is characterized in that: The flight situation data includes flight status data and air-to-air situation data, such as speed, acceleration, altitude, inclination angle, pitch angle, number and position of enemy targets, and number and position of enemy missiles; The operation command data includes the pilot's HOTAS commands on the throttle lever, flight stick, and pedals, as well as touch commands on the human-machine interface; The eye movement behavior data includes the spatiotemporal trajectory of the pilot's gaze point.

6. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 5 is characterized in that: The data-driven intention recognition model is modeled using a long-short time neural network method; wherein, the data feature extraction adopts a sliding window technology, and its window width and sliding step size are determined based on an offline visual analysis of simulated air combat human factors experimental data, and are assumed to be w seconds and s seconds respectively.

7. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 6 is characterized in that: The expert knowledge modeling includes the following steps: first, an air combat mission expert analyzes the pilot's operation intention set and the conversion paths between them, and preliminarily constructs a pilot intention path network; and the preliminary model is modified according to the occurrence frequency of the pilot's intention and the frequency of conversion between intentions in the air combat simulation human factors experiment data, and finally a path network probability model of the pilot's operation intention is formed; the model stipulates the rules and probabilities of mutual conversion between different pilot operation intentions.

8. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 7 is characterized in that: The human-machine fusion modeling includes the following steps: using a pilot operation intention path network probability model based on expert knowledge and experimental statistics, correcting the output result of the data-driven deep learning intention recognition model, and outputting the final pilot operation intention label.

9. The method for identifying pilot intention based on multimodal data of human-machine fusion according to claim 8 is characterized in that: The correction method is specifically as follows: given the recognition result output by the data-driven deep learning intention recognition model in the current time window, combined with the pilot's operation intention output by the model in the previous time window, the constraint conditions of the intention conversion in the pilot's operation intention path network probability model are calculated to obtain the pilot's operation intention with the highest probability in the current time window.

Images