An aircraft cockpit multi-channel human-machine intelligent interaction system and method

Abstract

The application provides an airplane cockpit multi-channel man-machine intelligent interaction system and method, and the system comprises a multi-channel data access module, a multi-channel interaction instruction matching module, a multi-channel relay communication function module, a multi-channel priority sorting module, a multi-channel interaction information adjusting module and a cockpit display control interface module. The natural cooperation and efficiency optimization of multi-channel interaction modes such as touch, voice, eye movement, gesture and electroencephalogram in the airplane cockpit interaction environment are realized, a scientific scheme for organizing airplane cockpit multi-channel man-machine intelligent interaction information is provided, an important basis for establishing the airplane cockpit multi-channel man-machine intelligent interaction system architecture is provided, and the design and development of the airplane cockpit multi-channel man-machine intelligent interaction system can be effectively supported.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human-computer interaction of complex systems, and relates to a multi-channel human-computer intelligent interaction system and method for an aircraft cockpit. BACKGROUND

[0002] Multi-channel human-computer intelligent interaction is to make the human-computer interaction process knowledge-based, including knowledge acquisition, knowledge representation, knowledge reasoning and other links, to excavate and extract human thinking patterns and represent them in a specific form, so as to make them computer-operable objects, so that the complex system has certain human intelligence. The development of the human-computer intelligent interaction system is reflected in information input / output devices, operation feedback methods, interface visualization methods, human-computer collaboration degrees and the like. With the further development of artificial intelligence technology, touch interaction, voice interaction, eye movement interaction, body sensation interaction, mixed reality interaction, brain-computer interaction, human and intelligent agent interaction and other technologies are becoming the most novel design features of the human-computer interaction system.

[0003] At present, in the field of aviation, the related researches on touch interaction technology and voice interaction technology are relatively mature, and have been successfully applied to the cockpits of some series of aircrafts. The cockpit human-computer system comprehensively integrates flight display, flight planning, airborne system control and other control functions, effectively reduces the workload of pilots by using direct voice input and touch control, and the multi-channel human-computer intelligent interaction will become the main human-computer interaction mode of the future aircraft cockpit. The aircraft cockpit integrating the multi-channel human-computer intelligent interaction technology has advantages such as diversification of interaction channels, dynamicization of function allocation, and collaboration of human-computer decision-making, but there are also new problems that have not been solved, mainly reflected in that the system architecture of the multi-channel human-computer intelligent interaction system for the aircraft cockpit has not been established, no scientific solution has been proposed for how to organize the multi-channel human-computer intelligent interaction information, and there is a lack of effective methods to support the design of the multi-channel human-computer intelligent interaction system for the aircraft cockpit. SUMMARY

[0004] In view of the lack of effective technical means to support the design of the multi-channel human-computer intelligent interaction system for the aircraft cockpit, the application provides a multi-channel human-computer intelligent interaction system and method for an aircraft cockpit, which can reasonably adjust the key node information flow load of the multi-channel human-computer intelligent interaction, and provide effective technical support for the design and optimization of the multi-channel human-computer intelligent interaction system for the aircraft cockpit.

[0005] The first aspect of the application provides a multi-channel human-computer intelligent interaction system for an aircraft cockpit, comprising: a multi-channel data access module, a multi-channel interaction instruction matching module, a multi-channel relay communication function module, a multi-channel priority sorting module, a multi-channel interaction information adjustment module, and a cockpit display control interface module.

[0006] The multi-channel data access module takes physiological and behavioral data of the pilot as input, extracts basic characteristic values of the physiological and behavioral data through time-frequency analysis and pattern matching, the basic characteristic values including touch coordinates, tone and speed, gaze duration, body posture, and brain region activation, further extracts fusion characteristic values from the basic characteristic values, the fusion characteristic values including target icons, hot word semantics, target regions, gesture semantics, and cognitive load, and transmits the fusion characteristic values to a multi-channel interaction instruction matching module.

[0007] The multi-channel interaction instruction matching module matches control semantics of the fusion characteristic values according to a mapping relationship between the fusion characteristic values and the control semantics, and transmits all successfully matched control semantics to a multi-channel relay communication function module; the control semantics include switch type continuous adjustment controlled object state type control, flight control type continuous adjustment controlled object state type control, and discrete adjustment controlled object state type control.

[0008] The multi-channel relay communication function module classifies and groups the received control semantics according to trigger conditions, stores the control semantics that first arrive at the module according to the order of arrival of the control semantics at the module, in combination with the current task stage or system state of the aircraft, and generates a control instruction when all trigger conditions for triggering a control operation are met, and transmits the control instruction to a multi-channel priority sorting module.

[0009] The multi-channel priority sorting module evaluates and judges the importance and conflict possibility of the multi-channel control instructions in combination with the current task stage or system state of the aircraft, sorts the control instructions according to the importance to generate a feasible control sequence for the control instructions without conflicts, and sorts only the control instruction with the highest importance to generate a feasible control sequence for the control instructions with conflicts, and transmits the generated feasible control sequence to a multi-channel interaction information adjustment module.

[0010] The multi-channel interaction information adjustment module analyzes different feasible control sequences, takes a human-machine interaction object as a node, generates a directed edge according to a control or controlled relationship, takes a relationship complexity between different nodes as a weight of the directed edge, constructs an aircraft cockpit multi-channel human-machine intelligent interaction information flow network, compares information flow loads of pilot nodes in different information flow networks, selects a feasible control sequence with the lowest information flow intensity of the pilot nodes as an optimal control scheme, and transmits the optimal control scheme to a cockpit display control interface module.

[0011] The cockpit display control interface module provides all display control interfaces of flight control functions, flight management functions, system monitoring functions, system control functions, and communication functions in the aircraft cockpit, transmits control signals to each system interface according to the optimal control scheme generated by the multi-channel interaction information adjustment module in an optimal sequence, and executes the control signals.

[0012] Optionally, the switch type continuous adjustment controlled object state type control of the multi-channel interactive instruction matching module refers to the combination of tactile serial operation and tactile activation for continuous knob operation, including the interactive scene of setting the permitted height on the flight control panel;

[0013] The flight control type continuous adjustment controlled object state type control of the multi-channel interactive instruction matching module refers to the continuous handle operation by using tactile activation, including the interactive scene of disconnecting the automatic thrust in the automatic landing stage and pushing the thrust handle to the idle speed position.

[0014] The discrete adjustment controlled object state type control of the multi-channel interactive instruction matching module refers to the knob operation, discrete knob operation and discrete handle operation by using tactile activation, including the interactive scene of switching the navigation mode on the control panel, turning on the landing light switch and adjusting the descent rate by using the speed brake handle at different heights / speeds.

[0015] Optionally, the trigger condition of the multi-channel relay communication function module for triggering the control operation includes flight parameters and state information, and provides logical operator connection, and the control instruction is generated when the trigger condition judgment statement is true; the flight parameters include flight height, airspeed, heading and attitude angle; the state information includes landing gear position, flap position and switch position.

[0016] The logical operation includes logical and, logical or and logical not.

[0017] Optionally, the multi-channel priority sorting module sorts the importance of the multi-channel control instructions according to the system level safety evaluation results of the aircraft development stage, including functional hazard evaluation, preliminary system safety evaluation and common cause analysis, based on the potential possibility of non-expected functional failure / fault in the aircraft system or non-expected functional failure / fault between related systems, and combined with different consequence severity of disaster, danger, greater impact, smaller impact and no safety impact.

[0018] Optionally, the pilot node information flow intensity adjustment strategy of the multi-channel interactive information adjustment module is to use tactile serial operation, tactile activation and non-contact interaction modes of voice, eye movement and gesture from the aspects of information simplification and intelligence.

[0019] Optionally, the pilot physiological and behavioral data include touch, voice, eye movement, gesture and electroencephalogram.

[0020] Optionally, the functional architecture of the multi-channel data access module includes a collection terminal, a data layer, a feature layer, a fusion layer and an external interface.

[0021] At the terminal layer, the terminal for monitoring the physiological state of the pilot includes an eye movement camera, an electroencephalogram amplifier, an electrocardiogram sensor, and an electromyography sensor, and the terminal for monitoring the behavior state of the pilot includes a voice recorder, a gesture catcher, and a touch recorder;

[0022] At the data acquisition layer, the original data collected by the terminal are processed by the controller, and the structured storage, timeline synchronization, multi-source physiological data, multi-modal human-computer interaction action data, and instruction data are sent out;

[0023] At the feature extraction layer, the corresponding physiological state features, behavior state features, and corresponding instruction semantics are extracted;

[0024] At the feature fusion layer, the physiological state feature fusion and the behavior state feature fusion are realized, and the further fusion of the two types of features is realized in combination with the context of the system state and the task state;

[0025] At the external interface layer, in combination with the task demand, the function architecture of the module provides an output interface of target icons, hot word semantics, target areas, gesture semantics, and cognitive load, and simultaneously accesses a multi-channel interaction instruction matching module.

[0026] The second aspect of the present application provides a multi-channel human-computer intelligent interaction method for an aircraft cockpit, which is executed by using the system according to any one of the first aspect.

[0027] The present application provides a multi-channel human-computer intelligent interaction system and method for an aircraft cockpit, which realizes the natural cooperation and efficiency optimization of multi-channel interaction modes such as touch, voice, eye movement, gesture, and electroencephalogram in the interaction environment of the aircraft cockpit, provides a scientific solution for organizing multi-channel human-computer intelligent interaction information for the aircraft cockpit, provides an important basis for establishing the system architecture of the multi-channel human-computer intelligent interaction system for the aircraft cockpit, and effectively supports the design and development of the multi-channel human-computer intelligent interaction system for the aircraft cockpit. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 Fig. 1 is a structural diagram of an intelligent human-computer system for an aircraft cockpit;

[0029] Figure 2 Fig. 2 is a multi-channel human-computer intelligent interaction loop diagram for an aircraft cockpit;

[0030] Figure 3 Fig. 3 is a function architecture of a multi-channel data access module. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0032] The features and illustrative embodiments of various aspects of the present application will be described below in detail. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one of ordinary skill in the art that the present application can be practiced without some of these specific details. The following description of embodiments is merely exemplary in nature and is intended to provide a better description of the application. The present application is not limited to any particular setting or method as set forth below, but covers any modifications, substitutions, and changes in structure, methods, devices, etc. without departing from the spirit of the present application. In the drawings and the following description, well-known structures and techniques are not shown to avoid unnecessary obscuring of the present application.

[0033] It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict, and each embodiment can be mutually referenced and quoted. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] The present application will be described in further detail below in combination with the embodiments and the accompanying drawings, but the implementation of the present application is not limited thereto.

[0035] The present application provides an aircraft cockpit multi-channel human-computer intelligent interaction method, comprising the following steps:

[0036] (1) An aircraft cockpit multi-channel human-computer intelligent interaction system is adopted, which comprises a multi-channel data access module, a multi-channel interaction instruction matching module, a multi-channel relay communication function module, a multi-channel priority sorting module, a multi-channel interaction information adjustment module, and a cockpit display control interface module;

[0037] (2) Pilot physiological data and behavior data such as touch, voice, eye movement, gesture, and brain electricity generated in the human-computer interaction process are collected through sensors and terminal devices, and the original data are transmitted to the multi-channel data access module, from which basic characteristic values of different data are extracted and fused, and the fused characteristic values are transmitted to the multi-channel interaction instruction matching module;

[0038] (3) The multi-channel interaction instruction matching module establishes a mapping relationship from the fused characteristic values to the control semantics, and transmits all the matched control semantics to the multi-channel relay communication function module;

[0039] (4) The received control semantics are combined and classified according to conditions by the multi-channel relay communication function module, and when all the conditions triggering the control operation are met, a control instruction is generated and transmitted to the multi-channel priority sorting module;

[0040] (5) The importance and conflict possibility of the multi-channel control instruction are evaluated and judged by the multi-channel priority sorting module, a feasible control sequence is generated according to the importance, and the generated feasible control sequence is transmitted to the multi-channel interactive information adjustment module;

[0041] (6) The multi-channel interactive information adjustment module analyzes different feasible control sequences, constructs a multi-channel human-machine intelligent interactive information flow network of the aircraft cockpit, compares the information flow load of the pilot node in different information flow networks based on the relationship complexity between different nodes, selects the feasible control sequence with the lowest information flow intensity of the pilot node as the optimal control scheme, and transmits the optimal control scheme to the cockpit display control interface module;

[0042] (7) The cockpit display control interface module transmits the control signal to each airborne system interface to perform flight control, flight management, system monitoring, system control, communication and other functions, and realizes the multi-channel human-machine intelligent interactive closed-loop control of the aircraft cockpit.

[0043] As shown in Figure 1 , in the intelligent human-machine system structure of the aircraft cockpit, the multi-interaction channels such as touch, voice, eye movement, gesture, and electroencephalogram are located at the interface position between the pilot and the airborne system, and act as intermediaries between the receptors, effectors of the pilot and the airborne system. The multi-channel human-machine intelligent interaction system of the aircraft cockpit is the general term of the cockpit sending device, receiving device and multi-channel human-machine interface.

[0044] As shown in Figure 2 , the multi-channel human-machine intelligent interaction loop of the aircraft cockpit involves the pilot, the airborne system, and the multi-channel human-machine intelligent interaction system. The multi-channel human-machine intelligent interaction system of the aircraft cockpit includes a multi-channel data access module, a multi-channel interactive instruction matching module, a multi-channel relay communication function module, a multi-channel priority sorting module, a multi-channel interactive information adjustment module, and a cockpit display control interface module.

[0045] The multi-channel data access module takes the physiological and behavioral data of pilots such as touch, voice, eye movement, gesture, and brain waves as input, extracts basic characteristic values of different data through time-frequency analysis and pattern matching, including touch coordinates, tone and speed, gaze duration, body posture, and brain region activation, further extracts high-level fusion characteristic values from the basic characteristic values, including target icons, hot word semantics, target regions, gesture semantics, and cognitive load, and transmits the fusion characteristic values to the multi-channel interactive instruction matching module.

[0046] The multi-channel data access module has a functional architecture as shown in Figure 3 The functional architecture of the multi-channel data access module includes a collection terminal, a data layer, a feature layer, a fusion layer, and an external interface.

[0047] In the collection terminal layer, the terminals for monitoring the physiological state of the pilot include an eye movement camera, a brain wave amplifier, an electrocardiogram sensor, and an electromyography sensor, and the terminals for monitoring the behavioral state of the pilot include a voice recorder, a gesture capture device, and a touch recorder.

[0048] In the data collection layer, the raw data collected by the terminals are processed by the controller to send out structured storage, timeline-synchronized multi-source physiological data, multi-modal human-computer interaction action data, and instruction data.

[0049] In the feature extraction layer, the corresponding physiological state features, behavioral state features, and instruction semantics are extracted.

[0050] In the feature fusion layer, the physiological state feature fusion and the behavioral state feature fusion are realized, and the fusion of the two types of features at a higher level is realized in combination with the situational context of the system state and the task state.

[0051] In the external interface layer, in combination with the task requirements, the functional architecture of the module provides output interfaces for features such as target icons, hot word semantics, target regions, gesture semantics, and cognitive load, and simultaneously accesses the multi-channel interactive instruction matching module.

[0052] The multi-channel interactive instruction matching module establishes a mapping relationship between the fusion characteristic values and the control semantics, including continuous adjustment (switch type) controlled object state type control, continuous adjustment (flight control type) controlled object state type control, and discrete adjustment controlled object state type control, and transmits all the matched control semantics to the multi-channel relay communication function module.

[0053] The continuous adjustment (switch type) controlled object state type control includes a single control mode of continuous adjustment of haptics, a combination control mode of haptics serial operation and haptics activation, and a continuous knob operation mode, and is suitable for interactive scenarios such as setting the permitted height on the flight control panel.

[0054] The continuous adjustment (flight control type) controlled object state type control includes a haptic continuous adjustment control mode, a haptic activation control mode and a continuous handle operation mode, and is suitable for interactive scenes such as automatic thrust disconnection, thrust handle pushing to idle position and the like in the automatic landing stage.

[0055] The discrete adjustment controlled object state type control includes a haptic activation control mode, a haptic discrete adjustment control mode, a knob operation mode, a discrete rotary knob operation mode, a discrete handle operation mode and the like, and is suitable for interactive scenes such as electronic flight instrument system control panel navigation mode switching in the descent preparation stage, landing light switch in the descent stage, and the like.

[0056] The multi-channel relay communication function module combines and classifies the received control semantics according to conditions, stores the control semantics that first arrive at the module according to the order of arrival of the control semantics at the module in combination with the current task stage or system state of the aircraft, and generates a control instruction when all conditions for triggering the control operation are met and transmits the control instruction to the multi-channel priority sorting module.

[0057] The multi-channel priority sorting module evaluates and judges the importance and conflict possibility of the multi-channel control instructions in combination with the current task stage or system state of the aircraft, generates a feasible control sequence according to the importance for the control instructions without conflict, and generates a feasible control sequence according to the importance of the control instruction with the highest importance for the control instructions with conflict, and transmits the generated feasible control sequence to the multi-channel interactive information adjustment module.

[0058] The multi-channel interactive information adjustment module analyzes different feasible control sequences, takes human-machine interaction objects as nodes, generates directed edges according to the control or controlled relationship, takes the relationship complexity between different nodes as the weight of the directed edges, constructs an aircraft cockpit multi-channel human-machine intelligent interactive information flow network, compares the information flow load of the pilot node in different information flow networks, selects the feasible control sequence with the lowest information flow intensity of the pilot node as the optimal control scheme, and transmits the optimal control scheme to the cockpit display control interface module.

[0059] The cockpit display control interface module provides all display control interfaces of flight control functions, flight management functions, system monitoring functions, system control functions, communication functions and the like in the aircraft cockpit, transmits control signals to each system interface according to the optimal control scheme generated by the multi-channel interactive information adjustment module in the optimal sequence, and executes the control signals.

[0060] A cockpit multi-channel human-machine intelligent interaction method is as follows:

[0061] 1. Collecting the physiological data and behavior data of pilots such as touch, voice, eye movement, gesture, and brain electricity generated in the process of human-computer interaction through sensors and terminal devices, transmitting the original data to a multi-channel data access module, extracting basic characteristic values of different data from the module and fusing them, and transmitting the fused characteristic values to a multi-channel interaction instruction matching module;

[0062] 2. Establishing a mapping relationship between the fused characteristic values and control semantics through the multi-channel interaction instruction matching module, and transmitting all matched control semantics to a multi-channel relay communication function module;

[0063] 3. The multi-channel relay communication function module combines and classifies the received control semantics according to conditions, generates a control instruction when all conditions for triggering control operations are met, and transmits the control instruction to a multi-channel priority sorting module;

[0064] 4. The multi-channel priority sorting module evaluates and judges the importance and conflict possibility of the multi-channel control instruction, sorts the importance to generate a feasible control sequence, and transmits the generated feasible control sequence to a multi-channel interaction information adjustment module;

[0065] 5. The multi-channel interaction information adjustment module analyzes different feasible control sequences, constructs a multi-channel human-machine intelligent interaction information flow network of the aircraft cockpit, compares the information flow load of the pilot node in different information flow networks based on the relationship complexity between different nodes, selects the feasible control sequence with the lowest information flow intensity of the pilot node as the optimal control scheme, and transmits the optimal control scheme to a cockpit display control interface module;

[0066] 6. The cockpit display control interface module transmits the control signal to each airborne system interface to perform flight control, flight management, system monitoring, system control, communication, and other functions, and realizes multi-channel human-machine intelligent interaction closed-loop control of the aircraft cockpit.

[0067] The above is only a further embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and concept of the present application within the scope disclosed by the present application, which belongs to the protection scope of the present application.

Claims

1. An aircraft cockpit multi-channel human-machine intelligent interaction system, characterized in that, Comprise: Multi-channel data access module, multi-channel interactive instruction matching module, multi-channel relay communication function module, multi-channel priority sorting module, multi-channel interactive information adjustment module, cockpit display control interface module; The multi-channel data access module takes the pilot physiological and behavioral data as input, extracts basic characteristic values of physiological and behavioral data through time-frequency analysis and pattern matching means, the basic characteristic values include touch coordinates, tone and speed, gaze duration, body posture, brain activation, further extracts fusion characteristic values from the basic characteristic values, the fusion characteristic values include target icon, hot word semantics, target area, gesture semantics, cognitive load, and the fusion characteristic values are transmitted to the multi-channel interactive instruction matching module; The multi-channel interactive instruction matching module matches the control semantics of the fusion characteristic values according to the mapping relationship from the fusion characteristic values to the control semantics, and transmits all matched control semantics to the multi-channel relay communication function module; the control semantics include: switch type continuous adjustment controlled object state type control, flight control type continuous adjustment controlled object state type control, discrete adjustment controlled object state type control The multi-channel relay communication function module combines and classifies the received control semantics according to the trigger conditions, according to the order of the control semantics reaching the module, combines the current task stage or system state of the aircraft, stores the control semantics reaching the module first, and generates a control instruction when all trigger conditions for triggering control operation are met, and transmits the control instruction to the multi-channel priority sorting module; The multi-channel priority sorting module evaluates and judges the importance and conflict possibility of multi-channel control instructions according to the current task stage or system state of the aircraft, sorts the control instructions according to the importance for generating a feasible control sequence, and only retains the control instruction with the highest importance for generating a feasible control sequence, and transmits the generated feasible control sequence to the multi-channel interactive information adjustment module; The multi-channel interactive information adjustment module analyzes different feasible control sequences, takes human-computer interaction objects as nodes, generates directed edges according to the control or controlled relationship, constructs an aircraft cockpit multi-channel human-computer intelligent interaction information flow network by taking the relationship complexity between different nodes as the weight of the directed edge, compares the information flow load of the pilot node in different information flow networks, selects the feasible control sequence with the lowest pilot node information flow intensity as the optimal control scheme, and transmits it to the cockpit display control interface module; The cockpit display control interface module provides all display control interfaces of flight control function, flight management function, system monitoring function, system control function and communication function in the aircraft cockpit, and transmits the control signals to each system interface according to the optimal control scheme generated by the multi-channel interactive information adjustment module.

2. The cockpit multi-channel human intelligence interaction system of claim 1, wherein, The switch type continuous adjustment controlled object state type control of the multi-channel interactive instruction matching module refers to the combination of touch serial operation and touch activation for continuous knob operation, including setting an interactive scene of permitted height on the flight control panel; The fly-by-wire type control of the multi-channel interactive instruction matching module continuously adjusts the state of the controlled object, which means that the continuous handle operation is performed in the way of haptic activation, including the interactive scenarios of disconnecting the automatic thrust in the automatic landing stage and pushing the thrust handle to the idle position; The discrete adjustment of the multi-channel interactive instruction matching module controls the state of the controlled object, which means that the knob operation, discrete knob operation and discrete handle operation are performed in the way of haptic activation, including the interactive scenarios of switching the navigation mode on the control panel, turning on the landing light switch and adjusting the descent rate by using the speed brake handle at different altitudes / speeds.

3. The cockpit multi-channel human intelligence interaction system of claim 1, wherein, The trigger condition of the multi-channel relay communication function module triggers the control operation, including flight parameters and state information, and provides logical operator connection, and generates control instructions when the trigger condition judgment statement is true. The flight parameters include flight altitude, airspeed, heading and attitude angle; the state information includes landing gear position, flap position and switch position. The logical operation includes logical AND, logical OR and logical NOT.

4. The cockpit multi-lane human intelligence interaction system of claim 1, wherein, The multi-channel priority sorting module sorts the importance of the multi-channel control instructions according to the system-level safety evaluation results of the aircraft development stage, including functional hazard evaluation, preliminary system safety evaluation and common cause analysis, based on the potential possibility of unexpected functional failure / fault in the aircraft system or unexpected functional failure / fault between related systems, and combining the different consequence severity of disasters, dangers, greater impact, smaller impact and no safety impact.

5. The cockpit multi-lane human intelligence interaction system of claim 1, wherein, The pilot node information flow intensity adjustment strategy of the multi-channel interactive information adjustment module is to use haptic serial operation, haptic activation and non-contact interaction modes such as voice, eye movement and gesture from the perspective of information simplification and intelligence.

6. The cockpit multi-lane human intelligence interaction system of claim 1, wherein, The pilot physiological and behavioral data include touch, voice, eye movement, gesture and electroencephalogram.

7. The cockpit multi-lane human intelligence interaction system of claim 6, wherein, The functional architecture of the multi-channel data access module includes acquisition terminal, data layer, feature layer, fusion layer and external interface. In the acquisition terminal layer, the terminals for monitoring the physiological state of the pilot include eye movement camera, electroencephalogram amplifier, electrocardiogram sensor and electromyography sensor, and the terminals for monitoring the behavioral state of the pilot include voice recorder, gesture capture device and touch recorder; In the data acquisition layer, the original data collected by the terminal are processed by the controller to send out structured storage, timeline synchronization multi-source physiological data, multi-modal human-computer interaction action data and instruction data; In the feature extraction layer, the corresponding physiological state features, behavioral state features and corresponding instruction semantics are extracted; In the feature fusion layer, the physiological state feature fusion and the behavioral state feature fusion are realized, and the further fusion of the two types of features is realized by combining the system state and the task state context; In the external interface layer, the functional architecture of the module provides the output interfaces of target icons, hot word semantics, target areas, gesture semantics and cognitive load, and simultaneously accesses the multi-channel interactive instruction matching module.

8. An aircraft cockpit multi-channel human-machine intelligent interaction method, characterized in that, The system is executed by using any one of claims 1-7.

Images