Control system with state change detection and multimodal interpretation

Abstract

Control system with state change detection and multimodal interpretation The present invention relates to a control system (14) comprising: - a modal state change detection module (32) configured to receive distinct modal inputs; the detection module being configured to detect a discrete state change of each modal input and to determine characteristic data of the current discrete state; the detection module also being configured to detect a user request; - a multimodal interpretation module (34) configured, at each detected state change, to store and update the characteristic data of the current discrete state of the modal input following the detected state change;The multimodal interpretation module is configured, for each incomplete query detected, to complete the incomplete query using up-to-date characteristic data stored from the current discrete state of another modal input. Figure for the abstract: 1;

Description

Title of the invention: Control system with state change detection and multimodal interpretation

[0001] The present invention relates to a control system based on multimodal interactions of a user.

[0002] The user, an aircraft pilot for example, can interact with a system through different means of communication involving different modalities, each modality being relative to a human sense with which the user can communicate, such as speech, sight, touch, or movement.

[0003] When the user issues a request via a first modality, such as a voice recognition device, it is known to manage certain modalities via a time window of events. This time window allows a temporal correlation between the inputs / events of the different modalities, in particular in the case of modalities which provide continuous states such as for example the movement of a cursor or a direction of gaze.

[0004] However, these known mechanisms for managing modalities are not satisfactory, in particular since they require a very complex definition of the time window for the modalities.

[0005] Furthermore, some known mechanisms only handle a limited number of modalities, typically only speech and touch, and are not adaptable to other modalities.

[0006] An aim of the invention is therefore to provide a control system making it possible to manage at least two modal inputs generated from the same modality or from respectively at least two distinct generic modalities and making it possible to manage requests from a user in a simple manner.

[0007] An additional aim of the invention is to provide a multimodality architecture capable of managing as many different input modalities as necessary, in a generic manner.

[0008] To this end, the invention relates to a control system comprising:

[0009] - a modal state change detection module configured to receive at at least two distinct modal inputs, each modal input being generated from the same modality or from at least two distinct modalities respectively;

[0010] the detection module being configured, for each modal input received, to detect a change of discrete state of the modal input from a previously assigned discrete state to a current discrete state newly assigned to the modal input and to determine data characteristic of the current discrete state. the modal entry following the detected state change;

[0011] the detection module also being configured to detect a user request from at least one of the received modal inputs;

[0012] - a multimodal interpretation module configured, at each change of state detected from one of the modal inputs, to store and update the characteristic data of the current discrete state of the modal input following the detected change of state;

[0013] the multimodal interpretation module also being configured, for each detected user request, to determine whether said request is sufficiently complete to be implemented, and, in the event of an incomplete request, to complete the incomplete request from at least one of the stored up-to-date characteristic data of the current discrete state of another modal input distinct from each modal input from which the request was detected.

[0014] The system according to the invention may comprise one or more of the following characteristics, taken in isolation or in any technically possible combination:

[0015] - at least two of the distinct modal inputs are generated respectively by distinct modalities, the multimodal interpretation module being configured, in the event of an incomplete query, to complete the incomplete intentional query from at least one of the stored up-to-date characteristic data of the current discrete state of the modal input generated by another modality, the other modality being distinct from each modality having generated the modal input from which the intentional query was detected;

[0016] - each discrete state change is detected based on at least one criterion of detection; and, preferably, the newly assigned current discrete state is chosen from a list of at least two possible discrete states; each detection criterion and each possible discrete state being advantageously stored in a file configurable externally by an operator;

[0017] - the multimodal interpretation module is configured, in the event of an incomplete request, to complete the incomplete query only from one or more up-to-date characteristic data stored from one or more current discrete states;

[0018] - at least one of the modal inputs is continuous, the continuous modal input having a continuous evolution over time, the detection module preferably being configured to detect a discrete change of state of the continuous modal input as a function of a degree of ongoing variation of the continuous modal input, the detection module advantageously being configured to detect a discrete change of state of the continuous modal input at least if the variation of the continuous modal input remains below a predetermined variation threshold for a predetermined minimum duration, for example at least if the variation is zero for said minimum duration predetermined;

[0019] - at least one of the modal inputs is discrete, the discrete modal input having a discrete evolution over time, the detection module being configured to detect a discrete change of state of the discrete modal input at each variation of the discrete modal input;

[0020] - the detection module is configured to detect a request from the user to from at least one discrete state change of one of the received modal inputs; and, preferably, the detection module comprises a database of queries and associated trigger conditions, each trigger condition relating to the current discrete state of at least one of the modal inputs, said database being advantageously stored in a file configurable externally by an operator;

[0021] - the detection module is further configured to convert each discrete state into newly assigned course following a detected change of state and each detected request into a transmission vector and to send said transmission vector to the multimodal interpretation module, each transmission vector having the same formatting common to all the transmission vectors sent by the detection module, said common formatting comprising at least fields, the detection module being configured, for each detected change of state, to fill the content of each field of the associated transmission vector according to said characteristic data associated with the newly assigned current discrete state following the detected change of state, the detection module being configured, for each detected request,to fill the contents of each field of the associated transmission vector based on the characteristic data associated with the newly assigned current discrete state following each detected state change from which the request was detected; ,

[0022] - the fields include a query identification field and at least one essential request field, the content of the request identification field being representative of the fact that the transmission vector is associated with a request, the multimodal interpretation module being configured to determine whether said request is sufficiently complete to be implemented at least if one of the essential fields of the associated transmission vector is empty;

[0023] - the multimodal interpretation module is configured, in the event of an incomplete request, to complete each empty essential field of the transmission vector associated with the incomplete query, each empty essential field being completed from the contents of the corresponding field of the transmission vector associated with the current discrete state of the modal input generated by said other modality;

[0024] - the detection module is configured to receive at least three modal inputs distinct, the multimodal interpretation module being configured to detect an ambiguity to complete the incomplete query, the ambiguity being between the current discrete states of at least two other modal inputs, said other modal inputs being distinct from each modal input from which the incomplete query was detected, the system further comprising a disambiguation module configured to provide a disambiguation directive to the multimodal interpretation module in the event of a detected ambiguity, the multimodal interpretation module being configured to complete the incomplete query based on said disambiguation directive provided by the disambiguation module, the disambiguation directive preferably comprising an order of priority on the modal inputs to complete the query;

[0025] - the disambiguation directive includes a query request of the user, the multimodal interpretation module then being configured to question the user according to the questioning request via a human-machine interface;

[0026] - the multimodal interpretation module is configured to detect an ambiguity between the current discrete states of said at least two other modal inputs, at least when the stored up-to-date characteristic data of said current discrete states, which correspond to the same empty essential field, are contradictory;

[0027] - the disambiguation directive is stored on a configurable file in a way external by an operator, the file preferably in xml format;

[0028] - the disambiguation module stores at least two disambiguation directives separate disambiguation directives, each disambiguation directive being associated with a separate user, the disambiguation module being configured to provide the disambiguation directive based on the user who originated the incomplete query;

[0029] - the control system comprises at least two distinct modalities, each modality comprising a technical means for acquiring at least one piece of sensory information associated with the user, each modality being configured to generate at least one modal input from the sensory information acquired by the technical acquisition means, and being configured to send each generated modal input to the detection module; and, preferably, one of the modalities is an acoustic modality, the detection module being configured to detect a request from the user from at least one discrete change of state of the modal input generated by the acoustic modality, and at least one other of the modalities is chosen from a tactile modality, an optical modality or a kinesthetic modality, and, preferably at least one other of the modalities is chosen from a tactile modality, an optical modality or a kinesthetic modality;

[0030] - the user is a pilot of an aircraft, for example an airplane or a drone.

[0031] The invention also relates to a control method comprising the following steps:

[0032] - generation of at least two distinct modal inputs, each modal input being generated from the same modality or from at least two distinct modalities respectively

[0033] - detection, for each modal input, of a discrete change of state of the input modal from a previously assigned discrete state to a newly assigned current discrete state at the modal input, and determining characteristic data of a current discrete state of the modal input following the detected state change;

[0034] - detecting a user query from at least one of the inputs modals;

[0035] the method comprising, at each detected change of state of one of the modal inputs, the storage and updating of the data characteristic of the current discrete state of the modal input following the detected change of state;

[0036] the method comprising, for each detected user request, determining whether said request is sufficiently complete to be implemented, and, in the event of an incomplete request, the incomplete request is then completed from at least one of the stored up-to-date characteristic data of the current discrete state of another modal input distinct from each modal input from which the request was detected.

[0037] The invention will be better understood on reading the following description, given solely by way of example, and made with reference to the appended drawings, in which:

[0038] [Fig-1] [Fig.l] is a schematic flowchart of an aircraft comprising a control system according to an exemplary embodiment of the invention; and

[0039] [Fig.2] [Fig.2] is a schematic flowchart of a control method according to an exemplary embodiment of the invention.

[0040] An example of aircraft 10 is illustrated in [Fig.l].

[0041] The aircraft is for example an airplane or a drone.

[0042] The aircraft 10 comprises a central avionics system 12.

[0043] The aircraft 10 also comprises a control system 14 for a user. The user here is for example a pilot of aircraft 10.

[0044] The aircraft 10 also comprises functional systems including, for example, systems 20 for measuring the aircraft 10, systems 22 for external communication, and systems 24 for actuating the controls of the aircraft 10.

[0045] The measurement systems 20 comprise, for example, components comprising sensors for measuring parameters external to the aircraft 10, such as temperature, pressure or speed, sensors for measuring parameters internal to the aircraft 10 and its various functional systems, and positioning sensors, such as GPS sensors, inertial units, and / or an altimeter.

[0046] The external communication systems 22 include, for example, components comprising radio systems, VOR / LOC, ADS, DME, ILS, radar systems, and / or satellite communication systems such as “SATCOM”.

[0047] The control systems 24 are capable of controlling flight parameters of the aircraft 10 and avionic states of the aircraft 10.

[0048] The control systems 24 include, for example, components comprising actuators capable of actuating controls of the aircraft 10, such as flaps, control surfaces, pumps, or even mechanical, electrical and / or hydraulic circuits, and software actuators.

[0049] The different systems 20 to 24 are connected to the central avionics system 12, for example digitally, by at least one data bus circulating on a network internal to the aircraft 10.

[0050] The central avionics system 12 comprises a central avionics unit 16 and at least one display unit 18, the display unit 18 being for example placed in the cockpit of the aircraft 10.

[0051] The cockpit of the aircraft 10 is for example located in the aircraft 10 itself, or in a remote control room of the aircraft 10.

[0052] The display unit 18 comprises a screen 26 and a display management assembly 28 on the screen 26.

[0053] The display management assembly 28 comprises, for example, at least one processor, and a memory comprising software modules capable of being executed by the processor to manage the display on the screen 26.

[0054] The display management assembly 28 is configured to display graphic elements on the screen 26, for example based on information coming from the measurement systems 20 and / or the external communication systems 22 and / or the control systems 24.

[0055] The control system 14 is connected to the central avionics system 12, by means of a data link.

[0056] The data link is for example a wired link or a wireless link.

[0057] Generally, the control system 14 comprises at least one module 32 modal state change detection and a multimodal interpretation module 34.

[0058] The control system 14 also preferably comprises a disambiguation module 36.

[0059] The control system 14 advantageously comprises a processing unit 38 including said modules 32, 34, 36.

[0060] The control system 14 preferably comprises at least two distinct modalities 30A-30X.

[0061] In an exemplary embodiment illustrated in [Fig.l], the control system 14 comprises at least three distinct modalities 30A-30X, preferably at least four distinct modalities or advantageously any number strictly greater than two of distinct modalities. In a variant not illustrated, the control system 14 comprises only two distinct modalities.

[0062] Generally, each modality 30A-30X comprises a technical means for acquiring at least one piece of sensory information associated with the user, and each modality 30A-30X is configured to generate at least one modal input from the sensory information acquired by the technical acquisition means.

[0063] Each modality 30A-30X generates a single modal input or at least two modal inputs from the same sensory information acquired by the technical acquisition means.

[0064] Each sensory information relates to a human sense with which the user can communicate, such as speech, sight, touch, movement or any other sense.

[0065] Each modal input generated by each modality 30A-30X evolves over time, depending on the user and their actions.

[0066] Each modality 30A-30X is configured to send each generated modal input to the detection module 32.

[0067] Preferably, at least one of the modalities 30A-30X is configured to generate at least one continuous modal input, the continuous modal input having a continuous evolution over time.

[0068] By "continuous" we mean, for example, that, between two successive states of the modal input, we can always assume an intermediate state.

[0069] In an exemplary embodiment, at least one of the continuous modal inputs continuously exhibits variation over time. An example of such a continuous modal input will be given below.

[0070] In an exemplary embodiment, at least one of the continuous modal inputs is capable of exhibiting zero variation for a minimum period. An example of such a continuous modal input will be given below.

[0071] Preferably, at least one of the modalities 30A-30X is configured to generate at least one discrete modal input, the discrete modal input having a discrete evolution over time.

[0072] The term “discrete” is here to be understood in opposition to the term “continuous”.

[0073] For example, discrete evolution is formed by a succession of stages.

[0074] Generally, in order to acquire sensory information and / or generate each modal input, each modality 30A-30X includes for example at least one computer processing device operatively connected to a computer memory, for example, a digital signal processor (DSP), a microcontroller, a field programmable gate array (FPGA) and / or a dedicated integrated circuit (ASIC) capable of performing various data processing operations and functions.

[0075] More specific examples of modalities 30A-30D will now be described. Each of these modalities 30A-30D is separately and individually known to those skilled in the art.

[0076] Said modalities thus preferably comprise at least one acoustic modality 30A, and / or at least one optical modality 30B, and / or at least one tactile modality 30C, and / or at least one kinesthetic modality 30D.

[0077] For example, said modalities comprise at least one acoustic modality 30A, at least one optical modality 30B, and at least one tactile modality 30C.

[0078] In the example illustrated in [Fig.l], said modalities comprise at least one acoustic modality 30A, at least one optical modality 30B, and at least two distinct tactile modalities 30C.

[0079] Within the framework of the invention, the control system 14 may comprise several distinct modalities of the same type or any other 30X modality conceivable by those skilled in the art.

[0080] Each acoustic modality 30A is configured to acquire acoustic information from the user and to generate at least one audio modal input from the acquired acoustic information.

[0081] To do this, the technical means of acquiring each acoustic modality 30A is an acoustic acquisition device 40.

[0082] Each acoustic modality 30A also includes an audio analysis device 42.

[0083] The acoustic acquisition device 40 is configured to acquire said acoustic information from the user.

[0084] The acoustic information includes, for example, sound waves, ultrasonic waves, other vibrations, and / or any combination of these waves.

[0085] The acoustic acquisition device 40 comprises for example at least one microphone and / or any other device capable of acquiring acoustic information.

[0086] The audio analysis device 42 is configured to convert the acquired acoustic information into said audio modal input.

[0087] The audio analysis device 42 is preferably configured to use a voice recognition technology, for example by automatic speech recognition, more commonly called RAP (or ASR in English). Such technology is known to those skilled in the art and will not be described in more detail here.

[0088] The audio analysis device 42 is configured to detect and identify different user sounds in the acquired acoustic information.

[0089] The user sounds may include phonetic sounds (e.g., words, sentences), emotional sounds (e.g., laughter, crying), musical sounds (e.g., clapping, humming), other sounds, and / or any combination of these sounds. Preferably, the user sounds include words.

[0090] The audio modal input is for example representative of user sounds, detected and identified by the audio analysis device 42, or of the absence of sound from the user.

[0091] Each optical modality 30B is configured to acquire optical information from the user and to generate at least one optical modal input from the acquired optical information.

[0092] To do this, the technical means of acquisition of each optical modality 30B is an optical acquisition device 44.

[0093] Each optical modality 30B also includes an optical analysis device 46.

[0094] The optical acquisition device 44 is configured to acquire said optical information from the user.

[0095] The optical information comprises, for example, electromagnetic radiation belonging to at least part of the electromagnetic spectrum.

[0096] Said part of the electromagnetic spectrum includes for example at least part of the visible light spectrum, and / or at least part of the infrared spectrum, and / or at least part of the ultraviolet spectrum.

[0097] The optical acquisition device 44 comprises for example at least one camera, a scanner, a light sensor, any other device capable of acquiring optical information and / or any combination of these elements.

[0098] The optical analysis device 46 is configured to convert the acquired optical information into said optical modal input.

[0099] The optical analysis device 46 is preferably configured to use computer vision technology. Such technology is known to those skilled in the art and will not be described in further detail here.

[0100] The optical analysis device 46 is configured to detect, from the acquired optical information, a movement, an absence of movement, a spatial orientation, a spatial location, and / or any combination of this information, of at least one part of the user's body.

[0101] The user's body parts include, for example, a hand, a head, a face, an eye, a mouth, any other part of the user's body and / or any combination of these parts.

[0102] Body part movements can be used to communicate and may include hand movements (e.g., gestures, sign language), head movements (e.g., nodding), facial movements (e.g., smiling), eye movements (e.g., blinking, looking in a specific direction), mouth movements (e.g., lip reading), other movements, and / or any combination of these movements.

[0103] In a preferred embodiment, the optical modal input is representative of an area pointed at by at least one part of the user's body. For example, the area pointed at is an area of ​​a display, for example an area of ​​the screen 26 of the display unit of the central avionics system 12.

[0104] The optical modal input is then in this example continuous, and for example continuously presents a variation over time.

[0105] Indeed, the parts of the user's body (such as the eyes) continually exhibit movements.

[0106] In an exemplary embodiment, the optical acquisition device 44 and the optical analysis device 46 of the optical modality 30B form an eye tracking device. The part of the body is then an eye of the user, and said pointed area is the area looked at by the eye.

[0107] The optical modal input includes, for example, a two-dimensional or three-dimensional format. Each optical modal input includes, for example, at least two positioning parameters of the pointed area which vary over time.

[0108] Each touch modality 30C is configured to acquire touch information from the user and to generate at least one touch modal input from the acquired touch information.

[0109] To do this, the technical means of acquiring each tactile modality 30C is a tactile acquisition device 48.

[0110] Each tactile modality 30C also includes a tactile analysis device.

[0111] Each tactile acquisition device 48 is configured to acquire said tactile information from the user.

[0112] At least one of the tactile acquisition devices 48 is for example included in a human-machine interface of the central avionics system 12.

[0113] Each tactile acquisition device 48 comprises for example a keyboard, a push button, and / or a pointing device and / or any combination of these elements.

[0114] Each keyboard comprises at least one key, for example hardware or software.

[0115] Each pointing device comprises, for example, a computer mouse, a controller 50 such as a joystick, a touchpad, and / or a touchscreen 52, and / or any combination of these elements.

[0116] In the case of a 30C touch mode comprising a keyboard or a push button, For example, the touch modal input is discreet.

[0117] In the example of [Fig. 1], the control system 14 comprises two distinct touch modalities 30C. The touch acquisition device 48 of a first touch modality 30C comprises a joystick 50 and the touch acquisition device 48 of a second touch modality 30C comprises a touch screen 52.

[0118] The tactile analysis device is configured to convert the acquired tactile information into said tactile modal input.

[0119] In the case of a touch modality 30C in which the touch acquisition device 48 comprises a pointing device (such as the joystick 50 or the touch screen 52 of [Fig. 1]), the touch modal input is representative of an area pointed at by the user. For example, the area pointed at is an area of ​​a display, for example an area of ​​the screen 26 of the display unit of the central avionics system 12.

[0120] The touch modal input is then continuous in this example.

[0121] The tactile modal input includes, for example, a two-dimensional or three-dimensional format. Each tactile modal input includes, for example, at least two positioning parameters of the pointed area that vary over time.

[0122] Each kinesthetic modality 30D is configured to acquire kinesthetic information from the user and to generate at least one kinesthetic modal input from the acquired kinesthetic information.

[0123] To do this, the technical means of acquiring each kinesthetic modality 30D is a kinesthetic acquisition device 54.

[0124] Each 30D kinesthetic modality also includes a kinesthetic analysis device.

[0125] Each kinesthetic acquisition device 54 is configured to acquire kinesthetic information from the user.

[0126] Kinesthetic information includes, for example, movement, lack of movement, spatial orientation, spatial location, and / or any combination of these information of at least one part of the user's body.

[0127] The user's body parts include, for example, a hand, a head, a face, a mouth, any other part of the user's body and / or any combination of these parts.

[0128] Each kinesthetic acquisition device 54 comprises, for example, at least one accelerometer, a gyroscope, a proximity detector, and / or any combination of these elements.

[0129] The kinesthetic analysis device is configured to convert the acquired kinesthetic information into said kinesthetic modal input.

[0130] Thus, the kinesthetic modal input is for example similar to what is described above for the optical modal input. The 30D kinesthetic modality differs from the optical modality 30B in that the technical acquisition means implemented are distinct.

[0131] The processing unit 38 comprises, for example, a computer processing device operatively connected to a computer memory, for example, a digital signal processor (DSP), a microcontroller, a field programmable grid array (FPGA) and / or a dedicated integrated circuit (ASIC) capable of performing various data processing operations and functions, in particular capable of performing at least the functions of the modules described below.

[0132] The computer processing device comprises, for example, a single processor. Alternatively, the computer processing device comprises several processors, which are located in the same geographical area, or are, at least partially, located in different geographical areas and are then able to communicate with each other.

[0133] Each occurrence of the term "memory" means any volatile or non-volatile computer memory suitable for the subject matter currently disclosed, such as random access memory (RAM), read only memory (ROM) or other electronic, optical, magnetic or other computer-readable storage media on which the data and functions of the modules described herein are, for example, stored.

[0134] Therefore, memory is a tangible storage medium where the data and functions of the modules described herein are for example stored in a non-transitory form.

[0135] The detection module 32 is configured to receive each modal input generated by each modality.

[0136] The detection module 32 is thus configured to receive over time the evolution of each modal input generated by each modality.

[0137] The detection module 32 is configured to process in parallel the modal inputs generated by each modality, i.e. separately and simultaneously.

[0138] In other words, the detection module 32 is capable of allowing the multimodal interpretation module 34 to be generic and adaptable to any type of modality and to any number of distinct modalities, that is to say adaptable for example to the or each acoustic modality 30A, and / or to the or each optical modality 30B, and / or to the or each tactile modality 30C, and / or to the or each kinesthetic modality 30D, and / or to any other modality 30X where appropriate.

[0139] The detection module 32 is configured, for each modal input, to detect a change in discrete state of the modal input from a previously assigned discrete state to a current discrete state newly assigned to the modal input, and to determine data characteristic of the current discrete state of the modal input following the detected change in state.

[0140] The current discrete state corresponds to a new discrete state that differs from the previously assigned discrete state.

[0141] The detection takes place in real time, for example during the piloting of the aircraft, in particular during the flight of the aircraft.

[0142] The discrete state change is detected based on at least one detection criterion.

[0143] Preferably, the detection module 32 is configured, for each modal input, to assign a current discrete state to the modal input chosen from a list of at least two possible discrete states.

[0144] The newly assigned current discrete state is in particular chosen from the list of possible discrete states.

[0145] Each possible discrete state is for example associated with a specific detection criterion.

[0146] In a preferred embodiment, for each modal input, each criterion of detection and each possible associated discrete state are stored in a file that can be configured externally by an operator.

[0147] The set of detection criteria and possible discrete states is suitable for being modified upstream of the operation of the control system 14 by the user, in particular by the addition, in the case of a new modality to be integrated into the control system, of new possible discrete states for any modal input generated by this new modality. Said set is then suitable for being loaded into the detection module 32.

[0148] Thus, the detection module 32 can be easily adapted to any new modality.

[0149] The detection module 32 is configured to assign and detect a current discrete state change of the modal input, independently of the nature of the evolution of the modal input (continuous or discrete in particular).

[0150] In other words, the detection module 32 only assigns and detects discrete state changes although the modal input may evolve continuously.

[0151] More specific examples of detection for modal inputs will now be described.

[0152] In the case of an acoustic modality 30A generating an audio modal input, the associated possible discrete states comprise states corresponding respectively to language elements suitable for being interpreted in the detected and identified user sounds of the audio modal input.

[0153] The language elements preferably correspond to a user intention or a user-designated element.

[0154] For each of these possible discrete states, the associated detection criterion corresponds thus at least to the recognition of at least one interpreted language element in the detected and identified user sounds of the audio modal input.

[0155] To do this, the detection module 32 is for example configured to implement a natural language understanding (NLU) processing. Such processing is for example capable of determining at least one interpreted language element corresponding to an intention of the user or an element designated by the user, from the detected and identified user sounds of the audio modal input.

[0156] The detection module 32 assigns the current discrete state corresponding to the interpreted language element(s) in the audio modal input.

[0157] The characteristic data of each possible discrete state are then, for example, representative of the interpreted language element(s).

[0158] The characteristic data then includes, for example, a material or virtual element designated vocally by the user or, as detailed below, a vocal request from the user.

[0159] The possible discrete states associated with the audio modal input also preferably include at least one state corresponding to an absence of user sounds or an absence of interpretation.

[0160] Thus, when the audio analysis device 42 no longer detects user sounds or the natural language understanding processing NLU no longer interprets language element(s), the detection module 32 also detects a change of state because the audio modal input varies, and the detection module 32 assigns a current discrete state corresponding to an absence of user sounds or an absence of interpretation.

[0161] The characteristic data of this current discrete state are then, for example, representative of an absence of user sounds or an absence of interpretation.

[0162] In the case of a 30C touch modality comprising a keyboard or a push button generating a touch modal input, the associated possible discrete states comprise states corresponding respectively to the actuation of each key of the keyboard or the actuation of the push button.

[0163] For each of these possible discrete states, the associated detection criterion thus corresponds at least to the actuation of the associated key or push button.

[0164] Thus, the detection module 32 detects a discrete state change of the touch modal input each time the keyboard or push button is actuation. The detection module 32 assigns the current discrete state corresponding to the key or push button actuated to the touch modal input.

[0165] The characteristic data of this current discrete state are then, for example, representative of said actuation.

[0166] Possible discrete states associated with the tactile modal input also include preferably at least one state corresponding to an absence of actuation.

[0167] Thus, when the user no longer actuates the keyboard or the push button, the detection module 32 also detects a change in state of the touch modal input, and assigns the current discrete state corresponding to an absence of actuation to the touch modal input.

[0168] The characteristic data of this current discrete state are then, for example, representative of an absence of actuation.

[0169] In the case of a tactile modality 30C comprising a pointing device, such as a joystick 50 or the touch screen 52, generating a tactile modal input representative of a pointed area, the associated possible discrete states comprise at least one state corresponding to a pointing of a target.

[0170] The target is for example a virtual element displayed by the screen 26.

[0171] Preferably, the associated detection criterion relates to a variation of the input tactile modal, for example on a degree of ongoing variation of the tactile modal input.

[0172] The detection criterion associated with the pointing state is for example verified at least if the variation of the tactile modal input relative to a target remains below a predetermined variation threshold for a non-zero predetermined minimum duration.

[0173] The target pointed at is for example fixed relative to a reference, the reference corresponding for example to the edges of the screen 26. The variation threshold is then for example zero.

[0174] In other words, the detection module 32 detects a discrete state change of the touch modal input when the pointing device is no longer moving and therefore the absolute variation of the associated touch modal input is zero. The detection module 32 assigns the current discrete state corresponding to the pointing of the fixed target to the touch modal input.

[0175] The pointed target is for example mobile relative to the reference. The variation threshold is then for example non-zero, to take into account a possible variation of the pointing device around the mobile pointed target.

[0176] In other words, the detection module 32 detects a discrete state change of the touch modal input when the pointing device moves and follows a moving target, the relative variation of the touch modal input with respect to the pointed target being below the non-zero variation threshold. The detection module 32 assigns the current discrete state corresponding to the pointing of the moving target to the touch modal input.

[0177] The characteristic data of the discrete state being pointed at are then, for example, representative of the pointed target, for example of the positioning parameters of said pointed target.

[0178] The possible discrete states associated with said tactile modal input also include preferably at least one state corresponding to an absence of pointing.

[0179] In particular, the detection criterion associated with the state of absence of pointing is verified at least if the variation of the continuous modal input passes above said predetermined variation threshold.

[0180] Thus, when the pointing device varies from the previously pointed area or target, the detection module 32 detects a change of state because the variation of the optical modal input passes above the variation threshold. The detection module 32 detects in particular a change of state and attributes the current discrete state of absence of pointing to the tactile modal input.

[0181] The characteristic data of this current discrete state are then, for example, representative of an absence of a pointed zone.

[0182] In a preferred variant, the state of absence of pointing is associated with another detection criterion, which is verified at least if the target remains pointed beyond a predetermined non-zero maximum duration.

[0183] In other words, the detection module 32 detects a discrete state change of the touch modal input when the pointing device no longer moves beyond the predetermined non-zero maximum duration. The detection module 32 assigns the current discrete state of no pointing to the touch modal input.

[0184] Such a detection criterion subsequently allows the multimodal interpretation module to ignore the target pointed at by the pointing device, beyond a certain duration.

[0185] In the case of an optical modality 30B comprising a gaze tracking device generating an optical modal input representative of an area pointed at by the user's eye, the associated possible discrete states comprise at least one state corresponding to a pointing of a target.

[0186] The target is for example a virtual element displayed by the screen 26.

[0187] Preferably, the associated detection criterion relates to a variation of the input optical modal, for example on a degree of ongoing variation of the optical modal input.

[0188] The detection criterion associated with the pointing state is for example verified at least if the variation of the optical modal input relative to a target remains below a predetermined variation threshold for a non-zero predetermined minimum duration.

[0189] The variation threshold is then, for example, non-zero, to take into account the continual movement of the eye even when it is pointing at the target.

[0190] The pointed target is for example fixed relative to a reference, the reference corresponding for example to the edges of the screen 26.

[0191] In other words, the detection module 32 detects a change in state of the optical modal input when the eye is no longer moving significantly, and therefore the user's eye fixes a specific area for a prolonged period, i.e. for a predetermined minimum duration. The detection module 32 assigns the current discrete state corresponding to the pointing of the fixed target to the optical modal input.

[0192] The pointed target is for example mobile relative to the reference.

[0193] In other words, the detection module 32 detects a discrete change of state of the optical modal input when the eye moves and follows a moving target, the relative variation of the optical modal input with respect to the pointed target being below the non-zero variation threshold. The detection module 32 assigns the current discrete state corresponding to the pointing of the moving target to the optical modal input.

[0194] The characteristic data of the discrete state being pointed at are then, for example, representative of the pointed target, for example the positioning parameters of said pointed target.

[0195] The possible discrete states associated with said optical modal input also preferably include at least one state corresponding to an absence of pointing.

[0196] In particular, the detection criterion associated with the state of absence of pointing is verified at least if the variation of the optical modal input passes above said predetermined variation threshold.

[0197] Thus, when the user's eye varies from the previously pointed area or target, the detection module 32 detects a change of state because the variation of the optical modal input passes above the predetermined variation threshold. The detection module 32 detects in particular a change of state and assigns a discrete state of no pointing to the optical modal input in which the user's eye does not point to any specific area.

[0198] The characteristic data of this current discrete state are then, for example, representative of an absence of a pointed zone.

[0199] The detection module 32 is also configured to detect an intentional request from the user from at least one of the received modal inputs.

[0200] The request relates to an action on at least one object.

[0201] The request is for example a request for information about the object, the action being the request for information. The request for information is for example of an open type and calls for a response other than yes / no.

[0202] The request is for example a command from the user relating to the object, the action being the command.

[0203] The command aims for example to intervene on said object in order to modify at least one flight parameter of the aircraft 10 during flight and / or at least one avionics state of the aircraft 10, by means of at least one of the control systems 24.

[0204] Any other request could be considered within the framework of the invention.

[0205] Any type of method for detecting such a request is conceivable within the framework of the invention.

[0206] Preferably, the detection module 32 is configured to detect such a request from at least one discrete change of state of one of the received modal inputs.

[0207] The detection of such a request takes place in real time, for example during the piloting of the aircraft, in particular during the flight of the aircraft.

[0208] To detect a request, the detection module 32 comprises, for example, a database of requests and associated trigger conditions.

[0209] In a preferred embodiment, said database is stored on a file configurable externally by an operator.

[0210] All of the requests and associated trigger conditions can be modified upstream of the operation of the control system 14 by the user, and can then be loaded into the detection module 32.

[0211] Each trigger condition preferably relates to at least one possible discrete state of at least one of the modal inputs.

[0212] The detection module 32 is for example configured to compare, with said database, the newly assigned current discrete state following each detected change of state of at least one of the modal inputs.

[0213] At least one query is for example detected in a unimodal manner.

[0214] Said request is then detected by one or more unimodal triggering conditions relating to only one of the modal inputs, i.e. relating to only a change of state of only one of the modal inputs.

[0215] The unimodal trigger conditions include, for example, an acoustic trigger condition relating to at least one discrete state change of the audio modal input. For example, the acoustic trigger condition relates to a user intention or a user-designated element forming data characteristic of the assigned current discrete state of the audio modal input following a state change of the audio modal input.

[0216] In other words, the detection module 32 is configured to detect a user request from at least one discrete change of state of the audio modal input generated by the acoustic modality 30A.

[0217] Unimodal trigger conditions also include, for example, at least one tactile trigger condition, one optical trigger condition, and / or one kinesthetic trigger condition.

[0218] At least one request is for example detected in a multimodal manner. Said request is then detected by at least two unimodal trigger conditions relating respectively to two distinct modal inputs, for example two modal inputs generated by two distinct modalities.

[0219] The detection module 32 is configured to send each current discrete state newly assigned following a detected state change, and each detected request to the multimodal interpretation module 34.

[0220] The detection module 32 is thus interposed between the multimodal interpretation module 34 and each modality, and only sends elements to the multimodal interpretation module 34 upon change of state of the modal inputs.

[0221] To do this, in a preferred embodiment, the detection module 32 is configured to convert each newly assigned current discrete state following a detected state change and each detected request into a transmission vector and to send said transmission vector to the multimodal interpretation module 34.

[0222] Advantageously, each transmission vector has the same formatting common to all the transmission vectors sent by the detection module 32 to the multimodal interpretation module 34.

[0223] The formatting is in particular the same whether the transmission vector is associated with a detected request or with a detected state change which is not associated with any request.

[0224] In other words, all modal inputs are managed under the same formalism. It is possible to have an architecture of the control system 14 which remains generic, without depending on the nature of the modalities considered. In particular, it is possible to easily add or delete modalities.

[0225] Common formatting is configurable per external file.

[0226] The common formatting then includes at least query and state change fields.

[0227] The detection module 32 is configured, for each detected change of state, to fill the content of each field of the associated transmission vector according to said characteristic data associated with the newly assigned current discrete state following the detected change of state.

[0228] In addition, the detection module 32 is configured, for each detected request, to fill the content of each field of the associated transmission vector according to the characteristic data associated with the newly assigned current discrete state following each detected change of state from which the request was detected.

[0229] In particular, in each case, the detection module 32 is configured to determine, among these characteristic data, the data relating to the content of each field.

[0230] The detection module 32 is for example configured to fill all the fields not corresponding to any of the characteristic data with information representative of an empty character of the field.

[0231] The fields comprise, for example, at least one field storing information representative of the original modal input.

[0232] The fields preferably include a query identification field.

[0233] The content of the request identification field is representative of whether the transmission vector is associated with a request or not.

[0234] The content of the request identification field is also, for example, representative of the action forming the request detected where applicable (for example, the fact that a distance is requested).

[0235] The fields preferably include at least one essential query field for implementing the query.

[0236] In particular, as explained below, in the case where at least one of the essential fields stores information representative of the empty nature of the field, the request is incomplete and cannot be implemented.

[0237] In one embodiment, the essential fields comprise, for example, at least one field relating to an identification of the object concerned (for example, the object whose distance the user wishes to know) and, for example, a field relating to a type of object concerned.

[0238] For example, the fields also include at least one field that is not essential to implement the query.

[0239] For example, such a non-essential field relates to additional complementary information on the object concerned by the request.

[0240] The multimodal interpretation module 34 is interposed between the detection module 32 and the central avionics system 14.

[0241] The multimodal interpretation module 34 is configured to send requests to the central avionics system 14 for implementation, the requests only being sent to the central avionics system 14 if they are sufficiently complete to be implemented.

[0242] To do this, the multimodal interpretation module 34 is configured, at each detected change of state of one of the modal inputs, to store and update the characteristic data of the newly assigned current discrete state of the modal input following the detected change of state.

[0243] In other words, for each modal input, the multimodal interpretation module 34 is configured to store in memory the last discrete state change that took place for the modal input.

[0244] In particular, for each change of state detected by the detection module 32, the multimodal interpretation module 34 is configured to receive each associated transmission vector and to update said characteristic data of the newly assigned current discrete state as a function of the transmission vector.

[0245] In an alternative embodiment, the multimodal interpretation module 34 is configured to store a history of the current discrete states successfully assigned- sively over time.

[0246] The multimodal interpretation module 34 is configured, for each detected user request, to determine whether said request is sufficiently complete to be implemented.

[0247] In an exemplary embodiment, the multimodal interpretation module 34 is configured to determine whether said request is sufficiently complete to be implemented at least if one of the essential fields of the associated transmission vector is empty.

[0248] In the event of an incomplete query, the multimodal interpretation module 34 is configured to complete the incomplete query from at least one of the stored up-to-date characteristic data of the current discrete state of another modal input, for example a modal input generated by another modality. The other modal input, and where appropriate the other modality, is distinct from the one(s) from which the query was detected.

[0249] More specifically, the multimodal interpretation module 34 is configured to compare the incomplete query with the stored up-to-date characteristic data of the current discrete state of each other modal input.

[0250] The query is completed following this comparison.

[0251] In an exemplary embodiment, the multimodal interpretation module 34 is configured to complete the incomplete query solely from one or more stored up-to-date characteristic data of one or more current discrete states.

[0252] In other words, the multimodal interpretation module 34 does not require a time window to complete the requests, but is based on the up-to-date discrete states, i.e. on the last detected state changes of the modal inputs.

[0253] Thus, in the event of an incomplete query, the multimodal interpretation module 34 is configured to complete the incomplete query without necessarily determining a complex time window for the modal inputs.

[0254] Alternatively, the multimodal interpretation module 34 is configured to complete the incomplete query from one or more up-to-date characteristic data stored from one or more current discrete states, and from a time window comprising a time correlation between the different modal inputs. Such a time window is for example determined by the detection module 32.

[0255] In an exemplary embodiment, the multimodal interpretation module 34 is configured to complete the incomplete request by completing each empty essential field of the transmission vector associated with the incomplete request.

[0256] Each empty essential field is completed from the contents of the corresponding field associated with the current discrete state of said other modal input.

[0257] The multimodal interpretation module 34 is for example capable of using the states discrete in progress of at least two distinct modal inputs to respectively complete different empty essential fields of the incomplete query.

[0258] After completing the previously incomplete request, the multimodal interpretation module 34 is configured to send the completed request to the central avionics system 14 for implementation.

[0259] In an advantageous embodiment in which the or all of the modalities generates in total at least three distinct modal inputs which are received by the detection module 32, the multimodal interpretation module 34 is further configured to detect an ambiguity to complete the incomplete query.

[0260] This is for example the example illustrated in [Fig.l], in which at least three distinct modalities respectively generate at least one modal input.

[0261] The detected ambiguity is an ambiguity between the current discrete states of at least two other modal inputs, for example generated respectively by two other modalities, said other modal inputs being distinct from the one(s) from which the incomplete query was detected.

[0262] To do this, the multimodal interpretation module 34 is for example configured to detect such ambiguity following the comparison of the incomplete query with the stored up-to-date characteristic data of the current discrete state of each other modal input.

[0263] Ambiguity is detected at least when the stored up-to-date characteristic data of at least two current discrete states, which correspond to the empty essential field associated with the incomplete query, are contradictory.

[0264] In the event of a detected ambiguity, the multimodal interpretation module 34 is configured to call upon the disambiguation module 36.

[0265] The disambiguation module 36 is configured to provide a disambiguation directive to the multimodal interpretation module 34 in the event of a detected ambiguity.

[0266] The disambiguation module 36 stores in particular said disambiguation directive.

[0267] The disambiguation directive preferably includes a priority order on the modal inputs to complete the query.

[0268] The disambiguation directive additionally or alternatively includes a user query request. In an exemplary embodiment, the query request is part of the priority order.

[0269] In a preferred embodiment, the disambiguation directive is stored on a file externally configurable by an operator.

[0270] The file is in xml format for example.

[0271] Thus, the file is suitable for being modified upstream of the operation of the control system 14 by the user, in particular according to his preferences, and is then suitable for loading into the disambiguation module 36.

[0272] Preferably, the disambiguation module 36 stores at least two distinct disambiguation directives, each disambiguation directive being associated with a distinct user.

[0273] Preferably, the disambiguation module 36 stores as many distinct disambiguation directives as there are distinct users.

[0274] The disambiguation module 36 is configured to provide the disambiguation directive based on the user who originated the incomplete query.

[0275] The two separate directives have, for example, separate orders of priority.

[0276] Following the provision of the disambiguation directive, the multimodal interpretation module 34 is configured to complete the incomplete query based on said provided disambiguation directive.

[0277] In particular, the multimodal interpretation module 34 completes the incomplete request by following the order of priority on the modal inputs.

[0278] More specifically, the multimodal interpretation module 34 is configured to complete the empty essential field with the contents of the corresponding non-empty field associated with the current discrete state of the modal input given by the priority order.

[0279] Furthermore, if the disambiguation directive includes a query request from the user, the multimodal interpretation module 34 is then configured to query the user based on the query request, for example via the central avionics system 14 which then includes a human-machine interface 56 for this purpose.

[0280] The central avionics system 14 is capable of acquiring the user's response to the questioning request and sending the acquired response to the multimodal interpretation module 34.

[0281] The multimodal interpretation module 34 is capable of completing the incomplete request based on the acquired response.

[0282] A control method 100 will now be described with reference to [Fig.2].

[0283] The control method 100 is for example implemented by computer and in particular by the control system 14 described above. The control method 100 then comprises the provision 102 of the control system 14 described above.

[0284] The method 100 comprises the implementation of the functions of the different modules 32, 34, 36 described above.

[0285] The method 100 comprises generating at least two distinct modal inputs, each modal input being generated from the same modality or from at least two distinct modalities respectively.

[0286] Each modal input generated by each modality 30A-30X evolves over time, depending on the user and their actions.

[0287] The method 100 comprises detecting 104, for each modal input, a change in discrete state of the modal input from a previously assigned discrete state to a current discrete state newly assigned to the modal input and determining data characteristic of the current discrete state of the modal input following the detected change in state.

[0288] More specifically, the allocation step 104 comprises, for each modal input, the detection of a change in discrete state of the modal input from a previously allocated discrete state to a new current discrete state.

[0289] The detection 104 is done in a similar manner to what was described above for the functions of the detection module 32.

[0290] The detection 104 takes place in real time, for example during the piloting of the aircraft, in particular during the flight of the aircraft.

[0291] The method 100 comprises, at each detected change of state of one of the modal inputs, a step 106 of storing and updating the data characteristic of the current discrete state of the modal input following the detected change of state.

[0292] The method 100 also comprises detecting 108 a user query from at least one of the modal inputs.

[0293] Any type of method for detecting such a request is conceivable within the framework of the invention.

[0294] Preferably, the detection 108 of a request is from at least one discrete change of state of the modal input generated by one of the modalities.

[0295] The detection 108 is done in a similar manner to what was described above for the functions of the detection module 32.

[0296] The detection 108 takes place in real time, for example during the piloting of the aircraft, in particular during the flight of the aircraft.

[0297] The method 100 further comprises, for each detected user request, a step 110 of determining whether said request is sufficiently complete to be implemented.

[0298] In the event of an incomplete query, the incomplete query is then completed (step 112) from at least one of the stored up-to-date characteristic data of the current discrete state of another modal input, the other modal input being distinct from the one(s) from which the query was detected.

[0299] The storage and updating step 106, the determination step 110 and the step 112 during which the incomplete query is completed are carried out in a similar manner to what was described above for the functions of the multimodal interpretation module 34.

[0300] In an advantageous embodiment in which the modality(ies) generates in total at least three distinct modal inputs, the method 100 comprises in in addition to detecting 114 an ambiguity to complete the incomplete query, and providing 116 a disambiguation directive, the incomplete query being completed (step 118) based on said provided disambiguation directive.

[0301] These steps 114, 116 are carried out in a similar manner to what was described above for the functions of the disambiguation module 36.

[0302] After being completed, the previously incomplete request is sent to the central avionics system 14 for implementation (step 120).

[0303] More specific examples of situations will now be described in the context of the aircraft 10 of [Fig.l].

[0304] The user is then a pilot of the aircraft 10, and the screen 26 displays waypoints.

[0305] A first example corresponds to the situation in which the touch screen 52 is not manipulated by the user, the joystick 50 does not point to any specific area or points to an area without a virtual element, the user fixes one of the waypoints displayed on the screen 26, designated by the terms waypoint 1, and the pilot says “what is the distance to the waypoint?”.

[0306] The detection module 32 then detects a change in state of the optical modal input from a state corresponding to an absence of pointing, to a current discrete state corresponding to the pointing of a fixed target by the user's eye, which is here a precise zone of the screen 26 including the virtual element corresponding to the waypoint 1. The characteristic data of this current discrete state include the waypoint 1 located in the zone pointed at by the eye.

[0307] The detection module 32 does not detect a user request from this optical modal input, since this change of state does not correspond in this particular example to any triggering condition.

[0308] The associated transmission vector sent by the detection module 32 includes in this example the following fields:

[0309] • Modal input: optical

[0310] • Request ID: empty

[0311] • Type of object concerned: waypoint

[0312] • Identifier of the object concerned: waypoint 1

[0313] • Additional information on the object concerned: Empty

[0314] The multimodal interpretation module 34 stores and updates accordingly the characteristic data of the current discrete state of the optical modal input following this detected change of state.

[0315] The audio analysis device 42 detects and identifies, in parallel, different user sounds in the acquired acoustic information corresponding here to the sequence of terms “what is the distance to the waypoint?” and forming the audio modal input.

[0316] By means of the NLU natural language understanding processing, the detection module 32 identifies language elements in this audio modal input, corresponding here to a query relating to a distance, and to a type of object concerned by the query, here a waypoint.

[0317] The detection module 32 then detects in parallel a change of state of the audio modal input from a previously assigned discrete state corresponding to an absence of user sounds or an absence of interpretation, to a current discrete state corresponding to the language elements interpreted in the audio modal input. The characteristic data of this current discrete state are then representative of these interpreted language elements.

[0318] The detection module 32 detects a user query from the audio modal input, and in particular from this detected change of state of the audio modal input. This query is detected from the comparison of this detected change of state of the audio modal input with the query database, at least one of the interpreted language elements being recognized as one of the trigger conditions of the database.

[0319] The transmission vector associated with the request sent by the detection module 32 includes in this example the following fields:

[0320] • Modal input: audio

[0321] • Request ID: distance

[0322] • Type of object concerned: waypoint

[0323] • Identifier of the object concerned: empty

[0324] • Additional information on the object concerned: Empty

[0325] The multimodal interpretation module 34 determines that the request is not sufficiently complete to be implemented, since the essential field corresponding to the identifier of the object concerned is empty.

[0326] The multimodal interpretation module 34 completes the incomplete query only from the stored up-to-date characteristic data of the current discrete state of the optical modal input.

[0327] The current characteristic data of the updated discrete states of the other modal inputs, in this case the two touch modal inputs associated with the controller 50 and the touch screen 52, do not allow the request to be completed.

[0328] Indeed, the current discrete states stored by the multimodal interpretation module 34 for the modal inputs associated with the joystick 50 and the touch screen 52 are states corresponding to an absence of pointing.

[0329] The multimodal interpretation module 34 completes the empty essential field of the identifier of the object concerned with the characteristic data “waypoint 1” of the current discrete state of the optical modal input.

[0330] The multimodal interpretation module 34 sends the complete request, corresponding to the request “what is the distance to waypoint 1?”, to the central avionics system 14.

[0331] The central avionics system 14 implements the complete request provided by the multimodal interpretation module 34. In this case, the central avionics system 14 provides the user with a response to the completed request.

[0332] A second example corresponds to the situation in which the touch screen 52 is not manipulated by the user, the user is looking at one of the waypoints displayed on the screen 26, designated by the terms waypoint 1, and has manipulated the joystick 50 so that it points to another of the waypoints displayed on the screen 26, designated by the terms waypoint 2, and the pilot says "what is the distance to the waypoint?".

[0333] The optical modal input is processed as in the first example.

[0334] The detection module 32 then detects in parallel a change of state of the tactile modal input associated with the joystick 50 from a state corresponding to an absence of pointing, to a current discrete state corresponding to a pointing of a target, which is here a precise zone of the screen 26 including the virtual element corresponding to the waypoint 2. The detection was made because the joystick 50 presents a zero variation with respect to said waypoint 2. The characteristic data of this current discrete state include the waypoint 2 located in the zone pointed by the joystick 50.

[0335] The associated transmission vector sent by the detection module 32 includes in this example the following fields:

[0336] • Modal input: touch - controller

[0337] • Request ID: empty

[0338] • Type of object concerned: waypoint

[0339] • Identifier of the object concerned: waypoint 2

[0340] • Additional information on the object concerned: Empty

[0341] The multimodal interpretation module 34 stores and updates accordingly the data characteristic of the current discrete state of the tactile modal input associated with the controller 50 following this detected change of state.

[0342] As in the first example, the multimodal interpretation module 34 determines that the request associated with the acoustic modality 30A is not sufficiently complete to be implemented, insofar as the essential field corresponding to the identifier of the object concerned is empty.

[0343] However, the multimodal interpretation module 34 detects an ambiguity to complete the incomplete query. Indeed, the stored up-to-date characteristic data of the current discrete state of the optical modal input and the current discrete state of the tactile modal input associated with the joystick 50 are contradictory, between waypoint 1 and waypoint 2.

[0344] The multimodal interpretation module 34 then calls upon the disambiguation module 36 which provides a disambiguation directive.

[0345] In one embodiment, the disambiguation directive comprises a priority order, in which the optical modality 30B takes priority over the tactile modality 30C.

[0346] Consequently, the multimodal interpretation module 34 completes the empty essential field of the identifier of the object concerned with the characteristic data “waypoint 1” of the current discrete state of the optical modal input.

[0347] The multimodal interpretation module 34 sends the complete request, corresponding to the request “what is the distance to waypoint 1?”, to the central avionics system 14.

[0348] The central avionics system 14 implements the complete request provided by the multimodal interpretation module 34. In this case, the central avionics system 14 provides the user with a response to the completed request.

[0349] In another embodiment, the disambiguation directive comprises a user questioning request.

[0350] Accordingly, the multimodal interpretation module 34 interrogates the user according to the questioning request, via the central avionics system 14. The central avionics system 14 acquires the user's response to the questioning request and sends the acquired response to the multimodal interpretation module 34.

[0351] The multimodal interpretation module 34 is capable of completing the incomplete request based on the acquired response. For example, if the user's response is to prioritize the touch modal input associated with the controller 50, the multimodal interpretation module 34 completes the empty essential field of the identifier of the object concerned with the characteristic data “waypoint 2” of the current discrete state of the touch modal input 30C associated with the controller 50.

[0352] The multimodal interpretation module 34 then sends the complete request, corresponding to the request “what is the distance to waypoint 2?”, to the central avionics system 14.

[0353] Thanks to the characteristics described above, the system of the invention is based on a modal input management mechanism detecting state changes and operating on the basis of current discrete states assigned to complete user requests, without requiring the use of a time window to temporally correlate the modal inputs with each other. This drastically simplifies the management of multimodality.

[0354] It is always possible for the control system to take into account a temporality between the modal inputs, but this is not necessary for the management of the modal inputs in the invention.

[0355] Furthermore, the system of the invention presents a multimodality architecture, corresponding to modules 32 and 34, which can take as input as many different modalities as necessary, in a generic manner.

[0356] Modules 32 and 34 form a multimodality architecture which is functionally independent of the part of the system which applies the control. It can therefore easily be adapted to an existing system, without modifying the architecture of this existing system.

[0357] Furthermore, the system of the invention is also preferably based on a configurable disambiguation mechanism, making it possible to manage ambiguous situations by prioritizing the modalities in relation to each other and making it possible to know when to question the user to disambiguate.

Claims

Claims

1. Control system (14) comprising: - a modal state change detection module (32) configured to receive at least two distinct modal inputs, each modal input being generated from the same modality or from respectively at least two distinct modalities; the detection module (32) being configured, for each modal input received, to detect a discrete state change of the modal input from a previously assigned discrete state to a current discrete state newly assigned to the modal input and to determine data characteristic of the current discrete state of the modal input following the detected state change; the detection module (32) also being configured to detect a user request from at least one of the modal inputs received;- a multimodal interpretation module (34) configured, at each detected change of state of one of the modal inputs, to store and update the characteristic data of the current discrete state of the modal input following the detected change of state; the multimodal interpretation module (34) also being configured, at each detected user request, to determine whether said request is sufficiently complete to be implemented, and, in the event of an incomplete request, to complete the incomplete request from at least one of the stored up-to-date characteristic data of the current discrete state of another modal input distinct from each modal input from which the request was detected.;

2. A control system (14) according to claim 1, wherein each discrete state change is detected based on at least one detection criterion; and, preferably, the newly assigned current discrete state is chosen from a list of at least two possible discrete states; each detection criterion and each possible discrete state being advantageously stored on a file externally configurable by an operator.

3. Control system (14) according to one of claims 1 or 2, in which the multimodal interpretation module (34) is configured, in the event of an incomplete request, to complete the incomplete request solely from one or more up-to-date characteristic data. stored in one or more current discrete states.

4. A control system (14) according to any preceding claim, wherein at least one of the modal inputs is continuous, the continuous modal input having a continuous evolution over time, the detection module (32) preferably being configured to detect a discrete change of state of the continuous modal input as a function of a current degree of variation of the continuous modal input, the detection module (32) advantageously being configured to detect a discrete change of state of the continuous modal input at least if the variation of the continuous modal input remains below a predetermined variation threshold for a predetermined minimum duration, for example at least if the variation is zero for said predetermined minimum duration.

5. A control system (14) according to any preceding claim, wherein at least one of the modal inputs is discrete, the discrete modal input having a discrete evolution over time, the detection module (32) being configured to detect a discrete change in state of the discrete modal input at each variation of the discrete modal input.

6. A control system (14) according to any preceding claim, wherein the detection module (32) is configured to detect a user request from at least one discrete state change of one of the received modal inputs; and, preferably, the detection module (32) comprises a database of requests and associated trigger conditions, each trigger condition relating to the current discrete state of at least one of the modal inputs, said database being advantageously stored on a file externally configurable by an operator.

7. The control system (14) of claim 6, wherein the detection module (32) is further configured to convert each newly assigned current discrete state following a detected state change and each detected request into a transmission vector and to send said transmission vector to the multimodal interpretation module (34), each transmission vector having the same formatting common to all the transmission vectors sent by the detection module (32), said common formatting comprising at least fields, the detection module (32) being configured, for each detected change of state, to fill the content of each field of the associated transmission vector according to said characteristic data associated with the newly assigned current discrete state following the detected change of state, the detection module (32) being configured, for each detected request, to fill the content of each field of the associated transmission vector according to the characteristic data associated with the newly assigned current discrete state following each detected change of state from which the request was detected.

8. The control system (14) of claim 7, wherein the fields comprise a request identification field and at least one request essential field, the content of the request identification field being representative of whether the transmission vector is associated with a request, the multimodal interpretation module (34) being configured to determine whether said request is sufficiently complete to be implemented at least if one of the essential fields of the associated transmission vector is empty.

9. A control system (14) according to claim 8, wherein the multimodal interpretation module (34) is configured, in the event of an incomplete request, to complete each empty essential field of the transmission vector associated with the incomplete request, each empty essential field being completed from the contents of the corresponding field of the transmission vector associated with the current discrete state of the modal input generated by said other modality.

10. A control system (14) according to any preceding claim, wherein the detection module (32) is configured to receive at least three distinct modal inputs, the multimodal interpretation module (34) being configured to detect an ambiguity to complete the incomplete query, the ambiguity being between the current discrete states of at least two other modal inputs, said other modal inputs being distinct from each modal input from which the incomplete query was detected, the system further comprising a disambiguation module (36) configured to provide a disambiguation directive to the multimodal interpretation module (34) in the event of a detected ambiguity, the multimodal interpretation module (34) being configured to complete the incomplete query based on said disambiguation directive. ambiguation provided by the disambiguation module (36), the disambiguation directive preferably including a priority order on the modal inputs to complete the query.

11. The control system (14) of claim 10, wherein the disambiguation directive comprises a questioning request from the user, the multimodal interpretation module (34) then being configured to question the user based on the questioning request via a human-machine interface (56).

12. A control system (14) according to any one of claims 10 or 11, wherein the multimodal interpretation module (34) is configured to detect an ambiguity between the current discrete states of said at least two other modal inputs, at least when the stored up-to-date characteristic data of said current discrete states, which correspond to the same empty essential field, are contradictory.

13. A control system (14) according to any one of claims 10 to 12, wherein the disambiguation directive is stored on a file externally configurable by an operator, the file preferably having an xml format.

14. The control system (14) of any one of claims 10 to 13, wherein the disambiguation module (36) stores at least two distinct disambiguation directives, each disambiguation directive being associated with a distinct user, the disambiguation module (36) being configured to provide the disambiguation directive based on the user originating the incomplete query.

15. Control system (14) according to any one of the preceding claims, wherein the control system comprises at least two distinct modalities (30A-30X), each modality (30A-30X) comprising a technical means for acquiring at least one piece of sensory information associated with the user, each modality (30A-30X) being configured to generate at least one modal input from the sensory information acquired by the technical acquisition means, and being configured to send each generated modal input to the detection module (32); and, preferably, one of the modalities (30A-30X) is an acoustic modality (30A), the detection module (32) being configured to detect a request from the user from at least one discrete change of state of the modal input generated by the acoustic modality (30A), and at least one other of the modalities is selected from a tactile modality (30C), an optical modality (30B) or a kinesthetic modality (30D), and, preferably at least one other of the modalities is selected from a tactile modality (30C), an optical modality (30B) or a kinesthetic modality (30D).

16. A control method comprising the following steps: - generation of at least two distinct modal inputs, each modal input being generated from the same modality or from at least two distinct modalities respectively - detecting, for each modal input, a change in discrete state of the modal input from a previously assigned discrete state to a current discrete state newly assigned to the modal input, and determining data characteristic of a current discrete state of the modal input following the detected change in state; - detecting a user query from at least one of the modal inputs; the method comprising, at each detected change of state of one of the modal inputs, the storage and updating of the data characteristic of the current discrete state of the modal input following the detected change of state; the method comprising, for each detected user query, determining whether said query is sufficiently complete to be implemented, and, in the event of an incomplete query, the incomplete query is then completed from at least one of the stored up-to-date characteristic data of the current discrete state of another modal input distinct from each modal input from which the query was detected.

Images