Intelligent monitoring and estimation of crew communications

Abstract

The invention relates to intelligent monitoring and estimation of crew communications, and discloses techniques for monitoring communications in a physical environment. The techniques include receiving a textual representation based on verbal communications relating to a physical environment from one or more users. The techniques also include generating a communication state of the physical environment based on the textual representation, and determining an actual state of the physical environment based on one or more measured characteristics of the environment. The techniques also include automatically providing an alert to at least one user of the one or more users based on comparing the communication status of the physical environment to the actual status of the physical environment.

Description

Technical Field

[0001] Aspects of the present disclosure relate to intelligent monitoring of communications in an environment. Background Art

[0002] The training and operation of aircraft systems is critical. Currently, human instructors are typically responsible for monitoring and evaluating students' communications skills during aviation training activities. The assessment of crew communications during a training session conducted in a flight training device (e.g., a flatbed trainer, a full flight simulator, or any other suitable flight training device) is performed by a human instructor who facilitates the training session. This human-in-the-loop activity introduces subjectivity into the evaluation process, potentially leading to misclassification of communications performance and other errors. Furthermore, access to professional human instructors is limited, which in turn limits the amount of training that can be performed.

[0003] Automated training systems are designed to provide students with more opportunities to participate in training activities by reducing reliance on professional human instructors. However, without the ability to provide automated assessment of crew communications, these systems currently provide students with training opportunities limited to single-user training activities. Since the majority of aviation training courses are designed for crew activities, automated training systems currently have limited applicability for training. Summary of the invention

[0004] Aspects include a computer-implemented method. The method includes receiving a textual representation based on verbal communications from one or more users related to a physical environment. The method also includes generating a communicated state of the physical environment based on the textual representation. The method also includes determining an actual state of the physical environment based on one or more measured characteristics of the environment. The method also includes automatically providing an alert to at least one of the one or more users based on comparing the communicated state of the physical environment to the actual state of the physical environment.

[0005] Aspects also include a non-transitory computer readable medium containing computer program code that, when executed by operation of one or more computer processors, performs operations. The operations include receiving a textual representation based on verbal communications from one or more users related to a physical environment. The operations also include generating a communicated state of the physical environment based on the textual representation. The operations also include determining an actual state of the physical environment based on one or more measured characteristics of the environment. The operations also include automatically providing an alert to at least one of the one or more users based on comparing the communicated state of the physical environment to the actual state of the physical environment.

[0006] Aspects also include a system. The system includes a computer processor and a memory having instructions stored thereon, the instructions performing operations when executed on the computer processor. The operations include receiving a textual representation based on verbal communications from one or more users related to a physical environment. The operations also include generating a communicated state of the physical environment based on the textual representation. The operations also include determining an actual state of the physical environment based on one or more measured characteristics of the environment. The operations also include automatically providing an alert to at least one of the one or more users based on comparing the communicated state of the physical environment with the actual state of the physical environment.

[0007] In further aspects, in conjunction with any of the above or below examples, the communicated state of the physical environment includes one or more communicated characteristics of the aircraft, and the actual state of the physical environment includes one or more measured characteristics of the aircraft.

[0008] In other aspects, in combination with any of the above or below examples, generating a communicated state of the physical environment based on the textual representation includes: identifying one or more expected checklist items or procedures for the aircraft, and determining the communicated state of the physical environment by probabilistically matching a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures.

[0009] In yet other aspects, in conjunction with any of the above or below examples, identifying the one or more expected checklist items or procedures for the aircraft includes retrieving the one or more expected checklist items or procedures from an electronic repository.

[0010] In yet other aspects, in conjunction with any of the above or below examples, one or more expected checklist items or procedures for the aircraft include a blank variable associated with a particular variable type, and the probabilistic match is based on determining that at least a portion of the first phrase matches the particular variable type.

[0011] In yet other aspects, in conjunction with any of the above or below examples, the particular variable type comprises a numeric variable type.

[0012] In some further aspects, in conjunction with any of the above or below examples, the probabilistic matching uses a hidden Markov model.

[0013] In some further aspects, in combination with any of the above or below examples, the method or operation includes: generating a user task list including multiple tasks for one or more users in a physical environment; and determining that one or more users have completed at least a portion of the user task list based on the text representation.

[0014] In yet further aspects, in combination with any of the above or below examples, the method or operations include determining, based on the user task list, that a workload threshold of a user among the one or more users has been reached, and providing an alert to the user based on the determination.

[0015] In further aspects, in combination with any of the above or below examples, comparing the communicated state of the physical environment to the actual state of the physical environment includes determining that a difference between a first state value in the communicated state and a second state value in the actual state exceeds a threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order that the manner in which the above-described features may be understood in detail, a more particular description, briefly summarized above, may be had by reference to example aspects, some of which are illustrated in the accompanying drawings.

[0017] Figure 1 An environment for intelligent monitoring and estimation of flight crew communications according to one aspect is described.

[0018] Figure 2 A controller for intelligent monitoring and estimation of crew communications is shown according to one aspect.

[0019] Figure 3A-3C Intelligent monitoring and estimation of crew communications according to one aspect is shown.

[0020] Figure 4 is a flow chart illustrating intelligent monitoring and estimation of crew communications according to one aspect. DETAILED DESCRIPTION

[0021] Automated training systems allow users to participate in training activities without the direct involvement of a human instructor (let alone the physical presence of a human instructor). For example, the delivery of training content and the evaluation of user performance can be automatically performed by a computerized system. This provides a number of benefits, one of which is that training courses can be more easily completed by crew members (e.g., more than one person) at the same time. Extending the application of automated training systems to crew training activities can greatly improve the training of crew members by providing the ability to automatically monitor and evaluate crew communications.

[0022] One or more of the techniques discussed below improve upon these existing solutions by providing techniques for evaluating crew communications to be used in training or operational scenarios. In one aspect, an electronic system obtains a transcription of words spoken by a user (e.g., provided by a speech-to-text engine) and performs a probabilistic determination to match the spoken words to known procedures or checklists used for aircraft flight training or operations. This allows the initiation and completion of a checklist or procedure to be automatically monitored by a computer system (rather than by a human instructor).

[0023] In addition, the state of the training or operating environment is monitored by the system, and the communications made by the users about the state of the environment and the interactions with the environment are verified to ensure that they are accurate. This allows the system to build a model of the state of the environment that has been shared among multiple crew members. Comparing this shared model with the true state of the environment can reveal gaps in the users' shared knowledge of the training or operating system. The comparison of the shared model with the true state of the environment can be used for training purposes, or to raise alarms in operational scenarios.

[0024] In addition, data describing the state of the training or operating environment is analyzed by a suitable software engine to identify the current environmental scenario. For example, the system can analyze flight simulator data to determine the current flight phase (e.g., the current environmental scenario). The identification of the current environmental scenario allows the system to determine relevant known rules and generate task expectations for each user of a given scenario. For example, once the system identifies that the current flight phase is the "climb phase", the system can fill the user's (e.g., aircraft crew member) task list with associated tasks (e.g., "request flap configuration", "configure aircraft flaps", and "complete post-takeoff checklist"). The integration of these elements allows the monitoring and evaluation of user communications to be completed programmatically by a computer system. In one aspect, this increases access to training by reducing reliance on limited professional human instructor time and allows crew-based training activities to be conducted using automated training systems.

[0025] Although the examples discussed above focus on flight training, this is merely an example. As discussed further below, one or more of the techniques disclosed herein may be used to automate the assessment of crew communications during aircraft operations, to automate the assessment of crew communications during operations of other vehicles or machinery, or to automate the assessment of multi-person communications in any structured environment, including but not limited to repair, maintenance, assembly, or any combination thereof.

[0026] Figure 1 Intelligent monitoring and estimation of crew communications according to one aspect is depicted. In one aspect, one or more users 122A-N participate in training or operation of aircraft 110. As part of the training or operation, users 122A-N participate in communication 124 (e.g., verbal communication using voice). For example, users 122A-N may communicate with each other, may detail training or operation status and queries, and may participate in any other suitable communication.

[0027] The computing environment 120 includes a sensor layer 130 for capturing the communications. For example, the sensor layer 130 may include one or more audio sensors (e.g., for capturing verbal communications and other sounds), one or more visual sensors (e.g., for capturing eye tracking and gaze of a user), one or more motion sensors, and any other suitable sensors. In one aspect, the controller 140 monitors and evaluates the communications of the users (e.g., crew members) (e.g., to ensure that the communications are proceeding as expected), provides any necessary alerts, and performs any other suitable tasks. The following with respect to FIG. 3- Figure 4 Further discussion on this.

[0028] Although Figure 1 The training or operation of aircraft 110 is shown, but this is only an example. One or more of the techniques discussed herein may be applied to intelligent monitoring and estimation of communications in any suitable environment. For example, any suitable vehicle (e.g., a ship, submarine, automobile, truck, bus, train or other track-based vehicle, military or tactical vehicle, spacecraft) or any other suitable environment (e.g., a manufacturing environment or any other suitable indoor or outdoor environment) may be used.

[0029] Figure 2 A controller for intelligently monitoring and estimating communications of users (e.g., crew members) according to one aspect is shown. In one aspect, the controller 140 includes a processor 202, a memory 210, and a network component 220. The memory 210 can take the form of any non-transitory computer-readable medium. The processor 202 generally retrieves and executes programming instructions stored in the memory 210. The processor 202 represents a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a graphics processing unit (GPU) with multiple execution paths, etc.

[0030] The network component 220 includes components necessary for the controller 140 to interface with a suitable communication network. For example, the network component 220 may include wired, Wi-Fi, and / or cellular network interface components and associated software. Although the memory 210 is shown as a single entity, the memory 210 may include one or more memory devices (e.g., random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory) having memory blocks associated with physical addresses.

[0031] Memory 210 generally includes program code for performing various functions associated with the use of controller 140. Program code is generally described as various functional "applications" or "modules" within memory 210, although alternative embodiments may have different functions and / or combinations of functions. Within memory 210, assessment service 212 facilitates intelligent monitoring and evaluation of crew communications. Figure 4 This is further explained.

[0032] although Figure 2 The evaluation service 212 is depicted as being located in the memory 210, but for clarity, this representation is provided only as an illustration. More generally, the controller 140 may include one or more computing platforms (such as computer servers that may be co-located), or may form an interactively linked but distributed system (e.g., a cloud-based system (e.g., a public cloud, a private cloud, a hybrid cloud, or any other suitable cloud-based system)). Thus, the processor 202 and the memory 210 may correspond to distributed processor and memory resources within a computing environment.

[0033] Figure 3A-3C Intelligent monitoring and estimation of crew communications according to one aspect is shown. In one aspect, one or more users 301 (e.g., crew members) are connected to a common session (e.g., a training or operation monitoring session). For example, users 301 can participate in a training or operation scenario and make physical or digital inputs to an environment 302 (e.g., a training or operation environment), as well as receive visual, auditory, and tactile information from the environment 302 (e.g., a training or operation environment). In one aspect, at box 303, audio information from each user 301 generates a textual representation (e.g., a text transcript). In addition, light reflected from the eyes of user 301 can be captured by an eye tracking system 320 (e.g., an eye tracking camera system).

[0034] In one aspect, environment 302 includes systems or hardware that user 301 uses to provide input during the scene they are completing. For example, environment 302 can be a digital environment (e.g., a virtual environment) displayed on a desktop computer, laptop computer, tablet computer, mobile device, head-mounted device, or other computing device. As another example, environment 302 can be a physical environment. This can include a flight training simulator, a flight deck, an autonomous vehicle command and control station, or any other suitable physical environment. In one aspect, the physical environment can contain digital elements (e.g., a computing device). As a third example, environment 302 can be a physical environment enhanced with virtual elements displayed by a mixed reality device. These are just examples, and environment 302 can be any suitable environment.

[0035] Although FIG. 3 focuses on users (such as crew members) in a shared physical environment, this is only an example for the sake of illustration. One or more of the techniques described herein can be used for communication between different crew members (e.g., different crew members in a vehicle), or for communication between users located in different physical locations. For example, one or more of these techniques can be applied to communication between users located in different physical locations but training or operating together. This can be used to identify mismatches in communication or environmental understanding, among other improvements.

[0036] In one aspect, environment state monitor 310 queries environment 302, returning one or more variables describing the current state of the environment. In addition, environment 302 can receive alerts from alert system 312. In one aspect, these alerts are presented to user 301 through visual, auditory, or tactile information or using any other suitable technology. In addition, physical and digital interactions can be sent to data storage system 318 for long-term storage.

[0037] At block 303, an evaluation service (e.g., Figure 2 ) or any other suitable software service generates a textual representation (e.g., a text transcript). For example, the evaluation service can provide a text-based transcription of words spoken by one or more users 301. The evaluation service can also identify each user 301 in the session.

[0038] In one aspect, speech audio can be received from a given user 301 and can be sent to any or all of a speech-to-text engine 304, a speaker recognition engine 305, or a data storage system 318. For example, the speech-to-text engine 304 can return a text-based representation (e.g., a text transcript) of the spoken words present in the audio. The speaker recognition engine 305 can return the identity of the speaker who spoke the words present in the audio. In one aspect, the returned speaker identity is attached to the transcribed text as metadata. The transcribed text with metadata can then be passed to any or all of a box 306 for checklist item identification, a user task list monitor 316, and a data storage system 318.

[0039] In one aspect, speech-to-text engine 304 is responsible for providing transcription of spoken words present in the audio received from box 303. Speech-to-text engine 304 can derive auditory information from spoken words using linguistic algorithms. Speech-to-text engine 304 can then segment the audio waves associated with the spoken words into small independent files at a scale of one hundredth of a second. These small audio files can be matched with known phonemes of a given language. The identified phonemes are input into a machine learning (ML) network and matched with known words and phrases to provide a text-based transcription, which is returned to box 303. In one aspect, speech-to-text engine 304 can be any suitable speech-to-text engine (e.g., an existing third-party engine).

[0040] In one aspect, speaker recognition engine 305 is responsible for identifying the user 301 who spoke the words present in the audio provided by block 303. For example, speaker recognition engine 305 can compare the voice characteristics present in the audio file with the voice characteristics matched with known speakers. In one aspect, speaker recognition engine 305 can be any suitable speaker recognition engine (e.g., an existing third-party engine). Alternatively or additionally, the user associated with the words present in the audio can be determined by associating each audio source with a specific user 301.

[0041] At box 306, the assessment service performs checklist item identification. In one aspect, the assessment service tracks the progress of one or more users 301 through the checklist or process from the transcribed speech, thereby associating the transcription with each item of the checklist or process. For example, at box 307, the assessment service can retrieve the process (e.g., from a process database 308) and provide the retrieved process to the assessment service for checklist item identification. The assessment service can also use the transcribed text and speaker identity provided by box 303. In one aspect, the process database 308 can be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0042] In one aspect, the progress of the checklist observed through the transcribed speech is represented as a hidden Markov model, and a dynamic programming algorithm is used to probabilistically match the transcribed speech to the checklist and determine the most likely checklist item taking into account the transcribed text. One such example of a dynamic programming algorithm that can be used to determine the checklist items is the Viterbi algorithm. The checklist items represent individual steps that are required to be completed by one or more users 301 (e.g., crew members) in a checklist or process. Once a checklist item is detected from the transcribed text, the checklist item and the associated ID are sent to any one or all of the box 309 for communication verification, the user task list monitor 316, and the data storage system 318. In one aspect, if the transcribed text does not match the checklist item, the text is marked as a non-checklist communication and is sent to any one or all of the box 309 for communication verification, the user task list monitor 316, and the data storage system 318.

[0043] At block 307, the assessment service loads the process files (e.g., from process database 308) and passes them to block 306. In one aspect, process database 308 stores all processes, checklists, or both that may be performed by one or more users 301 (e.g., crew members) during a session. For example, processes and checklists may be stored as process files that contain detailed information describing the expected execution of the processes and checklists. These process files may contain information such as, but not limited to, the process or checklist title, each spoken phrase required, the user required to generate each spoken phrase, each interaction with the environment required, the user required to perform each interaction with the environment, and one or more environmental elements associated with each step. In one aspect, process database 308 receives the query from block 307 and returns the appropriate process file.

[0044] At block 309, the assessment service performs communication verification. In one aspect, the assessment service verifies the communication made by the user 301 regarding the state of the environment. For example, the assessment service can verify the communication by comparing the enunciation of the state of the environment with the true state of the environment. As an example, the assessment service can compare the spoken state of a given variable (e.g., the spoken fuel level) with the actual current state of a reference variable (e.g., the state of the fuel measured from an instrument in the environment) to determine whether the communicated state is consistent with the true state.

[0045] In one aspect, at box 309, as part of communication verification, the assessment service receives any or all of the checklist items and checklist item IDs or non-checklist communications from box 306. For each checklist item (or non-checklist communication) received, the assessment service determines the variable referenced in the speech (e.g., the speech reference variable defined in the process file sent from box 306 or 306). In one aspect, the assessment service then queries the environment state monitor 310 using the speech reference variable ID and receives the current state of the speech reference variable. The assessment service compares the current state of the reference variable with the speech reference variable and sends the result of the comparison to the data storage system 318. In one aspect, if the result of the comparison meets the predefined criteria, the assessment service can send an alarm message to the alarm system 312 to trigger an alarm to the user 301. In addition, the assessment service can send the speech reference variable to the conveyed system state engine 311.

[0046] In one aspect, the environment state monitor 310 monitors the environment 302 (e.g., a training or operating environment) and records changes in the environment. For example, when a change occurs in the environment 310, the environment state monitor 310 updates the current environment state and sends the current environment state to the context estimation engine 313 and the data storage system 318. In one aspect, the current environment state maintained by the environment state monitor 310 includes all elements in the environment and their associated states. This is just an example, and the environment state monitor 310 can be related to any suitable subset of the elements in the environment and any suitable subset of the associated states.

[0047] As part of the communication verification at box 309, the assessment service can query the environment state monitor 310 using the voice reference variable ID. In response, the environment state monitor 310 can provide the current state of the associated environment variable. In addition, the communicated system state engine 311 can query the environment state monitor 310, and in response, the environment state monitor 320 can return the current environment state. In one aspect, the environment state monitor 310 is also responsible for detecting user (e.g., crew member) input to the environment 302. In one aspect, the user task list monitor 316 receives any or all of the detected user interactions and the user ID of the user who completed the interaction as input.

[0048] In one aspect, the communicated system state engine 311 is responsible for comparing the environmental state that has been represented by the communication between the users 301 with the actual environmental state measured from the environment 302. In addition, as part of the communication verification at box 309, the assessment service can provide a voice reference variable to the communicated system state engine 311, which updates the communicated environmental state. In one aspect, the communicated system state engine 311 queries the environmental state monitor 310 at a specific frequency (e.g., a configurable set frequency or sampling rate) and receives the current environmental state. The communicated system state engine 311 compares the communicated environmental state and the actual environmental state, and if a difference threshold (e.g., a predefined difference threshold) of the environmental element is reached, the communicated system state engine 311 can send an alarm message to the alarm system 312 to trigger an alarm to the user 301 (e.g., a crew member). The actual and communicated state differences can also be sent to the data storage system 318.

[0049] In one aspect, alarm system 312 is responsible for providing alarm to user 301 (e.g., through environment 302). Alarm system 312 can deliver visual information, auditory information, tactile information or any other suitable information (e.g., delivered to user 301). In addition, many other aspects can provide alarm messages. For example, as part of the communication verification at box 309, the assessment service can provide an alarm message for the required alarm (e.g., when the voice reference variable does not match the environmental state). As another example, the communication system state engine 311 can provide an alarm (e.g., when the difference threshold between the environmental state communicated and the actual environmental state is reached), or the workload monitor 317 can provide an alarm (e.g., when the workload estimation threshold is reached), or any other suitable aspect can provide an alarm. In addition, alarms and alarm messages can also be delivered to data storage system 318.

[0050] In one aspect, the context estimation engine 313 is responsible for identifying known environmental scenarios from data describing the current environmental state. For example, the context estimation engine 313 can receive the current environmental state from the environmental state monitor 310 and can query the scene characteristics database 324. If the current environmental state matches a known scene defined in the scene characteristics database 324, the current environmental scenario is returned, and a set of expected tasks corresponding to the given scenario can be retrieved from the expected task database 314 and sent to the user task list 315. In one aspect, the scene characteristics database 324 and the expected task database 314 can be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0051] In one aspect, the expected task database 314 contains expected tasks corresponding to known environmental scenarios. When the context estimation engine 313 uses the environmental scenario for querying, the expected task database 314 returns the expected tasks associated with the specific scenario. For example, the expected task can contain information describing the task to be completed (including completion criteria).

[0052] The user task list 315 may include a list of expected tasks for each user 301 in a session (e.g., a training or operational session). For example, the expected tasks for each user 301 may be populated by the context estimation engine 313. The expected tasks included in the list may include tasks such as, but not limited to, required interactions with the environment, required communications from the user 301 (e.g., crew members), commands to be issued, or checklists and processes to be completed. The list of expected tasks may be sent to both the user task list monitor 316 and the workload monitor 317. In addition, information received from the user task list monitor 316 may cause tasks in the task list to be marked as completed, or a new task to be created and added to the user task list 315. When the user task list 315 changes, the current task list may be sent to the data storage system 318.

[0053] In one aspect, the user task list monitor 316 is responsible for monitoring the actions and communications of the user 301 within the environment 302. The user task list monitor 316 receives any one or all of the following: the transcript generated at the box 303 (e.g., representing the communication of the user 301), one or more checklist items (e.g., from the box 30), one or more checked element IDs (IEIDs) from the element identifier processor 322 (e.g., representing the visual gaze of the user 301), and one or more user interactions, user IDs, or both (UIDs) from the environment state monitor 310 (e.g., representing the interaction of the user 301 with the environment). If the information received at the user task list monitor (e.g., any one or all of the transcript, one or more checklist items, one or more IEIDs, one or more UIDs, or any other suitable information) matches the completion criteria of the expected task, the task completion flag is sent to the user task list 315 to mark the task as completed. If the received transcript does not match the task completion criteria, the user task list monitor 316 queries the task assignment database 325 to determine whether the information represents the task being issued. If an assigned task is received from the task assignment database 325, the task is created and added to the user task list 315. In one aspect, the task assignment database 325 can be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0054] In one aspect, workload monitor 317 is responsible for evaluating the workload of each user during a session. Workload monitor 317 receives the current task list from user task list 315 and determines each workload by evaluating information (e.g., the number and type of tasks included in each task list). The workload estimate can then be sent to data storage system 318. If a workload threshold is reached, an alert message can be sent to alert system 312 to display an alert to user 301.

[0055] The data storage system 318 may log the relevant data (e.g., all relevant data or a portion of the relevant data) to long-term storage in the session database 319. The data storage system 318 receives any or all of the following: interaction data from the environment 302, audio and transcript from the box 303, checklist status and checklist step ID from the box 306, communication verification results from the box 309, current environment status from the environment status monitor 310, current task list from the user task list 315, workload estimates from the workload monitor 317, and actual and communicated state differences from the communicated system state engine 311. In one aspect, the input (e.g., all input or a portion of the input) is sent to the session database 319 for long-term storage.

[0056] In one aspect, session database 319 is a long-term storage device for relevant data (e.g., all relevant data or a portion of relevant data) captured during a session. Session database 319 receives data from data storage system 318 and submits the data to long-term storage. In one aspect, session database 319 can be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0057] In one aspect, eye tracking camera system 320 is responsible for generating an image of the eyes of user 301 (e.g., crew member) to determine the user's gaze point. For example, eye tracking camera system 320 may include an infrared light source that illuminates the eyes of user 301 and a camera that captures the location of glints generated from the reflection of the infrared light on the cornea of ​​user 301. The image of the eye including the corneal reflection is sent to gaze tracking processor 321.

[0058] In one aspect, gaze tracking processor 321 is responsible for identifying the gaze point of user 301 (e.g., crew member). Gaze tracking processor 321 receives an image of the user's eye from eye tracking camera system 320 and applies image processing techniques to the image to enable detection of the position of the pupil and the position where infrared light is reflected from the user's cornea. The gaze point of user 301 is calculated based on any or all of the following: the known geometry of the camera and infrared light source in eye tracking camera system 320, the eye of user 301, and the reflection of light on the user's cornea. The gaze point can be represented in 2D or 3D Cartesian coordinates depending on environment 302. In one aspect, the gaze point is sent to element identifier processor 322.

[0059] In one aspect, element identifier processor 322 is responsible for identifying whether user 301 (e.g., crew member) is visually looking at an element within environment 302. Element identifier processor 322 receives Cartesian coordinates describing the user's gaze point and queries element location database 323, which stores known locations of elements within environment 302. If the Cartesian coordinates are associated with a known location of an element, the ID of the element being checked is returned and sent to user task list monitor 316. In one aspect, element location database 323 can be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0060] Element location database 323 may store known locations of elements (e.g., all elements or a portion of elements) within environment 302. Each element in the database is associated with a set of Cartesian coordinates that describe the location of the element in environment 302. Element location database 323 receives a set of Cartesian coordinates from element identifier processor 322, and if the set of Cartesian coordinates is associated with a known element in the database, an element ID of the inspected element is returned.

[0061] The scene characteristics database 324 can store known scenes described by the state of elements within the environment. For example, a 55,000 kg aircraft, with landing gear down, flaps set to 5, airspeed of 150 kts, and a positive climb rate, will be associated with the climb phase of flight. When the context estimation engine 313 queries the scene characteristics database 324 using the current environment state, if an associated environment scene is found, the environment scene is returned. In one aspect, the scene characteristics database 324 can be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, a field repository, or any other suitable electronic repository.

[0062] The task assignment database 325 can store known task assignment phrases. The task assignment phrases are associated with the expected tasks and completion criteria that can populate the user task list 315. When queried by the user task list monitor 316, if the assigned tasks match the query, the task assignment database 325 returns the assigned tasks. In one aspect, the task assignment database 325 can be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0063] Figure 4 4 is a flow chart illustrating intelligent monitoring and estimation of crew communications according to one aspect. At block 402, an assessment service (e.g., Figure 2) captures the communication. For example, as described above with respect to Figure 3A-3C As discussed, the assessment service may capture verbal communications between users (eg, crew members) in a training or operational environment of an aircraft. This is merely an example, and the assessment service may capture any suitable communications.

[0064] At block 404, the assessment service converts the communication into text. Figure 3A-3C As discussed above at 304 shown in , the assessment service may use a speech-to-text engine to generate text from the spoken communication.

[0065] It should be understood that speech-to-text engines have an expected error rate in transcription (e.g., 10%). In addition, spoken communications often include additional sounds or words that do not reflect the meaning of the communication (e.g., repeated words, verbal filler phrases, and other sounds or words). It is straightforward to compare the transcribed communication to the expected checklist (e.g., as described above with respect to Figure 3A-3C 306 as discussed above) may result in significant errors.

[0066] In one aspect, this can be improved by using probabilistic matching from the transcribed communication to the expected checklist items. Thus, at block 406, the assessment service performs probabilistic matching on the text. For example, a hidden Markov model (e.g., as described above with respect to Figure 3A-3C As discussed above, the transcribed communication of the word "maxi" may be used to determine to what extent a given transcribed word matches an expected word and, based on the context, determine a likely match. For example, assume that the checklist contains the word "taxi." A transcribed communication of the word "taxi" would clearly match the expected word in the checklist. In contrast, a transcribed communication of the word "fish" would be very unlikely to match. However, a transcribed communication of the word "maxi" may be a probabilistic match to the expected word "taxi." Similarly, it may be expected that users (e.g., crew members) will continue through the checklist in sequence (e.g., on average), and therefore the next expected communication may be identified by following the checklist in sequence. As discussed above with respect to Figure 3A-3C As discussed, hidden Markov models and dynamic programming algorithms (eg, Viterbi algorithm) may be used for probabilistic matching.

[0067] At block 408, the evaluation service generates a communication status. Figure 3A-3CAs discussed, in one aspect, the assessment service compares a communication state of the environment (e.g., generated based on communications from a user) to an actual state of the environment (e.g., based on aircraft instruments or other direct sensors). In one aspect, establishing a communication state can be challenging. For example, the techniques described above with respect to block 406 can be used to probabilistically match communications of checklist words and phrases. However, a user (e.g., a crew member) can also be expected to communicate variables (e.g., unknown values). For example, a user can communicate the airspeed or fuel level of the aircraft. The airspeed or fuel level value is not known in advance and, therefore, may not be directly matched to an expected checklist value.

[0068] In one aspect, one or more blank variables can be used as part of probabilistic matching. For example, a given checklist process can be defined as a blank variable that includes an expected time. As an example, a checklist process can expect the user to specify "Heading [Heading_Value] (heading [heading value])". Heading_Value is unknown in advance, but is expected to be a number in a specific format (e.g., degrees and direction). The checklist process can include a blank variable that expects the numeric value of Heading_value. This can be used for probabilistic matching to improve transcription errors (e.g., non-numeric transcription values ​​may be probabilistically matched with numeric values). For example, if a numeric blank variable is included, the transcription value can be corrected to match the expected number (e.g., based on the inclusion of a numeric blank variable, the transcription value of "done fate nero" can be corrected to "oneeight zero").

[0069] Additionally, a blank variable may define a specific format for a numeric value. A heading may be expected to be specified with individual digits (e.g., "one eight zero (1-8-0)"), while a fuel level may be expected to be specified as a complete number (e.g., "one eighty (1-80)" or "one hundred eighty (180)" instead of "one eight zero (1-8-0)"). Numbers are just one example, and any suitable blank variable type and format may be used. For example, a selection of states may be defined, alphanumeric values ​​may be expected, Boolean values ​​(e.g., yes or no) may be expected, or any other suitable blank variable type and format may be expected.

[0070] At block 410, the assessment service compares the environment state to the communication state. Figure 3A-3CAs discussed, the assessment service can identify a difference between a communicated environmental state (e.g., a spoken phrase "fuel level 6000 kg") and an actual environmental state (e.g., an actual fuel level of 600 kg). In the event that the difference exceeds a threshold, the assessment service can generate an appropriate alert or notification for a user (e.g., a crew member).

[0071] At block 412, the assessment service creates a user task list. For example, as described above with respect to Figure 3A-3C As discussed in 315 and 316 shown in , the user task list can contain a list of expected tasks for the user in a given training or operation session. For example, the expected tasks for each user can be populated. The expected tasks can include: required interactions with the environment, required communications from the user, commands to be issued, or checklists and processes to be completed.

[0072] In the current disclosure, reference is made to various aspects. However, it should be understood that the present disclosure is not limited to the specifically described aspects. On the contrary, any combination of the following features and elements (whether or not related to different aspects) is contemplated as realizing and practicing the teachings provided herein. In addition, when describing the elements of various aspects in the form of "at least one of A and B", it should be understood that it is conceivable to include only element A, only element B, and each aspect including element A and B, respectively. In addition, although some aspects can achieve advantages that are superior to other possible solutions and / or superior to the prior art, whether a specific advantage is achieved by a given aspect does not limit the present disclosure. Therefore, the aspects, features, aspects, and advantages disclosed herein are merely illustrative and are not considered to be elements or limitations of the appended claims unless they are clearly described in detail in the claims. Similarly, reference to "the present invention" should not be interpreted as a summary of any inventive subject matter disclosed herein, and should not be considered to be elements or limitations of the appended claims unless they are clearly described in detail in the claims.

[0073] As will be appreciated by those skilled in the art, the various aspects described herein may be embodied as systems, methods, or computer program products. Thus, the various aspects may take the form of entirely hardware aspects, entirely software aspects (including firmware, resident software, microcode, etc.), or aspects combining software and hardware aspects, all of which may be generally referred to herein as "circuits," "modules," or "systems." Additionally, the various aspects described herein may take the form of a computer program product embodied in one or more computer-readable storage media having computer-readable program code embodied thereon.

[0074] Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0075] The computer program code for performing the operations of various aspects of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Java, Smalltalk, C++, etc.) and traditional procedural programming languages ​​(e.g., "C" programming language or similar programming languages). The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0076] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of various aspects of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device / means (means) for implementing the function / action specified in the frame of the flowchart and / or block diagram.

[0077] These computer program instructions may also be stored in a computer-readable medium, which may direct a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in each box in the flowchart and / or block diagram.

[0078] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device provide a process for implementing the functions / actions specified in the boxes in the flowchart and / or block diagram.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of systems, methods and computer program products according to different aspects of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, fragment or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions shown in the box may not occur in the order shown in the accompanying drawings. For example, in fact, two boxes shown in succession may be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order or out of order, depending on the functions involved. It should also be noted that each box of the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that performs a specified function or action.

[0080] While the foregoing is directed to various aspects of the present disclosure, other and further aspects of the disclosure may be envisaged without departing from the basic scope thereof, and the scope of which is to be determined by the appended claims.

Claims

1. A computer-implemented method comprising: receiving a textual representation (404) based on a verbal communication (402) from one or more users (301) related to a physical environment; generating a communicated state of the physical environment based on the textual representation (408); determining an actual state of the physical environment based on one or more measured characteristics of the environment (410); and Based on comparing the communicated state of the physical environment with the actual state of the physical environment (309), an alert (312) is automatically provided to at least one of the one or more users (301).

2. The computer-implemented method of claim 1, wherein the communicated state of the physical environment includes one or more communicated characteristics (110) of an aircraft, and Wherein the actual state of the physical environment includes one or more measured characteristics of the aircraft (320).

3. The computer-implemented method of claim 1, Wherein generating the communicated state of the physical environment based on the textual representation comprises: Identifying one or more expected checklist items or processes for the aircraft (306); as well as The communicated state of the physical environment is determined by probabilistically matching a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures (406).

4. The computer-implemented method of claim 3 , wherein identifying one or more expected checklist items or procedures for the aircraft comprises: The one or more expected checklist items or procedures are retrieved from an electronic repository (308).

5. The computer-implemented method of claim 3, wherein the one or more expected checklist items or procedures for the aircraft include a blank variable associated with a particular variable type (408), and Wherein the probabilistic match is based on determining that at least a portion of the first phrase matches the particular variable type (408).

6. The computer-implemented method of claim 5, wherein the particular variable type comprises a numeric variable type (408).

7. The computer-implemented method of claim 3, wherein the probability matching uses a Hidden Markov Model (306).

8. The computer-implemented method of claim 1 , further comprising: generating a user task list, the user task list comprising a plurality of tasks for the one or more users in the physical environment (315, 412); as well as A determination is made based on the textual representation that the one or more users have completed at least a portion of the user task list (316).

9. The computer-implemented method of claim 8, further comprising: A determination is made based on the user task list that a workload threshold of a user among the one or more users has been reached, and an alert is provided to the user based on the determination (317).

10. The computer-implemented method of claim 1 , wherein comparing the communicated state of the physical environment to the actual state of the physical environment comprises: It is determined that a difference between a first state value in the communicated state and a second state value in the actual state exceeds a threshold value (311).

11. A non-transitory computer readable medium containing computer program code, which when executed by operation of one or more computer processors performs operations comprising: receiving a textual representation (404) based on a verbal communication (402) from one or more users (301) related to a physical environment; generating a communicated state of the physical environment based on the textual representation (408); determining an actual state of the physical environment based on one or more measured characteristics of the environment (410); and Based on comparing the communicated state of the physical environment with the actual state of the physical environment (309), an alert (312) is automatically provided to at least one of the one or more users (301).

12. The non-transitory computer readable medium of claim 11, wherein the communicated state of the physical environment includes one or more communicated characteristics (110) of an aircraft, and Wherein the actual state of the physical environment includes one or more measured characteristics of the aircraft (320).

13. The non-transitory computer readable medium of claim 11, Wherein generating the communicated state of the physical environment based on the textual representation comprises: Identifying one or more expected checklist items or processes for the aircraft (306); as well as The communicated state of the physical environment is determined by probabilistically matching a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures (406).

14. The non-transitory computer readable medium of claim 13, wherein the one or more expected checklist items or procedures for the aircraft include a blank variable associated with a particular variable type (408), and Wherein the probabilistic match is based on determining that at least a portion of the first phrase matches the particular variable type (408).

15. The non-transitory computer readable medium of claim 11, further comprising: generating a user task list (315, 412), the user task list comprising a plurality of tasks for the one or more users in the physical environment; as well as A determination is made based on the textual representation that the one or more users have completed at least a portion of the user task list (316).

16. A system comprising: Computer processors; and a memory having stored thereon instructions that, when executed on the computer processor, perform operations including: receiving a textual representation (404) based on a verbal communication (402) from one or more users (301) related to a physical environment; generating a communicated state of the physical environment based on the textual representation (408); determining an actual state of the physical environment based on one or more measured characteristics of the environment (410); and Based on comparing the communicated state of the physical environment with the actual state of the physical environment (309), an alert (312) is automatically provided to at least one of the one or more users (301).

17. The system according to claim 16, wherein the communicated state of the physical environment includes one or more communicated characteristics (110) of an aircraft, and Wherein the actual state of the physical environment includes one or more measured characteristics of the aircraft (320).

18. The system according to claim 16, Wherein generating the communicated state of the physical environment based on the textual representation comprises: Identifying one or more expected checklist items or processes for the aircraft (306); as well as The communicated state of the physical environment is determined by probabilistically matching a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures (406).

19. The system according to claim 18, wherein the one or more expected checklist items or procedures for the aircraft include a blank variable associated with a particular variable type (408), and Wherein the probabilistic match is based on determining that at least a portion of the first phrase matches the particular variable type (408).

20. The system of claim 16, further comprising: generating a user task list (314, 412), the user task list comprising a plurality of tasks for the one or more users in the physical environment; as well as A determination is made based on the textual representation that the one or more users have completed at least a portion of the user task list (316).

Images