Intelligent monitoring and evaluation of crew communication

Abstract

To provide techniques for monitoring communication in a physical environment.SOLUTION: The techniques include receiving a textual representation based on verbal communication from one or more users relating to a physical environment. The techniques further include generating a communicated state of the physical environment based on the textual representation, and determining an actual state of the physical environment based on one or more measured characteristics of the physical environment. The techniques further include automatically providing an alert to at least one of the one or more users, based on comparing the communicated state of the physical environment with the actual state of the physical environment.SELECTED DRAWING: None

Description

[Technical field]

[0001] Aspects of the present disclosure relate to intelligent monitoring of communications within an environment. [Background technology]

[0002]

[0002] Training and operation of aircraft systems is critical. Currently, monitoring and evaluation of student communication skills during aviation training activities is typically performed by human instructors. Evaluation of flight crew communications during training sessions conducted on a flight training device (e.g., a flat panel trainer, a full flight simulator, or any other suitable flight training device) is performed by a human instructor facilitating the training session. Such human-in-the-loop activities introduce subjectivity into the evaluation process and can lead to misclassification of communication performance and other errors. Furthermore, access to expert human instructors is limited, limiting the amount of training that can be performed.

[0003]

[0003] Automated training systems are designed to increase the opportunities for students to engage in training activities by reducing reliance on expert human instructors. However, lacking the ability to provide automated evaluation of flight crew communications, the training opportunities these systems currently provide students are limited to single-user training activities. Because much of the aviation training curriculum is designed for flight crew activities, the application of automated training systems to training is currently limited. Summary of the Invention

[0004]

[0004] Aspects include a computer-implemented method. The method includes receiving a textual representation based on a verbal communication from one or more users associated with a physical environment. The method further includes generating a communicated state of the physical environment based on the textual representation. The method further includes determining an actual state of the physical environment based on one or more measured characteristics of the environment. The method further includes automatically providing an alert to at least one of the one or more users based on comparing the communicated state of the physical environment to the actual state of the physical environment.

[0005]

[0005] Aspects further include a non-transitory computer readable medium including computer program code that, when executed by operation of one or more computer processors, performs a plurality of operations. The plurality of operations includes receiving a textual representation based on a verbal communication from one or more users associated with a physical environment. The plurality of operations further includes generating a communicated state of the physical environment based on the textual representation. The plurality of operations further includes determining an actual state of the physical environment based on one or more measured characteristics of the environment. The plurality of operations further includes automatically providing an alert to at least one of the one or more users based on comparing the communicated state of the physical environment to the actual state of the physical environment.

[0006]

[0006] Aspects further include a system. The system includes a computer processor and a memory having instructions stored therein. The instructions, when executed on the computer processor, perform a plurality of operations. The plurality of operations includes receiving a textual representation based on a verbal communication from one or more users associated with the physical environment. The plurality of operations further includes generating a communicated state of the physical environment based on the textual representation. The plurality of operations further includes determining an actual state of the physical environment based on one or more measured characteristics of the environment. The plurality of operations further includes automatically providing an alert to at least one of the one or more users based on comparing the communicated state of the physical environment to the actual state of the physical environment.

[0007]

[0007] In some further aspects, in combination with any embodiment above or below, the communicated state of the physical environment includes one or more communicated characteristics of the aircraft, and the actual state of the physical environment includes one or more measured characteristics of the aircraft.

[0008]

[0008] In some further aspects, in combination with any embodiment above or below, generating a communicated state of the physical environment based on the textual representation includes identifying one or more expected checklist items or procedures for the aircraft, and determining the communicated state of the physical environment by probabilistically matching a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures.

[0009]

[0009] In some further aspects, in combination with any embodiment above or below, identifying one or more expected checklist items or procedures for the aircraft includes retrieving the one or more expected checklist items or procedures from an electronic repository.

[0010]

[0010] In some further aspects, in combination with any embodiment above or below, one or more anticipated checklist items or procedures for the aircraft include a blank variable associated with a particular variable type, and the probabilistic matching is based on determining that at least a portion of the first phrase matches the particular variable type.

[0011]

[0011] In some further aspects, in combination with any embodiment above or below, the specific variable type includes a numeric variable type.

[0012]

[0012] In some further aspects, in combination with any embodiment above or below, the probabilistic matching uses a hidden Markov model.

[0013]

[0013] In some further aspects, in combination with any embodiment above or below, the method or operation includes generating a user task list including a plurality of tasks for one or more users in the physical environment, and determining that one or more users have completed at least a portion of the user task list based on the textual representation.

[0014]

[0014] In some further aspects, in combination with any embodiment above or below, the method or operation includes determining that a workload threshold has been reached for a user of one or more users based on the user work list, and based on so determining, providing a warning to the user.

[0015]

[0015] In some further aspects, in combination with any embodiment above or below, comparing the communicated state of the physical environment with the actual state of the physical environment includes determining that a difference between a first state value in the communicated state and a second state value in the actual state exceeds a threshold value.

[0016]

[0016] So that the above-mentioned features of the present disclosure can be understood in detail, a more detailed description of the present disclosure than that briefly summarized above can be made by reference to several exemplary aspects, some of which are illustrated in the accompanying drawings. [Brief description of the drawings]

[0017] [Figure 1]

[0017] According to one aspect, an environment for intelligent monitoring and evaluation of crew communications is shown. [Diagram 2]

[0018] 1 illustrates a controller for intelligent monitoring and evaluation of crew communications, according to one aspect. [Figure 3A]

[0019] 3A-3C illustrate intelligent monitoring and evaluation of crew communications according to one embodiment. [Figure 3B] 3A-3C illustrate intelligent monitoring and evaluation of crew communications according to one embodiment. [Figure 3C] 3A-3C illustrate intelligent monitoring and evaluation of crew communications according to one embodiment. [Figure 4]

[0020] 1 is a flow chart illustrating intelligent monitoring and evaluation of crew communications according to one aspect. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018]

[0021] Automated training systems allow users to engage in training activities without the direct involvement (much less physical presence) of a human instructor. For example, the delivery of training content and the evaluation of user performance may be performed automatically by a computerized system. Among many benefits, this may make training curricula much easier to manage by a crew (e.g., two or more individuals) simultaneously. By providing the ability to automatically monitor and evaluate crew communications, training for crew members may be significantly improved, broadening the application of automated training systems to crew training activities.

[0019]

[0022] One or more techniques described below improve upon these prior solutions by employing techniques for the evaluation of flight crew communications in training or operational scenarios. In one aspect, an electronic system takes a transcript of the words spoken by a user (e.g., provided by a speech-to-text engine) and makes probabilistic decisions to match the spoken words to known procedures or checklists for flight training or operation of the aircraft. This allows the initiation and completion of checklists or procedures to be monitored automatically by a computer system, rather than by a human instructor.

[0020]

[0023] Additionally, the state of the training or operational environment is monitored by the system and communications made by users regarding the state of the environment and interactions with the environment are verified to ensure they are accurate. This allows the system to build a model of the environmental state shared among multiple crew members. Comparison of this shared model to the true environmental state may reveal gaps in the users' shared knowledge of the training or operational system. Comparison of the shared model to the true environmental state may be used for training purposes or to generate warnings in operational scenarios.

[0021]

[0024] Additionally, data describing the state of the training or operational environment is analyzed by a suitable software engine to identify a current environmental scenario. For example, the system may analyze flight simulator data to determine the current phase of flight (e.g., current environmental scenario). Identification of the current environmental scenario allows the system to determine relevant known rules and generate task predictions for each user for the given scenario. For example, if the system identifies that the current phase of flight is the "climb phase," the system may populate the task list of the user (e.g., flight crew) with the relevant tasks (e.g., "request flaps configuration," "configure aircraft flaps," and "complete post-takeoff checklist"). The integration of these elements allows for programmatic monitoring and evaluation of user communications by the computer system. In one aspect, increasing access to training opportunities by reducing reliance on limited expert human instructor time and enabling automated training systems to conduct crew-based training activities.

[0022]

[0025] Although the embodiments described above focus on flight training, this is by way of example only. As described further below, one or more techniques disclosed herein may be used for automated assessment of crew communications during operation of an aircraft, including, but not limited to, repair, maintenance, assembly, or any combination thereof, automated assessment of crew communications during operation of other vehicles or machinery, or automated assessment of multi-person communications in any structured environment.

[0023]

[0026] 1 illustrates intelligent monitoring and evaluation of flight crew communications, according to one aspect. In one aspect, one or more users 122A-N are engaged in training or operation of an aircraft 110. As part of the training or operation, the users 122A-N engage in communications 124 (e.g., oral communication using speech). For example, the users 122A-N may communicate with one another, may state the status of the training or operation, may provide inquiries, or may engage in any other suitable communication.

[0024]

[0027] Computing environment 120 includes a sensor layer 130 for capturing this communication. For example, sensor layer 130 may include one or more audio sensors (e.g., for capturing verbal communication and other sounds), one or more visual sensors (e.g., for capturing a user's eye movements and gaze), one or more motion sensors, and any other suitable sensors. In one aspect, controller 140 evaluates the user(s) (e.g., flight crew) communication (e.g., to ensure that the communication is progressing as expected), provides any necessary alerts, and performs any other suitable actions. This is described further below with respect to FIGS. 3-4.

[0025]

[0028] 1 illustrates training or operation of an aircraft 110, but this is by way of example only. One or more techniques described herein may be applied to intelligent monitoring and evaluation of communications in any suitable environment. For example, any suitable vehicle (e.g., a ship, submarine, automobile, truck, bus, train or other rail-based vehicle, military or tactical vehicle, spacecraft) or any other suitable environment (e.g., a manufacturing environment, or any other suitable indoor or outdoor environment) may be used.

[0026]

[0029] 2 illustrates a controller for intelligent monitoring and evaluation of user (e.g., flight crew) communications, according to one aspect. In one aspect, controller 140 includes a processor 202, a memory 210, and a network component 220. Memory 210 may take the form of any non-transitory computer-readable medium. Processor 202 generally reads and executes programming instructions stored in memory 210. Processor 202 is typically a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a graphics processing unit (CPU) with multiple execution paths, and the like.

[0027]

[0030] Network components 220 include components necessary for controller 140 to interact with an appropriate communications network. For example, network components 220 may include wired, WiFi, and / or cellular network interface components and associated software. Although memory 210 is illustrated as a single entity, memory 210 may include one or more memory devices having blocks of memory associated with physical addresses, such as random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory.

[0028]

[0031] Memory 210 generally includes program code for performing various functions associated with the use of controller 140. The program code is generally described as various functional "applications" or "modules" within memory 210, although alternative implementations may have different functionality and / or combinations of functionality. Within memory 210, evaluation service 212 facilitates intelligent monitoring and evaluation of crew communications, as further illustrated below with respect to Figures 3-4.

[0029]

[0032] 2 shows the assessment service 212 located in memory 210, that representation is provided merely as an example for clarity. More generally, the controller 140 may include one or more computing platforms, such as, for example, computer servers, which may be co-located to form an interactively linked but distributed system, such as a cloud-based system (e.g., a public cloud, a private cloud, a hybrid cloud, or any other suitable cloud-based system). As a result, the processor 202 and the memory 210 may correspond to distributed processor and memory resources within a computing environment.

[0030]

[0033] 3A-3C illustrate intelligent monitoring and evaluation of crew communications, according to one embodiment. In one embodiment, one or more users 301 (e.g., crew members) are connected to a common session (e.g., a training or operational monitoring session). For example, the users 301 are engaged in a training or operational scenario and can provide physical or digital inputs into and receive visual, auditory, and tactile information from an environment 302 (e.g., a training or operational environment). In one embodiment, audio information from each user 301 generates a text representation (e.g., a text transcript) in block 303. Additionally, light reflected from the eyes of the users 301 can be captured by an eye-tracking system 320 (e.g., an eye-tracking camera system).

[0031]

[0034] In one aspect, the environment 302 includes systems or hardware with which the user 301 provides input in the course of the scenario they are completing. For example, the environment 302 can be a digital environment (e.g., a virtual environment displayed on a desktop, laptop, tablet, mobile device, headset, or other computing device). As another example, the environment 302 can be a physical environment. This can include a flight training simulator, a cockpit, an autonomous vehicle command and control station, or any other suitable physical environment. In one aspect, the physical environment can incorporate a digital environment such as a computing device. As a third example, the environment 302 can be a physical environment augmented with virtual elements displayed by a mixed reality device. These are merely examples, and the environment 302 can be any suitable environment.

[0032]

[0035] While FIG. 3 focuses on users (e.g., crew members) in a shared physical environment for illustrative purposes, this is by way of example only. One or more of the techniques described herein may be used for communication between different crew members (e.g., different crew members in a vehicle) or for communication between users located in different physical locations. For example, one or more of these techniques may be applied to communication between users who train or operate together while being in different physical locations. This may be used to identify mismatches in communication or environmental understanding, among other improvements.

[0033]

[0036] In one aspect, an environment state monitor 310 queries the environment 302, which returns one or more variables describing the current state of the environment. Additionally, the environment 302 may receive alerts from an alert system 312. In one aspect, these alerts are presented to the user 301 through visual, auditory, or tactile information, or using any other suitable technique. Additionally, physical and digital interactions may be transmitted to a data storage system 318 for long-term storage.

[0034]

[0037] In block 303, a rating service (e.g., rating service 212 shown in FIG. 2), or any other suitable software service, generates a text representation (e.g., a text transcript). For example, the rating service may provide a text-based transcription of words spoken by one or more users 301. The rating service may also identify each user 301 in the session.

[0035]

[0038] In one aspect, audio may be received from a given user 301 and sent to any or all of the speech to text engine 304, the speaker identification engine 305, or the data storage system 318. For example, the speech to text engine 304 may return a text-based representation (e.g., a text transcript) of the spoken words present in the audio. The speaker identification engine 305 may return the identity of the speaker who spoke the words present in the audio. In one aspect, the returned speaker identity is attached to the transcribed text as metadata. The transcribed text with metadata may then be passed to any or all of the block for checklist item recognition 306, the user worklist monitor 316, and the data storage system 318.

[0036]

[0039] In one aspect, the speech-to-text engine 304 is responsible for providing a transcription of the spoken words present in the audio received from the block 303. The speech-to-text engine 304 may use linguistic algorithms to derive auditory information from the spoken words. The speech-to-text engine 304 may then split the audio waves for the spoken words into small individual files of 1 / 100ths of a second. These small audio files may be matched against known phonemes of a given language. The identified phonemes are input into a machine learning (ML) network and matched against known words and phrases to provide a text-based transcription, which is returned to the block 303. In one aspect, the speech-to-text engine 304 may be any suitable speech-to-text engine (e.g., an existing third-party engine).

[0037]

[0040] In one aspect, the speaker identification engine 305 is responsible for identifying the user 301 who spoke the words present in the audio provided by block 303. For example, the speaker identification engine 305 may compare voice characteristics present in the audio file with voice characteristics that match known speakers. In one aspect, the speaker identification engine 305 may be any suitable speaker identification engine (e.g., an existing third party engine). Alternatively or additionally, the user associated with the words present in the audio may be determined by associating each audio source with a particular user 301.

[0038]

[0041] In block 306, the assessment service performs checklist item recognition. In one aspect, the assessment service tracks one or more users 301's progress through the checklist or procedure from the transcribed text and associates the transcription with each item in the checklist or procedure. For example, in block 307, the assessment service may retrieve the procedure (e.g., from the procedure database 308) and provide the retrieved procedure to the assessment service for checklist item recognition. The assessment service may further use the transcribed text and speaker identities provided by block 303. In one aspect, the procedure database 308 may be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0039]

[0042] In one aspect, the checklist progression observed by the transcribed audio is represented as a hidden Markov model, and a dynamic programming algorithm is used to probabilistically match the transcribed conversation to the checklist and determine the most likely checklist items based on the transcribed text. One such example of a dynamic programming algorithm that can be used to determine checklist items is the Viterbi algorithm. Checklist items represent individual steps in a checklist or procedure that one or more users 301 (e.g., flight crew) need to complete. When a checklist item is detected in the transcribed text, the checklist item and associated ID are sent to any or all of the block 309 for communication validation, the user worklist monitor 316, and the data storage system 318. In one aspect, if the transcribed text does not match a checklist item, the text is labeled as a non-checklist communication and sent to any or all of the block 309 for communication validation, the user worklist monitor 316, and the data storage system 318.

[0040]

[0043] In block 307, the evaluation service loads procedure files (e.g., from procedure database 308) and passes them to block 306. In one aspect, procedure database 308 stores all procedures, checklists, or both that may be executed by one or more users 301 (e.g., flight crew) during a session. For example, procedures and checklists may be stored as procedure files. Procedure files contain detailed information that describes the intended execution of the procedures and checklists. These procedure files may include information such as, but not limited to, the title of the procedure or checklist, each spoken phrase required, the user required to generate each spoken phrase, each environmental interaction required, the user required to execute each environmental interaction, and one or more environmental elements associated with each step. In one aspect, procedure database 308 receives a query from block 307 and returns the appropriate procedure file.

[0041]

[0044] In block 309, the rating service performs a communication validation. In one aspect, a rating service validates communications made by user 301 regarding the environmental state. For example, the rating service may validate the communications by comparing the environmental state statements to the true state of the environment. As one example, the rating service may compare the spoken state of a given variable (e.g., spoken fuel level) to the actual current state of the mentioned variable (e.g., fuel state measured from an instrument in the environment) to determine whether the communicated state matches the true state.

[0042]

[0045] In one aspect, in block 309, as part of the communication validation, the evaluation service receives the checklist items and checklist item IDs or any or all of the non-checklist items from block 306. For each received checklist item (or non-checklist communication), the evaluation service determines the variables mentioned in the conversation (e.g., the variables mentioned by the conversation defined in the procedure file sent from block 306). In one aspect, the evaluation service then queries the environmental state monitor 310 with the variable IDs mentioned by the conversation and receives the current state of the variables mentioned by the conversation. The evaluation service compares the current state of the variables mentioned with the variables mentioned by the conversation and sends the results of the comparison to the data storage system 318. In one aspect, if the results of the comparison meet the predefined criteria, the evaluation service may send a warning message to the alert system 312 to trigger an alert to the user 301. Additionally, the evaluation service may send the variables mentioned by the conversation to the communicated system state engine 311.

[0043]

[0046] In one aspect, environmental state monitor 310 monitors environment 302 (e.g., a training or operational environment) and records changes in the environment. For example, when a change occurs in environment 310, environmental state monitor 310 updates the current environmental state and transmits the current environmental state to context evaluation engine 313 and data storage system 318. In one aspect, the current environmental state maintained by environmental state monitor 310 includes all elements in the environment and their associated states. This is by way of example only, and environmental state monitor 310 may be associated with any suitable subset of elements in the environment and any suitable subset of their associated states.

[0044]

[0047] As part of the communication validation in block 309, the rating service can query the environmental state monitor 310 with the variable IDs that the conversation refers to. In response, the environmental state monitor 310 can provide the current state of the associated environmental variables. Additionally, the communicated system state engine 311 can query the environmental state monitor 310, which in turn can return the current environmental state. In one aspect, the environmental state monitor 310 is also responsible for detecting user (e.g., crew) inputs to the environment 302. In one aspect, a user worklist monitor 316 receives as input any or all of the detected user interactions and the user ID of the user who completed the interaction.

[0045]

[0048] In one aspect, the communicated system state engine 311 is responsible for comparing the environmental state expressed through the communication between the users 301 with the actual environmental state measured from the environment 302. Additionally, as part of the communication validation in block 309, the evaluation service may provide the variables to which the conversation refers to the communicated system state engine 311. The communicated system state engine 311 updates the communicated environmental state. In one aspect, the communicated system state engine 311 queries the environmental state monitor 310 at a specific frequency (e.g., a configurable set frequency or sampling rate) to receive the current environmental state. The communicated system state engine 311 compares the communicated environmental state with the actual environmental state, and if a difference threshold (e.g., a predefined difference threshold) is reached for the environmental state, the communicated system state engine 311 may send a warning message to the warning system 312 to trigger a warning to the user 301 (e.g., the flight crew). The difference between the actual state and the communicated state may also be transmitted to the data storage system 318 .

[0046]

[0049] In one aspect, the alert system 312 is responsible for providing an alert to the user 301 (e.g., via the environment 302). The alert system 312 may deliver visual, audio, tactile, or any other suitable information (e.g., to the user 301). Furthermore, many other aspects may provide the alert message. For example, as part of the communication validation in block 309, the evaluation service may provide an alert message for a requested alert (e.g., when a variable to which the conversation refers does not match the environmental state). As another example, the communicated system state engine 311 may provide an alert (e.g., when a difference threshold between the communicated environmental state and the actual environmental state is reached), or the workload monitor 317 may provide an alert (e.g., when a workload evaluation threshold is reached), or any other suitable aspect may provide an alert. Furthermore, the alerts and alert messages may also be passed to the data storage system 318.

[0047]

[0050] In one aspect, the context evaluation engine 313 is responsible for identifying known environmental scenarios from data describing the current environmental conditions. For example, the context evaluation engine 313 can receive the current environmental conditions from the environmental condition monitor 310 and query the scenario characteristics database 324. If the current environmental conditions match a known scenario defined in the scenario characteristics database 324, the current environmental scenario can be returned and a set of expected tasks corresponding to the given scenario can be retrieved from the expected tasks database 314 and sent to the user tasks list 315. In one aspect, the scenario characteristics database 324 and the expected tasks database 314 can each be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0048]

[0051] In one aspect, expected tasks database 314 includes expected tasks corresponding to known environmental scenarios. When queried by context evaluation engine 313 with an environmental scenario, expected tasks database 314 returns expected tasks associated with a particular scenario. For example, expected tasks may include information describing tasks to be completed, including completion criteria.

[0049]

[0052] The user work list 315 may include a list of expected tasks for each user 301 in a session (e.g., a training or operational session). For example, the expected tasks for each user 301 may be input by the context evaluation engine 313. The expected tasks included in this list may include, but are not limited to, interactions required by the environment, communications required from the user 301 (e.g., crew), commands to be issued, or checklists and procedures to be completed. The list of expected tasks may be sent to both the user work list monitor 316 and the workload monitor 317. Additionally, information received from the user work list monitor 316 may cause tasks in the work list to be marked as completed or new tasks to be created or new tasks to be added to the user work list 315. When the user work list 315 changes, the current work list may be sent to the data storage system 318.

[0050]

[0053] In one aspect, the user worklist monitor 316 is responsible for monitoring the activities and communications of the user 301 within the environment 302. The user worklist monitor 316 receives any or all of the following: the transcribed text generated in block 303 (e.g., representing the communication of the user 301), one or more checklist items (e.g., from block 306), one or more inspected element IDs (IEIDs) (e.g., representing the visual fixations of the user 301) from the element identifier processor 322, and one or more user interactions, user IDs, or both (UIDs) (e.g., representing the interactions of the user 301 with the environment) from the environment state monitor 310. If the information received at the user worklist monitor (e.g., the transcribed text, one or more checklist items, one or more IEIDs, one or more UIDs, or any other suitable information) meets completion criteria for the expected task, a task completion flag is sent to the user worklist 315 to mark the task as completed. If the received transcribed text does not meet the completion criteria, the user work list monitor 316 queries the work assignment database 325 to determine if the information represents an outstanding task. If assigned work is received from the work assignment database 325, the work is generated and added to the user work list 315. In one aspect, the work assignment database 325 may be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0051]

[0054] In one aspect, the workload monitor 317 is responsible for assessing the workload of each user during a session. The workload monitor 317 receives current work lists from the user work lists 315 and determines each workload through evaluation of information (e.g., the amount and type of work contained within each work list). The workload assessment may then be sent to the data storage system 318. If a workload threshold is reached, a warning message may be sent to the warning system 312 to display a warning to the user 301.

[0052]

[0055] The data storage system 318 can record the relevant data (e.g., all of the relevant data or a portion of the relevant data) for long-term storage in a session database 319. The data storage system 318 receives any or all of the interaction data from the environment 302, the audio and transcribed text from block 303, the checklist state and checklist step ID from block 306, the communication validation provided from block 309, the current environment state from the environment state monitor 310, the current work list from the user work list 315, the workload assessment from the workload monitor 317, and the difference between the actual state and the communicated state from the communicated system state engine 311. In one aspect, the input (e.g., all of the input or a portion of the input) is sent to the session database 319 for long-term storage.

[0053]

[0056] In one aspect, the session database 319 is a long-term store for relevant data captured during a session (e.g., all of the relevant data or a portion of the relevant data). The session database 319 receives data from the data storage system 318 and routes the data to long-term storage. In one aspect, the session database 319 may be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0054]

[0057] In one embodiment, the eye tracking camera system 320 is responsible for generating an image of the eye of the user 301 (e.g., a flight attendant) to determine the user's point of gaze. For example, the eye tracking camera system 320 may include an infrared light source and a camera. The infrared light source illuminates the eye of the user 301. The camera captures the location of the glint generated from the reflection of the infrared light on the cornea of ​​the user 301. The image of the eye, including the corneal reflection, is transmitted to the gaze tracking processor 321.

[0055]

[0058] In one embodiment, the gaze tracking processor 321 is responsible for identifying the gaze point of the user 301 (e.g., a flight attendant). The gaze tracking processor 321 receives an image of the user's eye from the eye tracking camera system 320 and applies image processing techniques to the image to enable detection of both the position of the pupil and the reflection of infrared light from the user's cornea. The gaze point of the user 301 is calculated from the known geometry of the camera and infrared light source in the eye tracking camera system 320, the user's 301 eye, and / or the reflection of light on the user's cornea. The gaze point may be expressed in 2D or 3D Cartesian coordinates depending on the environment 302. In one embodiment, the gaze point is sent to the element identifier processor 322.

[0056]

[0059] In one aspect, the element identifier processor 322 is responsible for identifying whether the user 301 (e.g., a flight attendant) has visually fixated an element within the environment 302. The element identifier processor 322 receives Cartesian coordinates describing the user's point of gaze and queries an element location database 323 that stores known locations of elements within the environment 302. If the Cartesian coordinates are associated with the known locations of the elements, the examined element ID is returned and transmitted to the user worklist monitor 316. In one aspect, the element location database 323 may be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0057]

[0060] The element location database 323 may store known locations of elements (e.g., all elements or a portion of multiple elements) within the environment 302. Each element in the database is associated with a set of Cartesian coordinates that describe the element's location within the environment 302. The element location database 323 receives the set of Cartesian coordinates from the element identifier processor 322, and if the set of Cartesian coordinates is associated with a known element in the database, an element ID for the examined element is returned.

[0058]

[0061] The scenario characteristics database 324 may store known scenarios described by the state of elements in the environment. For example, a 55000 kg aircraft with landing gear down, flaps set at 5, airspeed 150 kts, and positive climb rate may be associated with the climb phase of flight. When the scenario characteristics database 324 is queried by the context evaluation engine 313 with the current environmental state, if an associated environmental scenario is found, the environmental scenario is returned. In an aspect, the scenario characteristics database 324 may be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0059]

[0062] The work assignment database 325 may store known work assignment phrases. Work assignment phrases are associated with expected work and completion criteria that may populate the user work list 315. When queried by the user work list monitor 316, the work assignment database 325 returns the assigned work if the assigned work matches the query. In one aspect, the work assignment database 325 may be any suitable electronic repository, including an electronic database (e.g., a relational database or a graph database), a cloud repository, an on-site repository, or any other suitable electronic repository.

[0060]

[0063] 4 is a flowchart 400 illustrating intelligent monitoring and evaluation of crew communications, according to one aspect. At block 402, an evaluation service (e.g., evaluation service 212 shown in FIG. 2 or any other suitable software service) captures communications. For example, as described above in connection with FIGS. 3A-3C, the evaluation service may capture verbal communications between users (e.g., crew members) within a training or operational environment for an aircraft. This is by way of example only, and the evaluation service may capture any suitable communications.

[0061]

[0064] The rating service converts the communication to text at block 404. For example, as described above in connection with 304 shown in Figures 3A-3C, the rating service can use a speech-to-text engine to generate text from the spoken communication.

[0062]

[0065] It is understood that speech-to-text engines have an expected error rate in transcriptions (e.g., 10%). Furthermore, spoken communication often includes additional sounds or words (e.g., repeated words, verbal filler phrases, and other sounds or words) that do not reflect the meaning of the communication. Directly comparing a transcribed communication to an expected checklist (e.g., as part of the checklist item recognition described above in connection with block 306 shown in Figures 3A-3C) can result in significant errors.

[0063]

[0066] In one aspect, this may be improved by using probabilistic matching from the transcribed communication to the expected checklist items. Thus, in block 406, the evaluation service performs probabilistic matching on the text. For example, a hidden Markov model (e.g., as described above in connection with FIGS. 3A-3C) may be used to determine both how likely a given transcribed word is to match an expected word, and how likely a match is based on the context. For example, assume a checklist includes the word "taxi." A transcribed communication of the word "taxi" may unambiguously match an expected word in the checklist. In contrast, a transcribed communication of the word "fish" is highly unlikely to match. However, a transcribed communication of the word "maxie" may probabilistically match the expected word "taxi." Similarly, since a user (e.g., a flight attendant) is expected to work through the checklist in a sequential order (e.g., on average), the next expected communication may be identified by working through the checklist in a sequential order. As described above in connection with FIGS. 3A-3C, hidden Markov models and dynamic programming algorithms (eg, the Viterbi algorithm) may be used for probabilistic matching.

[0064]

[0067] At block 408, the evaluation service generates a communication state. As described above in connection with FIGS. 3A-3C, in one aspect, the evaluation service compares the communication state of the environment (e.g., generated based on a communication from a user) to the actual state of the environment (e.g., based on aircraft instruments or other direct sensors). In one aspect, the communication state may be difficult to construct. For example, a communication of checklist words or phrases may be probabilistically matched using the techniques described above in connection with block 406. However, a user (e.g., a flight crew member) may also be expected to communicate variables (e.g., unknown values). For example, a user may communicate the airspeed or fuel level of an aircraft. This airspeed or fuel level value is not known in advance and therefore cannot be directly matched to expected checklist values.

[0065]

[0068] In one aspect, one or more blank variables may be used as part of the probabilistic matching. For example, a given checklist step may be defined to include a blank variable for expected time. As an example, a checklist step may expect the user to repeat "Heading [heading_value]". Heading_value is not known a priori, but is expected to be a numeric value in a particular format (e.g., degrees or direction). The checklist step may include a blank variable that predicts a numeric value for the heading_value. This may be used in the probabilistic matching to improve transcription errors (e.g., a transcribed value that is not numeric may be probabilistically matched with a numeric value). For example, if a numeric blank variable is included, the transcribed value may be corrected to match the expected numeric value (e.g., a transcribed value of "done fate nero" may be corrected to "one eight zero" based on the inclusion of a numeric blank variable).

[0066]

[0069] Additionally, the blank variable may specify a particular format for the numeric value. Heading may be expected to be expressed in individual numeric values ​​(e.g., "one eight zero"), while fuel level may be expected to be expressed in full numeric values ​​(e.g., "one eighty" or "one hundred eighty" rather than "one eight zero"). Numeric values ​​are merely examples, and any suitable blank variable type and format may be used. For example, state options may be defined, alphanumeric values ​​may be expected, Boolean values ​​(e.g., yes or no) may be expected, or any other suitable blank variable type and format may be defined.

[0067]

[0070] At block 410, the rating service compares the environmental state to the communicated state. For example, as described above in connection with Figures 3A-3C, the rating service may identify a discrepancy between the communicated environmental state (e.g., the spoken phrase "fuel level 6000 kg") and the actual environmental state (e.g., actual fuel level 600 kg). If the discrepancy exceeds a threshold, the rating service may generate an appropriate alert or notification to a user (e.g., flight crew).

[0068]

[0071] In block 412, the assessment service builds a user task list. For example, as described above in connection with 315 and 316 shown in Figures 3A-3C, the user task list may include a list of expected tasks for users in a given training or operational session. For example, expected tasks for each user may be entered. The expected tasks may include interactions required by the environment, communications required from the user, commands to be issued, or checklists or procedures to be completed.

[0069]

[0072] In the present disclosure, various embodiments are referenced. However, it should be understood that the disclosure is not limited to the specific described embodiments. Instead, any combination of the following features and elements, whether related to various embodiments or not, is contemplated for implementing and practicing the teachings provided herein. Furthermore, when an element of an embodiment is described in the form of "at least one of A and B," it should be understood that an embodiment including element A only, element B only, and elements A and B are each contemplated. Furthermore, while some embodiments may realize other potential solutions and / or advantages over the prior art, whether or not a particular advantage is realized by a given embodiment does not limit the disclosure. Thus, the embodiments, features, and advantages disclosed herein are merely exemplary and should not be considered elements of or limiting the scope of the appended claims unless expressly recited in the claim(s). Similarly, references to "the present invention" should not be construed as a generalization of all inventive subject matter disclosed herein, and should not be considered an element of or as limiting the scope of any accompanying claim(s) unless expressly recited in the claim(s).

[0070]

[0073] As will be appreciated by one of ordinary skill in the art, aspects described herein may be embodied as a system, method, and / or computer program product. Accordingly, aspects may take the form of entirely hardware aspects, entirely software aspects (including firmware, resident software, microcode, etc.), or aspects combining software and hardware aspects, all of which may be broadly referred to herein as "circuits," "modules," or "systems." Additionally, aspects described herein may take the form of a computer program product embodied in one or more computer-readable storage medium(s) having computer-readable program code embodied therein.

[0071]

[0074] The program code embodied in the computer readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0072]

[0075] Computer program code for carrying out operations of aspects of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar. The program code may run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0073]

[0076] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to aspects of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose or special purpose computer, or other programmable data processing device, to produce a machine. These instructions, executed via the processor of the computer or other programmable data processing device, thereby create means for performing the function / acts identified in the block(s) of the flowcharts and / or block diagrams.

[0074]

[0077] The computer program instructions may also be stored on a computer-readable medium that may direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, whereby the instructions stored in the computer-readable medium produce an article of manufacture. The instructions include instructions that implement the function / act identified in the flowchart and / or block diagram block(s).

[0075]

[0078] The computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable data processing apparatus, or other device provide a process for implementing the functions / acts identified in the flowchart and / or block diagram block(s).

[0076]

[0079] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the disclosure. As such, each block in the flowcharts and block diagrams may represent a module, a segment, or a portion of code, including one or more executable instructions for implementing a particular logical function(s). In some alternative implementations, the functions shown in the blocks may not be performed in the order described in the figures. For example, in practice, two blocks shown in succession may be executed substantially simultaneously, or the blocks may be executed in reverse or out of order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a special-purpose hardware-based system that performs a particular function or function, or by a combination of special-purpose hardware and computer instructions.

[0077]

[0080] While the foregoing is directed to aspects of the disclosure, other and further aspects of the disclosure may be devised without departing from the basic scope thereof, the scope of which is defined by the following claims.

Claims

1. receiving (402) a textual representation (404) based on verbal communication from one or more users (301) associated with a physical environment; generating a communicated state of the physical environment based on the textual representation (408); determining (410) an actual state of the physical environment based on one or more measured characteristics of the environment; and and automatically providing an alert (312) to at least one of the one or more users (301) based on comparing (309) the communicated state of the physical environment to the actual state of the physical environment.

2. the communicated state of the physical environment includes one or more communicated characteristics of the aircraft (110); The computer-implemented method of claim 1 , wherein the actual state of the physical environment comprises one or more measured characteristics (320) of the aircraft.

3. Generating the communicated state of the physical environment based on the textual representation includes: Identifying (306) one or more anticipated checklist items or procedures for the aircraft; and 2. The computer-implemented method of claim 1, comprising determining the communicated state of the physical environment by probabilistically matching (406) a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures.

4. Identifying one or more anticipated checklist items or procedures for the aircraft includes:

4. The computer-implemented method of claim 3, comprising retrieving the one or more anticipated checklist items or procedures from an electronic repository (308).

5. the one or more anticipated checklist items or procedures for the aircraft include blank variables associated with a particular variable type (408); 4. The computer-implemented method of claim 3, wherein the probabilistic matching is based on determining that at least a portion of the first phrase matches the particular variable type (408).

6. The computer-implemented method of claim 5 , wherein the particular variable type comprises a numeric variable type (408).

7. 4. The computer-implemented method of claim 3, wherein the probabilistic matching uses a hidden Markov model (306).

8. generating (315, 412) a user task list including a plurality of tasks for the one or more users within the physical environment; and 2. The computer-implemented method of claim 1, further comprising determining (316) that the one or more users have completed at least a portion of the user work list based on the textual representation.

9. 10. The computer-implemented method of claim 8, further comprising: determining that a workload threshold has been reached for a user of the one or more users based on the user work list; and providing (317) a warning to the user based on the determining.

10. Comparing the communicated state of the physical environment with the actual state of the physical environment comprises:

2. The computer-implemented method of claim 1, further comprising determining (311) that a difference between a first state value in the communicated state and a second state value in the actual state exceeds a threshold.

11. A non-transitory computer readable medium containing computer program code, the computer program code, when executed by operation of one or more computer processors, performing a plurality of operations, the plurality of operations including: Receiving (402) a textual representation (404) based on verbal communication from one or more users (301) associated with a physical environment; generating a communicated state of the physical environment based on the textual representation (408); determining (410) an actual state of the physical environment based on one or more measured characteristics of the environment; and and automatically providing an alert (312) to at least one of the one or more users (301) based on comparing (309) the communicated state of the physical environment to the actual state of the physical environment.

12. the communicated state of the physical environment includes one or more communicated characteristics of the aircraft (110); The non-transitory computer-readable medium of claim 11 , wherein the actual state of the physical environment includes one or more measured characteristics (320) of the aircraft.

13. Generating the communicated state of the physical environment based on the textual representation includes: Identifying (306) one or more anticipated checklist items or procedures for the aircraft; and 12. The computer-implemented method of claim 11, comprising determining the communicated state of the physical environment by probabilistically matching (406) a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures.

14. the one or more anticipated checklist items or procedures for the aircraft include blank variables associated with a particular variable type (408); 14. The non-transitory computer readable medium of claim 13, wherein the probabilistic matching is based on determining that at least a portion of the first phrase matches the particular variable type (408).

15. generating (315, 412) a user task list including a plurality of tasks for the one or more users within the physical environment; and The non-transitory computer-readable medium of claim 11 , further comprising determining (316) that the one or more users have completed at least a portion of the user task list based on the textual representation.

16. Computer processors, 1. A system comprising: a memory having instructions stored thereon, the instructions, when executed by the computer processor, performing a plurality of operations, the plurality of operations including: Receiving (402) a textual representation (404) based on verbal communication from one or more users (301) associated with a physical environment; generating a communicated state of the physical environment based on the textual representation (408); determining (410) an actual state of the physical environment based on one or more measured characteristics of the environment; and and automatically providing an alert (312) to at least one of the one or more users (301) based on comparing (309) the communicated state of the physical environment to the actual state of the physical environment.

17. the communicated state of the physical environment includes one or more communicated characteristics of the aircraft (110); The system of claim 16 , wherein the actual state of the physical environment includes one or more measured characteristics (320) of the aircraft.

18. Generating the communicated state of the physical environment based on the textual representation includes: Identifying (306) one or more anticipated checklist items or procedures for the aircraft; and 17. The system of claim 16, further comprising determining the communicated state of the physical environment by probabilistically matching (406) a first phrase in the textual representation with a second phrase in the one or more expected checklist items or procedures.

19. the one or more anticipated checklist items or procedures for the aircraft include blank variables associated with a particular variable type (408); 20. The system of claim 18, wherein the probabilistic matching is based on determining that at least a portion of the first phrase matches the particular variable type (408).

20. generating (315, 412) a user task list including a plurality of tasks for the one or more users within the physical environment; and 17. The system of claim 16, further comprising determining (316) that the one or more users have completed at least a portion of the user work list based on the textual representation.

Images