Human-machine interface and related interaction method

Abstract

The invention relates to a human-machine interface and related interaction method. The human-machine interface (HMI), the method of interacting with the HMI and the corresponding computer program product facilitate the interaction of a user with the HMI. In the method, a face interacting with the HMI is detected and also a gesture made by the person relative to the HMI is detected. The method analyzes the information about the detected face to determine whether the person is wearing personal protective equipment. In case it is determined that the person is not wearing personal protective equipment, the method suspends the response to the gesture made by the person. However, in case it is determined that the person is wearing personal protective equipment, the method analyzes the gesture made by the person relative to the HMI and converts the gesture into a corresponding command to a system associated with the HMI.

Description

Technical Field

[0001] In an exemplary embodiment, a human-machine interface, a method for interacting with the human-machine interface, and a computer program product configured to interact with the human-machine interface are provided, wherein the human-machine interface and the associated method and computer program product are configured to determine whether a person attempting to interact with the human-machine interface is wearing personal protective equipment before responding to a gesture made by a person. Background Technology

[0002] Human-machine interfaces (HMIs) are widely used in a variety of applications to receive user input and control the systems associated with the HMI accordingly. For example, HMIs can be used in conjunction with a wide range of industrial, commercial, manufacturing, and transportation applications. For instance, an aircraft may include one or more HMIs to receive input from pilots or other crew members and / or passengers.

[0003] HMIs typically include a user interface, such as a touchscreen. To provide input via the HMI to control the associated system, the user touches the user interface, and the HMI translates the touch detected by the touchscreen into corresponding input for controlling the system associated with the HMI. This reliance on the touchscreen to receive user input can limit the input options available to the user, such as restricting input options to those that can be presented on the touchscreen at any time for the user to select. Furthermore, the need for physical contact with the touchscreen to provide input via the HMI can expose users to various health risks. For example, various particles, such as those carrying bacteria or viruses, can accumulate on the touchscreen through transfer from previous users, or by previously airborne particles remaining on the touchscreen. To reduce the potential risks to users of touchscreen-equipped HMIs, the touchscreen can be cleaned repeatedly; however, such cleaning increases the time and cost associated with using the HMI. Additionally, cleaning the touchscreen may damage the screen and / or reduce its accuracy or performance. Summary of the Invention

[0004] According to exemplary embodiments, a human-machine interface (HMI), a method for interacting with the HMI, and corresponding computer program products are provided to facilitate user interaction with the HMI. In the exemplary embodiments, the HMI, method, and associated computer program products are configured to detect gestures and act upon them, rather than requiring the user to touch the HMI's user interface. By relying on gestures, the HMI can be configured to receive a large number of different types of input from the user, thereby increasing the flexibility and level of detail in controlling the systems associated with the HMI. Furthermore, the HMI, method, and computer program products of the exemplary embodiments are configured to detect whether a user is wearing personal protective equipment (PPE) when the user attempts to interact with the HMI. Therefore, the HMI, method, and associated computer program products of this exemplary embodiment can modulate any response of the HMI (including any systems associated with the HMI) to input provided by a user wearing PPE, thereby reducing the health risks associated with the user and other subsequent users using the HMI. In addition, the HMI, method, and computer program products of the exemplary embodiments can facilitate contact tracing by logging information identifying the user of the HMI and whether the user is wearing PPE.

[0005] In an exemplary embodiment, a method is provided for interacting with a human-machine interface (HMI) of a system. The method includes detecting a face attempting to interact with the HMI and detecting a gesture made by the person relative to the HMI. The method also includes analyzing information about the detected face to determine whether the person is wearing personal protective equipment, such as a face mask. If it is determined that the person is not wearing personal protective equipment, the method suspends response to the gesture made by the person relative to the HMI. However, if it is determined that the person is wearing personal protective equipment, the method analyzes the gesture made by the person relative to the HMI and translates the gesture into a corresponding command to the system associated with the HMI.

[0006] If it has been previously determined that the person is not wearing personal protective equipment (PPE), the method in the exemplary embodiment further includes subsequently detecting a face attempting to interact with the HMI and analyzing the subsequently detected information about the face to determine whether the person is now wearing PPE. If it is determined based on the subsequent detection of the face that the person is now wearing PPE, the method in this exemplary embodiment terminates the pause in the response to the gesture and analyzes the gesture made by the person relative to the HMI and translates the gesture into a corresponding command to the system associated with the HMI.

[0007] A method in an exemplary embodiment analyzes information about a face by estimating a point cloud representing the face to determine whether the person is wearing personal protective equipment (PPE). In an exemplary embodiment, the method detects a face attempting to interact with the HMI by detecting a face based on signals received by a first sensor and detecting a pose made by the person relative to the HMI based on signals received by a second sensor different from the first sensor. In this exemplary embodiment, the first sensor may be a near-infrared (NIR) sensor, and the second sensor may be an electro-optical (EO) sensor. In an exemplary embodiment, the method utilizes one or more convolutional neural networks to detect faces attempting to interact with the HMI, analyzes information about the detected faces to determine whether the person is wearing PPE, analyzes the pose made by the person relative to the HMI, and translates the pose into the corresponding command.

[0008] In another exemplary embodiment, a human-machine interface (HMI) is provided, including processing circuitry configured to detect a face attempting to interact with the HMI and to detect a gesture made by the person relative to the HMI. The processing circuitry is also configured to analyze information about the detected face to determine whether the person is wearing personal protective equipment, such as a face shield. If it is determined that the person is not wearing personal protective equipment, the processing circuitry is configured to send a response to the gesture made by the person relative to the HMI. However, if it is determined that the person is wearing personal protective equipment, the processing circuitry is configured to analyze the gesture made by the person relative to the HMI and translate the gesture into a corresponding command for the system associated with the HMI.

[0009] If it has been previously determined that the person is not wearing personal protective equipment (PPE), the processing circuitry in the exemplary embodiment is further configured to subsequently detect a face attempting to interact with the HMI and analyze the subsequently detected information about the face to determine whether the person is now wearing the PPE. If it is determined based on the subsequent detection of the face that the person is now wearing PPE, the processing circuitry in this exemplary embodiment is also configured to terminate the pause in the response to the gesture, analyze the gesture made by the person relative to the HMI, and translate the gesture into a corresponding command for the system associated with the HMI.

[0010] In an exemplary embodiment, the processing circuitry is configured to analyze information about a face by estimating point cloud data representing a face to determine whether the person is wearing personal protective equipment (PPE). The HMI in the exemplary embodiment also includes a first sensor and a second sensor, different from the first sensor. The first sensor is configured to provide the processing circuitry with a signal indicating that a face attempting to interact with the HMI has been detected, and the second sensor is configured to provide the processing circuitry with a signal indicating that a gesture made by the person relative to the HMI has been detected. The first sensor may be a near-infrared (NIR) sensor, and the second sensor may be an electro-optical (EO) sensor. In an exemplary embodiment, the processing circuitry includes one or more convolutional neural networks configured to detect a face attempting to interact with the HMI, analyze information about the detected face to determine whether the person is wearing PPE, analyze the gesture made by the person relative to the HMI, and translate the gesture into a corresponding command.

[0011] In another exemplary embodiment, a computer program product is provided, comprising at least one non-transitory computer-readable storage medium storing computer-executable program code instructions, wherein the computer-executable program code instructions include program code instructions for detecting a face attempting to interact with an HMI and program code instructions for detecting a gesture made by the person relative to the HMI. The computer-executable program code instructions also include program code instructions for analyzing information about the detected face to determine whether the person is wearing personal protective equipment (such as a face shield). The computer-executable program code instructions further include program code instructions for suspending the response to the gesture made by the person relative to the HMI if it is determined that the person is not wearing personal protective equipment. Furthermore, the computer-executable program code instructions include program code instructions for analyzing the gesture made by the person relative to the HMI and for translating the gesture into a corresponding command for a system associated with the HMI if it is determined that the person is wearing personal protective equipment.

[0012] If it has been previously determined that the person is not wearing personal protective equipment (PPE), the computer-executable program code instructions further include program code instructions for subsequently detecting a face attempting to interact with the HMI, and program code instructions for analyzing subsequently detected information about the face to determine whether the person is now wearing the PPE. If it is determined based on subsequent face detection that the person is now wearing PPE, the computer-executable program code instructions of this exemplary embodiment also include program code instructions for terminating a pause in the response to a gesture, and program code instructions for analyzing the gesture made by the person relative to the HMI and translating the gesture into a corresponding command for the system associated with the HMI.

[0013] In one exemplary embodiment, the program code instructions for analyzing information about a face include program code instructions for estimating point cloud data representing a face to determine whether the person is wearing personal protective equipment. In another exemplary embodiment, the program code instructions for detecting a face attempting to interact with the HMI and for detecting a pose made by the person relative to the HMI include program code instructions for detecting the face based on signals received by a first sensor and program code instructions for detecting a pose made by the person relative to the HMI based on signals received by a second sensor different from the first sensor. In this exemplary embodiment, the first sensor may be a near-infrared (NIR) sensor, and the second sensor may be an electro-optical (EO) sensor. In yet another exemplary embodiment, the program code instructions include one or more convolutional neural networks configured to detect a face attempting to interact with the HMI, analyze information about the detected face to determine whether the person is wearing personal protective equipment, analyze a pose made by the person relative to the HMI, and translate the pose into a corresponding command. Attached Figure Description

[0014] Some exemplary embodiments of this disclosure have been described in general terms. Reference will be made below to the accompanying drawings, which are not necessarily drawn to scale, and in which:

[0015] Figure 1 The diagram illustrates an HMI configured to respond to gestures according to an exemplary implementation.

[0016] Figure 2 It is a block diagram of an HMI that includes processing circuitry that can be specifically configured according to an exemplary implementation.

[0017] Figure 3 This is an illustration based on an exemplary implementation, such as by... Figure 2 A flowchart of the operations performed by the computing device in the process; and

[0018] Figure 4 This is a flowchart depicting an interaction with an HMI according to an exemplary implementation. Detailed Implementation

[0019] This disclosure will now be described more fully below with reference to the accompanying drawings, which show some, but not all, aspects. In fact, this disclosure may be embodied in many different forms and should not be construed as limited to the aspects set forth herein. Rather, these aspects are provided so that this disclosure will meet the requirements of applicable law. The same reference numerals refer to the same elements throughout.

[0020] According to an exemplary embodiment, a human-machine interface (HMI), a method for interacting with the HMI, and a computer program product configured to interact with the HMI are provided. The HMI includes a user interface that receives user input, which is detected and then translated into corresponding commands to guide the operation of a system associated with the HMI. The HMI can be associated with any of a wide range of systems and used in any of a wide range of different industries and applications. For example, the HMI can be used in conjunction with systems used in the business, industrial, retail, manufacturing, and transportation industries.

[0021] For example, in Figure 1 The text describes, by way of example and not limitation, the user interface 10 of an HMI used in the transportation industry, and more specifically, the user interface associated with an aircraft cabin display during flight. Figure 1 As shown, the user interface includes a first area 12 where information content, such as that for passengers on board the aircraft, is presented. In the illustrated embodiment, a representation of the aircraft's flight path relative to highlighted terrain is depicted. However, in addition to... Figure 1 The flight path information shown is either outside or replaced by Figure 1 The flight path information shown can be used to provide other types of information to the user in the first area of ​​the user interface.

[0022] Figure 1 The user interface 10 also includes several other areas, including area 14 which provides a menu of functions that can be executed in response to user input. Furthermore, the user interface of this exemplary embodiment includes area 16, which is configured to provide notifications and / or various instructions to the user, such as instructions to keep passengers seated and fasten their seatbelts. The user interface of this exemplary embodiment also includes area 18, which provides information about the personal protective equipment (such as a face mask) worn by the user. Figure 1 The user interface provided here is provided as an example of one type of HMI user interface that may be provided according to an exemplary implementation, but the HMI user interface may be configured in a variety of other ways and may not need to include different areas, or if areas are included, any number or configuration of areas may be included.

[0023] Figure 2A block diagram of an HMI 20 according to an exemplary embodiment is depicted herein. The HMI can be embodied by any of a variety of computing devices, such as servers, computer workstations, networks of distributed computing devices, personal computers, tablet computers, etc. Therefore, the HMI does not require a specific hardware design, and any of a variety of computing devices can be configured to operate as described herein. However, regardless of the type of computing device embodying the HMI, the illustrated embodiment of the HMI includes processing circuitry 22, storage device 24, user interface 26, and one or more sensors, associated with or otherwise communicating with them.

[0024] Processing circuitry 22 may be embodied in various means, including one or more microprocessors, one or more coprocessors, one or more multi-core processors, one or more controllers, one or more computers, various other processing elements, including integrated circuits, such as ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), or some combination thereof. In some exemplary embodiments, the processing circuitry is configured to execute instructions stored in storage device 24 or otherwise accessible to the processing circuitry. These instructions, when executed by the processing circuitry, can cause HMI 20 to perform one or more functions described herein. Thus, HMI may include an entity capable of performing operations according to embodiments of this disclosure when appropriately configured. Thus, for example, when the processing circuitry is embodied in an ASIC, FPGA, etc., the processing circuitry and the corresponding HMI may include hardware specifically configured to perform one or more operations described herein. Alternatively, as another example, when the processing circuitry is embodied in an instruction executor, such as one that may be stored in storage device, instructions may specifically configure the processing circuitry and, consequently, the computing device to perform one or more algorithms and operations described herein.

[0025] Storage device 24 may include, for example, non-volatile memory. Storage devices may include, for example, hard disks, random access memory, cache memory, flash memory, optical discs (e.g., compressed optical disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM), etc.), circuitry configured to store information, or some combination thereof. In this respect, storage devices may include any non-transitory computer-readable storage medium. According to exemplary embodiments of this disclosure, storage devices may be configured to store information, data, application programs, instructions, etc., enabling HMI 20, such as processing circuitry 22, to perform various functions. For example, storage devices may be configured to store program instructions executed by processing circuitry.

[0026] User interface 26 can communicate with processing circuitry 22 and storage device 24 to receive user input instructions, such as gestures performed by the user, and / or provide auditory, visual, mechanical, or other outputs to the user. Therefore, the user interface may include, for example, a display for providing visual and auditory outputs to the user and one or more speakers.

[0027] While HMI 20 may include various sensors to detect the user's face and the user's posture relative to the HMI, the exemplary embodiment of the HMI includes a first sensor 28 and a second sensor 30. The first and second sensors are of different types. In one embodiment, the first sensor is a near-infrared (NIR) sensor and the second sensor is an electro-optical (EO) sensor. At this point, Figure 1 The HMI user interface 10 depicted also includes a first sensor and a second sensor, each comprising a NIR sensor and an EO sensor, respectively. In some embodiments, the HMI may define virtual privacy partitions that restrict the respective fields of view of the first and second sensors. In this regard, the first and second sensors are configured and / or the HMI is configured to analyze signals provided by the first and second sensors such that the respective fields of view of the first and second sensors are restricted in such a way that the person interacting with the HMI is detected, rather than a person in an adjacent seat.

[0028] Now for reference Figure 3 , depicting, for example, by Figure 2 The operations performed by HMI 20. Now refer to Figure 3 In box 40, the HMI, such as the first sensor 28, such as an NIR sensor, is configured to detect a face attempting to interact with the HMI. Additionally, the HMI, such as the second sensor 30, such as an EO sensor, is configured to detect the person's pose relative to the HMI. The detection of the face attempting to interact with the HMI and the detection of the person's pose relative to the HMI can be performed simultaneously or within a predefined time window to increase the probability that the person whose face is detected and who is attempting to interact with the HMI is the same person as the person whose pose relative to the HMI is also detected.

[0029] In one implementation, the first sensor 28 and the second sensor 30 of the HMI 20 can be configured to continuously monitor their respective fields of view to detect the face of a person attempting to interact with it and the presence of one or more gestures performed by that person relative to the HMI. Optionally, the HMI can be configured to be triggered, for example, by detecting a predefined gesture of the user (e.g., waving), and then the HMI begins monitoring the face of the person attempting to interact with it and providing another gesture intended to guide the operation of the system associated with the HMI. In this exemplary implementation, the second sensor of the HMI, such as an EO sensor, can be continuously or repeatedly activated to detect the predefined trigger gesture. However, the first sensor, such as a NIR sensor, can remain inactive until the trigger gesture is detected, after which the first sensor is also activated to detect the face, and then the person makes a gesture to command the system associated with the HMI.

[0030] This discussion of hand gestures performed by a person interacting with the HMI 20 is by way of example, not limitation. In some implementations, a person may perform gestures in different ways, such as by using their head, for example, nodding, turning right, left, tilting their head forward, etc. For example, a person interacting with the HMI may indicate that they do not wish to use gestures, or the person may have a disability that limits their ability to perform gestures, causing the person to then perform gestures by moving or positioning their head. Therefore, the reference here to detected and analyzed gestures also applies to gestures performed by this person, such as by their head, in different ways.

[0031] HMI 20, such as processing circuitry 22, is also configured to analyze information about a face, for example, detected by the first sensor 28, in order to determine whether the person is wearing personal protective equipment. See also Figure 3 Box 42. The HMI, such as processing circuitry, can be configured to determine whether the person is wearing any of a variety of different types of personal protective equipment (PPE), including, for example, a face shield, a protective mask, etc. In some embodiments, the type of PPE being analyzed depends on the role or job performed by the person. For example, if the person is a surgeon, the PPE being analyzed includes a hood and a face shield. Alternatively, if the person is a passenger on an airplane, train, or bus, the PPE being analyzed may be a face shield that does not require a hood.

[0032] Processing circuitry 22 can be configured to analyze information about a person's face to determine whether the person is wearing personal protective equipment (PPE) in various ways. For example, processing circuitry can be configured to perform one or more image analysis and / or image recognition techniques to detect whether the person is wearing PPE. As described below regarding the analysis of signals provided by the second sensor 30 to identify performed gestures, the processing circuitry in an exemplary embodiment may include or embody one or more convolutional neural networks (CNNs) trained to detect faces and determine whether the person is wearing PPE (such as a face shield).

[0033] In another exemplary embodiment, a first sensor 28, such as an NIR sensor, is configured to capture point cloud data representing a face attempting to interact with the HMI 20. In this exemplary embodiment, the HMI, such as processing circuitry 22, is configured to, for example, estimate the point cloud data representing the face in real time to determine whether the person is wearing personal protective equipment (PPE). The point cloud data provides key points of the face in three-dimensional (3D) space, from which eyes, ears, nose, mouth, chin, and other unique features can be identified. Depending on the type of PPE worn, some of these key points may either not be detected, such as the mouth of a user covered by a mask, or have a low probability of detection, such as when the mouth's position is estimated based on standard distances between the mouth and other facial features(s), thus allowing determination of whether the person is wearing PPE.

[0034] If it is determined that the person is not wearing personal protective equipment, the HMI 20 (e.g., processing circuitry 22) is configured to suspend execution of responses to gestures made by the person relative to the HMI and detected, for example, by the second sensor 30. See also Figure 3 Box 44. By pausing the response to the gesture, not performing a response, and the HMI (e.g., processing circuitry) waiting until the person is wearing personal protective equipment (PPE), it then performs a response to the detected gesture. For example, in one embodiment, the HMI (e.g., processing circuitry) is also configured to subsequently detect a face attempting to interact with the HMI, in cases where it has been previously determined that the person is not wearing PPE. See box 46. At this point, the subsequent detection is performed after a person attempting to interact with the HMI but not wearing PPE has been previously detected. In some embodiments, the HMI (e.g., processing circuitry) is configured to require that subsequent detection of a face attempting to interact with the HMI occur within a predefined time period relative to the previous detection of a person attempting to interact with the HMI. If subsequent detection of a face attempting to interact with the HMI does not occur within the predefined time period, the HMI (e.g., processing circuitry) of this exemplary embodiment is configured to not allow resumption of the paused response to the previous gesture; instead, the user must restart the process once the user is wearing PPE.

[0035] However, after a face attempting to interact with the HMI is subsequently detected, such as within a predefined time period, the HMI 20 in the exemplary implementation (e.g., processing circuitry 22) is configured to determine whether the person attempting to interact with the HMI is the same person, for example, based on face recognition or other comparisons between detected face representations. In the case of a different face subsequently detected, the HMI (e.g., processing circuitry) in this exemplary implementation is configured not to allow resumption of the paused response to the previous gesture; instead, the user must restart the process once they are wearing personal protective equipment (PPE). However, if the HMI (e.g., processing circuitry) determines that the same person is attempting to interact with the HMI, the HMI (e.g., processing circuitry) is configured to analyze information about a face subsequently detected, such as by the first sensor 28, to determine whether this person is now wearing PPE. See box 48. If the person is still not wearing PPE, the HMI (e.g., processing circuitry) is configured to continue inducing a pause in the response to gestures previously made by the person relative to the HMI.

[0036] However, if subsequent detection of the face determines that the person is now wearing personal protective equipment, the HMI 20 (e.g., processing circuitry 22) is configured to terminate the pause in the response to the gesture. See also Figure 3 Box 50. After the pause in the response to the gesture is terminated, and as described below, the HMI (e.g., processing circuitry) is configured to analyze the gesture made by the person relative to the HMI and translate the gesture into a corresponding command for the system associated with the HMI. The system is then directed to execute the gesture-related function. Although, to save or at least delay the utilization of processing resources and time, the gesture may not need to be analyzed and translated into a corresponding command for the system until the pause in the response is terminated, the gesture may be analyzed and translated into a corresponding command before the pause in the response is terminated, as long as the system is not directed to execute the function commanded by the gesture until the pause is terminated. For example, the gesture may be analyzed and translated into a corresponding command when the gesture is detected and / or during the pause in the response, but the system is not directed to execute the function commanded by the gesture until the pause is terminated.

[0037] Return to reference Figure 3 In box 42, if initial analysis of the detected face information determines that the person is wearing personal protective equipment, the HMI 20 (e.g., processing circuitry 22) is configured to analyze the person's pose relative to the HMI and translate that pose into a corresponding command for the system associated with the HMI. See also Figure 3Box 52. The system is then directed to perform pose-related functions. The HMI (such as processing circuitry) can be configured to analyze poses in various ways, including the execution of any of a variety of image analysis and / or recognition techniques. However, in an exemplary embodiment, the processing circuitry includes or embodies one or more convolutional neural networks trained to identify and distinguish individual poses from a variety of poses of the system in response to the HMI and subsequently associated with the HMI.

[0038] While the convolutional neural network can be configured to receive either of a variety of signals representing a face and a pose performed by the person, respectively, from the first sensor 28 and the second sensor 30, these signals may optionally have undergone sensor data fusion, the processing circuitry 22 in the exemplary embodiment includes or embodies one or more convolutional neural networks that receive several different versions of information provided by the second sensor. In one exemplary embodiment, where the second sensor provides three different signal streams representing the pose performed by the person in three different colors (e.g., red, green, and blue), the convolutional neural network (one or more) may receive a first version of the information provided by the second sensor, i.e., the three different signal streams, and may be trained to recognize features such as poses based thereon.

[0039] In this exemplary embodiment, the processing circuit 22 may also be configured to receive a second version of information provided by the second sensor in the form of a signal representing the optical flow from the second sensor 30. Here, the optical flow represents the motion pattern of the person's hand performing a posture caused by the relative movement between the person's hand and the second sensor. Based on the signal representing the optical flow, the processing circuit is configured to identify features, such as posture, such as the training results of one or more convolutional neural networks embodied by the processing circuit, to identify posture based on the signal representing the optical flow. Furthermore, the processing circuit in this exemplary embodiment may be configured to receive a third version of information provided by the second sensor, in the form of signals representing the posture performed by the person via three different colors through three different channels and signals representing pixels captured by the second sensor via a fourth channel after semantic segmentation. As a result of semantic segmentation, pixels belonging to the same object class have been grouped together. In embodiments where the processing circuit includes or embodies one or more convolutional neural networks, the convolutional neural networks (one or more) have also been trained to identify features, such as posture, based on signals representing the three colors and signals after semantic segmentation. In this exemplary embodiment, the convolutional neural network (one or more) is also configured to process features identified based on three different versions of information provided by a second sensor, and then predict features performed by the person relative to the HMI, such as pose. This prediction can be performed in various ways, including by using support vector machines (one or more) and / or a softmax function.

[0040] After analyzing the posture as described above, the processing circuit 22 can also be configured to translate the posture into a corresponding command for the system associated with the HMI 20. In this way, the system can then execute functions corresponding to the posture made by the person relative to the HMI.

[0041] Reference to depictions of aircraft cockpit displays during flight Figure 1 In the HMI 20, a masked passenger can make gestures relative to the HMI's user interface 10. These gestures instruct the HMI to change the content depicted in the first area 12 and display information such as estimated arrival time, current aircraft speed, and elapsed time since takeoff, instead of depicting the flight path. This exemplary embodiment of the HMI is configured to detect the passenger's face based on analysis of signals provided by the first sensor 28 to determine if the passenger is wearing personal protective equipment (PPE). Consequently, this exemplary embodiment of the HMI also analyzes gestures detected, for example, by the second sensor 30, and translates these gestures into commands that change the content depicted in the first area of ​​the user interface to the desired content. However, if the passenger makes the same gesture but is not wearing a mask or other PPE, the HMI determines that the passenger is not wearing PPE and suspends any response to the gesture, thus continuing to present... Figure 1 The flight path shown is displayed, and the content presented in the first area of ​​the display is not changed in a manner that corresponds to the posture instruction. In this case, the HMI (such as processing circuitry 22) can also be configured to provide the passenger with information about any response to the posture and can accordingly provide information describing the personal protective equipment that the passenger must wear so that the HMI responds to the posture performed by the passenger. For example, it can be... Figure 1 This information is depicted in area 18 of the user interface.

[0042] In an exemplary implementation, HMI 20 is configured to repeatedly detect faces attempting to interact with the HMI to determine whether the person is wearing personal protective equipment (PPE), and if not, to not only suspend consideration of the gestures performed by the person but also provide a notification. For example, the HMI (e.g., processing circuitry 22) may be configured to provide a notification to the person attempting to interact with the HMI in response to each of a predetermined number of cases in which the person is detected not wearing PPE. In this example, in the case of the HMI on an aircraft, if the person attempting to interact with the HMI continues not to wear PPE after receiving a predetermined number of personal notifications, the HMI (e.g., processing circuitry) may also be configured to provide system-level notifications, such as notifying the crew. The HMI may define the frequency at which faces attempting to interact with the HMI are detected and analyzed to determine whether the person is wearing PPE in any of a variety of ways; however, in one implementation, the HMI determines that the person is wearing PPE once every 2 to 5 seconds, for example, once every 3 seconds.

[0043] Now for reference Figure 4 This diagram depicts an operational flowchart performed by, for example, processing circuitry 22 of an HMI 20 according to an exemplary embodiment of this disclosure. As shown, a second sensor 30, such as an EO sensor, is configured to provide a signal indicative of the passenger and any posture performed by the passenger within the field of view of the EO sensor. While the EO sensor can provide any of a variety of different signals, the EO sensor in this exemplary embodiment is configured to provide a signal indicative of finger coordinates, such as coordinates of key points of the finger, and confidence values ​​associated with these coordinates. See also... Figure 4 Box 54. In this exemplary embodiment, the EO sensor may first provide a signal to the processing circuitry, and more specifically to the attention detection module 60, which is implemented, for example by one or more CNNs, and embodied by the processing circuitry, to recognize situations where a passenger faces the HMI user interface and performs a predefined trigger gesture (such as by waving). Once the trigger gesture is detected, the signal detected by the EO sensor is provided to the processing circuitry, and more specifically to a gesture recognition module 62, for example by one or more convolutional neural networks. The gesture recognition module of this exemplary embodiment is trained to recognize one or more gestures performed by the passenger.

[0044] Furthermore, a first sensor 28, such as an NIR sensor, is configured to provide signals in the form of point cloud data, from which the processing circuitry 22 can estimate the passenger's face. See block 56. This information can be provided to the processing circuitry, such as a mask detection module 64 embodied by one or more convolutional neural networks. The mask detection module is used to detect whether the passenger is wearing personal protective equipment (PPE). The processing circuitry in this exemplary embodiment also includes a messaging system 66 configured to receive signals from the attention detection module 60, the gesture recognition module 62, and the mask detection module, and accordingly communicate with the passenger via, for example, the user interface 10 and / or systems associated with the HMI 20. For example, if the passenger is not wearing PPE, the messaging system in this exemplary embodiment can be configured to provide information via, for example, the user interface, reminding the passenger that the expected response to the gesture has been suspended until the passenger puts on PPE. However, if it is determined that the passenger is wearing PPE, the gesture recognized by the gesture recognition module is translated into a corresponding command, and the messaging system commands the system associated with the HMI to perform its associated functions.

[0045] In an exemplary embodiment, the first sensor 28, such as an NIR sensor, may also be configured to scan boarding passes or other markers that identify passengers. Processing circuitry 22, for example... Figure 4The optical authentication system 68 in the embodiment is configured to verify that a passenger will be seated in the seat associated with HMI 20 (e.g., by placing it on the seatback facing each seat). Although the processing circuitry can make such a determination in various ways, the processing circuitry of the exemplary embodiment, such as the optical authentication system, is configured to use natural language processing and optical character recognition to identify the passenger and the seat assigned to that passenger, such as the passenger identified on the boarding pass, and to verify that the passenger is seated in the correct seat. If the processing circuitry, such as the optical authentication system, determines that the passenger is not in the correct seat, the processing circuitry, such as the messaging system 66, may be configured to provide a message to the passenger, for example via the user interface 10, reminding the passenger that they may be seated in the wrong seat. Additionally or alternatively, the processing circuitry, such as the messaging system, may be configured to notify the crew that the passenger may be seated in the wrong seat. However, once the processing circuitry, such as the optical authentication system, determines that the passenger is seated correctly and the processing circuitry, such as the attention detection module 60, determines that a trigger action has been performed, the processing circuitry, such as the gesture recognition module 62 and the mask detection module 64, can then process the signals provided by the EO sensor and the NIR sensor to determine whether the passenger is wearing personal protective equipment, and if so, to determine the posture performed by the passenger. The processing circuitry, such as the messaging system, can then be configured to accordingly command the system associated with the HMI to perform posture-related functions.

[0046] As described above, an HMI 20, methods for interacting with the HMI, and corresponding computer program products are provided to facilitate user interaction with the HMI. The HMI, methods, and associated computer program products are configured to detect gestures and act upon them, rather than requiring the user to touch the HMI's user interface. By relying on gestures, the HMI can be configured to receive a large number of different types of input from the user, thereby increasing the flexibility and level of detail in controlling the systems associated with the HMI. Furthermore, the HMI, methods, and computer program products are configured to detect whether the user is wearing personal protective equipment (PPE) when attempting to interact with the HMI and can modulate any response of the HMI (including any systems associated with the HMI) to input provided by the user wearing PPE. Therefore, the HMI, methods, and computer program products encourage the user to wear PPE, thereby reducing the health risks associated with the user and other subsequent users using the HMI. Additionally, the HMI, methods, and computer program products can be configured to log, i.e., maintain a record of the user and whether the user is wearing PPE, thereby facilitating contact tracing relative to other users of the HMI or others nearby when the user interacts with the HMI.

[0047] As mentioned above, Figure 3 and Figure 4 A flowchart illustrating an exemplary embodiment of an HMI 20, method, and computer program product according to this disclosure is provided. It should be understood that each block of the flowchart, and combinations thereof, can be implemented in various ways, such as hardware and / or a computer program product including one or more computer-readable storage media having computer-readable program instructions stored thereon. For example, one or more processes described herein can be embodied by computer program instructions of a computer program product. In this respect, a computer program product (one or more) embodying the processes described herein can be stored by one or more storage devices 24 of the HMI and executed by the processing circuitry 22 of the HMI. In some embodiments, computer program instructions including a computer program product (one or more) embodying the processes described above can be stored by multiple storage devices. It will be understood that any such computer program product can be loaded onto a computer or other programmable device to produce a machine, such that the computer program product including instructions executable on the computer or other programmable device creates means for implementing the functions specified in the flowchart blocks. Furthermore, the computer program product can include one or more computer-readable storage media on which computer program instructions can be stored, such that the one or more computer-readable storage media can instruct a computer or other programmable device to operate in a particular manner, such that the computer program product includes an article of manufacture that implements the functions specified in the flowchart blocks. Computer program instructions of one or more computer program products may also be loaded onto a computing system or other programmable device to cause a series of operations to be performed on the computing system or other programmable device to produce a computer-implemented process, such that the instructions executed on the computing system or other programmable device implement the functions specified in the flowchart boxes.

[0048] Therefore, the blocks or steps of a flowchart support combinations of means for performing a specified function and combinations of steps for performing a specified function. It should also be understood that one or more blocks of a flowchart, as well as combinations of blocks in a flowchart, can be implemented by a dedicated hardware-based computer system or a combination of dedicated hardware and computer program products that perform the specified function or steps.

[0049] The functions described above can be performed in a variety of ways. For example, any suitable means for performing each of the functions described above can be used to implement embodiments of this disclosure. In one embodiment, a suitably configured computing system 20 may provide all or part of the elements of this disclosure. In another embodiment, all or part of the elements may be configured and operated under the control of a computer program product. A computer program product for performing embodiments of this disclosure includes a computer-readable storage medium, such as a non-volatile storage medium, and a computer-readable program code portion embodied in the computer-readable storage medium, such as a series of computer instructions.

[0050] Furthermore, this disclosure includes implementation methods according to the following terms:

[0051] Clause 1. A method for interacting with a human-machine interface (HMI) (20) of a system, the method comprising:

[0052] (40) Detect the face of a person attempting to interact with the HMI and detect the pose of the person relative to the HMI;

[0053] (42) Analyze information about detected faces to determine whether the person is wearing personal protective equipment;

[0054] If it is determined that the person is not wearing personal protective equipment, (44) suspend the response to the person's gestures relative to the HMI; and

[0055] If it is determined that the person is wearing personal protective equipment, (52) the person’s posture relative to the HMI is analyzed and the posture is translated into a corresponding command to the system associated with the HMI.

[0056] Clause 2. The method according to Clause 1, wherein, in cases where it has been previously determined that the person is not wearing personal protective equipment, the method further comprises:

[0057] (46) Then detect faces attempting to interact with the HMI (20);

[0058] (48) Analyze subsequently detected facial information to determine whether the person is currently wearing the personal protective equipment; and

[0059] If, based on subsequent face detection, it is determined that the person is now wearing personal protective equipment, (50) the pause in the response to the gesture is terminated and (52) the gesture made by the person relative to the HMI is analyzed and the gesture is translated into the corresponding command to the system associated with the HMI.

[0060] Clause 3. The method described under any of the preceding clauses, wherein the personal protective equipment includes a face shield.

[0061] Clause 4. The method described under any of the preceding clauses, wherein (42) analyzing information about a face includes estimating point cloud data representing a face to determine whether the person is wearing the personal protective equipment.

[0062] Clause 5. The method according to any of the preceding clauses, wherein (40) detecting a face attempting to interact with the HMI (20) and detecting the person’s gesture relative to the HMI comprises detecting the face based on signals received by a first sensor (28) and detecting the person’s gesture relative to the HMI based on signals received by a second sensor (30) different from the first sensor.

[0063] Clause 6. The method according to Clause 5, wherein the first sensor (28) comprises a near-infrared (NIR) sensor and the second sensor (30) comprises an electro-optical (EO) sensor.

[0064] Clause 7. The method according to any of the preceding clauses, wherein (40) detecting a face attempting to interact with the HMI (20), (42) analyzing information about the detected face to determine whether the person is wearing personal protective equipment, and (52) analyzing the person's gesture relative to the HMI and converting the gesture into the corresponding command are performed using one or more convolutional neural networks.

[0065] Clause 8. A human-machine interface (HMI) (20) for a system, said HMI including processing circuitry (22), said processing circuitry being configured to:

[0066] (40) Detect the face of a person attempting to interact with the HMI and detect the pose of the person relative to the HMI;

[0067] (42) Analyze information about detected faces to determine whether the person is wearing personal protective equipment;

[0068] If it is determined that the person is not wearing personal protective equipment, (44) suspend the response to the person's gestures relative to the HMI; and

[0069] If it is determined that the person is wearing personal protective equipment, (52) analyze the person’s posture relative to the HMI and translate the posture into a corresponding command of the system associated with the HMI.

[0070] Clause 9. The human-machine interface (20) according to Clause 8, wherein, in the case of prior determination that the person is not wearing personal protective equipment, the processing circuit (22) is further configured as follows:

[0071] (46) Then detect faces attempting to interact with the HMI;

[0072] (48) Analyze subsequently detected facial information to determine whether the person is currently wearing the personal protective equipment; and

[0073] If, based on subsequent face detection, it is determined that the person is now wearing personal protective equipment, (50) the pause in the response to the gesture is terminated and (52) the gesture made by the person relative to the HMI is analyzed and the gesture is translated into the corresponding command to the system associated with the HMI.

[0074] Clause 10. The human-machine interface (20) according to any one of Clauses 8-9, wherein the personal protective equipment includes a face shield.

[0075] Clause 11. The human-machine interface (20) according to any one of Clauses 8-10, wherein the processing circuit (22) is configured (42) to analyze information about a face by estimating point cloud data representing a face to determine whether the person is wearing the personal protective equipment.

[0076] Clause 12. The human-machine interface (20) according to any one of Clauses 8-11 further includes:

[0077] A first sensor (28) configured to provide the processing circuitry (22) with a signal detecting a face attempting to interact with the HMI; and

[0078] A second sensor (30), different from the first sensor, is configured to provide a signal to the processing circuitry that detects the person’s posture relative to the HMI.

[0079] Clause 13. The human-machine interface (20) according to Clause 12, wherein the first sensor (28) comprises a near-infrared (NIR) sensor and the second sensor (30) comprises an electro-optical (EO) sensor.

[0080] Clause 14. The human-machine interface (20) according to any one of Clauses 8-13, wherein the processing circuitry (22) comprises one or more convolutional neural networks configured to (40) detect a face attempting to interact with the HMI, (42) analyze information about the detected face to determine whether the person is wearing personal protective equipment, and (52) analyze the gesture made by the person relative to the HMI and convert the gesture into the corresponding command.

[0081] Clause 15. A computer program product comprising at least one non-transitory computer-readable storage medium storing computer-executable program code instructions therein, the computer-executable program code instructions comprising program code instructions to:

[0082] (40) Detect the face of a person attempting to interact with the HMI (20) and detect the pose of the person relative to the HMI;

[0083] (42) Analyze information about detected faces to determine whether the person is wearing personal protective equipment;

[0084] If it is determined that the person is not wearing personal protective equipment, (44) suspend the response to the person's gestures relative to the HMI; and

[0085] If it is determined that the person is wearing personal protective equipment, (52) the person’s posture relative to the HMI is analyzed and the posture is translated into a corresponding command to the system associated with the HMI.

[0086] Clause 16. The computer program product pursuant to Clause 15, wherein, in the event that it has been previously determined that the person is not wearing personal protective equipment, the computer-executable program code instructions further include program code instructions to:

[0087] (46) Then detect faces attempting to interact with the HMI (20);

[0088] (48) Analyze subsequently detected facial information to determine whether the person is currently wearing the personal protective equipment; and

[0089] If, based on subsequent face detection, it is determined that the person is now wearing personal protective equipment, (50) the pause in the response to the gesture is terminated and (52) the gesture made by the person relative to the HMI is analyzed and the gesture is translated into the corresponding command of the system associated with the HMI.

[0090] Clause 17. The computer program product according to any one of Clauses 15-16, wherein the personal protective equipment includes a face shield.

[0091] Clause 18. A computer program product according to any one of Clauses 15-17, wherein (42) program code instructions for analyzing information about a face include program code instructions for estimating point cloud data representing a face to determine whether the person is wearing the personal protective equipment.

[0092] Clause 19. A computer program product according to any one of Clauses 15-18, wherein the program code instructions for (40) to detect a face attempting to interact with the HMI (20) and to detect the person’s gesture relative to the HMI include program code instructions for detecting the face based on signals received by a first sensor (28) and for detecting the person’s gesture relative to the HMI based on signals received by a second sensor (30) different from the first sensor.

[0093] Clause 20. A computer program product according to any one of Clauses 15-19, wherein the program code instructions comprise one or more convolutional neural networks configured to (40) detect a face attempting to interact with the HMI (20), (42) analyze information about the detected face to determine whether the person is wearing personal protective equipment, and (52) analyze the gesture made by the person relative to the HMI and translate the gesture into the corresponding command.

[0094] Many modifications and other aspects of the disclosure set forth herein will come to mind for those skilled in the art upon which this disclosure pertains, taking advantage of the teachings presented in the foregoing description and the accompanying drawings. Therefore, it should be understood that this disclosure is not limited to the specific aspects disclosed, and that modifications and other aspects are intended to be included within the scope of the appended claims. Although specific terminology is used herein, it is used only in a general and descriptive sense and not for limiting purposes.

Claims

1. A computer-implemented method for interacting with a human-machine interface (HMI) (20) of a system, the method comprising: (40) Detect the face of a person attempting to interact with the HMI and detect the pose of the person relative to the HMI; (42) Analyze information about detected faces to determine whether the person is wearing personal protective equipment; If it is determined that the person is not wearing the personal protective equipment, (44) suspend the response to the person’s gestures relative to the HMI; and If it is determined that the person is wearing the personal protective equipment, (52) the person's posture relative to the HMI is analyzed and the posture is translated into a corresponding command to the system associated with the HMI. The detection of a face attempting to interact with the HMI (20) and the detection of a person’s gesture relative to the HMI (40) include detecting a face based on a signal received by a first sensor (28) and detecting a person’s gesture relative to the HMI based on a signal received by a second sensor (30) different from the first sensor; and wherein the second sensor is configured to be continuously or repeatedly activated to detect a predefined trigger gesture and the first sensor is configured to remain inactive until the trigger gesture is detected, after which the first sensor is also configured to be activated to detect a face, and then the person makes the gesture to command the system of the HMI; and The personal protective equipment mentioned therein includes face shields, protective shields, or face shields and head coverings.

2. The method of claim 1, wherein, in cases where it has been previously determined that the person is not wearing the personal protective equipment, the method further comprises: (46) Then detect faces attempting to interact with the HMI (20); (48) Analyze the information about the face that is subsequently detected to determine whether the person is currently wearing the personal protective equipment; and If, based on subsequent detection of the face, it is determined that the person is now wearing the personal protective equipment, (50) the pause in the response to the gesture is terminated and (52) the gesture made by the person relative to the HMI is analyzed and the gesture is translated into a corresponding command to the system associated with the HMI.

3. The method according to any one of claims 1-2, wherein (42) analyzing information about a face includes estimating point cloud data representing a face to determine whether the person is wearing the personal protective equipment.

4. The method according to any one of claims 1-2, wherein the first sensor (28) comprises a near-infrared (NIR) sensor and the second sensor (30) comprises an electro-optic (EO) sensor.

5. The method according to any one of claims 1-2, wherein (40) detecting a face attempting to interact with the HMI (20), (42) analyzing information about the detected face to determine whether the person is wearing personal protective equipment, and (52) analyzing the person's gesture relative to the HMI and converting the gesture into the corresponding command are performed using one or more convolutional neural networks.

6. A human-machine interface (HMI) (20) for a system, the HMI including a processing circuit (22), the processing circuit (22) being configured as follows: (40) Detect the face of a person attempting to interact with the HMI and detect the pose of the person relative to the HMI; (42) Analyze information about detected faces to determine whether the person is wearing personal protective equipment; If it is determined that the person is not wearing the personal protective equipment, (44) suspend the response to the person’s gestures relative to the HMI; and If it is determined that the person is wearing the personal protective equipment, (52) the person's posture relative to the HMI is analyzed and the posture is translated into a corresponding command to the system associated with the HMI. The HMI further includes: A first sensor (28) is configured to provide the processing circuitry (22) with a signal that it detects of a face attempting to interact with the HMI; and A second sensor (30), different from the first sensor, is configured to provide the processing circuit with a signal of the posture the person has made relative to the HMI. The second sensor is configured to be continuously or repeatedly activated to detect a predefined trigger gesture, and the first sensor is configured to remain inactive until the trigger gesture is detected, after which the first sensor is also configured to be activated to detect a face, and then the person makes the gesture to command the HMI system; and The personal protective equipment mentioned therein includes face shields, protective shields, or face shields and head coverings.

7. The human-machine interface (20) according to claim 6, wherein, in the case that it has been previously determined that the person is not wearing the personal protective equipment, the processing circuit (22) is further configured to: (46) Then detect faces attempting to interact with the HMI; (48) Analyze subsequently detected facial information to determine whether the person is currently wearing the personal protective equipment; and If, based on subsequent detection of the face, it is determined that the person is now wearing the personal protective equipment, (50) the pause in the response to the gesture is terminated and (52) the gesture made by the person relative to the HMI is analyzed and the gesture is translated into a corresponding command to the system associated with the HMI.

8. The human-machine interface (20) according to any one of claims 6-7, wherein the processing circuit (22) is configured (42) to analyze information about a face by estimating point cloud data representing a face to determine whether the person is wearing the personal protective equipment.

9. The human-machine interface (20) according to any one of claims 6-7, wherein the first sensor (28) comprises a near-infrared (NIR) sensor and the second sensor (30) comprises an electro-optic (EO) sensor.

10. The human-machine interface (20) according to any one of claims 6-7, wherein the processing circuit (22) comprises one or more convolutional neural networks configured to (40) detect a face attempting to interact with the HMI, (42) analyze information about the detected face to determine whether the person is wearing personal protective equipment, and (52) analyze the gesture made by the person relative to the HMI and convert the gesture into the corresponding command.

11. The human-machine interface (20) of claim 10, wherein the convolutional neural network is configured to receive a first version of information provided by the second sensor, which includes three different signal streams representing the posture in three different colors.

12. The human-machine interface (20) according to claim 11, wherein the three different colors include red, green and blue.

13. The human-computer interface (20) of claim 11, wherein the convolutional neural network is configured to be trained to identify features including the pose based on the first version.

14. A computer program product comprising at least one non-transitory computer-readable storage medium storing computer-executable program code instructions therein, the computer-executable program code instructions including program code instructions that, when executed by processing circuitry (22), cause the processing circuitry (22) to perform the steps of the method according to any one of claims 1-5.

Images