Redundant touchless input for automation systems

Abstract

The present disclosure relates to a human-machine interface for an industrial automation control system and a method for the same. In one embodiment, the human-machine interface for the industrial automation control system includes at least one touchless input device adapted to be in a first state or a second state, in the first state the human-machine interface provides a first input to the industrial automation control system, in the second state the human-machine interface provides a second input to the industrial automation control system. The at least one touchless input device includes a first touchless input sensor and a second touchless input sensor, each touchless input sensor configured to detect a gesture of a hand of an operator to provide an input to the human-machine interface based on the gesture. In one implementation, the touchless input device provides an emergency stop (Estop) switch device.

Description

BACKGROUND

[0001] Industrial automation systems often include a human machine interface (HMI) that provides one of a number of paths through which a human operator of the automation system provides input to the system and receives output from the system. Thus, the HMI often includes conventional computer system input / output devices such as a keyboard, a keypad, one or more switches, a visual display such as a touch screen display for combined input and output, indicator lights, etc. These known HMI devices require physical contact between the operator's body (e.g., his or her fingers) and the HMI device, which in some situations - e.g., in clean room environments, food service environments, and other environments where contamination of the operator's hands and / or the HMI itself is highly undesirable - can be undesirable. Moreover, physical contact with the HMI can increase the risk of transmission of pathogens such as the Covid-19 virus from person to person.

[0002] In industrial automation systems, the HMI often also includes an emergency stop (Estop) switch (button) through which an operator can stop or at least slow down a machine being controlled in an emergency or other situation. As with other HMI inputs, this Estop switch must be physically contacted by the operator's hand / finger, which as noted above is sometimes undesirable. Moreover, these known Estop switches often include redundant (dual) mechanical electrical contacts, which results in increased cost, size, and wiring requirements. Such mechanical contacts in known Estop switches and other HMI switches are also susceptible to physical wear or contamination from liquids, dust, vapors, corrosion, etc.

[0003] To date, touchless input devices have been considered unsuitable for industrial automation applications that require redundancy. Known industrial automation HMI devices have not included touchless input devices that implement a redundant operating scheme to prevent inadvertent actuation of the touchless input device that can result in unsafe or otherwise undesirable conditions.

[0004] Thus, while known HMI and Estop devices have been found to be generally satisfactory, it has been determined that there is a need for new and improved HMI and Estop systems / devices that overcome the above-referenced deficiencies and other deficiencies while providing superior overall results. SUMMARY

[0005] According to one aspect of the present development, a human-machine interface for an industrial automation control system includes at least one touchless input device adapted to be in a first state or a second state, in the first state the human-machine interface provides a first input to the industrial automation control system, in the second state the human-machine interface provides a second input to the industrial automation control system. The at least one touchless input device includes a first touchless input sensor and a second touchless input sensor, each touchless input sensor configured to detect a gesture of a hand of an operator to provide an input to the human-machine interface based on the gesture. The first touchless input sensor and the second touchless input sensor can be the same or different relative to each other.

[0006] According to another aspect of the present development, a method of controlling a human-machine interface for an industrial automation control system includes setting a state of a touchless input device of the human-machine interface to a first state and providing a first input to the industrial automation control system. The method further includes detecting a first gesture using a first touchless sensor and a second touchless sensor, and changing the state of the touchless input device from the first state to a second state and providing a second input to the industrial automation control system when both the first touchless sensor and the second touchless sensor detect the first gesture.

[0007] According to yet another aspect of the present development, the method includes changing the state of the touchless input device from the second state back to the first state and providing the first input to the industrial automation control system when at least one of the first touchless sensor and the second touchless sensor detects a second gesture different from the first gesture.

[0008] According to another aspect of the present development, a method of controlling a human-machine interface for an industrial automation control system includes setting a state of a touchless Estop device of the human-machine interface to a first state and providing a first input to the industrial automation control system. The method includes detecting a first gesture using a first touchless sensor and a second touchless sensor, and changing the state of the touchless Estop device from the first state to a second state and providing a second input to the industrial automation control system when at least one of the first touchless sensor and the second touchless sensor detects the first gesture. The second input to the industrial automation control system corresponds to an emergency stop command. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 (Background Art) shows an example of a known human-machine interface (HMI) device for an industrial automation system including a known emergency stop (Estop) device.

[0010] Figure 1AA detailed view of an Estop switch of the HMI of Figure 1

[0011] Figure 2 A human-machine interface (HMI) device for an industrial automation system is shown that provides and includes one or more integrated non-touch input devices such as an emergency stop (Estop) input device in accordance with embodiments of the present development.

[0012] Figure 2A A detailed view of a non-touch input switch device of the HMI of Figure 2

[0013] Figure 3 An embodiment of a non-touch input device of the HMI of Figure 2

[0014] Figure 4 Similar to Figure 2 but showing an HMI having one or more non-touch input devices formed in accordance with a second embodiment of the present disclosure.

[0015] Figure 4A A detailed view of a non-touch input switch device of the HMI of Figure 4

[0016] An embodiment of a non-touch input device of the HMI of Figure 5 Figure 4 A table showing examples of suitable input gestures for actuating

[0017] Figure 6 A flowchart providing an example of a process for implementing a non-touch emergency stop (Estop) device / system for an industrial automation system using any of the HMI non-touch input devices of Figures 2 to 5

[0018] A flowchart providing an example of a process for implementing a non-touch HMI input device / system for an industrial automation system using any of the HMI non-touch input devices of Figure 7 Figures 2 to 5

[0019] Figure 8 DETAILED DESCRIPTION Figures 2 to 5

[0020] Figure 1 ​​​​​​​​An example of a known human-machine interface (HMI) device 10 for use in an industrial automation system is shown. The HMI 10 is particularly suitable for use in conjunction with an industrial automation control system S that controls an industrialized machine M and / or a process P. Thus, the HMI 10 typically includes conventional computer system input / output devices such as a keyboard, a keypad, one or more switches, a visual display such as a touch screen display for combined input and output, indicator lights, etc. The shown HMI 10 includes a housing 12 that supports a conventional touch screen display 14 for outputting visual information to a human operator and for receiving touch screen input from the operator. In one example, the touch screen display 14 is a touch resistive or touch capacitive display that requires physical contact from an exposed finger of the operator. As is generally known, the touch screen display 14 includes one or more visual icons or other graphical elements 14i that provide visual information to a human operator user, and the user physically touches one or more of these graphical elements 14i to provide input to the HMI and control system S. The known HMI 10 also includes a conventional mechanical emergency stop (Estop) switch 16 (or "button") by which the operator can stop or at least slow down the machine M or process P being controlled in an emergency or other situation. This Estop switch 16 must be physically contacted and moved / actuated by the operator's hand / finger, which, as noted above, is sometimes undesirable. Moreover, as shown, such known Estop switch 16 typically includes a cover 16c that engages one or redundant (dual) mechanical electrical contacts CX1, CX2 when in operation, which results in increased cost, size, and wiring requirements. When manually actuated, the electrical contacts CX1, CX2 provide an emergency stop signal to the industrial automation control system S. Such mechanical contacts CX1, CX2 are also susceptible to physical wear, damage, and / or contamination from liquids, dust, vapors, corrosion, etc. when actuated. Figure 1A

[0021] Figure 2 ​A human-machine interface (HMI) device 110 is shown that is provided in accordance with an embodiment of the present development and that includes one or more redundant non-touch input devices 120. As shown by the dashed lines at 16, in addition to the one or more redundant non-touch input devices 120, the HMI device 110 can optionally include a conventional mechanical emergency stop (Estop) switch 16 that is connected to the control system S when in operation, as described above. The HMI 110 is particularly well suited for use in conjunction with an industrial automation control system S that controls an industrialized machine M and / or process P. In the example shown, the HMI 110 includes a housing 112 that supports a touchscreen display 114, which can be the same as or similar to the touchscreen display 14 described above, except as otherwise described and / or shown herein. In one example, the touchscreen display 114 is a touch resistive or touch capacitive display. As is generally known, the touchscreen display 114 includes one or more visual icons or other graphical elements 114i that provide visual information to a human operator user, and the user physically touches one or more of these graphical elements 114i to provide input to the HMI and control system S. The HMI can also include conventional computer system input / output devices such as a keyboard, keypad, one or more switches, a visual display, one or more status indicator lights (e.g., one or more LED indicator lights 117 connected to the housing 112), etc. However, unlike a conventional HMI 10, the HMI 110 includes one or more non-touch input devices 120 that are used to provide non-touch input to the industrial automation control system S and its controlled machine M and / or process P for input and output of data related thereto. As shown herein, the non-touch input devices 120 are supported by the housing 112 but are separate from the touchscreen display 114, although some or all of the non-touch input devices 120 can alternatively or additionally be integrated into and provided as part of the touchscreen display 114, or the non-touch input devices 120 can be detached from the main HMI housing 112 and located in one or more locations adjacent to or remote from the HMI housing 112. Similarly, as shown herein, the visual status indicator lights 117 are shown herein as being connected to the main HMI housing 112, separate from the touchscreen display 114, but the indicator lights 117 can alternatively be provided by illuminated portions of the touchscreen display 114, illuminated portions of the non-touch input devices 120, and / or located elsewhere. In one example, the touchscreen display 114 is entirely replaced by one or more non-touch input devices 120 for providing input to the system S and controlled machine / process M / P.In another implementation, one or more visual status indicator lights 117 are included as part of the non-touch input device 120 itself (e.g., as one or more pixels of the video display screen), such that when the indicator light 117 is activated, part or all of the non-touch input device 120 is illuminated in the color of the status indicator light 117 (e.g., red, green, or another color).

[0022] Each non-touch input device 120 is constructed and configured to allow a human operator to provide one or more non-touch (contactless) gesture input commands to the HMI 110 for input to the industrial automation control system S and machines and / or processes M, P controlled thereby. In this manner, each non-touch input device 120 provides and functions as a virtual switch or virtual button suitable for non-touch (contactless) operator input to the system S. As shown herein, in accordance with the present development, at least one of the non-touch input devices 120 is configured and adapted to function as a non-touch emergency stop (Estop) input device 116 to stop or at least slow down any machine M and / or process P controlled by the control system S to which the HMI 110 is operated.

[0023] In Figure 2A and Figure 3 are shown, respectively, a non-touch input device (e.g., a non-touch Estop device 116) that includes a base 140, such as a printed circuit board (PCB), that includes at least one, preferably first and second redundant non-touch input sensors or sensor devices 142a, 142b connected thereto, and an electronic processor, such as a microcontroller (MCU). The non-touch input sensors 142a, 142b can be covered by a sheet of glass or transparent polymer. Each non-touch input sensor 142a, 142b is configured and adapted to detect a non-touch gesture of a human operator to provide an input control signal to the HMI and control system S without any physical contact between the human operator's hand or finger and the sensor 142a, 142b or any other component. In Figure 2A implementations, the non-touch input sensors 142a, 142b are provided as time-of-flight (ToF) sensors connected in a parallel, redundant arrangement. The dual ToF sensors 142a, 142b can be operated to scan the sensing field or target in a repetitive, alternating arrangement to avoid interference of one sensor with the other, or they can be operated to scan simultaneously in parallel with one another. As one example, suitable ToF sensors are commercially available from STMicroelectronics under the trademark FLIGHTSENSE, but the present disclosure is not intended to be limited to any particular brand of sensor. The non-touch input sensors 142a, 142b each include a photon emitter and a photon sensor. As shown inFigure 3 As shown, the photon emitter emits photons Y that are incident on a target (a human operator's hand H including a finger F). The photons Y incident on the target are reflected back to the touchless input sensors 142a, 142b, where the reflected photons Y' are detected by the photon sensors. Photons Y associated with the first sensor 142a are shown with solid lines, while photons Y associated with the second sensor 142b are shown with dashed lines. The touchless input sensors 142a, 142b include or are otherwise operatively connected to an electronic processor 146 such as a microcontroller MCU that calculates the time of flight required for each photon Y emitted to travel from the photon emitter to the photon sensor, and the processor 146 uses the time data to derive the distance D and / or changes in distance between the target H, F and the touchless input sensors 142a, 142b. The processor 146 is configured to detect and understand movement or gestures of the target (hand H / finger F) based on changes in the time of flight and / or based on changes in the distance D. The processor 146 is operatively connected to one or more control circuits 148 and communication channels 150 of the industrial control system S (e.g., the emergency power interrupt relays Rl, R2 shown), and the processor 146 provides output to the control circuits 148 and communication channels 150 that varies according to the gestures and / or changes in gestures derived by the processor 146 from the time of flight and distance data, such that the gestures derived by the processor 146 provide input to the control system S through the HMI 110. The processor 146 also controls the color and / or on / off state of the one or more LEDs 117 to provide a visual indication of the status of the HMI 110 and the control system S to the human operator.

[0024] Figure 4 、 Figure 4A and Figure 5 correspond to Figure 2 、 Figure 2A and Figure 3 , respectively, but show an alternative embodiment 210 of the HMI. The HMI 210 is the same as the HMI 110 except as otherwise shown and / or described herein, and thus certain features are identified accordingly but not described again here. One or more of the touchless input devices 120 of the HMI 210 include touchless input sensors 242a, 242b that are alternatives to the touchless input sensors 142a, 142b of the HMI 110. As shown by the dashed lines at 16, the HMI device 110 can optionally include a traditional mechanical emergency stop (Estop) switch 16 as described above in addition to the one or more redundant touchless input devices 120. Unlike the HMI 110, the touchless input sensors 242a, 242b differ relative to one another. In particular as Figure 5As shown, one of the first non-touch input sensors 242a is provided as a time-of-flight (ToF) sensor, the same as sensor 142a of HMI 110, while one of the second non-touch input sensors 242b is provided as a non-touch sensor that is not a time-of-flight sensor, such that the first and second input sensors are different from one another. In the illustrated embodiment of HMI 210, the second non-touch input sensor 242b comprises an electric field (E-Field) proximity sensor such as commercially available from Microchip Technology under the registered trademark GESTIC(R) or another electric field sensor. The electric field proximity sensor 242b generates an electric field EF in the immediate vicinity of the sensor 242b Figure 5 ). The first and second sensors 242a, 242b can be operated simultaneously in parallel or alternatively in a fast-alternating arrangement. When the user's hand H (including its fingers F) enters the electric field EF, the electric field is perturbed and distorted in a manner that can be used by the processor 146 to determine the position and changes in position (gestures) of the hand H and fingers. The first non-touch input sensor 242a and the second non-touch input sensor 242b are redundant and operate in parallel or alternately to detect gesture input from the user's hand H and fingers F. The processor 146 is configured to detect and understand movement or gestures of the object (hand H / fingers F) based on the detected position of the hand / fingers H, F and / or changes in the detected position of the hand / fingers H, F. The processor 146 operates in connection with one or more control circuits 148 and communication channels 150 of the HMI 210 (e.g., the illustrated emergency power interrupt relays Rl, R2) and the processor 146 provides output to the control circuits 148 and communication channels 150 that varies in accordance with the gestures and / or changes in gestures derived by the processor 146 from the non-touch sensors 242a, 242b, such that the gestures derived by the processor 146 provide input to the control system S through the HMI 210.

[0025] Figure 6Examples of a plurality of gestures that can be determined by each touchless input device 120 of the HMI 110, 210 according to embodiments of the present disclosure are provided, including: (i) single tap or single press (single fore-and-aft motion); (ii) double tap or double press (repeating the fore-and-aft motion twice in succession within a selected time period (e.g., 500 milliseconds (ms) / half-second)); (iii) single swipe (unidirectional swipe left or right, either laterally or sideways); and (iv) double swipe (bidirectional swipe left then right or right then left, either laterally or sideways, within a selected time period (e.g., half-second 500 ms or 1000 ms (1 whole second))). Each of these gestures is associated with a unique output signal generated by the HMI 110, 210 for input to the control system S. In one example, the single tap and single swipe gestures can be associated with low-priority controls of the machine M or process P that are non-critical and / or non-safety related. Thus, the single tap or single swipe gestures can be used to control auxiliary functions where erroneous inputs will have little or no adverse effect. On the other hand, the double tap and double swipe gestures, which require more purposeful action on the part of the human operator, are associated with high-priority controls of the machine M or process P that are safety related or otherwise critical. Thus, the double tap or double swipe gestures can be used to control primary functions of the machine M and / or process P (e.g., emergency stop (E-stop) functions) where erroneously detected inputs will be undesirable.

[0026] Figure 7 is provided for using a Figures 2 to 5The flowchart of an example of a process for an industrial automation system S for a non-touch emergency stop (Estop) device / system is implemented with any of the HMI non-touch input devices 120. In the illustrated example, the HMI 110, 210 sets the E-stop state of the HMI 110, 210 and the control system S to "on" (running) as a start condition (step SI), such that the machine M and / or process P is activated. If the first non-touch sensor 142a, 242a detects a double tap gesture of a user's hand in step S2, or if the second non-touch sensor 142b, 242b senses a double tap gesture of a user's hand H (i.e., two quick successive motions toward and away from the non-touch sensor) in step S3, the HMI 110, 210 sets the E-stop state to "off" in step S4, such that the machine M and / or process P is disabled (or slowed down), and the HMI 110, 210 controls the status indicator 117 to be illuminated in red or another selected color in step S5. As described above, the visual indicator 117 can be integrated into the non-touch input device 120, such that the non-touch input device is illuminated in the color of the visual indicator 117. Once the Estop state is set to "off" to disable or slow down the machine or process M, P, any further sensing of the double tap gesture will be ignored. If both the first and second non-touch input sensors 142a, 142b (for the HMI 110) or the first and second non-touch input sensors 242a, 242b (for the HMI 210) detect a double (bi-directional) swipe gesture caused by a backward and forward lateral motion of a user's hand in step S6, the HMI 110, 210 performs step S7, in which the status indicator 117 changes from red to another color (e.g., green) or is extinguished, and control returns to step SI, in which the HMI 110, 210 changes the state of the Estop back to "on" (running), such that the machine or process M, P is activated or returned to full speed. Once the Estop state is set to "on" to activate the machine or process M, P, any further sensing of the double swipe gesture will be ignored.

[0027] Figure 7 The E-stop functionality is just one example of the use of the non-touch input devices 120 according to the present disclosure. Figure 8 is to provide another embodiment for using Figures 2 to 5The flowchart of FIG. 10 illustrates an example of a process for a non-touch HMI input device / system for an industrial automation system S implemented with any one of the HMI non-touch input devices 120. In step T1, the HMI 110, 210 sets the state of the non-touch input device 120 or "button" to "off" as a start-up or default condition and thus provides input from the HMI 110, 210 to the industrial control system S corresponding to the "off state of the non-touch input device 120. In step T2, the HMI 110, 210 illuminates the visual indicator 117 in a first color (e.g., red) to indicate the "off state of the non-touch input device 120. As described above, the visual indicator 117 can be integrated into the non-touch input device 120 such that the non-touch input device is illuminated in the color of the visual indicator 117. In step T3, if both non-touch input sensors 142a, 142b (for HMI 110) or both non-touch input sensors 242a, 242b (for HMI 210) detect a double-tap gesture (two consecutive forward and backward hand movements within a selected time period), then in step T4, the HMI 110, 210 changes the operating state of the non-touch input device 120 to "on" and provides input to the industrial control system S corresponding to the "on state of the non-touch input device 120. Any further sensing of the double-tap gesture is ignored once the state of the non-touch input device 120 is set to "on" (activated) for the control system S. In step T5, the HMI 110, 210 changes the color of the visual indicator 117 to a second color (e.g., green) or extinguishes the visual indicator 117 to indicate the "on state of the non-touch input device 120. In steps T6 and T7, if either of the non-touch sensors 142a, 142b (for HMI 110) or either of the non-touch sensors 242a, 242b (for HMI 210) of the input device 120 senses a double-swipe gesture (bidirectional hand movement within a selected time period) of the user's hand H, the HMI 110, 210 returns control to step T1 and again sets the state of the non-touch input device 120 to "off, and the process continuously repeats itself. Any further sensing of the double-swipe gesture is ignored once the state of the non-touch input device 120 is set to "off for the control system S.

[0028] In the foregoing specification, various implementations have been described. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader scope of the application as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A human-machine interface (HMI) for an industrial automation control system, the HMI comprising at least one non-touch input device, the at least one non-touch input device being adapted to be in a first state or a second state, wherein in the first state the HMI provides a first input to the industrial automation control system, and in the second state the HMI provides a second input to the industrial automation control system, wherein... The at least one non-touch input device includes a first non-touch input sensor and a second non-touch input sensor that are identical to each other and operate alternately, each non-touch input sensor being configured to detect the gesture of an operator's hand in order to provide input to the human-machine interface based on the gesture; The first non-touch input sensor and the second non-touch input sensor are each configured to detect at least a first gesture and a second gesture, wherein the first gesture corresponds to a first input to the industrial automation control system, the second gesture corresponds to a second input to the industrial automation control system, and the at least one non-touch input device ignores the first gesture when the at least one non-touch input device is in the indicated first state.

2. The human-machine interface for an industrial automation control system according to claim 1, wherein, One of the first gesture and the second gesture includes a double tap gesture, which includes two consecutive hand movements toward and away from the first non-touch input sensor and the second non-touch input sensor within a first selected time period, and wherein the other of the first gesture and the second gesture includes a double swipe gesture, which includes two consecutive lateral hand movements across the first non-touch input sensor and the second non-touch input sensor within a second selected time period.

3. The human-machine interface for an industrial automation control system according to claim 2, wherein, Both the first selected time period and the second selected time period are within the range of 500 milliseconds to 1000 milliseconds.

4. The human-machine interface for an industrial automation control system according to claim 1, wherein, The first non-touch input sensor and the second non-touch input sensor are each time-of-flight sensors.

5. The human-machine interface for an industrial automation control system according to claim 1, wherein, The first non-touch input sensor and the second non-touch input sensor are each time-of-flight photon sensors.

6. The human-machine interface for an industrial automation control system according to claim 1, wherein, The first non-touch input sensor and the second non-touch input sensor are electric field proximity sensors.

7. The human-machine interface for an industrial automation control system according to claim 1, further comprising at least one visual indicator light, the at least one visual indicator light being configured to illuminate a selected color when the non-touch input device is in one of the first state and the second state.

8. The human-machine interface for an industrial automation control system according to claim 1 further includes a mechanical emergency stop switch, which is connected to the industrial automation control system during operation and includes at least one mechanical electrical contact, which provides an emergency stop signal to the industrial automation control system when actuated.

9. A method for controlling a human-machine interface for an industrial automation control system, the method comprising: The non-touch input device of the human-machine interface is set to a first state and a first input is provided to the industrial automation control system. The first gesture is detected using a first non-touch sensor and a second non-touch sensor that are identical to each other and operate alternately. When both the first non-touch sensor and the second non-touch sensor detect the first gesture, the state of the non-touch input device is changed from the first state to the second state, and a second input is provided to the industrial automation control system. In the case that the non-touch input device is in the second state, the non-touch input device ignores the first gesture.

10. The method for controlling a human-machine interface for an industrial automation control system according to claim 9, further comprising: When at least one of the first non-touch sensor and the second non-touch sensor detects a second gesture that is different from the first gesture, the state of the non-touch input device is changed from the second state back to the first state and the first input is provided to the industrial automation control system.

11. The method for controlling a human-machine interface for an industrial automation control system according to claim 10, wherein, When the non-touch input device is in the first state, the non-touch input device ignores the second gesture.

12. A method for controlling a human-machine interface for an industrial automation control system, the method comprising: Set the state of the non-touch Estop device of the human-machine interface to a first state and provide a first input to the industrial automation control system; The first gesture is detected using a first non-touch sensor and a second non-touch sensor that are identical to each other and operate alternately. When at least one of the first non-touch sensor and the second non-touch sensor detects the first gesture, the state of the non-touch Estop device is changed from the first state to the second state and a second input is provided to the industrial automation control system, wherein, when the non-touch Estop device is in the second state, the non-touch Estop device ignores the first gesture; The second input to the industrial automation control system corresponds to an emergency stop command.

13. The method for controlling a human-machine interface for an industrial automation control system according to claim 12, further comprising: When the non-touch input device is in the second state, the visual indicator is illuminated in red.

Images