Aerial operating device

Abstract

To provide an aerial operation apparatus configured to suppress malfunctions.SOLUTION: An aerial operation apparatus comprises: a recognition unit that recognizes whether a button 11 has been pressed on the basis of the position of an object F detected by a sensor 12; and a signal output unit that outputs a control signal corresponding to the button 11 pressed by a user of the aerial operation apparatus 1 to equipment outside the aerial operation apparatus 1 to control the operation of the equipment. When the recognition unit recognizes that the object F detected by the sensor 12 is located at a position of a button 11, the recognition unit disables the operation of a plurality of buttons 11 so as to suppress the output of the control signal output by the signal output unit. When the recognition unit recognizes that the object F detected by the sensor 12 has separated from the button 11, the recognition unit enables the operation of the button 11 so as to allow the signal output unit to output the control signal.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an aerial operation device in which operations are performed on a display surface displayed as a virtual image in the air. [Background technology]

[0002] Japanese Patent Application Laid-Open No. 2019-109637 describes a non-contact input device including a device housing, a display disposed inside the device housing, an optical plate that forms a real image of the display in the air outside the device housing, and a non-contact position detection unit. The non-contact position detection unit includes an invisible light emitting element and a light receiving element that detects reflected light, and detects light emitted from the invisible light emitting element and reflected on a detection surface with the light receiving element. The detection surface is disposed approximately parallel to the image plane of the aerial image formed by the optical plate.

[0003] The non-contact input device has a first detection surface and a second detection surface located on the back side of the first detection surface. A finger is placed on each of the first detection surface and the second detection surface as a pointing object. When the finger touches the first detection surface, the XY position coordinates of the finger pointing are displayed on the aerial image as a pointer. This allows the user to recognize where on the aerial image the finger is pointing.

[0004] When the finger touches the second detection surface, the point in time when the finger touches the second detection surface is regarded as the time of input, and thereafter, the indicated position is corrected based on the position coordinates detected on the first detection surface. In other words, the position of the finger thereafter is corrected based on the point in time when the finger touches the second detection surface, so that the user can perform input operations without worrying about misalignment of the finger. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-109637 A Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, when a detection plane is set substantially parallel to the image plane of the aerial image and an object such as a finger is detected on the detection plane, erroneous detection of the object may be a problem. When multiple buttons are displayed as multiple detection planes, a malfunction may occur when a detection plane corresponding to a button other than the pressed button detects an object. That is, a button other than the pressed button may react. In addition, when a display surface with buttons is displayed as a virtual image in the air, an unintended operation may be performed when something other than a finger, such as the user's clothing, touches the button. In this way, there is a concern that a malfunction may occur or an unintended operation may be performed.

[0007] The present disclosure aims to provide an aerial operation device that can suppress malfunctions. [Means for solving the problem]

[0008] An aerial manipulating device according to one aspect of the present disclosure includes a display device that displays information by irradiating light, an aerial imaging device that reflects light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air, and a sensor that detects the position of an object approaching the button. The aerial manipulating device includes a recognition unit that recognizes whether the button has been pressed based on the position of the object detected by the sensor, and a signal output unit that outputs a control signal corresponding to the button pressed by a user of the aerial manipulating device to an external device of the aerial manipulating device and controls the operation of the device. When the recognition unit recognizes that the object detected by the sensor is at the position of a button, it disables the multiple buttons to suppress output of a control signal by the signal output unit, and when it recognizes that the object detected by the sensor is moving away from the button, it enables the button to allow output of a control signal by the signal output unit.

[0009] In this aerial operation device, an aerial imaging device displays a display surface with multiple buttons as a virtual image in the air, and a sensor detects the position of an object such as a finger approaching a button. The recognition unit recognizes whether or not a button has been pressed based on the position of the object detected by the sensor. The signal output unit outputs a control signal corresponding to the pressed button to an external device of the aerial operation device. The recognition unit disables the multiple buttons when the object is at the position of the button. Therefore, by disabling the multiple buttons at the timing when the button is pressed, it is possible to suppress an unintended button from reacting and causing an unintended operation. In addition, when the object leaves the button, the recognition unit enables the button to allow the signal output unit to output a control signal. Therefore, by enabling the button at the timing when the object leaves the button, only the intended button can be reacted to realize the intended operation. Therefore, it is possible to suppress erroneous operations and improve the operability of the buttons.

[0010] The multiple buttons may form a group including at least one button, and the recognition unit may disable all the buttons in the group when it recognizes that an object detected by the sensor is located at the position of a button in the group. In this case, the recognition unit disables all the buttons in the group when the object is located at the position of a button belonging to the group. By disabling all the buttons in the group when a button in the group is pressed, it is possible to prevent other buttons in the group from reacting and causing an unintended operation.

[0011] An aerial manipulating device according to another aspect of the present disclosure includes a display device that displays information by emitting light, an aerial imaging device that reflects the light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air, and a sensor that detects the position of an object approaching the button, and includes a recognition unit that recognizes whether the button has been pressed based on the position of the object detected by the sensor, and a signal output unit that outputs a control signal to an external device to control the operation of the device, wherein the sensor detects the position of the object multiple times, and the signal output unit outputs a control signal to the device only when the sensor detects multiple times that the object is located between a first surface and a second surface that sandwich the display surface.

[0012] In this aerial operation device, when the sensor detects multiple times that an object is located between the first surface and the second surface that sandwich the display surface on which the image is formed by the aerial imaging device, the signal output unit outputs a control signal to the device. That is, only when the sensor detects multiple times that an object is located between the first surface and the second surface, the signal output unit outputs a control signal to the device and the device operates. Therefore, if the sensor does not detect multiple times that an object located between the first surface and the second surface, the signal output unit does not output a control signal, so that malfunction of the device can be suppressed.

[0013] The above-mentioned aerial manipulation device includes a notification unit that notifies a user of the aerial manipulation device that the recognition unit has recognized the object, and the notification unit may notify the user that the button has been pressed when the sensor detects multiple times that an object is located between the first and second surfaces that sandwich the display surface. In this case, the user is notified that the button has been pressed when the sensor detects multiple times that an object is located between the first and second surfaces. Therefore, the user can be notified that the button has been pressed correctly, thereby improving the operability of the button.

[0014] The sensor may measure the distance from the sensor to the object multiple times, and the signal output unit may output a control signal to the device only when the average value of the measured distances from the sensor to the object is equal to or greater than the distance from the sensor to the second surface and equal to or less than the distance from the sensor to the first surface. In this case, the signal output unit outputs a control signal to the device only when the average value of the measured distances is equal to or greater than the distance from the sensor to the second surface and equal to or less than the distance from the sensor to the first surface. Therefore, when the average value of the distance is shorter than the distance from the sensor to the second surface or longer than the distance from the sensor to the first surface, the signal output unit does not output a control signal. Therefore, it is possible to suppress erroneous detection of the object and more reliably suppress malfunction.

[0015] The above-mentioned aerial manipulation device includes a plurality of sensors, and the signal output unit may not output a control signal to the device when the plurality of sensors detect an object. In this case, the output of the control signal is suppressed when the plurality of sensors detects an object, so that erroneous detection and malfunction can be more reliably suppressed.

[0016] An aerial manipulating device according to another aspect of the present disclosure includes a display device that displays information by emitting light, an aerial imaging device that reflects the light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air, and a sensor that detects the position of an object approaching the button, and includes a recognition unit that recognizes whether the button has been pressed based on the position of the object detected by the sensor, and a signal output unit that outputs a control signal to an external device of the aerial manipulating device to control the operation of the device, wherein the sensor detects the position of the object multiple times, and the signal output unit outputs a control signal to the device only when the recognition unit recognizes that the position of the object detected by the sensor is a first surface and then recognizes that the position of the object detected by the sensor is a second surface that is closer to the sensor than the first surface.

[0017] In this aerial manipulation device, the signal output unit outputs a control signal to the device only when the recognition unit recognizes that the object is located on the second surface closer to the sensor than the first surface after the recognition unit recognizes that the object is located on the first surface. That is, a control signal is output to the device only when it is recognized that the object has moved from the first surface to the second surface closer to the sensor. Therefore, a control signal is not output to the device except when the object moves from the first surface to the second surface, so that erroneous detection of the object and malfunction of the device can be suppressed.

[0018] The above-mentioned aerial operation device may include a plurality of sensors, and the distance from the first surface to the second surface may be half the distance between two adjacent sensors. In this case, a control signal for the device is output only when the object moves half the distance between the two sensors from the first surface and reaches the second surface. Therefore, the output of the control signal for the device can be limited, and malfunctions can be suppressed.

[0019] The above-mentioned aerial operation device includes a dummy sensor that detects the position of an object approaching a position adjacent to the button on the display surface, and the signal output unit does not need to output a control signal to the device when the position of the object is detected by the dummy sensor. In this case, the output of the control signal to the device is restricted when the dummy sensor detects an object adjacent to the button. Therefore, it is possible to more reliably suppress erroneous detection of the object and malfunction of the device. Effect of the Invention

[0020] According to the present disclosure, malfunctions can be suppressed. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a perspective view showing an application example of an aerial operation device according to an embodiment. [Diagram 2] FIG. 1 is a perspective view showing a state in which the aerial operating device according to the embodiment displays a display surface with buttons. [Diagram 3] FIG. 2 is a vertical cross-sectional view of the aerial operation device according to the embodiment. [Figure 4] FIG. 2 is a block diagram showing the functions of the aerial operation device according to the embodiment. [Diagram 5] 3A to 3C are diagrams illustrating detection of an object by a sensor according to an embodiment. [Figure 6] 4 is a block diagram showing an example of a relationship between a button and a recognition unit according to the embodiment; FIG. [Figure 7] 10 is a flowchart showing steps of a button recognition method in the aerial operating device according to the embodiment. [Figure 8] FIG. 13 is a perspective view showing an aerial operation device according to a modified example. [Figure 9] FIG. 11 is a vertical cross-sectional view showing a modified embodiment of the aerial operation device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Various embodiments of the aerial manipulation device according to the present disclosure will be described below with reference to the drawings. In the description of the drawings, the same or corresponding elements are given the same reference numerals, and duplicated descriptions are omitted as appropriate. In addition, the drawings may be partially simplified or exaggerated for ease of understanding, and the dimensional ratios and angles are not limited to those shown in the drawings.

[0023] Fig. 1 is a perspective view showing an example of an aerial manipulating device 1 according to this embodiment. Fig. 2 is a perspective view showing a state in which the aerial manipulating device 1 displays a display surface 10 with buttons 11 in the air. For example, the aerial manipulating device 1 according to this embodiment is an operation input device for operating equipment.

[0024] As an example, the aerial operation device 1 is provided in a toilet T, and the buttons 11 on the display surface 10 are switches that operate various parts of the toilet T. The display surface 10 with the buttons 11 is a floating image that is displayed as if it is floating in the air. In other words, the display surface 10 with the buttons 11 is displayed as a virtual image in the air. Therefore, a user of the aerial operation device 1 can operate various parts of the toilet T in a hygienic manner, since they do not have to directly touch buttons, etc.

[0025] For example, the toilet T includes a toilet bowl B and a wall W adjacent to the toilet bowl B. For example, the wall W is provided on the right side as viewed from a user seated on the toilet bowl B. An aerial operation device 1 and toilet paper E are fixed to the exemplary wall W, and a washbasin M may also be provided thereon. The toilet bowl B includes, for example, a Washlet (registered trademark), and the Washlet of the toilet T can be operated from a button 11 on a display surface 10. For example, the display surface 10 includes a plurality of buttons 11, and the user can operate each part of the Washlet by operating the plurality of buttons 11.

[0026] The aerial operation device 1 is fixed to, for example, a wall W. The aerial operation device 1 may be embedded in the wall W, or may be detachable from the wall W. The exemplary aerial operation device 1 includes a housing 2 and a protruding portion 3 protruding from a lower portion of the housing 2. A plurality of sensors 12 are embedded in the protruding portion 3 to detect an object F approaching a button 11.

[0027] In this disclosure, the "object" refers to an object that operates a device by operating a button 11, for example, a user's finger. The sensors 12 are provided corresponding to the buttons 11, and each button 11 is displayed diagonally above the corresponding sensor 12. As an example, the number of buttons 11 and sensors 12 is seven.

[0028] The display surface 10 is displayed so as to be oriented diagonally upward with respect to both the up-down direction D1 of the aerial operation device 1 and the protruding direction D3 from the wall W. This makes it easier for the user to view the display surface 10. For example, since the display surface 10 is displayed glowing, the display surface 10 can be easily viewed regardless of the brightness of the surroundings. The display surface 10 has a shape that extends long in the left-right direction D2, for example.

[0029] As an example, the display surface 10 is displayed in a rectangular shape, and the vertical length L1 of the display surface 10 is shorter than the horizontal length L2 of the display surface 10. If the aspect ratio of the length L1 to the length L2 is 1:X (X is a real number), then, for example, the value of X is 2 or more and 10 or less. The lower limit of the value of X may be 3, and the upper limit of the value of X may be 5, 6, 7, 8, or 9.

[0030] As an example, the value of X above may be 4, and the aspect ratio of length L1 to length L2 may be 1:4. Button 11 may have, for example, a rectangular shape. As an example, button 11 may have a square shape. The length of one side of button 11 is, for example, 18 mm or more and 24 mm or less. However, the shape and size of button 11 may be changed as appropriate.

[0031] The display surface 10 may display something other than the plurality of buttons 11. For example, the display surface 10 may include the plurality of buttons 11 and text information 13. As an example, the text information 13 may include at least any one of date information, time information, temperature information, temperature information, and atmospheric pressure information.

[0032] On the display surface 10, for example, a plurality of buttons 11 may be displayed lined up horizontally on the lower side of the display surface 10, and text information 13 may be displayed above the plurality of buttons 11. The buttons 11 may be arranged such that their centers 11b are positioned at a position shifted from the center 10b of the display surface 10. In the present embodiment, as an example, each of the plurality of buttons 11 is displayed at a position offset from a reference line V that passes through the center 10b and extends in the longitudinal direction (horizontal direction) of the display surface 10.

[0033] Fig. 3 is a cross-sectional view showing a schematic internal structure of the aerial operation device 1. As shown in Fig. 2 and Fig. 3, for example, the housing 2 has a horizontally elongated box shape. As an example, the housing 2 has a front surface 2b extending in the up-down direction D1 and the left-right direction D2 when attached to the wall part W, a pair of left and right side surfaces 2c extending in the up-down direction D1 and in the protruding direction D3 from the wall part W, an upper surface 2d extending in the left-right direction D2 and the protruding direction D3, and a lower surface 2f facing the opposite side to the upper surface 2d.

[0034] The protruding portion 3 has, for example, a rectangular plate shape. As an example, the protruding portion 3 has a main surface 3b that protrudes from the lower side of the front surface 2b and faces upward, an end surface 3c that faces the left-right direction D2 or the protruding direction D3, and a back surface 3d that faces the opposite side to the main surface 3b. As described above, the protruding portion 3 has a built-in sensor 12. This makes it easier for the sensor 12 to detect the object F.

[0035] The aerial operation device 1 includes an aerial imaging device 14 that displays a display surface 10 with buttons 11, a display device 15 arranged inside the housing 2, and a control unit 20. The aerial imaging device 14 and the display device 15 correspond to a virtual image display unit that displays the display surface 10 as a virtual image in the air. The aerial imaging device 14 is, for example, a retroreflective member (retroreflector) fixed to an opening 2g located inside the front surface 2b of the housing 2.

[0036] The opening 2g is defined by, for example, a pair of first plates 2h constituting the front surface 2b of the housing 2, and a pair of second plates 2j fixed to the inside of the housing 2 of the first plates 2h. As an example, the first plate 2h and the second plate 2j are provided as a pair above and below. The aerial imaging device 14 is fixed to the housing 2 in a state where it is sandwiched between the first plate 2h and the second plate 2j.

[0037] The aerial imaging device 14 is, for example, rectangular in shape with long sides extending in the left-right direction D2 and short sides extending in the up-down direction D1. One long side and the other long side of the aerial imaging device 14 are sandwiched between the first plate portion 2h and the second plate portion 2j. The display device 15 is disposed obliquely with respect to the aerial imaging device 14.

[0038] As an example, the display device 15 is a liquid crystal display (LCD). The display device 15 is disposed diagonally above the aerial imaging device 14 inside the housing 2. The display device 15 has a screen for displaying an image. The screen of the display device 15, for example, irradiates light C as an image diagonally downward toward the aerial imaging device 14. The aerial imaging device 14 reflects the light C from the display device 15 multiple times (for example, twice) inside the aerial imaging device 14, and forms an image on the display surface 10 in a space in front of the aerial imaging device 14 as viewed from the user of the aerial operation device 1.

[0039] Sensor 12 may be exposed on, for example, main surface 3b of protrusion 3. As an example, sensor 12 is a depth sensor. For example, sensor 12 is provided on a virtual straight line extending from button 11, that is, in a position in front of button 11, which is a virtual image. Sensor 12 acquires distance image data including information on the position (two-dimensional position) of object F on a plane perpendicular to the virtual straight line and information on distance K from sensor 12 to object F. Sensor 12 acquires, for example, distance image data. Sensor 12 outputs the acquired distance image data to control unit 20 at a predetermined period (for example, 1 / 30 seconds).

[0040] As a specific example, the sensor 12 irradiates light rays (or infrared rays) to each point on an object present in a shooting area including the target object F, and receives the light rays reflected from each point on the object. Then, the sensor 12 measures the distance between the sensor 12 and each point on the object based on the received light rays, and outputs the measured distance for each pixel. The distance between the sensor 12 and each point on the object may be measured by, for example, a Light Coding method.

[0041] In the Light Coding method, the sensor 12 irradiates light rays in a random dot pattern to each point on an object present in a shooting area including the target object F. The sensor 12 then receives the light rays reflected from each point on the object and measures the distance between the sensor 12 and each point on the object by detecting distortion of the pattern of the reflected light rays. The sensor 12 detects information on the two-dimensional position of each point on the object and information on the distance from the sensor 12 to each point on the object as multiple pixels, and outputs the detected multiple pixels to the control unit 20.

[0042] The control unit 20 is capable of communicating with each of the sensor 12 and the display device 15. The control unit 20 includes, for example, a CPU (Central Processing Unit) that executes a program, a storage unit including a ROM (Read Only Memory) and a RAM (Random Access Memory), an input / output unit, and a driver. Each function of the control unit 20 is realized by operating the input / output unit under the control of the CPU and reading or writing data in the storage unit. There are no particular limitations on the form and location of the control unit 20.

[0043] Fig. 4 is a functional block diagram of the control unit 20. As shown in Fig. 3 and Fig. 4, the control unit 20 has, as functional components, an image output unit 21, an object detection unit 22, a recognition unit 23, a signal output unit 24, and a notification unit 25. The image output unit 21 outputs a control signal of image data of an image to be displayed on the display device 15 to the display device 15. The display device 15 is capable of displaying various types of images based on the control signal of the image data from the image output unit 21.

[0044] The object detection unit 22 detects the object F based on the range image data output from the sensor 12. When the object detection unit 22 detects the object F, it outputs position data indicating the position of the object F to the recognition unit 23. Based on the position data output from the object detection unit 22, the recognition unit 23 recognizes that the button 11 has been pressed by the object F.

[0045] The notification unit 25 receives an operation signal from the recognition unit 23 when the recognition unit 23 recognizes that the button 11 has been pressed. The notification unit 25 is a notification means that notifies the user of the aerial operation device 1 that the button 11 has been operated when the recognition unit 23 recognizes that the button 11 has been operated. The notification unit 25 has, for example, an audio output unit 25b and a color change unit 25c.

[0046] As an example, the audio output unit 25b is a speaker, and outputs audio when an operation signal is received from the recognition unit 23. By hearing the audio from the audio output unit 25b, the user can understand that the button 11 has been operated. The color change unit 25c generates a color change signal when an operation signal is received from the recognition unit 23, for example, and outputs the color change signal to the display device 15.

[0047] When the display device 15 receives the color change signal from the color change unit 25c, it changes the color of the button 11 pressed by the user, for example. When the user visually confirms that the color of the button 11 has been changed, the user can understand that the button 11 has been operated. Note that the display device 15 may change the color of a portion other than the button 11 when it receives the color change signal from the color change unit 25c.

[0048] Providing the above-described notification unit 25 allows the user to know that the button 11 has been operated. However, at least one of the audio output unit 25b and the color change unit 25c may be omitted from the notification unit 25. Furthermore, the notification unit 25 may notify the user that the button 11 has been operated in a manner other than audio output by the audio output unit 25b or color change by the color change unit 25c.

[0049] When the recognition unit 23 recognizes that the button 11 has been pressed, the signal output unit 24 generates a control signal based on the pressing operation of the button 11. The signal output unit 24 transmits the generated control signal to a device 30 external to the aerial operation device 1, and causes the device 30 to operate by transmitting the control signal to the device 30.

[0050] The device 30 may be, for example, a toilet B with a washlet (see FIG. 1), each part of the washlet, or a washbasin M. As an example, the device 30 may be the toilet B, which may operate to flush the toilet when a control signal is received from the signal output unit 24. In this way, in the exemplary aerial operation device 1, each part of the toilet T can be operated by operating the button 11 displayed as a virtual image. Therefore, since there is no need to directly (physically) press a button or the like to operate each part of the toilet T, operation can be achieved in a hygienic manner.

[0051] For example, the recognition unit 23 may disable the button 11 to suppress output of a control signal by the signal output unit 24 when it recognizes that the object F detected by the sensor 12 is at the position of the button 11. In addition, the recognition unit 23 may enable the button 11 to allow output of a control signal by the signal output unit 24 when it recognizes that the object F detected by the sensor 12 is moving away from the button 11.

[0052] The recognition unit 23 may also determine whether the distance K between the sensor 12 and the object F is equal to or less than a threshold Y1, and whether the distance K between the sensor 12 and the object F is equal to or less than a threshold Y2 that is smaller than the threshold Y1. In this case, the recognition unit 23 determines whether the object F has reached a first surface Z1 that is a pressing determination surface, where the distance K from the sensor 12 is the threshold Y1, and whether the object F has reached a second surface Z2 that is a pressing determination surface, where the distance K from the sensor 12 is the threshold Y2.

[0053] Each of the first surface Z1 and the second surface Z2 is a virtual surface formed at a location at a constant distance from the sensor 12. As shown in Fig. 5, each of the first surface Z1 and the second surface Z2 has, for example, a circular shape. Each of the first surface Z1 and the second surface Z2 is provided, for example, at a position close to the button 11. At least one of the first surface Z1 and the second surface Z2 may coincide with the position of the button 11, or both the first surface Z1 and the second surface Z2 may not coincide with the position of the button 11.

[0054] The distance from the first surface Z1 to the second surface Z2 is, for example, half the distance between two adjacent sensors 12. Note that Fig. 3 shows an example in which the first surface Z1 is provided closer to the user than the button 11, and the second surface Z2 is provided closer to the sensor 12 than the button 11.

[0055] As described above, the sensor 12 detects the position of the object F multiple times. As an example, the signal output unit 24 outputs a control signal to the device 30 only when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 sandwiching the display surface 10. The sensor 12 also measures the distance K from the sensor 12 to the object F multiple times. For example, the notification unit 25 notifies the user of the aerial operation device 1 that the button 11 has been pressed when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 sandwiching the display surface 10.

[0056] Then, the signal output unit 24 outputs a control signal to the device 30 only when the average value of the measured distance K from the sensor 12 to the object F is equal to or greater than the distance from the sensor 12 to the second surface Z2 (threshold Y2) and equal to or less than the distance from the sensor 12 to the first surface Z1 (threshold Y1). As a specific example, when the sensor 12 measures the distance K n times (n is a natural number), the signal output unit 24 outputs a control signal to the device 30 when the average value of the distance K is equal to or greater than the threshold Y1 and equal to or less than the threshold Y2. The value of n is, for example, equal to or greater than 3 and equal to or less than 10 (5, as an example).

[0057] The signal output unit 24 may output a control signal to the equipment 30 only when the recognition unit 23 recognizes that the position of the object F detected by the sensor 12 is the first surface Z1 and then the recognition unit 23 recognizes that the position of the object F detected by the sensor 12 is the second surface Z2 closer to the sensor 12 than the first surface Z1.

[0058] The signal output unit 24 may be configured not to output a control signal to the device 30 when the multiple sensors 12 simultaneously detect an object F. Furthermore, at least one of the multiple sensors 12 may be a dummy sensor, and the signal output unit 24 may be configured not to output a control signal to the device 30 when the position of the object F is detected by the dummy sensor. The dummy sensor detects, for example, the position of the object F approaching a position adjacent to the button 11 on the display surface 10. As described above, by configuring the signal output unit 24 not to output a control signal under a predetermined condition, it is possible to suppress malfunction of the device 30.

[0059] FIG. 6 is a block diagram showing a schematic example of the relationship between the multiple buttons 11 and the recognition unit 23. As exemplified in FIG. 6, the multiple buttons 11 form a group including at least one button 11. The number of groups may be single or multiple. A group may be made up of multiple buttons 11 that are in a touchable relationship on the same line. In the example of FIG. 6, group G1 including buttons 11A and 11B, group G2 including buttons 11C, 11D, and 11E, and group GX (X is a natural number equal to or greater than 3) including button 11N are formed.

[0060] Fig. 7 is a flowchart of a recognition method in which the recognition unit 23 recognizes the operation of the buttons 11 grouped as described above. As shown in Fig. 7, first, the recognition unit 23 recognizes the group of the touched button 11. For example, when the object F detected by the sensor 12 is at the position of the button 11A (or button 11B), the recognition unit 23 recognizes the area of ​​the group G1. At this time, for example, the recognition unit 23 recognizes the area on the same touch line in the group G1 (step S1).

[0061] When the recognition unit 23 recognizes that the object F detected by the sensor 12 is at the position of the button 11A (or button 11B) of the group G1, it invalidates all the buttons 11 (button 11A and button 11B) of the group G1. That is, it invalidates all the buttons in the aforementioned area (step S2). Then, the recognition unit 23 sets all the buttons (button 11A and button 11B) in the group G1 (for example, the aforementioned area) as possible valid candidates (step S3).

[0062] Incidentally, for example, the user may first intend to press the button 11A, but may realize that he or she intended to press the button 11B when he or she pressed it. At that time, the user may not remove the object F, such as a finger, from the display surface 10, but may move the object F to the button 11B while keeping the object F on the display surface 10, and may want to operate the button 11B when the object F is removed from the button 11B. At this time, the recognition unit 23 determines whether the object F detected by the sensor 12 is removed from the button 11B, that is, whether the position of the object F is at the touch release button coordinates (step S4). When the recognition unit 23 recognizes that the object F detected by the sensor 12 is removed from the button 11B, that is, when the position of the object F is at the touch release button coordinates, the recognition unit 23 enables the button 11B to allow the signal output unit 24 to output a control signal (step S5), and ends the series of steps. On the other hand, when the recognition unit 23 does not recognize that the object F detected by the sensor 12 is being removed from the button 11B, that is, when the position of the object F is not at the touch-release button coordinates, the recognition unit 23 ends the series of steps.

[0063] Next, the effects of the aerial operation device 1 according to this embodiment will be described. In the aerial operation device 1, the aerial imaging device 14 displays the display surface 10 with multiple buttons 11 as a virtual image in the air, and the sensor 12 detects the position of an object F, such as a finger, approaching the button 11. The recognition unit 23 recognizes whether the button 11 has been pressed or not based on the position of the object F detected by the sensor 12. The signal output unit 24 outputs a control signal corresponding to the pressed button 11 to an external device 30 of the aerial operation device 1. The recognition unit 23 disables the operation of the multiple buttons 11 when the object F is at the position of the button 11. Therefore, by disabling the multiple buttons 11 at the timing when the button 11 is pressed, it is possible to prevent an unintended operation caused by an unintended button 11 reacting.

[0064] Furthermore, when the object F leaves the button 11, the recognition unit 23 enables the button 11 to allow the signal output unit 24 to output a control signal. Therefore, by enabling the button 11 at the timing when the object F leaves the button 11, it is possible to make only the intended button 11 react and realize the intended operation. Therefore, it is possible to suppress erroneous operations and improve the operability of the button 11.

[0065] The multiple buttons 11 may be configured into groups G1, G2, and GX each including at least one button 11, and the recognition unit 23 may disable the operation of all the buttons 11 in the groups G1, G2, and GX when the object F detected by the sensor 12 is at the position of the button 11 in the groups G1, G2, and GX. For example, the recognition unit 23 disables all the buttons 11 (buttons 11A and 11B) in the group G1 when the object F is at the position of the button 11A belonging to the group G1. By disabling all the buttons 11 belonging to the group G1 at the timing when a button in the group G1 is pressed, it is possible to prevent other buttons 11 in the group G1 from reacting and causing an unintended operation. As a specific example, it is possible to prevent the user from feeling that the button 11B reacted when he or she intended to press the button 11A.

[0066] As illustrated in FIG. 3, in the aerial operation device 1, the signal output unit 24 outputs a control signal to the device 30 only when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 sandwiching the display surface 10 on which an image is formed by the aerial imaging device 14. That is, only when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2, the signal output unit 24 outputs a control signal to the device 30, and the device 30 operates. Therefore, if the sensor 12 does not detect the object F located between the first surface Z1 and the second surface Z2 multiple times, the signal output unit 24 does not output a control signal, so that erroneous detection of the object F can be suppressed. As a result, erroneous operation of the device 30 can be suppressed.

[0067] The aerial operation device 1 includes a notification unit 25 that notifies the user of the aerial operation device 1 that the recognition unit 23 has recognized the button 11, and the notification unit 25 may notify the user that the button 11 has been pressed when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2 that sandwich the display surface 10. In this case, the user is notified that the button 11 has been pressed when the sensor 12 detects multiple times that the object F is located between the first surface Z1 and the second surface Z2. Therefore, the user can be notified that the button 11 has been correctly pressed, thereby improving the operability of the button 11.

[0068] The sensor 12 may measure the distance K from the sensor 12 to the object F multiple times, and the signal output unit 24 may output a control signal to the device 30 only when the average value of the measured distances K from the sensor 12 to the object F is equal to or greater than the distance from the sensor 12 to the second surface Z2 and equal to or less than the distance from the sensor 12 to the first surface Z1. In this case, the signal output unit 24 outputs a control signal to the device 30 only when the average value of the measured distances K is equal to or greater than the distance from the sensor 12 to the second surface Z2 and equal to or less than the distance from the sensor 12 to the first surface Z1. Therefore, when the average value of the distance K is shorter than the distance from the sensor 12 to the second surface Z2 or longer than the distance from the sensor 12 to the first surface Z1, the signal output unit 24 does not output a control signal. Therefore, it is possible to suppress erroneous detection of the object F and more reliably suppress malfunction.

[0069] The aerial operation device 1 includes a plurality of sensors 12, and the signal output unit 24 does not have to output a control signal to the equipment 30 when the plurality of sensors 12 detect an object F. In this case, the output of a control signal is suppressed when the plurality of sensors 12 detects an object F, so that erroneous detection and malfunction can be more reliably suppressed.

[0070] In the aerial operation device 1, the signal output unit 24 may output a control signal to the device 30 only when the recognition unit 23 recognizes that the position of the object F is the first surface Z1, and then recognizes that the position of the object F is the second surface Z2 closer to the sensor 12 than the first surface Z1. In this case, a control signal is output to the device 30 only when it is recognized that the object F has moved from the first surface Z1 to the second surface Z2 closer to the sensor 12. Therefore, since a control signal is not output to the device 30 except when the object F moves from the first surface Z1 to the second surface Z2, it is possible to suppress erroneous detection of the object F and malfunction of the device. As a specific example, it is possible to suppress erroneous detection of the button 11 when the clothes of a user using the toilet T pass over the display surface 10, when the hand of a user trying to pick up toilet paper E passes over the display surface 10, when the hand passes over the display surface 10 when cleaning the toilet T, and when a small insect passes over the display surface 10.

[0071] The aerial operation device 1 may include a plurality of sensors 12, and the distance from the first surface Z1 to the second surface Z2 may be half the distance between two adjacent sensors 12. In this case, a control signal for the device 30 is output only when the target object F moves half the distance between the two sensors 12 from the first surface Z1 to reach the second surface Z2. Therefore, the output of the control signal for the device 30 can be limited, thereby suppressing malfunction.

[0072] The aerial operation device 1 includes a dummy sensor that detects the position of an object F approaching a position adjacent to the button 11 on the display surface 10, and the signal output unit 24 does not need to output a control signal to the device 30 when the position of the object F is detected by the dummy sensor. In this case, the output of a control signal to the device 30 is restricted when the dummy sensor detects the object F in a position adjacent to the button 11. Therefore, erroneous detection of the object F and malfunction of the device 30 can be more reliably suppressed.

[0073] The above describes the embodiment of the aerial manipulation device according to the present disclosure. However, the aerial manipulation device according to the present disclosure is not limited to the above-mentioned embodiment, and may be modified or applied to other things within the scope of the gist described in each claim. That is, the configuration, shape, size, number, material and arrangement of each part of the aerial manipulation device can be appropriately changed within the scope of the gist described above.

[0074] For example, in the above embodiment, the sensor 12 measures the distance between the sensor 12 and each point on the object by the Light Coding method, but the method is not limited to this method. For example, the sensor 12 may measure the distance between the sensor 12 and each point on the object by the TOF (Time Of Flight) method. In the TOF method, the sensor 12 calculates the flight time (delay time) of a light ray from being reflected at each point on the object until it reaches the sensor 12, and measures the distance between the sensor 12 and each point on the object from the calculated flight time and the speed of light. Even in such a form, the same effect as the above embodiment is achieved. Furthermore, the type of the sensor is not limited to a depth sensor. In other words, instead of the sensor 12 which is a depth sensor, an infrared sensor or an ultrasonic sensor may be provided, and the type of the sensor can be changed as appropriate.

[0075] In the above embodiment, an example has been described in which the aerial imaging device 14 and the display device 15 display the display surface 10 on which a plurality of buttons 11 are arranged in a horizontal row at the bottom and text information 13 is displayed above the buttons 11. However, the layout of the buttons and the like on the display surface is not limited to the above example and can be changed as appropriate.

[0076] Next, an aerial manipulation device 41 according to a modified example will be described with reference to Figs. 8 and 9. Fig. 8 is a perspective view showing the aerial manipulation device 41 according to the modified example. Fig. 9 is a vertical cross-sectional view showing a schematic internal structure of the aerial manipulation device 41. As shown in Figs. 8 and 9, the aerial manipulation device 41 displays a display surface 50 having a plurality of buttons 51, which are a numeric keypad, as a virtual image in the air. The aerial manipulation device 41 has a columnar shape. The aerial manipulation device 41 extends in a first direction A1, a second direction A2, and a third direction A3.

[0077] The first direction A1 corresponds to the depth direction (front-back direction) of the aerial manipulating device 41, the second direction A2 corresponds to the width direction of the aerial manipulating device 41, and the third direction A3 corresponds to the height direction of the aerial manipulating device 41. The length (height) of the third direction A3 of the aerial manipulating device 41 is longer than the length (depth) of the first direction A1 of the aerial manipulating device 41 and the length (width) of the second direction A2 of the aerial manipulating device 41. This makes it possible to further improve the visibility of the display surface 50 to the user.

[0078] The aerial operation device 41 includes a housing 42 that constitutes the outer surface of the aerial operation device 41. The housing 42 includes, for example, a top surface 42b facing vertically upward, a pair of side surfaces 42c aligned along the second direction A2, a front surface 42d facing one side of the first direction A1, a back surface 42f facing the other side of the first direction A1, and a bottom surface 42g facing vertically downward. Furthermore, the housing 42 includes an inclined surface 42h located above the front surface 42d.

[0079] A rectangular opening is formed on the inclined surface 42h, through which the aerial imaging device 54 is exposed. The inclined surface 42h is oriented obliquely upward, for example. The display surface 50 on which an image is formed by the aerial imaging device 54 is displayed at a position closer to the user than the inclined surface 42h, and therefore the visibility of the display surface 50 can be improved.

[0080] The buttons 51 on the display surface 50 are buttons for operating an automated teller machine (ATM). The display surface 50 with the buttons 51 is a floating image displayed as if it is floating in the air. In other words, the display surface 50 with the buttons 51 is displayed as a virtual image in the air. Therefore, a user of the aerial operation device 41 can operate the automated teller machine in a hygienic manner since he or she does not need to directly touch a touch panel or the like.

[0081] For example, display surface 50 includes a plurality of buttons 51, and a user can operate an automated teller machine by operating the plurality of buttons 51. Housing 42 has embedded therein, for example, a sensor 43 that detects an object F approaching button 51. The configuration of sensor 43 is similar to the configuration of sensor 12 described above.

[0082] The aerial operation device 41 includes, for example, a plurality of sensors 43. The sensors 43 are provided corresponding to, for example, the buttons 51, and each button 51 is displayed diagonally above the corresponding sensor 43. As an example, the number of buttons 51 and sensors 43 is 12. The display surface 50 is displayed at a position closer to the user than the aerial imaging device 54, and is displayed diagonally above the aerial imaging device 54. This makes it easier for the user to view the display surface 50.

[0083] For example, the display surface 50 may be displayed with light, in which case the display surface 50 can be easily viewed regardless of the brightness of the surroundings. The display surface 50 has, for example, a shape that is elongated vertically. As an example, the display surface 50 is displayed in a rectangular shape, and the vertical length of the display surface 50 is longer than the horizontal length of the display surface 50. However, the shape and size of the display surface 50 are not limited to the above example, and are not particularly limited. The display surface 50 may display something other than the button 51. For example, the display surface 50 may include multiple buttons 51 and text information. As an example, the text information may include at least one of date information and time information.

[0084] On the display surface 50, for example, a plurality of buttons 51 are displayed in a grid arrangement. As one example, the plurality of buttons 51 are displayed as a numeric keypad. In this case, by pressing the button 51 of the aerial operation device 41, it becomes possible to designate an amount to be debited, an amount to be deposited, an amount to be transferred, or a PIN number for the automated teller machine.

[0085] The shape of the button 51 is, for example, a square shape (one example is a rectangle with rounded corners). However, the shape of the button 51 may be a circle, an oval, or a polygon, and is not particularly limited. For example, the button 51 may include at least one of a decision button and a back button. In this case, the decision button corresponds to a button that decides an operation, and the back button corresponds to a button that goes back to the previous operation. However, the type of the button 51 is not limited to the above examples, and is not particularly limited.

[0086] The aerial operation device 41 includes an aerial imaging device 54 that displays a display surface 50 with buttons 51, a display device 55 arranged inside the housing 42, and a control unit 60. The aerial imaging device 54 and the display device 55 correspond to a virtual image display unit that displays the display surface 50 as a virtual image in the air. The aerial imaging device 54 is, for example, a retroreflective member (retroreflector) fixed to an opening formed in the inclined surface 42h of the housing 42. The display device 55 is arranged obliquely with respect to the aerial imaging device 54. The configurations of the aerial imaging device 54, the display device 55, and the control unit 60 are the same as the configurations of the aerial imaging device 14, the display device 15, and the control unit 20 described above.

[0087] The aerial manipulation device 41 of the modified example installed in an automated teller machine has been described above, but the aerial manipulation device 41 also provides the same operational effects as the aerial manipulation device 1. The operation input device according to the present disclosure can be further modified with respect to the aerial manipulation device 41 of the modified example. For example, in the above-mentioned modified example, the aerial manipulation device 41 installed in an automated teller machine has been described. However, the aerial manipulation device according to the present disclosure may be installed in a kitchen, a bathroom, a hospital, a public institution, a station, an airport, or the like.

[0088] For example, in an aerial operation device installed in a kitchen, a display surface with buttons is displayed as a virtual image in the air, so that a cooking app can be operated without directly touching the display with fingers with ingredients on them. In other words, in the past, in order to touch the display of a mobile terminal or the like, one had to take the trouble of washing one's hands before operating a cooking app displayed on the display, but in the aerial operation device according to the present disclosure, a display surface with buttons is displayed as a virtual image in the air, so that the cooking app can be operated by operating the buttons in the air. Therefore, since the cooking app can be operated without taking the trouble of washing one's hands, an aerial operation device with high operability can be obtained.

[0089] For example, in an aerial operation device installed in an ATM, hospital, public institution, station, or airport, a display surface with buttons is displayed as a virtual image in the air, making it possible to operate an ATM, or a reception machine at a hospital or airport, without directly touching the shared display. Therefore, since it is not necessary to touch the shared display with fingers, operation can be performed hygienically. [Explanation of symbols]

[0090] 1,41...Aerial operation device, 2,42...Casing, 2b...Front surface, 2c...Side surface, 2d...Top surface, 2f...Bottom surface, 2g...Opening, 2h...First plate part, 2j...Second plate part, 3...Protrusion part, 3b...Main surface, 3c...End surface, 3d...Back surface, 10,50...Display surface, 10b...Center, 11,11A ,11B,11C,11D,11E,11N,51...Button, 11b...Center, 12,43...Sensor, 13...Text information, 14,54...Aerial imaging device, 15,55...Display device, 20,60...Control section, 21...Image output section, 22...Target detection section, 23...Recognition unit, 24...signal output unit, 25...notification unit, 25b...audio output unit, 25c...color change unit, 30...device, 42b...top, 42c...side, 42d...front, 42f...rear, 42g...bottom, 42h...inclined surface, A1...first direction, A2...second direction, A3...third direction, B...toilet bowl, C...light, D1...up / down direction, D2...left / right direction, D3...projecting direction, E...toilet paper, F...object, G1,G2,GX...group, K...distance, M...sink, T...toilet, V...reference line, W...wall, Y1,Y2...threshold, Z1...first surface, Z2...second surface.

Claims

1. A display device that displays information by emitting light; an aerial imaging device that reflects light from the display device multiple times to display a display surface with multiple buttons as a virtual image in the air; a sensor for detecting a position of an object approaching the button; An aerial control device comprising: a recognition unit that recognizes whether the button has been pressed or not based on the position of the object detected by the sensor; A signal output unit that outputs a control signal corresponding to the button pressed by a user of the aerial operation device to an external device of the aerial operation device and controls the operation of the external device; Equipped with The recognition unit is disabling the plurality of buttons so as to suppress output of the control signal by the signal output unit when it is recognized that the object detected by the sensor is located at the position of the button; enabling the button to allow the signal output unit to output the control signal when it is recognized that the object detected by the sensor is moving away from the button; The sensor detects the position of the object multiple times; the signal output unit outputs a control signal to the device only when the recognition unit recognizes that the object is removed from the button, enables the button, and the sensor detects a plurality of times that the object is located between a first surface and a second surface sandwiching the display surface. Aerial control device.

2. the plurality of buttons constitute a group including at least one of the buttons; the recognition unit invalidates all of the buttons in the group when it recognizes that the object detected by the sensor is the position of the button in the group. The aerial control device according to claim 1 .

3. A notification unit that notifies a user of the aerial operation device that the recognition unit has recognized, the notification unit notifies the user that the button has been pressed when the sensor detects a plurality of times that the object is located between a first surface and a second surface sandwiching the display surface. The aerial control device according to claim 1 or 2.

4. The sensor measures a distance from the sensor to the object multiple times; the signal output unit outputs a control signal to the device only when an average value of the measured distances from the sensor to the object is equal to or greater than a distance from the sensor to the second surface and equal to or less than a distance from the sensor to the first surface. The aerial operation device according to any one of claims 1 to 3.

5. A plurality of the sensors are provided, The signal output unit does not output a control signal to the device when the plurality of sensors detects the object. The aerial operation device according to any one of claims 1 to 4.

Images