Aerial image display input device and aerial image display input method

Abstract

To realize a highly convenient input device allowing a user to intuitively operate an aerial image with a simple device configuration.SOLUTION: An aerial image display input device 1 comprises: an aerial image projection unit 2 having a rectangle display 3 for displaying a predetermined image inside thereof, having a rectangle image transmission plate 4 for projecting the image displayed on the display 3 onto a three-dimensional space projection surface 10 which is visible to a user outside thereof, and for forming the image displayed on the display 3 as an input guide screen (input operation button 30) for the user in the air; an input detection sensor 5 for detecting the user's aerial operation on the input guide screen; and a control part 7 for performing prescribed control.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an aerial image display input device and an aerial image display input method, and is suitable for use in an aerial image display input device and an aerial image display input method for inputting user operations on an image (aerial image) displayed in the air. [Background technology]

[0002] In recent years, communication methods that do not involve real images, such as virtual reality or augmented reality, have become widespread.

[0003] For example, as a technology for displaying an image in the air by focusing an image displayed from an image display device in the air, Patent Document 1 discloses an aerial image display device that displays an aerial image by arranging an image display device, a half mirror, and a retroreflective material.

[0004] Furthermore, for example, Patent Document 2 discloses a gesture operation device that displays an internally installed display object in an external three-dimensional space using an image combining plate, and recognizes hand movement gesture operations from information from a camera that captures an image of the user's hand and a distance sensor.

[0005] Also, for example, Patent Document 3 discloses a non-contact operation detection device equipped with a detection means for detecting whether or not an object is present at a plurality of attention points set on an aerial image displayed in space by an aerial image display device equipped with a light source such as an image, a retroreflective member, and a light branching member. Patent Document 3 further discloses that the detection means includes a detection light source that emits infrared light and is provided in the light source such as an image, and a photodiode that detects the infrared light that is returned from the attention points.

[0006] For example, Patent Document 4 discloses a proposal to place an input / output device equipped with an imaging mechanism having a display unit that displays an image to be imaged in the air, in which an input detection sensor is built in, and a proposal to equip the device with a detection device that detects the operator's face or line of sight. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2016 / 088683 [Patent Document 2] Japanese Patent Application Publication No. 2017-062709 [Patent Document 3] Japanese Patent Publication No. 2020-067707 [Patent Document 4] Japanese Patent Publication No. 2020-067838 Summary of the Invention [Problem to be solved by the invention]

[0008] As mentioned above, various proposals have been made for image display technologies for forming and displaying aerial images, and input detection technologies for allowing users to input operations into the displayed aerial image information. However, there have been issues with the convenience of operation, such as the fact that how users can input information into the image content displayed in the air is not widely used, particularly in three-dimensional space such as the air, and that inexperienced users are unable to determine whether their input operations are appropriate.

[0009] Furthermore, when detecting input using images captured by a camera or the like, a control unit with high processing power is required to perform image recognition processing to recognize a user's gesture as a specified input operation, which poses challenges in terms of processing time and product price.

[0010] The present invention has been made in consideration of the above points, and aims to propose an aerial image display input device and an aerial image display input method that realize a highly convenient input device that allows users to intuitively operate aerial images with a simple device configuration. [Means for solving the problem]

[0011] In order to solve the above problems, the present invention provides the following aerial image display input devices [1] to

[14] , and aerial image display input methods using these aerial image display input devices. [1] an aerial image projection unit having an internal rectangular display unit for displaying a predetermined image, an external rectangular image transmission plate for projecting the image displayed on the display unit onto a three-dimensional space projection surface visible to a user, and projecting the image displayed on the display unit into the air as an input guide screen for the user; an input detection sensor that detects an aerial operation on the input guide screen by the user; a control unit that performs predetermined control; In a floating image display input device having the three-dimensional space projection surface is a rectangular projection surface that faces a user's line of sight from a vicinity of one side of the rectangle of the image-transmitting plate at a predetermined angle with the image-transmitting plate, the input detection sensor is disposed on a straight line parallel to another side of the rectangle of the image-transmitting plate, the other side being located near the other side opposite to the one side of the rectangle of the image-transmitting plate; a plane formed by a line segment connecting the vicinity of the midpoint of a pair of sides of the rectangular projection surface that are not parallel to the image transmission plate and a line on which the input detection sensor is disposed is disposed as an input detection area; the display unit that displays the predetermined image displays input operation buttons in an input guide screen near a line segment that connects the midpoints of the rectangular projection surface; the input detection sensor continuously detects distance information indicating a distance to an object that falls within a detectable area obtained by extending the input detection area toward the rectangular projection surface; The control unit determines whether the input operation button has been pressed based on how the distance to the object changes over time relative to the distance to the input operation button on the three-dimensional space projection surface, based on distance information of the object continuously detected by the input detection sensor, and when it is determined that the input operation button has been pressed, changes the input operation button on the display unit to indicate that the button is in a pressed state. 1. A floating image display input device. [2] The input detection sensors are reflective distance measurement sensors, and N of the sensors are arranged in a line, and the display unit displays N or less input operation buttons, The control unit determines that the input operation button has been pressed when a change in the output signals of the N reflective distance measuring sensors indicates that the distance to the object has changed from a first range in which the distance is farther than the projection surface of the three-dimensional space to a second range in which the distance is closer than the projection surface of the three-dimensional space. The aerial image display input device according to [1] above, [3] a movement operation of moving an object in a predetermined operation direction parallel to the input guide screen is provided as the aerial operation for requesting a change of input content that can be input on the input guide screen; one or more of the N or less input operation buttons displayed by the display unit is a display change request button indicating the operation direction of the movement operation, The control unit determines whether the display change request button has been pressed by the movement operation based on change information between a first sensor output signal that detects the display change request button and a second sensor output signal that detects a portion adjacent to the display change request button, among the output signals of the N reflective distance measuring sensors, and when it determines that the display change request button has been pressed, changes the display of input operation buttons other than the display change request button on the display unit. The aerial image display input device according to [2] above, [4] The control unit determining that the movement operation on the display change request button has been performed when the distance of the object does not change from the first range to the second range even if there is a change in the first sensor output signal, and when the change in the second sensor output signal indicates that the distance of the object is within the first range; determining that a pressing operation has been performed on the display change request button when a change in the first sensor output signal indicates that the distance to the object has changed from the first range to the second range; When it is determined that either the moving operation or the pressing operation on the display change request button has been performed, it is determined that the display change request button has been pressed, and the display of the input operation buttons other than the display change request button on the display unit is changed. The aerial image display input device according to [3] above, [5] The aerial image projection unit and the input detection sensor are an integrated structure, the image transmission plate is substantially vertical, and the three-dimensional space projection surface is arranged to form a surface inclined toward the user's line of sight with respect to a horizontal plane, and the angle of the integrated structure is variable. The aerial image display input device according to any one of [1] to [4] above, [6] The aerial image display input device is connected to a handling processing device that performs media processing or a predetermined task with a user, and the control unit switches the image of the input operation button displayed on the display unit based on information from the handling processing device, and outputs determination information on pressing of the input operation button to the handling processing device. The aerial image display input device according to any one of [1] to [5] above, [7] the input detection sensors are arranged on M straight lines, two or more of which are located near another side of the rectangle of the image-transmitting plate that is opposite to the one side, and are parallel to the other side; M surfaces formed by line segments connecting the vicinity of the (M+1)th equidistant point of a pair of sides of the rectangular projection surface that are not parallel to the image transmission plate and M placement lines of the input detection sensors are arranged as the input detection area; The display unit that displays the predetermined image displays input operation buttons in an input guide screen near a line segment that connects the neighborhoods of the (M+1)-division points on the rectangular projection surface. The aerial image display input device according to any one of [1] to [6] above. [8] The three-dimensional space projection surface is a rectangular projection surface that faces the user's line of sight from the vicinity of one side of the rectangle of the image-transmitting plate, at an angle of approximately 75 degrees with the image-transmitting plate. The aerial image display input device according to any one of [1] to [7] above, [9] the display unit that displays the predetermined image displays the input operation buttons in the input guide screen as a three-dimensional perspective image having a shadow image in the thickness direction on the side opposite to one side of the rectangle of the image-transmitting plate; When the control unit determines that a pressing operation has been performed on the input operation button, the control unit displays the input operation button displayed by the display unit as a planar image without a shadow image in the thickness direction at a position moved by the length of the shadow image in the thickness direction of the input operation button when not in a pressed state, thereby indicating that the input operation button is in a pressed state. The aerial image display input device according to any one of [1] to [8] above,

[10] The input detection sensor has a plurality of pairs of infrared light emitting elements and light receiving elements, and is a reflected light distance sensor that calculates the distance to an object by the principle of triangulation based on the light receiving direction when the light receiving elements receive light emitted by the infrared light emitting elements and reflected by the object, and of two lines that are located near the other side opposite to the one side of the rectangle of the image-transmitting plate and are parallel to the other side, a plurality of the infrared light emitting elements are arranged on the line closer to the image-transmitting plate, and a plurality of the light receiving elements are arranged on the line farther from the image-transmitting plate, and the line connecting each pair of the infrared light emitting elements and the light receiving elements is arranged so as to intersect the two parallel lines at a right angle. The aerial image display input device according to any one of [1] to [9] above,

[11] The input detection sensor a projection surface touch sensor that detects, on the same surface as the three-dimensional space projection surface, a position of an object associated with an aerial operation by the user, when one side direction of the rectangle of the image-transmitting plate is defined as an X direction and a direction perpendicular to the X direction is defined as a Y direction, at least the position in the Y direction; a plurality of distance sensors arranged on a straight line parallel to the other side of the rectangle of the image-transmitting plate, the distance sensors being located near the other side opposite to the one side of the rectangle of the image-transmitting plate, and calculating the distance to the object; the display unit that displays the predetermined image displays input operation buttons in an input guide screen; the input detection sensor detects the presence or absence of an object in an area of ​​the input operation button by the projection surface touch sensor, and detects the amount of movement of the object in a pressing direction relative to the area of ​​the input operation button in the detectable area by the plurality of distance sensors; The control unit determines whether a pressing operation has been performed on the input operation button based on detection information on the presence or absence of an object detected by the input detection sensor and detection information on the amount of movement of the object in the pressing direction, and when it determines that a pressing operation has been performed on the input operation button, changes the input operation button displayed by the display unit to indicate that it is in a pressed state. The aerial image display input device according to any one of [1] to

[10] above.

[12] the aerial image projection unit and the input detection sensor are integrally structured, the image transmission plate is substantially vertical, and the three-dimensional space projection surface is disposed to form a surface inclined toward a user's line of sight with respect to a horizontal plane; The aerial image projection unit further includes triangular side guides on both sides thereof that cover the input detection sensor, the image transmission plate, and the three-dimensional space projection surface. The aerial image display input device according to any one of [1] to

[11] above. [Effects of the Invention]

[0012] According to the present invention, a highly convenient input device that allows a user to intuitively operate aerial images can be realized with a simple device configuration. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a perspective view of the appearance of an aerial image display input device 1 according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a side view of the aerial image display input device 1. [Figure 3] 1 is a block diagram showing an example of the configuration of an aerial image display input device 1. FIG. [Figure 4] 1A to 1C are diagrams illustrating examples of angle changes in the aerial image display input device 1. [Figure 5] 10 is a diagram illustrating an example of the arrangement and operation of reflected light distance sensors 51, 52, and 53. FIG. [Figure 6] 10 is a flowchart illustrating an example of a processing procedure for input determination processing. [Figure 7] 10A to 10C are diagrams illustrating another example of the arrangement and operation of reflected light distance sensors 51, 52, and 53. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure for a second input determination process. [Figure 9] FIG. 10 is an external view of the system when the aerial image display input device 1 is connected to a reception and settlement machine 94 for a hospital. [Figure 10] 10 is a diagram schematically showing processing between the aerial image display input device 1 and a control unit of a reception and settlement machine 94 in the system shown in FIG. [Figure 11] FIG. 10 is an external perspective view of an aerial image display input device 1A according to a second embodiment of the present invention. [Figure 12] 10 is a diagram showing a display example of input operation buttons 30 in the third embodiment. FIG. [Figure 13] FIG. 10 is an external perspective view of an aerial image display input device 1B according to a fourth embodiment. [Figure 14] FIG. 2 is a side view of the aerial image display input device 1B. [Figure 15] FIG. 10 is an external perspective view of an aerial image display input device 1C according to a fifth embodiment. [Figure 16]1A and 1B are diagrams showing examples of images displayed on a three-dimensional space projection surface 10 by aerial image display input device 1C. [Figure 17] FIG. 17 is a diagram showing the relationship between the display image shown in FIG. 16 and the arrangement of reflected light distance sensors of the aerial image display input device 1C. [Figure 18] FIG. 13 is an external perspective view of an aerial image display input device 1D according to a sixth embodiment. [Figure 19] FIG. 1 is a side view of aerial image display input device 1D. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. [Example]

[0015] Fig. 1 is a perspective view of the appearance of an aerial image display input device 1 according to a first embodiment (Example 1) of the present invention. Fig. 2 is a side view of the aerial image display input device 1. Note that the aerial image display input device 1 in Fig. 2 is shown in cross section as viewed from the side, making the internal structure easier to understand. Fig. 3 is a block diagram showing an example of the configuration of the aerial image display input device 1.

[0016] As shown in FIG. 1, the aerial image display input device 1 includes an aerial image projection unit 2, an input detection sensor 5, and a control unit 7, which are mounted in a housing 11 mounted on a housing base 12. FIG. 1 also schematically shows input operation buttons 30 and a user's operating hand 80 on a three-dimensional space projection surface 10. The three-dimensional space projection surface 10 is a projection surface in three-dimensional space on which an aerial image (input operation buttons 30) is displayed by the aerial image display input device 1, and the hand 80 is an example of a body part of a user 8 using the aerial image display input device 1. As will be described in detail later, in this embodiment, the user 8 can perform an input operation on the aerial image display input device 1 by placing his or her hand 80 at the position of the input operation button 30 projected on the three-dimensional space projection surface 10.

[0017] 3, the aerial image display input device 1 is configured to include an aerial image projection unit 2, an input detection sensor 5, and a control unit 7. The aerial image projection unit 2 has a display unit 3 and an image transmission plate 4 (light branching member 40, retroreflective member 41). The input detection sensor 5 has reflected light distance sensors 51, 52, and 53, which are reflective distance measurement sensors. The control unit 7 has a built-in speaker 74, and includes an input determination processing unit 71, an input / output I / F processing unit 72, and a screen control processing unit 73. The aerial image display input device 1 is connectable to a separate handling processing device 9, which is configured to include a control unit 91, a display unit 92, and a processing unit 93.

[0018] Each component of the aerial image display input device 1 will be described in detail below.

[0019] The aerial image projection unit 2 is a unit that projects an image from a display unit 3, such as a liquid crystal display, through an image transmission plate 4 by internal light reflection and transmission, and forms the image on a three-dimensional space projection surface 10. In this embodiment, within the aerial image display input device 1, a high-brightness liquid crystal display unit 3 is disposed at the top, and a light branching member 40, such as a half mirror, is disposed on the image transmission plate 4 at an angle of approximately 45 degrees relative to the display unit 3. Furthermore, a retroreflective member 41 is disposed that retroreflects light reflected by the light branching member 40 in the same direction. With this structure, the reflected light passes through the light branching member 40 and is formed on the three-dimensional space projection surface 10, forming an aerial image. Generally, the display unit 3 and the aerial image on the three-dimensional space projection surface 10 are line-symmetrical with respect to the image transmission plate 4.

[0020] Aerial image projection technology using the above configuration has been widely disclosed in the prior art documents mentioned above, and may employ configurations other than that of this embodiment. For example, in this embodiment, a light branching member 40 such as a half mirror is used in the image transmission plate 4, and an aerial image is formed in combination with the retroreflective member 41. However, as an alternative embodiment, an image transmission plate 4 that does not use the light branching member 40 has also been devised, and such an alternative embodiment can also achieve the formation of an aerial image in the same way as this embodiment.

[0021] In this embodiment, a small, 5-inch high-brightness LCD is disposed in the display unit 3, and the light branching member 40 is formed in a rectangular shape with a vertical surface as shown in Fig. 2, thereby forming a rectangular display area on the three-dimensional space projection surface 10 that is approximately the same size as the screen size of the 5-inch LCD in the display unit 3, and the angle θ1 (see Fig. 2) formed between the light branching member 40 and the three-dimensional space projection surface 10 is approximately 75 degrees. Note that the rectangle of the display area can also be expressed as a rectangle A1-A2-A3-A4 using the vertices A1, A2, A3, and A4 of the rectangle (see Fig. 1), and this notation method is also used for other rectangles described later.

[0022] In this embodiment, the input detection sensor 5 includes three reflected light distance sensors 51, 52, and 53. Each reflected light distance sensor 51, 52, and 53 has a light-emitting element (e.g., an infrared light-emitting element) and a light-receiving element, and is capable of detecting the distance to an object located in the optical axis direction of the light-emitting element and the light-receiving element. As shown in FIG. 1, the three reflected light distance sensors 51, 52, and 53 are arranged in a straight line (on the line segment B1-B2 in FIG. 1). Furthermore, as shown in FIG. 2, the reflected light distance sensors 51, 52, and 53 are arranged near the lower end of the light branching member 40, and are arranged so that an input detection surface 61 (i.e., a rectangle B1-B2-B3-B4 shown in FIG. 1) including an input detection direction 6 determined by the optical axis directions of the light-emitting element and the light-receiving element intersects with the three-dimensional space projection surface 10. Furthermore, the input detection direction 6 (input detection surface 61) and the light branching member 40 form an angle θ2. Line segments B3 and B4 of the intersection between the input detection surface 61 and the three-dimensional space projection surface 10 are positioned so as to be substantially identical to the line segment connecting the midpoints of sides A1-A4 and A2-A3 of the three-dimensional space projection surface 10. In this embodiment, the angle θ2 between the three-dimensional space projection surface 10 and the light branching member 40 is approximately 10 degrees. Furthermore, the reflected light distance sensors 51, 52, and 53 can acquire distance information when an object is present in the direction facing each light-emitting element and light-receiving element (see the solid arrowed line in FIG. 1 ). Therefore, when a user's hand 80, which is located near the three-dimensional space projection surface 10, is in the area indicated by the arrow, distance information to the hand 80 is acquired and the movement of the hand 80 is detected as an input.

[0023] As shown in the block diagram of Fig. 3, the control unit 7 has an input determination processing unit 71, an input / output I / F processing unit 72, and a screen control processing unit 73. The input determination processing unit 71 is connected to the input detection sensor 5 consisting of reflected light distance sensors 51, 52, and 53 and to the built-in speaker 74, and performs input determination processing to process inputs operated by the user on the input operation buttons 30. The input / output I / F processing unit 72 is connected to the handling processing device 9, a separate device that processes media and performs predetermined tasks with the user, and performs input / output I / F processing to send and receive screen information and input information. The screen control processing unit 73 is connected to the display unit 3, and performs screen control processing to control the screen display on the display unit 3.

[0024] The handling processing device 9 is a device that receives a user operation (input operation) on the aerial image display input device 1 and executes the handling processing desired by the user, and as shown in the block diagram of Fig. 3, includes a control unit 91, a display unit 92, and a processing unit 93. The control unit 91 controls the display unit 92 having a touch panel and the processing unit 93 that performs multiple handling operations.

[0025] Regarding the aerial image display input device 1 according to this embodiment configured as above, the operation of the user 8 on the aerial image display input device 1 and the internal processing of the aerial image display input device 1 will be described below.

[0026] In order to execute a desired handling on handling processing device 9, for example, when two types of handling (handling A and handling B) are prepared, user 8 begins the operation by selecting either handling A or handling B. At this time, control unit 91 of handling processing device 9 displays a selection screen for handling A or handling B and transmits a request for selection information for either to control unit 7. In control unit 7, screen control processing unit 73 displays two input operation buttons indicating handling A and handling B on display unit 3. As a result, aerial image projection unit 2 displays two input operation buttons 30 indicating the selection of handling A or handling B on three-dimensional space projection surface 10, as shown in FIG. 1 .

[0027] Next, the user 8 recognizes the input operation buttons 30 displayed on the three-dimensional space projection surface 10, holds his / her hand 80 over the top of either the input operation button 30 corresponding to the desired operation, either operation A or operation B (here, operation A), and performs an action as if pressing the button (button pressing action). This button pressing action is detected by the user's hand 80 via a reflected light distance sensor (here, reflected light distance sensor 51) that detects the selected input operation button 30 among the reflected light distance sensors 51, 52, and 53. Based on the detection by this reflected light distance sensor 51, the input determination processing unit 71 in the control unit 7 detects movement in the input detection direction 6 and determines that the button for operation A has been pressed. Then, in the control unit 7, the screen control processing unit 73 controls the screen display of the display unit 3 to change the color and shape of the button display and outputs a predetermined sound from the speaker 74 to notify the user that the displayed button has been pressed. Furthermore, in the control unit 7, the input / output I / F processing unit 72 sends selection information of operation A to the operation processing device 9.

[0028] According to this embodiment, user 8 can input the handling selection of handling processing device 9 in a touchless manner, without touching the display unit 92 (touch panel) provided on handling processing device 9 with his / her hand, thereby making handling processing device 9 and aerial image display input device 1 clean equipment. Furthermore, even in cases where operation using a touch panel is difficult, for example, even users wearing gloves or users with disabilities in their hands can easily operate it in a touchless manner, making it possible to provide a highly convenient input device.

[0029] Furthermore, the aerial image display input device 1 according to the present embodiment not only provides the above-mentioned effect of enabling general touchless operation, but also provides the following effect.

[0030] First, the three-dimensional space projection surface 10 that forms the aerial image is positioned at an angle θ1 (approximately 75 degrees) with the light branching member 40, so that it faces the line of sight below the viewpoint of the user 8, as shown in Figure 2. This makes it less susceptible to the influence of external intrusion light or reflected light, resulting in an aerial image with good visibility.

[0031] Furthermore, when the three-dimensional space projection surface 10 is near the user's hand 80 and is in an approximately horizontal position where the user 8 can naturally hold their palm over it as shown in Figure 2, the palm (hand 80) and the three-dimensional space projection surface 10 are nearly parallel, so the user can easily hold their hand 80 over the position of the input operation button 30, and it is also easy to naturally lower their palm in a vertical direction, thereby further improving operability.

[0032] Furthermore, the input detection direction 6 is at an angle θ3 of approximately 90 degrees with respect to the nearly horizontal direction in which the user 8 naturally holds his / her palm as shown in FIG. 2, and the input detection surface 61 is parallel to the direction in which the user 8 moves his / her palm (hand 80) to press a button, thereby improving the detection sensitivity (accuracy) of the "press" operation by the user 8.

[0033] In this embodiment, the input detection sensor 5 uses reflected light distance sensors 51, 52, and 53 to detect the movement of the palm of the user 8, but a device with similar ease of operation can also be obtained by using other sensor means such as a camera. In that case, however, the arrangement and detection method of the sensor means such as a camera will differ from when the reflected light distance sensors 51, 52, and 53 are used.

[0034] Furthermore, after pressing a button, the color or shape of the button display changes, and sound is output, allowing the user 8 to recognize that the operation has been recognized by the device visually or audibly instead of by touch.

[0035] By achieving the various effects described above, the aerial image display input device 1 of this embodiment can provide an input device with a simple device configuration that allows users to intuitively operate aerial images, is highly convenient, and has good response (operation sensitivity).

[0036] Furthermore, in this embodiment, as described above, a relatively small 5-inch LCD is provided on the display unit 3, and two input operation buttons 30 are displayed on the horizontal line segment B3-B4 on the three-dimensional space projection surface 10, thereby improving the user's operational feel. Specifically, in terms of the user's operational feel, if the three-dimensional space projection surface 10 is too large, the aerial image will be too far away from the background light branching member 40, making it difficult for some people to grasp the sense of distance of the aerial image. Also, if the three-dimensional space projection surface 10 is too small, the input operation buttons 30 will be too close to the light branching member 40, and there is a possibility that the hand 80 will touch the panel of the image-transmitting plate 4 while operating the buttons. Furthermore, if the input operation buttons 30 are arranged in the front-to-back direction on the three-dimensional space projection surface 10, the hand 80 will have to move diagonally in consideration of the front-to-back tilt of the three-dimensional space projection surface 10 (angle θ1 in FIG. 2 ), making operation difficult. The above-described configuration of this embodiment eliminates such a deterioration in the user's operational feel, allowing user 8 to operate in a situation with good visibility by using light branching member 40 in the background as a reference, and furthermore, the movement of hand 80 required for operation is a small, simple pressing motion without having to consider depth or tilt. In other words, the aerial image display input device 1 of this embodiment can achieve highly operable touchless operation.

[0037] The detailed configuration and processing method of the aerial image display input device 1 according to this embodiment will be described below with reference to FIGS.

[0038] FIG. 4 is a diagram illustrating an example of angle change in the aerial image display input device 1. As shown in FIG. 4(A), the aerial image display input device 1 is configured by mounting a housing 11 on a housing base 12. As shown in FIG. 4(B), the angle can be changed along the arc of the outer shape of the housing 11. In this embodiment, the light branching member 40 is movable from the vertical direction to a range of angle θ4 (e.g., approximately 10 degrees) in a clockwise rotation direction. Although not shown in detail, the aerial image display input device 1 of this embodiment may also be movable along the arc of the outer shape of the housing 11 in a similar range of angle θ4 (e.g., approximately 10 degrees) in the opposite direction (counterclockwise rotation direction) from FIG. 4(B). By configuring the angle of the housing 11 to be changeable in this way, visibility and operability can be optimized when the aerial image display input device 1 is to be positioned further below or above the line of sight of the user 8 shown in FIG. 2. More specifically, when it is desired to dispose the aerial image display input device 1 lower than in the state shown in Fig. 2, the visibility and operability can be made favorable by rotating the aerial image display input device 1 clockwise. On the other hand, when it is desired to dispose the aerial image display input device 1 higher than in the state shown in Fig. 2, the visibility and operability can be made favorable by rotating the aerial image display input device 1 counterclockwise. This will be described in detail below with reference to Figs. 4(C) and 4(D).

[0039] 4(C) is a diagram showing the line-of-sight angle θ5 (θ5A, θ5B) of the user 8 relative to the three-dimensional space projection surface 10 (10A, 10B) when the position and angle of the aerial image display input device 1 are changed in combination, and FIG. 4(D) is a diagram schematically showing the positional relationship between the user 8 and the aerial image display input device 1 when the above combination is changed. When attempting to position the aerial image display input device 1 in FIG. 2 lower than the state in FIG. 2 without changing the angle, the position of the three-dimensional space projection surface 10 simply moves downward, changing the line-of-sight angle θ5 of the user 8 relative to the three-dimensional space projection surface 10, and it is expected that the aerial image will be difficult for the user 8 to see. Here, when the aerial image display input device 1 is rotated clockwise, the three-dimensional space projection surface 10A produced by the aerial image display input device 1 after the movement is tilted clockwise from the three-dimensional space projection surface 10 before the movement, so that the line-of-sight angle θ5A from the user 8 can be maintained close to the line-of-sight angle θ5 at the initial position (in the case of FIG. 2), making it easier for the user 8 to see the aerial image (see FIG. 4(C)). Furthermore, if the aerial image display input device 1 is simply positioned downward, the distance from the user 8 to the aerial image (three-dimensional space projection surface 10A) becomes greater than in the state of FIG. 2, which may cause the user 8 to feel uncomfortable operating the aerial image. For this reason, when the aerial image display input device 1 is positioned downward, it is preferable to position it closer to the user 8 from the viewpoint of improving operability. 4(D), if it is desired to position the aerial image display input device 1 lower than in the state shown in Fig. 2, the aerial image can be displayed at an angle that is easy to see and at a position where the user 8 can easily operate it by rotating the aerial image display input device 1 clockwise and positioning it closer (front), thereby optimizing visibility and operability. On the other hand, if it is desired to position the aerial image display input device 1 in Fig. 2 higher than in the state shown in Fig. 2, without changing the angle, the position of the three-dimensional space projection surface 10 will simply move upward, changing the line-of-sight angle θ5 from the user 8 relative to the three-dimensional space projection surface 10, and it is expected that the aerial image will be difficult for the user 8 to see.Here, when the aerial image display input device 1 is rotated counterclockwise, the three-dimensional space projection surface 10B produced by the aerial image display input device 1 after the movement is tilted counterclockwise from the three-dimensional space projection surface 10 before the movement. This allows the line-of-sight angle θ5B from the user 8 to maintain a value close to the line-of-sight angle θ5 at the initial position (in the case of FIG. 2), making it easier for the user 8 to see the aerial image (see FIG. 4(C)). Furthermore, if the aerial image display input device 1 is simply positioned upward, the distance from the user 8 to the aerial image (three-dimensional space projection surface 10B) may become too close compared to the state of FIG. 2, which may cause the user 8 to feel uncomfortable operating the aerial image. For this reason, when the aerial image display input device 1 is positioned upward, it is preferable to position it toward the back as viewed from the user 8 in order to improve operability. Taking the above into consideration, as shown in Figure 4(D), if it is desired to position the aerial image display input device 1 higher than in the state shown in Figure 2, the aerial image display input device 1 can be rotated counterclockwise and positioned at the back (rear), thereby displaying the aerial image at an easy-to-view angle in a position where the user 8 can easily operate it with their outstretched hand 80, thereby optimizing visibility and operability.

[0040] 5A and 5B are diagrams illustrating an example of the arrangement and operation of reflected light distance sensors 51, 52, and 53. FIG. 5A schematically illustrates an example of the arrangement of two input operation buttons 30 and three reflected light distance sensors 51, 52, and 53, and further illustrates an image of a hand 80 of a user 8 pressing the left input operation button 30 (the "left" button) to perform an input operation. FIG. 5B is a graph showing the change in distance information detected by reflected light distance sensors 51, 52, and 53 in the situation shown in FIG. 5A, i.e., when user 8 performs an input operation by pressing the left button. The graph has time on the horizontal axis and distance information (sensor output, distance) on the vertical axis.

[0041] In the following description of the graphs, the distance information detected by the reflected light distance sensors 51, 52, and 53 will be referred to as distance information L, C, and R, respectively. In the graph of FIG. 5B, distance information L is represented by a dashed line, distance information C by a dashed-dotted line, and distance information R by a solid line. In the graph of FIG. 5B, the sensor output represents the distance from each of the reflected light distance sensors 51, 52, and 53. Specifically, the distance corresponding to the sensor output of "H2" is the distance to the aerial image (three-dimensional space projection surface 10). Therefore, when the sensor output changes below H2, it can be determined that a pressing action on the aerial image has been performed.

[0042] According to Figure 5(B), distance information L starts to change at 0.2 seconds and changes to the pressed position (distance of the aerial image) over the next 0.2 seconds. By acquiring this distance information L, reflected light distance sensor 51 can detect the pressing of left input operation button 30. At this time, adjacent distance information C also changes at the same time as distance information L, but distance information C does not change to the pressed position. Furthermore, distance information R does not change at all.

[0043] The above changes in the distance information L, C, and R are just an example, but there are various ways in which the user 8 presses the input operation button 30, resulting in various changes in the distance information L, C, and R. In response to such various changes in the distance information L, C, and R, the input determination processing unit 71 of the control unit 7 determines the input information for the input operation button 30 by performing the input determination process as follows.

[0044] In this example, in order to simplify the input determination process, for example, as shown on the vertical axis of Fig. 5(B), four determination levels (thresholds) of H0, H1, H2, and H3 are set for the distance information (sensor output), and the state is classified depending on which determination level the distance information (sensor output) falls between. Specific states to be classified are when the distance information (sensor output) is H1 to H0, it is a "no detection state," when it is H2 to H1, it is a "detection state above the aerial image," and when it is H3 to H2, it is a "detection state below the vicinity of the aerial image."

[0045] 6 is a flowchart showing an example of the processing procedure for the input determination process. The input determination processing unit 71 executes the processing according to the flowchart shown in FIG. 6 to determine the input information for the input operation button 30 from the distance information L, C, R using the determination levels H0, H1, H2, and H3.

[0046] 6, the processing group of steps S101 to S104 and S111 to S113 is represented as step S10, and the processing group of steps S201 to S208 and S211 to S218 (excluding steps S204 and S214) is represented as step S20. Step S10 is a process for performing button detection, and more specifically, a process for detecting that the hand 80 of the user 8 has started to operate any of the input operation buttons 30, which are aerial images. Step S20 is a process for detecting the pressed state of a button, and more specifically, a process for detecting that a pressing action has been performed on the input operation button 30 detected in step S10.

[0047] An overview of the processing in step S10 is as follows. The input determination processing unit 71 determines whether distance information L or distance information R has been detected (step S101). If distance information L is detected in step S101, it determines whether distance information L is a value between H2 and H1 and is smaller than distance information C, thereby detecting the start of an operation on the input operation button 30 (the "left" button) corresponding to distance information L (steps S102 to S104). On the other hand, if distance information R is detected in step S101, it performs a determination process similar to the process for distance information L described above on distance information R, thereby detecting the start of an operation on the input operation button 30 (the "right" button) corresponding to distance information R (steps S111 to S113).

[0048] Next, the input determination processing unit 71 executes the process of step S20 based on the process result of step S10 (steps S104 and S113). The process of step S20 is outlined below.

[0049] When the start of an operation on the input operation button 30 (the "left" button) corresponding to the distance information L is detected in step S10, the input determination processing unit 71 detects a pressing of the "left" button by using a timer (a 100 ms timer in this example) for determining a pressing operation to determine whether the distance information L is below H2 (the distance indicated by the distance information L is closer than the distance to the aerial image) for 100 ms or more (steps S201 to S203, S205 to S208). When a pressing operation is detected in the detection result, the input determination processing unit 71 outputs an input determination processing result indicating that the "left" button has been pressed (step S204).

[0050] On the other hand, if the start of an operation on the input operation button 30 (the "right" button) corresponding to the distance information R is detected in step S10, the input determination processing unit 71 detects a pressing action of the "right" button by performing a determination process on the distance information R similar to the process on the distance information L described above (steps S211 to S213, S215 to S218). If a pressing action is detected in the detection result, the input determination processing unit 71 outputs an input determination processing result indicating that the "right" button has been pressed (step S214).

[0051] In the aerial image display input device 1 according to the present embodiment, the input determination process described above is performed using three distance sensors (reflected light distance sensors 51-53). When detecting a user 8 pressing two buttons (input operation buttons 30) displayed on an aerial image, the pressing of the two buttons can be detected by one output from each of the distance sensor corresponding to the left (L) (reflected light distance sensor 51) and the distance sensor corresponding to the right (R) (reflected light distance sensor 53). This detection method is extremely accurate and can be performed using simple logic, allowing for high-sensitivity, high-speed processing. Furthermore, the distance sensors and control unit 7 do not require complex configurations for detection, making it possible to realize a compact device with a simple configuration.

[0052] Furthermore, in the detection method using the above-mentioned input determination process, the output of the sensor for distance information C (reflected light distance sensor 52) is monitored to detect the movement of the hand 80 or the like on parts other than the buttons, thereby preventing erroneous detection of erroneous operations other than button operations.

[0053] Note that the infrared light reflected light distance sensors 51-53 have limited detection areas, so if the display buttons (input operation buttons 30) are larger than the detection area, there may be a range where the input operation (button pressing action) cannot be detected, or if the user 8 operates with a slender fingertip, the detectable area may be small. Therefore, taking these situations into consideration, it is preferable that the button images displayed on the display unit 3 be buttons of a position and size that approximately matches the detection area of ​​the sensors. By configuring the button images displayed on the display unit 3 in this way, it is possible to realize an input device with excellent operability for the aerial image (input operation buttons 30) projected from the display unit 3 onto the three-dimensional space projection surface 10.

[0054] In this embodiment, we have explained up to this point the input operation of selecting handling input information by pressing a button as input by the user 8, but the input operations that the aerial image display input device 1 of this embodiment can handle are not limited to input operations by pressing a button, and for example, when there are multiple pieces of selectable information such as "Handling A," "Handling B," "Handling C," and "Handling D," it can also handle an operation to display an handling that is not displayed on the screen (hereinafter referred to as a "flipping" operation).

[0055] FIG. 7 is a diagram illustrating another example of the arrangement and operation of reflected light distance sensors 51, 52, and 53. Similar to FIG. 5A, FIG. 7A illustrates an example of the arrangement of three reflected light distance sensors 51, 52, and 53. However, while FIG. 5A shows two circular buttons (a "left" button and a "right" button) as input operation buttons 30, FIG. 7A shows a circular "left" button on the left side and an arrow-shaped "turn" button on the right side as input operation buttons 30. The "turn" button is a button that, when user 8 performs a predetermined "turn" operation, transitions the operations displayed on the "left" button in a predetermined order, thereby enabling the operation desired by user 8 to be displayed on the "left" button. For example, in FIG. 7A, assume that "Operation A" is displayed on the "left" button. Note that, although detailed processing procedures will be described later in Fig. 8, in this example, pressing the "turn" button (pressing operation) or moving horizontally from near the "turn" button according to the displayed arrow direction (swiping operation) can be accepted as a predetermined "turn" operation for the "turn" button. Fig. 7(B) is a graph showing the situation shown in Fig. 7(A), i.e., the change in distance information detected by reflected light distance sensors 51, 52, 53 when user 8 performs a swipe operation on the "turn" button, and the graph has time on the horizontal axis and distance information (sensor output, distance) on the vertical axis.

[0056] 7(B), distance information L starts to change at 0.2 seconds, then distance information C starts to change about 0.2 seconds later, and then distance information R starts to change about 0.2 seconds later. Each sensor (reflected light distance sensors 51 to 53) acquires this information.

[0057] Then, in response to changes in the distance information L, C, and R, the input determination processing unit 71 of the control unit 7 performs the following input determination process (referred to as second input determination process when distinguished from the input determination process described in FIGS. 5 and 6) to determine the input information for the input operation button 30. Note that in the second input determination process, in order to simplify the process, four determination levels (thresholds) H0, H1, H2, and H3 are provided to classify the distance information into three states, just like the input determination process described in FIGS. 5 and 6.

[0058] Fig. 8 is a flowchart showing an example of the processing procedure of the second input determination process. Note that the processing procedure shown in Fig. 8 has many parts in common with the processing procedure of the input determination process shown in Fig. 6, and therefore, a description of these common parts will basically be omitted.

[0059] 8, the processing group of steps S101 to S104, S111, and S113 is represented as step S30, and the processing group of steps S201 to S208, S211 to S213, and S402 to S407 (excluding steps S204 and S403) is represented as step S40. Step S30 is a process for detecting that hand 80 of user 8 has started to operate any of input operation buttons 30, which are aerial images, and step S40 is a process for detecting that a predetermined action (a pressing action in the case of the "left" button, or a pressing action or a swipe action in the case of the "turn" button) has been performed on the input operation button 30 detected in step S30.

[0060] The detailed processing procedure of step S30 is the processing procedure of step S10 in Fig. 6 with the processing of step S112 deleted. The processing of step S112 is to determine whether the distance information R corresponding to the right button is at a distance farther than the aerial image (button) as a pre-determination for detecting a pressing action when the distance information R corresponding to the right button is detected, but in the case of step S30, the right button is a "turn" button and there is no need to determine a pressing action (it is sufficient that the object to be detected (hand 80) is near the button), so step S112 is deleted. The detailed processing procedure is omitted to avoid repetition, but the processing of step S30 detects the start of an operation on the input operation button 30 corresponding to the distance information L or distance information R.

[0061] Next, the input determination processing unit 71 executes the process of step S40 based on the process result of step S30 (steps S104 and S113). The process of step S40 is outlined below.

[0062] If the start of operation on the input operation button 30 ("left" button) corresponding to the distance information L is detected in step S30, the input determination processing unit 71 detects a pressing action of the "left" button by performing the same process as in step S20 of Fig. 6 (steps S201 to S203, S205 to S208). If the input determination processing unit 71 detects a pressing action in the above detection result, it outputs an input determination processing result that the "left" button has been pressed (step S204).

[0063] On the other hand, if the start of an operation on the input operation button 30 (the "turn" button) corresponding to the distance information R is detected in step S30, the input determination processing unit 71 performs a process of detecting a pressing action of the "turn" button based on a change in the distance information R (steps S211 to S213) or a process of detecting a swipe action of the "turn" button based on a change in the distance information C (steps S402, S404 to S407) in order to detect whether a predetermined "turn" operation has been performed on the "turn" button. If the input determination processing unit 71 detects a pressing action in the detection result, it outputs an input determination processing result of "turn" pressing (step S401). If the input determination processing unit 71 detects a swipe action in the detection result, it outputs an input determination processing result of "turn" swipe (step S403). When the determination result of the "turn" operation is output in step S401 or step S403, the screen control processing unit 73 of the control unit 7 performs a process of switching the display image of the "left" button on the display unit 3 to the next action in accordance with a predetermined transition order. Specifically, for example, the display image is switched from "Handling A" to "Handling B."

[0064] In the aerial image display input device 1 according to the present embodiment, by performing the second input determination process as described above, even when selecting from three or more operations, the user 8 can select a desired operation by simply repeating two intuitive and simple operations: a swipe action to move horizontally and a press action to press vertically, using only two buttons (input operation buttons 30) based on the aerial image, one of which is a "turn" button, thereby enabling the user 8 to select a desired operation. Therefore, even users who are accustomed to swiping operations on smartphones, tablets, etc., or users who are unfamiliar with swipe operations and only accustomed to press operations, can easily perform touchless operation.

[0065] Next, an example of how to use the aerial image display input device 1 according to this embodiment will be described.

[0066] FIG. 9 is an external view of the system when the aerial image display input device 1 is connected to a reception and settlement machine 94 for a hospital.

[0067] The hospital reception and settlement machine 94 is an example of a handling processing device 9, and is a device that allows the user, a patient, to present their patient registration card, select the medical department to receive the data from, and carry out the reception process before receiving the data, and after the examination, check the consultation fee, insert cash equivalent to the settlement amount, and carry out the payment process to settle the consultation fee.

[0068] In the case of Fig. 9, the aerial image display input device 1 is placed to the right of the cash slot of the reception and settlement machine 94, so that it faces the user who operates it and is located below the user's line of sight as shown in Fig. 2. As shown in Fig. 9, the user can operate both input operation buttons 96 for "reception" and "settlement" displayed on the display unit 95 of the reception and settlement machine 94 and input operation buttons 30 for "reception" and "settlement" displayed as aerial images by the aerial image display input device 1. Therefore, the user can select a transaction by operating either button.

[0069] Fig. 10 is a diagram schematically illustrating the processing between the control units of the aerial image display input device 1 and the reception and settlement machine 94 in the system shown in Fig. 9. Fig. 10 shows the processing between the control units when user 8 selects the "reception" handling option.

[0070] In this example, three types of display screens (screens A1 to A3) that can be displayed on the aerial image display input device 1 are predefined between the control units of the aerial image display input device 1 and the reception and settlement machine 94. Furthermore, four types of display screens (screens B1 to B4) that can be displayed by the reception and settlement machine 94 are predefined.

[0071] As shown in Fig. 10, when a user starts operation, the aerial image display input device 1 receives an instruction to display screen A1, which is a selection screen between "reception" and "payment," from the reception and settlement machine 94, and displays screen A1 on the display unit 3. When an input operation is performed on screen A1, the aerial image display input device 1 transmits input key information to the reception and settlement machine 94. In addition, the reception and settlement machine 94 displays screen B1 on its own display unit 95.

[0072] When "Reception" is selected on screen A1 or screen B1, the reception and settlement machine 94 displays screen B2, which is a guide for reading the patient registration card, on the display unit 95, and transmits a display instruction for screen A2, which is a screen on which "Cancel" can be selected, to the aerial image display input device 1. In response to this display instruction, the aerial image display input device 1 displays screen B2 on the display unit 3.

[0073] Thereafter, when the reception and settlement machine 94 has finished reading the patient registration card, it displays screen B3, which is a screen for selecting a medical department, on the display unit 95, and transmits an instruction to display screen A3, which is a screen on which a medical department can be selected, to the aerial image display input device 1. In response to this display instruction, the aerial image display input device 1 displays screen A3 on the display unit 3. It is preferable that screen A3 displayed by the aerial image display input device 1 be a display screen that uses the "turn over" button, as described with reference to FIGS. 7 and 8. As shown in FIG. 10, when the "turn over" button on the right is operated, the treatment (medical department) is transitioned and displayed on the button on the left. When an input operation (an operation to select a medical department) is performed on screen A3, the aerial image display input device 1 transmits input key information to the reception and settlement machine 94.

[0074] Then, when a medical department is selected on screen B3 displayed on its own display unit 3, or when input key information for screen A3 (input information for the selected medical department) is received from aerial image display input device 1, reception and settlement machine 94 internally executes a predetermined reception process, and upon completion, displays screen B4 notifying completion of reception on display unit 95. This completes the reception process for one customer (patient).

[0075] As described above, the aerial image display input device 1 of this embodiment not only provides a user with an intuitive, responsive, and easy-to-use input device for aerial images in the form of a small device with a simple configuration, but also makes it possible to realize an easy-to-use touchless input device by connecting it to an existing handling processing device.

[0076] Furthermore, when the aerial image display input device 1 of this embodiment is connected to the handling processing device 9 (reception and settlement machine 94), the user can operate either the aerial image display input device 1 or the handling processing device 9. Therefore, even if the user is not suited to operating aerial images, they can simply operate the handling processing device 9, thereby providing a highly convenient input system.

[0077] Furthermore, in this embodiment, by configuring the aerial image display input device 1 separately from the handling processing device 9, it becomes easy to add the aerial image display input device 1 to various existing handling processing devices. In this case, not only in terms of hardware but also in terms of software, the screen and operation method specifications of the application software need not be changed, and only logic for processing input information from the aerial image display input device 1 needs to be added, which facilitates software modifications, simplifies implementation, and provides advantages in terms of cost. Furthermore, because the aerial image display input device 1 is a compact device that simply displays input buttons on the screen, it can be easily placed near each operation unit of various types of processing devices, minimizing the overall increase in installation space while ensuring operability. [Example]

[0078] 11 is a perspective view of the appearance of an aerial image display input device 1A according to a second embodiment (Example 2) of the present invention. The aerial image display input device 1A of Example 2 is different from Example 1 in the arrangement relationship between the input detection sensors and the input operation buttons, specifically, three input operation buttons 31 are arranged opposite three reflected light distance sensors 51, 52, and 53, which differs from the aerial image display input device 1 of Example 1 in that two input operation buttons 30 are arranged.

[0079] As shown in Figure 11, in the aerial image display input device 1A, the distance between each input operation button 31 is short, but the number of selectable types of handling is increased from two to three, which has the advantage of broadening the range of use of handling.

[0080] In the first embodiment described above, a total of three input detection sensors 5 (reflected light distance sensors 51, 52, 53) are arranged linearly on both sides of the two input operation buttons 30 and between them, and the sensors on both sides (reflected light distance sensors 51, 53) detect pressing operations of the respective input operation buttons 30, and the remaining sensor (reflected light distance sensor 52) detects the movement (swipe operation) of the hand 80 between the two input operation buttons 30. This makes it possible to prevent erroneous inputs caused by the movement of an object larger than the hand 80, such as the movement of the user's luggage or a part of the user other than the hand 80, through the input determination process, which is a simple logical process, shown in Fig. 6. On the other hand, in the second embodiment, three input detection sensors 5 (reflected light distance sensors 51, 52, 53) are arranged linearly facing the three input operation buttons 31, and there is a risk that inappropriate input determination processing will be performed in response to the above-mentioned erroneous operation due to the buttons being too close to each other. Therefore, in order to prevent this, it is preferable to provide a detection means and an input determination process different from those in the first embodiment.

[0081] In the aerial image display input device 1 (1A) according to the present invention, the same relationship between the number of buttons and the number of sensors basically holds even if the number of input operation buttons 30 (31) is increased. That is, in consideration of device performance and component installation costs, it is most efficient to adopt a configuration in which two or more (N) input detection sensors 5 are arranged in a line and less than N ([N-1] or less) input operation buttons 30 (31) are displayed. However, in principle, it is also possible to adopt a configuration in which N input detection sensors 5 are arranged and the same number N of input operation buttons 30 (31) are displayed.

[0082] In the present invention, if the number of input operation buttons is increased to three or more, the number of reflected light distance sensors constituting the input detection sensor may be increased accordingly. In this case, the reflected light distance sensor may be an integrated line-type detection sensor in which elements are arranged in a line. [Example]

[0083] In a third embodiment (Example 3) of the present invention, a configuration is assumed in which the aerial image display input device 1 described in Example 1 is connected to call buttons installed on each floor of an elevator in a building.

[0084] FIG. 12 is a diagram showing a display example of input operation buttons 30 in Example 3. In Example 3, the aerial image display input device 1 connected to the call buttons displays a call button image on the display unit 3, thereby displaying the input operation buttons 30 on the three-dimensional space projection surface 10. More specifically, as shown in FIG. 12 , an upper call button 30A for calling an elevator car for upward movement and a lower call button 30B for calling an elevator car for downward movement are displayed on the three-dimensional space projection surface 10. When a user 8 performs a predetermined operation (basically a press operation, but a swipe operation may also be accepted) on the upper call button 30A or the lower call button 30B, the aerial image display input device 1 is configured to directly output an input based on the operation to an elevator control device as button information of the elevator call button, thereby making it possible to output button information as a substitute for the elevator call button. As a result, the aerial image display input device 1 according to the third embodiment can switch the call push buttons installed on each floor of an existing elevator to touchless operation. [Example]

[0085] In a fourth embodiment (Embodiment 4) of the present invention, an aerial image display input device 1B in which the number of input operation buttons is increased compared to the above-described embodiments will be described.

[0086] Fig. 13 is a perspective view of the appearance of an aerial image display input device 1B according to Example 4. Fig. 14 is a side view of the aerial image display input device 1B. In Figs. 13 and 14, the same reference numerals are used to designate components common to Figs. 1 and 2, and descriptions thereof will be omitted.

[0087] Compared to the aerial image display input device 1 shown in FIGS. 1 and 2, the aerial image display input device 1B shown in FIGS. 13 and 14 has an input detection sensor 5B instead of the single input detection surface 61, and two input detection surfaces 62 and 63. The input detection sensor 5B is configured by five sensors (distance sensors each consisting of a light-emitting element and a light-receiving element) aligned on one axis and arranged across two axes. More specifically, five sensors are arranged on each of the line segments B1-B2 and C1-C2, thereby forming two input detection areas each consisting of an input detection surface 62 (B1-B2-B3-B4) and an input detection surface 63 (C1-C2-C3-C4). The three-dimensional space projection surface 10 is a surface including the line segments B1-B2 and C1-C2, and displays, for example, ten input operation buttons 32. Each button is assigned a number from "0" to "9," allowing a single-digit number to be selected and input.

[0088] The aerial image display input device 1B described above can be realized with the same configuration as the aerial image display input device 1 of Example 1, except for the increased number of sensors constituting the input detection sensor 5B. However, due to the two input detection surfaces 62 and 63, when a user 8 attempts to operate an input operation button 32 on the line segment B3-B4 displayed on the rear input detection surface 62, the hand 80 of the user 8 also blocks light from the front input detection surface 63, changing the output of the two-axis distance sensor. However, this problem can be technically easily solved by adding logic to the input determination process by the input determination processing unit 71 to determine which button is to be operated. Therefore, the aerial image display input device 1B of Example 4 can achieve the effect of increasing the number of input selection options and expanding the range of application, in addition to the effects achieved by the other examples. [Example]

[0089] In a fifth embodiment (Example 5) of the present invention, a description will be given of an aerial image display input device 1C in which five input operation buttons are arranged.

[0090] Fig. 15 is a perspective view of the appearance of an aerial image display input device 1C according to Example 5. Fig. 16 is a diagram showing an example of a display image on a three-dimensional space projection surface 10 by the aerial image display input device 1C. Fig. 17 is a diagram showing the relationship between the display image shown in Fig. 16 and the arrangement of reflected light distance sensors of the aerial image display input device 1C.

[0091] In aerial image display input device 1C of Example 5, the positional relationship between the input detection sensors (reflected light distance sensors) and the input operation buttons is changed from that of aerial image display input device 1 of Example 1. Specifically, five input operation buttons 33 to 37 are arranged opposite five reflected light distance sensors 54 to 58. Of these, the three input operation buttons 33, 34, and 35 arranged opposite three reflected light distance sensors 55, 56, and 57 are operation buttons corresponding to an input operation (selection operation) by pressing the button, and the two input operation buttons 36 and 37 arranged opposite two reflected light distance sensors 54 and 58 are operation buttons corresponding to a "flip" operation.

[0092] In addition, the side guides 13 shown in Figures 15 and 17 are, for example, plate-shaped members arranged on both sides of the housing 11, and have a projection surface portion 14 located on the side of the three-dimensional space projection surface 10 and a sensor surface portion 15 located on the side of the input detection sensor 5 (reflected light distance sensors 54 to 58).

[0093] Of the display images of the three-dimensional space projection surface 10 shown in Figure 16, the display image (three-dimensional space projection surface 10) in Figure 16(A) is the screen before operating the input operation buttons 33 to 37, and the display image (three-dimensional space projection surface 10) in Figure 16(B) is the screen after pressing the input operation button 35.

[0094] As shown in Figure 15, in the aerial image display input device 1C, the distance between each of the input operation buttons 33 to 37 is closer than in Example 1, but by having input operation buttons 36 and 37 that correspond to the "turning over" operation, the number of selectable handling types increases from two (handling A and handling B) to five or more (for example, handling A to handling E), which has the advantage of broadening the range of use of the handling.

[0095] In the fifth embodiment, the five input detection sensors 5 (reflected light distance sensors 54-58) are arranged at intervals of, for example, approximately 25 mm. Each of the reflected light distance sensors 54-58 has an infrared LED, which is a type of infrared light emitting element, built in on the light emitting side (as the light emitting element), and a position sensitive detector (PSD) built in on the light receiving side (as the light receiving element), and is a sensor that outputs an output according to the distance to the object using a triangulation method based on the light receiving direction when the PSD receives light emitted by the infrared LED and reflected by the object. As shown in Fig. 15, each sensor is arranged with the light emitting side (light emitting element) closer to the image transmission plate 4 and the light receiving side (light receiving element) farther away.

[0096] The five input detection sensors 5 (reflected light distance sensors 54-58) in Example 5 are arranged with the light-emitting and light-receiving sides rotated 90 degrees compared to the three input detection sensors 5 (reflected light distance sensors 51-53) shown in Example 1. The reflected light distance sensor has a detection area that is wide in the direction of the light-emitting and light-receiving elements and narrow in the direction perpendicular to that. The reflected light distance sensor used in this example has a detection area that is approximately 40 mm in the direction of the light-emitting and light-receiving elements and approximately 20 mm in the direction perpendicular to that, at a distance of approximately 100 mm from the element. To detect the operation of multiple buttons on the image, as in this example, it is preferable to prevent the detection areas of adjacent sensors from overlapping. For example, the distance between each pair of light-emitting and light-receiving elements (light-emitting element and light-receiving element) is approximately 25 mm, and the arrangement direction of the light-emitting and light-receiving elements in each pair is perpendicular (vertical) to the arrangement direction of multiple elements of the same type (light-emitting element, light-receiving element). As can be seen from Figures 15 and 17, multiple identical elements are arranged near the bottom edge of the image-transmitting plate 4, each on a straight line parallel to the bottom edge. Therefore, in this embodiment, it can also be said that the arrangement direction of the light-receiving and light-emitting elements in each set is perpendicular (vertical) to the bottom edge direction of the image-transmitting plate 4.

[0097] Furthermore, even when the input detection sensor 5 is made up of five reflected light distance sensors 54 to 58 as in this embodiment, by assigning operation buttons corresponding to input operations (selection operations) by pressing a button and operation buttons corresponding to a "turning over" operation to positions opposite each sensor, as in the first embodiment, it is possible to perform processing similar to the input determination processing described in the first embodiment.

[0098] Furthermore, by forming a surface that coincides with the three-dimensional space projection surface 10, the projection surface 14 of the side guide 13 can serve as a reference for the user's field of view of the operation image floating in the air (forming both ends of the operation image), as shown in Fig. 17, thereby improving the visibility of the operation image. Furthermore, by forming the sensor surface 15, the side guide 13 has the effect of preventing external light from the side from making the image difficult to see and preventing the input detection sensor 5 from malfunctioning.

[0099] In addition, in the display image shown in Fig. 16, the arranged input operation buttons 33 to 37 are displayed as a stereoscopic view showing three-dimensional thickness in the forward direction, and the pressed state of input operation button 35 (handling D) shown in Fig. 16(B) displays only the selected input operation button 35 at a position where a plan view with no three-dimensional thickness has been moved forward a distance equivalent to the thickness. By displaying this on the aerial image display input device 1C as shown in Fig. 17, the user can visually understand that the displayed button "D" has been pressed. [Example]

[0100] In a sixth embodiment (Example 6) of the present invention, a description will be given of an aerial image display input device 1D in which ten input operation buttons are arranged.

[0101] Fig. 18 is a perspective view of the appearance of an aerial image display input device 1D according to Example 6. Fig. 19 is a side view of the aerial image display input device 1D.

[0102] The aerial image display input device 1D of Example 6, like the aerial image display input device 1C of Example 5, has five reflected light distance sensors 54, 55, 56, 57, and 58 arranged as input detection sensors 5, and further has a configuration different from that of the aerial image display input device 1C of Example 5 in that a projection surface touch sensor 16 is arranged near the three-dimensional space projection surface 10.

[0103] The projection surface touch sensor 16 has an infrared emitting element and a light receiving element. The infrared emitting element emits infrared light in an upward direction on the three-dimensional space projection surface 10, and the light receiving elements are arranged in a line in the width direction of the three-dimensional space projection surface 10, and output the position on the three-dimensional space projection surface 10 in an obstructed state as an XY coordinate, with the width direction being X and the direction perpendicular to that being Y. Specifically, for example, a zForce (registered trademark) AIR touch sensor from neonodo Inc. can be used as the projection surface touch sensor 16.

[0104] In Example 6, when ten input operation buttons 38 as shown in FIG. 18 are displayed, the projection surface touch sensor 16 detects whether an object is blocking the position of each input operation button 38 on the three-dimensional space projection surface 10, and the five reflected light distance sensors 54 to 58 detect the distance the object detected by the projection surface touch sensor 16 moves in the pressing direction, thereby detecting the pressing operation of each input operation button 38.

[0105] According to this embodiment, it is possible to increase the number of types of operation selection compared to the fifth embodiment, and at the same time, even if the number of types of operation selection increases, by providing the projection surface touch sensor 16, pressing detection is performed only when a pressing operation is performed, thereby preventing malfunctions and erroneous operations.

[0106] In this embodiment, the projection surface touch sensor 16 is disposed near the three-dimensional space projection surface 10, and is a sensor of the type that outputs the position on the three-dimensional space projection surface 10 when an object is obstructing the projection surface 10 in XY coordinates, with the width direction X and the direction perpendicular to the width direction Y. However, as an alternative, the projection surface touch sensor 16 may be a sensor of the type that has a light-emitting element disposed on one side and a light-receiving element disposed on the other side of the width direction X of the three-dimensional space projection surface 10, and outputs the Y coordinate of the position on the three-dimensional space projection surface 10 when an object is obstructing the projection surface 10 in the direction Y perpendicular to the width direction X. This is because the position in the X direction can be detected using the reflected light distance sensors 54 to 58.

[0107] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0108] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0109] In addition, the control lines and information lines in the drawings are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In practice, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0110] 1, 1A, 1B, 1C, 1D Aerial image display input device 2 Aerial image projection unit 3 Display section 4. Image transmission plate 5,5B Input detection sensor 6 Input detection direction 7 Control Unit 8 User 9 Handling and processing equipment 10 Three-dimensional space projection surface 11. Housing 12. Case base 13 Side guide 14 Projection surface section 15 Sensor surface 16 Projection surface touch sensor 30~38, 30A, 30B Input operation buttons 40 Optical branching component 41 Retroreflective material 51~58 Reflected light distance sensor 61, 62, 63 Input detection surface 71 Input determination processing unit 72 Input / output I / F processing section 73 Screen control processing section 74 Speaker 80 moves 91 Control Unit 92 Display section 93 Processing Unit 94 Reception and payment machine 95 Display section 96 Input operation buttons

Claims

1. an aerial image projection unit having an internal rectangular display unit for displaying a predetermined image, an external rectangular image transmission plate for projecting the image displayed on the display unit onto a three-dimensional space projection surface visible to a user, and projecting the image displayed on the display unit into the air as an input guide screen for the user; an input detection sensor that detects an aerial operation on the input guide screen by the user; a control unit that performs predetermined control; In a floating image display input device having the three-dimensional space projection surface is a rectangular projection surface that faces a user's line of sight from a vicinity of one side of the rectangle of the image-transmitting plate at a predetermined angle with the image-transmitting plate, the input detection sensor is disposed on a straight line parallel to another side of the rectangle of the image-transmitting plate, the other side being located near the other side opposite to the one side of the rectangle of the image-transmitting plate; a plane formed by a line segment connecting the vicinity of the midpoint of a pair of sides of the rectangular projection surface that are not parallel to the image transmission plate and a line on which the input detection sensor is disposed is disposed as an input detection area; the display unit that displays the predetermined image displays input operation buttons in an input guide screen near a line segment that connects the midpoints of the rectangular projection surface; the input detection sensor continuously detects distance information indicating a distance to an object that falls within a detectable area obtained by extending the input detection area toward the rectangular projection surface; the control unit determines whether the input operation button has been pressed based on how the distance to the object changes over time relative to the distance to the input operation button on the three-dimensional space projection surface, based on distance information of the object continuously detected by the input detection sensor, and when it is determined that the input operation button has been pressed, changes the input operation button on the display unit to indicate that the button is in a pressed state; The input detection sensor is a reflected light distance sensor that has a plurality of pairs of infrared light emitting elements and light receiving elements, and calculates the distance to an object by the principle of triangulation based on the light receiving direction when the light receiving elements receive light emitted by the infrared light emitting elements and reflected by the object, and is arranged such that of two straight lines located near one side of the rectangle of the image-transmitting plate opposite to the other side and parallel to the other side, a plurality of the infrared light emitting elements are arranged on the straight line closer to the image-transmitting plate, and a plurality of the light receiving elements are arranged on the straight line farther from the image-transmitting plate, and the straight line connecting each pair of the infrared light emitting elements and the light receiving elements intersects the two parallel straight lines at a right angle.

1. A floating image display input device comprising:

2. the input detection sensors are reflective distance measurement sensors, and N of the sensors are arranged in a line, where N is two or more; the display unit displays N or less input operation buttons, The control unit determines that the input operation button has been pressed when a change in the output signals of the N reflective distance measuring sensors indicates that the distance of the object has changed from a first range in which the distance is farther than the projection surface of the three-dimensional space to a second range in which the distance is closer than the projection surface of the three-dimensional space.

2. The aerial image display input device according to claim 1, wherein:

3. a movement operation of moving an object in a predetermined operation direction parallel to the input guide screen is provided as the aerial operation for requesting a change of input content that can be input on the input guide screen; one or more of the N or less input operation buttons displayed by the display unit is a display change request button indicating the operation direction of the movement operation, The control unit determines whether the display change request button has been pressed by the movement operation based on change information between a first sensor output signal that detects the display change request button and a second sensor output signal that detects a portion adjacent to the display change request button, among the output signals of the N reflective distance measuring sensors, and when it determines that the display change request button has been pressed, changes the display of input operation buttons other than the display change request button on the display unit.

3. The aerial image display input device according to claim 2.

4. The control unit determining that the movement operation on the display change request button has been performed when the distance of the object does not change from the first range to the second range even if there is a change in the first sensor output signal, and when the change in the second sensor output signal indicates that the distance of the object is within the first range; determining that a pressing operation has been performed on the display change request button when a change in the first sensor output signal indicates that the distance to the object has changed from the first range to the second range; When it is determined that either the moving operation or the pressing operation on the display change request button has been performed, it is determined that the display change request button has been pressed, and the display of the input operation buttons other than the display change request button on the display unit is changed.

4. The aerial image display input device according to claim 3.

5. The aerial image projection unit and the input detection sensor are an integrated structure, the image transmission plate is substantially vertical, and the three-dimensional space projection surface is arranged to form a surface inclined toward the user's line of sight with respect to a horizontal plane, and the angle of the integrated structure is variable.

5. The aerial image display input device according to claim 1, wherein:

6. The device is connected to a handling processing device that processes media or performs a predetermined task with the user, and the control unit switches the image of the input operation button displayed on the display unit based on information from the handling processing device, and outputs determination information on pressing of the input operation button to the handling processing device.

6. The aerial image display input device according to claim 1, wherein:

7. the input detection sensors are arranged on M straight lines, two or more of which are located near another side of the rectangle of the image-transmitting plate that is opposite to the one side, and are parallel to the other side; M surfaces formed by line segments connecting the vicinity of (M+1) equal division points of a pair of sides of the rectangular projection surface that are not parallel to the image transmission plate and M arrangement lines of the input detection sensors are arranged as the input detection area; The display unit that displays the predetermined image displays input operation buttons in an input guide screen near a line segment that connects the neighborhoods of the (M+1)-division points on the rectangular projection surface.

7. The aerial image display input device according to claim 1, wherein:

8. The three-dimensional space projection surface is a rectangular projection surface that faces the user's line of sight from the vicinity of one side of the rectangle of the image-transmitting plate, at an angle of 75 degrees with the image-transmitting plate.

8. The aerial image display input device according to claim 1, wherein:

9. the display unit that displays the predetermined image displays the input operation buttons in the input guide screen as a three-dimensional perspective image having a shadow image in the thickness direction on the side opposite to one side of the rectangle of the image-transmitting plate; When the control unit determines that a pressing operation has been performed on the input operation button, the control unit displays the input operation button displayed by the display unit as a planar image without a shadow image in the thickness direction at a position moved by the length of the shadow image in the thickness direction of the input operation button when not in a pressed state, thereby indicating that the input operation button is in a pressed state.

9. The aerial image display input device according to claim 1, wherein:

10. The input detection sensor a projection surface touch sensor that detects, on the same surface as the three-dimensional space projection surface, a position of an object associated with an aerial operation by the user, when one side direction of the rectangle of the image-transmitting plate is defined as an X direction and a direction perpendicular to the X direction is defined as a Y direction, at least the position in the Y direction; a plurality of distance sensors arranged on a straight line parallel to the other side of the rectangle of the image-transmitting plate, the distance sensors being located near the other side opposite to the one side of the rectangle of the image-transmitting plate, and calculating the distance to the object; the display unit that displays the predetermined image displays input operation buttons in an input guide screen; the input detection sensor detects the presence or absence of an object in an area of ​​the input operation button by the projection surface touch sensor, and detects the amount of movement of the object in a pressing direction relative to the area of ​​the input operation button in the detectable area by the plurality of distance sensors; The control unit determines whether a pressing operation has been performed on the input operation button based on detection information on the presence or absence of an object detected by the input detection sensor and detection information on the amount of movement of the object in the pressing direction, and when it determines that a pressing operation has been performed on the input operation button, changes the input operation button displayed by the display unit to indicate that it is in a pressed state.

10. The aerial image display input device according to claim 1, wherein:

11. the aerial image projection unit and the input detection sensor are integrally structured, the image transmission plate is substantially vertical, and the three-dimensional space projection surface is disposed to form a surface inclined toward a user's line of sight with respect to a horizontal plane; The aerial image projection unit further includes triangular side guides on both sides thereof that cover the input detection sensor, the image transmission plate, and the three-dimensional space projection surface.

11. The aerial image display input device according to claim 1, wherein:

12. an aerial image projection unit having an internal rectangular display unit for displaying a predetermined image, an external rectangular image transmission plate for projecting the image displayed on the display unit onto a three-dimensional space projection surface visible to a user, and projecting the image displayed on the display unit into the air as an input guide screen for the user; an input detection sensor that detects an aerial operation on the input guide screen by the user; a control unit that performs predetermined control; An aerial image display input method using an aerial image display input device having the three-dimensional space projection surface is a rectangular projection surface that faces a user's line of sight from a vicinity of one side of the rectangle of the image-transmitting plate at a predetermined angle with the image-transmitting plate, the input detection sensor is disposed on a straight line parallel to another side of the rectangle of the image-transmitting plate, the other side being located near the other side opposite to the one side of the rectangle of the image-transmitting plate; a plane formed by a line segment connecting the vicinity of the midpoint of a pair of sides of the rectangular projection surface that are not parallel to the image transmission plate and a line on which the input detection sensor is disposed is disposed as an input detection area; the display unit that displays the predetermined image displays input operation buttons in an input guide screen near a line segment that connects the midpoints of the rectangular projection surface; the input detection sensor continuously detects distance information indicating a distance to an object that falls within a detectable area obtained by extending the input detection area toward the rectangular projection surface; the control unit determines whether the input operation button has been pressed based on how the distance to the object changes over time relative to the distance to the input operation button on the three-dimensional space projection surface, based on distance information of the object continuously detected by the input detection sensor, and when it is determined that the input operation button has been pressed, changes the input operation button on the display unit to indicate that the button is in a pressed state; The input detection sensor is a reflected light distance sensor that has a plurality of pairs of infrared light emitting elements and light receiving elements, and calculates the distance to an object by the principle of triangulation based on the light receiving direction when the light receiving elements receive light emitted by the infrared light emitting elements and reflected by the object, and is arranged such that of two straight lines located near one side of the rectangle of the image-transmitting plate opposite to the other side and parallel to the other side, a plurality of the infrared light emitting elements are arranged on the straight line closer to the image-transmitting plate, and a plurality of the light receiving elements are arranged on the straight line farther from the image-transmitting plate, and the straight line connecting each pair of the infrared light emitting elements and the light receiving elements intersects the two parallel straight lines at a right angle.

1. A method for displaying and inputting an aerial image, comprising:

Images