Input display apparatus

Abstract

To provide an input display apparatus which enables an input operation into an aerial image having high interactivity.SOLUTION: An input display apparatus of the present invention comprises: display means for displaying an aerial image Q by utilizing retroreflection; a sensor 130 for detecting the fact that a finger U approaches the aerial image or the fact that the finger U separates from the aerial image Q; and a sound output unit 160 for outputting the sound of a sound source A responsive to the detection of the approach of the finger U to the aerial image Q, and outputting the sound of a sound source B responsive to the detection of the separation of the finger U from the aerial image Q. A combination of the sound of the sound source A and the sound of the sound source B is, e.g., "ding dong", and by making the sounds twice with the approach and separation relative to the aerial image Q, interactivity increases, and operational feeling and decisiveness can be fed back to a user.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an input display device that performs an input operation on an aerial image using retroreflection.

Background Art

[0002] Aerial Imaging by Retro-Reflection (AIRR) is known. For example, the input device of Patent Document 1 recognizes that a user is bringing an object such as a finger close to an image formed in space, and notifies the user to that effect by a change in the image. Further, the aerial operation device of Patent Document 2 determines whether a button has been pressed based on the position of an object detected by a sensor, notifies the user of the determination result, and changes the display mode of the button to a first mode when it is determined that the position of the object is a first position.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] As an aerial interface, the practical application of an input display device that performs an input operation on an aerial image by combining an AIRR-type aerial optical system and a sensor that detects the proximity of an object has been promoted. In such an input display device, the operation of the input switch has been modularized by adding functions such as turning on / off illumination and switch sound, and the production of such a device has also been advanced.

[0005] Fig. 1(A) is a perspective view showing an example of a conventional input display device, and Fig. 1(B) is a diagram showing an example of an operation on the aerial image of the input display device. The input display device 10 houses an optical system such as a light source and a retroreflective member in a housing 120 with a beam splitter mounted on its surface, and displays an aerial image Q above the housing 120. Further, the input display device 10 includes a sensor 130 for detecting the approach of the user's finger to the aerial image Q, a speaker that outputs a switch sound when the approach of the finger to the aerial image Q is detected, and the like. For example, when the design P of the original video is generated by the light source, the aerial image Q is displayed in the space floating from the housing 120. In this example, the aerial image Q shows the numbers 1, 2, 3, and 4.

[0006] As shown in Fig. 1(B), the user can perform an input operation on the aerial image Q by approaching the finger U to the aerial image Q. For example, when it is desired to input to "1", the finger is approached to "1" as if pressing the aerial image Q of "1". The approach of the finger U to the aerial image Q is detected by the sensor 130, and in response to this detection, to inform the user that the input operation to the aerial image Q has been recognized, the image of the aerial image Q is changed, or a switch sound (or operation sound) such as "pip" is emitted from the speaker.

[0007] However, the conventional input display device has the following problems. Since the aerial image Q has no physical entity, there is a problem that it is still difficult for the user (operator) to feel the operation feeling and determination feeling like a physical switch simply by making a sound or changing the image. This is considered to be due to the fact that the feedback to the operator is insufficient compared to the operation of a mechanical switch, etc., and the interactivity is not exerted.

[0008] The present invention solves such conventional problems and aims to provide an input display device that enables an input operation with high interactivity to an aerial image. operation.

Means for Solving the Problems

[0009] The input display device according to the present invention is capable of displaying an aerial image using retroreflection, and includes a first detection means for detecting that an operating object such as a finger approaches the aerial image, a second detection means for detecting that the operating object has moved away from the aerial image, and a sound output control means for outputting a first sound signal in response to the detection by the first detection means that the operating object has approached, and outputting a second sound signal in response to the detection by the second detection means that the operating object has moved away.

[0010] In one aspect, the first sound signal is different from the second sound signal. In one aspect, the sound output control means includes a first sound source for generating the first sound signal and a second sound source for generating the second sound signal. In one aspect, the sound output control means sets a silent period between the output of the first sound signal and the output of the second sound signal. In one aspect, the sound output control means further measures a first time in a state where the operating object is approaching in response to the detection by the first detection means that the operating object is approaching, and when the measured first time is shorter than the reproduction time of the first sound, performs a fade-out process on the first sound signal. In one aspect, the sound output control means inserts a silent period between the fade-out processed first sound signal and the second sound signal, and outputs the second sound signal after the silent period. In one aspect, the sound output control means further measures a second time in a state where the operating object is moving away in response to the detection by the second detection means that the operating object has moved away, and when the measured second time is shorter than the reproduction time of the second sound, performs a fade-out process on the second sound signal. In one aspect, the sound output control means inserts a silent period between the fade-out processed second sound signal and the first sound signal, and outputs the first sound signal after the silent period.

Advantages of the Invention

[0011] According to the present invention, since the first sound signal is output when the operating object approaches and the second sound signal is output when the operating object moves away, the interactivity during the input operation to the aerial image is improved, and a higher sense of operation and determination can be fed back to the user.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

MODE FOR CARRYING OUT THE INVENTION

[0013] The input display device according to the present invention displays an aerial image using retroreflection in a three-dimensional space without wearing special glasses or the like. Further, the input display device according to the present invention enables the provision of a non-contact user interface with high interaction with respect to the aerial image. It should be noted that the drawings referred to in the following description of the embodiments include exaggerated displays for ease of understanding of the invention and do not represent the actual shape and scale of the product as it is.

Embodiment

[0014] Next, embodiments of the present invention will be described in detail. FIG. 2 is a diagram for explaining the outline of the input display device according to the first embodiment of the present invention. In the input display device 100 of this embodiment, for example, in a state where an aerial image Q is displayed above a beam splitter 120 attached to the upper surface of the housing, as shown in FIG. 2(A), when the user's finger U passes near the aerial image Q, a sound like "pin" is made, and as shown in FIG. 2(B), when the user performs an operation of pulling the finger U from the aerial image Q, a sound like "pon" is made again. The approach of the finger U to the aerial image Q and the operation of pulling the finger U therefrom are detected by the sensor 130.

[0015] The sound made when approaching the aerial image Q and the sound made when leaving the aerial image Q may be different sounds, and the combination of sounds is arbitrary. For example, in addition to the combination of "pin" and "pon" as described above, a combination of simple bell sounds or melodies with different frequencies may also be used. Also, the combination of the loudness of the sounds is arbitrary. For example, the sound made when approaching may be made louder or softer than the sound made when leaving. Furthermore, the combination of the lengths of the sounds to be output is also arbitrary. For example, the sound made when approaching may be made longer or shorter than the sound made when leaving.

[0016] In addition, it is also possible to prepare a plurality of combinations of sounds. For example, as shown in FIG. 1(A), when a plurality of numbers 1, 2, 3, and 4 are displayed as the aerial image Q and the user brings a finger close to a number to select any of the numbers, the sounds emitted during the operation of each number may be made different. For example, the frequency of the sound emitted during the operation of number 1 is made relatively high, and the frequency of the sound emitted during the operation of number 2 is made relatively low.

[0017] Furthermore, when a sound is emitted when approaching and moving away from the aerial image Q, it is also possible to change the video of the aerial image in conjunction with this. For example, when a finger approaches, the color of the aerial image is changed, or the color is darkened and returned to the original color when the finger is moved away.

[0018] Thus, according to this embodiment, when performing a non-contact operation on the aerial image Q, by generating two different sounds in conjunction with the movement of approaching / moving away from the aerial image Q, the interactivity is enhanced, and the operation feeling and determination feeling during the input operation can be more effectively feedback to the operator. In addition, by generating two different sounds, the auditory cross-modal effect can be enhanced. Furthermore, by enabling any combination of sounds, sound effects suitable for the image and function in various scenes can be achieved.

[0019] Next, the specific configuration of the input display device 100 of this embodiment will be described. FIG. 3 is a diagram showing a configuration example of an optical system for displaying an aerial image in the input display device 100. The input display device 100, for example, attaches a beam splitter 120 to the upper surface of a rectangular housing 110, arranges a retroreflective member 140 and a light source 150 inside thereof, and displays an aerial image Q above the beam splitter 120.

[0020] The beam splitter 120 is an optical member that separates incident light into transmitted light and reflected light. For example, a half mirror or a polarizing beam splitter is used when using polarized light.

[0021] The retroreflective member 140 is an optical member that reflects light in the same direction as the incident light, and is composed of, for example, a prism-type retroreflective element such as a triangular pyramid-type retroreflective element or a full cube corner-type retroreflective element, or a bead-type retroreflective element. The illustrated retroreflective member 140 has a curved or spherical retroreflective surface, but the retroreflective surface may be planar.

[0022] The light source 150 is not particularly limited in its configuration as long as it has a function of generating a video or image of the design P that is the original image of the aerial image Q. The design P may be various types of button displays such as characters, figures, icons, etc., and may be not only a still image but also a moving image. The light source 150 is, for example, a display light source such as a liquid crystal display device, an organic EL display device, a projection display device, or an LED light source. Further, an opening or through-hole for generating the design P may be formed in the retroreflective member, and the back side of the retroreflective member may be irradiated with an LED light source.

[0023] The light (design P) emitted from the light source 150 is incident on the beam splitter 120, the light reflected there is retroreflected by the retroreflective member 140, the retroreflected light passes through the beam splitter 120, and the transmitted light forms an image to generate the aerial image Q. The aerial image Q is displayed at a position symmetric to the light source 150 with respect to the beam splitter 120, and the user can visually recognize the aerial image Q within the range where the user can observe the retroreflective member 140.

[0024] FIG. 4 is a block diagram showing the electrical configuration of the input display device 100 of this embodiment. The input display device includes a sensor 130 that detects the approach of the finger U to the aerial image Q or the separation of the finger from the aerial image Q, a light source 150 for generating the design P, a sound output unit 160 that outputs sound from a speaker, a memory 170, and a controller 180.

[0025] The sensor 130 is not particularly limited as long as it can detect the approach and separation of the finger U to and from the aerial image Q. For the sensor 130, for example, a distance measuring sensor, a stereo camera, a capacitance sensor, etc. for measuring the three-dimensional position of the finger U can be used. The distance measuring sensor includes, for example, a combination of an infrared light emitting element and an infrared light receiving element, and enables detection of the three-dimensional position of the finger U on the housing 110 based on the reflected light from the finger U irradiated with infrared light. The stereo camera images the finger U on the housing 110 from the left and right directions, and enables detection of the three-dimensional position of the finger U by analyzing the captured image data. The capacitance sensor is, for example, a touch panel mounted on a liquid crystal display device or the like. The touch panel is attached to, for example, the upper surface of the housing 110, and enables detection of the XY coordinates of the finger U and the Z coordinate which is the distance from the upper surface of the housing 110. The detection result of the sensor 130 is provided to the controller 180.

[0026] The memory 170 stores the audio data of the sound source A for outputting a sound when the finger approaches the aerial image Q and the audio data of the sound source B for outputting a sound when the finger is separated from the aerial image Q. The types and formats of the audio data of the sound source A and the sound source B are arbitrary. The controller 180 controls the overall operation of the input display device 100, and includes, for example, a microcontroller or a microprocessor including a ROM / RAM or the like, and is controlled in operation by a program stored in the ROM / RAM. Further, the controller 180 holds the three-dimensional coordinates of the displayed aerial image Q. The three-dimensional coordinates of the aerial image Q can be geometrically determined from the three-dimensional coordinates of the beam splitter 120 and the light source 150. For example, the three-dimensional coordinates of 1, 2, 3, and 4 of the aerial image Q shown in FIG. 1(A) are known in the controller 180.

[0027] FIG. 5 is a flowchart for explaining the operation of the input display device 100. The controller 180 drives the light source 150 to display the aerial image Q of the design P (S100). Thereby, the user visually recognizes the aerial image Q in the viewing direction. The aerial image Q is an image or video for the user to perform an input operation, and here, the aerial image Q may be referred to as a button for the user to perform an input operation.

[0028] The controller 180 receives the detection result from the sensor 130 and determines whether the user's finger U has approached the aerial image Q (S110). For example, when the sensor 130 is a capacitance-type touch panel, the touch panel is attached to the upper surface of the housing 110, and when the finger U approaches the housing 110, it is possible to detect the three-dimensional position of the finger U due to the change in capacitance. When the distance D1 between the position of the finger U detected by the sensor 130 and the position of the aerial image Q decreases and the distance D1 reaches a first threshold or less (D1 ≤ the first threshold), the controller 180 determines that the finger U has approached the aerial image Q.

[0029] When the controller 180 determines that the finger U has approached the aerial image Q, it reads the audio data of the sound source A from the memory 170 and provides the reproduction data of the sound source A to the sound output unit 160. Thereby, the sound of the sound source A is output from the speaker (S120).

[0030] Next, the controller 180 monitors the detection result of the sensor 130 and determines whether the finger U has left the aerial image Q (S130). For example, when the distance D2 between the position of the finger U and the position of the aerial image Q increases and the distance D2 becomes a second threshold or more (D2 ≥ the second threshold), the controller 180 determines that the finger U has left the aerial image Q.

[0031] When the controller 180 determines that the finger U has left the virtual image Q, it reads the audio data of the sound source B from the memory 170 and provides the playback data of the sound source B to the sound output unit 160. As a result, the sound of the sound source B is output from the speaker (S140). Note that when providing the playback data of the sound source B to the sound output, the controller 180 may set a blank period (silence) for a certain period and provide the playback data of the sound source B after the elapse of the blank period. Thereby, the user can clearly recognize the boundary between the sound of the sound source A and the sound of the sound source B.

[0032] Next, a second embodiment of the present invention will be described. In the first embodiment, the sound source A is reproduced when a finger approaches the virtual image Q, and the sound source B is reproduced when the finger leaves the virtual image Q. However, depending on the time during which the user presses the button (virtual image Q) with the finger U, the sound of the sound source A may be interrupted in the middle, or the boundary between the sound of the sound source A and the sound of the sound source B may be difficult to hear.

[0033] FIG. 6(A) is a graph showing the waveform of the sound output when the button press time ≧ the reproduction time of the sound source A, and FIG. 6(B) is a graph showing the waveform of the sound output when the button press time < the reproduction time of the sound source A. The vertical axis represents the amplitude, and the horizontal axis represents the time.

[0034] As shown in FIG. 6(A), when the button press time of the button (virtual image Q) is longer than the reproduction time of the sound source A, the sound of the sound source A is output within the button press time. Therefore, the sound of the sound source A is then clearly separated from the sound of the sound source B output when the finger leaves the button, and there is no sense of discomfort. For example, the "peen" of the sound source A and the "poon" of the sound source B can be heard naturally.

[0035] On the other hand, as shown in FIG. 6(B), when the button press time is shorter than the reproduction time of the sound source A, the sound source B is reproduced in the middle of the reproduction of the sound source A, resulting in a sense of discomfort in the way it sounds. For example, the "peen" of the sound source A is interrupted and heard as "pi", "poon".

[0036] Therefore, in the second embodiment, when the button press time is shorter than the playback time of sound source A, the sounds of sound source A and sound source B are played so as not to cause a sense of discomfort. FIG. 7 is a diagram for explaining the outline of the input display device 100A of the second embodiment. FIG. 7(A) shows the operation when a finger approaches, and FIG. 7(B) shows the operation when a finger is withdrawn.

[0037] (1) When the button is pressed: Play sound source A. The fixed time during which the button is pressed is extended while fading the volume. As shown in FIG. 7(A), for example, a sound like "pee~nn" can be heard. When the playback time of sound source A > the button press time, the end of the playback of sound source A is faded out so that the volume gradually decreases. (2) The operation of releasing the finger from the button: Play sound source B. When the playback time of sound source A > the button press time, when playing sound source B after fading out sound source A, a blank time (silent period) is inserted immediately before playing sound source B.

[0038] Thereby, when the button press time is shorter than the playback time of sound source A, the interruption of the sound of sound source A is suppressed by the fade-out of sound source A, and the sounds of sound source A and sound source B can be heard more naturally without a sense of discomfort due to the insertion of the blank period.

[0039] FIG. 8(A) is a graph showing the output waveform of the sound before the countermeasure when the button press time < the playback time of sound source A, and FIG. 8(B) is a graph showing the output waveform of the sound after the countermeasure when the button press time < the playback time of sound source A.

[0040] As shown in FIG. 8(A), when the button press time is shorter than the playback time of sound source A, since sound source B is played at the timing when the finger leaves the button, the sound of sound source A is interrupted in the middle. On the other hand, as shown in FIG. 8(B), in the second embodiment, when the button press time is shorter than the playback time of sound source A, in order to prevent the sound of sound source A from being interrupted in the middle, the playback signal is fade-out processed so that the volume gradually decreases at the end of sound source A, a blank period (silent period) is inserted immediately before the playback of sound source B, and sound source B is played after the blank period.

[0041] FIG. 9 is a block diagram showing the electrical configuration of the input display device according to the second embodiment, and the same components as those in the first embodiment are denoted by the same reference numerals. In the second embodiment, the controller 200 includes a time measurement unit 210 for measuring the button press time (i.e., the time since it is detected that the finger U approaches the virtual image) and the non-press time when the button is not pressed (i.e., the time since it is detected that the finger U leaves the virtual image), and a signal processing unit 220 for processing the playback signals of sound source A and sound source B.

[0042] FIG. 10 is a flowchart for explaining the operation of the input display device according to the second embodiment. The controller 200 monitors whether the button (virtual image Q) is pressed based on the detection result of the sensor 130 with the virtual image Q being displayed (S200). That is, it monitors whether the distance D1 between the position of the finger U and the position of the virtual image Q decreases and whether the distance D1 reaches a first threshold or less (D1 ≤ the first threshold).

[0043] When the controller 200 determines that the button has been pressed, it reads the audio data of sound source A from the memory 170, provides the playback data of sound source A to the sound output unit 160, and causes the sound output unit 160 to output the sound of sound source A, in the same manner as in the first embodiment (S210). In parallel with this, the controller 200 causes the time measurement unit 210 to measure the button press time at the timing when the distance D1 reaches the first threshold value (S220). When the distance D1 between the finger U and the virtual image Q starts to increase or when the distance D1 becomes greater than the first threshold value, the controller 200 determines that the button press has ended and terminates the measurement by the time measurement unit 210.

[0044] Next, the controller 200 determines whether the button press time ≥ the playback time of sound source A based on the measurement result of the time measurement unit 210 (S230). When the button press time is longer than the playback time of sound source A, it monitors whether the finger has been released from the button (S240). That is, it monitors whether the distance D2 between the position of the finger U and the position of the virtual image Q has increased and whether the distance D2 has become equal to or greater than the second threshold value (D2 ≥ the second threshold value).

[0045] On the other hand, when the button press time ≥ the playback time of sound source A is not satisfied, that is, when it is determined that the button press time is shorter than the playback time of sound source A (S230), the controller 200 monitors whether the finger has been released from the button in the same manner as in step S240 (S250). When the controller 200 determines that the finger has been released from the button, it causes the signal processing unit 220 to perform a fade-out process on the playback signal of sound source A so that the volume of sound source A gradually decreases (S260). As a result, the sound output unit 160 outputs the fade-out processed sound of sound source A.

[0046] In response to the finger leaving the button (S240, S250), the controller 200 reads the data of sound source B from the memory 170, inserts a certain blank time (silent time) before providing the playback data of sound source B to the sound output unit 160 (S270), and then provides the playback signal of sound source B to the sound output unit 160. As a result, the sound output unit 160 outputs the sound of sound source B (S280).

[0047] Further, while the sound of sound source B is being output, the controller 200 causes the time measurement unit 210 to measure the non-pressing time during which the button is not pressed at the timing when the distance D2 reaches the second threshold value (S290). When the distance D2 between the finger U and the virtual image Q starts to decrease or when the distance D2 becomes smaller than the second threshold value, the controller 200 regards the non-pressing of the button as ended and ends the measurement by the time measurement unit 210.

[0048] Next, the controller 200 determines whether the non-pressing time of the button ≥ the reproduction time of sound source B based on the measurement result of the time measurement unit 210 (S300). If the non-pressing time of the button is longer than the reproduction time of sound source B, the controller 200 causes the signal processing unit 220 to insert a blank time (S310) and ends the process.

[0049] On the other hand, if the non-pressing time of the button is shorter than the reproduction time of sound source B (S300), it is monitored whether the button has been pressed (S320). This monitoring is the same process as in step S200. When it is determined that the button has been pressed (S320), the controller 200 causes the signal processing unit 220 to perform a fade-out process on the reproduction signal of sound source B so that the volume of sound source B gradually decreases (S330). As a result, the faded-out sound of sound source B is output from the sound output unit 160.

[0050] In response to the button being pressed (S320), the controller 200 reads the audio data of sound source A from the memory 170, inserts a certain blank time (silent time) before providing the reproduction data of sound source A to the sound output unit 160 (S340), and then provides the reproduction signal of sound source A to the sound output unit 160. As a result, the sound of sound source A is output from the sound output unit 160 (S210). The processes after step S220 are the same as above.

[0051] Components of sound that are easy for people with hearing impairments to hear (beep - boop) are defined as sound signs in international standards. Figure 9(B) is a diagram showing the amplitude waveform of the guiding sound sign in public facilities defined by international standards. In this figure, 1 is the audio signal, 2 is the start part, 3 is the end part, 4 is the silent interval, the x - axis represents time, and the y - axis represents amplitude.

[0052] The fade - out process of the volume of sound source A in this embodiment (step S260) corresponds to section 3 in Figure 9(B), the insertion of blank time (S270) corresponds to section 4, the fade - out process of the volume of sound source B (step 330) corresponds to section 5, and the insertion of blank time (S340) corresponds to section 6. The silent intervals in sections 4 and 6 correspond to the "n" in "beep - boop". Thus, the sound output when operating on the aerial image according to this embodiment conforms to international standards.

[0053] When the button - pressing time is short, by fading out the volume of the first sound source A (the beep part) and inserting a blank time before playing the latter - half sound source B (the boop part), a sound that is easy to hear and has little sense of discomfort can be realized. And since it is linked with the operation, a sufficient operating feeling can be obtained.

[0054] The input - display device of this embodiment can be applied to the display of information of all devices and user input. For example, it can be applied to computer devices, in - vehicle electronic devices, ATMs in banks, ticket - purchasing machines at stations, input buttons of elevators, etc.

[0055] As described above in detail for the preferred embodiments of the present invention, the present invention is not limited to specific embodiments, and various modifications and changes are possible within the scope of the gist of the invention described in the claims.

Explanation of Reference Numerals

[0056] 100, 100A: Input - display device 110: Housing 120: Beam splitter 130: Sensor 140: Retro - reflective member 150: Light source 160: Sound output unit 170: Memory 180, 200: Controller P: Design Q: Aerial image

Claims

1. An input display device capable of displaying a virtual image using retroreflection, comprising: a first detection means for detecting that an operating object such as a finger approaches the virtual image; a second detection means for detecting that the operating object moves away from the virtual image; a sound output control means for outputting a first sound signal in response to detection of the approach of the operating object by the first detection means, and outputting a second sound signal in response to detection of the departure of the operating object by the second detection means; An input display device having the above components.

2. The input display device according to claim 1, wherein the first sound signal is different from the second sound signal.

3. The input display device according to claim 1, wherein the sound output control means includes a first sound source for generating the first sound signal and a second sound source for generating the second sound signal.

4. The input display device according to claim 1, wherein the sound output control means sets a silent period between the output of the first sound signal and the output of the second sound signal.

5. The input display device according to claim 1, wherein the sound output control means further measures a first time when the operating object is in an approaching state in response to detection of the approach of the operating object by the first detection means, and if the measured first time is shorter than the reproduction time of the first sound, performs a fade-out process on the first sound signal.

6. The input display device according to claim 5, wherein the sound output control means inserts a silent period between the fade-out processed first sound signal and the second sound signal, and outputs the second sound signal after the silent period.

7. The input display device according to claim 1, wherein the sound output control means further measures a second time when the operating object is in a departing state in response to detection of the departure of the operating object by the second detection means, and if the measured second time is shorter than the reproduction time of the second sound, performs a fade-out process on the second sound signal.

8. The input display device according to claim 7, wherein the sound output control means inserts a silent period between the fade-out processed second sound signal and the first sound signal, and outputs the first sound signal after the silent period.

Images