Airborne display device

Abstract

This allows for more accurate detection of touch operations performed by observers on aerial images. [Solution] The aerial display device includes a display element 10 that displays an image, an optical element 30 positioned to receive light from the display element 10 and configured to form an aerial image in the air opposite the display element 10, an input device 40 configured to form a detection area on the aerial image plane including the area where the aerial image is displayed and to detect objects present in the detection area, and a control unit 50 that controls the operation of the display element 10 and the input device 40. The control unit 50 sequentially executes a first operation mode and a second operation mode. In the first operation mode, the control unit 50 displays a first image at a reference position on the display element 10 and calculates the amount of deviation between the reference position and the detection position based on the detection position by the input device 40. In the second operation mode, the control unit 50 corrects the position of the image to be displayed based on the amount of deviation and displays a second image on the display element 10.

Description

[Technical Field]

[0001] This invention relates to an aerial display device. [Background technology]

[0002] In environments where cleanliness and hygiene are paramount, such as industrial and medical applications, contactless aerial display technology is highly anticipated. One proposed aerial display technology involves an aerial display device in which an imaging element and a display element are arranged in parallel (Patent Document 1).

[0003] The aerial display device uses sensing elements to detect the observer's input. The sensing elements detect objects present in the aerial image area and determine when the observer touches the aerial image. Sensing elements include infrared TOF sensors or capacitive sensors. The observer can touch the aerial image displayed by the aerial display device without touching the device itself.

[0004] When an observer's viewpoint moves relative to an aerial display device, the aerial image may move in accordance with the change in viewpoint. In this case, the position of the aerial image and the position of the detection area used to detect objects within the aerial image's region may become misaligned. In this situation, the sensing element may mistakenly determine that a different aerial image than the one the observer intended has been touched. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 7184220 [Overview of the project] [Problems that the invention aims to solve]

[0006] This invention provides an aerial display device that can more accurately detect touch operations performed by an observer on an aerial image. [Means for solving the problem]

[0007] According to a first aspect of the present invention, an aerial display device is provided, comprising: a display element for displaying an image; an optical element positioned to receive light from the display element and configured to form an aerial image in the air opposite to the display element; an input device configured to form a detection region on the aerial image plane including the region where the aerial image is displayed, and to detect an object present in the detection region; and a control unit for controlling the operation of the display element and the input device, wherein the control unit sequentially executes a first operation mode and a second operation mode, in the first operation mode, displays a first image at a reference position on the display element, calculates the amount of deviation between the reference position and the detection position based on the detection position by the input device, and in the second operation mode, corrects the position of the image to be displayed based on the amount of deviation, thereby displaying a second image on the display element.

[0008] According to a second aspect of the present invention, an aerial display device according to the first aspect is provided, wherein the control unit calculates the amount of displacement using mutually orthogonal X and Y coordinates in the second operating mode, and corrects the position of the image to be displayed in a direction in which the sign of the amount of displacement is reversed.

[0009] According to a third aspect of the present invention, an aerial display device according to the first aspect is provided, wherein the input device forms the detection area in the region that overlaps with the image before position correction in a plan view in the second operating mode.

[0010] According to a fourth aspect of the present invention, an aerial display device is provided, comprising: a display element for displaying an image; an optical element positioned to receive light from the display element and configured to form an aerial image in the air opposite to the display element; an input device configured to form a detection region on the aerial image plane including the region where the aerial image is displayed, and to detect an object present in the detection region; and a control unit for controlling the operation of the display element and the input device, wherein the control unit sequentially executes a first operation mode and a second operation mode, in the first operation mode, displays a first image at a reference position on the display element and calculates the amount of deviation between the reference position and the detection position based on the detection position by the input device; and in the second operation mode, displays a second image on the display element and corrects the position of the detection region based on the amount of deviation.

[0011] According to a fifth aspect of the present invention, an aerial display device according to the fourth aspect is provided, wherein the control unit calculates the amount of displacement using mutually orthogonal X and Y coordinates in the second operating mode and corrects the position of the detection area in the same direction as the sign of the amount of displacement.

[0012] According to a sixth aspect of the present invention, an aerial display device according to the fourth aspect is provided, wherein the control unit causes the second image to be displayed in the area that overlaps with the detection area before position correction in a plan view in the second operating mode.

[0013] According to a seventh aspect of the present invention, an aerial display device according to the first or fourth aspect is provided, further comprising a motion sensor for detecting when an observer has been replaced, wherein the control unit re-executes the first operating mode based on the information from the motion sensor.

[0014] According to an eighth aspect of the present invention, an aerial display device according to the first or fourth aspect is provided, wherein the input device includes a capacitive sensor unit.

[0015] According to a ninth aspect of the present invention, the optical element includes a planar base material and a plurality of optical elements provided under the base material, each extending in a first direction and arranged in a second direction perpendicular to the first direction, and each of the plurality of optical elements is inclined with respect to the normal direction of the base material and has an incident surface and a reflection surface that are in contact with each other, and an aerial display device according to the first or fourth aspect is provided.

[0016] According to a tenth aspect of the present invention, an aerial display device according to the first or fourth aspect is provided, in which the display element and the optical element are arranged parallel to each other.

[0017] According to an eleventh aspect of the present invention, an aerial display device according to the first or fourth aspect is further provided with a light control element arranged between the display element and the optical element and configured to transmit an obliquely directed light component of the light from the display element.

[0018] According to a twelfth aspect of the present invention, the light control element includes a plurality of transparent members and a plurality of light-shielding members. The plurality of transparent members each extend in a first direction and are arranged in a second direction perpendicular to the first direction. The plurality of light-shielding members each extend in the first direction and are arranged in the second direction. The plurality of transparent members and the plurality of light-shielding members are arranged alternately, and the plurality of light-shielding members are inclined with respect to the normal of the light control element, and an aerial display device according to the eleventh aspect is provided.

Advantages of the Invention

[0019] According to the present invention, it is possible to provide an aerial display device capable of more accurately detecting a touch operation on an aerial image by an observer.

Brief Description of the Drawings

[0020] [Figure 1] FIG. 1 is a perspective view of an aerial display device according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a perspective view of the light control element shown in FIG. 1. [Figure 3]Figure 3 is a perspective view of the optical element shown in Figure 1. [Figure 4] Figure 4 is a block diagram of the aerial display device. [Figure 5] Figure 5 is a side view of the aerial display device in the XZ plane. [Figure 6] Figure 6 is a perspective view illustrating the reflection of light in an optical element. [Figure 7] Figure 7 is a side view of the XZ plane illustrating the reflection of light in an optical element. [Figure 8] Figure 8 is a side view of the YZ plane illustrating the reflection of light in an optical element. [Figure 9] Figure 9 illustrates the angular conditions of the incident and reflective surfaces in an optical element. [Figure 10] Figure 10 illustrates the display operation of an aerial display device. [Figure 11] Figure 11 illustrates the display operation of an aerial display device when the observer moves their viewpoint. [Figure 12] Figure 12 is a flowchart illustrating the operation of the aerial display device. [Figure 13] Figure 13 is a diagram illustrating the standby screen in standby mode. [Figure 14] Figure 14 illustrates the operation of the aerial display device in standby mode. [Figure 15] Figure 15 illustrates the operation of the aerial display device in standby mode when the observer's viewpoint is off-center. [Figure 16] Figure 16 is a schematic diagram illustrating the process of generating corrected image data. [Figure 17] Figure 17 is a schematic diagram illustrating the observer's input operations in input mode. [Figure 18] Figure 18 illustrates an example of an input screen displayed by a display element. [Figure 19] Figure 19 is a flowchart illustrating the operation of the aerial display device according to the second embodiment of the present invention. [Figure 20]Figure 20 is a schematic diagram illustrating the operation of correcting the position of the detection area. [Figure 21] Figure 21 is a schematic diagram illustrating the observer's input operations in input mode. [Figure 22] Figure 22 is a block diagram of an aerial display device according to a third embodiment of the present invention. [Figure 23] Figure 23 is a flowchart illustrating the operation of the aerial display device. [Figure 24] Figure 24 is a flowchart illustrating the operation of the aerial display device according to the fourth embodiment of the present invention. [Figure 25] Figure 25 is a perspective view of an aerial display device according to a fifth embodiment of the present invention. [Modes for carrying out the invention]

[0021] The embodiments will be described below with reference to the drawings. However, the drawings are schematic or conceptual, and the dimensions and proportions in each drawing are not necessarily the same as those in reality. Furthermore, even when the same part is represented between drawings, the relationship between dimensions and proportions may be represented differently. In particular, the embodiments shown below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not determined by the shape, structure, arrangement, etc. of the components. In the following description, elements having the same function and configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0022] [1] First Embodiment [1-1] Configuration of the aerial display device 1 Figure 1 is a perspective view of an aerial display device 1 according to the first embodiment of the present invention. In Figure 1, the X direction is the direction along one side of the aerial display device 1, the Y direction is the direction perpendicular to the X direction in the horizontal plane, and the Z direction is the direction perpendicular to the XY plane (also called the normal direction).

[0023] The aerial display device 1 is a device that displays images (including video and moving images). The aerial display device 1 displays an aerial image in the air above its light-emitting surface. "Displaying an aerial image" is synonymous with "forming an aerial image." The light-emitting surface of the aerial display device 1 refers to the upper surface of the uppermost component among the multiple components that make up the aerial display device 1. An aerial image is a real image formed in the air.

[0024] The aerial display device 1 comprises a display element 10, a light control element 20, an optical element 30, and an input device 40. The display element 10, the light control element 20, the optical element 30, and the input device 40 are arranged in this order along the Z direction and are parallel to each other. The display element 10, the light control element 20, the optical element 30, and the input device 40 are fixed in desired positions with fixing members (not shown) so as to maintain a desired distance from each other. The display element 10, the light control element 20, the optical element 30, and the input device 40 are housed in a housing (not shown).

[0025] The display element 10 displays a desired image on its display surface. The display element 10 emits display light for displaying the image. The display element 10 is composed of, for example, a liquid crystal display element. The driving mode of the display element 10 is not particularly limited, and TN (Twisted Nematic) mode, VA (Vertical Alignment) mode, or homogeneous mode can be used.

[0026] The optical control element 20 has a function to reduce unwanted light. Unwanted light is light components that do not contribute to the generation of an aerial image and includes light components that are transmitted through the optical element 30 in the normal direction. The optical control element 20 is configured to transmit light components within a predetermined angular range centered on an oblique direction with respect to the normal direction (angle θ1 described later) in the X direction (i.e., viewed from the Y direction), while blocking light components outside of the above angular range.

[0027] The optical element 30 reflects light incident from its bottom side to its top side. It also reflects incident light incident from its bottom side at an oblique angle to the normal direction perpendicular to the plane, approximately in the front direction (normal direction). The optical element 30 forms an aerial image in the air in front of the aerial display device 1. The aerial image is parallel to the element plane of the optical element 30 and is a two-dimensional image. The element plane refers to a virtual plane extending in the in-plane direction of the optical element 30. The term "element plane" has the same meaning as "in-plane." The same meaning applies to the element planes of other elements. An observer in front of the optical element 30 can see the aerial image.

[0028] The input device 40 is configured to detect the observer's input operation. The input device 40 is configured to detect objects without contact. The input device 40 forms a detection area in the two-dimensional spatial region where the aerial image 2 displayed by the aerial display device 1 is located. The input device 40 detects objects (things) present in the detection area and also detects the position of the objects. The input device 40 is composed of, for example, a capacitive sensing element. A capacitive sensing element can detect objects located in the air above it without contact. Control wiring 41 is connected to the input device 40.

[0029] [1-1-1] Configuration of the light control element 20 Figure 2 is a perspective view of the optical control element 20 shown in Figure 1. Figure 2 also shows a partially enlarged view of the optical control element 20.

[0030] The light control element 20 comprises a plurality of transparent members 21 and a plurality of light-shielding members 22. Each of the plurality of transparent members 21 and the plurality of light-shielding members 22 extends in the Y direction. The plurality of transparent members 21 and the plurality of light-shielding members 22 are arranged alternately along the X direction such that adjacent members are in contact with each other.

[0031] The transparent member 21 extends in an oblique direction at an angle θ1 with respect to the normal direction (Z direction) in the XZ plane. The transparent member 21 is a parallelogram with its side surface inclined by an angle θ1 in the XZ plane. The transparent member 21 transmits light. Glass or a transparent resin (including acrylic resin) can be used as the transparent member 21.

[0032] The light-shielding member 22 extends in an oblique direction at an angle θ1 with respect to the normal direction in the XZ plane. The light-shielding member 22 is a parallelogram with its side surface inclined by an angle θ1 in the XZ plane. The light-shielding member 22 blocks light. The thickness of the light-shielding member 22 is set to be thinner than the thickness of the transparent member 21. For example, two adjacent light-shielding members 22 are arranged so that their ends slightly overlap in the Z direction. As the light-shielding member 22, for example, a resin mixed with black dye or pigment is used.

[0033] The multiple transparent members 21 and the multiple light-shielding members 22 may be provided on a transparent substrate with a rectangular planar shape.

[0034] The light control element 20 configured in this way can transmit display light such that the light intensity peaks in the oblique direction at an angle θ1 with respect to the normal direction. The light component transmitted through the transparent member 21 at an angle θ1 is emitted from the light control element 20 at an emission angle (refraction angle) θ2. For example, the light control element 20 is configured to block light components outside the range of 30°±30° with respect to the normal direction. Preferably, the light control element 20 is configured to block light components outside the range of 30°±20° with respect to the normal direction.

[0035] As a modified example, the aerial display device 1 may be configured by omitting the light control element 20.

[0036] [1-1-2] Configuration of the optical element 30 Figure 3 is a perspective view of the optical element 30 shown in Figure 1. Figure 3 also shows a magnified view of a portion of the optical element 30.

[0037] The optical element 30 comprises a substrate 31 and a plurality of optical elements 32. The substrate 31 is planar in the XY plane and has a rectangular parallelepiped.

[0038] Multiple optical elements 32 are provided on the bottom surface of the base material 31. Each of the multiple optical elements 32 is composed of a triangular prism. The optical elements 32 are arranged such that three sides of the triangular prism are parallel to the XY plane, and one side is in contact with the base material 31. Each of the multiple optical elements 32 extends in the Y direction and is arranged in a line in the X direction. In other words, the multiple optical elements 32 have a sawtooth shape in the XZ plane.

[0039] Each of the multiple optical elements 32 has an incident surface 33 and a reflecting surface 34. Viewed from the Y direction, the left side is the incident surface 33, and the right side is the reflecting surface 34. The incident surface 33 is the surface onto which light from the display element 10 enters. The reflecting surface 34 is the surface that reflects light incident on the incident surface 33 from the outside within the optical element 32. The angle between the incident surface 33 and the reflecting surface 34 is θ p It has.

[0040] The base material 31 and the optical element 32 are made of a transparent material. The optical element 32 is formed integrally with the base material 31, for example, using the same transparent material as the base material 31. Alternatively, the base material 31 and the optical element 32 may be formed separately, and the optical element 32 may be bonded to the base material 31 using a transparent adhesive. Glass or a transparent resin (including acrylic resin) can be used as the transparent material constituting the base material 31 and the optical element 32.

[0041] The optical element 30 configured in this way reflects incident light internally to form a real image in the air. The optical element 30 also forms an aerial image at a position directly in front of the element surface.

[0042] [1-1-3] Block configuration of the aerial display device 1 Figure 4 is a block diagram of the aerial display device 1. The aerial display device 1 comprises a control unit 50, a storage unit 51, an input / output interface (input / output IF) 52, a display element 10, and an input device 40. The control unit 50, the storage unit 51, and the input / output interface 52 are connected to each other via a bus 53.

[0043] The input device 40 comprises a sensor unit 42 and a sensor controller 43. The sensor unit 42 is composed of a capacitive sensor. The sensor unit 42 is constructed by stacking a plurality of first transparent electrodes, a transparent insulating layer, and a plurality of second transparent electrodes in that order. The sensor unit 42 detects an object non-contactually based on the change in capacitance between the first transparent electrodes and the second transparent electrodes. The sensor controller 43 applies a voltage to the sensor unit 42 and controls the operation of the sensor unit 42.

[0044] The display element 10 displays an image on its display surface. The input / output interface 52 is connected to the display element 10 and the input device 40. The input / output interface 52 performs interface processing for each of the display element 10 and the input device 40 according to a predetermined standard.

[0045] The control unit 50 is composed of one or more processors, such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 50 implements various functions by executing programs stored in the memory unit 51. The control unit 50 includes a display processing unit 50A, a sensing processing unit 50B, and a correction unit 50C.

[0046] The display processing unit 50A controls the operation of the display element 10. The display processing unit 50A transmits an image signal to the display element 10 and causes the display element 10 to display an image. The display processing unit 50A generates the image to be displayed by the display element 10. The display processing unit 50A can use image data stored in the storage unit 51. The display processing unit 50A may also acquire image data from an external source using a communication function (not shown). The display processing unit 50A can select the image to be displayed by the display element 10 based on the information received by the input device 40.

[0047] The sensing processing unit 50B controls the operation of the input device 40. The sensing processing unit 50B causes the input device 40 to form a detection area. Based on the detection results from the input device 40, the sensing processing unit 50B determines the position of objects present in the detection area.

[0048] The correction unit 50C calculates the amount of deviation between the reference position and the observer's touch position using the position of the image displayed on the display element 10 (reference position) and the detected position detected by the input device 40. The correction unit 50C corrects the original image data using the calculated amount of deviation and generates corrected image data.

[0049] The storage unit 51 includes non-volatile storage devices such as ROM (Read Only Memory), HDD (Hard Disk Drive), and SSD (Solid State Drive), and volatile storage devices such as RAM (Random Access Memory) and registers. The storage unit 51 stores the program executed by the control unit 50. The storage unit 51 stores various data necessary for controlling the control unit 50. The storage unit 51 stores the image data displayed by the aerial display device 1.

[0050] [1-2] Operation Next, the operation of the aerial display device 1 configured as described above will be explained.

[0051] [1-2-1] Basic operation of the aerial display device 1 First, the basic operation of the aerial display device 1 will be explained. Figure 5 is a side view of the aerial display device 1 in the XZ plane. The arrows in Figure 5 represent the approximate optical path. For simplicity, the refraction angle of the components is not considered in Figure 5. The basic operation described here is the operation of the aerial display device 1 when viewed from the Y direction.

[0052] The display element 10 displays an image on its screen (display surface). The display element 10 emits light radially from any point on its screen, due to the image. The light emitted from any point on the display element 10 is incident on the light control element 20. Of the light emitted from the display element 10, the light component at angle θ2 (specifically, including the light component in a predetermined angular range centered on angle θ3) is transmitted through the light control element 20. The oblique light that has been transmitted through the light control element 20 is incident on the optical element 30. The optical element 30 reflects the incident light to the opposite side of the light control element 20, forming an aerial image 2 in the air. The aerial image 2 is a two-dimensional image parallel to the display element 10.

[0053] Figure 6 is a perspective view illustrating the reflection of light in the optical element 30. Figure 7 is a side view of the XZ plane illustrating the reflection of light in the optical element 30. Figure 7 shows the optical element 30 as seen by observer 3 with both eyes (i.e., the line connecting both eyes) parallel to the X direction. Figure 8 is a side view of the YZ plane illustrating the reflection of light in the optical element 30. Figure 8 shows the optical element 30 as seen by observer 3 with both eyes parallel to the Y direction.

[0054] Light emitted from any point "o" of the light control element 20 enters the incident surface 33 of the optical element 30 and reaches the reflection surface 34. Light that reaches the reflection surface 34 at an angle greater than the critical angle with respect to the normal direction of the reflection surface 34 is totally reflected by the reflection surface 34 and is emitted from the plane opposite to the side of the optical element 32 of the optical element 30. The critical angle is the smallest angle of incidence beyond which total reflection occurs. The critical angle is the angle with respect to the perpendicular to the incident surface.

[0055] In the XZ plane of FIG. 7, among the light emitted from point "o", the light component contributing to the display is totally reflected by the reflection surface 34 of the optical element 32, and this light forms an image in the air to generate an aerial image.

[0056] In the YZ plane of FIG. 8, among the light emitted from point "o", the light component not contributing to the display is not reflected by the reflection surface 34 of the optical element 32, and since this light does not form an image in the air, it does not contribute to the generation of the aerial image.

[0057] That is, the condition for the observer 3 to visually recognize the aerial image is that the two eyes of the observer 3 are parallel in the X direction or in a state close thereto (for example, within ±10 degrees with respect to the X direction). Also, when the two eyes of the observer 3 move the viewing point along the Y direction in a state parallel to the X direction or close thereto, the observer 3 can always recognize the aerial image.

[0058] FIG. 9 is a diagram for explaining the angular conditions of the incident surface 33 and the reflection surface 34 in the optical element 30.

[0059] Let the angle of the incident surface 33 with respect to the Z direction (the direction perpendicular to the element surface) be θ3, the angle of the reflection surface 34 with respect to the Z direction be θ4, and the angle formed by the incident surface 33 and the reflection surface 34 be θ p Then. The angle θ p is represented by the following formula (1). θ p =θ3 + θ4 ··· (1)

[0060] The light emitted from the light control element 20 at an angle θ2 is incident on the incident surface 33. Let the refractive index of the material of the optical element 30 be n p , and the refractive index of air be 1. Let the incident angle on the incident surface 33 be θ5 and the refraction angle be θ6. Let the incident angle on the reflection surface 34 be θ7 and the reflection angle be θ8 (= θ7). Let the incident angle on the upper surface of the optical element 30 be θ9 and the refraction angle be θ 10 Then. The refraction angle θ 10 is the exit angle of the optical element 30. The exit angle θ 10 is represented by the following formula (2). θ 10 = sin -1 (n p * sin(sin-1 ((1 / n p )*sin(90°-(θ2+θ3)))+θ3+2θ4-90°)) ···(2)

[0061] The critical angle at the reflective surface 34 is expressed by the following equation (3). Critical angle<θ7(=θ8) Critical angle=sin -1 (1 / n p ) ···(3)

[0062] In other words, the angle of incidence θ7 at the reflective surface 34 is set to be greater than the critical angle at the reflective surface 34. To put it another way, the angle θ4 of the reflective surface 34 is set so that the angle of incidence of light incident on the reflective surface 34 is greater than the critical angle.

[0063] Furthermore, the light incident on the incident surface 33 is set so that it is not totally reflected at the incident surface 33. In other words, the angle θ3 of the incident surface 33 is set so that the angle of incidence of the light incident on the incident surface 33 is smaller than the critical angle.

[0064] The angle between the element surface of the optical element 30 and the plane of the aerial image 2, and the distance between the element surface of the optical element 30 and the plane of the aerial image 2, can be adjusted by optimally setting the angle of light incident on the optical element 30, the refractive index of the optical element 30, the angle of the incident surface 33 of the optical element 30, and the angle of the reflective surface 34 of the optical element 30.

[0065] [1-2-2] Display operation Next, the display operation of the aerial display device 1 will be described.

[0066] Figure 10 illustrates the display operation of the aerial display device 1. In Figure 10, the aerial display device 1 is positioned so that the Y direction is vertical. That is, the aerial display device 1 is positioned so that the aerial image is displayed in front of an upright observer 3. Figure 10 shows how observer 3 views the aerial display device 1 from the front (specifically, how observer 3 views the center of the aerial display device 1).

[0067] The aerial image 2 is, for example, composed of a push button. The plane on which the aerial image 2 is located is called the aerial image plane 60. That is, the aerial image 2 is displayed so as to be located on the aerial image plane 60. The aerial image plane 60 is a virtual plane and is parallel to the element plane of the optical element 30.

[0068] The input device 40 forms a detection area 44 in the two-dimensional region occupied by the aerial image 2. The detection area 44 is located on the aerial image surface 60 and has the same area as the aerial image 2. For example, when observer 3 touches the aerial image 2 with their finger 3A, the finger 3A passes through the detection area 44. The input device 40 detects the finger 3A in the detection area 44 and determines that the aerial image 2 has been touched. The sensing processing unit 50B determines whether or not the detected finger 3A is within the detection area 44.

[0069] Specifically, the input device 40 detects objects present on the aerial image plane 60. The sensing processing unit 50B stores the detection area 44 on the aerial image plane 60 as data, and determines that the aerial image 2 has been touched if the position detected by the input device 40 is included in the detection area 44. Note that the position of the detection area 44 does not have to be the same as the position of the aerial image 2; it may be positioned slightly offset from the aerial image 2 towards the input device 40.

[0070] Figure 11 illustrates the display operation of the aerial display device 1 when observer 3 moves their viewpoint. Observer 3 is assumed to move their viewpoint along the Y direction. For example, an observer viewing from above the center line of the aerial display device 1 (shown as a horizontal dashed line in Figure 11) corresponds to a tall person or adult, while an observer viewing from below the center line of the aerial display device 1 corresponds to a short person or child.

[0071] The optical element 30 forms an aerial image 2 so as to follow the viewpoint of observer 3. This is due to the structure of the optical element 30, which does not reflect light in the Y direction. The aerial image 2 is displayed at the position where a straight line connecting the image on the screen of the display element 10 and the viewpoint of observer 3 intersects with the aerial image plane 60. In the example in Figure 11, observer 3 viewing from above the centerline of the aerial display device 1 sees the aerial image 2 at a position above the centerline, and observer 3 viewing from below the centerline of the aerial display device 1 sees the aerial image 2 at a position below the centerline.

[0072] Assume that the detection area 44 of the input device 40 is fixed in the position shown in Figure 11. For example, if an observer 3 viewing from above the center line of the aerial display device 1 touches the aerial image 2, the position of the aerial image 2 and the detection area 44 may be misaligned, making it impossible to detect the touch operation by the observer 3. Therefore, in this embodiment, the position of the image displayed on the display element 10 (i.e., the position of the aerial image 2) is corrected according to the position of the observer 3's viewpoint.

[0073] Figure 12 is a flowchart illustrating the operation of the aerial display device 1. The control unit 50 can execute the standby mode (also called the first operation mode) and the input mode (also called the second operation mode) in this order. The standby mode is an operation mode in which the position of the observer 3's viewpoint is determined. The input mode is an operation mode in which input operations from the observer 3 are accepted.

[0074] First, the control unit 50 enters standby mode. Specifically, the display processing unit 50A displays a standby screen on the display element 10 (step S100).

[0075] Figure 13 illustrates the standby screen in standby mode. Figure 13 schematically shows the standby screen displayed on the display element 10.

[0076] The screen of the display element 10 is defined by XY coordinates. XY coordinates (orthogonal coordinates) have orthogonal X and Y coordinates. For example, the center of the screen of the display element 10 is the origin (0,0). In a plan view from the Z direction, the origin (0,0) is similarly defined for each element of the aerial display device 1. The origin is sometimes called the center. In Figure 13, the two orthogonal dashed lines represent the X and Y axes. The standby screen includes a point image 11. The point image 11 is an image that serves as a marker for the observer 3 to touch with their finger. In the example in Figure 13, the point image 11 is an image of a finger. The point image 11 is positioned, for example, at the reference position of the screen of the display element 10 (in this embodiment, the origin (0,0)).

[0077] Figure 14 illustrates the operation of the aerial display device 1 in standby mode. In Figure 14, the light control element 20 and optical element 30 of the aerial display device 1 are omitted from the illustration. The same omissions apply to Figures 15 to 17.

[0078] In Figure 14, observer 3 is observing the aerial image 2 from the front of the aerial display device 1. The display element 10 is displaying a standby screen including a point image 11. In Figure 14, the point image 11 is simplified and shown as a circle. Due to the action of the optical element 30, the aerial image 2 corresponding to the point image 11 is displayed on the aerial image plane 60. In a plan view from the Z direction, the aerial image 2 and the point image 11 are positioned at the same location. That is, in a plan view, the aerial image 2 and the point image 11 overlap.

[0079] Figure 15 illustrates the operation of the aerial display device 1 in standby mode when the observer's viewpoint is off-center. Figure 15 shows observer 3 viewing the aerial image 2 from above the centerline of the aerial display device 1.

[0080] Observer 3 sees the aerial image 2, which is shifted above the center line of the aerial display device 1 (towards the positive side in the Y direction).

[0081] The input device 40 detects an object present in the aerial image plane 60 and also detects the observer 3's touch operation (step S101). If the observer 3's touch operation is detected (step S101 = Yes), the sensing processing unit 50B determines the touch position based on the detection result of the input device 40 (step S102).

[0082] Next, the correction unit 50C calculates the amount of deviation Δy from the center (reference position) of the aerial image plane 60 to the touch position (step S103).

[0083] Next, the control unit 50 executes the input mode. That is, the correction unit 50C corrects the original image data using the displacement amount Δy calculated in step S103 and generates corrected image data (step S104). For example, if the original image is located at the origin (0,0), the correction unit 50C corrects the original image data so that the image displayed on the display element 10 is located at coordinates (0,-Δy).

[0084] Next, the display processing unit 50A displays an input screen on the display element 10 (step S105). The input screen is generated using corrected image data. The input screen is a screen that accepts input operations from the observer 3 and is an operation screen for displaying the original aerial image.

[0085] Figure 16 is a schematic diagram illustrating the process of generating corrected image data. The input screen includes an input image 12. The input image 12 is, for example, a push button. In Figure 16, the input image 12 is simplified and shown as a circle. In the image data before correction, the input image 12 is assumed to be located at the origin (0,0). The correction unit 50C corrects the position of the input image 12 so that it is shifted by an amount Δy in the direction opposite in sign to the amount Δy calculated in standby mode. Sign refers to positive or negative sign. In the example in Figure 16, the position of the input image 12 is corrected so that it is shifted by an amount Δy in the negative direction of the Y coordinate.

[0086] In other words, the correction unit 50C generates corrected image data such that the input image 12 is positioned at a location where the line connecting the observer's viewpoint and the position where the uncorrected input image is displayed as an aerial image (i.e., the position of the detection area 44) intersects with the screen of the display element 10.

[0087] Next, the input device 40 forms a detection area 44. The detection area 44 is located on the aerial image plane 60. In a plan view from the Z direction, the detection area 44 is positioned at the same location as the input image before correction. The input device 40 and the sensing processing unit 50B detect objects present on the aerial image plane 60 and also detect the touch operation of the observer 3 (step S106).

[0088] Figure 17 is a schematic diagram illustrating the input operations of observer 3 in input mode. The display element 10 displays an input screen including the input image 12 using corrected image data. Observer 3 perceives the input image 12 as an aerial image 2 through the action of the optical element 30.

[0089] The input device 40 and the sensing processing unit 50B form a detection area 44. In the example shown in Figure 17, the detection area 44 is positioned at the same origin (0,0) as the input image before correction. When observer 3 looks at the aerial image 2, the detection area 44 is positioned at the same location as the aerial image 2. In the state shown in Figure 17, if observer 3 touches the aerial image 2 with finger 3A, finger 3A passes through the detection area 44. Therefore, the input device 40 and the sensing processing unit 50B can accurately detect finger 3A in the detection area 44.

[0090] Figure 18 illustrates an example of an input screen displayed by the display element 10. The input screen includes nine input images "Function" arranged in a matrix.

[0091] Figure 18(a) shows the input screen when there is no displacement. In other words, Figure 18(a) shows the input screen when observer 3 views the aerial image from the front of the aerial display device 1.

[0092] Figure 18(b) shows the input screen when the amount of displacement in the negative direction of the Y coordinate is corrected. In other words, Figure 18(b) shows the input screen when observer 3 views the aerial image from above the center line of the aerial display device 1.

[0093] Figure 18(c) shows the input screen when the amount of displacement in the positive direction of the Y coordinate is corrected. In other words, Figure 18(c) shows the input screen when observer 3 views the aerial image from below the center line of the aerial display device 1.

[0094] Next, if a touch operation by observer 3 is detected (step S106 = Yes), the display processing unit 50A displays a different input screen corresponding to the touch operation on the display element 10 (steps S104 and S105).

[0095] If no touch operation by observer 3 is detected (step S106 = No), the control unit 50 determines whether or not an end operation has occurred (step S107). For example, the control unit 50 generates an input screen that includes an end button. The end button is visible to the observer as an aerial image. If the end button is touched, the control unit 50 determines that an end operation has occurred. Alternatively, the aerial display device 1 may be equipped with an end button (not shown) consisting of a manual push button, and the control unit 50 may determine that an end operation has occurred when the end button is pressed by the observer.

[0096] If no termination operation is performed (step S107=No), the control unit 50 continues the input mode. On the other hand, if a termination operation is performed (step S107=Yes), the aerial display device 1 terminates the process.

[0097] In this embodiment, the amount of displacement in the Y direction has been described, but the amount of displacement in the X direction can also be determined and corrected in the same way as in the Y direction. In this case, the amount of displacement is determined and corrected in both the X and Y coordinates.

[0098] [1-3] Effects of the first embodiment According to the first embodiment, the amount of deviation between the detection area 44 formed by the input device 40 and the aerial image 2 can be determined using the standby mode. Furthermore, in the input mode, the position of the aerial image 2 can be corrected according to the position of the observer's viewpoint 3. This makes it possible to more accurately detect touch operations on the aerial image 2 by the observer 3.

[0099] Furthermore, it is possible to display a two-dimensional aerial image parallel to the display element 10. In addition, multiple elements constituting the aerial display device 1 can be arranged in parallel. This makes it possible to realize an aerial display device 1 that can be miniaturized in the Z direction.

[0100] [2] Second embodiment In the second embodiment, the position of the detection area formed by the input device 40 is corrected using the amount of deviation calculated in standby mode.

[0101] Figure 19 is a flowchart illustrating the operation of the aerial display device 1 according to the second embodiment of the present invention. The operation in standby mode is the same as in the first embodiment. That is, the operation in steps S200 to S203 in Figure 19 is the same as the operation in steps S100 to S103 in Figure 12.

[0102] Next, the control unit 50 executes the input mode. That is, the display processing unit 50A displays the input screen on the display element 10 (step S204).

[0103] Next, the correction unit 50C corrects the position of the detection area 44 (step S205). The correction unit 50C corrects the position of the detection area 44 by a deviation amount Δy of the same sign.

[0104] Figure 20 is a schematic diagram illustrating the operation of correcting the position of the detection area 44. In Figure 20, the light control element 20 and optical element 30 of the aerial display device 1 are omitted from the illustration. The same omission applies to Figure 21. Figure 20 shows how observer 3 views the aerial image 2 from above the center line of the aerial display device 1.

[0105] The display element 10 displays an input screen. The input screen includes an input image 12. The input image 12 is, for example, a push button. In Figure 20, the input image 12 is shown in a simplified form as a circle. The input image 12 is assumed to be located at the origin (0,0). The aerial image 2 is visible to the observer 3, shifted upward from the center of the aerial image plane 60. Specifically, the aerial image 2 is displayed at the position where the line connecting the observer 3's viewpoint and the input image 12 of the display element 10 intersects the aerial image plane 60.

[0106] The correction unit 50C corrects the position of the detection area 44 so that the detection area is shifted by an amount Δy in the same direction as the shift amount Δy calculated in standby mode. In the example in Figure 20, the correction unit 50C corrects the position of the detection area 44 so that it is shifted by an amount Δy in the positive direction of the Y coordinate. In other words, the correction unit 50C corrects the position so that the detection area 44 is located at the point where the line connecting the observer's viewpoint 3 and the input image 12 of the display element 10 intersects with the aerial image plane 60.

[0107] The input device 40 and the sensing processing unit 50B form a detection area 44 based on the correction information corrected by the correction unit 50C. The input device 40 and the sensing processing unit 50B detect objects present on the aerial image plane 60 and also detect the touch operation of the observer 3 (step S206).

[0108] Figure 21 is a schematic diagram illustrating the input operations of observer 3 in input mode. The display element 10 displays an input screen including the input image 12. Observer 3 perceives the input image 12 as an aerial image 2 through the action of the optical element 30.

[0109] The input device 40 and the sensing processing unit 50B form a detection area 44. In the example shown in Figure 21, the detection area 44 is positioned offset upward by a displacement Δy from the center of the aerial image plane 60. When observer 3 looks at the aerial image 2, the detection area 44 is positioned at the same location as the aerial image 2. In the state shown in Figure 21, if observer 3 touches the aerial image 2 with their finger 3A, the finger 3A passes through the detection area 44. Therefore, the input device 40 can accurately detect the finger 3A that is present in the detection area 44.

[0110] Next, if a touch operation by observer 3 is detected (step S206 = Yes), the display processing unit 50A displays a different input screen corresponding to the touch operation on the display element 10 (step S204).

[0111] If no touch operation by observer 3 is detected (step S206 = No), the control unit 50 determines whether or not an termination operation has occurred (step S207). The operation in step S207 is the same as the operation in step S107 in Figure 12.

[0112] According to the second embodiment, the operation of the display element 10 can be corrected only for the position of the detection area 44 while maintaining normal operation. Other effects are the same as in the first embodiment.

[0113] As a variation, both the position correction of the input image 12 on the input screen and the position correction of the detection area 44 may be performed. For example, the position correction of the input image 12 on the input screen may be performed by a correction amount Δy / 2, and the position correction of the detection area 44 may be performed by a correction amount Δy / 2. This makes it possible to reduce the correction amount for both the input image and the detection area.

[0114] [3] Third embodiment The third embodiment uses a motion sensor to detect changes in the observer, and repeats the standby mode based on the detection result.

[0115] Figure 22 is a block diagram of an aerial display device 1 according to a third embodiment of the present invention. The aerial display device 1 further comprises a human presence sensor 54.

[0116] The motion sensor 54 detects an observer in front of the aerial display device 1. "In front of the aerial display device 1" refers to the area from the optical element 30 toward the aerial image. The motion sensor 54 is composed of, for example, an infrared sensor. The motion sensor 54 is positioned, for example, near the optical element 30 and is housed in, for example, a housing (not shown).

[0117] The motion sensor 54 is connected to the input / output interface 52. The input / output interface 52 performs interface processing on the motion sensor 54 according to a predetermined standard.

[0118] The control unit 50 further includes a mode switching unit 50D. The mode switching unit 50D controls the system to re-execute the standby mode based on the detection result of the human presence sensor 54.

[0119] Figure 23 is a flowchart illustrating the operation of the aerial display device 1. The control unit 50 executes the standby mode (step S300). Subsequently, the control unit 50 executes the input mode (step S301). The operation of the standby mode and the input mode is the same as in the first or second embodiment.

[0120] Next, the motion sensor 54 detects an observer in front of the aerial display device 1. Based on the detection result of the motion sensor 54, the mode switching unit 50D determines whether or not the observer in front of the aerial display device 1 has changed (step S302).

[0121] If the observer in front of the aerial display device 1 changes (step S302 = Yes), the mode switching unit 50D repeats the standby mode again.

[0122] If the observer in front of the aerial display device 1 has not changed (step S302=No), the control unit 50 determines whether or not a termination operation has occurred (step S303). If no termination operation has occurred (step S303=No), the control unit 50 continues the input mode. On the other hand, if a termination operation has occurred (step S303=Yes), the aerial display device 1 terminates the process.

[0123] According to the third embodiment, each time the observer changes, the position of the aerial image 2 or the detection area 44 can be corrected to match the observer's height.

[0124] As a variation, the aerial display device 1 may be equipped with a toggle button instead of the human presence sensor 54. The toggle button consists of a manual push button. The toggle button is provided at any position on the aerial display device 1 facing the observer 3. When the toggle button is pressed, the mode switching unit 50D repeats the standby mode again.

[0125] [4] Fourth Embodiment The fourth embodiment determines the observer's height in standby mode and selects the image to display according to the determination result.

[0126] Figure 24 is a flowchart illustrating the operation of the aerial display device 1 according to the fourth embodiment of the present invention. The block diagram of the aerial display device 1 is the same as in Figure 4. The operation in standby mode (steps S400 to S403) is the same as in the first embodiment (steps S100 to S103 in Figure 12).

[0127] Next, the display processing unit 50A determines whether the touch position is higher than a threshold (step S404). Specifically, the display processing unit 50A compares the Y coordinate of the touch position with the threshold for the Y coordinate. The threshold is used to distinguish whether the observer is relatively tall or short. Relatively tall height assumes an adult observer, while relatively short height assumes a child observer or an observer using a wheelchair. As an example, step S404 determines whether the observer is an adult or a child.

[0128] If the touch position is higher than the threshold (step S404 = Yes), the control unit 50 executes an input mode for the first input screen (step S405). In step S405, the display processing unit 50A selects an image for the first input screen. For example, the first input screen includes an image with content that is easy for adults to understand. The specific operation of the input mode is the same as steps S104 to S107 described in the first embodiment.

[0129] If the touch position is below a threshold (step S404 = No), the control unit 50 executes an input mode for the second input screen (step S406). In step S406, the display processing unit 50A selects an image for the second input screen. For example, the second input screen may include an image with content that is easy for children to understand, and an image that uses many hiragana and katakana characters. The second input screen may also include an image suitable for an observer using a wheelchair. The specific operation of the input mode is the same as steps S104 to S107 described in the first embodiment.

[0130] According to the fourth embodiment, the observer's viewpoint height can be determined using the standby mode. Then, an image corresponding to the observer's viewpoint height can be displayed as an aerial image.

[0131] [5] Fifth embodiment The fifth embodiment involves changing the size of the display element 10 and the optical element 30.

[0132] Figure 25 is a perspective view of an aerial display device 1 according to a fifth embodiment of the present invention. The area of ​​the display element 10 is set to be larger than the area of ​​the optical element 30. For example, the area of ​​the display element 10 is set to be 1.2 times or more and 1.5 times or less the area of ​​the optical element 30.

[0133] The area of ​​the light control element 20 is set to be approximately the same as the area of ​​the display element 10. The area of ​​the input device 40 (specifically, the area of ​​the sensor part) is set to be approximately the same as the area of ​​the optical element 30.

[0134] The image displayed on the display element 10 is displayed with, for example, a correction to its Y-coordinate. If an observer views the aerial image from a more oblique angle, there is a possibility that the observer may not be able to see the entire aerial image that should be displayed. According to the fifth embodiment, a sufficient range can be secured for correcting the position of the input image displayed on the display element 10. This makes it possible to display the entire aerial image more reliably.

[0135] Furthermore, as the distance between the display element 10 and the optical element 30 increases, for example, the amount of floating of the aerial image (distance from the optical element 30 to the aerial image) increases, but the amount of displacement Δy and the correction amount when an observer views the aerial image from an oblique direction also increase. Therefore, it is desirable to set the ratio of the area of ​​the display element 10 to the area of ​​the optical element 30 to increase as the distance between the display element 10 and the optical element 30 increases. This allows for the optimal setting of the size of the aerial display device 1.

[0136] [6] Variant In each of the above embodiments, the display element 10 and the optical element 30 are arranged in parallel. However, the invention is not limited to this, and the display element 10 may be arranged diagonally with respect to the optical element 30. The angle between the display element 10 and the optical element 30 is set to a range greater than 0 degrees and less than 45 degrees. In this modified example, the light control element 20 can be omitted.

[0137] In each of the embodiments described above, the left side of the optical element 32 is defined as the incident surface 33, and the right side is defined as the reflective surface 34. However, the invention is not limited to this, and the incident surface 33 and the reflective surface 34 may be configured in reverse. In this case, the operation of the aerial display device 1 described in the embodiments will also be reversed left and right.

[0138] In each of the above embodiments, the optical element 30 is not limited to the above configuration, and other types of imaging elements can be used. For example, the optical element 30 may be configured with a two-sided corner reflector array in which two-sided corner reflectors are arranged in an array. When a two-sided corner reflector array is used, an aerial image is formed at a position symmetrical to the display element.

[0139] In the embodiments described above, a liquid crystal display element is used as an example for the display element 10, but it is not limited to this, and various types of display elements can be used. For example, the display element 10 can be a self-emissive organic EL (electroluminescence) display element or a micro-LED (light-emitting diode) display element. A micro-LED display element is a display element that emits R (red), G (green), and B (blue) light, which constitute the pixels, using LEDs.

[0140] The present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention. [Explanation of symbols]

[0141] 1...Aerial display device, 2...Aerial image, 3...Observer, 10...Display element, 11...Point image, 12...Input image, 20...Light control element, 21...Transparent member, 22...Light-shielding member, 30...Optical element, 31...Substrate, 32...Optical element, 33...Incident surface, 34...Reflective surface, 40...Input device, 41...Wiring, 42...Sensor unit, 43...Sensor controller, 44...Detection area, 50...Control unit, 50A...Display processing unit, 50B...Sensing processing unit, 50C...Correction unit, 50D...Mode switching unit, 51...Storage unit, 52...Input / output interface, 53...Bus, 54...Human presence sensor, 60...Aerial image surface.

Claims

1. A display element that displays an image, An optical element is positioned to receive light from the display element and configured to form an aerial image in the air opposite to the display element, An input device configured to form a detection region on the aerial image plane including the region where the aerial image is displayed, and to detect objects present in the detection region, A control unit that controls the operation of the display element and the input device, It is equipped with, The control unit executes the first operation mode and the second operation mode in sequence. In the first operating mode, the first image is displayed at the reference position of the display element. Based on the detection position by the input device, the amount of deviation between the reference position and the detection position is calculated. In the second operating mode, the position of the image to be displayed is corrected based on the amount of displacement, and the second image is displayed on the display element. Aerial display device.

2. In the second operating mode, the control unit calculates the amount of displacement using mutually orthogonal X and Y coordinates, and corrects the position of the image to be displayed in a direction in which the sign of the amount of displacement is reversed. The aerial display device according to claim 1.

3. In the second operating mode, the input device forms the detection region in the region that overlaps with the image before position correction in a plan view. The aerial display device according to claim 1.

4. A display element that displays an image, An optical element is positioned to receive light from the display element and configured to form an aerial image in the air opposite to the display element, An input device configured to form a detection region on the aerial image plane including the region where the aerial image is displayed, and to detect objects present in the detection region, A control unit that controls the operation of the display element and the input device, It is equipped with, The control unit executes the first operating mode and the second operating mode in sequence. In the first operating mode, the first image is displayed at the reference position of the display element. Based on the detection position by the input device, the amount of deviation between the reference position and the detection position is calculated. In the second operating mode, the second image is displayed on the display element, and the position of the detection area is corrected based on the amount of displacement. Aerial display device.

5. In the second operating mode, the control unit calculates the amount of displacement using mutually orthogonal X and Y coordinates, and corrects the position of the detection area in the same direction as the sign of the amount of displacement. The aerial display device according to claim 4.

6. In the second operating mode, the control unit displays the second image in the region that overlaps with the detection region before position correction in a plan view. The aerial display device according to claim 4.

7. It is further equipped with a motion sensor that detects when the observer changes, The control unit, based on the information from the human presence sensor, executes the first operating mode again. The aerial display device according to claim 1 or 4.

8. The input device includes a capacitive sensor unit. The aerial display device according to claim 1 or 4.

9. The optical element includes a planar substrate and a plurality of optical elements provided beneath the substrate, each extending in a first direction and arranged in a second direction perpendicular to the first direction. Each of the plurality of optical elements is inclined with respect to the normal direction of the substrate and has an incident surface and a reflective surface that are in contact with each other. The aerial display device according to claim 1 or 4.

10. The display element and the optical element are arranged parallel to each other. The aerial display device according to claim 1 or 4.

11. The system further comprises a light control element disposed between the display element and the optical element, configured to transmit oblique light components from the display element. The aerial display device according to claim 1 or 4.

12. The light control element includes a plurality of transparent members and a plurality of light-shielding members, Each of the aforementioned plurality of transparent members extends in a first direction and is arranged in a second direction perpendicular to the first direction. Each of the plurality of light-shielding members extends in the first direction and is arranged in the second direction. The plurality of transparent members and the plurality of light-shielding members are arranged alternately. The plurality of light-shielding members are inclined with respect to the normal of the light control element. The aerial display device according to claim 11.

Images