Display device, display method, and program

Abstract

To easily make the sight line match with an aerial image.SOLUTION: A display device comprises: a display unit which displays an aerial image obtained by forming an image displayed on a display surface in the air; a detection unit which detects whether or not an object exists in a monitoring area in a display area displaying the aerial image; and a display control unit which causes the monitoring area to display the aerial image when the object exists in the monitoring area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] Devices that display three-dimensional images in midair have been developed (see, for example, Patent Document 1). Using a midair image display device, a variety of human interfaces can be realized. For example, operations such as rotating or moving a midair image can be performed while virtually touching the midair image with hands or fingers. Also, buttons or the like can be displayed as midair images, and various operations can be performed while virtually touching the midair image with hands or fingers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-67933 Summary of the Invention [Problem to be solved by the invention]

[0004] When building a human interface using a mid-air image display device, it is assumed that the user focuses their gaze on the physical display device in the initial state when the mid-air image is not displayed. Therefore, when a mid-air image is displayed, it is difficult for the user to focus their gaze on the displayed mid-air image, which creates a problem in that it is difficult for the user to recognize the mid-air image.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device, a display method, and a program that can make it easier to align the line of sight with an aerial image. [Means for solving the problem]

[0006] A display device according to one aspect of the present invention includes a display unit for displaying an aerial image formed in the air by imaging an image displayed on a display surface;a display area provided on the front surface of the display surface; In the surveillance area located , manipulating the aerial image a detection unit that detects whether an object is present; The detection unit In the monitoring area The aforementioned Object exists was detected as In this case, the monitoring area In accordance with the detected position of the object in the display screen, a position adjacent to the detected position of the object in the depth direction perpendicular to the display screen is displayed. and a display control unit that displays the aerial image.

[0007] A display method according to one aspect of the present invention is a display method performed by a display device that is a computer, the display unit displaying a mid-air image by imaging an image displayed on a display surface in mid-air, and a detection unit detecting a display area in which the mid-air image is displayed. a display area provided on the front surface of the display surface; In the surveillance area located , manipulating the aerial image The display control unit detects whether an object is present or not. The detection unit In the monitoring area The aforementioned Object exists was detected as In this case, the monitoring area In accordance with the detected position of the object in the display screen, a position adjacent to the detected position of the object in the depth direction perpendicular to the display screen is displayed. The aerial image is displayed.

[0008] A program according to one aspect of the present invention causes a display device that is a computer to display an aerial image by forming an image displayed on a display surface in the air, and displays the aerial image in a display area a display area provided on the front surface of the display surface; In the surveillance area located , manipulating the aerial image Detect whether an object is present in the monitoring area The aforementioned Object exists was detected as In this case, the monitoring area In accordance with the detected position of the object in the display screen, a position adjacent to the detected position of the object in the depth direction perpendicular to the display screen is displayed. A program for displaying the aerial image. [Effects of the Invention]

[0009] According to the present invention, a mid-air image is displayed near the user's hand or finger, making it easier to focus the user's gaze on the mid-air image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing an overall configuration of a display device according to an embodiment; [Figure 2A] FIG. 2 is an explanatory diagram illustrating an outline of the operation of the display device according to the embodiment. [Figure 2B]FIG. 2 is an explanatory diagram illustrating an outline of the operation of the display device according to the embodiment. [Figure 2C] FIG. 2 is an explanatory diagram illustrating an outline of the operation of the display device according to the embodiment. [Figure 3] FIG. 1 is an explanatory diagram of the principle of a lens-type aerial image display device. [Figure 4] FIG. 10 is an explanatory diagram for setting a display position on a lens-type aerial image display device. [Figure 5] FIG. 1 is an explanatory diagram of a mirror-type aerial image display device using a concave mirror. [Figure 6] FIG. 1 is an explanatory diagram of a mirror-type aerial image display device using retroreflective material. [Figure 7] FIG. 1 is an explanatory diagram of position detection using a normal monocular camera. [Figure 8] FIG. 2 is a diagram illustrating an example of the external appearance of a stereo camera. [Figure 9] FIG. 10 is an explanatory diagram illustrating a case where a position in the depth direction is detected using a monocular camera. [Figure 10] FIG. 10 is an explanatory diagram illustrating a case where a position in the depth direction is detected using a monocular camera. [Figure 11] FIG. 10 is an explanatory diagram illustrating a case where an interruption sensor is used to detect a position in the depth direction. [Figure 12] 5 is a flowchart illustrating a process performed by a display control unit in the display device according to the embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a still aerial image. [Figure 14] FIG. 10 is a diagram showing an example of a tracking aerial image. [Figure 15] 10 is a flowchart showing the processing in the display control unit when a still aerial image is displayed. [Figure 16] 10A and 10B are diagrams illustrating changes in display clarity of a still aerial image. [Figure 17] 10 is a flowchart showing processing by a display control unit when a tracking aerial image is displayed. [Figure 18] FIG. 1 is an explanatory diagram of a specific example of a man-machine interface using a display device according to an embodiment. [Figure 19]10 is a flowchart showing a process for realizing a man-machine interface using the display device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing the overall configuration of a display device 1 according to an embodiment. As shown in Fig. 1, the display device 1 is composed of a display unit 10, a display control unit 20, and a detection unit 30. In the following description, the horizontal direction, the vertical direction, and the depth direction of a display surface 10a of the display unit 10 are defined as the X-axis direction, the Y-axis direction, and the Z-axis direction, respectively.

[0012] The display unit 10 forms an image displayed on the display surface in mid-air, thereby displaying an optical image light that appears to float in space as an aerial image IMG. The aerial image IMG is used, for example, for museum exhibition images, sales promotion images, medical images, 3D movies, games, etc. In this embodiment, such aerial image IMG can be used as a man-machine interface. The display unit 10 can set the display position of the aerial image IMG in the horizontal direction (X-axis direction) and vertical direction (Y-axis direction), as well as the position in the depth direction (Z-axis direction). Details of the display unit 10 will be described later.

[0013] The display control unit 20 is, for example, a PC (Personal Computer) and performs various processes based on a program. In this embodiment, the display control unit 20 implements the functions of an aerial image generation unit 21, a position detection unit 22, a display position control unit 23, a motion detection unit 24, and an operation setting unit 25. The aerial image generation unit 21 implements the function of displaying an aerial image IMG of a desired shape on the display unit 10. The position detection unit 22 implements the function of detecting the position of an object in front of the display unit 10 based on a detection signal from the detection unit 30. More specifically, the object in front of the display unit 10 is the user USR's hand or fingers, and the detection unit 30 detects the user USR's hand or fingers as the object. The display position control unit 23 implements the function of controlling the display position of the aerial image IMG according to the position of the object. The motion detection unit 24 detects the movement of the object in front of the display unit 10, i.e., the user USR's hand or fingers, and performs processing to set various operations according to the movement of the object. The operation setting unit 25 executes processing according to the set operation.

[0014] The detection unit 30 detects the position of an object, i.e., the hand or fingers of the user USR, in a monitoring area in the display area where the aerial image is displayed. In this embodiment, the detection unit 30 used can detect the position of an object not only in the horizontal direction (X-axis direction) and vertical direction (Y-axis direction) but also in the depth direction (Z-axis direction). The detection unit 30 may also be configured using multiple devices. Details of the detection unit 30 will be described later.

[0015] 2A to 2C are explanatory diagrams illustrating an outline of the operation of the display device 1 according to the embodiment.

[0016] 2A shows an initial state. In the initial state, the detection unit 30 has not yet detected the hands or fingers of the user USR as objects. Also, in the initial state, the aerial image IMG is not yet displayed.

[0017] As shown in Fig. 2B, when the user USR extends his / her hand in front of the display unit 10, the hand or fingers of the user USR enter the monitoring area, and the hand or fingers of the user USR are detected as an object from the detection signal of the detection unit 30. When the hand or fingers of the user USR are detected in front of the display unit 10, the display control unit 20 calculates the position La of the hand or fingers of the user USR from the detection signal of the detection unit 30. Then, as shown in Fig. 2C, the display control unit 20 displays an aerial image IMG near the position La of the hand or fingers of the user USR.

[0018] In addition, taking into consideration that the user USR will operate the aerial image IMG to set various operations, it is desirable that the aerial image IMG be positioned slightly closer to the position of the display unit 10 than near the position La of the user USR's hand and fingers.

[0019] In this way, in the display device 1 according to the embodiment, when the user USR extends his / her hand in front of the display unit 10, the aerial image IMG is displayed near the position of the user USR's hand and fingers. This allows the user USR to direct his / her line of sight toward the aerial image IMG and view it.

[0020] 2A, in the initial state, the user USR does not extend his / her hand, and no object is detected in the monitoring area of ​​the detection unit 30. Also, at this time, the user USR is usually looking at the display unit 10, and it is assumed that the line of sight LS of the user USR is in a direction toward the display unit 10.

[0021] Now, suppose that the user USR reaches out his / her hand in front of the display unit 10. Typically, the user USR moves his / her hand while looking at the tip of his / her hand. Therefore, when the user USR reaches out his / her hand in front of the display unit 10, it is assumed that the line of sight LS of the user USR is in a direction toward the tip of his / her hand, as shown in FIG. 2B.

[0022] In this embodiment, when the user USR reaches out in front of the display unit 10 and their hand or fingers enter the monitoring area of ​​the detection unit 30, an aerial image IMG is displayed near the hand or fingers of the user USR. Therefore, as shown in Fig. 2C, the aerial image IMG is displayed at the position of the line of sight LS of the user USR, and the user USR can view the aerial image IMG without moving their line of sight.

[0023] Next, the display unit 10 in the display device 1 according to this embodiment will be described. In the display device 1 according to the present embodiment, the display unit 10 displays an aerial image IMG near the position of the hand or fingers of the user USR. Devices capable of displaying an aerial image include lens-type, mirror-type, and holographic-type devices.

[0024] Figure 3 is an explanatory diagram of the principle of a lenticular aerial image display device. As shown in Figure 3, the lenticular aerial image display device comprises a display 101 and a convex lens 102. Image light displayed on the display 101 is focused by the convex lens 102 to generate an optical real image that becomes the aerial image IMG. In this example, a Fresnel lens is used as the convex lens 102.

[0025] In the case of a lenticular aerial image display device, moving the position of the display 101 also moves the focal position, making it possible to move the position of the optical real image that becomes the aerial image IMG. Figure 4 is an explanatory diagram for setting the display position in a lenticular aerial image display device. In the case of a lenticular aerial image display device, as shown in Figure 4, the display 101 is movably fixed by a uniaxial positioner 106, and the position in the depth direction (Z-axis direction) can be set by moving the uniaxial positioner 106 according to a control signal from the display control unit 20.

[0026] FIG. 5 is an explanatory diagram of a mirror-type aerial image display device using a concave mirror. Mirror-type aerial image display devices include those that use a concave mirror and those that use a retroreflective material. This example uses a concave mirror. As shown in FIG. 5, the mirror-type aerial image display device using a concave mirror comprises a display 111 and a concave mirror 112. The image light displayed on the display 111 is reflected by the concave mirror 112 and focused to generate an optical real image that becomes the aerial image IMG. Note that the hatched frame in this figure is a cover that makes the aerial image IMG easier to view, and is formed, for example, by an opaque box. In addition, in this example, the image displayed on the display 111 and the aerial image IMG are upside down. The principle of such a mirror-type aerial image display device using a concave mirror can be understood, for example, from Japanese Patent Application No. 2001-177784.

[0027] Fig. 6 is an explanatory diagram of a mirror-type aerial image display device using a retroreflective material. As shown in Fig. 6, the mirror-type aerial image display device using a retroreflective material comprises a display 121, a retroreflective material 122, and a half mirror 123, and generates an optical real image by two reflections. That is, in the device using a retroreflective material, image light displayed on the display 121 is reflected by the half mirror 123 and the retroreflective material 122, and is focused via the half mirror 123 to generate an optical real image that becomes the aerial image IMG. Even in the case of such a mirror-type display device, the position of the optical real image that becomes the aerial image IMG can be moved in the depth direction (Z-axis direction) by moving the positions of the displays 111 and 121.

[0028] A holographic aerial image display device displays an aerial image by irradiating a hologram that records an interference pattern with a reconstructed light. A holographic aerial image display device can make the reconstructed image appear to move by distorting the image depending on the position of the reconstructed light. Therefore, the position of the optical real image that becomes the aerial image IMG can be set by moving the light source. Alternatively, a holographic aerial image display device may be equipped with multiple light sources and switch between them depending on the position of the image.

[0029] Additionally, in the case of holographic aerial image display devices, there is also a method for adjusting the position where the image appears by dynamically changing the interference pattern. That is, a device has been developed that displays an interference pattern on a spatial light modulator (SLM) and reconstructs the hologram by irradiating it with reconstruction light. An SLM is a device that can change the state of its pixels. Using an SLM, it is possible to adjust the position where the image appears by dynamically changing the interference pattern.

[0030] Next, the detection unit 30 in the display device 1 according to this embodiment will be described. In the display device 1 according to this embodiment, the positions of the hands and fingers of the user USR are detected as objects. FIG. 7 is an explanatory diagram of position detection using a normal monocular camera. As shown in FIG. 7, the horizontal (X-axis direction) and vertical (Y-axis) positions can be detected using a normal monocular camera 300. However, it is difficult to detect the position in the depth direction (Z-axis direction) using only one monocular camera 300.

[0031] A depth camera is a camera that can detect the position in the depth direction (Z-axis direction). Such a depth camera can be used as the detection unit 30. Depth cameras include a stereo camera, a ToF (Time Of Flight) camera, and a structured lighting (projector type) camera.

[0032] Fig. 8 is a diagram showing the external appearance of an example of a stereo camera. As shown in Fig. 8, a stereo camera 301 is equipped with two or more cameras 302a and 302b, and can recognize distance from the parallax between the cameras.

[0033] A ToF camera has a light source located near a single camera, and recognizes the distance to an object by measuring the time it takes for the light emitted from the light source to hit the object being photographed, reflect, and be captured by the camera. ToF cameras are broadly classified into pulse and continuous light types. The pulse type simply measures the time it takes for light from a light source that is emitted for an extremely short time to hit the object and return. The continuous light type irradiates a CW (continuous wave) as a reference light, and calculates distance by measuring the phase difference between the light that hits the object and the reflected light that returns.

[0034] Structured lighting obtains depth information by projecting various patterns onto an object from a projector, causing the patterns to distort on the object's surface. There are various patterns available, such as grids and dot matrices. In addition, invisible wavelengths such as infrared can be used as the light source for the ToF camera or the light emitted using structured lighting, which makes it possible to prevent the user from realizing that their position is being detected.

[0035] The position in the depth direction (Z-axis direction) can also be detected using a monocular camera. FIGS. 9 and 10 are explanatory diagrams of detecting the position in the depth direction using one monocular camera 310. When detecting the position in the depth direction, as shown in FIG. 9, the monocular camera 310 is positioned so that its optical axis direction is perpendicular to the depth direction (Z-axis direction). That is, the monocular camera 310 is positioned so as to capture an image of the user USR from the side. In this case, as shown in FIG. 10, the position in the depth direction of the user's hand and fingers can be detected from the horizontal position Ax of the image of the hand of the user USR captured by the monocular camera 310.

[0036] Position detection in the depth direction (Z-axis direction) can also be performed using an interruption sensor. FIG. 11 is an explanatory diagram of a case where an interruption sensor is used for position detection in the depth direction. The interruption sensor positions a light-emitting element and a light-receiving element directly opposite each other to detect whether an object is present between them. In this example, as shown in FIG. 11, multiple light-emitting elements 321, 321, ... and multiple light-receiving elements 322, 322, ... are arranged side by side in the depth direction (Z-axis direction). The light-emitting elements 321, 321, ... emit infrared light, and the light-receiving elements 322, 322, ... detect the infrared light. As shown in FIG. 11, when the user USR extends his / her hand, the light from the light-emitting elements 321, 321, ... is interrupted depending on the position of the user's hand. As a result, the position of the user USR's hand and fingers in the depth direction (Z-axis direction) can be detected from the detection signals of the light-receiving elements 322, 322, ...

[0037] Next, the processing performed by the display control unit 20 in the display device 1 according to this embodiment will be described. FIG. 12 is a flowchart showing the processing in the display control unit 20 of the display device 1 according to the embodiment.

[0038] (Step S1) The display control unit 20 determines whether or not an object is detected in the monitoring area in front of the display unit 10 based on the detection signal of the detection unit 30. In the initial state, a message such as "Please hold your hand over" may be displayed to prompt the user USR to take action.

[0039] (Step S2) When an object is detected from the detection signal of the detection unit 30 (Step S1: Yes), the display control unit 20 calculates the position of the detected object. Specifically, the detected object is the hand or fingers of the user USR, and the display control unit 20 calculates the position of the hand or fingers of the user USR.

[0040] (Step S3) The display control unit 20 calculates the display position of the aerial image according to the position of the object calculated in step S2. The display position of the aerial image is near the position of the user USR's hand or fingers detected in step S2. Note that, considering that the user USR will operate the aerial image IMG to set various actions, it is desirable to display the aerial image IMG at a position slightly closer to the display unit 10 than near the position La of the user USR's hand or fingers.

[0041] (Step S4) The display control unit 20 displays the aerial image IMG at the position of the object calculated in step S3.

[0042] As described above, in this example, by performing the processing from step S1 to step S4, when the user USR extends his / her hand in front of the display unit 10, the position of the user USR's hand and fingers is detected, and an aerial image IMG is displayed near the user USR's hand and fingers.

[0043] When constructing a man-machine interface using the display device 1 according to this embodiment, the user USR will be able to virtually press, touch, and move the aerial image IMG. Two types of aerial image IMG can be considered for realizing such a man-machine interface: a still aerial image and a tracking aerial image.

[0044] A still aerial image is an aerial image that is fixed and displayed at a predetermined position. FIG. 13 is a diagram showing an example of a still aerial image. This example is an aerial image of an operation panel. This aerial image is a still aerial image IMG1, and the user USR moves their hand or finger to operate a desired button. At this time, if the entire aerial image moves along with the movement of the user USR's hand or finger, it will be difficult for the user USR to operate the desired button. Therefore, a still aerial image IMG1 that includes multiple options like this needs to be fixed and still at a predetermined position.

[0045] A tracked aerial image is an aerial image that moves by tracking the movements of the user USR's hands and fingers. FIG. 14 is a diagram showing an example of a tracked aerial image. This example is an aerial image of an animal. With such a tracked aerial image IMG2, when the user USR moves their hands or fingers, the tracked aerial image IMG2 moves in accordance with the movements.

[0046] FIG. 15 is a flowchart showing the processing performed by the display control unit 20 when a still aerial image is displayed.

[0047] (Step S101) The display control unit 20 determines, from the detection signal of the detection unit 30, whether or not an object is detected in the monitoring area in front of the display unit .

[0048] (Step S102) When an object is detected from the detection signal of the detection unit 30 (step S101: Yes), the display control unit 20 calculates the position of the detected object.

[0049] (Step S103) The display control unit 20 calculates the display position of the aerial image according to the position of the object calculated in step S102.

[0050] (Step S104) The display control unit 20 displays the still aerial image IMG1 near the position of the object calculated in step S103.

[0051] (Step S105) The display control unit 20 determines whether a predetermined waiting time has elapsed, and if the predetermined waiting time has not elapsed (step S105: No), returns the process to step S102.

[0052] (Step S106) If it is determined in step S105 that the predetermined waiting time has elapsed (step S105: Yes), the display control unit 20 moves the still aerial image IMG1 to a specified position (designated position) and then freezes (fixes) the still aerial image IMG1 at the specified position.

[0053] As described above, in this example, by performing the processes from step S101 to step S104, in the initial state, the still aerial image IMG1 is displayed near the hand or fingers of the user USR. Therefore, the still aerial image IMG1 is displayed at the eye level of the user USR, and the user USR can view the still aerial image IMG1 without moving their line of sight. Then, after a predetermined waiting time has elapsed in step S105, the position of the still aerial image IMG1 is moved to and fixed at a specified position in step S106. The waiting time here is the time from when the still aerial image IMG1 is displayed until the user USR's eyes are able to recognize the still aerial image IMG1. Furthermore, when the still aerial image IMG1 is moved to a designated position, it is moved at a speed that the user USR can follow with their eyes. This allows the user USR to once recognize the still aerial image IMG1 displayed at the user USR's line of sight, and then continue to recognize the moving still aerial image IMG1 by following it with their eyes. Therefore, the user USR can recognize that the still aerial image IMG1 has moved to the designated position and perform an operation on the still aerial image IMG1. After the position of the still aerial image IMG1 has been moved to the specified position, the still aerial image IMG1 remains stationary even if the user USR moves their hands or fingers. Therefore, if the still aerial image IMG1 is an operation panel, for example, the user can easily operate a specified button.

[0054] In the case of still aerial image IMG1, the display clarity of the still aerial image IMG1 may be changed depending on the distance between the user USR's hands or fingers and the still aerial image IMG1. FIG. 16 is an explanatory diagram illustrating changes in the display clarity of the still aerial image IMG1. As shown in FIG. 16, in the case of still aerial image IMG1, the image may become clearer as the user USR moves their hands or fingers closer to the still aerial image IMG1, and become less clear as the user USR moves their hands or fingers away from the still aerial image IMG1. In this example, the still aerial image IMG1 is displayed when the user USR's hands or fingers enter the monitoring area in the display area where the aerial image is displayed, and as the hands or fingers move closer to the still aerial image IMG1, the still aerial image IMG1 becomes more clear. The control of displaying the aerial image clearly or less clearly can be achieved by increasing or decreasing the brightness of the still aerial image IMG1 or increasing or decreasing the contrast of the still aerial image IMG1. The color of the still aerial image IMG1 may also be changed.

[0055] FIG. 17 is a flowchart showing the processing performed by display control unit 20 when displaying a follow-up aerial image.

[0056] (Step S201) The display control unit 20 determines, from the detection signal of the detection unit 30, whether or not an object is detected in the monitoring area in front of the display unit .

[0057] (Step S202) When an object is detected from the detection signal of the detection unit 30 (step S201: Yes), the display control unit 20 calculates the position of the detected object.

[0058] (Step S203) The display control unit 20 calculates the display position of the aerial image according to the position of the object calculated in step S202.

[0059] (Step S204) Display control unit 20 displays tracking aerial image IMG2 near the position of the object calculated in step S203.

[0060] (Step S205) The display control unit 20 detects whether or not there is an action on the following aerial image IMG2. For example, to move the following aerial image IMG2, the user USR performs an operation to move the following aerial image IMG2. When such an operation is performed, it is determined in step S205 that there is an action on the following aerial image IMG2.

[0061] (Step S206) If it is determined in step S205 that there is an action on the following aerial image IMG2 (step S205: Yes), the display control unit 20 moves the position of the following aerial image IMG2 in response to an operation by the user USR, and returns the process to step S202.

[0062] As described above, in this example, in the initial state, the following aerial image IMG2 is displayed near the hand or fingers of the user USR. Then, by repeatedly performing the processes from step S202 to step S206, the following aerial image IMG2 moves in accordance with the movement of the hand or fingers of the user USR.

[0063] FIG. 18 is an explanatory diagram of a specific example of a man-machine interface using the display device 1 according to the embodiment.

[0064] 18, when the user USR reaches out his / her hand in front of the display unit 10, a still aerial image IMG1 is displayed near the user USR's hand and fingers (state ST1). In this example, the still aerial image IMG1 is an operation panel, and buttons B1, B2, and B3 can be used to select "cow," "bird," or "pig."

[0065] The user USR performs an operation such as pressing a desired button on this still aerial image IMG1 to select the animal he or she wants to display (state ST2). In this example, the user USR presses button B2 to select "bird."

[0066] When the user USR presses button B2 on the still aerial image IMG1 with his / her finger to select "bird," the still aerial image IMG1 disappears and a following aerial image IMG2 corresponding to the "bird" appears (state ST3).

[0067] When the user USR moves his / her hands or fingers, the tracking aerial image IMG2 corresponding to the "bird" can move to follow the movements of the user USR's hands and fingers (state ST4).

[0068] The user USR can set various operations by hand movements (state ST5). Here, the user USR erases the following aerial image IMG2 by making a special hand movement, for example, by pressing the following aerial image IMG2 with three fingers.

[0069] FIG. 19 is a flowchart showing the process for realizing the above-mentioned man-machine interface.

[0070] (Step S501) The display control unit 20 determines, from the detection signal of the detection unit 30, whether or not an object is detected in the monitoring area in front of the display unit .

[0071] (Step S502) When an object is detected from the detection signal of detection unit 30 (step S501: Yes), display control unit 20 performs display processing of still aerial image IMG1. As shown in Fig. 15, the display processing of still aerial image IMG1 involves displaying still aerial image IMG1 near the detected position of the object, and after a predetermined time has elapsed, moving and fixing the position of still aerial image IMG1 to a specified position. This processing realizes the processing of state ST1 in Fig. 18.

[0072] (Step S503) The display control unit 20 determines whether or not there is an operation on the still aerial image IMG 1. If there is no operation on the still aerial image IMG 1 (step S503: No), the process proceeds to step S504.

[0073] (Step S504) The display control unit 20 determines whether a predetermined time has elapsed. If the predetermined time has not elapsed (step S504: No), the process returns to step S502. If the predetermined time has elapsed (step S504: Yes), the process proceeds to step S505.

[0074] (Step S505) The display control unit 20 erases the still aerial image IMG1, and returns the process to step S501.

[0075] (Step S506) If it is determined in step S503 that an operation has been performed on still aerial image IMG1 (step S503: Yes), display control unit 20 erases still aerial image IMG1, and proceeds to step S507.

[0076] (Step S507) The display control unit 20 performs processing to display the following aerial image IMG2 in response to the operation on the still aerial image IMG1. The display processing of the following aerial image IMG2 involves moving the position of the following aerial image IMG2 so as to follow the detected position of the object, as shown in Fig. 17. This processing realizes the processing from states ST3 to ST4 in Fig. 15.

[0077] (Step S508) The display control unit 20 determines whether a specific operation has been performed on the following aerial image IMG2. If a specific operation has not been performed on the following aerial image IMG2 (step S508: No), the display control unit 20 returns the process to step S507 and continues display processing of the following aerial image IMG2. If a specific operation has been performed on the following aerial image IMG2 (step S508: Yes), the display control unit 20 proceeds to step S509.

[0078] (Step S509) In response to the specific operation, display control unit 20 erases tracking aerial image IMG2 and returns the process to step S501. By this process, the process of state ST5 in FIG.

[0079] As described above, the display device 1 of the embodiment includes a display unit 10, a detection unit 30, and a display control unit 20. The display unit 10 displays an aerial image IMG (a still aerial image IMG1 and / or a tracking aerial image IMG2). The aerial image IMG is an image formed in mid-air from an image displayed on a display surface. The detection unit 30 detects whether an object (e.g., a hand or finger of the user USR, or a pointing device) is present in a monitoring area in the display area where the aerial image IMG is displayed. If an object is present in the monitoring area, the display control unit 20 displays the aerial image IMG in the monitoring area. As a result, the display device 1 of the embodiment displays the aerial image near the hand or finger of the user USR. Therefore, the user USR can view the aerial image without moving their line of sight.

[0080] Furthermore, the display device 1 of the embodiment controls the display position of the aerial image IMG according to the position of the object, thereby making it possible for the display device 1 of the embodiment to display the aerial image IMG at a position that is easy to operate.

[0081] Furthermore, in the display device 1 of the embodiment, the display position of the aerial image IMG is near the position where the object is located. This allows the display control unit 20 of the embodiment to display the aerial image IMG at a position that is easy for the user USR to operate. This allows the display device 1 of the embodiment to display the aerial image IMG at a position that allows the user USR to operate it without moving his or her line of sight.

[0082] Furthermore, in the display device 1 of the embodiment, the display position of the aerial image is a position near the position where the object is located and on the side of the display unit 10. As a result, in the display device 1 of the embodiment, when the user USR presses the aerial image IMG with his / her finger, the aerial image IMG can be displayed so that the finger of the user USR approaches the aerial image IMG.

[0083] Furthermore, in the display device 1 of the embodiment, when the aerial image IMG is a still aerial image IMG1, the display control unit 20 fixes the position of the aerial image IMG regardless of whether the position of the object has changed. As a result, when the display device 1 of the embodiment displays the aerial image IMG, if it is preferable that the display position of the aerial image IMG does not move in accordance with the content of the aerial image IMG or the movement of the finger of the user USR, the display position can be fixed.

[0084] Furthermore, in the display device 1 of the embodiment, the display control unit 20 controls the display position of the aerial image IMG according to the position of the object, and fixes the position of the aerial image IMG after a predetermined waiting time has elapsed since the aerial image IMG was displayed, regardless of whether the position of the object has changed. As a result, in the display device 1 of the embodiment, after the time has elapsed from the time the still aerial image IMG1 is displayed until the user USR's eyes can recognize the still aerial image IMG1, the display position of the still aerial image IMG1 can be fixed by the time the user USR performs a selection operation, allowing the user USR to select a desired option. Alternatively, the still aerial image IMG1 may be moved to a specified position after a predetermined waiting time has elapsed. This allows the still aerial image IMG1 to be displayed at the eye level of the user USR and then moved to a specified position, making it easier for the user USR to recognize the position of the still aerial image IMG1 and allowing the user USR to accept operations on the still aerial image IMG1 at a position where the still aerial image IMG1 does not interfere with other displays.

[0085] Furthermore, in the display device 1 of the embodiment, the display control unit 20 changes the display mode of the aerial image IMG so that the aerial image IMG becomes clearer as the object approaches the aerial image IMG. This allows the display device 1 of the embodiment to make the user USR easily recognize that an operation has been accepted.

[0086] Furthermore, in the display device 1 of the embodiment, the still aerial image IMG1 is an aerial image including a plurality of options. This allows the display position of the aerial image including a plurality of options to be fixed in the display device 1 of the embodiment, thereby preventing a situation in which the display position of the aerial image IMG moves in accordance with the movement of the user USR's finger, preventing the user USR from selecting a desired option.

[0087] Furthermore, in the display device 1 of the embodiment, when the aerial image IMG is the tracking aerial image IMG2, the display control unit 20 changes the position of the aerial image to track a change in the position of the object. As a result, when the display device 1 of the embodiment displays the aerial image IMG, if it is desirable for the display position of the aerial image IMG to move in accordance with the movement of the finger of the user USR in accordance with the content of the aerial image IMG, the display position can be made to track the movement.

[0088] In the above description, an operation switch is shown as the still aerial image IMG1, but the still aerial image IMG1 is not limited to an operation switch and may be an object that realizes functions in place of various pointing devices. Also, an animal image is shown as the following aerial image IMG2, but the following aerial image IMG2 is not limited to this.

[0089] Furthermore, the display device according to the embodiment of the present invention may be linked to other devices. For example, by combining it with a tactile presentation technology called haptics, a more diverse human interface that is closer to the real world can be realized.

[0090] All or part of the display device 1 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet or a telephone line, or devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may also be designed to implement some of the functions described above, or may be capable of implementing the functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0091] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0092] 10: Display unit, 20: Display control unit, 30: Detection unit

Claims

1. a display unit that displays an aerial image by projecting an image displayed on a display surface into the air; a detection unit that detects whether or not an object manipulating the aerial image is present in a monitoring area in a display area that displays the aerial image and is provided in front of the display surface; and a display control unit that, when the detection unit detects that the object is present in the monitoring area, displays the aerial image at a position adjacent to the detection position of the object in a depth direction perpendicular to the display surface, in accordance with the detection position of the object in the monitoring area; A display device comprising:

2. The display position of the aerial image is near the position where the object is present and is located closer to the display unit in the depth direction than the detection position of the object. The display device according to claim 1 .

3. When the aerial image is a still aerial image including a plurality of options, the display control unit fixes the position of the aerial image regardless of whether the position of the object has changed. The display device according to claim 1 .

4. The display control unit displays the aerial image at an initial position near the detected position of the object, and after a predetermined waiting time has elapsed corresponding to a time from when the aerial image is displayed at the initial position until the user can view the aerial image displayed at the initial position, moves the display position of the aerial image to a designated position different from the initial position and at which the user can operate a plurality of options included in the aerial image, and fixes the display position of the aerial image at the designated position regardless of whether the position of the object has changed. The display device according to claim 3 .

5. The display control unit changes at least one of brightness, contrast, and color of the aerial image as the object approaches the aerial image, thereby changing the display mode of the aerial image so that the aerial image becomes clearer. The display device according to claim 3 or 4.

6. When the aerial image is a tracking aerial image that tracks a user's operation, the display control unit changes the display position of the aerial image in accordance with a change in the position of the object. The display device according to claim 2 .

7. The object is a user's hand or finger, The display control unit erases the displayed aerial image in response to a specific movement of the user's hand or finger. The display device according to claim 6.

8. A display method performed by a display device that is a computer, a display unit displays a mid-air image by projecting the image displayed on the display surface into mid-air; a detection unit detects whether or not an object manipulating the aerial image is present in a monitoring area in a display area where the aerial image is displayed and which is provided in front of the display surface; a display control unit, when the detection unit detects that the object is present in the monitoring area, causing the aerial image to be displayed at a position close to the detection position of the object in a depth direction perpendicular to the display surface, in accordance with the detection position of the object in the monitoring area. Display method.

9. When the aerial image is a still aerial image including a plurality of options, the display control unit displays the aerial image at an initial position close to the detected position of the object, and after a predetermined waiting time corresponding to the time from when the aerial image is displayed at the initial position until the user can view the aerial image displayed at the initial position has elapsed, moves the display position of the aerial image to a designated position different from the initial position and at which the user can operate the plurality of options included in the aerial image, and fixes the display position of the aerial image at the designated position regardless of whether the position of the object has changed. The display method according to claim 8.

10. The display control unit changes the display mode of the aerial image so that the aerial image becomes clearer by changing at least one of the brightness, contrast, and color of the aerial image as the object approaches the aerial image. The display method according to claim 9.

11. When the aerial image is a tracking aerial image that tracks a user's operation, the display control unit changes the display position of the aerial image in accordance with a change in the position of the object. The display method according to claim 10.

12. On the display device which is a computer, The image displayed on the display surface is projected in mid-air to display a mid-air image. detecting whether or not an object manipulating the aerial image is present in a monitoring area in a display area for displaying the aerial image, the monitoring area being provided in front of the display surface; when the presence of the object in the monitoring area is detected, the aerial image is displayed at a position close to the detected position of the object in a depth direction perpendicular to the display surface, in accordance with the detected position of the object in the monitoring area; program.

Images