Video display system and video control method

Abstract

An image display system (1) comprises: an aerial image display unit (10) that displays an aerial image (71, 72), which is a real image, in a predetermined displayable region (70) in a three-dimensional space; a sensor unit (20) that detects a user (81, 82) in the vicinity of the displayable region (70) and outputs detection information; and a control unit (30) that determines a gesture including at least one of the motion and attitude of a physical part of the user (81, 82) on the basis of the detection information, and that, if the gesture is a predetermined position designation gesture for designating a display position of the aerial image (71, 72) in the displayable region (70), controls the aerial image display unit (10) so that the aerial image (71, 72) is displayed at the designated display position that is the display position designated by the position designation gesture.

Description

[Technical Field]

[0001] The present disclosure relates to a video display system, a video control method, and a video control program. [Background technology]

[0002] Patent document 1 proposes an aerial video interaction device that includes an imaging optical system that images a real image of an object to be projected, which is placed on one side of a ray bending surface, as an aerial image on the other side of the ray bending surface, and an object identification means that identifies the position, or the position and shape, of an object close to the real image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5212991 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned device has a problem in that the imaging position of the aerial image may not be suitable for the observer (i.e., the user of the device).

[0005] The present disclosure aims to display an aerial image at a display position appropriate for the user. [Means for solving the problem]

[0006] of the present disclosure In one aspect The video display system includes an aerial video display unit that displays a real aerial video within a predetermined displayable area in a three-dimensional space, a sensor unit that detects a user near the displayable area and outputs detection information, and a display unit that displays a real aerial video of the user based on the detection information. The position of the eyes is identified, a visual range is calculated, and a gesture of closing the eyes of the user is determined and the visual range is displayed at a designated display position. and a control unit that controls the aerial image display unit so that the aerial image is displayed.

[0007] Other aspects of the present disclosure Related to the aspect The video display system includes an aerial video display unit that displays a real aerial video within a predetermined displayable area in three-dimensional space; a sensor unit that detects a user near the displayable area and outputs detection information; and a control unit that determines a gesture including at least one of a movement and a posture of a body part of the user based on the detection information, and, when the gesture is a predetermined position designation gesture that designates a display position of the aerial video within the displayable area, controls the aerial video display unit to display the aerial video at a designated display position that is the display position designated by the position designation gesture. a camera that captures an image of a target and outputs image data of the target, and when the gesture is determined to be an object designation gesture that designates the target to be displayed at the designated display position, the control unit controls the aerial image display unit so that an image of the target designated by the object designation gesture is displayed at the designated display position as the aerial image. It is characterized by: Another aspect of the video display system of the present disclosure is characterized by having an aerial video display unit that displays a real aerial video within a predetermined displayable area in three-dimensional space; a sensor unit that detects a user in the vicinity of the displayable area and outputs detection information, photographs the target, and outputs video data; and a control unit that determines a gesture including at least one of a movement and posture of a body part of the user based on the detection information, and when the gesture is determined to be a target designation gesture that designates the target, controls the aerial video display unit so that an image of the target designated by the target designation gesture is displayed as the aerial video. [Effects of the Invention]

[0008] According to the present disclosure, an aerial image can be displayed at a display position appropriate for the user. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram schematically illustrating a configuration of a video display system according to a first embodiment. [Figure 2] 1 is a diagram showing an example of a usage state of the video display system according to the first embodiment. [Figure 3] 10A and 10B are diagrams showing examples of designated display positions, which are display positions of aerial images designated by a user. [Figure 4] 2 is a block diagram schematically showing the configuration of a control unit of the video display system according to the first embodiment. FIG. [Figure 5] 1 is a diagram illustrating an example of a hardware configuration of a video display system according to a first embodiment. [Figure 6] 5 is a flowchart showing the operation of a control unit of the video display system according to the first embodiment. [Figure 7] FIG. 10 is a diagram schematically illustrating a configuration of a video display system according to a modification of the first embodiment. [Figure 8]10 is a block diagram schematically showing the configuration of a control unit of a video display system according to a modified example of the first embodiment. FIG. [Figure 9] 10 is a flowchart showing the operation of a control unit of a video display system according to a modification of the first embodiment. [Figure 10] FIG. 10 is a diagram schematically illustrating a configuration of a video display system according to a second embodiment. [Figure 11] FIG. 10 is a block diagram schematically illustrating the configuration of a control unit of a video display system according to a second embodiment. [Figure 12] 10 is a flowchart showing the operation of a control unit of the video display system according to the second embodiment. [Figure 13] FIG. 10 is a diagram schematically illustrating a configuration of a video display system according to a modified example of the second embodiment. [Figure 14] 10 is a flowchart showing the operation of a control unit of a video display system according to a modification of the second embodiment. [Figure 15] 10(A) and 10(B) are diagrams showing the configuration and operation of a video display system according to a third embodiment. [Figure 16] 10 is a flowchart showing the operation of a control unit of the video display system according to the third embodiment. [Figure 17] 10(A) and 10(B) are diagrams showing examples of size designation gestures in a video display system according to a fourth embodiment. [Figure 18] 13(A) and 13(B) are diagrams showing examples of size designation gestures in a video display system according to a modification of the fourth embodiment. [Figure 19] 10 is a flowchart showing the operation of a control unit of the video display system according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a video display system, a video control method, and a video control program according to embodiments will be described with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified.

[0011] First Embodiment Fig. 1 is a diagram schematically showing the configuration of video display system 1 according to embodiment 1. As shown in Fig. 1, video display system 1 has aerial image display unit 10 as an image display device that displays (i.e., projects) real images as aerial images 71, 72, sensor unit 20, and control unit 30 as a control device that controls the operation of aerial image display unit 10.

[0012] Aerial image display unit 10 displays one or more aerial images (e.g., aerial images 71, 72, etc.) within a predetermined displayable area 70 in three-dimensional space. Aerial image display unit 10 has display unit 11 as a display device that displays images, and real image optical system 12 that focuses the images displayed on display unit 11 as aerial images 71, 72 within displayable area 70. Display unit 11 is, for example, a liquid crystal display or an LED (Light Emitting Diode) display, and displays images based on input image data (e.g., video content).

[0013] The real image optical system 12 has a beam splitter 13, which is an optical element that reflects and transmits incident light, and a retroreflective sheet 14, which is a retroreflective element that retroreflects the light reflected by the beam splitter 13. The light retroreflected by the retroreflective sheet 14 travels in the opposite direction to the traveling direction of the light that entered the retroreflective sheet 14, passes through the beam splitter 13, and forms an image within the displayable area 70.

[0014] The beam splitter 13 may be any optical element capable of reflecting and transmitting light to separate it, such as a half mirror, a glass plate, etc. The retroreflective sheet 14 may be, for example, an encapsulated lens type reflective sheet formed by embedding the lower halves of numerous transparent minute glass beads (i.e., microbeads) in a reflective film, or a microprism type reflective sheet formed by embedding numerous transparent microprisms in a reflective film.

[0015] The sensor unit 20 has, for example, a camera as an imaging device or a detector that detects objects. Specific examples of the camera of the sensor unit 20 include an infrared camera, an infrared line sensor, a stereo camera, and a visible light camera. Specific examples of the detector include a distance measurement sensor such as LiDAR that uses light, millimeter-wave radar that uses radio waves, or sonar that uses sound. The detector is capable of detecting the distance to an object and the shape of the object. The sensor unit 20 detects one or more users (e.g., users 81 and 82) in the vicinity of the displayable area 70 and outputs detection information (e.g., a video signal, a detection signal).

[0016] Control unit 30 determines a gesture including at least one of a movement and a posture of a body part of users 81, 82 based on the detection information, and if the gesture is a predetermined position designation gesture that designates the display position of aerial images 71, 72 within displayable area 70, controls aerial image display unit 10 so that aerial images 71, 72 are displayed at designated display positions that are display positions designated by the position designation gesture. Specifically, control unit 30 controls control unit 30 of aerial image display unit 10 so that aerial images 71, 72 are displayed at the display positions designated by the position designation gesture. Control unit 30 sets video data to be displayed on display unit 11 (for example, selects video content from a database) in order to display aerial images 71, 72 at the designated display positions within displayable area 70.

[0017] When control unit 30 determines that the gestures include a first gesture by user 81 as a first user and a second gesture by user 82 as a second user, and that both the first gesture and the second gesture are position designation gestures that designate the display positions of aerial images 71, 72 within displayable area 70, control unit 30 controls aerial image display unit 10 to display aerial images 71, 72 at the first display position designated by the first gesture and the second display position designated by the second gesture. For example, control unit 30 sets image data to be displayed on display unit 11 in order to display aerial images 71, 72 at the designated display positions within displayable area 70.

[0018] The control unit 30 may detect the skeletons of the users 81 and 82 based on the detection information, and determine the display position specified by the position designation gesture based on whether the skeletons are in a predetermined state or not.

[0019] Furthermore, control unit 30 may determine that a state in which users 81 and 82 extend their first fingers in a first direction and their second fingers in a second direction intersecting the first direction is a position designation gesture, and perform control to display aerial images 71 and 72 at designated display positions designated by the first and second fingers. Here, the first and second fingers are the index finger and thumb, and the crossing angle is approximately 90 degrees.

[0020] Furthermore, when the facial expression (including changes in expression) of users 81, 82 matches a predetermined expression, control unit 30 may determine the facial expression to be a position designation gesture, and perform control to display aerial images 71, 72 at a designated display position designated by the expression. Facial expressions that can be used as gestures include, for example, opening and closing the mouth, the shape of the mouth, blinking, winking, facial expression, and the direction of gaze of the eyes.

[0021] FIG. 2 is a diagram showing an example of how the video display system 1 is used. FIGS. 3A and 3B are diagrams showing examples of designated display positions, which are display positions of the aerial image 71 designated by a user 81. In FIG. 2, users 81 and 82 extend their index fingers in a vertical direction (height direction in FIG. 2) as a first direction and their thumbs in a horizontal direction (horizontal direction in FIG. 2) as a second direction intersecting the first direction, which is regarded as a position designation gesture. In this example, the aerial images 71 and 72 are displayed in rectangular areas having vertical sides roughly parallel to the index fingers and horizontal sides roughly parallel to the thumbs. Therefore, by changing the position of the position designation gesture made with the index fingers and thumbs, the user 81 can move the aerial images 71 and 72 as shown in FIGS. 3A and 3B.

[0022] Fig. 4 is a block diagram schematically showing the configuration of the control unit 30 of the video display system 1 according to Embodiment 1. As shown in Fig. 4, the control unit 30 includes a three-dimensional position detection unit 31, a visually visible area determination unit 32, a video data control unit 33, a gesture determination unit 34, and a display control unit 35.

[0023] Three-dimensional position detection unit 31 detects the positions and movements of users 81, 82 who are viewers of aerial images 71, 72. For example, three-dimensional position detection unit 31 identifies the position and viewpoint position (i.e., eye position) of the user near aerial image display unit 10.

[0024] Visible area determination unit 32 determines the visual range, which is the area that users 81, 82 can see (i.e., can observe). Visible area determination unit 32 calculates the visual range based on, for example, the positional relationship between aerial image display unit 10 and the eyes of users 81, 82. Specifically, visual area determination unit 32 calculates the visual range of users 81, 82 from the placement information of aerial image display unit 10 and the three-dimensional position information from three-dimensional position detection unit 31.

[0025] Video data control unit 33 controls the video data for displaying aerial images 71, 72 (i.e., selects video content from a database) according to the visual range of aerial images 71, 72. For example, the video data is controlled so that it is displayed only in a specific area visible to users 81, 82 from an area that can be displayed by one real image optical system 12.

[0026] Gesture determination unit 34 determines gestures (i.e., commands issued by the users) that are spatial operations performed by users 81 and 82. Gesture determination unit 34 detects specific gestures performed by users 81 and 82 after users 81 and 82 view aerial images 71 and 72, and determines the content that the users wish to operate by the gesture.

[0027] Display control unit 35 controls the video data in accordance with gestures made by users 81 and 82. The control content includes transition to a screen in accordance with the gesture or selection of video content. Display control unit 35 outputs the video data (i.e., video content) to aerial video display unit 10.

[0028] FIG. 5 is a diagram illustrating an example of a hardware configuration of a video display system 1 according to the first embodiment. The video display system 1 includes a display unit 11, a sensor unit 20, and a control unit 30. The control unit 30 is, for example, a computer that can execute the display control program according to the first embodiment. The control unit 30 includes a processor 101, a memory 102, and a storage device 103. The processor 101 is, for example, a central processing unit (CPU). The memory 102 is, for example, a volatile semiconductor memory such as a random access memory (RAM). The storage device 103 is a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The control unit 30 may also include a communication unit that communicates with other devices via a network, and an operation unit such as a keyboard that serves as an input interface.

[0029] Each function of the control unit 30 is realized by a processing circuit. The processing circuit may be dedicated hardware, or may be a processor 101 that executes a program stored in a memory 102. The processor 101 may be any of a processing device, an arithmetic device, a microprocessor, a microcomputer, and a DSP (Digital Signal Processor).

[0030] When the processing circuitry is dedicated hardware, the processing circuitry may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or any combination thereof.

[0031] When the processing circuit is the processor 101, the display control program according to the first embodiment is realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 102. The processor 101 can realize the functions of each unit shown in FIG. 4 by reading and executing the display control program stored in the memory 102. Note that the control unit 30 may be partially realized by dedicated hardware and partially realized by software or firmware. In this way, the processing circuit can realize the functions of each functional block shown in FIG. 1 by hardware, software, firmware, or any combination thereof.

[0032] 6 is a flowchart showing the operation of control unit 30 of video display system 1 according to Embodiment 1. First, control unit 30 receives detection information from sensor unit 20 (step S11), detects the three-dimensional positions of body parts of users 81 and 82 (step S12), and determines the visually accessible areas of users 81 and 82 (step S13). Control unit 30 sets video data (e.g., video content) (step S14), determines a gesture based on the detection information (e.g., video data) received from sensor unit 20 (step S15), and causes aerial image display unit 10 to display aerial images 71 and 72 at designated display positions, which are display positions indicated by the gesture (step S16).

[0033] As described above, according to the first embodiment, aerial images 71, 72 can be displayed at display positions appropriate for users 81, 82 according to gestures of users 81, 82. For example, the display positions of aerial images 71, 72 can be changed to appropriate positions depending on whether user 81 is an adult, user 82 is a child, or a user who uses a wheelchair (not shown).

[0034] Fig. 7 is a diagram schematically illustrating the configuration of video display system 1a according to a variation of embodiment 1. In Fig. 7, components that are the same as or correspond to those shown in Fig. 1 are assigned the same reference numerals as those shown in Fig. 1. Video display system 1a differs from video display system 1 in Fig. 1, which determines the display positions of aerial images 71, 72 based on the gestures of users 81, 82, in that video display system 1a determines the display positions of aerial images 71, 72 based on the positions and postures of users 81, 82.

[0035] 7, video display system 1a includes aerial video display unit 10, sensor unit 20 that detects users 81, 82 in the vicinity of displayable area 70 and outputs detection information, and control unit 30a as a display control device. Control unit 30a determines the three-dimensional positions of the body parts of users 81, 82 based on the detection information output from sensor unit 20, determines the display positions of aerial videos 71, 72 based on the determination results, and controls aerial video display unit 10 so that aerial videos 71, 72 are displayed at the display positions.

[0036] Control unit 30a determines the three-dimensional positions of the eyes or faces of users 81, 82, determines the display positions of aerial images 71, 72 based on the position and line of sight of the eyes, or the position and orientation of the face, and controls aerial image display unit 10 so that aerial images 71, 72 are displayed at the display positions. That is, in image display system 1a, when users 81, 82 simply approach the vicinity of aerial image display unit 10, aerial images 71, 72 are displayed, for example, near the position of the eyes of users 81, 82. The three-dimensional positions of the eyes or faces of users 81, 82 are examples of the position and orientation of users 81, 82, and the position and orientation of users 81, 82 may also be determined based on the state of body parts other than the eyes or face.

[0037] FIG. 8 is a block diagram schematically illustrating the configuration of control unit 30a of video display system 1a according to a variation of Embodiment 1. In FIG. 8, components that are the same as or correspond to those shown in FIG. 4 are assigned the same reference numerals as those shown in FIG. 4. Video display system 1a differs from video display system 1 of FIG. 1, which determines the display positions of aerial images 71 and 72 based on gestures, which are movements of users 81 and 82, in that video display system 1a determines the display positions of aerial images 71 and 72 based on the positions and postures of users 81 and 82. Control unit 30a differs from control unit 30 in that it includes user position and posture determination unit 34a but does not include the gesture determination unit of FIG. 4. User position and posture determination unit 34a of control unit 30a determines the three-dimensional positions of the eyes or faces of users 81 and 82, determines the display positions of aerial images 71 and 72 based on the positions and orientations of the eyes or faces, and controls aerial image display unit 10 so that aerial images 71 and 72 are displayed at the display positions. The control unit 30a may include both the user position / posture determination unit 34a and the gesture determination unit 34.

[0038] 9 is a flowchart showing the operation of control unit 30a of video display system 1a according to a variation of Embodiment 1. In FIG. 9, steps that are the same as or correspond to steps shown in FIG. 6 are assigned the same reference numerals as those shown in FIG. 6. In video display system 1a, control unit 30a receives detection information from sensor unit 20 (step S11), detects the three-dimensional positions of body parts of users 81 and 82 (step S12), and determines the visually accessible areas of users 81 and 82 (step S13). Control unit 30a sets video data (e.g., video content) (step S14a), determines the positions and orientations of users 81 and 82 based on the detection information (step S15a), and displays aerial images 71 and 72 in areas corresponding to the positions and orientations of users 81 and 82 (step S16a).

[0039] As described above, video display system 1a according to the variation of embodiment 1 can display aerial images 71 and 72 at display positions appropriate for users 81 and 82 depending on the state of the body parts of users 81 and 82 (for example, eye position, line of sight direction, and facial orientation). In other words, video display system 1a can display aerial images 71 and 72 at positions appropriate for each of users 81 and 82, tailored to each individual.

[0040] Furthermore, the video display system 1a according to the modified example of the first embodiment can extract people at specific positions from a crowd and display specific screens of aerial images 71 and 72 at positions suitable for each extracted person.

[0041] In video display system 1a according to a modified example of embodiment 1, control unit 30a may, after displaying aerial images 71, 72 at display positions determined based on the results of the determination, determine whether aerial images 71, 72 are making a gesture including at least one of the movement and posture of a body part of users 81, 82 based on the detection information, and if the gesture is a predetermined position designation gesture that newly designates the display position of aerial images 71, 72 within displayable area 70, control aerial image display unit 10 so that aerial images 71, 72 are displayed at the display position designated by the position designation gesture.

[0042] Furthermore, the image display system 1 or 1a can be combined with any of the configurations of the second, third, and fourth embodiments described below.

[0043] Second Embodiment FIG. 10 is a diagram schematically illustrating a configuration of video display system 2 according to the second embodiment. In video display system 2 according to the second embodiment, sensor unit 20 has a camera that captures an image of user 81 and target 90 to be displayed as aerial image 71 and outputs the image data. Target 90 may be something other than an object, and may be, for example, an image (which may include a moving image) displayed on a monitor or character string information displayed on a monitor. In other words, target 90 is not limited to an actual object. When control unit 40 determines that the gesture is an object designating gesture, which is a predetermined gesture that designates target 90 to be displayed at the display position of aerial image 71, control unit 40 controls aerial image display unit 10 so that the image of target 90 designated by the object designating gesture is displayed as aerial image 71 at the display position.

[0044] As shown in Fig. 10, the target designation gesture is a gesture in which user 81 points at target 90 with his / her finger. In Fig. 10, by simply pointing with the index finger at target 90 that user 81 wants to display in front of him / her as aerial image 71, aerial image 71 of target 90 can be displayed in front of user 81 (for example, at the display position in embodiment 1). Based on the detection information, control unit 40 determines that an object that intersects with vector F in the direction pointed by user 81's finger is target 90 that should be displayed. Target 90 may be an object that exists in reality, an object displayed on a large screen, or the like.

[0045] Fig. 11 is a block diagram showing a schematic configuration of the control unit 40 of the video display system 2 according to the second embodiment. In Fig. 11, components that are the same as or correspond to those shown in Fig. 4 are assigned the same reference numerals as those in Fig. 4. In the control unit 40, the gesture determination unit 34b determines the target 90 that the user 81 is pointing at with his / her finger as a body part, and the video data control unit 33 obtains the video data of the target 90 from a database of previously captured videos or from the video data captured by the sensor unit 20, processes the video data into a video format that is to be displayed at the display position, and provides the video data to the display control unit 35.

[0046] Fig. 12 is a flowchart showing the operation of control unit 40 of video display system 2 according to embodiment 2. In Fig. 12, steps S21 to S23 are the same as steps S11 to S13 in Fig. 6. Control unit 40 of video display system 2 determines target 90 at which user 81 is pointing with his / her finger as a body part (step S24), acquires video data of target 90 from a database of pre-captured videos or from video data captured by sensor unit 20 (step S25), determines the position and orientation of user 81 (step S26), and displays aerial video 71 of target 90 in an area corresponding to the position and orientation of user 81 (step S27).

[0047] Fig. 13 is a diagram schematically showing the configuration of video display system 2a according to a modification of Embodiment 2. In Fig. 13, components that are the same as or correspond to those shown in Fig. 11 are assigned the same reference numerals as those in Fig. 11. Video display system 2a further has camera 21 that captures an image of target 90 to be displayed as aerial image 71 and outputs video data of target 90. When control unit 40 determines that the gesture is an object designating gesture that designates target 90 to be displayed at the display position of aerial image 71, control unit 40 controls aerial image display unit 10 so that the image of target 90 designated by the object designating gesture is displayed as aerial image 71 at the display position.

[0048] Figure 14 is a flowchart showing the operation of a control unit of video display system 2a according to a variation of embodiment 2. In Figure 14, steps that are the same as or correspond to steps shown in Figure 12 are given the same reference numerals as in Figure 12. Video display system 2a differs from video display system 2 in that video captured by camera 21 is displayed as aerial image 71 at a display position.

[0049] According to the second embodiment, it is possible to display aerial image 71 of target 90 pointed at by a finger at a display position appropriate for user 81.

[0050] In addition, it is possible to display an object 90 that is far away, an object 90 at a high position that is difficult to reach by hand, an object 90 within an image displayed on a large-screen monitor, etc., near the user 81 as an aerial image 71.

[0051] Moreover, it is possible to combine the configurations of any of the first, third, and fourth embodiments with the video display system 2 or 2a. For example, the control unit 40 shown in FIGS. 10 and 13 discriminates a gesture including at least either the movement or the posture of the body part of the user 81 based on the video information captured by the camera 20, and when the gesture is a predetermined position-specifying gesture (for example, as shown in FIG. 2, a gesture of forming an L shape or a horizontally reversed L shape with the thumb and index finger) that specifies the display position of the aerial image 71 within the displayable area 70, the aerial image display unit 10 may be controlled to perform an operation such that the aerial image 71 is displayed at the specified display position specified by the position-specifying gesture. The specification of the display position based on the position-specifying gesture may be performed either before or after the target-specifying gesture. Also, the position-specifying gesture may be another gesture as described in the first embodiment.

[0052] 《Embodiment 3》 FIGS. 15(A) and (B) are diagrams showing the configuration and operation of the video display system 3 according to Embodiment 3. FIG. 15(A) shows a state where the aerial image 71 is displayed at a display position separated from the aerial image display unit 10a by a distance L1, and FIG. 15(B) shows a state where the aerial image 71 is displayed at a display position separated from the aerial image display unit 10a by a distance L2 (L2 < L1).

[0053] Video display system 3 includes a sensor unit 20, an aerial image display unit 10a, and a control unit 30. Aerial image display unit 10a includes a first display unit 11a that displays a first image, a second display unit 11b that displays a second image, and a real image optical system 12 that displays the first image or the second image as aerial image 71, which is a real image. The distance from first display unit 11a to real image optical system 12 is longer than the distance from second display unit 11b to real image optical system 12. Furthermore, control unit 30 determines a gesture including at least one of a movement and a posture of a body part of user 81 based on the detection information, and, when the gesture is a predetermined depth designation gesture that designates a position in the depth direction of displayable area 70, controls aerial image display unit 10a so that aerial image 71 is displayed within displayable area 70 designated by the depth designation gesture.

[0054] The depth designation gesture is an action of designating a position in the depth direction of the displayable area 70 with a body part of the user 81. The depth designation gesture may be any gesture that can designate a position in the depth direction. For example, the depth designation gesture may be included in the position designation gesture described in the first and second embodiments. That is, the depth designation gesture may be a gesture performed simultaneously with the position designation gesture. For example, as shown in FIG. 2, a gesture of forming an L-shape or a mirrored L-shape with the thumb and index finger may designate a position in the depth direction in addition to a position in the height direction and a position in the left-right direction of the displayed image. Furthermore, the depth designation gesture may be a gesture separate from the position designation gesture. In this case, the depth designation gesture is performed before or after the position designation gesture.

[0055] Fig. 16 is a flowchart showing the operation of the control unit of video display system 3 according to embodiment 3. In Fig. 16, steps S31 to S34 are the same as steps S11 to S14 shown in Fig. 12. When a gesture specifies a position in the depth direction of displayable area 70 (step S35), video display system 3 moves displayable area 70 to the position specified by the gesture, and displays aerial images within displayable area 70 (step S36).

[0056] As described above, according to video display system 3 according to the third embodiment, aerial video 71 can be displayed at a display position appropriate for user 81.

[0057] Furthermore, the video display system 3 can be combined with any of the configurations of the first, second, and fourth embodiments.

[0058] 15(A) and 15(B) depict first display unit 11a and second display unit 11b as two display devices, this configuration can be replaced with a single display device and a movement mechanism (not shown) that moves this display device. That is, the movement mechanism can move the display device closer to or farther away from real image optical system 12. In this case, the display device changes its position in the depth direction of the aerial image by moving to either a first position shown as "first display unit 11a" in FIG. 15(A) or a second position shown as "second display unit 11b" in FIG. 15(B).

[0059] 15(A) and (B) can be replaced with a single display device and a movement mechanism (not shown) that moves real image optical system 12 (or a portion of the optical system). That is, the movement mechanism can move image optical system 12 (or a portion of the optical system) closer to or farther away from the display device. In this case, the display device changes its position in the depth direction of the aerial image by moving to either a first position shown as "first display unit 11a" in FIG. 15(A) or a second position shown as "second display unit 11b" in FIG. 15(B).

[0060] Fourth Embodiment 17(A) and (B) are diagrams showing examples of size designation gestures in the video display system according to embodiment 4. Fig. 17(A) shows an example of a size designation gesture in which user 81 narrows the gap between two fingers to reduce the display area of ​​aerial image 71, and Fig. 17(B) shows an example of a size designation gesture in which user 81 widens the gap between two fingers to increase the display area of ​​aerial image 71.

[0061] The configuration of the video display system according to the fourth embodiment is the same as the configuration of the video display system according to any one of the first to third embodiments. When the gesture is a predetermined size designation gesture for enlarging or reducing the size of the floating image 71 within the displayable area 70, the control unit of the video display system according to the fourth embodiment controls the floating image display unit 10 so that the floating image 71 is displayed at the size designated by the size designation gesture. Note that while the user 81 uses the fingers of each hand as the two fingers, a similar operation may also be performed with two fingers on one hand. Furthermore, a similar operation may also be performed with body parts other than fingers (for example, both arms, both legs, an arm and a finger, etc.).

[0062] 18(A) and (B) are diagrams showing examples of size designation gestures in a video display system according to a variation of embodiment 4. Fig. 18(A) shows an example of a size designation gesture in which user 81 brings a finger, as an example of a body part, closer to the display area to reduce the display area of ​​aerial image 71, and Fig. 18(B) shows an example of a size designation gesture in which user 81 moves a finger, as an example of a body part, away from the display area to increase the display area of ​​aerial image 71.

[0063] The configuration of the video display system according to the variation of Embodiment 4 is the same as the configuration of the video display system according to any one of Embodiments 1 to 3. When the gesture is a predetermined size designation gesture that enlarges or reduces the size of floating image 71 within displayable area 70, the control unit of Embodiment 4 controls floating image display unit 10 so that floating image 71 is displayed at the size designated by the size designation gesture. Note that, although user 81 makes the gesture using the fingers of their hand, similar operations may also be performed with body parts other than fingers (for example, both arms, both legs, an arm and fingers, etc.).

[0064] Fig. 19 is a flowchart showing the operation of a control unit of the video display system according to embodiment 4. In Fig. 19, steps S41 to S45 are the same as steps S11 to S15 shown in Fig. 6. After displaying aerial image 71, the video display system according to embodiment 4 can enlarge or reduce aerial image 71 using a size designation gesture (step S46).

[0065] As described above, according to the fourth embodiment, aerial image 71 of target 90 pointed at with a finger can be displayed at a desired size at a display position appropriate for user 81.

[0066] Furthermore, the video display system according to the fourth embodiment can be combined with any of the configurations of the first, second, and third embodiments.

[0067] Other Modifications The video display systems according to the first to fourth embodiments can be applied to non-contact operation panels used by many people, digital signage having a function to switch display content by user operation, and the like.

[0068] Moreover, the video display systems according to the first to fourth embodiments can also be used as display devices for automobile equipment. [Explanation of symbols]

[0069] 1, 1a, 2, 2a, 3 Image display system, 10, 10a Aerial image display unit, 11 Display unit, 11a First display unit, 11b Second display unit, 12 Real image optical system, 13 Beam splitter, 14 Retroreflective sheet (retroreflective material), 20 Sensor unit, 21 Camera, 30, 30a, 40 Control unit, 31 Three-dimensional position detection unit, 32 Visible area determination unit, 33 Image data control unit, 34, 34b Gesture determination unit, 34a User position / posture determination unit, 35 Display control unit, 70 Displayable area, 71, 72 Aerial image, 81, 82 User, 81a Hand, 90 Target

Claims

1. an aerial image display unit that displays aerial images, which are real images, within a predetermined displayable area in three-dimensional space; a sensor unit that detects a user in the vicinity of the displayable area and outputs detection information; a control unit that identifies eye positions of the user based on the detection information to calculate a visual range, and determines a gesture of the user closing the eyes to control the aerial image display unit so that the aerial image is displayed at a designated display position that is determined in accordance with the visual range; A video display system comprising:

2. an aerial image display unit that displays aerial images, which are real images, within a predetermined displayable area in three-dimensional space; a sensor unit that detects a user in the vicinity of the displayable area and outputs detection information; a control unit that determines a gesture including at least one of a movement and a posture of a body part of the user based on the detection information, and, when the gesture is a predetermined position designation gesture that designates a display position of the aerial image within the displayable area, controls the aerial image display unit so that the aerial image is displayed at a designated display position that is the display position designated by the position designation gesture; a camera that photographs an object and outputs video data of the object; and When the gesture is determined to be a target designation gesture for designating the target to be displayed at the designated display position, the control unit controls the aerial image display unit so that an image of the target designated by the target designation gesture is displayed as the aerial image at the designated display position. A video display system comprising:

3. The control unit changes the display position in the displayable area in response to the position designation gesture.

3. The video display system according to claim 2.

4. The control unit detects a skeletal state of the user based on the detection information, and determines the display position designated by the position designation gesture based on the skeletal state.

3. The video display system according to claim 2.

5. the sensor unit has at least a camera that captures an image of an object and outputs video data; The control unit determining a gesture including at least one of a movement and a posture of a body part of the user based on the detection information or the video data obtained by capturing an image of the user with the camera; When the gesture is determined to be a target designation gesture for designating the target to be displayed at the display position, the aerial image display unit is controlled so that an image of the target designated by the target designation gesture is displayed as the aerial image at the display position.

2. The video display system according to claim 1.

6. When the gesture is a predetermined position designation gesture that designates the display position of the aerial image within the displayable area, the control unit controls the aerial image display unit so that the aerial image is displayed at a designated display position that is the display position designated by the position designation gesture.

6. The video display system according to claim 5.

7. the sensor unit has a camera that captures an image of the user and an object and outputs video data; When the gesture is determined to be a target designation gesture for designating the target to be displayed at the designated display position, the control unit controls the aerial image display unit so that an image of the target designated by the target designation gesture is displayed as the aerial image at the designated display position.

2. The video display system according to claim 1.

8. When the control unit determines that the gesture is a target designation gesture for designating a target to be displayed at the designated display position, the control unit controls the aerial image display unit to use image information of the target from a database that stores image data of the target, the image having been captured in advance, so that the image of the target designated by the target designation gesture is displayed as the aerial image at the designated display position.

2. The video display system according to claim 1.

9. The target designation gesture is an action in which the user points a finger at the target.

3. The video display system according to claim 2.

10. the aerial image display unit includes a first display unit that displays a first image, a second display unit that displays a second image, and a real image optical system that displays the first image or the second image as the aerial image that is the real image, a distance from the first display unit to the real image optical system is longer than a distance from the second display unit to the real image optical system; The control unit determines a gesture including at least one of a movement and a posture of the body part of the user based on the detection information, and when the gesture is a predetermined depth designation gesture designating a position in a depth direction of the displayable area, controls the aerial image display unit so that the aerial image is displayed within the displayable area whose position in the depth direction is designated by the depth designation gesture.

5. The video display system according to claim 2 or 4.

11. When the gesture is a predetermined size designation gesture for enlarging or reducing the size of the aerial image within the displayable area, the control unit controls the aerial image display unit so that the aerial image is displayed at the size designated by the size designation gesture.

3. The video display system according to claim 1 or 2.

12. The size specification gesture is an action in which the user spreads or narrows the space between two fingers.

12. The video display system according to claim 11.

13. The size specification gesture is an action in which the user moves a finger closer to or further away from the displayable area.

12. The video display system according to claim 11.

14. an aerial image display unit that displays aerial images, which are real images, within a predetermined displayable area in three-dimensional space; a sensor unit that detects a user in the vicinity of the displayable area and outputs detection information, and captures an image of a target and outputs video data; a control unit that determines a gesture including at least one of a movement and a posture of a body part of the user based on the detection information, and when the gesture is determined to be a target designating gesture that designates the target, controls the aerial image display unit so that an image of the target designated by the target designating gesture is displayed as the aerial image; A video display system comprising:

15. 1. A video control method implemented by a video display system having: an aerial video display unit that displays a real aerial video within a predetermined displayable area in a three-dimensional space; and a sensor unit that detects a user near the displayable area and outputs detection information, Identifying the position of the user's eyes based on the detection information, calculating a visual range, and determining whether the user has made a gesture of closing the eyes; controlling the aerial image display unit so that the aerial image is displayed at a designated display position, which is a display position determined in accordance with the visual range; A video control method comprising:

16. A video control method implemented by a video display system having an aerial video display unit that displays a real aerial video within a predetermined displayable area in three-dimensional space, and a sensor unit that detects a user in the vicinity of the displayable area and outputs detection information, determining a gesture including at least one of a movement and a posture of a body part of the user based on the detection information; when the gesture is a predetermined position designation gesture that designates a display position of the aerial image within the displayable area, controlling the aerial image display unit so that the aerial image is displayed at a designated display position that is the display position designated by the position designation gesture; A step of photographing an object and outputting image data of the object; when the gesture is determined to be a target designating gesture for designating the target to be displayed at the designated display position, controlling the aerial image display unit so that an image of the target designated by the target designating gesture is displayed as the aerial image at the designated display position; A video control method comprising:

Images