Video display system, video control method, and video control program

Abstract

A video display system (1) is provided with: an aerial video display unit (10) that displays aerial videos (71, 72), which are real images, within a prescribed displayable region (70) in a three-dimensional space; a sensor unit (20) that detects users (81, 82) in the vicinity of the displayable area (70) and outputs detection information; and a control unit (30) that determines, on the basis of the detection information, a limb movement including at least one of the movement and posture of the body parts of the users (81, 82), and controls the aerial image display unit (10) when the limb movement is a predetermined position-specified limb movement that specifies the display position of the aerial images (71, 72) within the displayable area (70). The aerial images (71, 72) are displayed at a specified display position, which is a display position specified by the position-specifying limb movement.

Description

Technical Field

[0001] The present disclosure relates to an image display system, an image control method and an image control program. Background Art

[0002] Patent document 1 proposes an aerial image interaction device, which has: an imaging optical system that images the real image of a projected object arranged on one side of a light bending surface as an aerial image on the other side of the light bending surface; and an object determination unit that determines the position, or the position and shape of an object close to the real image.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 5212991 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, the above-mentioned device has a problem that the imaging position of the aerial image may not be an appropriate position for the observer (ie, the user of the device).

[0008] An object of the present disclosure is to display an aerial image at a display position appropriate for a user.

[0009] Means for solving problems

[0010] The image display system disclosed in the present invention is characterized in that the image display system comprises: an aerial image display unit, which displays an aerial image as a real image in a predetermined displayable area in three-dimensional space; a sensor unit, which detects a user near the displayable area and outputs detection information; and a control unit, which determines the three-dimensional position of the user's body part based on the detection information, determines the display position of the aerial image based on the result of the determination, and controls the aerial image display unit so that the aerial image is displayed at the display position.

[0011] Another image display system disclosed in the present invention is characterized in that the image display system comprises: an aerial image display unit, which displays an aerial image as a real image within a predetermined displayable area in three-dimensional space; a sensor unit, which detects a user near the displayable area and outputs detection information; and a control unit, which determines a limb movement of at least any one of the movement and posture of a body part of the user based on the detection information, and when the limb movement is a predetermined position-specified limb movement that specifies a display position of the aerial image within the displayable area, controls the aerial image display unit to display the aerial image at the display position specified by the position-specified limb movement, i.e., the specified display position.

[0012] Effects of the Invention

[0013] According to the present disclosure, it is possible to display an aerial image at a display position appropriate for the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram schematically showing the configuration of the image display system according to the first embodiment.

[0015] Figure 2 This is a diagram showing an example of a usage state of the image display system according to Embodiment 1.

[0016] Figure 3 (A) and (B) are diagrams showing examples of the display position of the aerial image designated by the user, that is, the designated display position.

[0017] Figure 4 This is a block diagram schematically showing the configuration of a control unit of the image display system according to the first embodiment.

[0018] Figure 5 This is a diagram showing an example of the hardware configuration of the video display system according to the first embodiment.

[0019] Figure 6 This is a flowchart showing the operation of the control unit of the image display system according to the first embodiment.

[0020] Figure 7 This is a diagram schematically showing the configuration of an image display system according to a modified example of the first embodiment.

[0021] Figure 8 This is a block diagram schematically showing the configuration of a control unit of a video display system according to a modified example of the first embodiment.

[0022] Fig. 9 This is a flowchart showing the operation of the control unit of the image display system according to the modified example of the first embodiment.

[0023] Fig.10This is a diagram schematically showing the structure of the image display system according to the second embodiment.

[0024] Fig.11 This is a block diagram schematically showing the structure of the control unit of the image display system according to the second embodiment.

[0025] Fig.12 This is a flowchart showing the operation of the control unit of the image display system according to the second embodiment.

[0026] Fig.13 This is a diagram schematically showing the structure of an image display system according to a modified example of embodiment 2.

[0027] Fig.14 This is a flowchart showing the operation of the control unit of the image display system according to the modified example of the second embodiment.

[0028] Fig.15 (A) and (B) are diagrams showing the structure and operation of the image display system of embodiment 3.

[0029] Fig.16 This is a flowchart showing the operation of the control unit of the image display system according to the third embodiment.

[0030] Fig.17 (A) and (B) are diagrams showing examples of size-specified limb movements in the image display system of embodiment 4.

[0031] Fig.18 (A) and (B) are diagrams showing examples of size-specified limb movements in an image display system of a variation of embodiment 4.

[0032] Fig.19 This is a flowchart showing the operation of the control unit of the image display system according to the fourth embodiment. DETAILED DESCRIPTION

[0033] Hereinafter, an image display system, an image control method, and an image control program according to an embodiment will be described with reference to the accompanying drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified.

[0034] Implementation Method 1

[0035] Figure 1 1 is a diagram schematically showing the structure of the image display system 1 according to the first embodiment. Figure 1 As shown, the image display system 1 includes an aerial image display unit 10 as an image display device that displays (ie projects) a real image as aerial images 71 , 72 , a sensor unit 20 , and a control unit 30 as a control device that controls the operation of the aerial image display unit 10 .

[0036] The aerial image display unit 10 displays one or more aerial images (e.g., aerial images 71, 72, etc.) in a predetermined displayable area 70 in three-dimensional space. The aerial image display unit 10 includes a display unit 11 as a display device for displaying images, and a real image optical system 12 for imaging the images displayed on the display unit 11 as aerial images 71, 72 in the displayable area 70. The display unit 11 is, for example, a liquid crystal display or an LED (Light Emitting Diode) display, and displays images based on input image data (e.g., image content).

[0037] The real image optical system 12 includes a beam splitter 13, which is an optical component that reflects and transmits incident light, and a retroreflective sheet 14, which is a retroreflective component that retroreflects the light reflected by the beam splitter 13. The light retroreflected by the retroreflective sheet 14 travels in a direction opposite to the direction of travel of the light incident on the retroreflective sheet 14, passes through the beam splitter 13, and forms an image in the displayable area 70.

[0038] The beam splitter 13 can be any optical component that can separate light by reflecting and transmitting, and can be, for example, a semi-transparent mirror, a glass plate, etc. In addition, the retro-reflective sheet 14 is, for example, a lens-enclosed reflective sheet formed by embedding the lower half of a plurality of transparent tiny glass beads (i.e., microbeads) in a reflective film, or a micro-prism-type reflective sheet formed by embedding a plurality of transparent micro-prisms in a reflective film.

[0039] The sensor unit 20 has, for example, a camera as an imaging device or a detector for detecting an object. Specific examples of the camera of the sensor unit 20 are an infrared camera, an infrared sensor, a stereo camera, a visible light camera, etc. Specific examples of the detector are, for example, a distance measuring sensor such as LiDAR using light, a millimeter wave radar using radio waves, or a sonar using sound. In addition, the detector can detect the distance to an object and the shape of the object. The sensor unit 20 detects one or more users (for example, users 81 and 82) near the displayable area 70 and outputs detection information (for example, an image signal, a detection signal).

[0040] The control unit 30 determines a body movement (gesture) including at least one of the movement and posture of the body parts of the users 81 and 82 based on the detection information, and when the body movement is a predetermined position-specifying body movement that specifies the display position of the aerial images 71 and 72 in the displayable area 70, the control unit 30 controls the aerial image display unit 10 so that the aerial images 71 and 72 are displayed at the display position specified by the position-specifying body movement, that is, the designated display position. Specifically, the control unit 30 controls the control unit 30 of the aerial image display unit 10 so that the aerial images 71 and 72 are displayed at the display position specified by the position-specifying body movement. In order to display the aerial images 71 and 72 at the designated display position in the displayable area 70, the control unit 30 sets the image data to be displayed on the display unit 11 (for example, selects image content from a database).

[0041] When it is determined that the body movement includes the first body movement of the user 81 as the first user and the second body movement of the user 82 as the second user, and both the first body movement and the second body movement are position-specifying body movements that specify the display positions of the aerial images 71 and 72 in the displayable area 70, the control unit 30 controls the aerial image display unit 10 to display the aerial images 71 and 72 at the first display position specified by the first body movement and the second display position specified by the second body movement, respectively. For example, in order to display the aerial images 71 and 72 at the specified display positions in the displayable area 70, the control unit 30 sets the image data to be displayed on the display unit 11.

[0042] The control unit 30 may detect the skeleton of the user 81, 82 based on the detection information and determine the display position specified by the position-specifying body movement based on the state of the skeleton or determine the display position specified by the position-specifying body movement based on whether the skeleton is in a predetermined state.

[0043] Furthermore, the control unit 30 may also determine that the state in which the user 81, 82 extends the first finger of the hand in the first direction and extends the second finger in the second direction intersecting the first direction is a position-specifying body movement, and control the display of the aerial images 71, 72 at the designated display position designated by the first finger and the second finger. Here, the first finger and the second finger are the index finger and the thumb, and the intersecting angle is about 90 degrees.

[0044] Furthermore, when the facial expression (including changes in expression) of the user 81, 82 is a predetermined expression, the control unit 30 may determine the facial expression as a position-specified body movement, and control the display of the aerial images 71, 72 at the designated display position designated by the expression. Expressions that can be used as body movements include, for example, opening and closing of the mouth, the shape of the mouth, blinking, blinking of one eye, facial expressions, and the direction of sight of the eyes.

[0045] Figure 2 1 is a diagram showing an example of a usage state of the video display system 1 . Figure 3 (A) and (B) of FIG. 8 are diagrams showing an example of a display position of the aerial image 71 designated by the user 81, that is, a designated display position. Figure 2 In the example, users 81 and 82 point their index fingers in the vertical direction as the first direction (in Figure 2 The thumb, which is another finger, is extended in a lateral direction (in the height direction) which is a second direction intersecting the first direction. Figure 2 In this example, aerial images 71 and 72 are respectively displayed in a rectangular area having a longitudinal side substantially parallel to the index finger and a transverse side substantially parallel to the thumb. Therefore, the user 81 changes the position of the position-specifying body movement formed by the index finger and the thumb, such as Figure 3 As shown in (A) and (B), the aerial images 71 and 72 can be moved.

[0046] Figure 4 1 is a block diagram schematically showing the structure of the control unit 30 of the image display system 1 according to the first embodiment. Figure 4 As shown, the control unit 30 includes a three-dimensional position detection unit 31 , a visible area determination unit 32 , an image data control unit 33 , a limb movement determination unit 34 , and a display control unit 35 .

[0047] The three-dimensional position detection unit 31 detects the position and movement of users 81 and 82 who are observers of the aerial images 71 and 72. For example, the three-dimensional position detection unit 31 specifies the position of the user near the aerial image display unit 10 and the viewpoint position (ie, the position of the eyes).

[0048] The visible area determination unit 32 determines the area visible (i.e., able to be observed) by the users 81 and 82, i.e., the visual range. The visible area determination unit 32 calculates the visual range, for example, based on the positional relationship between the aerial image display unit 10 and the eyes of the users 81 and 82. Specifically, the visible area determination unit 32 calculates the visual range of the users 81 and 82 based on the configuration information of the aerial image display unit 10 and the three-dimensional position information from the three-dimensional position detection unit 31.

[0049] The image data control unit 33 controls the image data for displaying the aerial images 71 and 72 (i.e., selects image content from the database) according to the visual range of the aerial images 71 and 72. For example, the image data is controlled so that it is displayed only in a specific area visible to the users 81 and 82 in the area that can be displayed by one real image optical system 12.

[0050] The body movement identification unit 34 identifies the spatial operation, i.e., body movement (i.e., the command issued by the user) performed by the user 81, 82. After the user 81, 82 sees the aerial images 71, 72, the body movement identification unit 34 detects the specific body movement performed by the user and identifies the content of the desired operation through the body movement.

[0051] The display control unit 35 controls the video data according to the body movements of the users 81 and 82. The control content is transition to a screen corresponding to the body movement or selection of video content. The display control unit 35 outputs the video data (ie, video content) to the aerial video display unit 10.

[0052] Figure 5 1 is a diagram showing an example of the hardware structure of the image display system 1 of embodiment 1. The image display system 1 has a display unit 11, a sensor unit 20, and a control unit 30. The control unit 30 is, for example, a computer capable of executing the display control program of embodiment 1. The control unit 30 has a processor 101, a memory 102, and a storage device 103. The processor 101 is a CPU (Central Processing Unit) or the like. The memory 102 is, for example, a volatile semiconductor memory such as a RAM (Random Access Memory). The storage device 103 is a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD). In addition, the control unit 30 may also have a communication unit for communicating with other devices via a network, an operating unit such as a keyboard as an input interface.

[0053] Each function of the control unit 30 is realized by a processing circuit. The processing circuit may be dedicated hardware or a processor 101 that executes a program stored in a memory 102. The processor 101 may be any of a processing device, a computing device, a microprocessor, a microcomputer, and a DSP (Digital Signal Processor).

[0054] When the processing circuit is dedicated hardware, the processing circuit is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a component obtained by combining any of them.

[0055] When the processing circuit is the processor 101, the display control program of the first embodiment is implemented by software, firmware, or a combination of software and firmware. Software and firmware are described as programs and stored in the memory 102. The processor 101 reads out and executes the display control program stored in the memory 102, thereby realizing Figure 4 The functions of each unit shown in the figure are shown in the figure. In addition, the control unit 30 may be partially implemented by dedicated hardware and partially implemented by software or firmware. In this way, the processing circuit can be implemented by hardware, software, firmware, or a combination of any of them. Figure 1 The functions of each functional block are shown.

[0056] Figure 6 1 is a flowchart showing the operation of the control unit 30 of the image display system 1 according to Embodiment 1. First, the control unit 30 receives detection information from the sensor unit 20 (step S11), detects the three-dimensional position of the body parts of the users 81 and 82 (step S12), and determines the visible area of ​​the users 81 and 82 (step S13). The control unit 30 sets the image data (e.g., image content) (step S14), determines the body movement based on the detection information (e.g., image data) received from the sensor unit 20 (step S15), and causes the aerial image display unit 10 to display the aerial images 71 and 72 at the display position indicated by the body movement, i.e., the designated display position (step S16).

[0057] As described above, according to the first embodiment, the aerial images 71 and 72 can be displayed at appropriate display positions for the users 81 and 82 according to the body movements of the users 81 and 82. For example, the display positions of the aerial images 71 and 72 can be changed to appropriate positions according to the user 81 who is an adult, the user 82 who is a child, or the user who is a wheelchair user (not shown).

[0058] Figure 7 FIG. 2 is a diagram schematically showing a configuration of an image display system 1a according to a modified example of the first embodiment. Figure 7 In, with Figure 1 The same or corresponding structures as shown are marked with Figure 1 The image display system 1a determines the display positions of the aerial images 71 and 72 according to the positions and postures of the users 81 and 82, which is different from the system that determines the display positions of the aerial images 71 and 72 according to the body movements of the users 81 and 82. Figure 1 The image display system 1 is different.

[0059] like Figure 7As shown, the image display system 1a includes an aerial image display unit 10, a sensor unit 20 for detecting users 81 and 82 near a displayable area 70 and outputting detection information, and a control unit 30a as a display control device. The control unit 30a determines the three-dimensional position of the body parts of the users 81 and 82 based on the detection information output from the sensor unit 20, determines the display position of the aerial images 71 and 72 based on the result of the determination, and controls the aerial image display unit 10 so that the aerial images 71 and 72 are displayed at the display position.

[0060] The control unit 30a determines the three-dimensional position of the eyes or faces of the users 81 and 82, determines the display position of the aerial images 71 and 72 based on the position and sight lines of the eyes or the position and orientation of the faces, and controls the aerial image display unit 10 so that the aerial images 71 and 72 are displayed at the display position. That is, in the image display system 1a, the aerial images 71 and 72 can be displayed near the positions of the eyes of the users 81 and 82, for example. The three-dimensional position of the eyes or faces of the users 81 and 82 is an example of the position and posture of the users 81 and 82, and the position and posture of the users 81 and 82 can also be determined based on the state of body parts other than the eyes or faces.

[0061] Figure 8 1 is a block diagram schematically showing the configuration of a control unit 30a of an image display system 1a according to a modified example of the first embodiment. Figure 8 In, with Figure 4 The same or corresponding structures as shown are marked with Figure 4 The image display system 1a determines the display positions of the aerial images 71 and 72 according to the positions and postures of the users 81 and 82, which is different from the system that determines the display positions of the aerial images 71 and 72 according to the actions, i.e., the body movements, of the users 81 and 82. Figure 1 The control unit 30a has a user position and posture determination unit 34a, but does not have Figure 4 The control unit 30a is different from the control unit 30 in that it has a body movement determination unit. The user position and posture determination unit 34a of the control unit 30a determines the three-dimensional position of the eyes or faces of the users 81 and 82, determines the display position of the aerial images 71 and 72 according to the position and orientation of the eyes or faces, and controls the aerial image display unit 10 to display the aerial images 71 and 72 at the display position. In addition, the control unit 30a may include both the user position and posture determination unit 34a and the body movement determination unit 34.

[0062] Fig. 9 2 is a flowchart showing the operation of the control unit 30a of the image display system 1a according to the modification of the first embodiment. Fig. 9 In, with Figure 6 Steps that are identical or correspond to the steps shown are marked with Figure 6 In the image display system 1a, the control unit 30a receives detection information from the sensor unit 20 (step S11), detects the three-dimensional position of the body parts of the users 81 and 82 (step S12), and determines the visible area of ​​the users 81 and 82 (step S13). The control unit 30a sets image data (for example, image content) (step S14a), determines the position and posture of the users 81 and 82 based on the detection information (step S15a), and displays the aerial images 71 and 72 in the area corresponding to the position and posture of the users 81 and 82 (step S16a).

[0063] As described above, according to the image display system 1a of the modified example of the first embodiment, it is possible to display the aerial images 71 and 72 at display positions appropriate for the users 81 and 82 according to the states of the body parts of the users 81 and 82 (e.g., the positions of the eyes, the direction of the sight lines, and the orientation of the faces). That is, the image display system 1a can display the aerial images 71 and 72 at positions appropriate for the users 81 and 82, respectively, in accordance with the respective persons.

[0064] Furthermore, the image display system 1 a according to the modified example of the first embodiment can extract people at specific positions from a crowd, and display the aerial images 71 and 72 at positions suitable for extracting the specific people.

[0065] In addition, in the image display system 1a of a variation of implementation mode 1, it may be that after the control unit 30a displays the aerial images 71, 72 at the display position determined according to the result of the judgment, it may judge the limb movement of at least any one of the movements and postures of the body parts of the users 81, 82 based on the detection information, and when the limb movement is a predetermined position-specified limb movement of the display position of the aerial images 71, 72 in the newly designated displayable area 70, the aerial image display unit 10 is controlled to display the aerial images 71, 72 at the display position specified by the position-specified limb movement.

[0066] In addition, any configurations of Embodiments 2, 3, and 4 described later may be combined with the image display system 1 or 1 a.

[0067] Implementation Method 2

[0068] Fig.10: is a diagram schematically showing the structure of the image display system 2 of the second embodiment. In the image display system 2 of the second embodiment, the sensor unit 20 has a camera that captures the user 81 and the object 90 displayed as the aerial image 71 and outputs image data. The object 90 may not be an object, for example, it may be an image projected on a monitor (which may also include a dynamic image) or information of a character string projected on a monitor. That is, the object 90 is not limited to an actual existing object. In the case where it is determined that the body movement is a predetermined body movement that specifies the object 90 that should be displayed at the display position of the aerial image 71, that is, an object-specified body movement, the control unit 40 controls the aerial image display unit 10 so that the image of the object 90 specified by the object-specified body movement is displayed as the aerial image 71 at the display position.

[0069] like Fig.10 As shown, the object-specifying body movement is a body movement in which the user 81 points to the object 90 with a finger. Fig.10 In the embodiment, by simply pointing to the object 90 that the user 81 wants to display in front of his / her eyes as the aerial image 71 with his / her index finger, the aerial image 71 of the object 90 can be displayed in front of the user 81 (for example, the display position in Embodiment 1). In addition, the control unit 40 determines, based on the detection information, an object that intersects the vector F in the direction pointed by the finger of the user 81 as the object 90 to be displayed. The object 90 is an object existing in reality, an object displayed on a large screen, or the like.

[0070] Fig.11 2 is a block diagram schematically showing the structure of the control unit 40 of the video display system 2 according to the second embodiment. Fig.11 In, with Figure 4 The same or corresponding structures as shown are marked with Figure 4 In the control unit 40, the body movement identification unit 34b identifies the object 90 pointed to by the user 81 using the finger as a body part, and the image data control unit 33 obtains the image data of the object 90 from a database of pre-shot images or image data shot by the sensor unit 20, processes it into an image format displayed at the display position, and provides it to the display control unit 35.

[0071] Fig.12 4 is a flowchart showing the operation of the control unit 40 of the video display system 2 according to the second embodiment. Fig.12 In step S21 to S23, Figure 6The control unit 40 of the image display system 2 determines the object 90 pointed to by the user 81 using the finger as a body part (step S24), obtains the image data of the object 90 from the database of pre-shot images or the image data shot by the sensor unit 20 (step S25), determines the position and posture of the user 81 (step S26), and displays the aerial image 71 of the object 90 in the area corresponding to the position and posture of the user 81 (step S27).

[0072] Fig.13 FIG. 2 is a diagram schematically showing the structure of an image display system 2a according to a modified example of Embodiment 2. Fig.13 In, with Fig.11 The same or corresponding structures as shown are marked with Fig.11 The image display system 2a further includes a camera 21 that captures an object 90 displayed as an aerial image 71 and outputs image data of the object 90. When it is determined that the body movement is an object-specified body movement that specifies the object 90 displayed at the display position of the aerial image 71, the control unit 40 controls the aerial image display unit 10 so that the image of the object 90 specified by the object-specified body movement is displayed as the aerial image 71 at the display position.

[0073] Fig.14 2 is a flowchart showing the operation of the control unit of the image display system 2a according to the modified example of the second embodiment. Fig.14 In, with Fig.12 Steps that are identical or correspond to the steps shown are marked with Fig.12 The image display system 2a is different from the image display system 2 in that an image captured by the camera 21 is displayed as an aerial image 71 at a display position.

[0074] According to the second embodiment, it is possible to display the aerial image 71 of the object 90 indicated by the finger at a display position suitable for the user 81 .

[0075] Furthermore, an object 90 located far away, an object 90 located at a high position that is difficult to reach by hand, an object 90 in an image displayed on a large-screen screen, etc. can be displayed near the user 81 using the aerial image 71 .

[0076] In addition, the image display system 2 or 2a may be combined with any of the structures in the first, third, and fourth embodiments. For example, Fig.10 and Fig.13The control unit 40 shown can also perform the following operations: determine a body movement of the user 81 including at least any one of the movement and posture of the body part from the image information captured by the camera 20, and when the body movement is a predetermined position-specifying body movement for specifying the display position of the aerial image 71 within the displayable area 70 (for example, as shown in Figure 2 the body movement of forming an L shape or a left-right reversed L shape using the thumb and index finger), control the aerial image display unit 10 so that the aerial image 71 is displayed at the specified display position, which is the display position specified by the position-specifying body movement. The specification of the display position based on the position-specifying body movement may be performed before or after the object-specifying body movement. Further, as described in Embodiment 1, the position-specifying body movement may be other body movements.

[0077] <<Embodiment 3>>

[0078] Fig.15 (A) and (B) of FIG. are diagrams showing the structure and operation of the image display system 3 according to Embodiment 3. Fig.15 (A) of FIG. shows a state where the aerial image 71 is displayed at a display position separated from the aerial image display unit 10a by a distance L1. Fig.15 (B) of FIG. shows a state where the aerial image 71 is displayed at a display position separated from the aerial image display unit 10a by a distance L2 (L2 < L1).

[0079] The image display system 3 includes a sensor unit 20, an aerial image display unit 10a, and a control unit 30. The aerial image display unit 10a includes a first display unit 11a that displays a first image, a second display unit 11b that displays a second image, and a real-image optical system 12 that displays the first image or the second image as a real-image aerial image 71. The distance from the first display unit 11a to the real-image optical system 12 is longer than the distance from the second display unit 11b to the real-image optical system 12. Further, the control unit 30 determines a body movement of the user 81 including at least any one of the movement and posture of the body part from the detection information, and when the body movement is a predetermined depth-specifying body movement for specifying the position in the depth direction of the displayable area 70, controls the aerial image display unit 10a so that the aerial image 71 is displayed within the displayable area 70 specified by the depth-specifying body movement.

[0080] The depth-specifying body movement is a movement that uses the body part of the user 81 to specify the position of the displayable area 70 in the depth direction. The depth-specifying body movement can specify the position in the depth direction and can be any body movement. For example, the depth-specifying body movement can also be included in the position-specifying body movement described in Embodiments 1 and 2. That is, the depth-specifying body movement can also be a body movement performed simultaneously with the position-specifying body movement. For example, Figure 2 As shown, the position in the depth direction can also be specified in addition to the position in the height direction and the left-right direction of the displayed image by using a body movement of an L shape or a left-right reversed L shape formed by the thumb and index finger. In addition, the depth-specifying body movement can also be a body movement different from the position-specifying body movement. In this case, the depth-specifying body movement is performed before or after the position-specifying body movement.

[0081] Fig.16 2 is a flowchart showing the operation of the control unit of the video display system 3 according to the third embodiment. Fig.16 In the steps S31 to S34 and Fig.12 When the position of the displayable area 70 in the depth direction is specified by the body movement (step S35), the image display system 3 moves the displayable area 70 to the position specified by the body movement and displays the aerial image in the displayable area 70 (step S36).

[0082] As described above, according to the video display system 3 of the third embodiment, it is possible to display the aerial video 71 at a display position appropriate for the user 81 .

[0083] In addition, any configurations of the first, second, and fourth embodiments may be combined with the image display system 3 .

[0084] In addition, Fig.15 In (A) and (B) of FIG. 1 , the first display unit 11a and the second display unit 11b are described as two display devices. However, this structure can be replaced by one display device and a moving mechanism (not shown) for moving the display device. That is, the moving mechanism can move the display device closer to the real image optical system 12 or farther away from the real image optical system 12. In this case, the display device is moved by moving the display device to the real image optical system 12. Fig.15 The first position indicated as "the first display portion 11a" in (A) and Fig.15 The second position indicated as the "second display unit 11 b " in (B) is moved to an arbitrary position, and the position of the aerial image in the depth direction is changed.

[0085] In addition, the real image optical system 12 (or a part of the optical system thereof) can be replaced by one display device and a moving mechanism (not shown) for moving the real image optical system 12. Fig.15That is, the moving mechanism can make the image optical system 12 (or a part of the optical system) approach the display device, or move away from the display device. In this case, the display device moves toward Fig.15 The first position indicated as "the first display portion 11a" in (A) and Fig.15 The position of the aerial image in the depth direction is changed by moving to any position in the second position indicated as the "second display unit 11b" in (B).

[0086] Implementation Method 4

[0087] Fig.17 (A) and (B) are diagrams showing examples of size-specified limb movements in the image display system of embodiment 4. Fig.17 (A) shows an example of specifying a body movement in which the user 81 reduces the size of the display area of ​​the aerial image 71 by reducing the distance between two fingers. Fig.17 (B) shows an example of specifying a body motion in which the user 81 increases the size of the display area of ​​the aerial image 71 by increasing the distance between two fingers.

[0088] The structure of the image display system of the fourth embodiment is the same as the structure of any of the image display systems in the above-mentioned first to third embodiments. In the case where the body movement is a predetermined size-specified body movement for enlarging or reducing the size of the aerial image 71 in the displayable area 70, the control unit of the image display system of the fourth embodiment controls the aerial image display unit 10 so that the aerial image 71 is displayed in the size specified by the size-specified body movement. In addition, the user 81 uses the fingers of the left and right hands as the two fingers, but the same operation can also be performed using the two fingers of one hand. In addition, the same operation can also be performed using body parts other than fingers (such as both arms, both feet, arms and fingers, etc.).

[0089] Fig.18 (A) and (B) are diagrams showing examples of size-specified limb movements in an image display system of a variation of embodiment 4. Fig.18 (A) shows an example of a body motion designated by the user 81 to reduce the size of the display area of ​​the aerial image 71 by bringing a finger, which is an example of a body part, close to the display area. Fig.17 (B) shows an example of a body motion designation in which the user 81 increases the size of the display area of ​​the aerial image 71 by moving a finger away from the display area.

[0090] The structure of the image display system of the modification of the fourth embodiment is the same as the structure of any of the image display systems in the first to third embodiments. When the body movement is a predetermined size-specifying body movement for enlarging or reducing the size of the aerial image 71 in the displayable area 70, the control unit of the fourth embodiment controls the aerial image display unit 10 to display the aerial image 71 in the size specified by the size-specifying body movement. In addition, the user 81 uses fingers to perform body movements, but the same operation can also be performed using body parts other than fingers (such as both arms, both feet, arms and fingers, etc.).

[0091] Fig.19 4 is a flowchart showing the operation of the control unit of the image display system according to the fourth embodiment. Fig.19 In step S41 to S45, Figure 6 The steps S11 to S15 shown are the same. After displaying the aerial image 71, the image display system of the fourth embodiment can enlarge or reduce the aerial image 71 by a size-designating body movement (step S46).

[0092] As described above, according to the fourth embodiment, the aerial image 71 of the object 90 indicated by the finger can be displayed at a display position suitable for the user 81 and in a desired size.

[0093] In addition, any structure of Embodiments 1, 2, and 3 can be combined with the image display system of Embodiment 4.

[0094] 《Other Variations》

[0095] The image display systems of the first to fourth embodiments described above can be applied to a non-contact operation panel used by multiple persons, a digital signage having a function of switching display contents by user operation, and the like.

[0096] Furthermore, the image display systems of the first to fourth embodiments described above can also be used as display devices for equipment in automobiles.

[0097] Description of symbols

[0098] 1, 1a, 2, 2a, 3: image display system; 10, 10a: aerial image display unit; 11: display unit; 11a: first display unit; 11b: second display unit; 12: real image optical system; 13: beam splitter; 14: retroreflective sheet (retroreflective component); 20: sensor unit; 21: camera; 30, 30a, 40: control unit; 31: three-dimensional position detection unit; 32: visible area determination unit; 33: image data control unit; 34, 34b: limb movement determination unit; 34a: user position and posture determination unit; 35: display control unit; 70: displayable area; 71, 72: aerial image; 81, 82: user; 81a: hand; 90: object.

Claims

1. An image display system, characterized in that: The image display system comprises: an aerial image display unit that displays an aerial image as a real image within a predetermined displayable area in a three-dimensional space; a sensor unit that detects a user near the displayable area and outputs detection information; as well as A control unit determines the three-dimensional position of the user's body part based on the detection information, determines the display position of the aerial image based on the result of the determination, and controls the aerial image display unit to display the aerial image at the display position.

2. The image display system according to claim 1, characterized in that: The control unit determines the three-dimensional position of the face or eyes of the user as the body part, The control unit determines the display position of the aerial image according to the position and orientation of the face or the position and sight line of the eyes, The control unit controls the aerial image display unit to display the aerial image at the display position.

3. The image display system according to claim 1 or 2, characterized in that: After displaying the aerial image at the display position determined according to the result of the determination, the control unit determines a body movement including at least one of a movement and a posture of a body part of the user based on the detection information, In a case where the body movement is a predetermined position-specifying body movement that specifies the display position of the aerial image within the displayable area, the control unit controls the aerial image display unit to display the aerial image at the display position specified by the position-specifying body movement, i.e., the designated display position.

4. An image display system, characterized in that: The image display system comprises: an aerial image display unit that displays an aerial image as a real image within a predetermined displayable area in a three-dimensional space; a sensor unit that detects a user near the displayable area and outputs detection information; and A control unit that determines a physical movement of at least any one of the movement and posture of the user's body parts based on the detection information, and when the physical movement is a predetermined position-specified physical movement that specifies the display position of the aerial image within the displayable area, controls the aerial image display unit to display the aerial image at the display position specified by the position-specified physical movement, i.e., the designated display position.

5. The image display system according to claim 3 or 4, characterized in that: The control unit changes the display position in the displayable area according to the position-designating body movement.

6. The image display system according to claim 3 or 4, characterized in that: When it is determined that the body movement includes a first body movement of a first user and a second body movement of a second user, and both the first body movement and the second body movement are body movements that specify the display position of the aerial image within the displayable area, the control unit controls the aerial image display unit so that the aerial image is displayed at the first display position specified by the first body movement and the second display position specified by the second body movement, respectively.

7. The image display system according to claim 6, characterized in that: The control unit changes the first display position and the second display position in the displayable area according to the position-specifying body movement.

8. The image display system according to any one of claims 3 to 7, characterized in that: The control unit detects the state of the user's skeleton based on the detection information, and determines the display position specified by the position-specifying limb movement based on the state of the skeleton.

9. The image display system according to any one of claims 3 to 7, characterized in that: The control unit determines, as the user's body movement, a state in which a first finger of the hand is extended in a first direction and a second finger of the hand is extended in a second direction intersecting the first direction as the position specifying body movement, The control unit displays the aerial image at the designated display position designated by the first finger and the second finger.

10. The image display system according to any one of claims 3 to 7, characterized in that: When the facial expression of the user is a predetermined expression, the control unit determines the facial expression as the position-specifying body movement. The aerial image is displayed at the designated display position designated by the expression.

11. The image display system according to claim 1, characterized in that: The sensor unit includes at least a camera for photographing an object and outputting image data. The control unit determines a body movement including at least one of a movement of a body part and a posture of the user based on the detection information or the image of the user captured by the camera, When it is determined that the body movement is an object-specifying body movement that specifies the object displayed at the display position, the control unit controls the aerial image display unit so that the image of the object specified by the object-specifying body movement is displayed as the aerial image at the display position.

12. The image display system according to claim 11, characterized in that: In a case where the body movement is a predetermined position-specifying body movement that specifies the display position of the aerial image within the displayable area, the control unit controls the aerial image display unit to display the aerial image at the display position specified by the position-specifying body movement, i.e., the designated display position.

13. The image display system according to claim 3 or 4, characterized in that: The sensor unit includes a camera that captures the user and the object and outputs image data. When it is determined that the body movement is an object-specified body movement that specifies the object displayed at the specified display position, the control unit controls the aerial image display unit so that the image of the object specified by the object-specified body movement is displayed as the aerial image at the specified display position.

14. The image display system according to claim 3 or 4, characterized in that: The image display system also has a camera for photographing an object and outputting image data of the object. When it is determined that the body movement is an object-specified body movement that specifies the object displayed at the specified display position, the control unit controls the aerial image display unit so that the image of the object specified by the object-specified body movement is displayed as the aerial image at the specified display position.

15. The image display system according to claim 3 or 4, characterized in that: When it is determined that the body movement is an object-specified body movement that specifies the object to be displayed at the designated display position, the control unit controls the aerial image display unit so that the image of the object specified by the object-specified body movement is displayed as the aerial image at the designated display position using image information of the object from a database storing image data pre-photographed of the object to be displayed as the aerial image.

16. The image display system according to any one of claims 11 to 15, characterized in that: The object designating body motion is a motion in which the user points to the object with a finger.

17. The image display system according to claim 16, characterized in that: The control unit determines that an object that intersects a vector in a direction indicated by the finger is the target.

18. The image display system according to any one of claims 3, 4, 8 to 16, characterized in that: The aerial image display unit includes a first display unit that displays a first image, a second display unit that displays a second image, and a real image optical system that displays the first image or the second image as the aerial image as the real image. The distance from the first display unit to the real image optical system is longer than the distance from the second display unit to the real image optical system. The control unit determines a physical movement of at least any one of the movement and posture of the body part of the user based on the detection information, and when the physical movement is a predetermined depth-specifying physical movement that specifies the position in the depth direction of the displayable area, the control unit controls the aerial image display unit to display the aerial image within the displayable area whose position in the depth direction is specified by the depth-specifying physical movement.

19. The image display system according to any one of claims 3, 4, 8 to 16, characterized in that: When the body movement is a predetermined size-specifying body movement for enlarging or reducing the size of the aerial image within the displayable area, the control unit controls the aerial image display unit to display the aerial image in the size specified by the size-specifying body movement.

20. The image display system according to claim 19, characterized in that: The size specifying body movement is a movement of the user increasing or decreasing the distance between two fingers.

21. The image display system according to claim 19, characterized in that: The size specifying body motion is an action of the user moving a finger closer to or farther away from the displayable area.

22. An image display system, characterized in that: The image display system comprises: an aerial image display unit that displays an aerial image as a real image within a predetermined displayable area in a three-dimensional space; a sensor unit that detects a user near the displayable area and outputs detection information, captures the subject and outputs image data; and A control unit that determines a limb movement of at least any one of the movement and posture of the user's body parts based on the detection information, and when it is determined that the limb movement is an object-specified limb movement that specifies the object, controls the aerial image display unit to display the image of the object specified by the object-specified limb movement as the aerial image.

23. The image display system according to claim 22, characterized in that: The sensor unit includes a camera that detects users near the displayable area and outputs detection information and photographs the object and outputs image data, or a sensor that detects users near the displayable area and outputs detection information and a camera that photographs the object and outputs image data, or a first camera that detects users near the displayable area and outputs detection information and a second camera that photographs the object and outputs image data.

24. An image control method, which is implemented by an image display system, the image display system having an aerial image display unit that displays an aerial image as a real image in a predetermined displayable area in a three-dimensional space, and a sensor unit that detects a user near the displayable area and outputs detection information, characterized in that: The image control method comprises the following steps: Determining a body movement including at least one of a movement and a posture of a body part of the user based on the detection information; and In a case where the body movement is a predetermined position-specifying body movement that specifies the display position of the aerial image within the displayable area, the aerial image display unit is controlled to display the aerial image at the display position specified by the position-specifying body movement, i.e., the designated display position.

25. An image control program, characterized in that: The image control program causes a computer that controls an image display system, the image display system having an aerial image display unit that displays an aerial image as a real image in a predetermined displayable area in a three-dimensional space, and a sensor unit that detects a user near the displayable area and outputs detection information, to execute the following steps: Determine, based on the detection information, a body movement including at least one of a movement and a posture of a body part of the user; as well as In a case where the body movement is a predetermined position-specifying body movement that specifies the display position of the aerial image within the displayable area, the aerial image display unit is controlled to display the aerial image at the display position specified by the position-specifying body movement, i.e., the designated display position.

Images