Spatial image display device, spatial image display method, and program

Abstract

A spatial image display device (1) of the present disclosure comprises: a plurality of display units (21) which display an image; a plurality of optical elements (22) which respectively reflect image light projected from the image displayed on the plurality of display units (21), and which are arranged so that the reflected image light reaches an observer's eye, thereby displaying a virtual image (VI) of the image, in the same direction from the observer and at a different distance; a determination unit (22) which determines a visual effect to be added to an object image containing an image of an object (OB), based on the position of the object (OB), which is a specific subject an image of which is included in the image; and a display control unit (34) which controls the display units (21) so that the image is displayed on the basis of the object image and the visual effect.

Description

[Technical field]

[0001] The present disclosure relates to a spatial image display device, a spatial image display method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is known technology for displaying a virtual image in space by reflecting image light, which is light emitted from an image displayed on a display device, and allowing the reflected image light to reach the eye of an observer.

[0003] It is also known to display a spatial image composed of multiple virtual images by displaying the multiple virtual images at different distances in the same direction from the observer (Patent Document 1). This allows the observer to observe the spatial image with a higher sense of three-dimensionality. In addition, the sense of three-dimensionality can be further enhanced by designing the distance between the multiple virtual images to be short.

[0004] Also, a stereoscopic image can be provided by displaying multiple images with different brightness at different distances in the same direction from the observer (Patent Document 2). This makes it appear to the observer as if a stereoscopic image is being constructed even in a position where no image exists, and as a result, the image can be observed with a high sense of three-dimensionality. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2017 / 038091 [Patent Document 2] JP 2009-237310 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology described in Patent Document 1, the multiple virtual images are each formed on a two-dimensional virtual image surface. Therefore, in a configuration in which multiple virtual image surfaces are located at different distances in the same direction as seen from an observer, when an object (for example, a ball used in a sports game) moves in a direction corresponding to the observer's observation direction, the virtual image of the object may be displayed on one virtual image surface at one time and on another virtual image surface at another time. In such a case, the observer sees the virtual image of the object as if it had moved from one virtual image surface to another virtual image surface in an instant, and the movement appears unnatural compared to the movement of the object in real space. Therefore, the observer may not be able to obtain a sufficient sense of realism.

[0007] Furthermore, when the technology described in Patent Document 2 is applied to an image showing a moving object, where a virtual image of the object is displayed on each of multiple virtual image screens, depending on the position of the observer, parallax may occur between the multiple virtual images, making it difficult to obtain a sufficient sense of realism.

[0008] The object of the present disclosure, made in consideration of such circumstances, is to provide a spatial image display device, a spatial image display method, and a program that can provide an observer with a sufficient sense of realism when displaying virtual images of an object on multiple virtual image surfaces. [Means for solving the problem]

[0009] In order to solve the above problems, the spatial image display device of the present disclosure comprises a plurality of display units for displaying images, a plurality of optical elements arranged to respectively reflect image light emitted from the images displayed on the plurality of display units so that the reflected image light reaches the eyes of an observer, thereby displaying a virtual image of the image at different distances in the same direction from the observer, a determination unit that determines a visual effect to be added to an object image including an image of an object based on the position of an object that is a specific subject whose image is included in the image, and a display control unit that controls the display units to display the image based on the object image and the visual effect.

[0010] In addition, in order to solve the above-mentioned problems, the spatial image display method of the present disclosure is a spatial image display method for a spatial image display device that includes a plurality of display units that display images, and a plurality of optical elements that are arranged to reflect image light emitted from the images displayed on the plurality of display units, respectively, so that the reflected image light reaches the eyes of an observer, thereby displaying a virtual image of the image at different distances in the same direction from the observer, and includes a step of determining a visual effect to be added to an object image including an image of an object based on the position of an object that is a specific subject whose image is included in the image, and a step of controlling the display units to display the image based on the object image and the visual effect.

[0011] In order to solve the above problem, a program according to the present disclosure causes a computer to function as the above-mentioned spatial image display device. Effect of the Invention

[0012] According to the aerial image display device, aerial image display method, and program of the present disclosure, it is possible to provide a sufficient sense of realism to an observer when displaying virtual images of an object on a plurality of virtual image surfaces. [Brief description of the drawings]

[0013] [Figure 1] 1 is a schematic diagram of a spatial image display device according to a first embodiment. [Diagram 2] 2 is a schematic diagram of a virtual image display unit shown in FIG. 1. [Diagram 3] 3 is a diagram for explaining a virtual image displayed by the virtual image display unit shown in FIG. 2. [Figure 4] FIG. 2 is a diagram showing a virtual image seen by an observer. [Diagram 5] 2 is a diagram showing an example of a visual effect stored in a visual effect storage unit shown in FIG. 1; [Figure 6A] FIG. 2 is a diagram showing an example of the position of an object in real space. [Figure 6B]6B is a diagram showing an example of a virtual image displayed based on the position of the object shown in FIG. 6A. [Figure 7A] FIG. 11 is a diagram showing another example of the position of the object in real space. [Figure 7B] 7B is a diagram showing an example of a virtual image displayed based on the position of the object shown in FIG. 7A. [Figure 8A] FIG. 13 is a diagram showing yet another example of the position of an object in real space. [Figure 8B] 8B is a diagram showing an example of a virtual image displayed based on the position of the object shown in FIG. 8A. [Figure 9] 2 is a flowchart showing an example of the operation of the spatial image display device shown in FIG. [Figure 10A] FIG. 11 is a diagram showing a first variant of the visual effect. [Figure 10B] 10B is a diagram showing an example of a virtual image displayed based on the visual effect shown in FIG. 10A. [Figure 10C] 10B is a diagram showing another example of a virtual image displayed based on the visual effect shown in FIG. 10A. [Figure 11A] FIG. 11 is a diagram showing a second variation of the visual effect. [Figure 11B] 11B is a diagram showing an example of a virtual image displayed based on the visual effect shown in FIG. 11A. [Figure 11C] 11B is a diagram showing another example of a virtual image displayed based on the visual effect shown in FIG. 11A. [Figure 12A] FIG. 11 is a diagram showing a third variant of the visual effect. [Figure 12B] 12B is a diagram showing an example of a virtual image displayed based on the visual effect shown in FIG. 12A. [Figure 12C] 12B is a diagram showing another example of a virtual image displayed based on the visual effect shown in FIG. 12A. [Figure 13] FIG. 11 is a schematic diagram of a spatial image display device according to a second embodiment. [Figure 14] 14 is a sequence diagram showing an example of the operation of the spatial image display device shown in FIG. [Figure 15] FIG. 11 is a schematic diagram of a spatial image display device according to a third embodiment. [Figure 16] 16 is a sequence diagram showing an example of the operation of the spatial image display device shown in FIG. [Figure 17] FIG. 2 is a hardware block diagram of the spatial image display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] <<First embodiment>> The overall configuration of the first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of a spatial image display device 1 according to the first embodiment.

[0015] As shown in FIG. 1, the spatial image display device 1 according to the first embodiment includes a virtual image display section 2 and a visual effect determination section 3.

[0016] <Configuration of virtual image display unit> As shown in FIG. 2, the virtual image display unit 2 includes a plurality of display units 21 and a plurality of optical elements 22. The display unit 21 is configured by an organic EL (Electro Luminescence), a liquid crystal panel, or the like. The organic EL, the liquid crystal panel, or the like may be attached to a device such as a tablet, a VR goggle, or the like. The optical element 22 is configured by, for example, a half mirror that transmits a part of the incident light and reflects the remaining part of the light. In the example shown in FIG. 2, the virtual image display unit 2 includes two plurality of display units 211 and 212 and two optical elements 221 and 222, but is not limited thereto, and may include three or more display units 21 and three or more optical elements 22.

[0017] The display unit 21 displays an image. Specifically, the display unit 21 displays an image based on the object image based on the control of the visual effect determination unit 3. The object image is a part of an image generated by an imaging device such as a camera, and is an image including an image of an object OB. The object OB is any object, but may be, for example, an object included in the image and that may be noticed by an observer. In an image generated by an imaging device capturing an image of a tennis match, the object OB is, for example, a first player PL1, a second player PL2 who is the opponent of the first player PL1, and a ball B used by the first player PL1 and the second player PL2 during the match. In this example, the imaging device is disposed such that the first player PL1 is located in front of the imaging surface of the imaging device, and the second player PL2 is located in the back.

[0018] The image based on the object image includes an image in which the visual effect determined by the visual effect determination unit 3 is added to the object image. The image based on the object image may include an image in which the visual effect determined by the visual effect determination unit 3 is not added to the object image, that is, the object image itself. That is, the display unit 21 displays an image in which the visual effect determined by the visual effect determination unit 3 is added to the object image. The display unit 21 may also display an image in which the visual effect is not added to the object image. Note that an image in which a visual effect is added to an object image is an image in which processing has been applied to the object image. The image is generated by processing the object image according to the type of visual effect, as will be described in detail later.

[0019] In the above-mentioned example in which an image captured by an imaging device of a tennis match is generated, as shown in FIG. 3, the display unit 211 displays an object image IM1 including an image of a first player PL1, which is one of the objects OB. The display unit 212 displays an object image IM2 including an image of a second player PL2, which is one of the objects OB. Furthermore, the display unit 212 may display an object image IM3 including an image of a ball B, which is one of the objects OB, based on the control of the visual effects determination unit 3. At this time, the display unit 211 may not display the object image IM3. Furthermore, the display unit 211 may display the object image IM3. At this time, the display unit 212 may not display the object image IM3.

[0020] The optical elements 221 and 222 reflect the image light emitted from the images displayed on the multiple display units 211 and 212, respectively. The optical elements 221 and 222 are arranged so that the reflected image light reaches the observer's eye, and a virtual image VI of the image is displayed at a different distance in the same direction from the observer. In the example shown in FIG. 2 and FIG. 3, the distance from the observer to the virtual image surface VF1, which is one virtual image surface VF, is different from the distance from the observer to the virtual image surface VF2, which is the other virtual image surface VF. Furthermore, the direction from the observer to the virtual image surface VF1 and the direction from the observer to the virtual image surface VF2 may be the same, and in such a configuration, the virtual image surface VF1 is located in front of the virtual image surface VF2 for the observer. The position of the virtual image surface VF1 can be calculated by a known method based on the positions of the display unit 211 and the optical element 221, and the display unit 211 and the optical element 221 are arranged so that the virtual image surface VF1 is located at a desired position. For example, since the virtual image plane VF1 is located on a plane symmetrical to the display surface of the display unit 211 across the optical element 221, the optical element 221 may be arranged so that the display surface of the display unit 211 is symmetrical to the desired position at which the virtual image plane VF1 is displayed. Similarly, the position of the virtual image plane VF2 can be calculated by a known method based on the positions of the display unit 212 and the optical element 222, and the display unit 212 and the optical element 222 are arranged so that the virtual image plane VF2 is arranged at the desired position. For example, since the virtual image plane VF2 is located on a plane symmetrical to the display surface of the display unit 212 across the optical element 222, the optical element 222 may be arranged so that the display surface of the display unit 212 is symmetrical to the desired position at which the virtual image plane VF2 is displayed.

[0021] In the above-mentioned example in which the imaging device captures an image of a tennis match, the optical elements 221 and 222 are arranged so that the virtual image plane VF1 and the virtual image plane VF2 are located on the real court. Specifically, the optical element 221 is arranged so that the virtual image plane VF1 is located closer to the observer than the real net on the real court. Also, the optical element 222 is arranged so that the virtual image plane VF2 is located farther from the observer than the real net on the real court. In this way, by combining a plurality of display units 211 and 212, and the optical elements 221 and 222, it is possible to project a virtual image into a space in which the real court, net, etc. are located, thereby enhancing the three-dimensional effect that the observer has.

[0022] As a result, as shown in Fig. 4, the observer observes the virtual image VI1 displayed on the virtual image surface VF1 on the near side of the observer, and the virtual image VI2 displayed on the virtual image surface VF2 on the far side of the observer. The virtual image VIb shown in Fig. 4 is displayed on either the virtual image surface VF1 or the virtual image surface VF2. The virtual image surface VF on which the virtual image VIb is displayed will be described in detail later. Therefore, the observer observes a spatial image composed of the virtual images VI1, VI2, and VIb. Such a virtual image is generally called a Pepper's ghost, and the observer sees as if the display unit 211 and the display unit 212 are present on the virtual image surface VF1 and the virtual image surface VF2, respectively.

[0023] <Configuration of visual effects determination unit> As shown in FIG. 1, the visual effect determination unit 3 includes a communication unit 31, a visual effect storage unit 32, a determination unit 33, and a display control unit .

[0024] The communication unit 31 is configured by a communication interface. For example, standards such as Ethernet (registered trademark), FDDI (Fiber Distributed Data Interface), and Wi-Fi (registered trademark) may be used for the communication interface. The visual effect storage unit 32 is configured by memories such as HDD (Hard Disk Drive), SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory), and RAM (Random Access Memory). The determination unit 33 and the display control unit 34 configure a control unit (controller). The control unit may be configured by dedicated hardware such as ASIC (Application Specific Integrated Circuit) and FPGA (Field-Programmable Gate Array), may be configured by a processor, or may be configured including both.

[0025] The communication unit 31 receives object video information indicating an object video and object position information indicating the position of an object OB from an external device. The object video is a video including an image of an object OB that is a specific subject. The external device may be a video processing device that generates the object video information and the object position information, or may be any device that acquires the object video information and the object position information generated by the video processing device.

[0026] As an example, the position of the object OB may be indicated by the distance in a predetermined direction from a reference plane, which is a plane in real space corresponding to the virtual image plane VF on which the virtual image is displayed, to the object OB. The predetermined direction is a normal direction of an imaging plane of a camera that images the object. Note that the positional relationship of a specific subject as seen from the camera in the predetermined direction is substantially the same as the positional relationship of a virtual image of the specific subject as seen from an observer. In addition, in the above-mentioned example in which the imaging device images a tennis match, the predetermined direction is a direction from the imaging device toward a predetermined position (e.g., a central position) in a range in which a first player PL1 located in the foreground is present, and a direction from the imaging device toward a predetermined position (e.g., a central position) in a range in which a second player PL2 is present.

[0027] The visual effect storage unit 32 stores the position of the object OB in the real space in association with the visual effect. The visual effect is a visual effect that is applied to the object image when the virtual image display unit 2 displays the virtual image VI of the object image indicated by the object image information. In the example shown in Fig. 5, the position is indicated by the distance from a reference plane to the object OB in the real space, and the visual effect storage unit 32 stores the distance in association with the visual effect.

[0028] The determination unit 33 determines a visual effect to be added to an object image including an image of the object OB based on the position of the object OB, which is a specific subject whose image is included in the image. The position of the object OB may be the position of the object OB in real space, or the position of the image of the object OB in image space. In the first embodiment, the determination unit 33 determines a visual effect to be added to an object image indicated by the image information based on object position information received by the communication unit 31, which corresponds to the object image information received by the communication unit 31.

[0029] 5, the determination unit 33 determines a visual effect based on the distance from a reference plane to the object OB in real space. Specifically, the determination unit 33 extracts a visual effect stored in the visual effect storage unit 32 in correspondence with the distance. Then, the determination unit 33 determines the extracted visual effect as the visual effect to be added to the object image.

[0030] In addition, in a configuration in which the position of an object OB is indicated by the distance from a reference plane to the object OB, the determination unit 33 may determine, based on the distance, the display unit 21 that is to display the object video from among the multiple display units 21. As an example, the determination unit 33 determines that the display unit 21 that is to display the object OB is the display unit 21 that is to display the virtual image VI on the virtual image surface VF that corresponds to the reference plane that is the shortest distance to the object OB among the multiple reference planes.

[0031] The display control unit 34 controls the multiple display units 21 to display images based on the object images. The virtual image display unit 2 controlled by the display control unit 34 may be changeable, for example, may be a tablet, and may be replaced with VR goggles later. That is, the display control unit 34 may be compatible with multiple devices.

[0032] Specifically, the display control unit 34 controls the display unit 21 to display an image based on the object image with a visual effect added. In this embodiment, the display control unit 34 controls the display unit 21 to display an image with a visual effect determined by the determination unit 33 added to an object image of the object OB indicated by the object image information received by the communication unit 31. At this time, in a configuration in which the determination unit 33 displays an object image from among a plurality of display units 21 based on the distance, the display control unit 34 controls the display unit 21 determined by the determination unit 33 to display an image with a visual effect added to the object image. In the example shown in FIG. 3, the display control unit 34 controls the display unit 21 determined by the determination unit 33 to display an image with a visual effect added to an object image IM3 including an image of a ball B, which is an object OB.

[0033] The display control unit 34 may also control the predetermined display unit 21 to display an object image of a predetermined object OB indicated by the object image information. In the example shown in Fig. 3, the display control unit 34 controls the display unit 211 to display an object image IM1 including an image of a first player PL1 who is the predetermined object OB. The display control unit 34 also controls the display unit 212 to display an object image IM2 including an image of a second player PL2 who is the predetermined object OB.

[0034] Here, an example of the processing of the determination unit 33 and the display control unit 34 in the example shown in FIG. 5 will be described in detail.

[0035] In the example shown in Fig. 6A, the object image includes images of a first player PL1, a second player PL2, and a ball B. In addition, in real space, the distance between a first reference plane S1 and a second reference plane S2 is 9. In addition, in real space, a ball B, which is one of the objects, is located between the first reference plane S1 and the second reference plane S2, and the distance between the first reference plane S1 and the ball B is 8.2.

[0036] As described above, the determination unit 33 determines the display unit 21 that displays the object OB as the display unit 21 that displays the virtual image VI on the virtual image surface VF corresponding to the reference surface having the shortest distance to the object OB. In this example, the distance from the first reference surface S1 to the ball B is 8.2, and the distance from the second reference surface S2 to the ball B is 0.8. Therefore, the determination unit 33 determines the display unit 21 that displays the virtual image VIb of the ball B as the display unit 212 that displays the virtual image VI2 on the virtual image surface VF2 (see FIG. 3) corresponding to the second reference surface S2 having the shortest distance to the ball B.

[0037] Then, the determination unit 33 determines that the visual effect to be applied to the object OB is “none”, which is a visual effect stored in the visual effect storage unit 32, in correspondence with the distance of 0.8 from the second reference plane S2 to the ball B, which is the object OB.

[0038] The display control unit 34 controls the display unit 212 to display an object image IM3 including an image of the ball B, which is an object OB, without applying any visual effects. Furthermore, the display control unit 34 controls the display unit 211 to display an object image IM1 including an image of the first player PL1, which is a predetermined object OB. Furthermore, the display control unit 34 controls the display unit 212 to display an object image IM2 including an image of the second player PL2, which is a predetermined object OB. As a result, as shown in FIG. 6B, the observer observes a virtual image VI1 of the first player PL1 on the near side, and a virtual image VI2 of the second player PL2 and a virtual image VIb of the ball B on the far side. Therefore, the observer can observe the virtual image VI1 of the first player PL1, the virtual image VI2 of the second player PL2, and the virtual image VIb of the ball B in a positional relationship similar to the positional relationship of each object OB in real space.

[0039] In the example shown in Fig. 7A, similar to the example shown in Fig. 6A, the object image includes images of the first player PL1, the second player PL2, and the ball B. Furthermore, in the real space, the distance between the first reference plane S1 and the second reference plane S2 is 9. Furthermore, in the real space, the ball B, which is one of the objects, is located between the first reference plane S1 and the second reference plane S2, and unlike the example shown in Fig. 6A, the distance between the first reference plane S1 and the ball B is 7.

[0040] In this example, the distance from the first reference plane S1 to the ball B is 7, and the distance from the second reference plane S2 to the ball B is 2. Therefore, the determination unit 33 determines the display unit 212, which displays the virtual image VI on the virtual image plane VF2 corresponding to the second reference plane S2 that has the shortest distance to the ball B, as the display unit 21 that displays the object video including the image of the ball B.

[0041] Then, the determination unit 33 determines the visual effect "enlargement / reduction" stored in the visual effect storage unit 32 as the visual effect to be added to the object OB in accordance with the distance 2 from the second reference plane S2 to the ball B, which is the object OB. As an example, the determination unit 33 determines to add a visual effect such that the object image including the image of the ball B is reduced as the distance between the ball B and the second reference plane S2 becomes shorter, and the object image including the image of the ball B is enlarged as the distance becomes longer. Note that, when the determination unit 33 determines the display unit 211 that displays the virtual image on the first reference plane S1 as the display unit 21 that displays the object image including the image of the ball B, and the distance between the ball B and the first reference plane S1 is within a predetermined range, the determination unit 33 may determine to add a visual effect such that the object image is enlarged as the distance between the ball B and the first reference plane S1 becomes shorter, and the object image is reduced as the distance becomes longer.

[0042] Accordingly, the display control unit 34 controls the display unit 212 to display an image to which a visual effect of enlarging or reducing the image of the ball B is added. The visual effect of enlarging or reducing the image of the ball B may be, for example, a visual effect in which the image of the ball B is enlarged as the ball B approaches the first player PL1, and the image of the ball B is reduced as the ball B moves away from the first player PL1. Furthermore, the display control unit 34 controls the display unit 211 to display an object image IM1 including an image of the first player PL1, which is a predetermined object OB, and controls the display unit 212 to display an object image IM2 including an image of the second player PL2, which is a predetermined object OB. As a result, the observer observes a virtual image VI1 of the first player PL1 on the near side, and a virtual image VI2 of the second player PL2 and a virtual image VIb of the ball B on the far side, as shown in FIG. 7B. Therefore, the observer can observe the virtual images of the first player PL1, the second player PL2, and the ball B in a positional relationship similar to their respective positional relationships in real space. Furthermore, since the virtual image display unit 2 makes the image of the ball B smaller or larger according to changes in distance, the observer can recognize the movement of the ball in real space with a more realistic feeling.

[0043] In the example shown in Fig. 8A, similarly to the examples shown in Fig. 6A and Fig. 7A, the object image includes images of the first player PL1, the second player PL2, and the ball B. In addition, in the real space, the distance between the first reference plane S1 and the second reference plane S2 is 9. In addition, in the real space, the ball B, which is one of the objects, is located between the first reference plane S1 and the second reference plane S2, and unlike the examples shown in Fig. 6A and Fig. 7A, the distance between the first reference plane S1 and the ball B is 5.

[0044] In this example, the distance from the first reference plane S1 to the ball B is 5, and the distance from the second reference plane S2 to the ball B is 4. Therefore, the determination unit 33 determines that the display unit 21 that displays the object video including the image of the ball B is the display unit 212 that displays a virtual image on the virtual image plane VF2 that corresponds to the second reference plane S2 that is the shortest distance to the ball B, which is the object OB.

[0045] Then, the determination unit 33 determines that the visual effect to be applied to the object OB is a "flash of light", which is a visual effect stored in the visual effect storage unit 32, in accordance with the distance 2 from the second reference plane S2 to the ball B.

[0046] Accordingly, the display control unit 34 controls the display unit 212 to display an object image including an image of the ball B with a visual effect (shaded portion in FIG. 8B) added such that the image of the ball B looks bright like a flash light. Furthermore, the display control unit 34 controls the display unit 211 to display an object image IM1 including an image of the first player PL1 who is a predetermined object OB, and controls the display unit 212 to display an object image IM2 including an image of the second player PL2 who is a predetermined object OB. As a result, as shown in FIG. 8B, the observer observes a virtual image VI1 of the first player PL1 on the near side, and a virtual image VI2 of the second player PL2 and a virtual image VIb of the ball B on the far side. Therefore, the observer can observe the virtual images of the first player PL1, the second player PL2, and the ball B in a positional relationship similar to the positional relationship in the real space. Furthermore, the observer can more strongly recognize that the ball B is approaching by observing the image of the ball B to which the visual effect of the flash light has been added.

[0047] <Operation of the spatial image display device> Here, the operation of the spatial image display device 1 according to the first embodiment will be described with reference to Fig. 9. Fig. 9 is a flow chart showing an example of the operation of the spatial image display device 1 according to the first embodiment. The operation of the spatial image display device 1 described with reference to Fig. 9 corresponds to an example of a method of displaying a spatial image of the spatial image display device 1 according to the first embodiment.

[0048] In step S11, the communication unit 31 receives object video information and object position information from an external device via a communication network.

[0049] In step S12, the determination unit 33 determines a visual effect to be added to an object image that is an image of an object included in the object image information, based on the position of the object that corresponds to the object that is a specific subject whose image is included in the image.

[0050] In step S13, the display control unit 34 determines, from among the multiple display units 21, the display unit 21 on which to display the object video including the image of the object OB.

[0051] In step S14, the display control unit 34 controls the display unit 21 to display the image based on the object image and the visual effect. Specifically, the display control unit 34 controls the display unit 21 determined in step S13 to display an image in which the visual effect determined by the determination unit 33 is added to the object image of the object OB indicated by the object image information. Furthermore, the display control unit 34 may control a predetermined display unit 21 to display an object image of a predetermined object OB indicated by the object image information.

[0052] In step S15, the display unit 21 displays the object image under the control of the display control unit 34. Specifically, the display unit 21 displays an image in which the visual effect determined by the determination unit 33 is added to the object image of the object OB indicated by the object image information. At this time, the display unit 21 may display the object image of a predetermined object.

[0053] As described above, according to the first embodiment, the aerial image display device 1 determines a visual effect to be added to an object image including an image of an object OB based on the position of the object, which is a specific subject whose image is included in the image, and controls the display unit 21 to display an image based on the object image and the visual effect. Therefore, the aerial image display device 1 can display virtual images of objects on multiple virtual image planes VF. Therefore, the aerial image display device 1 can provide a sufficient sense of presence to an observer who observes a spatial image formed by virtual images on multiple virtual image planes VF.

[0054] Furthermore, according to the first embodiment, the spatial image display device 1 receives object image information indicating an object image and object position information indicating the position of the object OB, and determines a visual effect to be added to the object image indicated by the object image information based on the position indicated by the object position information. Therefore, the spatial image display device 1 can display a virtual image in which a visual effect has been added to the object image included in the image, at a location remote from where the object OB is captured.

[0055] Moreover, according to the first embodiment, the spatial image display 1 determines a visual effect such as changing the luminance of each object image based on the distance. Therefore, the spatial image display 1 can display the virtual images VI of the moving object OB on a plurality of virtual image planes VF so that the observer can intuitively recognize the position of the object OB, thereby providing a sense of realism to the observer observing the stereoscopic image formed by the virtual images.

[0056] Furthermore, according to the first embodiment, the spatial image display device 1 determines, based on the distance, from among the multiple display units 21, the display unit 21 that is to display an image based on the object image, and controls the display unit 21 determined by the determination unit 33 to display an image in which a visual effect is added to the object image. Therefore, the spatial image display device 1 can display the virtual image VI of the moving object OB on multiple virtual image planes VF so that the observer can intuitively recognize the position of the object OB, thereby providing a greater sense of presence to the observer observing the stereoscopic image formed by the virtual images.

[0057] In the above-mentioned first embodiment, the spatial image display device 1 includes the virtual image display unit 2, but this is not limited to the above. For example, the spatial image display device 1 may not include the virtual image display unit 2, and may control an external virtual image display device to display an object image. In such a configuration, the spatial image display device 1 does not execute step S15 of the above-mentioned flowchart.

[0058] (First Modification) In the above embodiment, the position of the object OB is indicated by the distance from a reference plane to the object OB in real space, but this is not limited to this. For example, the position of the object OB may be indicated by the height of the object OB from a predetermined horizontal plane (for example, the ground plane) in real space.

[0059] In this configuration, the visual effect storage unit 32 stores heights and visual effects in association with each other, as shown in Fig. 10A. The determination unit 33 determines the visual effect based on the height. Furthermore, the display control unit 34 controls the display unit 21 to display an image in which the visual effect determined by the determination unit 33 is added to the object image.

[0060] In this example, when the height is 1.5, the determination unit 33 determines that there is "no" visual effect (see FIG. 10A). Then, the display control unit 34 controls the display unit 21 to display the object image without applying any visual effect. This allows the viewer to view a spatial image constituted by virtual images VI1, VI2, and VIb as shown in FIG. 10B.

[0061] Furthermore, when the height is 2.5, the determination unit 33 determines the visual effect to be "50% transparency" (see FIG. 10A), and controls the display unit 21 to change the transparency of the object image to 50% and display it. This allows the viewer to view a spatial image composed of virtual images VI1 and VI2 as shown in FIG. 10C, and a virtual image VIb of the object image to which a visual effect of 50% transparency has been applied.

[0062] Furthermore, if the height is 3 or greater, the determination unit 33 determines the visual effect to be "80% transparency" (see FIG. 10A), and controls the display unit 21 to change the transparency of the object image to 80% and display it.

[0063] (Second Modification) In the above-described embodiment, the determination unit 33 determines the visual effect based on the position of the object OB, but in this case, the determination unit 33 may determine the visual effect based on the rate of change in the position of the image of the object OB in the video. The rate of change in the position can be, for example, the change in the position of the image of the object OB in the unit number of frames constituting the video.

[0064] In such a configuration, the visual effect storage unit 32 stores the position change rate and the visual effect in association with each other, as shown in Fig. 11A. Furthermore, the display control unit 34 controls the display unit 21 to display an image in which the visual effect determined by the determination unit 33 is added to the object image.

[0065] In this example, when the rate of change of the position is less than the threshold, the determination unit 33 determines the visual effect to be "none" (see FIG. 11A). Then, the display control unit 34 controls the display unit 21 to display the object image without applying the visual effect. This allows the observer to observe a spatial image constituted by the virtual images VI1, VI2, and VIb as shown in FIG. 11B.

[0066] Furthermore, if the rate of change in position is equal to or greater than the threshold, the determination unit 33 determines the visual effect to be "superimposing multiple frames" (see FIG. 11A). Then, the display control unit 34 controls the display unit 21 to display an image with a visual effect of superimposing and displaying multiple frames of object images. This allows the observer to observe a spatial image constituted by virtual images VI1 and VI2 as shown in FIG. 11C, and a virtual image VIb of an image displayed by superimposing the object images of multiple frames.

[0067] Furthermore, if the rate of change in position cannot be determined, the determination unit 33 determines the visual effect to be "50% transparency." The display control unit 34 then controls the display unit 21 to change the transparency of the object video to 50% and display it.

[0068] (Third Modification) Furthermore, the determination unit 33 may determine the visual effect to be added to the object image based on whether or not the position of the object OB is within a predetermined range.

[0069] For example, as shown in Fig. 12A, the visual effect storage unit 32 stores whether or not the position of the object OB is within a predetermined range in association with a visual effect. In the example shown in Fig. 12A, the area inside and on the lines of a court used in a tennis match is within the predetermined range (IN), and the area outside the lines is outside the predetermined range (OUT). Furthermore, the display control unit 34 controls the display unit 21 to display an image in which the visual effect determined by the determination unit 33 is added to the object image.

[0070] In this example, when the position of the object OB in the real space is within a predetermined range, the determination unit 33 determines that the visual effect is "none." Then, the display control unit 34 controls the virtual image display unit 2 to display the object image without applying any visual effect. This allows the observer to observe a spatial image constituted by the virtual images VI1, VI2, and VIb as shown in FIG. 12B.

[0071] Furthermore, if the position of the object OB in the real space is outside a predetermined range, the determination unit 33 determines the visual effect to be "flash light." Then, the display control unit 34 controls the virtual image display unit 2 to display the object OB with a visual effect that makes the image of the object OB look bright like a flash light. This allows the observer to observe a spatial image composed of the virtual images VI1 and VI2 as shown in FIG. 12C, and the virtual image VIb of the image to which the visual effect that makes the image look bright like a flash light has been added.

[0072] Furthermore, when it is unclear whether the position of the object OB in the real space is within a predetermined range, the determination unit 33 determines the visual effect to be "50% transparency." Then, the display control unit 34 controls the virtual image display unit 2 to change the transparency of the object image to 50% and display it.

[0073] In the third modified example, the determination unit 33 determined the visual effect based on whether or not the position of the object OB in real space is within a predetermined range, but the determination unit 33 may also determine the visual effect based on whether or not the image of the object OB in the video space is within a predetermined range.

[0074] (Fourth Modification) In the above embodiment, the determination unit 33 determines the display unit 21 on which the object image is to be displayed based on the position of the object OB, but the present invention is not limited to this. For example, the determination unit 33 may determine that the object image is to be displayed on each of the multiple display units 21. In such a configuration, the position is indicated by the distance in a predetermined direction from a reference plane, which is a plane in real space corresponding to the virtual image plane on which the image is displayed, to the object, and the determination unit 33 determines a visual effect such as changing the luminance of the object image to be displayed on each of the multiple display units 21 based on the distance. Specifically, the determination unit 33 changes the luminance of the object image to be displayed on each display unit 21 for each display unit 21 according to the position of the object OB so that the observer can observe the spatial image with a higher sense of stereoscopicity.

[0075] (Fifth Modification) In the above-described embodiment, the determination unit 33 may further determine the irradiation light to be irradiated to a member defining the space in which the virtual image is displayed. The member defining the space in which the virtual image is displayed may be, for example, a floor, a wall, or a pillar. The irradiation light is visible light, and the determination unit 33 may determine, for example, the color, intensity, etc. of the visible light, or may determine a projection image formed by the visible light according to the irradiation position. In such a configuration, the display control unit 34 controls the irradiation device to irradiate the irradiation light determined by the determination unit 33. The irradiation device may be a lighting device, a projection projection device, etc.

[0076] For example, in the example of a video of a tennis match as described above, when the ball B is positioned outside the lines on the court, the determination unit 33 determines to irradiate the actual court with white light having a higher intensity than that before the ball B was positioned outside the lines on the court. Then, the display control unit 34 controls the irradiation device to irradiate the white light onto the court. In such an example, even if the observer cannot clearly perceive whether the ball B is positioned inside or on the lines on the court or outside the lines on the court due to, for example, the fast moving speed of the actual ball B, the observer can instantly recognize the range on the court where the ball B is positioned by the irradiation of the white light. Therefore, the observer can grasp the content of the match without missing the timing.

[0077] <<Second embodiment>> The overall configuration of the second embodiment will be described with reference to Fig. 13. Fig. 13 is a schematic diagram of a spatial image display device 1-1 according to the second embodiment. In the second embodiment, the same reference numerals are attached to the same functional units as those in the first embodiment, and the description thereof will be omitted.

[0078] As shown in FIG. 13, a spatial image display device 1-1 according to the second embodiment includes a virtual image display section 2, a visual effect determination section 3-1, and an image processing section 4-1.

[0079] <Video Processing Unit Configuration> The video processing unit 4-1 includes an input unit 41 and an object extraction unit 42. The input unit 41 is configured with an input interface that accepts input of information. The object extraction unit 42 constitutes a control unit.

[0080] The input unit 41 accepts input of video information indicating an image generated by an imaging device.

[0081] The object extraction unit 42 extracts an image of the object OB from the image information received by the input unit 41. The object extraction unit 42 may extract the object image using any method.

[0082] <Configuration of visual effects determination unit> As shown in FIG. 1, the visual effect determination unit 3-1 includes a communication unit 31-1, a visual effect storage unit 32, a determination unit 33-1, and a display control unit .

[0083] The communication unit 31-1 receives object position information. Specifically, the communication unit 31-1 receives object position information from an external device via a communication network.

[0084] The determination unit 33-1 determines a visual effect to be added to an object image including an image of the object OB based on the position of the object OB. Specifically, the determination unit 33-1 determines a visual effect to be added to an object image extracted by the object extraction unit 42 based on the position of the object OB indicated by the object position information received by the communication unit 31-1. A specific method in which the determination unit 33-1 determines a visual effect is similar to the specific method in which the determination unit 33 in the first embodiment determines a visual effect described above.

[0085] <Operation of the spatial image display device> Here, the operation of the spatial image display device 1-1 according to the second embodiment will be described with reference to Fig. 14. Fig. 14 is a flow chart showing an example of the operation of the spatial image display device 1-1 according to the second embodiment. The operation of the spatial image display device 1-1 described with reference to Fig. 14 corresponds to an example of a method of displaying a spatial image of the spatial image display device 1-1 according to the second embodiment.

[0086] In step S21, the input unit 41 accepts input of video information indicating an image generated by an imaging device.

[0087] In step S22, the object extraction unit 42 extracts object video information indicating an object video including an image of the object OB from the video information.

[0088] In step S23, the communication unit 31-1 receives object position information from an external device via the communication network.

[0089] Next, the spatial image display device 1-1 executes the processes from step S24 to step S27. The processes from step S25 to step S28 are the same as the processes from step S12 to step S15 in the first embodiment.

[0090] In the second embodiment, the visual effect determination unit 3-1 and the video processing unit 4-1 may be configured as separate entities. In such a configuration, the video processing unit 4-1 has a communication unit configured by a communication interface, and the communication unit transmits object video information indicating the object video extracted by the object extraction unit 42 to the communication unit 31-1 of the visual effect determination unit 3-1.

[0091] Moreover, the first to fifth modified examples of the first embodiment described above may be applied to the second embodiment.

[0092] <<Third embodiment>> The overall configuration of the third embodiment will be described with reference to Fig. 15. Fig. 15 is a schematic diagram of a spatial image display device 1-2 according to the third embodiment. The same functional units as those in the second embodiment are given the same reference numerals, and the description thereof will be omitted.

[0093] As shown in FIG. 15, a spatial image display device 1-2 according to the second embodiment includes a virtual image display section 2, a visual effect determination section 3-2, and a video processing section 4-2.

[0094] <Video Processing Unit Configuration> The video processing unit 4-2 includes an input unit 41, an object extraction unit 42, and an object position estimation unit 43.

[0095] The object position estimation unit 43 estimates the position of the object OB. The method by which the object position estimation unit 43 estimates the position of the object OB may be any method. For example, the object position estimation unit 43 may estimate the distance from a reference plane in real space to the object OB using deep learning that indicates the position of the object OB based on the video information input by the input unit 41.

[0096] <Configuration of visual effects determination unit> As shown in FIG. 15, the visual effect determination unit 3-2 includes a visual effect storage unit 32, a determination unit 33-2, and a display control unit .

[0097] The determination unit 33-2 determines a visual effect to be added to an object image including an image of the object OB based on the position of the object OB. Specifically, the determination unit 33-2 determines a visual effect to be added to an object image extracted by the object extraction unit 42 based on the position estimated by the object position estimation unit 43. A specific method in which the determination unit 33-2 determines the visual effect is similar to the specific method in which the determination unit 33 in the first embodiment determines the visual effect described above.

[0098] <Operation of the spatial image display device> Here, the operation of the spatial image display device 1-2 according to the third embodiment will be described with reference to Fig. 16. Fig. 16 is a sequence diagram showing an example of the operation of the spatial image display device 1-2 according to the third embodiment. The operation of the spatial image display device 1-2 described with reference to Fig. 16 corresponds to an example of a method of displaying a spatial image of the spatial image display device 1-2 according to the third embodiment.

[0099] In step S31, the input unit 41 accepts input of video information indicating an image generated by an imaging device such as a camera.

[0100] In step S32, the object extraction unit 42 extracts object video information indicating the video of the object OB from the video information.

[0101] In step S33, the object position estimation unit 43 estimates an object position indicating the position of the object OB.

[0102] Next, the spatial image display device 1-2 executes the processes from step S34 to step S37. The processes from step S34 to step S37 are the same as the processes from step S12 to step S15 in the first embodiment.

[0103] The first to fifth modified examples of the first embodiment described above may be applied to the third embodiment. In a configuration in which the first modified example is applied, the object position estimation unit 43 estimates the height of the object OB from a predetermined horizontal plane in real space, which indicates the position of the object OB. In a configuration in which the second modified example is applied, the object position estimation unit 43 estimates the rate of change of the position of the image of the object OB in the video. In a configuration in which the third modified example is applied, the object estimation unit 44 estimates whether the position of the object OB in real space is within a predetermined range.

[0104] <Program> The above-mentioned determination units 33, 33-1, 33-2, and display control unit 34 can be realized by a computer 100. A program for causing the determination units 33, 33-1, 33-2, and display control unit 34 to function may be provided. The program may be stored in a storage medium or provided through a network. FIG. 17 is a block diagram showing a schematic configuration of a computer 100 functioning as each of the determination units 33, 33-1, 33-2, and display control unit 34. Here, the computer 100 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. The program instructions may be program code, code segments, or the like for performing required tasks.

[0105] 17, computer 100 includes processor 110, read only memory (ROM) 120, random access memory (RAM) 130, storage 140, input unit 150, display unit 160, and communication interface (I / F) 170. Each component is connected to be able to communicate with each other via bus 180. Processor 110 is specifically a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a digital signal processor (DSP), a system on a chip (SoC), etc., and may be composed of multiple processors of the same type or different types.

[0106] The processor 110 controls each component and executes various arithmetic processing. That is, the processor 110 reads a program from the ROM 120 or the storage 140, and executes the program using the RAM 130 as a working area. The processor 110 controls each component and executes various arithmetic processing according to the program stored in the ROM 120 or the storage 140. In the above-described embodiment, the program according to the present disclosure is stored in the ROM 120 or the storage 140.

[0107] The program may be stored in a storage medium readable by the computer 100. By using such a storage medium, the program can be installed in the computer 100. Here, the storage medium in which the program is stored may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be in a form that is downloaded from an external device via a network.

[0108] The ROM 120 stores various programs and various data. The RAM 130 temporarily stores programs or data as a working area. The storage 140 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0109] The input unit 150 includes one or more input interfaces that receive an input operation from a user and acquire information based on the user's operation. For example, the input unit 150 is a pointing device, a keyboard, a mouse, etc., but is not limited to these.

[0110] The display unit 160 includes one or more output interfaces that output information. For example, the display unit 160 is a display that outputs information as a video or a speaker that outputs information as a sound, but is not limited to these. Note that, if the display unit 160 is a touch panel type display, it also functions as the input unit 150.

[0111] The communication interface (I / F) 170 is an interface for communicating with an external device.

[0112] The following supplementary notes are further disclosed regarding the above embodiment.

[0113] (Additional note 1) A plurality of display units for displaying images; a plurality of optical elements arranged to reflect image light emitted from the images displayed on the plurality of display units, respectively, and to display virtual images of the images at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eye; A control unit, The control unit is determining a visual effect to be added to an object image including an image of a specific object based on a position of the object, the image being a specific subject, the image of which is included in the object image; A spatial image display device that controls the display unit to display the image based on the object image and the visual effect. (Additional note 2) a communication unit for receiving object image information indicating the object image and object position information indicating a position of the object, The spatial image display device according to claim 1, wherein the control unit determines the visual effect to be added to the object image indicated by the object image information based on the position indicated by the object position information. (Additional note 3) The control unit is Estimating the position of the object; 3. The spatial image display device according to claim 1 or 2, wherein the visual effect is determined based on the position. (Additional note 4) the position is indicated by a distance from a reference plane, which is a plane in real space corresponding to a virtual image plane on which the virtual image is displayed, to the object; The control unit is determining a display unit on which to display the object image from among the plurality of display units based on the distance; 4. The spatial image display device according to any one of claims 1 to 3, wherein the display unit determined by the determination unit displays an image to which the visual effect has been applied to the object image. (Additional note 5) The position is indicated by a height of the object from a predetermined horizontal plane in real space; 5. The spatial image display device according to any one of claims 1 to 4, wherein the control unit determines the visual effect based on the height. (Additional note 6) 6. The spatial image display device according to any one of claims 1 to 5, wherein the control unit determines the visual effect based on a rate of change of the position. (Additional note 7) 7. The spatial image display device according to any one of claims 1 to 6, wherein the control unit determines the visual effect based on whether the position is within a predetermined range. (Additional note 8) the position is indicated by a distance in a predetermined direction from a reference plane, which is a plane in real space corresponding to a virtual image plane on which the virtual image is displayed, to the object; The spatial image display device according to claim 3, wherein the control unit determines the visual effect such that the luminance of the object images to be displayed on the multiple display units is changed, based on the distance. (Additional note 9) The control unit is determining an illumination light to be illuminated on a member defining a space in which the virtual image is to be displayed; 9. The spatial image display device according to any one of claims 1 to 8, further comprising: controlling an illumination device so as to illuminate the determined illumination light. (Additional note 10) 10. The spatial image display device according to any one of claims 1 to 9, wherein the control unit is compatible with multiple devices. (Additional note 11) A method for displaying an aerial image of a spatial image display device, comprising: a plurality of display units for displaying an image; and a plurality of optical elements arranged to reflect image light emitted from the images displayed on the plurality of display units, and to display virtual images of the images at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eyes, the method comprising: A step of determining a visual effect to be added to an object image including an image of a specific object based on a position of the object, the object being a specific subject, the image of which is included in the image; controlling the plurality of display units to display the image based on the object image and the visual effect; A method for displaying an aerial image comprising: (Additional note 12) A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing the program causing the computer to function as the determination unit and the display control unit described in any one of appended claims 1 to 10.

[0114] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0115] Although the above-mentioned embodiment has been described as a representative example, it is obvious to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, it is possible to combine multiple building blocks shown in the configuration diagram of the embodiment into one, or to divide one building block. [Explanation of symbols]

[0116] 1, 1-1, 1-2 Spatial image display device 2 Virtual image display unit 3, 3-1, 3-2 Visual Effects Division 4-1, 4-2 Video processing section 21, 211, 212 Display section 22, 221, 222 Optical elements 31, 31-1 Communications Department 32 Visual Effects Memory Department 33, 33-1, 33-2 Decision Section 34 Display control section 41 Input section 42 Object Extraction Unit 43 Object position estimation unit 100 Computers 110 Processor 120 ROM 130 RAM 140 Storage 150 Input section 160 Output section 170 Communication Interface 180 Bus

Claims

1. A plurality of display units for displaying images; a plurality of optical elements arranged to reflect image light emitted from the images displayed on the plurality of display units, respectively, and to display virtual images of the images at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eye; a determination unit which determines, based on a position of an object, which is a specific subject, whose image is included in the image, from among the plurality of display units, a display unit which displays a virtual image on a virtual image surface corresponding to a reference surface in real space corresponding to the virtual image surface on which the virtual image is displayed and on which the distance from the reference surface, which is a surface in real space corresponding to the display unit on which the object image is displayed, to the object, as a display unit which displays an object image including the image of the object, and which determines a visual effect to be added to the object image, based on the distance from the reference surface, which is a surface in real space corresponding to the display unit on which the object image is displayed, to the object; a display control unit that controls the display unit to display the image based on the object image and the visual effect; A spatial image display device comprising:

2. A plurality of display units for displaying images; a plurality of optical elements arranged to reflect image light emitted from the images displayed on the plurality of display units, respectively, and to display virtual images of the images at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eye; A determination unit that determines a visual effect to be added to an object image including an image of an object, the object being a specific subject, based on a position of the object that is included in the image; a display control unit that controls the display unit to display the image based on the object image and the visual effect; Equipped with The position is indicated by a height of the object from a predetermined horizontal plane in real space; The determination unit determines the visual effect based on the height.

3. A plurality of display units for displaying images; a plurality of optical elements arranged to reflect image light emitted from the images displayed on the plurality of display units, respectively, and to display virtual images of the images at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eye; A determination unit that determines a visual effect to be added to an object image including an image of an object, the object being a specific subject, based on a position of the object that is included in the image; a display control unit that controls the display unit to display the image based on the object image and the visual effect; Equipped with The determination unit determines the visual effect based on a rate of change of the position.

4. a communication unit for receiving object image information indicating the object image and object position information indicating a position of the object, The spatial image display device according to claim 1 , wherein the determination section determines the visual effect to be added to the object image indicated by the object image information, based on the position indicated by the object position information.

5. An object position estimation unit that estimates a position of the object, The spatial image display device according to claim 1 , wherein the determination section determines the visual effect based on the position estimated by the object position estimation section.

6. The spatial image display device according to claim 1 , wherein the determination section determines the visual effect based on whether the position is within a predetermined range.

7. the position is indicated by a distance in a predetermined direction from a reference plane, which is a plane in real space corresponding to a virtual image plane on which the virtual image is displayed, to the object; The spatial image display device according to claim 5 , wherein the determination section determines the visual effect such that luminance of each of the object images to be displayed on the plurality of display sections is changed based on the distance.

8. The determination unit determines illumination light to be irradiated onto a member defining a space in which the virtual image is displayed, The spatial image display device according to claim 1 , wherein the display control unit controls an illumination device to illuminate with the illumination light determined by the determination unit.

9. The spatial image display device according to claim 1 , wherein the display control unit is compatible with multiple devices.

10. A method for displaying an aerial image of a spatial image display device, comprising: a plurality of display units for displaying an image; and a plurality of optical elements arranged to reflect image light emitted from the images displayed on the plurality of display units, and to display virtual images of the images at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eyes, the method comprising: a step of determining, based on a position of an object, which is a specific subject whose image is included in the image, a display unit that displays a virtual image on a virtual image surface corresponding to a reference surface in real space that corresponds to the virtual image surface on which the virtual image is displayed and has the shortest distance from the reference surface to the object, from among the plurality of display units, as a display unit that displays an object image including the image of the object, and determining a visual effect to be added to the object image based on the distance from the reference surface, which is a surface in real space that corresponds to the display unit on which the object image is displayed, to the object; controlling the display unit to display the image based on the object image and the visual effect; A method for displaying an aerial image comprising:

11. A method for displaying a spatial image of a spatial image display device comprising a plurality of display units for displaying an image, and a plurality of optical elements arranged to reflect image light emitted from the images displayed on the plurality of display units, and to display virtual images of the images at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eyes, comprising: determining a visual effect to be added to an object image including an image of a specific object based on a position of the object, the object being a specific subject, the image of which is included in the image; controlling the display unit to display the image based on the object image and the visual effect; Including, The position is indicated by a height of the object from a predetermined horizontal plane in real space; A method for displaying aerial images, wherein the step of determining the visual effect determines the visual effect based on the height.

12. A method for displaying a spatial image of a spatial image display device comprising: a plurality of display units for displaying an image; and a plurality of optical elements arranged to reflect image light emitted from the image displayed on the plurality of display units, and to display a virtual image of the image at different distances in the same direction from the observer by causing the reflected image light to reach the observer's eye, comprising: determining a visual effect to be added to an object image including an image of a specific object based on a position of the object, the object being a specific subject, the image of which is included in the image; controlling the display unit to display the image based on the object image and the visual effect; Including, A method for displaying aerial images, wherein the step of determining the visual effect determines the visual effect based on a rate of change of the position.

13. A program for causing a computer to function as the determination unit and the display control unit according to any one of claims 1 to 9.

Images