Image Display System

Abstract

To superpose a virtual object on an object in a real space which is watched through a transmission type display device by a user to be displayed on condition that a relative position between a user's point of view and the transmission type display device is not fixed.SOLUTION: An image display system 1 comprises: a transmission type display device 30; a display device; and an image generation device 10. The image generation device comprises: a storage unit which stores position information and information of a three-dimensional shape of an object on a real space and a virtual object on a virtual space; and a processing unit which generates an image by projective transformation or perspective projection transformation on the basis of a viewpoint position, the position information, the three-dimensional shape information of the display device, and three-dimensional image data of the virtual object to be displayed on the display device so that the virtual object to be a view when the virtual object is viewed from the viewpoint position is superposed and displayed on the real space which is viewed through the display device when the display device is viewed from the viewpoint position 2A of a user 2.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technology for displaying a virtual object by superimposing it on an object in real space that is visible through a see-through display device. [Background technology]

[0002] Mixed reality (MR) is known as one of the technologies that superimposes real-space objects and virtual objects, and one of its devices is Hololens (registered trademark), a head-mounted display (HMD) developed by Microsoft Corporation (see Non-Patent Document 1). Hololens allows a user to see real space through the transparent eyepiece of the Hololens (HMD) worn on the head, while simultaneously superimposing a virtual CG object displayed in the eyepiece on the real space. Hololens is also equipped with a distance sensor, and while acquiring the position and shape of real-space objects in the external world of the HMD, it projects a virtual CG object, thereby accurately superimposing the real-space object and the virtual CG object. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] “Microsoft HoloLens 2”, Microsoft Corporation, [online], [Retrieved October 19, 2022], Internet<https: / / www.microsoft.com / ja-jp / hololens> Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned accurate superimposition of real-space objects and virtual CG objects by Hololens is possible only because the HMD as a display device is fixed to the user's head, that is, the positional relationship between the user's eyes and the image display device (eyepiece lens) is always fixed. To achieve a similar superimposition without fixing the display device to the user's head, it is necessary to accurately obtain changes in the positional relationship between the user's eyes and the image display device, while also simultaneously obtaining changes in the positional relationship between the image display device and real-space objects, and perform drawing calculations in accordance with both changes. To achieve this, a new methodology and the technology to realize it are required.

[0005] The main reason that technology for overlaying virtual CG objects on real-space objects without fixing a display device to the user's head has not yet become widespread is, simply put, that it is technically difficult to ensure the quality of the overlay display at a level that can be used by a general audience. In the case of Hololens, at least the relative positional relationship between the viewpoint (eye position) and the display device is fixed, so when overlaying a virtual object on a real-space object, all that is required is information about the positional relationship, distance, and shape from the display device to the real-space object. In other words, there is only one variable factor related to the relative position that involves large changes.

[0006] On the other hand, when performing a similar overlay display without fixing the display device to the user's head, in addition to acquiring information about the positional relationship, distance, and shape between the display device and the real-space object, it is necessary to reflect in the display on the display device conditions in which the distance and relative positional relationship between the user's viewpoint position and the display device vary greatly.In other words, there are now two variable factors related to the relative position that involve large changes, which increases the complexity, and errors accumulate for each variable factor, making it difficult to guarantee the quality of the overlay, and a specific methodology is required.

[0007] One objective of the present disclosure is to provide a methodology that enables virtual objects to be superimposed on real-space objects visible to a user through a see-through display device, under conditions where the relative position between the user's viewpoint and the see-through display device is not fixed. [Means for solving the problem]

[0008] An image display system according to one aspect of the present disclosure includes a transparent display device capable of displaying images, a memory unit that stores position information and three-dimensional shape information in a virtual space constructed on a predetermined reference coordinate system that is set by superimposing the coordinate system in real space for each of the display device, objects in real space surrounding the display device, and any virtual object, and a processing unit that generates an image and displays it on the display device by performing projective transformation or perspective projection transformation based on the viewpoint position, the position information and three-dimensional shape of the display device, and three-dimensional image data representing the virtual object, so that the virtual object is accurately superimposed on the real space seen through the display device when a user views the display device from a certain viewpoint position as it appears when viewing the virtual space from the viewpoint position. Effect of the Invention

[0009] According to one aspect of the present disclosure, it is possible to provide a technology that enables virtual objects to be accurately superimposed with high quality on real-space objects visible to a user through a transmissive display device, under conditions where the relative positions of the user and the transmissive display device are not fixed. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic configuration diagram of an image display system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic configuration diagram showing an example in which the image display system of the present embodiment is mounted on a vehicle. [Diagram 3]1A and 1B are diagrams for explaining how the image display system of the present embodiment displays an image according to a user's viewpoint position. [Figure 4] FIG. 11 is a conceptual diagram for explaining calculation of a viewpoint position. [Diagram 5] 1 is a block diagram showing a configuration of an image generating apparatus according to a first embodiment. [Figure 6] 4 is a flowchart showing the overall process by the image generating device of the first embodiment. [Figure 7] FIG. 13 is a diagram showing a display device in a modified example of the system according to the first embodiment, and an example of a display image on the display device. [Figure 8] FIG. 11 is a diagram showing a display device and the like in a system according to a second embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a display device in a modified example of the system according to the second embodiment, and an example of a display image on the display device. [Figure 10] FIG. 13 is a schematic diagram for explaining the configuration of a display device in a system according to a third embodiment. [Figure 11] 13 is a diagram illustrating a more detailed configuration of a display device according to a third embodiment and display operation control by the display device. FIG. [Figure 12] 13 is a diagram illustrating a more detailed configuration of a display device according to a third embodiment and display operation control by the display device. FIG. [Figure 13] 7 is a flowchart showing a detailed example of the display process in step S104 of FIG. 6 described in the first embodiment. [Figure 14] FIG. 13 is a diagram showing a display device in a modified example of the system according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [First embodiment] A first embodiment of the present disclosure will be described with reference to the drawings.

[0012] FIG. 1 is a schematic diagram of an image display system according to a first embodiment of the present disclosure. As shown in FIG. 1, an image display system (hereinafter, sometimes simply referred to as a "system") 1 of this embodiment mainly includes an image generating device 10 that generates a virtual object image to be presented to a user, a measuring device 20 that measures the viewpoint position of a user 2, a transmissive display device 30 that displays an image generated by the image generating device 10, and a real space information acquiring device 40 (hereinafter, sometimes simply referred to as an "acquisition device 40") that acquires environmental information (information on the position, shape, size, distance to an object, etc. of an object existing in the real world) that the user can see through the transmissive display device 30. As an example, the image generating device 10 is a computer that executes a software program with a processor, and also serves as a calculation unit (not shown) of the measuring device 20.

[0013] The transmissive display device 30 in this embodiment is a so-called "transmissive" display device, and can be configured, for example, by a transmissive organic EL (Electro Luminescence) display (transmissive OLED), a transmissive liquid crystal display (transmissive LCD), or a transmissive screen onto which an image is projected from an external projector or the like. The transmissive display device 30 of this embodiment configured by such a transmissive display is transparent or semi-transparent, and allows the user to see the real space environment on the opposite side through the transmissive display device 30, and by displaying an image on the transmissive display device 30, it is possible to provide a visual experience in which a virtual object represented by the displayed image exists in the real space seen through the transmissive display device 30. The transmissive display device 30 of this embodiment can be used as a device fixedly installed at any position, such as a window of a building, a window of a vehicle such as an automobile, a bus, a ship, or an airplane, or on an indoor table, and in this case, a virtual object can be superimposed and displayed on the outdoor or indoor real space scenery or landscape seen through the transmissive display device 30.

[0014] The measuring device 20 (sensor 22) is provided, for example, below or above the transmissive display device 30 located in front of the user 2, and measures the viewpoint position of the user 2 from in front of the user.

[0015] The measurement device 20 measures the viewpoint position 2A of the user 2 in a predetermined reference coordinate system while the user 2 is present within a range where the user 2 can see the transmissive display device 30. The viewpoint position 2A is a position corresponding to the position of the eyes. The specific viewpoint position 2A used in the processing is not particularly limited, but for example, the midpoint between the eyes of the user 2, the center point of the head, a position a predetermined distance inward from the center of the eyes, or the like can be used as the viewpoint position 2A.

[0016] In this embodiment, a use case in which the transmissive display device 30 is mainly installed on a window of a vehicle such as an automobile will be described using a world coordinate system that is fixed and defined in real space, and a local coordinate system that is defined with the vehicle as a reference. Both the world coordinate system and the local coordinate system are assumed to be Cartesian coordinate systems having three axes, xyz. If the position and direction of the vehicle in the world coordinate system are specified, the relationship between the world coordinate system and the local coordinate system is determined, and they can be converted into each other. It is arbitrary how the world coordinate system is set with respect to the real space. It is also arbitrary how the local coordinate system is set with respect to the vehicle.

[0017] FIG. 2 is a schematic diagram showing an example in which the image display system of the present embodiment is mounted on a vehicle. In the example shown in FIG. 2, an image display system 1 including an image generating device 10, a measuring device 20, a transmissive display device 30, and a real space information acquiring device 40 is mounted on a vehicle M, and provides a visual experience in which a real landscape and a virtual image are fused to a driver, a passenger, or other occupant (user) 2 of the vehicle M. As shown in FIG. 2, the vehicle M of the present embodiment is placed in a real space (real world) in which various real objects (hereinafter sometimes referred to as "real objects") RO exist, and is, for example, a small passenger vehicle for one person that runs in the real space. However, the vehicle M is not limited to a passenger vehicle, and may be a passenger car, a bus, a railroad vehicle, an airplane, a ship, a vehicle of an attraction facility, or the like.

[0018] 2 shows an orthogonal coordinate system having an X-axis pointing to the right of the user 2, a Y-axis pointing upward, and a Z-axis pointing backward, as an example of a local coordinate system based on the position and direction of the vehicle. The positions and attitudes of the sensor 22 (not shown in FIG. 2) in the measurement device 20 and the transmissive display device 30 are fixed in this local coordinate system. The attitude is represented by a pitch around the X-axis, a yaw around the Y-axis, and a roll around the Z-axis.

[0019] The image generating device 10 displays a virtual object VO in a virtual space on the transparent display device 30 based on the position and direction of the vehicle, the viewpoint position 2A of the user 2 in the vehicle, and the fixed position and attitude of the transparent display device 30. At this time, the virtual object VO is placed in a three-dimensional virtual space defined by a world coordinate system, and the world coordinate system of the real space and the world coordinate system of the virtual space are defined by being superimposed so that their positional relationship and enlargement / reduction ratio match. The user peeks into the real space through the transparent display device 30 from the viewpoint position 2A, but at the same time, he / she also peeks into the virtual space fixed to the same world coordinate system. At this time, in order to give the user 2 the illusion that the virtual object VO in the virtual space exists simultaneously on the real space, that is, to make it look as if the virtual object VO exists in the real space, a calculation process is performed to generate an image to be displayed on the display screen. Specifically, three-dimensional CG data in a virtual space defined three-dimensionally based on the coordinates of the viewpoint position 2A is subjected to projective transformation, perspective projection transformation, or similar calculation processing onto a two-dimensional display screen to generate a two-dimensional image, which is then displayed on the transmissive display device 30. As a result, the three-dimensional information in the virtual space is converted into two-dimensional information with geometrical precision and presented to the user 2.

[0020] The image generating device 10 stores three-dimensional CG data representing a virtual object to be superimposed on the real space seen through the transparent display device 30, for example, data of a cheetah drawn in three-dimensional CG, in advance in an internal memory unit 12 (see Figure 5) together with placement information in the virtual space, and generates a display image through calculation processing to appropriately display the three-dimensional CG on the transparent display device 30 based on the viewpoint position 2A so that the image appears realistic and natural with geometrical precision to give the user 2 the illusion that the virtual object exists in real space.

[0021] In the present specification and claims, the term "image" is used to include not only one or more still images, but also a moving image (video) made up of a plurality of images that are successive in time series.

[0022] According to this embodiment, as an example, an image in which a virtual object is superimposed on a view of real space seen from a vehicle window is generated according to the viewpoint position 2A of the user 2 and displayed on the transmissive display device 30, so that the user 2 can be given a sensation as if the virtual object is placed together with real-world objects in real space in the real world and exists simultaneously. In addition, this embodiment adopts a method of displaying an image on a screen fixed to a position other than the user's body part including the user's head, rather than a head-mounted display (HMD) that moves with the user's head, so that the display output unit 16 in the image generating device 10 can be used as a display device in mixed reality (MR) technology, and can allow the user to experience a realistic mixed reality in the same way as a head-mounted display, without giving the user 2 any inconvenience or discomfort due to the use of a head-mounted display.

[0023] The image display system 1 of this embodiment will be described in further detail below.

[0024] As an example, the measuring device 20 continuously measures the viewpoint position 2A, and the image generating device 10 generates a display image to follow the viewpoint position 2A and displays it on the screen of the transmissive display device 30. As a result, when the head of the user 2 moves, the display image displayed on the transmissive display device 30 changes to follow the viewpoint position 2A that moves with the head of the user 2, so that the user 2 can feel as if a virtual object exists in real space. The image generation associated with the tracking of the viewpoint position 2A is realized by continuously generating images that are the appearance of a virtual object in the virtual space when viewed through the display device from the viewpoint position 2A at that time. The method of generating the image at this time is as described above. In addition, although the description of tracking is given here, it is not limited to this, and the image may be generated by predicting the viewpoint position 2A a little ahead. When the viewpoint position 2A moves, if an image is generated based on the current position of 2A and the result is displayed, a delay occurs in the image display due to the time required for calculation, which makes the user feel uncomfortable. To solve this problem, a method may be used in which the position a little further in advance is predicted while taking time series data of the viewpoint position 2A, and image generation is performed in advance. Although the method of predicting the viewpoint position has been described here, prediction is not limited to the viewpoint position, and predictive calculations may also be used for the relative position of the local coordinate system with respect to the world coordinate system. For example, assuming a car as the vehicle, the position of the vehicle a little further in the future can be predicted with high accuracy based on the direction of travel, speed information, and map information. The prediction results may be reflected in the image generation described above.

[0025] FIG. 3 is a diagram for explaining a state in which the image display system according to the present embodiment displays an image according to a user's viewpoint position. In the explanation of FIG. 3, the user space on the front side of the transmissive display device 30 is defined as a real space, and the space on the back side is defined as a virtual space, based on the user's viewpoint position. The virtual space defined on the back side of the transmissive display device 30 is displayed on the transmissive display device 30 as an image seen from a position P1 where the head of the user 2 is located through a pseudo window (hereinafter also referred to as a "pseudo window") by the transmissive display device 30. The virtual object in the virtual space is defined by three-dimensional CG data to be described later. In the example of FIG. 3, six trees are arranged side by side as virtual objects. Note that here, the virtual space is defined on the back side of the transmissive display device 30 in FIG. 3 for the sake of explanation, but the virtual space may be defined on the front side of the transmissive display device 30, or the space including the front side and the back side of the transmissive display device 30 may be defined as the virtual space. This makes it possible to display on the transparent display device 30 not only an image in which a virtual object in the virtual space appears to be on the other side of a virtual window created by the transparent display device 30, but also an image in which a virtual object in the virtual space appears to be protruding in front of the transparent display device 30.

[0026] When the user 2 is at a position P1 in front of the transmissive display device 30, the field of view FoV1 of the virtual space seen through the pseudo window created by the transmissive display device 30 is wide, and all six trees are displayed on the transmissive display device 30 as if they are included in the field of view FoV1 (display D1). When the user 2 moves from the position P1 in the z direction to a position P2 away from the transmissive display device 30, the field of view FoV2 of the virtual space seen through the pseudo window created by the transmissive display device 30 narrows, and only the entire three trees and parts of the trees on both sides of them are displayed in the field of view FoV2 (display D2). When the user 2 moves from the position P1 in the -x (negative x) direction to a position P3, the field of view FoV3 of the virtual space seen through the pseudo window created by the transmissive display device 30 changes in the x direction. Only the three trees on the right edge are included in the field of view FoV3. In addition, in the field of view FoV3, the screen is viewed from an oblique direction rather than from the front of the transparent display device 30, but the horizontal thickness of the tree seen through the pseudo window needs to be the same as when viewed from the front (display D3'). For this reason, when displaying the tree on the transparent display device 30, an image that has been appropriately stretched so that it looks like display D3' to the user is displayed (display D3). In this way, as a process that makes the image look plausible and natural so that the user 2 has the illusion that the virtual object in the virtual space exists there, in this embodiment, when generating an image to be displayed on the transparent display device 30, a process (for example, a projective transformation, a perspective projection transformation, or a calculation similar thereto) is performed to project the virtual object in the virtual space defined in the three-dimensional CG data onto the transparent display device 30, that is, onto a two-dimensional surface. As another method, each point of the three-dimensional CG data may be projected onto a point where a straight line connecting each point and the user's viewpoint position intersects with the transparent display device 30 in the reference coordinate space. As another processing method for generating an image to be displayed on the transmissive display device 30, arithmetic operations of specific matrices or values ​​based on empirical rules may be performed on the image or three-dimensional parameters of the image.

[0027] Measuring device 20 of this embodiment is configured to have an imaging section that images user 2, and a calculation section that determines viewpoint position 2A based on imaging information of user 2 imaged by the imaging section. The imaging section is realized by sensor 22 installed in measuring device 20 shown in Fig. 1. The calculation section is realized by processing section 14 (see Fig. 5) of image generating device 10 shown in Fig. 1. Alternatively, the calculation section may be realized by a processing section (not shown) within measuring device 20.

[0028] As an example, in this embodiment, the sensor 22 (imaging unit) is a depth sensor that measures the depth from the sensor 22 to an object (here, the body of the user 2) at each pixel. The calculation unit realized by the processing unit 14 of the image generating device 10 estimates the shape of the human body based on the depth of each pixel measured by the sensor 22, and calculates the viewpoint position 2A based on the position of the head in the body. In this way, since the human body shape is estimated from the depth of each pixel and the head position in the body shape is used, the viewpoint position 2A can be identified with high accuracy even if the position and orientation of the body of the user 2 changes in various ways.

[0029] FIG. 4 is a conceptual diagram for explaining calculation of the viewpoint position. The position (X s ,Y s ,Z s ) and Pitch s ,Yaw s ,Roll s ) are set in advance. The coordinates (X h ,Y h ,Z h ) can be calculated. As shown in FIG. 4, the position (X m ,Y m ,Z m ), posture (Pitch m ,Yaw m ,Roll m ), and shape (Height m ,Width mIn this embodiment, as an example, each of the transmissive display devices 30 is rectangular or trapezoidal, and the shape of each of the transmissive display devices 30 is also set in advance. m ) and Width m The transmissive display device 30 may have any polygonal shape, or may have a curved or spherical surface.

[0030] The image captured by the sensor 22 may be a depth image, or may be a depth image and a visible image. For example, the imaging unit may include a sensor that captures a depth image and a camera that captures a visible image, and the calculation unit may calculate the viewpoint position 2A using both the depth image and the visible image. Alternatively, two visible images captured at different positions may be used. In this case, the viewpoint position 2A may be calculated using the parallax between the two visible images. Alternatively, the viewpoint position 2A may be estimated from a single visible image by image processing using AI.

[0031] 1 and 4, the measuring device 20 (sensor 22) is installed above or below the transmissive display device 30 in front of the user 2, but the present invention is not limited to this. The measuring device 20 (sensor 22) can also be arranged behind the user 2 at a position that does not overlap with the transmissive display device 30. Alternatively, the measuring device 20 (sensor 22) can be installed behind the transmissive display device 30 (at the back or rear side) and passes through the display portion of the transmissive display device 30 to measure the user's viewpoint position. Alternatively, the measuring device 20 may be measured using an image captured by any of these multiple sensors, or a viewpoint position obtained by integrating information from multiple sensors may be used.

[0032] As described above, the real space information acquisition device 40 in this embodiment is a device that acquires environmental information (information on the position, shape, size, distance to an object, etc., of an object existing in the real world) that is visible to the user through the transmissive display device 30. The acquisition device 40 includes a LiDAR sensor, for example, and is capable of measuring the distance to an object and the shape of the object, etc., of an object existing in the real space around the transmissive display device 30 as seen from the user. Here, LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) is one of the remote sensing technologies that mainly uses laser light, and measures scattered light in response to pulsed laser irradiation to acquire information on the distance to a distant object, the shape of the object, etc. The acquisition device 40 uses such a LiDAR sensor to scan the real space around the transmissive display device 30 and acquires information on the distance to an object existing in the space, the shape of the object, etc.

[0033] The acquisition device 40 may include an imaging camera equipped with an imaging element that generates images (visible images and depth images) in addition to or instead of the LiDAR sensor described above. By capturing an image of the real space visible through the transmissive display device 30 with the imaging camera and analyzing the captured image using any image processing technology, information regarding the position, size, shape, and the like of an object included in the image can be acquired. Examples of objects included in the image include buildings such as buildings and structures such as bridges that exist in the real space, distant mountains, moving vehicles, living things, and the like.

[0034] The acquisition device 40 may further include a GPS (Global Positioning System) sensor. By including the GPS sensor in the acquisition device 40, position information on the position where the real space information is acquired by the LiDAR sensor or the imaging camera of the acquisition device 40 can also be acquired. This position information is, for example, position information in the world coordinate space of a vehicle or the like on which the system 1 including the acquisition device 40 is mounted.

[0035] The acquisition device 40 may be configured to specify the position and direction of the vehicle using a visual positioning system (VPS). In this case, the acquisition device 40 captures an image of the real space outside the vehicle, and the processing unit 14 (see FIG. 5) of the image generation device 10 specifies the position and direction of the vehicle based on the image. For this purpose, many images whose captured positions and directions are specified are acquired in advance, visual features such as the contours of objects such as buildings appearing in those images are extracted, and the visual features are stored in a database of the storage unit 12 (see FIG. 5) of the image generation device 10 as searchable indexes together with information on the position and direction. Then, when an image captured by the acquisition device 40 is input while the vehicle is traveling, the processing unit 14 of the image generation device 10 extracts visual features from the acquired image and compares the extracted visual features with the visual features stored in the database to specify the position and direction at which the image was captured. The specified position and direction at which the image was captured are converted into the position and direction of the vehicle. This makes it possible to obtain the position and direction of the vehicle with high accuracy in real time. In this acquisition process, there is a possibility that a time delay occurs in acquiring the information in accordance with the actual movement of the vehicle. In order to solve this problem, the speed and acceleration vectors are calculated using the current and past positions and orientations of the vehicle, and the future position and orientation of the vehicle are predicted by adding vehicle body information (accelerator information, brake information, speed and steering angle information, etc.) that can be acquired directly from the vehicle without delay, thereby solving the problem of time delay.

[0036] In the above, an example has been described in which visual features are stored as searchable indexes in a database in the storage unit 12 (see FIG. 5) of the image generating device 10 together with information on position and direction, but the visual features may be stored in a database in an external device (not shown). In this case, image data acquired by the acquisition device 40 is transmitted from the acquisition device 40 to the external device. Then, the external device extracts visual features from the image and compares the extracted visual features with visual features stored in the database to identify the position and direction at which the image was captured. Information regarding the identified position and direction for the image is transmitted from the external device to the acquisition device 40.

[0037] Next, the image generating device 10 in this embodiment will be described. Fig. 5 is a block diagram showing the configuration of the image generating device in this embodiment. As shown in Fig. 5, the image generating device 10 has a storage unit 12, a processing unit 14, a display output unit 16, and a communication unit 18. In Fig. 5, the image generating device 10 is depicted as a single element, but the image generating device 10 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0038] The storage unit 12 includes a temporary or non-temporary storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), a SDD (Solid State Drive), etc. The storage unit 12 stores a computer program executed by the processing unit 14 and various data described later. The computer program stored in the non-temporary storage medium of the storage unit 12 includes instructions for carrying out each process of an image generating method by the processing unit 14, which will be described later with reference to FIG. 6 etc.

[0039] The storage unit 12 holds screen arrangement data indicating the position, attitude, and shape of the transmissive display device 30 in a predetermined reference coordinate space, measuring device / acquisition device arrangement data indicating the positions and attitudes of the measuring device 20 and the acquisition device 40 in the reference coordinate space, and three-dimensional CG data indicating a virtual object to be superimposed and displayed in the real space seen through the transmissive display device 30. The reference coordinate space is a space represented by a coordinate system (reference coordinate system) having a predetermined origin O that is the basis of the calculation in this embodiment, and is a space of an orthogonal coordinate system having three axes, xyz. How to set the reference coordinate space and its origin O is arbitrary. For example, the reference coordinate space may be fixed to the transmissive display device 30, may be fixed to the measuring device 20 and / or the acquisition device 40, may be fixed to the transmissive display device 30, the measuring device 20, and the acquisition device 40, or may not be fixed to any of them. In addition, when a vehicle is assumed, the reference coordinate space may be processed as a coordinate within the vehicle (local coordinate system) separately from the coordinate of the outside world (world coordinate system).

[0040] "Three-dimensional CG" is also called 3DCG because it is a virtual solid object in a three-dimensional space and differs from a two-dimensional image in that it contains depth (solid) information. One example of three-dimensional CG is a virtual solid object (modeling) composed of multiple faces created with vertices at points placed on the three-dimensional coordinates of the virtual space, and it may be expressed by providing information reproducing the material, etc., to each face and illuminating the object from an arbitrary light intensity, light source position, etc. The data of three-dimensional CG includes information regarding the placement position of the virtual object displayed in the three-dimensional CG in the virtual space. The storage unit 12 can store three-dimensional CG data related to various multiple virtual objects.

[0041] The processing unit 14 is configured with, for example, one or more central processing units (CPUs). The processing unit 14 may also include one or more graphics processing units (GPUs). The processing unit 14 executes a computer program stored in the storage unit 12 to perform each process of an image generating method described later with reference to FIG. 6 and the like.

[0042] The display output unit 16 is an output unit to which the transmissive display device 30 is connected and which outputs an image signal for displaying an image generated by the processing unit 14 on the transmissive display device 30. The display output unit 16 is configured with output terminals such as VGA, DVI, DisplayPort (trademark), HDMI (registered trademark), and USB (trademark) Type-C, and can be connected to the transmissive display device 30 by wire. Alternatively, the display output unit 16 may be configured to be wirelessly connected to the transmissive display device 30 using any wireless communication technology.

[0043] The communication unit 18 has a function of transmitting and receiving data between the image generating device 10 and an external device. In particular, in this embodiment, the communication unit 18 may transmit and receive data of an image captured by the sensor 22 of the measuring device 20 and data of information acquired by the acquisition device 40. The communication unit 18 may transmit image data acquired by the acquisition device 40 to the external device, or receive information on the position and direction at which the image was captured, which is specified in the external device, from the external device. In addition, the communication unit 18 may acquire only a necessary portion of data of a three-dimensional real space model that models an object existing in the real space through a network. Note that the communication between the external device and the communication unit 18 may be either wired communication or wireless communication.

[0044] Next, as an example, an image generating method by the image generating device 10 of this embodiment will be described with reference to Fig. 6 etc. Fig. 6 is an example of a flowchart showing the overall processing by the image generating device of this embodiment. Each process in Fig. 6 is executed by the processing unit 14. When the image generated by the image generating device 10 is a moving image consisting of a plurality of frame images that are successive in time series, calculations may be performed for each frame of the moving image. Furthermore, in the calculation process of each step of the flowchart, a predictive calculation may be performed using past data in time series to generate an image.

[0045] 6, the processing unit 14 of the image generating device 10 first obtains information about the real space that the user 2 sees through the transmissive display device 30 (step S101).

[0046] The processing unit 14 controls the operation of the acquisition device 40, and causes the acquisition device 40 to acquire information on the real space on the back side as seen from the user 2 of the transmissive display device 30. As an example of the specific process, the processing unit 14 first acquires GPS position information from the acquisition device 40, specifies the approximate position of the vehicle on which the system 1 including the acquisition device 40 is mounted, and further specifies VPS information (more accurate position and direction of the vehicle) from the acquisition device 40. Next, the processing unit 14 acquires data of a three-dimensional real space model in which an object existing in the real space is three-dimensionally modeled. The object existing in the real space is, for example, a building, a house, a bridge, a railway, a road, and other structures in the real world. The data of the three-dimensional real space model in which such an object is three-dimensionally modeled includes information on the position of each object in the real world and information on the shape and size of each object as information for reproducing the object in the virtual space. Currently, 3D city model data in which real spaces are three-dimensionally modeled for some cities and natural shapes are provided by a number of businesses and organizations. Known services that provide 3D city model data include, for example, "Project PLATEAU" by the Ministry of Land, Infrastructure, Transport and Tourism and "3D map data" by Zenrin Co., Ltd. In the future, it is expected that 3D model data for more cities and nature will be provided by various businesses and organizations. In this embodiment, as an example, the "three-dimensional real space model data" can be obtained by using a service provided by a business or organization. The communication unit 18 of the image generating device 10 connects to a service providing server of the business or organization via the Internet or the like, and downloads at least a necessary part of the three-dimensional real space model data from the server, thereby obtaining the three-dimensional real space model data. Alternatively, if the three-dimensional real space model data is already obtained in advance, the data can be used temporarily or permanently without connecting to the Internet. The obtained three-dimensional real space model data is stored in the storage unit 12.

[0047] In addition to or instead of the above processing, the processing unit 14 may acquire information on the real space on the far side as viewed from the user 2 of the transmissive display device 30 by a sensor (e.g., a LiDAR sensor) included in the acquisition device 40, and acquire information on the distance from the acquisition device 40 to the object and the size and shape of the object for an object existing in the real space. When there are multiple objects, information on the distance, size, shape, etc. of each object is acquired. When the acquisition device 40 is equipped with an imaging camera or a depth sensor, an image of the real space on the far side as viewed from the user 2 of the transmissive display device 30 may be captured by the imaging camera or acquired as a depth image. The processing unit 14 can acquire or estimate information on the position, shape, etc. of an object included in the image (e.g., buildings such as buildings existing in the real space, structures such as bridges, traffic lights, and telephone poles, distant mountains, etc.) by analyzing the acquired image of the real space using any known image analysis technology and / or artificial intelligence technology based on machine learning.

[0048] Next, the processing unit 14 acquires the viewpoint position 2A of the user 2 (step S102). The processing unit 14 calculates the viewpoint position 2A of the user 2 using the measurement device 20 by the method described with reference to FIG.

[0049] The processing of step S101 and the processing of step S102 by the processing unit 14 may be executed in a different order, or may be executed in parallel. Also, when the processing of the following steps is performed based on data acquired or estimated in the past in a time series, the necessary data has already been acquired without performing both processes, so this part may be skipped.

[0050] The processing unit 14 then performs processing to generate a display image on the transmissive display device 30, in which the virtual object is superimposed and displayed in the real space around the transmissive display device 30 as seen by the user 2 through the transmissive display device 30 (step S103).

[0051] The processing unit 14 continuously identifies the position and direction of the vehicle in the world coordinate system based on the real space environment information from the acquisition device 40, and also continuously identifies the viewpoint position 2A. The processing unit 14 performs the above-mentioned calculation process on the screen of the transparent display device 30 based on the viewpoint position 2A of the user 2 to display a display image that displays a virtual object so as to follow the position and direction of the vehicle and the viewpoint position 2A, thereby generating a display image with strict geometric accuracy that looks as if the virtual object exists together in the real world when viewed through the screen of the transparent display device 30 from the viewpoint position 2A of the user 2. When the display image is displayed on the transparent display device 30, it is an image that looks like a virtual object in the virtual space seen through the transparent display device 30 from the viewpoint position 2A. The display image displayed on the transparent display device 30 changes following the movement of the vehicle and the viewpoint position 2A that moves with the movement of the head of the user 2 inside the vehicle. This allows the user 2 to feel as if a virtual object exists in the real space. Tracking to the viewpoint position 2A is achieved by continuously generating images of the rendering contents as they appear when viewed from the viewpoint position 2A at that time.

[0052] By such a calculation process for converting three-dimensional information into two-dimensional information, the display image of the virtual object is projected onto the screen of the transmissive display device 30 with strict geometric accuracy taking into account the user's viewpoint position 2A, so that even when the user 2 looks into the screen of the transmissive display device 30 from an oblique direction, when the user 2 looks at the virtual object from the viewpoint position 2A, a display image is generated to be presented on the transmissive display device 30 as a natural image that seems to exist in reality at that position. The process performed here is not limited to projective transformation. A perspective projection transformation method, a calculation method similar to the above, or arithmetic processing of specific matrices and values ​​according to empirical rules may be used.

[0053] In addition, the virtual object may be displayed on the transmissive display device 30 so as to be positioned at a position where it appears to overlap partially or completely with a real-space object existing in the real space surrounding the transmissive display device 30 when viewed from a certain viewpoint position 2A. In this case, the virtual object is reflected in the display after a process of blocking partially or completely the virtual object is performed based on three-dimensional real-space model data corresponding to the object in the real space, or three-dimensional data of the object constructed based on the distance to the object and the shape acquired in real time by the real-space information acquisition device 40, thereby making it possible to accurately represent the overlap between the two.

[0054] The process of occluding a part or all of a virtual object includes a process of making transparent (or deleting, erasing, or making invisible) a region of the virtual object that is occluded by the real-space object when a part or all of a real-space object exists in front of the virtual object when the virtual object is viewed from a certain viewpoint position 2A. More specifically, when the distance from the viewpoint position 2A to a part constituting the virtual object is farther than the distance from the viewpoint position 2A to the part constituting the real-space object, a display image of the virtual object is generated in which a part corresponding to the occluded region occluded by the part constituting the real-space object is occluded.

[0055] The processing unit 14 transmits the signal of the display image thus generated to the transmissive display device 30 via the display output unit 16 of the image generating device 10, and causes the transmissive display device 30 to display the display image (step S104).

[0056] In this way, according to the system 1 of the present embodiment, display images of virtual objects displayed in three-dimensional CG and superimposed on the real space seen through the transmissive display device 30 are continuously generated following changes in the viewpoint position 2A of the user 2, and composite realistic moving images are provided to the user 2 through the transmissive display device 30. By superimposing and expressing the three-dimensional CG virtual objects on the scenery of the real space, the user 2 can be given the illusion that the virtual objects exist together in the real space of the real world seen through the transmissive display device 30.

[0057] In particular, according to this embodiment, not only the position information but also the information on the size, shape, etc., of an object in real space behind the transmissive display device 30 as seen from the user is acquired, so that it is possible to simulate the front-to-back relationship and the relative positional relationship when a virtual object to be superimposed and displayed on an object in real space is virtually arranged in the same space, and it is possible to generate an image in a form that reflects the expression including the expression by the occlusion process in which the virtual object is partially visible or hidden when the user's viewpoint position is used as the starting point. As a result, since the virtual object can be superimposed and displayed with higher quality on the object in real space, it is possible to provide the user 2 with a more realistic and natural image to give the user 2 the illusion that the virtual object exists in real space.

[0058] In the above, as one application example of the system 1 according to this embodiment, a use case has been described in which the transparent display device 30 is used as a window of a vehicle, and a virtual object is superimposed on the outside scenery seen through the window. However, the application of the system 1 is not limited to this example, and the system 1 can be applied to any other purpose.

[0059] As an example of other uses of the system 1, a use case in which the transparent display device 30 is used as a window of a building and a virtual object is superimposed on the outside scenery seen through the window, or a use case in which the transparent display device 30 is installed on an indoor table or the like and a virtual object is superimposed on a real space object (such as a shelf in the room) in the room seen through the transparent display device 30, can be considered. In these use cases, the system 1 including the transparent display device 30 is fixed and does not move in the world coordinate space. Therefore, the local coordinate system of the system 1 is fixed in the world coordinate system. Therefore, in these use cases, the GPS sensor and VPS-related functions for identifying the position of the system 1 in the local coordinate system may be omitted from the acquisition device 40 of the present embodiment described above.

[0060] In these use cases, the relative position between the transparent display device 30 and the shape of a real-space object of the scenery (buildings outside a window, etc.) or the scenery inside the room (shelves, etc.) seen through the transparent display device 30 does not change dynamically, so that the acquisition device 40 may be omitted if environmental information of the real space (real world) is acquired in advance. In this case, a device similar to the acquisition device 40 or a method such as photogrammetry may be used to acquire three-dimensional information of the surrounding real space in advance. Alternatively, the content creator or the like may set the scenery seen through the transparent display device 30 and the positions and shapes of real-space objects on the image generating device 10 and store them in the storage unit 12 as environmental information of the real space. In this case, three-dimensional real-space model data such as 3D city model data may be accurately arranged in advance in the virtual space based on the relative position with the position of the transparent display device 30 fixedly arranged on the world coordinate system, and stored in the storage unit 12 as environmental information of the real space. Due to such advance processing, S101 in the flowchart of FIG. 6 may be omitted in these use cases.

[0061] (Modification) Next, a modification of the system 1 according to the first embodiment will be described.

[0062] In the above-described first embodiment, an example has been shown in which a viewpoint position 2A for one user 2 is detected and a display image of a virtual object is generated according to the viewpoint position 2 A. In contrast, according to one modification of this embodiment, a means is provided for superimposing and displaying a virtual object on real space for multiple users 2.

[0063] FIG. 7 is a diagram showing a transmissive display device 30 in a modified example of the system according to the first embodiment, and an example of a display image on the transmissive display device 30. In FIG.

[0064] The transmissive display device 30 in this modification has a relatively large and horizontally elongated shape so that multiple users 2 (four users 2 in the illustrated example) lined up side by side can view the display at the same time. Alternatively, multiple transmissive displays are arranged side by side to function almost equivalently to one large, indefinite transmissive display 30. The real space behind the transmissive display device 30 in this modification can be viewed through the transmissive display device 30. According to the transmissive display device 30 in this modification, for example, by installing it at a window facing a bench in a bus or train, multiple users 2 seated on the bench can view the display at the same time.

[0065] In this modification, it is assumed that each user 2 looks in a predetermined line of sight (for example, in front of each user 2) from a predetermined position such as the bench. In this modification, the measurement device 20 determines a representative viewpoint position 2B based on a plurality of viewpoint positions 2A identified for each of the plurality of users 2. As an example, the representative viewpoint position 2B is the center of gravity of the plurality of viewpoint positions 2A. Alternatively, the representative viewpoint position 2B may be set in advance based on the position and shape of the seat. In this manner, in this modification, after the representative viewpoint position 2B is determined before or immediately after the user is in front of the transmissive display device 30, it is not necessary to continuously acquire the viewpoint position. Therefore, the other configurations and operations of the system 1 in this modification are the same as those of the system 1 in the first embodiment described above, but S102 in the flowchart of FIG. 6 may be skipped depending on the use case.

[0066] In the system 1 of this modified example, when the transmissive display device 30 is viewed from the representative viewpoint position 2B, a display image is generated and displayed on the transmissive display device 30 so as to give the illusion that the virtual object coexists with the real-space objects visible around it through the transmissive display device 30.

[0067] According to the system 1 of this modification, a display image of a virtual object displayed in three-dimensional CG and superimposed on the real space seen through the transparent display device 30 is generated by projective transformation (including perspective projection transformation or a calculation method similar thereto) using information such as the representative viewpoint position 2B and the arrangement, shape, and size of the transparent display device 30, and a composite realistic moving image in which the virtual object displayed in three-dimensional CG exists in the real space of the real world is provided to each user 2 via the transparent display device 30. Each user 2 who sees the image displayed on the transparent display device 30 can have the illusion that the virtual object exists in the real space of the real world seen through the transparent display device 30.

[0068] However, since the viewpoint position used in this modification is the representative viewpoint position 2B and the viewpoint position 2A of each user 2 is a fixed position, it is not possible to realize a superimposed representation with strict geometric accuracy as in the above-mentioned embodiment. In addition, when a real-space three-dimensional model is used for superimposing by performing occlusion processing, the appearance from the user's perspective lacks accuracy, such as a shift occurring in the outline of the overlapping of the real-space object and the virtual object. In the example shown in Figure 7, a state is shown in which multiple cheetahs, which are virtual objects, are drawn on the transmissive display device 30 so as to run in the direction of the illustrated arrows in front of a distant mountain range in the real space that spreads out behind the transmissive display device 30. In such an example of expression, for example, the mountain range and the cheetah do not overlap with each other in the front-to-back relationship with respect to the viewpoint position 2A of each user 2, and they exist independently, so there is no need to perform occlusion processing on the CG of the cheetah, which is a virtual object, and therefore, even if there is some lack of accuracy, the objective of creating an illusion as if a virtual object simultaneously exists in the real space of the real world seen through the transmissive display device 30 can be achieved to a certain extent. In particular, in a use case in which the transmissive display device 30 is assumed to be used as a window of a vehicle, the effect of creating an illusion for the user can be enhanced by devising the expression by adding real-world information as described in the above embodiment. For example, in the case where a virtual object cheetah displayed in 3D CG runs in a virtual space, and the cheetah in the virtual space appears to move slightly backwards in parallel as the vehicle moves forward, even if the configuration does not allow for strictly accurate overlay representation, by reproducing the relative speed between the vehicle's moving speed and the cheetah's running speed, it is possible to add an appearance of the vehicle overtaking the cheetah and running away, thereby giving each user 2 the illusion that the cheetah really exists in the real world.

[0069] In this modified example, since calculation processing is performed to generate a virtual object to be displayed for the representative viewpoint position 2B, it is not possible to provide a representation with geometric precision to all users 2 having different viewpoint positions 2A. However, compared to the case where an image is simply displayed on a transparent display, a virtual object located in the real space behind the transmissive display device 30 is represented on the transmissive display device 30 based on information such as the representative viewpoint position 2B and the position, size, and orientation of the transmissive display device 30, so that a more natural-looking display image can be provided to each user 2 to give the illusion that the virtual object exists in real space.

[0070] [Second embodiment] Next, a second embodiment of the present disclosure will be described.

[0071] In the above-described first embodiment, an example was described in which the image generating device 10 stores three-dimensional CG data for generating a display image representing a virtual object in advance in the storage unit 12, and generates a display image from the three-dimensional CG data by a calculation process that generates an image with geometric rigor that looks realistic and natural to give the user 2 the illusion that the virtual object actually exists. In addition to the configuration and operation of the system 1 of the first embodiment, the second embodiment provides a means for presenting a display image of a virtual object as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes.

[0072] 8 is a diagram showing a display device and the like in a system according to the second embodiment. The system 1 of this embodiment has a transmissive display device 30 that displays a display image of a virtual object on a screen, and a three-dimensional glasses device G worn by a user 2. Other configurations and operations of the system 1 in this embodiment are similar to those of the system 1 of the first embodiment described above.

[0073] The image generating device 10 in this embodiment generates two display images for the left and right eyes having parallax as a display image of a virtual object according to the viewpoint position 2A of the user 2, and displays them on the screen of the transmissive display device 30. The three-dimensional glasses device G shows the two display images to the left and right eyes of the user 2, respectively. By providing parallax to the left and right eyes of the image according to the viewpoint position 2A of the user 2, the user 2 can be given a three-dimensional effect of the display image of the virtual object, and can be given a higher sense of realism and immersion. In other words, in the first embodiment, the objects constituting the actual scenery seen by the user have various positions, especially in the depth direction, and the information received by the left and right eyes is strictly different due to the difference in the positions of the left and right eyes, whereas the virtual object is ultimately displayed as a unique image on the transmissive display 30 in front of the user's eyes even after calculation, so the parallax between the left and right eyes was not taken into consideration. As a result, in the first embodiment, the user feels a certain sense of discomfort in the depth direction when recognizing the existence of the virtual object. However, in this embodiment, different images that take parallax into account are shown to the left and right eyes through the transmissive display 30, which makes it possible to eliminate the sense of incongruity caused by the structural problems that arise in the first embodiment, and enables superimposed expression to be achieved with higher quality.

[0074] 8 shows an example in which the sun, which is a virtual object Vobj, is presented three-dimensionally on the transmissive display device 30 in the sky above a distant mountain range in real space that spreads behind the transmissive display device 30. A display image showing the virtual object Vobj is displayed on the transmissive display device 30 as two display images for the left and right eyes having parallax, and a user 2 wearing the three-dimensional glasses device G can have a visual experience as if the three-dimensional virtual sun object Vobj exists in the real space visible beyond the transmissive display device 30.

[0075] In this embodiment, similarly to the first embodiment described above, display images of virtual objects displayed in three-dimensional CG and superimposed on the real space seen through the transmissive display device 30 are continuously generated following changes in the viewpoint position 2A of the user 2, and a composite realistic moving image in which the virtual objects displayed in three-dimensional CG exist in the real space of the real world is provided to the user 2 through the transmissive display device 30. Therefore, the user 2 who sees the image displayed on the transmissive display device 30 can have a stronger illusion that the virtual objects exist in the real space of the real world seen through the transmissive display device 30, because the three-dimensional CG virtual objects are superimposed on the scenery of the real space, including the depth information due to the parallax between the left and right eyes.

[0076] Also in this embodiment, not only the position information but also the information on the size, shape, etc., of objects in the real space around the transmissive display device 30 as seen from the user is acquired, so that it is possible to simulate the front-to-back relationship and the relative positional relationship when a virtual object to be superimposed and displayed on an object in the real space is virtually arranged in the same space, and it is possible to generate an image in a form that reflects the expression including the expression by the occlusion process in which the virtual object is partially visible or hidden when the user's viewpoint position is used as the starting point. As a result, since the virtual object can be superimposed and displayed with higher quality on the object in the real space, it is possible to provide the user 2 with a more realistic and natural image to give the user 2 the illusion that the virtual object exists in the real space.

[0077] In particular, according to this embodiment, a display image that is continuously generated following changes in the viewpoint position 2A of the user 2 and displays the virtual object Vobj as if it exists in a real space in the real world is displayed on the transmissive display device 30 as two display images for the left and right eyes having parallax, so that the user 2 wearing the three-dimensional eyeglass device G can obtain a visual experience in which the virtual object Vobj exists three-dimensionally and naturally, including in the depth direction, in the real space visible beyond the transmissive display device 30.

[0078] The three-dimensional image system of the transmissive display device 30 and the three-dimensional glasses G according to this embodiment is not particularly limited. For example, it may be an anaglyph type, a polarized type, or a liquid crystal shutter type. When the liquid crystal shutter type is adopted as the three-dimensional image system, a synchronization signal for alternately switching the opening and closing of the left and right liquid crystal shutters of the three-dimensional glasses G in synchronization with the timing of alternately switching and displaying the left and right display images on the transmissive display device 30 is transmitted to the three-dimensional glasses G. In this case, the system 1 further includes, as an example, a synchronization signal sending device 50 that sends such a synchronization signal, and the synchronization signal sending device 50 is driven and controlled by the processing unit 14 of the image generating device 10. The synchronization signal sending device 50 sends a synchronization signal composed of, for example, infrared light toward the three-dimensional glasses G. In addition, since the three-dimensional glasses are generally smaller and lighter than the head-mounted display, they do not cause anxiety or discomfort like the head-mounted display, and the viewing angle at which the image can be seen is not narrow and limited like the current goggle-type MR devices represented by Hololens.

[0079] (Modification) Next, a modification of the system 1 according to the second embodiment will be described.

[0080] In the above-described second embodiment, an example was shown in which a viewpoint position 2A for one user 2 is detected, a display image consisting of two display images for the left and right eyes having a parallax of a virtual object is generated according to the viewpoint position 2A, and displayed on the transmissive display device 30, and a three-dimensional virtual object superimposed and displayed on the real space visible beyond the transmissive display device 30 is presented to the user 2 wearing the three-dimensional eyeglass device G. In contrast, according to one modification of this embodiment, a means is provided for presenting a three-dimensional virtual object superimposed and displayed on the real space to a plurality of users 2 wearing the three-dimensional eyeglass device G.

[0081] FIG. 9 is a diagram showing a transmissive display device 30 in a modified example of the system according to the second embodiment, and an example of a display image on the transmissive display device 30. In FIG.

[0082] The transparent display device 30 in this modification has a relatively large and horizontally elongated shape so that multiple users 2 (four users 2 in the illustrated example) lined up side by side can view it simultaneously. The transparent display device 30 in this modification is also a transparent display, and the real space behind it can be viewed through the transparent display device 30. The transparent display device 30 in this modification can be installed, for example, in a window facing a bench in a bus or train, so that multiple users 2 sitting on the bench can view it simultaneously.

[0083] In this modification, the measuring device 20 detects the viewpoint position 2A of each user 2, and performs projective transformation, perspective projection transformation, or similar calculation processing on the display image of the virtual object Vobj according to the viewpoint position 2A of each user 2. In addition, as an example, in this modification, a liquid crystal shutter type is adopted as a three-dimensional image method, and a synchronization signal sending device 50 is further provided that sends a synchronization signal for alternately switching the opening and closing of the left and right liquid crystal shutters of the three-dimensional glasses G in synchronization with the timing of alternately switching and displaying the left and right display images on the transmissive display device 30. The synchronization signal sending device 50 is driven and controlled by the processing unit 14 of the image generating device 10. The synchronization signal sending device 50 sends a synchronization signal, for example, composed of infrared light, to the three-dimensional glasses G.

[0084] Other configurations and operations of the system 1 in this modified example are similar to those of the system 1 in the above-described second embodiment.

[0085] In the system 1 of this modification, when each user 2 views the transmissive display device 30 from the viewpoint position 2A, a display image of the virtual object Vobj is generated and displayed on the transmissive display device 30 so that the virtual object Vobj is superimposed on the real space seen around the user 2 through the transmissive display device 30. The example shown in Fig. 9 shows a state in which the sun, which is the virtual object Vobj, is presented three-dimensionally on the transmissive display device 30 in the sky above a distant mountain range in the real space spreading behind the transmissive display device 30.

[0086] In addition, when a plurality of users 2 see the virtual object Vobj from their respective viewpoints, the processing unit 14 needs to perform display control of the display image of the virtual object Vobj for each user 2 so that the left and right display images having appropriate parallax for each user 2 are displayed on the transmissive display device 30. As an example, in this modified example, the transmissive display device 30 has a relatively high screen display rate (for example, 240 Hz), and the processing unit 14 controls the display images for four users to be displayed in sequence in a time-division manner so that the left and right display images (two images) for each user 2 are displayed at a display rate of 30 Hz. The display rate of 30 Hz is almost the same as the display rate of terrestrial digital television, and each user 2 can view the virtual object Vobj displayed at the display rate of 30 Hz without feeling uncomfortable. On the other hand, since a higher display rate makes it possible to make flicker caused by the operation of the liquid crystal shutter less noticeable, a 480 Hz display device may be used to display at a display rate of 60 Hz for each eye of each user 2, or other frequencies may be used.

[0087] In this manner, in this modified example, similarly to the above-described embodiment, display images of virtual objects displayed in three-dimensional CG and superimposed on the real space seen through the transmissive display device 30 are continuously generated following changes in the viewpoint position 2A of each user 2, and composite realistic moving images as if the virtual objects displayed in three-dimensional CG exist in the real space of the real world are provided to each user 2 through the transmissive display device 30. Therefore, each user 2 who sees the image displayed on the transmissive display device 30 can have a stronger illusion that the virtual objects exist in the real space of the real world seen through the transmissive display device 30, since the three-dimensional CG virtual objects are superimposed on the scenery of the real space, including the depth information due to the parallax between the left and right eyes.

[0088] Also in this modification, not only the position information but also the information on the size, shape, etc., of objects in the real space around the transmissive display device 30 as seen from each user is acquired, so that it is possible to simulate the front-to-back relationship and the relative positional relationship when a virtual object to be superimposed and displayed on an object in the real space is virtually arranged in the same space, and it is possible to generate an image in a form that reflects the expression including the expression by the occlusion process in which the virtual object is partially visible or hidden when the viewpoint position of each user is used as the starting point. As a result, since the virtual object can be superimposed and displayed with higher quality on the object in the real space, it is possible to provide each user 2 with a more realistic and natural image so that each user 2 has the illusion that the virtual object exists in the real space.

[0089] In particular, according to this modified example, a display image that is continuously generated following changes in the viewpoint position 2A of each user 2 and displays the virtual object Vobj as if it exists in a real space in the real world is displayed on the transmissive display device 30 as two display images for the left and right eyes having parallax. Therefore, each user 2 wearing the three-dimensional eyeglass device G can be individually provided with a visual experience in which the virtual object Vobj exists three-dimensionally and naturally, including in the depth direction, in the real space visible beyond the transmissive display device 30.

[0090] In addition, 3D glasses are generally smaller and lighter than head-mounted displays, so they do not cause the same sense of anxiety or discomfort as head-mounted displays, and the viewing angle of the image is not limited to a narrow angle as with current goggle-type MR devices such as Hololens.

[0091] [Third embodiment] Next, a third embodiment of the present disclosure will be described.

[0092] In the above-mentioned second embodiment, an example was described in which a means is provided for presenting a display image of a virtual object to a user wearing a three-dimensional glasses device as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes. In contrast, in this embodiment, a means is provided for presenting a display image of a virtual object to a naked-eye user not wearing a three-dimensional glasses device as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes.

[0093] FIG. 10 is a schematic diagram for explaining the configuration of a display device in the system of the third embodiment.

[0094] As shown in FIG. 10, the transmissive display device 30 of this embodiment is, for example, a display capable of simultaneously displaying different images in a plurality of display directions, for example, a type of display in which a lenticular lens is attached to the surface of a transmissive organic electroluminescence display (OLED) to realize naked-eye stereoscopic vision. The transmissive display device 30 of this embodiment includes a transparent substrate 301 having a plurality of pixel units 302 arranged in an array, and a plurality of lens elements 303 provided on each pixel unit 302 on the transparent substrate 301. As an example, each lens element 303 in this embodiment is a semi-cylindrical lens called a lenticular lens. As the lens element 303, in addition to a lenticular lens, a lens of any three-dimensional shape such as a hemispherical semi-ellipsoid shape that controls the direction of light may be used. Also, instead of using a lens, a film or plate that generates a parallax barrier may be used.

[0095] As an example, the lens element 303 can be provided on each pixel unit 302 by individually bonding an individually manufactured lens element 303 to the position of the transparent substrate 301 where each pixel unit 302 is provided with a transparent adhesive. Alternatively, the lens element 302 can be provided on each pixel unit 302 by discharging a transparent material to the position of the transparent substrate 301 where each pixel unit 302 is provided using 3D printing technology to form the lens element 302. Furthermore, the lens element 303 can be provided on each pixel unit 302 by forming a transparent film member in advance on which a plurality of lens elements 303 positioned with respect to each pixel unit 302 of the transparent substrate 301 are formed, and attaching the transparent film member in a predetermined position on the transparent substrate 301. Alternatively, the transparent substrate 301 may be configured by inserting an array of pixel units and electrode wires for supplying power and signals to the pixel units into an intermediate film portion of a plate having a structure in which a plurality of transparent substrates are bonded together, and at the same time, inserting a film or plate including a lens element, or a film or plate that generates a parallax barrier. Alternatively, a method may be used in which only the array-shaped pixel units are inserted into an intermediate film to form the transparent substrate 301, and then a film or plate containing lens elements or a film or plate that generates a parallax barrier is attached to the outside. Here, the transparent substrate may be a transparent resin plate including an acrylic plate, or glass. Any material may be used as long as it has a certain degree of transparency.

[0096] The pixel units 302 and the lens elements 303 are arranged at a predetermined interval from each other. In the area where the pixel units 302 and the lens elements 303 are not arranged, the real space behind the transmissive display device 30 can be seen through the transparent substrate 301. In the transmissive display device 30 of this embodiment, the pixel units 302 and the lens elements 303 are arranged relatively sparsely. For example, the proportion of the pixel units 302 and the lens elements 302 in the entire screen of the transmissive display device 30 is several tens of percent to several percent, and if the size of the pixel units can be made smaller through future technological developments, it can be made less than several percent. If the size of each pixel unit can be reduced or the light intensity of each pixel unit can be increased, it is possible to display an image that is visible to the user without being affected by the surrounding light while maintaining transparency, and in the application of superimposing and displaying a virtual object on the real space seen through the transmissive display device 30 as in this case, it is possible to provide the user viewing the virtual object with sufficient resolution and clarity in the image.

[0097] For example, the transparent substrate 301 is provided with electrode wires made of a highly transparent conductive material or very fine metal electrode wires (not shown) that are electrically connected to each pixel unit 302, and each pixel unit 302 is individually driven to emit light or controlled to transmit light by power and drive signals supplied via these electrodes. Here, each pixel unit may be a self-emitting element such as an organic EL or micro LED, or may be an element that controls the transmission of light using a liquid crystal mechanism. In addition, each pixel unit may be an element or luminous material that reflects projection light from outside to allow the user to recognize a difference in color.

[0098] Other than that, the configuration and operation of each part of the system 1 according to this embodiment are generally similar to those of the system 1 described in the first embodiment, but the configurations and operations that differ will be described below.

[0099] Next, a more detailed configuration of the transmissive display device 30 of this embodiment and display operation control by the transmissive display device 30 will be described with reference to FIGS.

[0100] In the transmissive display device 30 of this embodiment, as an example, each pixel unit 302 is configured as a group of 45 pixels. That is, one pixel unit 302 is made up of 45 pixels, and an RGB display element 302' is arranged in each pixel. The RGB display element 302' is made up of three sub-pixel display elements that display three colors, R (red), G (green), and B (blue). Alternatively, the RGB display element 302' may be a display element for four colors. The pixel unit 302 made up of 45 pixels corresponds to one pixel of a general display.

[0101] In addition, as an example, each pixel unit 302 is provided with a lens element 303 which is a semi-cylindrical lens called a lenticular lens for refracting the directions of the optical paths of the 45 RGB lights displayed on the RGB display elements 302' for 45 pixels in that pixel unit 302 into 45 display directions, for example 0 to 44 (hereinafter, these directions will be referred to as "display directions 501") shown in Figure 12.

[0102] Then, the image generating device 10 inputs the display image data generated by the image generating device 10 corresponding to the different viewpoint positions 2A of each of the users 21-24 to the RGB display element 500 of the pixel numbers corresponding to the directions 502A, 502B, 502C, 502D from a predetermined origin O (e.g., the center position of the transmissive display device 30) of the reference coordinate space (FIG. 4) toward the viewpoint positions 2A1-2A4 of each of the users 21-24. Hereinafter, these directions are referred to as user viewpoint position directions 502A, 502B, 502C, 502D. Alternatively, these directions are collectively referred to as user viewpoint position directions 502.

[0103] Here, the processing unit 14 of the image generating device 10 may generate a right-eye display image and a left-eye display image for each of the viewpoint positions 2A1-2A4 of each of the users 21-24 detected by the measurement device 20, so that when a three-dimensional virtual object in the virtual space is viewed from the right eye and the left eye of the user at a certain distance to the left and right of the viewpoint position through the transmissive display device 30, the image appears as a realistic and natural image with geometric rigor to give each user 2 the illusion that the virtual object exists in real space. When generating the right-eye display image and the left-eye display image to be displayed on the screen of the transmissive display device 30, the image generating device 10 may perform projective transformation, perspective projection transformation, or similar calculation processing to project the virtual object represented by three-dimensional CG data onto the screen of the transmissive display device 30, i.e., a two-dimensional surface, based on the viewpoint positions 2A1-2A4 of each of the users 21-24.

[0104] In this case, for each of the viewpoint positions 2A1 to 2A4 of each of the users 21 to 24, the processing unit 14 displays the above-mentioned right eye display image and left eye display image data generated corresponding to each viewpoint position on the RGB display element 302' with a pixel number corresponding to the right eye display direction and the RGB display element 302' with a pixel number corresponding to the left eye display direction, which are located on both sides of each user viewpoint position direction 502A, 502B, 502C, 502D corresponding to each viewpoint position.

[0105] As a result of the above control operation, as shown by the line of sight 500 connecting each of the right and left eyes with each of the lens elements 303 for each of the users 21 to 24 in Fig. 11, and as described as the displays D1, D2, D3, etc. in Fig. 3, it becomes possible to simultaneously display images in which the display state of the virtual object is controlled for each of the users 2 to the multiple users 21 to 24 at various positions with respect to the transmissive display device 30. In this case, for each of the users 21 to 24, an image having parallax between both eyes is incident on the right and left eyes, so that the user can experience a realistic stereoscopic view with the naked eye.

[0106] In this embodiment, the viewpoint position 2A of each user 2 is detected by the measurement device 20, and projective transformation, perspective projection transformation, or similar calculation processing is performed on the display image of the virtual object Vobj according to the viewpoint position 2A of each user 2.

[0107] FIG. 13 is a flow chart showing a detailed example of the display process of step S104 in FIG. 6 described in the first embodiment. Hereinafter, this process will be described with reference to the explanatory diagram of FIG. 12 described above. The processing unit 14 sets the left-eye display image data generated corresponding to the left-eye display direction of the first viewpoint position 2A (e.g., 2A1 in FIG. 12) to the RGB display element 302' of each pixel number corresponding to each display direction 501, for example, from 0th to 3rd in FIG. 12, between the 0th display direction 501 of the 45 display directions 501 from 0th to 44th described in FIG. 12 and the user viewpoint position direction 502 (e.g., 502A in FIG. 12) passing through the first viewpoint position 2A (e.g., 2A1 in FIG. 12) of one or more viewpoint positions 2A calculated in step S102 in FIG. 6 (step S901). This setting process is executed for the RGB display element 302' of each pixel number of each pixel unit 302 (see FIG. 11) constituting the transmissive display device 30.

[0108] Next, the processing unit 14 increments by +1 the variable value indicating the viewpoint position 2A stored in, for example, a RAM or a register (not shown) in the storage unit 12 of the image generating device 10 (step S905).

[0109] Next, the processing unit 14 judges whether or not the variable value indicating the viewpoint position 2A has exceeded a predetermined last user viewpoint position (step S906). The value indicating the last viewpoint position 2A can be predetermined according to the number of users 2 using the system 1 and stored in the RAM or register. In step S906, it is judged whether or not the variable value indicating the viewpoint position 2A has exceeded the last user viewpoint position depending on whether or not the variable value indicating the viewpoint position 2A has exceeded the predetermined and stored value.

[0110] If the last user's viewpoint position has not yet been processed and the determination in step S906 is NO, the processing unit 14 transfers control to the process of step S902 and executes the processes from step S902 to S904.

[0111] First, the processing unit 14 calculates a midpoint direction 503 (for example, 503AB in FIG. 12) passing through the midpoint between the previous viewpoint position 2A (for example, 2A1 in FIG. 12) and the current viewpoint position 2A (for example, 2A2 in FIG. 12) (step S902).

[0112] Next, the processing unit 14 sets the right-eye display image data generated corresponding to the right-eye display direction of the previous user's viewpoint position 2A (e.g., 2A1 in FIG. 12) to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from No. 4 to No. 10 in FIG. 12, between the user viewpoint position direction 502 (e.g., 502A in FIG. 12) passing through the previous user's viewpoint position 2A (e.g., 2A1 in FIG. 12) and the currently calculated midpoint direction 503 (e.g., 503AB in FIG. 12) (step S903). This setting process is executed for the RGB display elements 302' of each pixel number of each pixel unit 302 (see FIG. 11) constituting the transmissive display device 30.

[0113] Next, the processing unit 14 sets the left-eye display image data generated corresponding to the left-eye display direction of the current user's viewpoint position 2A (e.g., 2A2 in FIG. 12) to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from No. 11 to No. 16 in FIG. 12, between the midpoint direction 503 (e.g., 503AB in FIG. 12) calculated this time in step S902 and the user's viewpoint position direction 502 (e.g., 502B in FIG. 12) passing through the current user's viewpoint position 2A (e.g., 2A2 in FIG. 12) (step S904). This setting process is executed for the RGB display elements 302' of each pixel number of each pixel unit 302 (see FIG. 11) constituting the transmissive display device 30.

[0114] Thereafter, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by +1 (step S905), and determines whether or not the variable value has exceeded the predetermined last user viewpoint position 2A (step S906).

[0115] If the final viewpoint position 2A has not yet been processed and the determination in step S906 is NO, the processing unit 14 executes the above-mentioned processes from steps S902 to S904 again.

[0116] In step S902 similar to that described above, for example in FIG. 12, midpoint direction 503BC passing through the midpoint between previous viewpoint position 2A2 and current user viewpoint position 2A3 is calculated.

[0117] Next, in step S903 similar to that described above, the right-eye display image data generated corresponding to the right-eye display direction of the previous viewpoint position 2A2 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 of, for example, numbers 17 to 21 in Fig. 12 between the user viewpoint position direction 502B passing through the previous viewpoint position 2A2 and the currently calculated midpoint direction 503BC in Fig. 12. This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.

[0118] Next, in step S904 similar to the above, the left-eye display image data generated corresponding to the left-eye display direction of the current viewpoint position 2A3 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 of, for example, numbers 22 to 26 in Fig. 12 between the midpoint direction 503BC calculated this time in step S902 and the user viewpoint position direction 502C passing through the current viewpoint position 2A3 in Fig. 12. This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.

[0119] Thereafter, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by +1 (step S905), and determines whether or not the variable value has exceeded the predetermined last user viewpoint position 2A (step S906).

[0120] If the final viewpoint position 2A has not yet been processed and the determination in step S906 is NO, the processing unit 14 executes the above-mentioned processes from steps S902 to S904 again.

[0121] In step S902 similar to that described above, for example in FIG. 12, a midpoint direction 503CD passing through the midpoint between the previous viewpoint position 2A3 and the current viewpoint position 2A4 is calculated.

[0122] Next, in step S903 similar to that described above, the right-eye display image data generated corresponding to the right-eye display direction of the previous viewpoint position 2A3 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 of, for example, numbers 27 to 30 in Fig. 12 between a user viewpoint position direction 502C passing through the previous user viewpoint position 2A3 and the currently calculated midpoint direction 503CD in Fig. 12. This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.

[0123] 12, the left-eye display image data generated corresponding to the left-eye display direction of the current viewpoint position 2A4 is set to the RGB display elements 302' of each pixel number corresponding to each display direction 501 of, for example, numbers 31 to 34 in Fig. 12 between the midpoint direction 503CD calculated this time in step S902 and the user viewpoint position direction 502D passing through the current user viewpoint position 2A4, for example in Fig. 12. This setting process is executed for the RGB display elements 302' of each pixel number in each pixel unit 302 (see Fig. 11) constituting the transmissive display device 30.

[0124] Thereafter, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by +1 (step S905), and determines whether or not the variable value has exceeded the predetermined final viewpoint position 2A (step S906).

[0125] When the determination in step S906 becomes YES after processing up to the last user viewpoint position 2A, the processing unit 14 executes the processing in the final step S907. In step S907, the processing unit 14 sets the right-eye display image data generated corresponding to the right-eye display direction of the last user viewpoint position 2A (e.g., 2A4 in FIG. 12) to the RGB display elements of each pixel number corresponding to each display direction 501, for example, from No. 35 to No. 44 in FIG. 12, between the user viewpoint position direction 502 (e.g., 502D in FIG. 12) passing through the current viewpoint position 204 (e.g., 2A4 in FIG. 12) and the 44th display direction 501.

[0126] Thereafter, the processing unit 14 ends the display process of step S104 in FIG. 6, which is illustrated in the flowchart of FIG.

[0127] In the image display process exemplified in the flowchart of Fig. 13, the display image data for the right eye or the left eye generated corresponding to the display direction for the right eye or the left eye of the adjacent viewpoint position 2A is copied and set in each display direction 501 in the section divided by the user viewpoint position direction 502 of two viewpoint positions 2A in Fig. 12 and the midpoint direction 503 passing through the midpoint between those viewpoint positions 2A. On the other hand, by changing the algorithm of mapping the display pixels to each display direction 501 between the viewpoint position 2A of the user 2 close to the transmissive display device 30 and the viewpoint position 2A of the user 2 far from the transmissive display device 30, it is also possible to make the change sensitivity of the display pixels uniform according to the viewpoint position A of the user 2. Alternatively, instead of simply copying, display pixels calculated by performing an interpolation process according to each display direction 501 between two adjacent display images may be set.

[0128] In this way, according to this embodiment, for a plurality of users 2 with naked eyes who do not wear three-dimensional glasses, for each of the viewpoint positions 2A1-2A4 of each of the users 21-24, when the user views a three-dimensional stereoscopic virtual object in a virtual space through the transmissive display device 30 from the right eye and the left eye of the user who is a certain distance to the left and right from the viewpoint position, the display image of the virtual object can be presented as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes, superimposed on the real space seen through the transmissive display device 30 so that the image appears realistic and natural with geometric rigor to give each user 2 the illusion that the virtual object exists in real space. In addition, according to this embodiment, separate display images of the virtual object according to the viewpoint position 2A of each user 2 can be presented to each user 2 simultaneously.

[0129] As an application example of the transparent display device 30 in this embodiment, for example, the transparent display device 30 of this embodiment may be attached to the glass part of an automobile or the like, and a stereoscopic display image of a virtual object may be presented to each of a plurality of users 2 seated in the front and rear seats of the automobile. According to the transparent display device 30 of this embodiment, a display image of a virtual object can be individually displayed to a plurality of users 2 who are at different distances and in different viewing directions from the transparent display device 30, so that a virtual object corresponding to the view seen from the viewpoint position of each user 2 can be superimposed and presented on the real space seen around the transparent display device 30 as the glass of a moving automobile.

[0130] (Modification) FIG. 14 is a diagram showing a modified example of the display device in the system of the third embodiment.

[0131] In the display device 30 of the third embodiment shown in Fig. 10 etc., the lens element 303 is provided individually for each pixel unit 302. In contrast to this, as shown in this modification, the lens element 303 may have a vertically elongated shape arranged across a plurality of pixel units 302 in each column among the plurality of pixel units 302 arranged in a matrix. This makes it possible to reduce the number of manufacturing steps for the lens element 303 compared to the case where the lens element 303 is provided individually for each pixel unit 302.

[0132] The above-described embodiments and modifications of the present invention are merely examples for explaining the present invention, and are not intended to limit the scope of the present invention to these embodiments. Those skilled in the art can implement the present invention in various other forms without departing from the gist of the present invention. [Explanation of symbols]

[0133] 1... image display system, 2... user, 2A... viewpoint position, 10... image generating device, 20... measuring device, 30... display device, 40... real space environment information acquiring device, 50... synchronization signal transmitting device

Claims

1. a transmissive display device capable of displaying an image; a storage unit that stores position information and three-dimensional shape information on a predetermined reference coordinate system that is set by superimposing it on a coordinate system in real space for the display device, and three-dimensional image data for an arbitrary virtual object constructed on the reference coordinate system; a processing unit that generates an image and displays it on the display device by performing projective transformation or perspective projection transformation based on the user's viewing position, position information and three-dimensional shape information of the display device, and three-dimensional image data representing the virtual object, so that the virtual object is accurately superimposed on the real space that is visible through the display device when the user looks at the display device, as it would appear when the user looks at a virtual space; a plurality of 3D eyeglass devices respectively worn by the plurality of users; the processing unit generates, for each of the users, two display images for the left and right eyes having parallax, and displays the two display images for each of the users on the display device in a manner that the two display images can be seen by the left and right eyes of each of the users through the three-dimensional glasses device; Each of the three-dimensional eyeglass devices displays the two display images to the left and right eyes of the user wearing the device, respectively. Image display system.

2. a transmissive display device capable of displaying an image; a storage unit that stores position information and three-dimensional shape information on a predetermined reference coordinate system that is set by superimposing it on a coordinate system in real space for the display device, and three-dimensional image data for an arbitrary virtual object constructed on the reference coordinate system; a processing unit that generates an image and displays it on the display device by performing projective transformation or perspective projection transformation based on the user's viewing position, position information and three-dimensional shape information of the display device, and three-dimensional image data representing the virtual object, so that the virtual object is accurately superimposed on the real space that is visible through the display device when the user looks at the display device, as it would appear when the user looks at a virtual space; a measuring device for measuring the location of each of the one or more users; the display device is configured to be capable of simultaneously displaying different images in multiple display directions; The processing unit a display image for the right eye and a display image for the left eye are generated for each of the one or more users measured by the measurement device by performing projective transformation or perspective projection transformation based on the positions of the right eye and the left eye of each of the users, position information and three-dimensional shape information of the display device, and three-dimensional image data representing the virtual object, so that a virtual object that appears when the virtual space is viewed from each of the right eye and the left eye of each of the users is accurately superimposed on a real space seen through the display device by the right eye and the left eye of each of the users; a display image for each right eye of each user is displayed from the display device in a direction of a range including the right eye of each user, and a display image for each left eye of each user is displayed from the display device in a direction of a range including the left eye of each user. Image display system.

3. a transmissive display device capable of displaying an image; a measurement device for measuring the positions of each of the left and right eyes of one or more users; a storage unit that stores position information and three-dimensional shape information on a predetermined reference coordinate system that is set by superimposing it on a coordinate system in real space for the display device, and three-dimensional image data for an arbitrary virtual object constructed on the reference coordinate system; a processing unit that performs projective transformation or perspective projection transformation based on the positions of each of the users' left and right eyes, position information and three-dimensional shape information of the display device, and three-dimensional image data of the virtual object, so that a virtual object that appears when a virtual space is viewed from each of the users is accurately superimposed on a real space that is visible through the display device when each of the users views the display device, thereby generating two display images for the left and right eyes of each of the users, each having a parallax corresponding to the positions of the left and right eyes of the user, and displays the two display images for each of the users on the display device in a manner that allows them to be seen by the left and right eyes of the user through a three-dimensional glasses device; a plurality of 3D eyeglass devices worn by the plurality of users, each of which displays the two display images to the left and right eyes of the user; An image display system having:

4. a transmissive display device capable of simultaneously displaying different images in multiple display directions; a measurement device for measuring the positions of each of the left and right eyes of one or more users; a storage unit that stores position information and three-dimensional shape information on a predetermined reference coordinate system that is set by superimposing it on a coordinate system in real space for the display device, and three-dimensional image data for an arbitrary virtual object constructed on the reference coordinate system; a processing unit that generates images corresponding to the positions of the left and right eyes of each of the users by performing projective transformation or perspective projection transformation based on the positions of the left and right eyes of each of the users, position information and three-dimensional shape information of the display device, and three-dimensional image data of the virtual object, so that a virtual object that appears when a virtual space is viewed from each of the one or more users is accurately superimposed on a real space that is visible through the display device when the user views the display device, and that causes the display device to display a display image for the right eye of each of the users in a direction including the right eye of each of the users, and causes the display device to display a display image for the left eye of each of the users in a direction including the left eye of each of the users; An image display system having:

5. The range is determined so as not to overlap with a range in which a display image for the other eye of the user is displayed, a range in which a display image for the right eye of another user is displayed, and a range in which a display image for the left eye of another user is displayed.

5. The image display system according to claim 2 or 4.

Images