Display device

Abstract

This invention provides a technology that reduces user discomfort with light emanating from pixels in a display device capable of displaying different images from multiple viewpoints. [Solution] The display device includes a quasi-planar substrate including a flat or curved surface; a group of pixels including a plurality of display pixels at different positions in a first direction parallel to the tangent plane of the substrate; a first lens element configured to refract light incident from a plurality of the display pixels of the pixel group in a third direction perpendicular to the tangent plane of the substrate in separate directions within a plane including the first and third directions; and a second lens element configured to refract light incident from the display pixels in the third direction within a plane including a second direction and a third direction that intersect the first direction at a specific angle within the tangent plane of the substrate.

Description

Technical Field

[0001] The present disclosure relates to a display device capable of three-dimensional video representation.

Background Art

[0002] Patent Document 1 discloses an image display system that enables display of display images that reproduce the appearance of a three-dimensional space from each of a plurality of viewpoint positions.

[0003] The image display system disclosed in Patent Document 1 includes a display having a screen capable of simultaneously displaying different images in a plurality of display directions, screen arrangement data indicating the position, orientation, and shape of the screen in a virtual preset reference coordinate space, three-dimensional data representing a three-dimensional object in the reference coordinate space, user viewpoint position data indicating one or more user viewpoint positions in the reference coordinate space, a storage unit that holds these, an image generation unit that generates, for each of the one or more user viewpoint positions, a display image of the three-dimensional object that appears as if seen from the user viewpoint position through the screen based on the screen arrangement data and the three-dimensional data, and an image display unit that displays, for each of the one or more user viewpoint positions, the display image generated corresponding to the user viewpoint position from the display in the user viewpoint position direction directed toward the user viewpoint position.

[0004] [[ID=第十九]] Further, Patent Documents 2 and 3 disclose an image display system capable of displaying a display image that reproduces the appearance of a three-dimensional space from each of a plurality of viewpoint positions by superimposing it on the real space visible through a transmissive display that transmits light. The image display systems described in Patent Documents 2 and 3 include: a transparent display device capable of displaying images; a storage unit that stores position information and three-dimensional shape information in a virtual space constructed on a predetermined reference coordinate system set to be superimposed on the coordinate system in the real space for each object in the real space surrounding it and for any virtual object; and a processing unit that generates an image and displays it on the display device by performing a projection transformation or perspective projection transformation based on the viewpoint position, the position information and three-dimensional shape of the display device, and three-dimensional image data representing the virtual object, so that when a user views the display device from a certain viewpoint position, the virtual object is accurately superimposed on the real space visible through the display device as it would appear when the user views the virtual space from that viewpoint position. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] International Publication WO2023 / 277020A1 Specification [Patent Document 2] International Publication WO2024 / 095358A1 Specification [Patent Document 3] International Publication WO2024 / 096044A1 Specification [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The image display systems described in Patent Documents 1 and 2 generate two-dimensional display images corresponding to multiple viewpoint positions based on data of a three-dimensional object, and use a display device capable of displaying different images for multiple viewpoints to display each display image only in the direction of the viewpoint position corresponding to it. The display device includes a pixel unit having multiple pixels, and by limiting the direction in which the light emitted from each pixel in the pixel unit can be seen for each pixel, it displays different display images for each viewpoint direction. For example, a configuration is used in which multiple pixels in the pixel unit are arranged at different positions in the horizontal direction, a semicircular cylindrical lens that acts as a lens in the horizontal direction is placed on the pixel unit, and the light from each pixel in the pixel unit is refracted in different directions by the cylindrical lens. Furthermore, in Patent Document 3, for example, pixels are arranged not only horizontally but also vertically, and a matrix-like configuration is defined as a pixel group, and a configuration is used in which multiple lens elements are arranged to cross the pixel group at a certain angle θ tilted to the vertical direction. In these display devices, the point-like light emitted from pixels is magnified by a cylindrical lens in the horizontal direction or in a direction tilted at an angle θ to the horizontal direction, causing it to appear distorted into a shape with a longitudinal direction in a specific direction (a horizontally elongated shape), which may cause discomfort to the user. One objective included in this disclosure is to provide a technology that enables a reduction in user discomfort with light emanating from pixels in a display device capable of displaying different images for multiple viewpoints. [Means for solving the problem]

[0007] A display device according to one aspect of the present disclosure comprises a quasi-planar substrate including a planar or curved surface, A group of pixels including a plurality of display pixels at different positions in a first direction parallel to the tangent plane of the substrate, A first lens element is configured to refract light incident from a plurality of the display pixels of the pixel group in a third direction perpendicular to the tangent plane of the substrate in separate directions within a plane including the first and third directions. A second lens element is configured to refract light incident from the display pixel in the third direction within a plane that includes a second direction and a third direction that intersect the first direction at a specific angle within the tangent plane of the substrate, It has. [Effects of the Invention]

[0008] According to one aspect of the present disclosure, for example, the point-like light emitted from a pixel is refracted and widened by the second lens element in a plane including the second and third directions orthogonal to the first direction, and then refracted and widened in separate directions by the first lens element in a plane including the first and third directions, causing it to appear as if it has been deformed into a shape that has spread horizontally and vertically. Compared to a device composed solely of the first lens element, this significantly reduces the user's discomfort with the light from the pixel. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the image display system according to the first embodiment. [Figure 2] This is a schematic diagram showing an example of the image display system of the first embodiment being mounted on a vehicle. [Figure 3] This figure illustrates how the image display system according to the first embodiment displays an image corresponding to the user's viewpoint position. [Figure 4] This is a conceptual diagram to explain how the viewpoint position is calculated. [Figure 5] This is a block diagram showing the configuration of the image generation device according to the first embodiment. [Figure 6] This is an example of a flowchart showing the overall processing by the image generation device of the first embodiment. [Figure 7] This is a schematic diagram illustrating the configuration of the display device in the system of the second embodiment. [Figure 8]This is a diagram for explaining the more detailed configuration of the transmissive display device 30 of the second embodiment and the display operation control by the transmissive display device 30. [Figure 9] This is a diagram for explaining the more detailed configuration of the transmissive display device 30 of the second embodiment and the display operation control by the transmissive display device 30. [Figure 10] This is a flowchart showing a detailed example of the display process in step S104 of FIG. 6 described in the first embodiment. [Figure 11] This is a diagram showing a modified example of the display device in the system of the second embodiment. [Figure 12] This is a schematic diagram for explaining the configuration of the display device in the system of the third embodiment. [Figure 13] This is a diagram for explaining the more detailed configuration of the display device of the third embodiment and the display operation control by the display device. [Figure 14] This is a diagram for explaining the more detailed configuration of the display device of the third embodiment and the display operation control by the display device. [Figure 15] This is a flowchart of the calibration process. [Figure 16] This is a schematic diagram for explaining the configuration of the first modified example of the display device in the system of the third embodiment. [Figure 17] This is a schematic diagram for explaining the configuration of the second modified example of the display device in the system of the third embodiment. [Figure 18] This is a schematic diagram for explaining the configuration of the third modified example of the display device in the system of the third embodiment. [Figure 19] This is a diagram for explaining the configuration of the fourth modified example of the display device in the system of the third embodiment. [Figure 20] This is a schematic diagram for explaining the configuration of the display device in the system of the fourth embodiment. [Figure 21]This is a schematic diagram illustrating other configurations of the display device in the system of the fourth embodiment. [Figure 22] This is a schematic diagram illustrating other configurations of the display device in the system of the fourth embodiment. [Figure 23] This is a diagram illustrating the configuration of the electrode wires of the display device according to the fourth embodiment. [Figure 24] This figure illustrates a more detailed configuration of the display device according to the fourth embodiment and the control of the display operation by the display device. [Figure 25] This figure illustrates a more detailed configuration of the display device according to the fourth embodiment and the control of the display operation by the display device. [Figure 26] This figure illustrates a more detailed configuration of the display device according to the fourth embodiment and the control of the display operation by the display device. [Figure 27] This flowchart shows a detailed example of the display process in the fourth embodiment. [Figure 28] This diagram illustrates the configuration of a transparent display that shows parallax-laden three-dimensional images of virtual objects to multiple users in the second to fourth embodiments. [Figure 29] This diagram illustrates the configuration of a transparent display that shows parallax-laden three-dimensional images of virtual objects to multiple users in the second to fourth embodiments. [Figure 30] This figure illustrates the configuration of a transparent display that shows a parallax-laden three-dimensional image of a virtual object to multiple users in the fifth embodiment. [Figure 31] This figure illustrates the configuration of a transparent display that shows a parallax-laden three-dimensional image of a virtual object to multiple users in the fifth embodiment. [Figure 32] This is a schematic diagram showing the shape of the optical path of the transmissive display in the fifth embodiment. [Figure 33]This figure shows how light entering from a single pixel appears on the measurement surface of a display. [Figure 34] This figure illustrates another configuration of a transparent display that shows a parallax-laden three-dimensional image of a virtual object to multiple users in the fifth embodiment. [Figure 35] This figure illustrates another configuration of a transparent display that shows a parallax-laden three-dimensional image of a virtual object to multiple users in the fifth embodiment. [Figure 36] This figure illustrates another configuration of a transparent display that shows a parallax-laden three-dimensional image of a virtual object to multiple users in the fifth embodiment. [Figure 37] This figure shows an example of a display device in the system of the fifth embodiment. [Figure 38] This figure shows a modified example of the display device in the system of the fifth embodiment. [Figure 39] This figure shows a modified example of the display device in the system of the fifth embodiment. [Figure 40] This figure shows a modified example of the display device in the system of the fifth embodiment. [Figure 41] This figure shows a modified example of the display device in the system of the fifth embodiment. [Figure 42] This figure shows a modified example of the display device in the system of the fifth embodiment. [Figure 43] This figure shows a modified example of the display device in the system of the fifth embodiment. [Figure 44] This figure is used to illustrate a modified example of the display device in the system of the fifth embodiment. [Figure 45] This figure is used to illustrate a modified example of the display device in the system of the fifth embodiment. [Figure 46] This is a schematic diagram illustrating other configurations of the display device. [Figure 47]This figure shows a modified example of the display device in the system of the fifth embodiment. [Figure 48] This figure shows a modified example of the display device in the system of the fifth embodiment. [Modes for carrying out the invention]

[0010] [First Embodiment] A first embodiment of this disclosure will be described with reference to the drawings.

[0011] Figure 1 is a schematic diagram of an image display system according to the first embodiment of this disclosure. As shown in Figure 1, the image display system 1 of this embodiment (hereinafter sometimes simply referred to as "the system") mainly comprises an image generation device 10 that generates virtual object images to be presented to the user, a measuring device 20 that measures the viewpoint position of the user 2, a transparent display device 30 that displays the images generated by the image generation device 10, and a real-space information acquisition device 40 (hereinafter sometimes simply referred to as "acquisition device 40") that acquires environmental information of the real space (real world) that the user can see through the transparent display device 30 (information such as the position, shape, size, distance to objects existing in that real world). As an example, the image generation device 10 is a computer that executes a software program with a processor and also serves as the calculation unit (not shown) of the measuring device 20.

[0012] The transparent display device 30 in this embodiment is a so-called "transparent" display device, and can be composed of, for example, a transparent organic EL (Electro Luminescense) display (transparent OLED), a transparent liquid crystal display (transparent LCD), a transparent mini / microLED display, or a transparent screen onto which an image is projected from an external projector. The transparent display device 30 of this embodiment, composed of such a transparent display, is transparent or semi-transparent, allowing the user to see the environment of the real space on the opposite side through the transparent display device 30. Furthermore, by displaying an image on the transparent display device 30, it is possible to provide a visual experience in which virtual objects represented by the displayed image appear to exist in the real space visible through the transparent display device 30. The transparent display device 30 of this embodiment can be used as a device that is fixedly installed in any position, such as a window in a building, a window in a vehicle such as a car, bus, ship, or aircraft, or on a table indoors. In this case, virtual objects can be superimposed on the scenery of the real space outdoors or indoors visible through the transparent display device 30.

[0013] The measuring device 20 (sensor 22) is, for example, installed below, above, left, or right of the transparent display device 30 located in front of the user 2, and measures the user's viewpoint position from in front of the user 2.

[0014] The measuring device 20 positions the user 2's viewpoint position 2A in a predetermined reference coordinate system while the user 2 is within range of being able to see the transparent display device 30. The viewpoint position 2A is the position corresponding to the eye position. The specific viewpoint position 2A used in the processing is not particularly limited, but for example, the midpoint between the user 2's eyes, the center point of the head, or a position a predetermined distance inside the head from the center of the eyes can be used as the viewpoint position 2A.

[0015] In this embodiment, a use case in which the transparent display device 30 is installed in the window of a vehicle such as an automobile will be explained using a world coordinate system defined as fixed in real space and a local coordinate system defined relative to the vehicle. Both the world coordinate system and the local coordinate system are assumed to be orthogonal coordinate systems having three axes: x, y, and z. Once the position and orientation of the vehicle in the world coordinate system are determined, the relationship between the world coordinate system and the local coordinate system is established and mutual conversion becomes possible. How the world coordinate system is set relative to real space is arbitrary. Similarly, how the local coordinate system is set relative to the vehicle is also arbitrary.

[0016] Figure 2 is a schematic diagram showing an example of the image display system of this embodiment being mounted on a vehicle. In the example shown in Figure 2, the image display system 1, which includes an image generation device 10, a measuring device 20, a transparent display device 30, and a real-space information acquisition device 40, is mounted on a vehicle M, providing the occupants (users) 2, such as the driver or passengers of the vehicle M, with a visual experience that fuses real scenery with virtual images. As shown in Figure 2, the vehicle M of this embodiment is a small, single-seater passenger vehicle that is placed in a real space (real world) where various real objects (hereinafter sometimes referred to as "real objects") RO exist and travels within that real space. However, the vehicle M is not limited to a passenger vehicle, and may be a passenger car, bus, railway car, aircraft, ship, amusement park ride, etc.

[0017] Figure 2 shows an example of a local coordinate system based on the position and orientation of the vehicle, which is a Cartesian coordinate system with an X-axis pointing to the right of user 2, a Y-axis pointing upward, and a Z-axis pointing backward. The position and orientation of the sensor 22 (not shown in Figure 2) and the transmissive display device 30 in the measuring device 20 are fixed in this local coordinate system. Orientation is represented by Pitch around the X-axis, Yaw around the Y-axis, and Roll around the Z-axis.

[0018] The image generation device 10 displays a virtual object VO in the virtual space on the transparent display device 30 based on the position and orientation of the vehicle, the viewpoint position 2A of the user 2 inside the vehicle, and the fixed position and orientation of the transparent display device 30. At this time, the virtual object VO is placed in a three-dimensional virtual space defined in the world coordinate system, and the world coordinate system of the real space and the world coordinate system of the virtual space are defined by being superimposed so that their positional relationship and scaling ratio match. The user looks at the real space through the transparent display device 30 from viewpoint position 2A, but at the same time looks at the virtual space fixed in the same world coordinate system. At this time, the device performs calculation processing to generate an image to be displayed on the display screen so that the user 2 is made to believe that the virtual object VO in the virtual space exists in the real space at the same time, that is, so that it looks as if the virtual object VO exists in the real space. Specifically, three-dimensional CG data in a virtual space defined in three dimensions based on the coordinates of viewpoint position 2A is transformed into a two-dimensional image on a display screen, which is a two-dimensional surface, by performing projection transformation, perspective projection transformation, or similar calculation processing, and this image is displayed on the transparent display device 30. In this way, the three-dimensional information of the virtual space is transformed into two-dimensional information and presented to the user 2 with geometric precision.

[0019] The image generation device 10 stores in advance data of three-dimensional computer graphics (CG) representing virtual objects superimposed on the real space visible through the transparent display device 30, along with placement information in the virtual space, in its internal memory unit 12 (see Figure 5). It then generates a display image according to a calculation process to appropriately display the three-dimensional CG on the transparent display device 30 based on the viewpoint position 2A, so that the image appears geometrically precise, plausible, and natural, giving the user 2 the illusion that the virtual objects exist in the real space.

[0020] In this specification and in the claims, the term "image" is used to include not only one or more still images, but also moving images (videos) composed of multiple images that are sequentially arranged in time.

[0021] According to this embodiment, for example, an image in which a virtual object is superimposed on the real-world scenery visible from the window of a vehicle is generated according to the user's viewpoint position 2A and displayed on the transparent display device 30. This gives the user 2 the sensation that the virtual object is placed and exists simultaneously with real-world objects in real space within the real world. Furthermore, in this embodiment, instead of a head-mounted display (HMD) that moves with the user's head, a method is adopted in which the image is displayed on a screen fixed at a position other than the user's body part, including the user's head. Therefore, the display output unit 16 of the image generation device 10 can be used as a display device in mixed reality (MR) technology, allowing the user to experience realistic mixed reality similar to a head-mounted display, without causing the user 2 any inconvenience or discomfort associated with using a head-mounted display.

[0022] The image display system 1 of this embodiment will be described in more detail below.

[0023] The measuring device 20 continuously measures the viewpoint position 2A, for example, and the image generation device 10 generates a display image that follows the viewpoint position 2A and displays it on the screen of the transparent display device 30. As a result, when the user 2's head moves, the display image on the transparent display device 30 changes in accordance with the viewpoint position 2A that moves with it, giving the user 2 the sensation that a virtual object exists in real space. Image generation in accordance with tracking the viewpoint position 2A is achieved by continuously generating images that represent how a virtual object in virtual space would appear when viewed from the viewpoint position 2A at that moment via the display device. The method of image generation at this time is as described above. In addition, although it is described as tracking, it is not limited to this, and images may also be generated by predicting the viewpoint position 2A a little further ahead. If images are generated based on the current position of 2A when the viewpoint position 2A moves and the result is displayed, there will be a delay in image display due to the time required for calculation, which may cause the user to feel uncomfortable. To solve this, one possible method is to predict the position slightly ahead by taking time-series data of viewpoint position 2A and generate images in advance. Furthermore, while we have described a method for predicting the viewpoint position here, prediction is not limited to the viewpoint position; prediction calculations can also be used for the relative position of the local coordinate system with respect to the world coordinate system. For example, if we consider a car as a vehicle, the position of the vehicle slightly ahead can be predicted with high accuracy based on the direction of travel, speed information, and map information. This prediction result can then be reflected in the image generation described earlier.

[0024] Figure 3 is a diagram illustrating how the image display system according to this embodiment displays an image corresponding to the user's viewpoint position. In the explanation of Figure 3, the user space in front of the transparent display device 30 is defined as the real space, and the space behind it is defined as the virtual space, with the user's viewpoint position as the reference point. The virtual space defined behind the transparent display device 30 is displayed on the transparent display device 30 as an image visible from a position P1 where the user 2's head is located, through a pseudo-window (hereinafter also referred to as the "pseudo-window") provided by the transparent display device 30. Virtual objects in the virtual space are defined by three-dimensional CG data, which will be described later. In the example in Figure 3, six trees are arranged horizontally as virtual objects. Note that, in this example in Figure 3, the virtual space is defined behind the transparent display device 30 for illustrative purposes, but the virtual space may also be defined in front of the transparent display device 30, or the space including both the front and back of the transparent display device 30 may be defined as the virtual space. This allows the transparent display device 30 to display not only images in which virtual objects in the virtual space appear to be behind a virtual window created by the transparent display device 30, but also images in which virtual objects in the virtual space appear to be protruding from the transparent display device 30 towards the user.

[0025] When User 2 is at position P1, directly in front of the transparent display device 30, the field of view (FoV1) of the virtual space visible through the pseudo-window of the transparent display device 30 is wide, and all six trees are displayed on the transparent display device 30 as being within the field of view (Display D1). When User 2 moves from position P1 in the z direction to position P2, away from the transparent display device 30, the field of view (FoV2) of the virtual space visible through the pseudo-window of the transparent display device 30 narrows, and the field of view (FoV2) appears to contain only the entirety of three trees and parts of the trees on either side of them (Display D2). Furthermore, when User 2 moves from position P1 in the -x (minus x) direction to position P3, the field of view (FoV3) of the virtual space visible through the pseudo-window of the transparent display device 30 changes in the x direction. Only the three trees on the far right are contained in the field of view (FoV3). In addition, in field of view FoV3, the transparent display device 30 is viewed from an oblique angle rather than directly in front, but the horizontal thickness of the tree visible through the pseudo-window needs to be the same as when viewed from the front (display D3'). For this reason, when displaying the tree on the transparent display device 30, an image that has been appropriately stretched and compressed is displayed so that it appears to the user as display D3' (display D3). In this way, as a process to make the image appear plausible and natural to the user 2 so that they feel as if the virtual object in the virtual space is actually there, in this embodiment, when generating the image to be displayed on the transparent display device 30, a process is performed to project the virtual object in the virtual space defined in the three-dimensional CG data onto the transparent display device 30, i.e., a two-dimensional surface (for example, projection transformation, perspective projection transformation, or similar calculation). Alternatively, each point of the three-dimensional CG data in the reference coordinate space may be projected onto the point where the line connecting each point and the user's viewpoint intersects the transparent display device 30. Furthermore, as another method for generating images to be displayed on the transparent display device 30, arithmetic operations on specific matrices or numerical values, based on empirical rules, may be performed on the image and its three-dimensional parameters.

[0026] The measuring device 20 of this embodiment is configured to include an imaging unit that images the user 2, and a calculation unit that determines the viewpoint position 2A based on the imaging information of the user 2 captured by the imaging unit. The imaging unit is realized by a sensor 22 (see Figure 4) installed in the measuring device 20 shown in Figure 1. The calculation unit is realized by a processing unit 14 (see Figure 5) of the image generation device 10 shown in Figure 1. Alternatively, the calculation unit may be realized by a processing unit (not shown) within the measuring device 20.

[0027] As an example, in this embodiment, the sensor 22 (imaging unit) is a depth sensor that measures the depth from the sensor 22 to an object (in this case, the human body of user 2) at each pixel. The calculation unit, implemented by the processing unit 14 of the image generation device 10, estimates the shape of the human body based on the depth of each pixel measured by the sensor 22, and calculates the viewpoint position 2A based on the position of the head on that human body. With this method, since the shape of the human body is estimated from the depth of each pixel and the position of the head on that human body is used, the viewpoint position 2A can be identified with high accuracy even if the position and orientation of user 2's body change in various ways.

[0028] Figure 4 is a conceptual diagram illustrating the calculation of the viewpoint position. The position (Xs, Ys, Zs) and orientation (Pitchs, Yaws, Rolls) of the sensor 22 in the reference coordinate system are pre-set. The coordinates (Xh, Yh, Zh) of the viewpoint position 2A can be calculated from the depth of each pixel acquired by the sensor 22, and the position and orientation of the sensor 22. As shown in Figure 4, the position (Xm, Ym, Zm), orientation (Pitchm, Yawm, Rollm), and shape (Heightm, Widthm) of the transmissive display device 30 in the reference coordinate system are also pre-set. In this embodiment, as an example, each transmissive display device 30 is rectangular or trapezoidal, and its shape is expressed by its height (Heightm) and width (Widthm). The transmissive display device 30 may also be an arbitrary polygon, a curved surface, or a sphere.

[0029] The image captured by sensor 22 may be a depth image, or a depth image and a visible image. For example, the imaging unit may include a sensor that captures a depth image and a camera that captures a visible image, and the calculation unit may calculate the viewpoint position 2A using both the depth image and the visible image. Alternatively, there may be two visible images captured at different positions. In this case, the viewpoint position 2A may be calculated using the parallax between the two visible images. Alternatively, the viewpoint position 2A may be estimated from a single visible image by image processing using AI.

[0030] In the examples shown in Figures 1 and 4, the measuring device 20 (sensor 22) is shown to be installed above or below the transparent display device 30 in front of the user 2, but it is not limited to this. The measuring device 20 (sensor 22) can also be placed behind the user 2 in a position that does not overlap with the transparent display device 30. Alternatively, it is possible to install it behind the transparent display device 30 (back side, rear side) and measure the user's viewpoint position by passing through the display area of ​​the transparent display device 30. Furthermore, measurements may be taken using images captured by any of these multiple sensors, or the viewpoint position obtained by integrating information from multiple sensors may be used.

[0031] As described above, the real-space information acquisition device 40 in this embodiment is a device that acquires environmental information of the real space (real world) visible to the user through the transparent display device 30 (information such as the position, shape, size, and distance to objects existing in that real world). The acquisition device 40, as an example, is equipped with a LiDAR sensor and is capable of measuring the distance to objects and the shape of objects that exist in the real space around the transparent display device 30 as seen from the user's perspective. Here, LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) is a remote sensing technology that mainly uses laser light, and it measures scattered light from pulsed laser irradiation to acquire information such as the distance to an object at a distance and the shape of that object. The acquisition device 40 scans the real space around the transparent display device 30 with such a LiDAR sensor and acquires information such as the distance to objects and their shapes that exist in that space.

[0032] The acquisition device 40 may, in addition to the LiDAR sensor described above, or instead, include an imaging camera equipped with an image sensor that generates images (visible images and depth images). By analyzing the images acquired by capturing the real space visible through the transparent display device 30 with the imaging camera using any image processing technique, information regarding the position, size, shape, etc., of objects included in the image can be obtained. Examples of objects included in the image include buildings such as skyscrapers, structures such as bridges, distant mountains, moving vehicles, and living things that exist in real space.

[0033] The acquisition device 40 may also be equipped with a GPS (Global Positioning System) sensor. By equipping the acquisition device 40 with a GPS sensor, it is possible to acquire location information related to the location obtained by the LiDAR sensor and imaging camera of the acquisition device 40. This location information is also, for example, the location information in world coordinate space of a vehicle or the like on which System 1, including the acquisition device 40, is mounted.

[0034] The acquisition device 40 may also be configured to determine the vehicle's position and orientation using a Visual Positioning System (VPS). In this case, the acquisition device 40 captures images of the real space outside the vehicle, and the processing unit 14 of the image generation device 10 (see Figure 5) determines the vehicle's position and orientation based on these images. To this end, many images with identified positions and orientations are acquired in advance, visual features such as the outlines of objects like buildings appearing in these images are extracted, and these visual features are stored in the database of the storage unit 12 of the image generation device 10 (see Figure 5) as a searchable index along with the position and orientation information. Then, when images captured by the acquisition device 40 are input while the vehicle is in motion, the processing unit 14 of the image generation device 10 extracts visual features from the acquired images and compares the extracted visual features with the visual features stored in the database to determine the position and orientation in which the images were captured. This determined position and orientation in which the images were captured is then converted into the vehicle's position and orientation. This allows for the acquisition of the vehicle's position and orientation with high accuracy in real time. However, this acquisition process may involve a time delay in acquiring data in accordance with the vehicle's actual movement. To resolve this, the time delay can be solved by calculating velocity and acceleration vectors using the vehicle's current and past position and orientation, and then adding vehicle information (such as accelerator information, brake information, speed, and steering angle information) acquired directly from the vehicle without delay to predict the vehicle's future position and orientation.

[0035] In the above explanation, we described an example where visual features, along with location and orientation information, are stored in the database of the storage unit 12 (see Figure 5) of the image generation device 10 as a searchable index. However, visual features may also be stored in the database of an external device (not shown). In this case, the image data acquired by the acquisition device 40 is transmitted from the acquisition device 40 to the external device. The external device then extracts visual features from the image and identifies the location and orientation in which the image was captured by comparing the extracted visual features with the visual features stored in the database. The information regarding the identified location and orientation of the image is transmitted from the external device to the acquisition device 40.

[0036] Next, the image generation device 10 in this embodiment will be described. Figure 5 is a block diagram showing the configuration of the image generation device in this embodiment. As shown in Figure 5, the image generation device 10 includes a storage unit 12, a processing unit 14, a display output unit 16, and a communication unit 18. In Figure 5, the image generation device 10 is depicted as a single element, but the image generation device 10 does not necessarily have to be a single physical element, and may be composed of multiple physically separated elements.

[0037] The storage unit 12 includes temporary or non-temporary storage media such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and SDD (Solid State Drive). The storage unit 12 stores computer programs executed by the processing unit 14 and various data described later. The computer programs stored in the non-temporary storage media of the storage unit 12 include instructions for performing each process of the image generation method by the processing unit 14, as described later with reference to Figure 6, etc.

[0038] The memory unit 12 stores screen layout data indicating the position, orientation, and shape of the transparent display device 30 in a predetermined reference coordinate space, measuring device / acquisition device layout data indicating the position and orientation of the measuring device 20 and acquisition device 40 in the reference coordinate space, and three-dimensional CG data representing virtual objects superimposed on the real space visible through the transparent display device 30. The reference coordinate space is a space represented by a coordinate system (reference coordinate system) having a predetermined origin O that serves as the basis for calculations in this embodiment, and is a Cartesian coordinate system space having three axes: x, y, and z. How the reference coordinate space and its origin O are set is arbitrary; for example, the reference coordinate space may be fixed to the transparent display device 30, or fixed to the measuring device 20 and / or acquisition device 40, or fixed to the transparent display device 30, measuring device 20 and acquisition device 40, or not fixed to any of them. Furthermore, if a vehicle is assumed, the reference coordinate space may be processed separately from the coordinates of the outside world (world coordinate system) by using the coordinates within the vehicle (local coordinate system).

[0039] "Three-dimensional computer graphics (3D CG)" are virtual three-dimensional objects in three-dimensional space and differ from two-dimensional images in that they include depth (three-dimensional) information; therefore, they are also called 3DCG. As an example of three-dimensional CG, a virtual three-dimensional object is created by multiple faces (modeling) with points placed on a three-dimensional coordinate system in a virtual space as vertices. Information to reproduce materials and other properties is given to each face, and the object may be represented by illuminating it with arbitrary light intensity and light source position. The data of the three-dimensional CG includes information about the placement position of the virtual object displayed in the three-dimensional CG within the virtual space. The memory unit 12 can store three-dimensional CG data for various multiple virtual objects.

[0040] The processing unit 14 is composed of, for example, one or more CPUs (Central Processing Units). The processing unit 14 may also include one or more GPUs (Graphics Processing Units). The processing unit 14 executes the computer program stored in the memory unit 12 and performs each process of the image generation method described later, referring to Figure 6, etc.

[0041] The display output unit 16 is an output unit that outputs an image signal to the transparent display device 30, to which the transparent display device 30 is connected, and to which the image generated by the processing unit 14 is displayed on the transparent display device 30. The display output unit 16 can be configured with output terminals such as VGA, DVI, DisplayPort (trademark), HDMI (registered trademark), USB (trademark) Type-C, etc., and can be connected to the transparent display device 30 by wire. Alternatively, the display output unit 16 may be configured to be wirelessly connected to the transparent display device 30 using any wireless communication technology.

[0042] The communication unit 18 has the function of sending and receiving data to and from an external device of the image generation device 10. In particular, in this embodiment, the communication unit 18 may send and receive data of images captured by the sensor 22 of the measuring device 20, or data of information acquired by the acquisition device 40. The communication unit 18 may also transmit image data acquired by the acquisition device 40 to an external device, or receive information from the external device regarding the position and direction in which the image was captured, as identified by the external device. In addition, the communication unit 18 may acquire only the necessary parts of data from a three-dimensional real-space model that models objects existing in real space via a network. The communication between the external device and the communication unit 18 may be either wired communication or wireless communication.

[0043] Next, as an example, the image generation method by the image generation device 10 of this embodiment will be described with reference to Figure 6, etc. Figure 6 is an example of a flowchart showing the overall processing by the image generation device of this embodiment. Each process in Figure 6 is executed by the processing unit 14. If the image generated by the image generation device 10 is a moving image consisting of multiple frames that are sequentially arranged in time, the calculation may be performed for each frame of the moving image. In addition, in the calculation process of each step of the flowchart, prediction calculations may be performed using past data in the time series and the image may be generated.

[0044] Referring to Figure 6, the processing unit 14 of the image generation device 10 first acquires information about the real space that user 2 can see through the transparent display device 30 (step S101).

[0045] The processing unit 14 controls the operation of the acquisition device 40, causing the acquisition device 40 to acquire information about the real space behind the user 2 as seen from the transparent display device 30. As a specific example of this process, the processing unit 14 first acquires GPS location information from the acquisition device 40 to determine the approximate location of the vehicle equipped with System 1, including the acquisition device 40, and then identifies VPS information (the vehicle's more precise location and direction) from the acquisition device 40. Next, the processing unit 14 acquires data from a three-dimensional real-space model that is a three-dimensional model of an object existing in real space. Objects existing in real space are, for example, buildings, houses, bridges, railways, roads, and other structures in the real world. The data from the three-dimensional real-space model that is a three-dimensional model of such an object includes information for reproducing the object in virtual space, such as the location information of each object in the real world and information about the shape and size of each object. Currently, 3D city model data that is a three-dimensional model of real space for the shapes of several cities and natural areas is provided by multiple businesses and organizations. Services that provide 3D city model data include, for example, the Ministry of Land, Infrastructure, Transport and Tourism's "Project PLATEAU" and Zenrin Co., Ltd.'s "3D Map Data." In the future, it is expected that 3D model data for more cities and natural areas will be provided by various businesses and organizations. In this embodiment, as an example, "three-dimensional real-space model data" can be obtained by using a service provided by a business or organization. The communication unit 18 of the image generation device 10 connects to the service provider server of the business or organization via the internet, and the three-dimensional real-space model data can be obtained by downloading at least the necessary portion of the three-dimensional real-space model data from the server. Alternatively, if the three-dimensional real-space model data has already been obtained, it can be used without connecting to the internet temporarily or permanently. The obtained three-dimensional real-space model data is stored in the storage unit 12.

[0046] In addition to the above processing, or in lieu thereof, the processing unit 14 may acquire information about the real space behind the user 2 of the transparent display device 30 using a sensor (e.g., a LiDAR sensor) provided by the acquisition device 40, and acquire information about objects in that real space, such as the distance from the acquisition device 40 to the object, and information about the size and shape of the object. If there are multiple objects, information about the distance, size and shape of each object will be acquired. If the acquisition device 40 is equipped with an imaging camera or a depth sensor, an image of the real space behind the user 2 of the transparent display device 30 may be captured by the imaging camera or acquired as a depth image. The processing unit 14 can acquire or estimate information about the position and shape of objects included in the image (e.g., buildings such as skyscrapers, structures such as bridges, traffic lights, and telephone poles, distant mountains, etc.) by analyzing the acquired image of the real space using any known image analysis technique and / or artificial intelligence technique using machine learning.

[0047] The processing unit 14 then acquires the viewpoint position 2A of user 2 (step S102). The processing unit 14 uses the measuring device 20 to calculate the viewpoint position 2A of user 2 using the method described with reference to Figure 4, etc.

[0048] Furthermore, the processing in step S101 and step S102 by the processing unit 14 may be executed in a different order, or they may be executed in parallel. Also, if the subsequent steps are performed based on data acquired or estimated in the past in the time series, this part may be skipped as the necessary data has already been acquired without performing both processes.

[0049] The processing unit 14 then performs a process to generate a display image on the transparent display device 30 that displays virtual objects superimposed on the real space surrounding the transparent display device 30 as seen by user 2 through the transparent display device 30 (step S103).

[0050] The processing unit 14 continuously determines the position and orientation of the vehicle in the world coordinate system based on real-space environment information from the acquisition device 40, and also continuously determines the viewpoint position 2A. The processing unit 14 generates a display image for displaying virtual objects on the screen of the transparent display device 30 with geometrically precise accuracy, so that when viewed from the user 2's viewpoint position 2A through the screen of the transparent display device 30, the virtual object appears to coexist in the real world. The display image, when displayed on the transparent display device 30, is an image that represents how a virtual object in the virtual space would appear when viewed from viewpoint position 2A through the transparent display device 30. The display image displayed on the transparent display device 30 changes in accordance with the movement of the vehicle and the movement of the viewpoint position 2A, which moves with the movement of the user 2's head inside the vehicle. This gives the user 2 the sensation that a virtual object exists in real space. Tracking the viewpoint position 2A is achieved by continuously generating images of the rendering content as seen from viewpoint position 2A at that moment.

[0051] This computational process, which converts 3D information into 2D information, projects the virtual object's display image onto the screen of the transparent display device 30 with geometrically precise accuracy, taking into account the user's viewpoint position 2A. Therefore, even if user 2 views the screen of the transparent display device 30 from an oblique angle, the display image presented to the transparent display device 30 is one that appears to be a plausible and natural image, as if the virtual object were actually present at that viewpoint position 2A. The processing performed here is not limited to projection transformation. Perspective projection transformation methods may also be used, as well as similar computational methods or arithmetic operations on specific matrices or numerical values ​​based on empirical rules.

[0052] Furthermore, the virtual object may be displayed on the transparent display device 30 in such a position that, when viewed from a certain viewpoint position 2A, it partially or completely overlaps with real-space objects existing in the real space surrounding the transparent display device 30. In this case, the overlap between the two can be accurately represented by performing a process to occlude part or all of the virtual object based on three-dimensional real-space model data corresponding to the object in real space, or three-dimensional data of the object constructed based on the distance and shape to the object acquired in real time by the real-space information acquisition device 40, before reflecting the virtual object in the display.

[0053] The process of occluding part or all of a virtual object includes making transparent (or deleting, removing, or making invisible) the area of ​​the virtual object that is occluded by the real-space object when the virtual object is viewed from a certain viewpoint position 2A and part or all of the real-space object is in front of the virtual object. More specifically, when the distance from the viewpoint position 2A to the part constituting the virtual object is greater than the distance from the viewpoint position 2A to the part constituting the real-space object, a display image of the virtual object is generated with the part corresponding to the occluded area occluded by the part constituting the real-space object.

[0054] The processing unit 14 transmits the signal of the display image generated in this manner to the transparent display device 30 via the display output unit 16 of the image generation device 10, causing the transparent display device 30 to display the display image (step S104).

[0055] Thus, according to System 1 of this embodiment, the display image of a virtual object displayed as three-dimensional computer graphics superimposed on the real space visible through the transparent display device 30 is continuously generated in accordance with the change in the user's viewpoint position 2A, and a composite realistic moving image is provided to the user 2 via the transparent display device 30. By superimposing the three-dimensional computer graphics virtual object onto the real-world landscape, the user 2 can be made to feel as if the virtual object is present together with the real-world space visible through the transparent display device 30.

[0056] In particular, according to this embodiment, not only positional information but also information on size and shape of objects in the real space behind the transparent display device 30 as seen from the user's perspective is acquired. Therefore, it is possible to simulate the front-to-back relationship and relative positional relationship when virtual objects superimposed on real objects are virtually placed in the same space. It is possible to generate images that reflect this, including representations using occlusion processing, where virtual objects are partially visible or hidden from the user's viewpoint. As a result, virtual objects can be superimposed on real objects with higher quality, providing the user 2 with a more believable and natural image that creates the illusion that the virtual objects exist in real space.

[0057] In the above description, as one example of the application of System 1 according to this embodiment, a use case was explained in which the transparent display device 30 is used as a window of a vehicle and virtual objects are superimposed on the outside scenery visible through the window. However, the application of System 1 is not limited to this example, and it can be applied to any other desired use.

[0058] Other examples of applications for System 1 include a use case where the transparent display device 30 is used as a window in a building, and virtual objects are superimposed on the outside scenery visible through the window, or a use case where the transparent display device 30 is placed on a table or the like indoors, and virtual objects are superimposed on real-space objects in the room (such as shelves) visible through the transparent display device 30. In these use cases, System 1, including the transparent display device 30, is fixed in place and does not move in the world coordinate space. Therefore, the local coordinate system of System 1 is fixed in the world coordinate system. Accordingly, in these use cases, the GPS sensor and VPS-related functions for determining the position of System 1 in the local coordinate system among the acquisition device 40 of this embodiment described above may be omitted.

[0059] Furthermore, in these use cases, the relative position between the transparent display device 30 and the shape of real-space objects such as scenery (buildings outside the window, etc.) or interior scenery (shelves, etc.) visible through the transparent display device 30 does not change dynamically. Therefore, if environmental information of the real space (real world) is acquired in advance, the acquisition device 40 may be omitted. In this case, a device similar to the acquisition device 40 or a method such as photogrammetry may be used to acquire three-dimensional information of the surrounding real space in advance. Alternatively, the content creator may set the position and shape of scenery and real-space objects visible through the transparent display device 30 on the image generation device 10 and store them in the storage unit 12 as environmental information of the real space. In this case, three-dimensional real-space model data such as 3D city model data may be accurately pre-positioned in the virtual space based on its relative position to the transparent display device 30 fixedly placed on the world coordinate system, and stored in the storage unit 12 as environmental information of the real space. Due to such pre-processing, S101 in the flowchart of Figure 6 may be omitted in these use cases.

[0060] [Second Embodiment] Next, a second embodiment of the present disclosure will be described.

[0061] The first embodiment described above showed an example in which a viewpoint position 2A for one user 2 is detected and a display image of a virtual object is generated according to that viewpoint position 2A. In contrast, this embodiment provides a means for displaying a virtual object superimposed in real space for multiple users 2. In the first embodiment, the image generation device 10 pre-stores three-dimensional CG data for generating a display image representing a virtual object in a storage unit 12, and generates a display image from the three-dimensional CG data by a calculation process that generates a geometrically precise image that appears plausible and natural to the user 2 so as to give the illusion that the virtual object is there. The second embodiment provides, in addition to the configuration and operation of the system 1 of the first embodiment, a means for presenting the display image of the virtual object as a three-dimensional image (stereoscopic image) consisting of two images that give parallax between the left and right eyes.

[0062] In this embodiment, a means is provided for presenting a virtual object display image as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes, to a user with uncorrected vision who does not wear a head-mounted display or 3D glasses.

[0063] Figure 7 is a schematic diagram illustrating the configuration of the display device in the system of the second embodiment.

[0064] As shown in Figure 7, the transmissive display device 30 of this embodiment is a display that realizes naked-eye stereoscopic viewing, for example, by attaching a lenticular lens to the surface of a transmissive organic EL display (OLED), which can simultaneously display different images in multiple display directions. The transmissive display device 30 of this embodiment comprises a transparent substrate 301 having a plurality of pixel units 302 arranged in an array, and a plurality of lens elements 303 provided on each pixel unit 302 on the transparent substrate 301. The transparent substrate 301 in this embodiment is, for example, a transparent flat plate, and may be a transparent resin plate including an acrylic plate, or it may be glass. Any material is acceptable as long as it has a certain degree of transparency. It may also have a shape that is basically planar but has a certain degree of curvedness, such as the transparent glass used in the windshield of an automobile. In addition, each lens element 303 in this embodiment is, for example, a semi-cylindrical lens called a lenticular lens. As the lens element 303, in addition to a lenticular lens, any three-dimensional lens shape such as a hemispherical semi-ellipsoid or toroidal shape that controls the direction of light may be used. Alternatively, instead of using a lens, a film or plate that generates a parallax barrier may be used.

[0065] For example, the lens elements 303 can be mounted on each pixel unit 302 by individually manufacturing them and placing them at the positions where each pixel unit 302 is located on the transparent substrate 301. Alternatively, the lens elements 303 can be mounted on each pixel unit 302 by using 3D printing technology to extrude transparent material at the positions where each pixel unit 302 is located on the transparent substrate 301 to form the lens elements 303. Furthermore, the lens elements 303 can also be mounted on each pixel unit 302 by pre-forming a transparent film member on which multiple lens elements 303 positioned relative to each pixel unit 302 on the transparent substrate 301, and then attaching the transparent film member to the transparent substrate 301 in a predetermined position. In addition, the transparent substrate 301 may be constructed by inserting an array of pixel units and electrode wires that supply power and signals to the pixel units into the interlayer portion of a plate with a structure in which multiple transparent substrates are bonded together, and simultaneously inserting a film or plate containing lens elements and a film or plate that creates a parallax barrier. Alternatively, the transparent substrate 301 may be constructed by inserting only the array-shaped pixel units into the interlayer, and then a film or plate containing lens elements, or a film or plate that creates a parallax barrier, may be attached to the outside. Here, the transparent substrate may be a transparent resin plate including an acrylic plate, or it may be glass. Any material with a certain degree of transparency is acceptable.

[0066] Each pixel unit 302 and lens element 303 is arranged at a predetermined distance from each other. In areas where no pixel units 302 or lens elements 303 are located, the real space behind the transparent display device 30 can be seen through the transparent substrate 301. In the transparent display device 30 of this embodiment, the pixel units 302 and lens elements 303 are arranged relatively "sparsely". For example, the proportion of the entire screen of the transparent display device 30 that the pixel units 302 and lens elements 303 occupy is several tens of percent to several percent, and if the size of the pixel units can be made smaller with future technological advancements, it can be reduced to several percent or less. By reducing the size of each individual pixel unit or increasing the light intensity of each individual pixel unit, it becomes possible to display images that are visible to the user without being overwhelmed by ambient light, while maintaining transparency. In applications such as the present invention, where virtual objects are superimposed on the real space visible through the transparent display device 30, sufficient resolution and clarity of the image can be provided to the user viewing the virtual objects.

[0067] The transparent substrate 301 is provided with electrode wires made of a highly transparent conductive material, or very fine metal electrode wires (not shown), which are electrically connected to each pixel unit 302, for example. Each pixel unit 302 is individually driven to emit light or controlled to transmit light by power and drive signals supplied through these electrodes. Here, each pixel unit may be a self-emissive element such as an organic EL, mini-LED, or micro-LED, or it may be an element that controls light transmission using the mechanism of liquid crystal. It may also be an element or light-emitting material that reflects externally projected light to allow the user to perceive color differences.

[0068] Furthermore, the configuration and operation of each part of System 1 according to this embodiment are generally the same as those of System 1 described in the first embodiment, but any differences in configuration and operation will be described below.

[0069] Next, with reference to Figures 8 and 9, a more detailed configuration of the transparent display device 30 of this embodiment and the display operation control by the transparent display device 30 will be described.

[0070] In the transparent display device 30 of this embodiment, as an example, each pixel unit 302 is configured as a collection of units of 45 pixels. That is, one pixel unit 302 consists of 45 pixels, and an RGB display element 302' is arranged in each pixel. The RGB display element 302' consists of three subpixel display elements that display three colors: R (red), G (green), and B (blue). Alternatively, the RGB display element 302' may be a display element for four colors. A pixel unit 302 consisting of 45 pixels corresponds to one pixel in a typical display.

[0071] In addition, as an example, each pixel unit 302 is equipped with a lens element 303, which is a semicircular cylindrical lens called a lenticular lens, that refracts the direction of each optical path of the 45 RGB lights displayed by the 45 RGB display elements 302' for the 45 pixels within the pixel unit 302, in 45 display directions, for example, 0 to 44, as shown in Figure 9 (hereinafter, these directions will be referred to as "display directions 501").

[0072] Then, for each of the different viewpoint positions 2A of each user 21 to 24, the image generation device 10 inputs the data of the display image it generates corresponding to those viewpoint positions 2A1 to 2A4 to the RGB display elements 500 with pixel numbers corresponding to directions 502A, 502B, 502C, and 502D from a predetermined origin O in the reference coordinate space (Figure 4) (for example, the center position of the transmissive display device 30) toward the viewpoint positions 2A1 to 2A4 of each user 21 to 24.Hereafter, these directions will be referred to as user viewpoint position directions 502A, 502B, 502C, and 502D. Alternatively, these directions will be collectively referred to as user viewpoint position directions 502.

[0073] Here, the processing unit 14 of the image generation device 10 may generate a right-eye display image and a left-eye display image for each viewpoint position 2A1 to 2A4 of each user 21 to 24 detected by the measuring device 20, such that when a user 2 views a three-dimensional virtual object in virtual space through the transparent display device 30 from their right eye and left eye, respectively, at a certain distance to the left and right of that viewpoint position, the image generation device 10 may generate a right-eye display image and a left-eye display image so that the image appears geometrically precise and natural, creating the illusion for each user 2 that the virtual object exists in real space. When generating the right-eye display image and the left-eye display image to be displayed on the screen of the transparent display device 30, the image generation device 10 may perform projection transformations, perspective projection transformations, or similar calculation processes to project the virtual object, represented by three-dimensional CG data, onto the screen of the transparent display device 30, i.e., a two-dimensional surface, based on the viewpoint positions 2A1 to 2A4 of each user 21 to 24.

[0074] In this case, for each viewpoint position 2A1 to 2A4 of each user 21 to 24, the processing unit 14 displays the data of the right-eye display image and the left-eye display image generated in accordance with each viewpoint position on the RGB display elements 302' with pixel numbers corresponding to the right-eye display direction and the RGB display elements 302' with pixel numbers corresponding to the left-eye display direction, respectively, located on both sides of each user viewpoint position direction 502A, 502B, 502C, 502D corresponding to each viewpoint position.

[0075] As a result of the above control operations, for each user 21-24 in Figure 8, the display state of the virtual object is controlled for each user 2, as shown by the line of sight direction 500 connecting each right eye and each left eye to each lens element 303, and as explained by the displays D1, D2, D3, etc. in Figure 3. This image can be simultaneously displayed to multiple users 21-24 at various positions on the transparent display device 30. In this case, for each 21-24, an image with binocular parallax is incident on their respective right and left eyes, allowing them to experience realistic stereoscopic vision without glasses.

[0076] In this embodiment, the measuring device 20 detects the viewpoint position 2A of each user 2, and the displayed image of the virtual object Vobj is subjected to projection transformation, perspective projection transformation, or similar calculation processing according to the viewpoint position 2A of each user 2.

[0077] Figure 10 is a flowchart showing a detailed example of the display processing in step S104 of Figure 6, as described in the first embodiment. This processing will be described below in accordance with the explanatory diagram in Figure 9 described above. The processing unit 14 first sets the left-eye display image data, generated in accordance with the left-eye display direction of the first viewpoint position 2A (for example, 2A1 in Figure 9), in the RGB display elements 302' corresponding to each pixel number of each display direction 501, for example, from the 0th to the 44th display directions 501 described in Figure 9, between the 0th display direction 501 and the user viewpoint position direction 502 (for example, 502A in Figure 9) that passes through the first viewpoint position 2A (for example, 2A1 in Figure 9) among the one or more viewpoint positions 2A calculated in step S102 of Figure 6 (step S901). This setting process is performed on the RGB display elements 302' of each pixel number in each pixel unit 302 (see Figure 8) that constitutes the transmissive display device 30.

[0078] Next, the processing unit 14 increments the variable value indicating the viewpoint position 2A, which is stored in the memory unit 12 of the image generation device 10, for example in RAM or registers (not shown), by 1 (step S905).

[0079] Next, the processing unit 14 determines whether the variable value indicating the viewpoint position 2A has exceeded a predetermined last user viewpoint position (step S906). The value indicating the last viewpoint position 2A can be predetermined according to the number of users 2 using System 1 and stored in the RAM or registers mentioned above. In step S906, the processing unit 14 determines whether the variable value indicating the viewpoint position 2A has exceeded the predetermined value stored above.

[0080] If the processing has not yet reached the final user viewpoint position and the determination in step S906 is NO, the processing unit 14 transfers control to the processing in step S902 and executes the processing from steps S902 to S904.

[0081] First, the processing unit 14 calculates the midpoint direction 503 (for example, 503AB in Figure 9) that passes through the midpoint between the previous viewpoint position 2A (for example, 2A1 in Figure 9) and the current viewpoint position 2A (for example, 2A2 in Figure 9) (step S902).

[0082] Next, the processing unit 14 sets the right-eye display image data generated in accordance with the right-eye display direction of the previous user viewpoint position 2A (for example, 2A1 in Figure 9) in the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, numbers 4 to 10 in Figure 9, between the user viewpoint position direction 502 (for example, 502A in Figure 9) passing through the previous user viewpoint position 2A (for example, 2A1 in Figure 9) and the midpoint direction 503 (for example, 503AB in Figure 9) calculated this time (step S903). This setting process is performed on the RGB display elements 302' of each pixel number in each pixel unit 302 (see Figure 8) that constitutes the transparent display device 30.

[0083] Next, the processing unit 14 sets the left-eye display image data generated in accordance with the display direction of the left eye at the current user viewpoint position 2A (for example, 2A2 in Figure 9) to the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, numbers 11 to 16 in Figure 9, between the midpoint direction 503 (for example, 503AB in Figure 9) calculated in step S902 and the user viewpoint position direction 502 (for example, 502B in Figure 9) that passes through the current user viewpoint position 2A (for example, 2A2 in Figure 9) (step S904). This setting process is performed on the RGB display elements 302' of each pixel number in each pixel unit 302 (see Figure 8) that constitutes the transparent display device 30.

[0084] Subsequently, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1 (step S905), and determines whether that variable value has exceeded the predetermined last user viewpoint position 2A (step S906).

[0085] If the final viewpoint position 2A has not yet been processed and the determination in step S906 is NO, the processing unit 14 will execute the processes from steps S902 to S904 described above again.

[0086] By step S902, as described above, the midpoint direction 503BC passing through the midpoint between the previous viewpoint position 2A2 and the current user viewpoint position 2A3 is calculated, for example in Figure 9.

[0087] Next, in step S903, similar to that described above, for example in Figure 9, the right-eye display image data generated in accordance with the right-eye display direction of the previous viewpoint position 2A2 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 from 17 to 21 in Figure 9, between the user viewpoint position direction 502B passing through the previous viewpoint position 2A2 and the midpoint direction 503BC calculated this time. This setting process is performed on the RGB display elements 302' of each pixel number in each pixel unit 302 (see Figure 8) that constitutes the transparent display device 30.

[0088] Next, in step S904, similar to the one described above, for example in Figure 9, the left-eye display image data generated to correspond to the left-eye display direction at the current viewpoint position 2A3 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from 22 to 26 in Figure 9, between the midpoint direction 503BC calculated in step S902 and the user viewpoint position direction 502C passing through the current viewpoint position 2A3. This setting process is performed on the RGB display elements 302' of each pixel number in each pixel unit 302 (see Figure 8) that constitutes the transparent display device 30.

[0089] Subsequently, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1 (step S905), and determines whether that variable value has exceeded the predetermined last user viewpoint position 2A (step S906).

[0090] If the final viewpoint position 2A has not yet been processed and the determination in step S906 is NO, the processing unit 14 will execute the processes from steps S902 to S904 described above again.

[0091] By step S902, as described above, the midpoint direction 503CD passing through the midpoint between the previous viewpoint position 2A3 and the current viewpoint position 2A4 is calculated, for example in Figure 9.

[0092] Next, in step S903, similar to that described above, for example in Figure 9, the right-eye display image data generated in accordance with the right-eye display direction of the previous viewpoint position 2A3 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 from 27 to 30 in Figure 9, between the user viewpoint position direction 502C passing through the previous user viewpoint position 2A3 and the midpoint direction 503CD calculated this time. This setting process is performed on the RGB display elements 302' of each pixel number in each pixel unit 302 (see Figure 8) that constitutes the transparent display device 30.

[0093] Next, in step S904, similar to the one described above, for example in Figure 9, the left-eye display image data generated to correspond to the left-eye display direction at the current viewpoint position 2A4 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from 31 to 34 in Figure 9, between the midpoint direction 503CD calculated in step S902 and the user viewpoint position direction 502D passing through the current user viewpoint position 2A4. This setting process is performed on the RGB display elements 302' of each pixel number in each pixel unit 302 (see Figure 8) that constitutes the transparent display device 30.

[0094] Subsequently, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1 (step S905), and determines whether that variable value has exceeded the predetermined last viewpoint position 2A (step S906).

[0095] After processing has been completed up to the last user viewpoint position 2A, if the determination in step S906 is YES, the processing unit 14 executes the processing in the final step S907. In step S907, the processing unit 14 sets the right-eye display image data generated in accordance with the right-eye display direction of the last user viewpoint position 2A (for example, 2A4 in Figure 9) in the RGB display elements of each pixel number corresponding to each display direction 501, for example from 35 to 44 in Figure 9, between the user viewpoint position direction 502 (for example, 502D in Figure 9) passing through the current viewpoint position 204 (for example, 2A4 in Figure 9) and the 44th display direction 501.

[0096] Subsequently, the processing unit 14 terminates the display process of step S104 in Figure 6, as illustrated in the flowchart of Figure 10.

[0097] In the image display process illustrated in the flowchart of Figure 10, in Figure 9, the display image data for the right eye or left eye, generated to correspond to the display direction of the right or left eye of the adjacent viewpoint position 2A, is copied and set in each display direction 501 of the section delimited by the user viewpoint position direction 502 of the two viewpoint positions 2A and the midpoint direction 503 passing through the midpoint between those viewpoint positions 2A. Alternatively, by changing the algorithm for mapping display pixels to each display direction 501 between the viewpoint position 2A of user 2 that is close to the transparent display device 30 and the viewpoint position 2A of user 2 that is far from the transparent display device 30, the sensitivity of the display pixel changes can be made uniform according to the viewpoint position A of user 2. Or, instead of simply copying, display pixels calculated by interpolation processing according to each display direction 501 may be set between two adjacent display images. Furthermore, the specific viewpoint position 2A acquired by the measuring device 20 described above and used for processing is not particularly limited, but for example, the midpoint between each user 2's eyes, the center point of the head, or a position a predetermined distance inside the head from the center of both eyes can be used as the viewpoint position 2A. Another example is that the positions of each user 2's left and right eyes may be acquired separately and used for processing. Based on the separately acquired left and right eye position information, the displayed image of the virtual object Vobj may be subjected to projection transformation, perspective projection transformation, or similar calculation processing according to the left and right eye positions of each user 2.

[0098] Thus, according to this embodiment, for multiple users 2 who do not wear a head-mounted display or 3D glasses, at each of the viewpoint positions 2A1 to 2A4 of each user 21 to 24, for example, when the right and left eyes of users located a certain distance to the left and right of that viewpoint position view a three-dimensional, stereoscopic virtual object in the virtual space through the transparent display device 30, the display image of the virtual object can be superimposed onto the real space visible through the transparent display device 30 as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes, so that it appears to each user 2 as if the virtual object exists in the real space, with geometric rigor and plausibility. In addition, according to this embodiment, different display images of the virtual object corresponding to each user 2's viewpoint position 2A can be simultaneously presented to each user 2.

[0099] One example of applying the transparent display device 30 in this embodiment is to mount the transparent display device 30 on the glass portion of an automobile or the like, and present a three-dimensional display image of a virtual object to each of multiple users 2 seated in the front and rear seats of the vehicle. With the transparent display device 30 of this embodiment, it is possible to individually display a display image of a virtual object to multiple users 2 who have different distances and viewing directions from the transparent display device 30. Therefore, it is possible to superimpose a virtual object onto the real space visible around the transparent display device 30, which is the glass of a moving automobile, according to the view from each user 2's viewpoint.

[0100] (modified version) Figure 11 shows a modified example of the display device in the system of the second embodiment.

[0101] In the display device 30 of the second embodiment shown in Figure 7, etc., a lens element 303 is provided individually for each pixel unit 302. In contrast, as shown in this modified example, the lens element 303 may have a vertically elongated shape and be arranged across multiple pixel units 302 in each row of the matrix-arranged plurality of pixel units 302. This makes it possible to reduce the manufacturing man-hours for the lens element 303 compared to the case where a lens element 303 is provided individually for each pixel unit 302.

[0102] [Third Embodiment] In the second embodiment described above, an image display system was illustrated that shows a parallax-laden three-dimensional image of a virtual object to multiple users using a transparent display. In the transparent display in the second embodiment, a predetermined number of pixels arranged horizontally on a transparent substrate are grouped together as a pixel unit (pixel cluster), multiple pixel units are arranged in a matrix, and a lens element of a cylindrical lens with the same width as the pixel unit is provided on the pixel unit. In the third embodiment, an image display system using a transparent display with a different configuration from the second embodiment is illustrated. However, since the configuration and operation of the image display system in the third embodiment are generally the same as those of the third embodiment, the following description will focus on the configuration and operation that differ from the second embodiment.

[0103] <Overview of display configuration and manufacturing method> Figure 12 is a schematic diagram illustrating the configuration of the display device in the system of the third embodiment. Figure 12 shows a front view of the screen of the display device.

[0104] The transparent display device 30 of this embodiment, like that of the second embodiment, is transparent or semi-transparent and is a display capable of simultaneously displaying different images in multiple display directions.

[0105] As shown in Figure 12, the transparent substrate 301 has a plurality of pixel rows 300 arranged parallel to each other with a predetermined vertical spacing, and a plurality of lens elements 303 that cross the plurality of pixel rows 300 in the vertical direction (perpendicular direction).

[0106] The transparent substrate 301 is, for example, a transparent flat plate, and may be a transparent resin plate including an acrylic plate, or it may be glass. Any material is acceptable as long as it has a certain degree of transparency. It may also have a shape that is basically flat but has a certain degree of curvature, like the transparent glass used in the windshield of an automobile. For convenience, in the following explanation, such shapes will be included and described as "flat," but in the example using a curved plate, the same explanation will hold if the "flat" part is replaced with "tangent plane."

[0107] Within the pixel array 300, multiple pixels (not shown) are arranged in the longitudinal direction (horizontal direction). Each pixel is, for example, an RGB organic EL element. The lens element 303 has a width in the short direction (horizontal direction) corresponding to a predetermined number of pixels in the pixel array 300, and is a semicircular cylindrical lens that acts as a lens in the short direction. Here, for example, the width of the lens element 303 in the short direction corresponds to 45 pixels. For example, the lens element 303 has its longitudinal direction in the vertical direction of the screen and is arranged with predetermined intervals in the horizontal direction.

[0108] The OLED material for the pixel array 300 is attached to the surface of the transparent substrate 301. The OLED material is attached to the transparent substrate 301 by, for example, a vapor deposition method or a printing method.

[0109] For example, the lens elements 303 can be individually manufactured and placed at predetermined positions on the transparent substrate 301 on which the pixel rows 300 are arranged, so as to intersect with each pixel row 300. Alternatively, the lens elements 303 can be provided so as to intersect with each pixel row 300 by using 3D printing technology to extrude transparent material at predetermined positions on the transparent substrate 301 on which the pixel rows 300 are arranged to form a cylindrical lens. Furthermore, the lens elements 303 can also be provided so as to intersect with each pixel row 300 by forming a transparent film member in which multiple lens elements 303 are positioned and arranged as a single unit using a method such as heat press molding with a mold, or by forming a transparent film-glass integrated member on a glass substrate by pouring UV-curable resin onto a glass substrate, pressing a mold onto it, and then UV curing, and then attaching it to the transparent substrate 301 on which the pixel rows 300 are arranged at predetermined positions.

[0110] Alternatively, the transparent substrate 301 may be manufactured by inserting an array of pixel units and electrode wires that supply power and signals to the pixel units into the interlayer portion of a plate with a structure in which multiple transparent substrates are bonded together, while simultaneously inserting a film or plate containing a lens element and a film or plate that creates a parallax barrier.

[0111] Alternatively, a method may be used in which only the pixel array 300 is inserted into the interlayer to form a transparent substrate 301, and then a film or plate containing the lens element 303 is attached to the outside.

[0112] Each pixel array 300 is arranged at a predetermined interval from one another. Similarly, each lens element 303 is also arranged at a predetermined interval from one another. As a result, multiple lens elements 303 and multiple pixel arrays 300 intersect with each other, forming a grid. In areas where neither pixel arrays 300 nor lens elements 303 are present, the real space behind the transmissive display device 30 can be seen through the transparent substrate 301. Furthermore, in areas where no pixel arrays 300 are present but lens elements 303 are, the real space behind the transmissive display device 30 can be seen to some extent through the lens elements 303 and the transparent substrate 301.

[0113] In the transparent display device 30 of this embodiment, the pixel arrays 300 and lens elements 303 are arranged relatively "sparsely". For example, the proportion of the entire screen of the transparent display device 30 that is occupied by the pixel arrays 300 and lens elements 303 is several tens of percent to several percent, and if the size of the pixel units can be made smaller with future technological advancements, it can be reduced to several percent or less. If the size of each individual pixel unit can be reduced or the light intensity of each individual pixel unit can be increased, it will be possible to display images that are visible to the user without being overwhelmed by ambient light, while maintaining transparency. In applications such as the present invention, where virtual objects are superimposed on the real space visible through the transparent display device 30, it is possible to provide the user viewing the virtual objects with sufficient resolution and clarity in the image.

[0114] The transparent substrate 301 is provided with, for example, electrode wires made of a highly transparent conductive material or very fine metal electrode wires (not shown) electrically connected to each pixel of each pixel row 300. Each pixel of each pixel row 300 is individually driven to emit light or its light transmission is controlled by power and drive signals supplied through these electrodes.

[0115] Next, we will describe the more detailed configuration of the transparent display device 30 of this embodiment, the display operation control by the transparent display device 30, and the calibration of the transparent display device 30.

[0116] <Details of display configuration and overview of display processing> Figures 13 and 14 illustrate the more detailed configuration of the display device of the third embodiment and the display operation control by the display device.

[0117] In the transmissive display device 30 of this embodiment, as an example, 45 consecutive pixels (RGB display elements) 302' in the portion of the pixel row 300 that overlaps with the lens element 303 in a direction orthogonal to the screen are treated as a set of units called pixel units 302. However, the number of pixels in pixel unit 302 being 45 is just an example, and the number of pixels in pixel unit 302 is not limited to this.

[0118] In the third embodiment, similar to the second embodiment, multiple pixel units are arranged in a matrix. That is, one pixel unit 302 consists of 45 pixels, and each pixel is equipped with an RGB display element 302'. The RGB display element 302' is composed of three subpixel display elements that display three colors: R (red), G (green), and B (blue). Alternatively, the RGB display element 302' may be a display element for four colors, or a display element for one color. The pixel unit 302 consisting of 45 pixels corresponds to one pixel of a typical display. The 45 RGB light beams emitted from the 45 RGB display elements 302' in the pixel unit 302 are refracted by the lens element 303 in the direction of each optical path, for example, in one of 45 display directions from 0 to 44 shown in Figure 13 (hereinafter, these directions are referred to as "display directions 501").

[0119] Then, for each of the different viewpoint positions 2A of each user 21 to 24, the image generation device 10 inputs the data of the display image it generates corresponding to those viewpoint positions 2A1 to 2A4 to the RGB display elements 32' with pixel numbers corresponding to directions 502A, 502B, 502C, and 502D from the position of the pixel unit 302 in the reference coordinate space (Figure 4) toward the viewpoint positions 2A1 to 2A4 of each user 21 to 24. Hereinafter, these directions will be referred to as user viewpoint position directions 502A, 502B, 502C, and 502D. Alternatively, these directions will be collectively referred to as user viewpoint position directions 502.

[0120] Here, the processing unit 14 of the image generation device 10 may generate a right-eye display image and a left-eye display image for each viewpoint position 2A1 to 2A4 of each user 21 to 24 detected by the measuring device 20, such that when a user 2 views a three-dimensional virtual object in virtual space through the transparent display device 30 from their right eye and left eye, respectively, at a certain distance to the left and right of that viewpoint position, the image generation device 10 may generate a right-eye display image and a left-eye display image so that the image appears geometrically precise and natural, creating the illusion for each user 2 that the virtual object exists in real space. When generating the right-eye display image and the left-eye display image to be displayed on the screen of the transparent display device 30, the image generation device 10 may perform projection transformations, perspective projection transformations, or similar calculation processes to project the virtual object, represented by three-dimensional CG data, onto the screen of the transparent display device 30, i.e., a two-dimensional surface, based on the viewpoint positions 2A1 to 2A4 of each user 21 to 24.

[0121] In this case, for each viewpoint position 2A1 to 2A4 of each user 21 to 24, the processing unit 14 displays the data of the right-eye display image and the left-eye display image generated in accordance with each viewpoint position on the RGB display elements 302' with pixel numbers corresponding to the right-eye display direction and the RGB display elements 302' with pixel numbers corresponding to the left-eye display direction, respectively, located on both sides of each user viewpoint position direction 502A, 502B, 502C, 502D corresponding to each viewpoint position.

[0122] As a result of the above control operations, for each user 21-24 in Figure 13, the display state of the virtual object is controlled for each user 2, as shown by the line of sight direction 500 connecting each right eye and each left eye to each lens element 303, and as explained by the displays D1, D2, D3, etc. in Figure 3. This image can be simultaneously displayed to multiple users 21-24 at various positions on the transparent display device 30. In this case, for each user 21-24, images with binocular parallax are incident on their respective right and left eyes, allowing them to experience realistic stereoscopic vision without glasses.

[0123] In this embodiment, the measuring device 20 detects the viewpoint position 2A of each user 2, and the displayed image of the virtual object Vobj is subjected to projection transformation, perspective projection transformation, or similar calculation processing according to the viewpoint position 2A of each user 2.

[0124] <Details of display processing> A detailed example of the display processing in the third embodiment will be explained using Figure 10. The following explanation of this processing will follow the explanatory diagram in Figure 14 described above. The processing unit 14 first sets the left-eye display image data generated in accordance with the left-eye display direction of the first viewpoint position 2A (for example, 2A1 in Figure 14) in each pixel number corresponding to each display direction 501 in Figure 14, for example from the 0th to the 44th display directions 501 described in Figure 14, between the 0th display direction 501 and the user viewpoint position direction 502 (for example, 502A in Figure 14) that passes through the first viewpoint position 2A (for example, 2A1 in Figure 14) among the one or more viewpoint positions 2A calculated in step S102 in Figure 6 (step S901). This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13) that are determined according to the user viewpoint position direction 502 from the pixel unit 302 toward the first viewpoint position 2A.

[0125] Next, the processing unit 14 increments the variable value indicating the viewpoint position 2A, which is stored in the memory unit 12 of the image generation device 10, for example in RAM or registers (not shown), by 1 (step S905).

[0126] Next, the processing unit 14 determines whether the variable value indicating the viewpoint position 2A has exceeded a predetermined value indicating the last user viewpoint position (step S906). The value indicating the last viewpoint position 2A can be predetermined according to the number of users 2 using System 1 and stored in the RAM or registers mentioned above. In step S906, the processing unit 14 determines whether the variable value indicating the viewpoint position 2A has exceeded the predetermined value stored above, thereby determining whether the variable value indicating the viewpoint position 2A has exceeded the value indicating the last user viewpoint position.

[0127] If the processing has not yet reached the final user viewpoint position and the determination in step S906 is NO, the processing unit 14 transfers control to the processing in step S902 and executes the processing from steps S902 to S904.

[0128] First, the processing unit 14 calculates the midpoint direction 503 (for example, 503AB in Figure 14) from the pixel unit 302 toward the midpoint between the previous viewpoint position 2A (for example, 2A1 in Figure 14) and the current viewpoint position 2A (for example, 2A2 in Figure 14) (step S902).

[0129] Next, the processing unit 14 sets the right-eye display image data generated in accordance with the right-eye display direction at the previous user viewpoint position 2A (for example, 2A1 in Figure 14) in the RGB display elements 302' corresponding to each pixel number, for example, numbers 4 to 10 in Figure 14, between the user viewpoint position direction 502 (for example, 502A in Figure 14) passing through the previous user viewpoint position 2A (for example, 2A1 in Figure 14) and the midpoint direction 503 (for example, 503AB in Figure 14) calculated this time (step S903). This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13) constituting the transparent display device 30, between the user viewpoint position direction 502 relative to the pixel unit 302 and the midpoint direction 503 determined for the pixel unit 302 in step S902.

[0130] Next, the processing unit 14 sets the left-eye display image data generated in accordance with the display direction of the left eye at the current user viewpoint position 2A (for example, 2A2 in Figure 14) to the RGB display elements 302' corresponding to each display direction 501, for example, numbers 11 to 16 in Figure 14, between the midpoint direction 503 (for example, 503AB in Figure 14) calculated in step S902 and the user viewpoint position direction 502 (for example, 502B in Figure 14) that passes through the current user viewpoint position 2A (for example, 2A2 in Figure 14) (step S904). This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13) constituting the transparent display device 30, between the midpoint direction 503 determined for the pixel unit 302' in step S902 and the user viewpoint position direction 502 relative to the pixel unit 302'.

[0131] Subsequently, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1, as described above (step S905), and determines whether that variable value exceeds the predetermined value indicating the last user viewpoint position 2A (step S906).

[0132] If the final viewpoint position 2A has not yet been processed and the determination in step S906 is NO, the processing unit 14 will execute the processes from steps S902 to S904 described above again.

[0133] In step S902, as described above, for example in Figure 14, the midpoint direction 503BC, which is directed towards the midpoint between the previous viewpoint position 2A2 and the current user viewpoint position 2A3, is calculated from the pixel unit 302.

[0134] Next, in step S903, similar to the one described above, the right-eye display image data generated to correspond to the right-eye display direction at the previous viewpoint position 2A2 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 from 17 to 21 in Figure 14, for example, between the user viewpoint position direction 502B passing through the previous viewpoint position 2A2 and the midpoint direction 503BC calculated this time. This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13) constituting the transparent display device 30, between the user viewpoint position direction 502 relative to the pixel unit 302 and the midpoint direction 503 determined for the pixel unit 302 in step S902.

[0135] Next, in step S904, similar to the one described above, for example in Figure 14, the left-eye display image data generated to correspond to the left-eye display direction at the current viewpoint position 2A3 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 from 22 to 26 in Figure 14, between the midpoint direction 503BC calculated in step S902 and the user viewpoint position direction 502C passing through the current viewpoint position 2A3. This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13) constituting the transparent display device 30, between the midpoint direction 503 determined for the pixel unit 302' in step S902 and the user viewpoint position direction 502 relative to the pixel unit 302'.

[0136] Subsequently, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1, as described above (step S905), and determines whether that variable value exceeds the predetermined value indicating the last user viewpoint position 2A (step S906).

[0137] If the final viewpoint position 2A has not yet been processed and the determination in step S906 is NO, the processing unit 14 will execute the processes from steps S902 to S904 described above again.

[0138] In step S902, as described above, for example in Figure 14, the midpoint direction 503CD, which is directed towards the midpoint between the previous viewpoint position 2A3 and the current viewpoint position 2A4, is calculated from the pixel unit 302.

[0139] Next, in step S903, similar to the one described above, the right-eye display image data generated to correspond to the right-eye display direction of the previous viewpoint position 2A3 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501 from 27 to 30 in Figure 14, for example, between the user viewpoint position direction 502C passing through the previous user viewpoint position 2A3 and the midpoint direction 503CD calculated this time. This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13) constituting the transparent display device 30, between the user viewpoint position direction 502 relative to the pixel unit 302 and the midpoint direction 503 determined for the pixel unit 302 in step S902.

[0140] Next, in step S904, similar to the one described above, for example in Figure 14, the left-eye display image data generated to correspond to the display direction of the left eye at the current viewpoint position 2A4 is set in the RGB display elements 302' of each pixel number corresponding to each display direction 501, for example, from 31 to 34 in Figure 14, between the midpoint direction 503CD calculated in step S902 and the user viewpoint position direction 502D passing through the current user viewpoint position 2A4. This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13) constituting the transparent display device 30, between the midpoint direction 503 determined for the pixel unit 302' in step S902 and the user viewpoint position direction 502 for the pixel unit 302'.

[0141] Subsequently, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1, as described above (step S905), and determines whether that variable value has exceeded the predetermined value indicating the last viewpoint position 2A (step S906).

[0142] After processing has been completed up to the last user viewpoint position 2A, if the determination in step S906 is YES, the processing unit 14 executes the processing in the final step S907. In step S907, the processing unit 14 sets the right-eye display image data, which was generated to correspond to the right-eye display direction at the last user viewpoint position 2A (for example, 2A4 in Figure 14), to the RGB display elements of each pixel number corresponding to each display direction 501 from, for example, 35 to 44 in Figure 14, between the user viewpoint position direction 502 (for example, 502D in Figure 14) passing through the current viewpoint position 204 (for example, 2A4 in Figure 14) and the 44th display direction 501. This setting process is performed on the RGB display elements 302' of each pixel unit 302 (see Figure 13), which are determined according to the user viewpoint position direction 502 from the pixel unit 302 toward the last viewpoint position 2A.

[0143] Subsequently, the processing unit 14 terminates the display process of step S104 in Figure 6, as illustrated in the flowchart of Figure 10.

[0144] In the image display process illustrated in the flowchart of Figure 10, in Figure 14, the right-eye or left-eye display image data, generated to correspond to the display direction of the right or left eye of the adjacent viewpoint position 2A, is copied and set to each display direction 501 in the section delimited by the user viewpoint position direction 502 of the two viewpoint positions 2A and the midpoint direction 503 passing through the midpoint between those viewpoint positions 2A. Alternatively, by changing the algorithm for mapping display pixels to each display direction 501 between the viewpoint position 2A of user 2 that is close to the transparent display device 30 and the viewpoint position 2A of user 2 that is far from the transparent display device 30, the sensitivity of the display pixels to changes can be made uniform according to the viewpoint position A of user 2. Or, instead of simply copying, display pixels calculated by interpolation processing according to each display direction 501 between two adjacent display images may be set. Furthermore, the specific viewpoint position 2A acquired by the measuring device 20 described above and used for processing is not particularly limited, but for example, the midpoint between each user 2's eyes, the center point of the head, or a position a predetermined distance inside the head from the center of both eyes can be used as the viewpoint position 2A. Another example is that the positions of each user 2's left and right eyes may be acquired separately and used for processing. Based on the separately acquired left and right eye position information, the displayed image of the virtual object Vobj may be subjected to projection transformation, perspective projection transformation, or similar calculation processing according to the left and right eye positions of each user 2.

[0145] Thus, according to this embodiment, for multiple users 2 with naked eyes who do not wear the 3D glasses device, for each of the viewpoint positions 2A1 to 2A4 of each user 21 to 24, when the user's right eye and left eye, each at a certain distance to the left and right of that viewpoint position, view a three-dimensional, stereoscopic virtual object in the virtual space via the transparent display device 30, the display image of the virtual object can be superimposed onto the real space visible through the transparent display device 30 as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes, so that it appears to each user 2 as if the virtual object exists in the real space, with geometric rigor and plausibility. In addition, according to this embodiment, different display images of the virtual object corresponding to each user 2's viewpoint position 2A can be simultaneously presented to each user 2.

[0146] <Calibration process> Since the direction in which light emitted from a pixel is refracted by the lens element 303 differs depending on the pixel number of the pixel unit 302, it is required that the pixels and lens elements be in a desired positional relationship with high precision in the transmissive display device 30. However, it can be difficult to manufacture the transmissive display device 30 so that the pixels and lens elements are in a desired positional relationship with high precision. For example, when manufacturing the transmissive display device 30 by bonding a transparent sheet on which multiple cylindrical lens elements 303 are formed to a transparent substrate 301 on which a pixel row 300 is arranged, it can be difficult to accurately position the lens elements 303 relative to the pixels due to expansion and contraction of the transparent sheet and errors in the position and angle of the transparent sheet relative to the transparent substrate 301. Also, for example, when bonding multiple cylindrical lens elements 303 individually to the transparent substrate 301 on which the pixel row 300 is arranged, it can be difficult to accurately position and bond the lens elements 303 relative to the pixels. Furthermore, for example, when printing lens elements 303 onto a transparent substrate 301 on which pixel rows 300 are arranged, in a process separate from the formation of the pixel rows 300, it can be difficult to accurately position and adhere the lens elements 303 relative to the pixels.

[0147] Therefore, errors may occur in the transmissive display device 30 of this embodiment during manufacturing, etc. In particular, errors may occur in the position relative to the pixels 302' when the lens element 303 is provided on the transparent substrate 301. The transmissive display device 30 is configured to display an appropriate image toward the user's viewpoint by accurately refracting the light emitted from each pixel 302' in a predetermined direction using the lens element 303. Therefore, if there is an error in the relative position between the lens element 303 and the pixels 302', the image will deteriorate. The calibration process is a process to improve image deterioration caused by individual differences in the transmissive display device 30 due to manufacturing, etc. It is preferable to perform this process at least once for each individual transmissive display device 30.

[0148] Figure 15 is a flowchart of the calibration process. Calibration can be performed manually by an operator or automated. In the case of automation, for example, the calibration can be performed so that the image captured by the camera is the desired image. The basic process flow is the same for both manual and automated calibration. The basic process flow of calibration will be explained below with reference to Figure 15.

[0149] With the adjuster looking at the screen of the transparent display device 30, or with a calibration imaging device (camera) installed at that position, the processing unit 14 acquires information about the adjuster's viewpoint or the position of the imaging device (step S1001). Next, the processing unit 14 displays a specific image in a predetermined width range to the left and right, centered on the direction of the viewpoint. The specific image is not particularly limited. For example, vertical stripes, diagonal stripes, grid patterns, etc., can be used as the specific image. Subsequently, the processing unit 14 provides the adjuster with a parameter adjustment GUI that allows adjustment of which range of pixels 302' to use in the pixel unit 302, and prompts the adjuster to adjust the parameters so that the specific image is clearly visible from the adjuster's viewpoint or in the captured image from the imaging device, or automatically adjusts the parameters through calculation processing based on the specific image. Then, while moving the adjuster's viewpoint or the imaging device (step S1005), the processing unit 14 repeats the processes of steps S1001-S1003 until a predetermined termination condition is met (step S1004).

[0150] In this case, for example, the number of adjustments may be predetermined so that the viewpoint position or the position of the imaging device is evenly set to the left and right of the screen, and the adjustment may be repeated until that number of adjustments is reached to complete the calibration. Also, the process in step S1003 may be performed while viewing with both eyes, or each eye separately. The same may be done using an imaging device. Furthermore, the parameters adjusted in step S1003 are not particularly limited. For example, a combination of parameters may be used, such as a parameter to shift the pixels 302' used in the pixel unit 302 to the left and right across the entire screen, a parameter to adjust the angle at which the pixels 302' used in the pixel unit 302 are tilted vertically across the screen, and a parameter to adjust the degree to which the pixels 302' used in the pixel unit 302 are gradually shifted horizontally across the screen. These parameters may be adjusted for the entire screen, or they may be adjusted for each pixel unit 302 corresponding to each individual pixel.

[0151] The shape and arrangement of the lens element 303 of the transmissive display device 30 in this embodiment described above can be modified in various ways.

[0152] (Variation 1) Figure 16 is a schematic diagram illustrating the configuration of Modification 1 of the display device in the system of the third embodiment. Figure 16 shows a front view of the screen of the display device. This transmissive display device 30 is transparent or semi-transparent, as in the third embodiment, and is a display capable of simultaneously displaying different images in multiple display directions.

[0153] As shown in Figure 16, in Modification 1, similar to the third embodiment, a plurality of pixel rows 300 are arranged on the transparent substrate 301, having a longitudinal direction in the horizontal direction of the screen and spaced apart from each other at predetermined intervals vertically, and a plurality of lens elements 303 are arranged vertically across the plurality of pixel rows 300. However, in Modification 1, unlike the third embodiment, the plurality of lens elements 303 having a longitudinal direction in the vertical direction are arranged adjacent to each other without spacing in the horizontal direction. Therefore, the user will see the real space behind the transmissive display device 30 through the lens elements 303 and the transparent substrate 301 in areas where there are no pixel rows 300 but there are lens elements 303.

[0154] (Modification 2) Figure 17 is a schematic diagram illustrating the configuration of a modified example 2 of the display device in the system of the third embodiment. Figure 17 shows a front view of the screen of the display device. This transmissive display device 30, like that of the third embodiment, is transparent or semi-transparent and is a display capable of simultaneously displaying different images in multiple display directions.

[0155] As shown in Figure 17, in Modification 2, similar to the third embodiment, a plurality of pixel rows 300 are arranged on the transparent substrate 301, with their longitudinal direction in the horizontal direction of the screen and spaced apart from each other at predetermined intervals vertically, and a plurality of lens elements 303 are arranged to cross the pixel rows 300 vertically. However, in Modification 2, unlike the third embodiment, the plurality of lens elements 303 are arranged in a matrix with spacing in both the horizontal and vertical directions, so that each lens element crosses one pixel row 300 vertically. Therefore, the user has a larger area where neither pixel rows 300 nor lens elements 303 are arranged compared to the third embodiment.

[0156] (Variation 3) Figure 18 is a schematic diagram illustrating the configuration of a modified example 3 of the display device in the system of the third embodiment. Figure 18 shows a front view of the screen of the display device. This transmissive display device 30, like that of the third embodiment, is transparent or semi-transparent and is a display capable of simultaneously displaying different images in multiple display directions.

[0157] As shown in Figure 18, in Modification 3, similar to the third embodiment, a plurality of pixel rows 300 are arranged on the transparent substrate 301, having a longitudinal direction in the horizontal direction of the screen and spaced apart from each other at predetermined vertical intervals, and a plurality of lens elements 303 are arranged to cross the pixel rows 300 in the vertical direction. However, in Modification 3, unlike the third embodiment, the plurality of lens elements 303 are spaced apart vertically so as to cross one pixel row 300 in the vertical direction, but are arranged adjacent to each other without spacing in the horizontal direction. In areas where neither pixel rows 300 nor lens elements 303 are arranged, the real space behind the transmissive display device 30 can be seen through the transparent substrate 301. Also, in areas where pixel rows 300 are not arranged but lens elements 303 are arranged, the real space behind the transmissive display device 30 can be seen to some extent through the lens elements 303 and the transparent substrate 301.

[0158] (Modification 4) Figure 19 is a diagram illustrating the configuration of a modified example 4 of the display device in the system of the third embodiment. Figure 19 shows a front view of the screen of the display device. This transmissive display device 30, like that of the third embodiment, is transparent or semi-transparent and is a display capable of simultaneously displaying different images in multiple display directions.

[0159] As shown in Figure 19, in Modification 4, the transparent substrate 301 has a plurality of pixel rows 300 arranged in a matrix with a longitudinal direction in the horizontal direction of the screen and spaced apart from each other at predetermined intervals in the vertical and horizontal directions, and a plurality of lens elements 303 that cross vertically the plurality of pixel rows 300 that are arranged vertically within the matrix. The pixel row 300 in this Modification is a row of pixels 302' with a number of pixels greater than the number of pixels in the pixel unit 302 (45 in this embodiment). This makes it possible to adjust which part of the pixel row 300 becomes the pixel unit 302 by calibration. Also, in Modification 4, unlike the third embodiment, the plurality of pixel rows 300 are arranged in a matrix with spacing in both the horizontal and vertical directions, so there is a wider area where pixel rows 300 are not arranged than in the third embodiment.

[0160] One example of applying the transparent display device 30 in this embodiment is to mount the transparent display device 30 on the glass portion of an automobile or the like, and present a three-dimensional display image of a virtual object to each of multiple users 2 seated in the front and rear seats of the vehicle. With the transparent display device 30 of this embodiment, it is possible to individually display a display image of a virtual object to multiple users 2 who have different distances and viewing directions from the transparent display device 30. Therefore, it is possible to superimpose a virtual object onto the real space visible around the transparent display device 30, which is the glass of a moving automobile, according to the view from each user 2's viewpoint.

[0161] In this embodiment, the lens element 303 may be a lenticular lens made of a cylindrical lens, or any three-dimensional lens shape such as a hemispherical semi-ellipsoid or toroidal shape that controls the direction of light. Alternatively, instead of using a lens, a material such as a film or plate that generates a parallax barrier may be used.

[0162] In this embodiment, the pixels 302' of the pixel array 302 are, for example, organic EL elements, but other elements may be used. For example, pixels 302' may be self-emissive elements such as mini-LEDs or micro-LEDs, or elements that control light transmission using a liquid crystal mechanism. Furthermore, pixels 302' may be elements or light-emitting materials that reflect externally projected light to allow the user to perceive color. In this embodiment, RGB elements are used as examples for pixels 302', but single-color pixels such as green or red may also be used.

[0163] [Fourth Embodiment] In the second embodiment described above, an image display system was illustrated that shows a parallax-laden three-dimensional image of a virtual object to multiple users using a transparent display. In the transparent display in the second embodiment, a predetermined number of pixels arranged horizontally on a transparent substrate were grouped together as a pixel unit, multiple pixel units were arranged in a matrix, and a lens element of a cylindrical lens with the same width as the pixel unit was provided on the pixel unit. In the fourth embodiment, an image display system using a transparent display with a different configuration from the second embodiment is illustrated. However, since the configuration and operation of the image display system in the fourth embodiment are generally the same as those of the second embodiment, the following description will focus on the configuration and operation that differ from the second embodiment. <Overview of display configuration and manufacturing method> Figure 20 is a schematic diagram illustrating the configuration of the display device in the system of the fourth embodiment. Figure 20 shows a front view of the screen of the display device.

[0164] The transparent display device 30 of this embodiment, like that of the second embodiment, is transparent or semi-transparent and is a display capable of simultaneously displaying different images in multiple display directions.

[0165] As shown in Figure 20, the transparent substrate 301 has multiple pixel groups 304 arranged on it, each consisting of multiple pixels arranged in an 8x4 matrix, with, for example, 8 columns horizontally and 4 columns vertically at regular intervals. In addition, multiple lens elements 303 are arranged across the substrate at a fixed angle θ with respect to the vertical direction. As shown in Figure 20 as an example, each pixel group 304 is arranged at equal intervals with a gap e in the horizontal direction and a gap f in the vertical direction. However, pixel groups 304 arranged in the same vertical column are positioned further slid horizontally by a distance d. By appropriately setting the gaps e and f, the distance d, and the angle θ, multiple pixel groups 304 are arranged spaced apart from each other on a straight line inclined by an angle θ from the vertical, and the lens element 303 is positioned to extend along that straight line with a longitudinal direction, and the lateral relative positions of the pixel groups 304 with respect to the lens element 303 are the same for all pixel groups 304. Within each pixel group 304, each pixel is densely arranged, while relatively large gaps e and f exist between adjacent pixel groups 304. Here, the region 305 including these gaps and one pixel group 304 is defined as a CLED (Combined LED). In the fourth embodiment, the CLED 305 is treated as a broad unit pixel, and the transmissive display device 30 is constructed by arranging multiple CLEDs 305 in a matrix in the vertical direction and multiple CLEDs 305 in the horizontal direction.

[0166] The transparent substrate 301 is, for example, a transparent flat plate, and may be a transparent resin plate including an acrylic plate, or it may be glass. Any material is acceptable as long as it has a certain degree of transparency. It may also have a shape that is basically flat but has a certain degree of curvature, like the transparent glass used in the windshield of an automobile. For convenience, in the following explanation, such shapes will be included and described as "flat," but in the example using a curved plate, the same explanation will hold if the "flat" part is replaced with "tangent plane."

[0167] Each pixel constituting the pixel group 304 is, for example, an RGB miniLED element with a cubic shape of several tens to several hundreds of micrometers in length, width, and height, but is not limited to this. It may also be an organic EL element or a liquid crystal element. The size is also not limited to the example size and may be larger or smaller. The shape is not limited to a cube. The color may also be monochromatic. The lens element 303 is, for example, a semicircular cylindrical lens that has a width corresponding to a predetermined number of pixels in the pixel group 304 in the short-side direction (horizontal direction) and acts as a lens in the short-side direction, but is not limited to this. As the lens element 303, multiple lens elements that act as cylindrical lenses for the pixel group 304 as in Figure 20 may be arranged for each pixel group 304, spaced apart in the vertical direction as shown in Figure 21. In that case, each of the lens elements 303 may be a cylindrical lens, a hemisphere, or an arbitrary dome-shaped lens. Alternatively, instead of the lens element 303, a striped slit 307 having transparent and opaque portions at regular intervals, which acts as a parallax barrier, may be arranged as shown in Figure 22. Here, as an example, the width of the lens element 303 in the shorter direction is set to cover a portion of the pixel group 304 as shown in Figure 20, but it may also be set to cover the entire group. The lens element 303, as an example, has its longitudinal direction in a direction tilted at an angle θ with respect to the vertical direction of the screen, and is arranged at predetermined intervals in the horizontal direction.

[0168] The miniLED material for the pixel group 304 is attached to the surface of the transparent substrate 301. The miniLED material is attached to the transparent substrate 301 by methods such as precise pick-and-place using a robot or by firing it with a laser. If smaller microLED material is used as the miniLED that makes up the pixel group 304, it may be difficult for the robot to grasp the mini / microLED in the first place using pick-and-place, or it may be difficult to attach it to the surface of the transparent substrate 301 with precision. Also, when mass-producing, it would take a considerable amount of time to place each mini / microLED for all the pixels. In the laser firing method, the LED material is placed parallel to the transparent substrate 301 with a specific gap between them, and then a laser is shone on it to fire and attach it to the transparent substrate, which may overcome the shortcomings of the robot method. Another methodology is to use mini / microLED clusters, which are made by integrating mini / microLEDs in a vertical and horizontal array from the beginning. As in this embodiment, if a pixel group 304 composed of multiple pixels arranged in an 8x4 matrix is ​​packaged as a mini / microLED cluster and each package is attached to a transparent substrate 301, the attachment time can be simply reduced to 1 / 32. In this case, the mini / microLED cluster may simply be an assembly of LEDs arranged in a matrix, but in terms of design, it is also possible to reduce the number of wires embedded on the transparent substrate 301 by constructing it on the silicon wafer together with the LEDs, for example, by embedding a separate circuit, such as a TFT backplane circuit, beneath the LED layer. Designing the assembly on the silicon wafer also contributes to higher density, and as a result contributes to higher resolution and a higher parallax number (45 parallax in Figure 14, but this number can be increased) as basic specifications for display devices. By using a TFT backplane circuit, the LED driving method can also be an active matrix method. Since the area where the miniLEDs are attached will be convex when attached to the transparent substrate, a transparent resin may be poured in and solidified to make it flat.Alternatively, after pouring in the transparent resin, a thin transparent film can be placed over it.

[0169] For example, the lens elements 303 can be individually manufactured and placed at predetermined positions on the transparent substrate 301 on which the pixel groups 304 are arranged, so that they overlap each pixel group 304. Alternatively, the lens elements 303 can be provided overlapping each pixel group 304 by using 3D printing technology to extrude transparent material at predetermined positions on the transparent substrate 301 on which the pixel groups 304 are arranged, thereby forming cylindrical lenses, dome-shaped lenses, etc. Furthermore, the lens elements 303 can also be provided at positions overlapping each pixel group 304 by attaching a transparent film member or transparent flat plate member (including curved shapes) on which multiple lens elements 303 are positioned and arranged to the transparent substrate 301 on which the pixel rows 300 are arranged, at predetermined positions. A transparent film member on which multiple lens elements 303 are positioned and arranged together can be formed using methods such as hot press molding using a mold. A transparent flat plate member (including a curved shape) in which multiple lens elements 303 are positioned and arranged to form a single unit can be manufactured by pouring UV-curable resin onto a glass substrate or the like, pressing a mold onto it, and curing it with UV light to form the lens elements 303 on the glass substrate.

[0170] Alternatively, the transparent substrate 301 may be manufactured by inserting an array of pixel units and electrode wires that supply power and signals to the pixel units into the interlayer portion of a plate with a structure in which multiple transparent substrates are bonded together with an interlayer in between, and also by inserting a film or plate containing a lens element and a film or plate that creates a parallax barrier.

[0171] Alternatively, after forming a transparent substrate 301 by inserting only the pixel group 304 into the interlayer, a transparent film member or a transparent flat plate member (including a curved shape) on which the lens element 303 is arranged may be attached to the surface of the transparent substrate 301.

[0172] As described above, each pixel group 304 is arranged with predetermined intervals e and f between them, and offset by a distance d. Similarly, each lens element 303 is also arranged with predetermined intervals between them. As a result, in areas where neither pixel groups 304 nor lens elements 303 are present, the real space behind the transmissive display device 30 can be seen through the transparent substrate 301. Furthermore, in areas where no pixel groups 304 are present but lens elements 303 are present, the real space behind the transmissive display device 30 can be seen to some extent through the lens elements 303 and the transparent substrate 301.

[0173] In the transmissive display device 30 of this embodiment, the pixel groups 304 and lens elements 303 are arranged relatively "sparsely". For example, the area occupied by the pixel groups 304 and / or lens elements 303 within the entire screen of the transmissive display device 30 is several tens of percent to several percent, and can be reduced to several percent or less by further reducing the size of the pixel units. By reducing the size of each individual pixel unit or increasing the light intensity of each individual pixel unit, it is possible to display an image that is visible to the user without being overwhelmed by ambient light, while ensuring sufficient transparency for viewing real space. This makes it possible to provide the user viewing the virtual object with sufficient resolution and clarity in the image when displaying virtual objects superimposed on real space visible through the transmissive display device 30, as in this embodiment. Alternatively, in the transmissive display device 30 of this embodiment, the pixel groups 304 and lens elements 303 may not be arranged "sparsely" but densely throughout the entire display device. In this case, transparency can be ensured within each pixel group by ensuring that the gaps between non-transparent materials such as light-emitting parts like mini-LEDs and wiring are transparent. Alternatively, transparency can be ensured by using liquid crystals for the pixels themselves, allowing areas not used as pixels to be transparent due to the properties of liquid crystals. Therefore, making the pixel group 304 and lens element 303 "sparse" is just one example, and various methods can be considered to ensure the transparency of the display, and the type of display is not limited here.

[0174] As shown in Figure 23, the transparent substrate 301 is provided with electrode wires 306 made of a highly transparent conductive material, or very fine metal electrode wires 306, which are electrically connected to each pixel of each pixel group 304, for example. Each pixel in each pixel group 304 is individually driven to emit light or controlled to transmit light by power and drive signals supplied via these electrodes. Both passive matrix and active matrix control methods are possible, but are not limited to these.

[0175] Next, we will describe the more detailed configuration of the transparent display device 30 of this embodiment, the display operation control by the transparent display device 30, and the calibration of the transparent display device 30.

[0176] <Details of display configuration and overview of display processing> Figures 24, 25, and 26 illustrate the more detailed configuration of the display device of the fourth embodiment and the control of the display operation by the display device. The left side of Figure 24 shows the pixel group 304, pixels 304', and lens element 303 when the transmissive display device 30 is at the same angle as in Figure 23. The center side of Figure 24 shows the pixel group 304, pixels 304', and lens element 303 when the transmissive display device 30 is tilted by an angle θ such that the lens element 303 is vertical. The right side of Figure 24 shows an example of the optical path through which light from each pixel of the pixel group 304 passes through the lens element 303.

[0177] In the transmissive display device 30 of this embodiment, as an example, 32 pixels (RGB display elements) 304' located in the portion of the image group 304 that overlaps with the lens element 303 in a direction perpendicular to the screen are treated as a set of pixels called a pixel group 304. However, the number of pixels in the pixel group 304 being 32 is just an example, and the number of pixels in the pixel group 304 is not limited to this.

[0178] In the fourth embodiment, multiple pixel groups 304 are arranged in a matrix. That is, one pixel group 304 consists of 32 pixels, and each pixel is equipped with an RGB display element 304'. The RGB display element 304' is composed of three subpixel display elements that display three colors: R (red), G (green), and B (blue). Alternatively, the RGB display element 304' may be a display element for four colors, or a display element for one color. The pixel group 304 consisting of 32 pixels corresponds to one pixel in a typical display.

[0179] Unlike the pixel unit 302 of the fourth embodiment, the RGB display elements 304' within the pixel group 304 are arranged in a matrix in the vertical and horizontal directions, rather than in a single row. The lens element 303 is not perpendicular to the pixel group 304, but is tilted at an angle θ as shown in Figures 20, 23, and 24, and is positioned to overlap with it. When the pixel group 304 is observed from a sufficient distance relative to its size, the differences in the positions of each individual RGB display element 304' constituting the pixel group 304 are not perceived, and the pixel group 304 is perceived by the observer as approximately a single point. Light from each of the 32 RGB display elements 304' within the pixel group 304 is emitted in the Z-axis direction in Figure 24. However, the light from each RGB display element 304' enters the lens element 303 from different positions at equal intervals with respect to the width direction of the lens element 303 (X-axis direction in Figure 24) and follows the optical path within the lens, so each undergoes refraction at a different angle.

[0180] Of the 32 RGB light beams emitted from the 32 RGB display elements 304' for each pixel in the pixel group 304, the light beams from RGB display elements 04 to 29 in Figure 24 that enter the lens element 303 are refracted by the lens element 303 in the direction of each optical path, for example, in the 26 display directions 04 to 29 shown in Figure 26 (hereinafter referred to as "display direction 501").

[0181] Then, for each of the different viewpoint positions 2A of each user 21 to 24 shown in Figures 25 and 26, the image generation device 10 inputs the data of the display image it generates corresponding to those viewpoint positions 2A1 to 2A4 to the RGB display elements 304' with pixel numbers corresponding to directions 502A, 502B, 502C, and 502D from the position of the pixel group 304 in the reference coordinate space (Figure 4) toward the viewpoint positions 2A1 to 2A4 of each user 21 to 24.Hereafter, these directions will be referred to as user viewpoint position directions 502A, 502B, 502C, and 502D.Alternatively, these directions will be collectively referred to as user viewpoint position directions 502.

[0182] Here, the processing unit 14 of the image generation device 10 may generate a right-eye display image and a left-eye display image for each viewpoint position 2A1 to 2A4 of each user 21 to 24 detected by the measuring device 20, such that when a user 2 views a three-dimensional virtual object in virtual space through the transparent display device 30 from their right eye and left eye, respectively, at a certain distance to the left and right of that viewpoint position, the image generation device 10 may generate a right-eye display image and a left-eye display image so that the image appears geometrically precise and natural, creating the illusion for each user 2 that the virtual object exists in real space. When generating the right-eye display image and the left-eye display image to be displayed on the screen of the transparent display device 30, the image generation device 10 may perform projection transformations, perspective projection transformations, or similar calculation processes to project the virtual object, represented by three-dimensional CG data, onto the screen of the transparent display device 30, i.e., a two-dimensional surface, based on the viewpoint positions 2A1 to 2A4 of each user 21 to 24.

[0183] In this case, for each viewpoint position 2A1 to 2A4 of each user 21 to 24, the processing unit 14 displays the data of the right-eye display image and the left-eye display image generated in accordance with each viewpoint position on the RGB display elements 304' with pixel numbers corresponding to the right-eye display direction and the RGB display elements 304' with pixel numbers corresponding to the left-eye display direction, respectively, located on both sides of each user viewpoint position direction 502A, 502B, 502C, 502D corresponding to each viewpoint position.

[0184] As a result of the above control operations, for each user 21-24 in Figure 25, the display state of the virtual object is controlled for each user 2, as shown by the line of sight direction 500 connecting each right eye and each left eye to each lens element 303, and as explained by the displays D1, D2, D3, etc. in Figure 3. This image can be simultaneously displayed to multiple users 21-24 at various positions on the transparent display device 30. In this case, for each user 21-24, images with binocular parallax are incident on their respective right and left eyes, allowing them to experience realistic stereoscopic vision without glasses.

[0185] In this embodiment, the measuring device 20 detects the viewpoint position 2A of each user 2, and the displayed image of the virtual object Vobj is subjected to projection transformation, perspective projection transformation, or similar calculation processing according to the viewpoint position 2A of each user 2.

[0186] <Details of display processing> A detailed example of the display processing in the fourth embodiment will be explained using Figure 27. The following explanation of this process will follow the explanatory diagram in Figure 26 described above. The processing unit 14 first sets the display image data for the left eye, which is generated in accordance with the display direction of the left eye at the first viewpoint position 2A (for example, 2A1 in Figure 26), in the RGB display elements 304' corresponding to each pixel number of each display direction 501, for example, from the 4th to the 29th display directions 501 described in Figure 26, between the 4th display direction 501 and the user viewpoint position direction 502 (for example, 502A in Figure 26) that passes through the first viewpoint position 2A (for example, 2A1 in Figure 26) among the one or more viewpoint positions 2A calculated in step S102 of Figure 6 (step S1101). This set processing is performed on each pixel group 304 (see Figure 25) and the RGB display elements 304' determined according to the user viewpoint position direction 502 from the pixel group 304 toward the initial viewpoint position 2A.

[0187] Next, the processing unit 14 increments the variable value indicating the viewpoint position 2A, which is stored in the memory unit 12 of the image generation device 10, for example in RAM or a register (not shown), by 1 (step S1105).

[0188] Next, the processing unit 14 determines whether the variable value indicating the viewpoint position 2A has exceeded a predetermined value indicating the last user viewpoint position (step S1106). The value indicating the last viewpoint position 2A can be predetermined according to the number of users 2 using System 1 and stored in the RAM or registers mentioned above. In step S1106, the processing unit 14 determines whether the variable value indicating the viewpoint position 2A has exceeded the predetermined value stored above, thereby determining whether the variable value indicating the viewpoint position 2A has exceeded the value indicating the last user viewpoint position.

[0189] If the processing has not yet reached the final user viewpoint position and the determination in step S1106 is NO, the processing unit 14 transfers control to the processing in step S1102 and executes the processing from steps S1102 to S1104.

[0190] First, the processing unit 14 calculates the midpoint direction 503 (for example, 503AB in Figure 26) from the pixel group 304 toward the midpoint between the previous viewpoint position 2A (for example, 2A1 in Figure 26) and the current viewpoint position 2A (for example, 2A2 in Figure 26) (step S1102). Next, the processing unit 14 sets the right-eye display image data generated in accordance with the right-eye display direction of the previous user viewpoint position 2A (for example, 2A1 in Figure 26) in the RGB display elements 302' corresponding to each display direction 501, for example, numbers 06 to 09 in Figure 26, between the user viewpoint position direction 502 (for example, 502A in Figure 26) passing through the previous user viewpoint position 2A (for example, 2A1 in Figure 26) and the midpoint direction 503 (for example, 503AB in Figure 26) calculated this time (step S1103). This setting process is performed on the RGB display elements 304' of each pixel group 304 (see Figure 25) constituting the transparent display device 30, between the user viewpoint position direction 502 relative to the pixel group 304 and the midpoint direction 503 determined for the pixel group 304 in step S1102.

[0191] Next, the processing unit 14 sets the left-eye display image data generated in accordance with the display direction of the left eye at the current user viewpoint position 2A (for example, 2A2 in Figure 26) to the RGB display elements 302' corresponding to each display direction 501, for example, numbers 10 to 13 in Figure 26, between the midpoint direction 503 (for example, 503AB in Figure 26) calculated in step S1102 and the user viewpoint position direction 502 (for example, 502B in Figure 26) that passes through the current user viewpoint position 2A (for example, 2A2 in Figure 26) (step S1104). This setting process is performed on the RGB display elements 304' of each pixel group 304 (see Figure 25) constituting the transparent display device 30, between the midpoint direction 503 determined for the pixel group 304' in step S1102 and the user viewpoint position direction 502 for the pixel group 304'.

[0192] Subsequently, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1 (step S1105), and determines whether that variable value exceeds the predetermined value indicating the last user viewpoint position 2A (step S1106).

[0193] If the final viewpoint position 2A has not yet been processed and the determination in step S1106 is NO, the processing unit 14 will execute the processes from steps S1102 to S1104 described above again.

[0194] By step S1102, as described above, for example in Figure 26, the midpoint direction 503BC, which is directed towards the midpoint between the previous viewpoint position 2A2 and the current user viewpoint position 2A3, is calculated from the pixel group 304.

[0195] Next, in step S1103, similar to the one described above, the right-eye display image data generated to correspond to the right-eye display direction at the previous viewpoint position 2A2 is set in the RGB display elements 304' of each pixel number corresponding to each display direction 501 from, for example, number 14 to 15 in Figure 26, between the user viewpoint position direction 502B passing through the previous viewpoint position 2A2 and the midpoint direction 503BC calculated this time. This setting process is performed on the RGB display elements 304' of each pixel group 304 (see Figure 25) constituting the transparent display device 30, between the user viewpoint position direction 502 relative to the pixel group 304 and the midpoint direction 503 determined for the pixel group 304 in step S1102.

[0196] Next, in step S1104, similar to the one described above, for example in Figure 26, the left-eye display image data generated to correspond to the left-eye display direction at the current viewpoint position 2A3 is set in the RGB display elements 304' of each pixel number corresponding to each display direction 501 from 16 to 18 in Figure 26, between the midpoint direction 503BC calculated in step S1102 and the user viewpoint position direction 502C passing through the current viewpoint position 2A3. This setting process is performed on the RGB display elements 304' of each pixel group 304 (see Figure 25) constituting the transparent display device 30, between the midpoint direction 503 determined for the pixel group 304' in step S1102 and the user viewpoint position direction 502 relative to the pixel group 304'.

[0197] Subsequently, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1 (step S1105), and determines whether that variable value exceeds the predetermined value indicating the last user viewpoint position 2A (step S1106).

[0198] If the final viewpoint position 2A has not yet been processed and the determination in step S1106 is NO, the processing unit 14 will execute the processes from steps S1102 to S1104 described above again.

[0199] By step S1102, as described above, for example in Figure 26, the midpoint direction 503CD, which is directed towards the midpoint between the previous viewpoint position 2A3 and the current viewpoint position 2A4, is calculated from the pixel group 304.

[0200] Next, in step S1103, similar to the one described above, the right-eye display image data generated to correspond to the right-eye display direction of the previous viewpoint position 2A3 is set in the RGB display elements 304' of each pixel number corresponding to each display direction 501 from, for example, number 19 to 20 in Figure 26, between the user viewpoint position direction 502C passing through the previous user viewpoint position 2A3 and the midpoint direction 503CD calculated this time. This setting process is performed on the RGB display elements 304' of each pixel group 304 (see Figure 25) constituting the transparent display device 30, between the user viewpoint position direction 502 relative to the pixel group 304 and the midpoint direction 503 determined for the pixel group 304 in step S1102.

[0201] Next, in step S1104, similar to the one described above, for example in Figure 26, the left-eye display image data generated to correspond to the display direction of the left eye at the current viewpoint position 2A4 is set in the RGB display elements 304' of each pixel number corresponding to each display direction 501, for example, from 21 to 22 in Figure 26, between the midpoint direction 503CD calculated in step S1102 and the user viewpoint position direction 502D passing through the current user viewpoint position 2A4. This setting process is performed on the RGB display elements 304' of each pixel group 304 (see Figure 25) constituting the transparent display device 30, between the midpoint direction 503 determined for the pixel group 304' in step S1102 and the user viewpoint position direction 502 relative to the pixel group 304'.

[0202] Subsequently, as described above, the processing unit 14 increments the variable value indicating the viewpoint position 2A by 1 (step S1105), and determines whether that variable value has exceeded the predetermined value indicating the last viewpoint position 2A (step S1106).

[0203] After processing has been completed up to the last user viewpoint position 2A, if the determination in step S1106 is YES, the processing unit 14 executes the processing in the final step S1107. In step S1107, the processing unit 14 sets the right-eye display image data, which was generated in accordance with the display direction of the right eye at the last user viewpoint position 2A (for example, 2A4 in Figure 26), to the RGB display elements of each pixel number corresponding to each display direction 501 from, for example, 23 to 29 in Figure 26, between the user viewpoint position direction 502 (for example, 502D in Figure 26) passing through the current viewpoint position 2A (for example, 2A4 in Figure 26) and the 29th display direction 501. This setting process is performed on the RGB display elements 304' of each pixel group 304 (see Figure 25), which are determined according to the user viewpoint position direction 502 from the pixel group 304 toward the last viewpoint position 2A.

[0204] Subsequently, the processing unit 14 terminates the display process of step S104 in Figure 6, as illustrated in the flowchart of Figure 27.

[0205] In the image display process illustrated in the flowchart of Figure 27, in Figure 26, the right-eye or left-eye display image data, generated to correspond to the right-eye or left-eye display direction of the adjacent viewpoint position 2A, is copied and set in each display direction 501 of the section delimited by the user viewpoint position direction 502 of the two viewpoint positions 2A and the midpoint direction 503 passing through the midpoint between those viewpoint positions 2A. Alternatively, by changing the algorithm for mapping display pixels to each display direction 501 between the viewpoint position 2A of user 2 that is close to the transparent display device 30 and the viewpoint position 2A of user 2 that is far from the transparent display device 30, the sensitivity of the display pixel changes can be made uniform according to the viewpoint position A of user 2. Or, instead of simply copying, display pixels calculated by interpolation processing according to each display direction 501 may be set between two adjacent display images. Furthermore, the specific viewpoint position 2A acquired by the measuring device 20 described above and used for processing is not particularly limited, but for example, the midpoint between each user 2's eyes, the center point of the head, or a position a predetermined distance inside the head from the center of both eyes can be used as the viewpoint position 2A. Another example is that the positions of each user 2's left and right eyes may be acquired separately and used for processing. Based on the separately acquired left and right eye position information, the displayed image of the virtual object Vobj may be subjected to projection transformation, perspective projection transformation, or similar calculation processing according to the left and right eye positions of each user 2.

[0206] Thus, according to this embodiment, for multiple users 2 with naked eyes who do not wear the 3D glasses device, for each of the viewpoint positions 2A1 to 2A4 of each user 21 to 24, when the user's right eye and left eye, each at a certain distance to the left and right of that viewpoint position, view a three-dimensional, stereoscopic virtual object in the virtual space via the transparent display device 30, the display image of the virtual object can be superimposed onto the real space visible through the transparent display device 30 as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes, so that it appears to each user 2 as if the virtual object exists in the real space, with geometric rigor and plausibility. In addition, according to this embodiment, different display images of the virtual object corresponding to each user 2's viewpoint position 2A can be simultaneously presented to each user 2. <Calibration process> Since the direction in which light emitted from a pixel is refracted by the lens element 303 differs depending on the pixel number of the pixel group 304, it is required that the pixels and lens elements be in a desired positional relationship with high precision in the transmissive display device 30. However, it can be difficult to manufacture the transmissive display device 30 so that the pixels and lens elements are in a desired positional relationship with high precision. For example, when manufacturing the transmissive display device 30 by bonding a transparent sheet on which multiple cylindrical lens elements 303 are formed to a transparent substrate 301 on which a pixel row 300 is arranged, it can be difficult to accurately position the lens elements 303 relative to the pixels due to the expansion and contraction of the transparent sheet and errors in the position and angle of the transparent sheet relative to the transparent substrate 301. Also, for example, when bonding multiple cylindrical lens elements 303 individually to a transparent substrate 301 on which a pixel group 304 is arranged in a matrix, it can be difficult to accurately position and bond the lens elements 303 relative to the pixels. Furthermore, for example, when printing lens elements 303 onto a transparent substrate 301 on which pixel rows 300 are arranged, in a process separate from the formation of the pixel rows 300, it can be difficult to accurately position and adhere the lens elements 303 relative to the pixels.

[0207] Therefore, errors may occur in the transmissive display device 30 of this embodiment during manufacturing, etc. In particular, errors may occur in the position relative to the pixels 304' when the lens element 303 is provided on the transparent substrate 301. The transmissive display device 30 is configured to display an appropriate image toward the user's viewpoint by accurately refracting the light emitted from each pixel 304' in a predetermined direction using the lens element 303. Therefore, if there is an error in the relative position between the lens element 303 and the pixels 304', the image will deteriorate. The calibration process is a process to improve image deterioration caused by individual differences in the transmissive display device 30 due to manufacturing, etc. It is preferable to perform this process at least once for each individual transmissive display device 30.

[0208] Since the calibration process in this embodiment is basically the same as the calibration process in the third embodiment shown in Figure 15, the calibration process in this embodiment will be described with reference to Figure 15.

[0209] With the adjuster looking at the screen of the transparent display device 30, or with a calibration imaging device (camera) installed at that position, the processing unit 14 acquires information about the adjuster's viewpoint or the position of the imaging device (step S1001). Next, the processing unit 14 displays a specific image in a predetermined width range to the left and right, centered on the direction of the viewpoint. The specific image is not particularly limited. For example, vertical stripes, diagonal stripes, grid patterns, etc., can be used as the specific image. Subsequently, the processing unit 14 provides the adjuster with a parameter adjustment GUI that allows adjustment of which range of pixels 304' to use in the image group 304, and prompts the viewer to adjust the parameters so that the specific image is clearly visible from the adjuster's viewpoint or in the captured image from the imaging device, or automatically adjusts the parameters through calculation processing based on the specific image. Then, while moving the adjuster's viewpoint or the imaging device (step S1005), the processing unit 14 repeats the processes of steps S1001-S1003 until a predetermined termination condition is met (step S1004).

[0210] In this case, for example, the number of adjustments can be predetermined so that the viewpoint position or the position of the imaging device is evenly set to the left and right of the screen, and the calibration can be completed by repeating the adjustments until that number of adjustments is reached. Also, the process in step S1003 can be performed while viewing with both eyes, or separately with each eye. The same can be done using an imaging device. Furthermore, the parameters adjusted in step S1003 are not particularly limited. For example, a combination of parameters can be used, such as a parameter to shift the pixels 304' used in the pixel group 304 left and right across the entire screen, a parameter to adjust the angle at which the pixels 304' used in the image group 304 are tilted vertically across the screen, and a parameter to adjust the degree to which the pixels 304' used in the image group 304 are gradually shifted horizontally across the screen. These parameters can be adjusted for the entire screen, or they can be adjusted for each pixel group 304 corresponding to each individual pixel.

[0211] The shape and arrangement of the lens element 303 of the transmissive display device 30 in this embodiment described above can be modified in various ways.

[0212] In this embodiment, the pixels 304' of the pixel group 304 in the pixel array 302 are, for example, miniLEDs, but other elements such as organic EL elements may also be used. Furthermore, pixels 304' may be self-emissive elements such as miniLEDs, microLEDs, or organic EL elements, or they may be elements that control light transmission using a liquid crystal mechanism. Also, pixels 304' may be elements or light-emitting materials that reflect externally projected light to allow the user to perceive color. In this embodiment, RGB elements are exemplified as pixels 304', but monochromatic pixels such as green or red may also be used.

[0213] [Fifth Embodiment] The second to fourth embodiments described above illustrate an image display system that shows a parallax-laden three-dimensional image of a virtual object to multiple users via a transparent display. As an example of a common structure in these embodiments, as shown in Figure 28, a transparent member 303, either a lens element alone or including a lens element, is installed on a transparent substrate 301 having multiple pixels 304'. In this case, a lenticular lens (cylindrical lens) is used as an example of the lens element 303.

[0214] Figure 29 shows an apparatus formed by bonding a transparent substrate 301, which has one representative pixel 304' as shown in Figure 28, with a transparent member 303 equipped with a lenticular lens (cylindrical lens) as a lens element, as well as the optical path 401 of light incident perpendicular to the surface of the transparent substrate 301 from the pixel 304' toward the transparent member 303, and a measurement surface 402 that is virtually positioned parallel to the transparent substrate 301 at a certain distance away from the transparent substrate 301 toward the transparent member 303.

[0215] The measuring surface 402 is assumed to be the viewpoint position of a user observing the image displayed on this display device. In Figure 29, a transparent substrate with only one pixel 304' is assumed, but the point-like light incident from the pixel 304' passes through the cylindrical lens on the transparent member 303, is expanded horizontally, and deforms into a shape with a longitudinal direction in the horizontal direction (horizontally elongated shape) before passing through the measuring surface. The shape of the optical path 401 when it intersects the measuring surface is, that is, the shape of the light visible to the user when the user's viewpoint is placed on the measuring surface. Figure 29 shows an example of a transparent substrate 301 with only one pixel 304', but in embodiments 2 and 3, the pixels 304' are arranged continuously in the horizontal direction, for example, and in embodiment 4, the pixel unit is composed of, for example, an aggregate of 8 × 4 = 32 pixels in the horizontal and vertical directions, and one example of a configuration in which these are arranged in a matrix on the transparent substrate 301 is described. In both configurations, the image displayed to the user is determined by the illumination or de-illumination of multiple pixels. For each individual pixel constituting the image, an aspect ratio close to 1:1 results in a more natural-looking image. In a configuration where only one lenticular lens layer is passed through, as shown in Figure 29, the pixels appear extremely elongated horizontally, which may cause discomfort to the user when the image is formed from these pixels.

[0216] In the above explanation, an example is shown where the longitudinal direction of the lenticular shape is oriented vertically (Y direction), and it is explained that the point-like light incident from pixel 304' is magnified horizontally. In the fourth embodiment, the longitudinal direction of the lenticular shape is tilted by an angle θ, but the explanation is the same as above. In a configuration where only one layer of lenticular light passes through, the appearance of the pixels becomes extremely elongated in the direction tilted by an angle θ, which may cause discomfort to the user when the collection of pixels is formed into an image.

[0217] As a method to reduce the unnatural appearance of these pixels, in this embodiment, as shown in Figure 30, a new lens element of a different structure or a transparent member 403 containing a lens element is inserted between the transparent substrate 301 equipped with pixels and the lens element alone or the transparent member 303 containing the lens element. In the example in Figure 30, the transparent member 303 is a lenticular lens with its longitudinal direction in the Y direction, while 403 is a lenticular lens with its longitudinal direction in the X direction.

[0218] Figure 31 shows an apparatus comprising a transparent substrate 301 having one representative pixel 304' as shown in Figure 30, a lenticular lens having a longitudinal direction in the X direction bonded between a lenticular lens having a longitudinal direction in the Y direction as a lens element, an optical path 401 of light incident perpendicular to the surface of the transparent substrate 301 from the pixel 304' toward the transparent member 303, and a measurement surface 402 virtually positioned as a plane parallel to the transparent substrate 301 at a certain distance away from the transparent substrate 301 toward the transparent member 303.

[0219] The measurement surface 402 is assumed to be the viewpoint position of the user observing the image displayed on this display device. In Figure 31, a transparent substrate with only one pixel 304' is assumed, but the point-like light incident from pixel 304' passes through the cylindrical lens on the transparent member 403, is magnified vertically, and deforms into a shape with a longitudinal direction in the vertical direction (elongated shape). Subsequently, passing through the cylindrical lens on the transparent member 303, it is further magnified horizontally, the elongated shape is softened, and it passes through the measurement surface in a shape closer to a square. For the appearance of a single pixel that makes up the image, an aspect ratio close to 1:1 results in a more natural image, but in the configuration shown in Figure 31, where two lenticular layers are arranged in mutually intersecting directions, the aspect ratio of the pixel's appearance is close to 1:1, so when the image is formed from the collection of pixels, the discomfort to the user is reduced. To make the shape of the optical path in Figure 31 easier to understand, the appearance of the YZ plane and XZ plane is schematically shown in Figure 32.

[0220] Figure 33 shows the results of observing the appearance of light incident from one pixel on the measurement surface 402 after actually fabricating a display with a configuration similar to that of Figure 31. The left figure of Figure 33 shows the observation results of an illuminated pixel in a display composed only of a transparent substrate 301 equipped with pixels, the center figure of Figure 33 shows the observation results of an illuminated pixel in a display composed of a single layer of lenticular material similar to that of Figure 29, and the right figure of Figure 33 shows the observation results of an illuminated pixel in a display composed of two layers of lenticular material similar to that of Figure 31. The upper half of Figure 33 shows the actual imaging results, and the lower half of Figure 33 shows the outline of the part where the optical path of light from the pixel intersects with the measurement surface 402 in the upper half of the imaging results, with the outline emphasized with a solid line for explanatory purposes. As shown in the observation results on the right side of Figure 33, the actual appearance of the light is such that the brightness decreases as you move upward or downward from the center, and a long, thin, streaky shape is observed. This occurs because the radius of curvature of the lenticular structure on the transparent member 403, through which light passes first, is smaller than the radius of curvature of the lenticular structure on the transparent member 303, through which light passes later. The smaller the radius of curvature of the former, the thinner the streaky shape becomes, making it difficult for the observer to distinguish the streaky shape with the naked eye. Conversely, the larger the radius of curvature of the former, the thicker the streaky shape becomes.

[0221] In this embodiment, the transparent substrate 301 is shown as having only one pixel as an example, but it is not limited to this. As shown in Figure 34, the display may have pixels uniformly distributed across the entire surface or a wide area of ​​the substrate 301. It may be a transparent organic EL (OLED) display, mini-LED / micro-LED display, or liquid crystal display, or a projection method, or any method that controls how light is seen. It may also be a conventional display device that does not transmit light. In addition to the transparent displays described above, the display method can also be applied to non-transparent liquid crystal displays, organic EL (OLED) displays, mini-LED / micro-LED displays, and projection methods using projectors.

[0222] In this embodiment, various designs and combinations of lenticular pitches and radii of curvature are possible for each of the two lenticular layers. As shown in Figure 30, the design is not limited to using a lenticular layer with a large pitch and radius of curvature on the outer layer; shapes using a lenticular layer with a small pitch and radius of curvature on the outer layer, as shown in Figure 35, can also be designed. The thickness of each lenticular layer is also an important factor in the design, as it affects the appearance. The design should be appropriate to the application of this technology, taking into account the position of the measurement surface 402. Furthermore, while Figure 30 shows an example where the two lenticular layers are installed with their longitudinal directions perpendicular to each other, this is not the only option. While orthogonal alignment results in a light appearance closer to a square, it is also possible to design the appearance to be closer to a rhombus or parallelogram, depending on the application of this technology. One of the features of this technology is that it enables a user-friendly appearance, but what constitutes a user-friendly appearance varies depending on the application. It is also possible to use a multi-layer structure with three or four layers instead of two, resulting in a light appearance closer to a polygon or circle.

[0223] In this embodiment, particularly in applications where transparency is important, it is possible to arrange the lenticular layers in each layer separately, or to provide flat areas in between. Figure 36 shows an example of this, a combination of transparent members designed so that lenticular and flat areas alternate in the outer layer. In this example, the inner lenticular layer is also designed to be flat in the area that overlaps with the flat area of ​​the outer lenticular layer. As a result, the path of light in the Z-axis direction of Figure 36 is not greatly affected in the flat area, and this area can be used as a transparent area to see through to the back. Increasing the width of the flat area increases the overall transparency of the device, but on the other hand, it reduces the number of pixels used for image representation, lowering the image resolution and the number of image parallaxes that can be displayed simultaneously. When creating a device that allows transparency and superimposes the scenery in the background, a trade-off arises between this transparency and the quality and function of image representation, so it is necessary to design considering the size of the pixels and the application.

[0224] Similar to the second to fourth embodiments, this embodiment also provides a means for presenting a virtual object display image as a three-dimensional image (stereoscopic image) consisting of two images that provide parallax between the left and right eyes, to a naked-eye user who does not wear a head-mounted display or 3D glasses device.

[0225] Figure 37 shows an example of a display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 7 described in the second embodiment, another lens layer 403 is superimposed on the lens layer 303. This makes it possible to create a more natural viewing experience for the user compared to the case where only the lens layer 303 is provided.

[0226] (Variation 1) Figure 38 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 11 described in the second embodiment, another lens layer 403 is superimposed on the lens layer 303. This makes it possible to create a more natural viewing experience for the user compared to the case where only the lens layer 303 is provided.

[0227] (Modification 2) Figure 39 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 12 described in the third embodiment, another lens layer 403 is superimposed on the lens layer 303. This makes it possible to create a more natural viewing experience for the user compared to the case where only the lens layer 303 is provided.

[0228] (Variation 3) Figure 40 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 16 described in the third embodiment, another lens layer 403 is superimposed on the lens layer 303. This makes it possible to create a more natural viewing experience for the user compared to the case where only the lens layer 303 is provided.

[0229] (Modification 4) Figure 41 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 17 described in the third embodiment, another lens layer 403 is superimposed on the lens layer 303. This makes it possible to create a more natural viewing experience for the user compared to the case where only the lens layer 303 is provided.

[0230] (Variation 5) Figure 42 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 18 described in the third embodiment, another lens layer 403 is superimposed on the lens layer 303. This makes it possible to create a more natural viewing experience for the user compared to the case where only the lens layer 303 is provided.

[0231] (Experimental variation 6) Figure 43 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 19 described in the third embodiment, another lens layer 403 is superimposed on the lens layer 303. This makes it possible to create a more natural viewing experience for the user compared to the case where only the lens layer 303 is provided.

[0232] (Example 7) Figures 44 and 45 are diagrams used to illustrate a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 20 described in the fourth embodiment, this configuration involves layering another lens layer 403, as shown in Figure 44, on top of the lens layer 303 (Figure 45). This makes it possible to create a more natural viewing experience for the user compared to the structure in Figure 20, which only has the lens layer 303.

[0233] In Figure 20, the pixel group 304 does not fit entirely within the width of the lens element 303. As an alternative, as shown in Figure 46, a configuration in which the pixel group 304 fits entirely within the width of the lens element 303 can also be considered. This structure is conceivable not only in the fifth embodiment but also in the fourth embodiment.

[0234] (Variation 8) Figure 47 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 45 described in Modification 7 of the fifth embodiment, it has a structure in which pixels are uniformly arranged on a transparent substrate 301. In this structure, if the gaps between pixels are transparent, or if the pixels are made of liquid crystal that can control transparency and opacity, it is possible to have the same functions as in Modification 7.

[0235] (Extreme variation 9) Figure 48 shows a modified example of the display device in the system of the fifth embodiment. Based on the structure of the display device in Figure 45 described in Modification 7 of the fifth embodiment, the substrate 301 has a structure in which pixels are uniformly arranged. In addition, it is assumed that the substrate 301 is an opaque display. In this case, by having both the lens layer 303 and the lens layer 403 have a lenticular structure across their entire surface without a flat structure, it shows one possible structure that can be used as an opaque naked-eye stereoscopic display.

[0236] One example of applying the transparent display device 30 in this embodiment is to mount the transparent display device 30 on the glass portion of an automobile or the like, and present a three-dimensional display image of a virtual object to each of multiple users 2 seated in the front and rear seats of the vehicle. With the transparent display device 30 of this embodiment, it is possible to individually display a display image of a virtual object to multiple users 2 who have different distances and viewing directions from the transparent display device 30. Therefore, it is possible to superimpose a virtual object onto the real space visible around the transparent display device 30, which is the glass of a moving automobile, according to the view from each user 2's viewpoint.

[0237] In this embodiment, the lens element 303 may be a lenticular lens made of a cylindrical lens, or any three-dimensional lens shape such as a hemispherical semi-ellipsoid or toroidal shape that controls the direction of light. Alternatively, instead of using a lens, a material such as a film or plate that generates a parallax barrier may be used.

[0238] In this embodiment, pixel 304' is an organic EL element as an example, but other elements may be used. For example, pixel 304' may be a self-emissive element such as a mini-LED or micro-LED, or an element that controls light transmission using a liquid crystal mechanism. Furthermore, pixel 304' may be an element or light-emitting material that reflects externally projected light to allow the user to perceive color. In this embodiment, RGB elements are used as an example for pixel 304', but single-color pixels such as green or red may also be used.

[0239] [Note] The embodiments described above include the following; however, the embodiments described above are not limited to those described below.

[0240] (Item 1) The display device is A quasi-planar substrate including a planar or curved surface, A pixel population including a plurality of display pixels at different positions in a first direction parallel to the tangent plane of the substrate, A first lens element configured to refract light incident from the plurality of display pixels of the pixel population in a third direction perpendicular to the tangent plane of the substrate in different directions within a plane including the first direction and the third direction, A second lens element configured to refract light incident from the display pixels in the third direction in a plane including a second direction that intersects the first direction at a specific angle within the tangent plane of the substrate and the third direction, It has.

[0241] (Item 2) In the display device according to claim 1, In the substrate, There are a plurality of the pixel populations, which are arranged at a predetermined first distance at intervals in the first direction and at a predetermined second distance at intervals in the second direction, There are a plurality of the first lens elements, which are arranged at the first distance at intervals in the first direction so as to overlap the pixel population, There are a plurality of the second lens elements, which are arranged at the second distance at intervals in the second direction so as to overlap the pixel population.

[0242] (Item 3) In the display device according to Item 1, In the third direction, the first lens element is arranged between the pixel population and the second lens element.

[0243] (Item 4) In the display device according to Item 1, The first lens element is a cylindrical lens that acts as a lens in the first direction and does not act as a lens in the second direction, The second lens element is a lens element in which a plurality of cylindrical lenses that act as lenses in the second direction and do not act as lenses in the first direction are adjacent to each other in the second direction.

[0244] (Item 5) In the display device according to Item 1, A plurality of display pixels in the pixel group are arranged in a matrix according to a first axis and a second axis that are orthogonal to each other, The first lens element refracts light from each of the display pixels included in the pixel group in a direction corresponding to the position of the display pixel in a direction inclined by a predetermined angle with respect to the first axis.

[0245] (Item 6) In the display device according to Item 5, It further includes a plurality of first wirings parallel to the first axis and a plurality of second wirings parallel to the second axis, and has a passive matrix driving circuit that causes each of the display pixels in the pixel group to emit light in a passive matrix method.

[0246] (Item 7) In the display device according to Item 5, It further includes a plurality of first wirings parallel to the first axis, a plurality of second wirings parallel to the second axis, and switching elements connected between the first wiring and the second wiring, and has an active matrix driving circuit that causes each of the display pixels in the pixel group to emit light in an active matrix method.

[0247] (Item 8) In the display device according to Item 1, The display pixel is an LED element.

[0248] (Item 9) In the display device according to Item 1, The display pixel is an organic EL element.

[0249] (Item 10) In the display device described in item 1, The aforementioned display pixels are liquid crystal elements.

[0250] (Item 11) In the display device described in item 1, The aforementioned display pixels are elements illuminated by a projector.

[0251] The embodiments of the present invention described above are illustrative for the purpose of explaining the present invention and are not intended to limit the scope of the present invention to those embodiments only. Those skilled in the art can implement the present invention in various other forms without departing from the spirit of the invention. [Explanation of Symbols]

[0252] 1…Image display system, 2…User, 2A…Viewpoint position, 10…Image generation device, 20…Measurement device, 30…Display device

Claims

1. A semi-planar substrate including a flat or curved surface, A group of pixels including a plurality of display pixels having different positions in a first direction parallel to the tangent plane of the substrate, A first lens element is configured to refract light incident from a plurality of the display pixels of the pixel group in a third direction perpendicular to the tangent plane of the substrate in separate directions within a plane including the first and third directions. A second lens element is configured to refract light incident from the display pixel in the third direction within a plane that includes a second direction and a third direction that intersect the first direction at a specific angle within the tangent plane of the substrate, A display device having the following features.

2. In the substrate, there are a plurality of pixel groups, which are arranged at predetermined first distances with spacing in the first direction and at predetermined second distances with spacing in the second direction. There are multiple first lens elements, which are arranged at intervals of the first distance in the first direction so as to overlap with the pixel group, There are multiple second lens elements, which are arranged at intervals of the second distance in the second direction so as to overlap with the pixel group, The display device according to claim 1.

3. In the third direction, the first lens element is positioned between the pixel group and the second lens element. The display device according to claim 1.

4. The first lens element is a cylindrical lens that acts as a lens in the first direction and does not act as a lens in the second direction. The second lens element is a lens element in which a plurality of cylindrical lenses that act as lenses in the second direction but do not act as lenses in the first direction are arranged adjacent to each other in the second direction. The display device according to claim 1.

5. The multiple display pixels in the aforementioned pixel group are arranged in a matrix according to a first axis and a second axis that are orthogonal to each other. The first lens element refracts light from each of the display pixels included in the pixel group in a direction corresponding to the position of the display pixel in a direction tilted by a predetermined angle with respect to the first axis. The display device according to claim 1.

6. The device further includes a passive matrix drive circuit that includes a plurality of first wirings parallel to the first axis and a plurality of second wirings parallel to the second axis, and causes each of the display pixels in the pixel group to emit light in a passive matrix manner. The display device according to claim 5.

7. The present invention further comprises an active matrix driving circuit that includes a plurality of first wirings parallel to the first axis, a plurality of second wirings parallel to the second axis, and a switching element connected between the first wirings and the second wirings, and causes each of the display pixels of the pixel group to emit light in an active matrix manner. The display device according to claim 5.

8. The aforementioned display pixel is an LED element. The display device according to claim 1.

9. The aforementioned display pixel is an organic EL element. The display device according to claim 1.

10. The aforementioned display pixel is a liquid crystal element. The display device according to claim 1.

11. The aforementioned display pixels are elements illuminated by a projector. The display device according to claim 1.

Images