Aerial image generation device, distortion correction function generation method, and aerial image generation method

Abstract

To generate an aerial image in a predetermined space in which video image distortion due to a transparent or reflective object is compensated.SOLUTION: Each pixel in the angle of view of a CG camera is selected. A light ray having a pixel value of an ideal camera video image corresponding to the location of the selected pixel is emitted from the CG camera. After that, by recording the pixel values of the light rays having passed through transparent and / or reflective objects and retro-reflective transparent optical elements, an image for calculation is generated, from which a distortion correction function is generated. A generated distortion correction function is then used to give an input image signal distortion in accordance with an optical system to generate a distortion corrected image display light source. From the generated distortion corrected image display light source, an areal image is generated as a real image in a predetermined space.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an aerial image generation device, a distortion correction function generation method, and an aerial image generation method for displaying an aerial image when a transparent object or a reflective object exists on an optical path.

Background Art

[0002] There is known a technique of reflecting and refracting light emitted from a light source by an optical element or the like to form an aerial image (real image) at a predetermined position in a space where no screen or display is arranged. Since the aerial image gives an observer a strange feeling, it is used in various applications such as virtual reality (VR) and augmented reality (AR).

[0003] In video display in virtual reality and augmented reality, it is important not only to simply convey information to an observer but also to attract the observer's attention and realize a natural interaction with a real object. In that sense, video projection onto a real object has attracted attention in that it can give a dynamic and interactive expression to static buildings and natural landscapes.

[0004] Patent Document 1 describes an aerial image forming device with a high degree of freedom in installation location and in which the display content of the display unit is not directly visible from the observer side, and a technique of forming an aerial image by recursively transmitting incident light using a recursive transmission optical element composed of micro-mirror array plates (MMAPs).

[0005] Patent Document 2 describes an image processing device for generating a high-quality image for projection, which calculates correction parameters for correcting a projection image and corrects the pixel positions constituting the projection image based on the calculated correction parameters and data related to the installation state of the projection unit.

[0006] The technique described in Patent Document 2 prepares an image for calculating correction parameters such as a white image and a lattice image, and calculates the correction parameters necessary for geometrically transforming the image projected onto the output device. Further, virtual space data including data on the shape and reflection characteristics of a virtual subject installed in the virtual space and position data of the virtual camera is generated. The technique described in Patent Document 2 corrects the pixel positions constituting the projected image from these correction parameters and the virtual space data.

[0007] In addition, as a technique for displaying an image within a transparent object, there is a technique described in Non-Patent Document 1. Also in the optical system described in this Non-Patent Document 1, recursive transmission optical elements (MMAPs) are used, and using these MMAPs, light from a light source object is imaged within the transparent object. The technique described in Non-Patent Document 1 further arranges a mirror image of the transparent object that is the image display target on the light source side to compensate for the distortion caused by the refraction of this transparent object.

[0008] The optical system described in Non-Patent Document 1 is an optical system for presenting a realistic aerial image (real image) in which an image floats within a transparent object without impairing the appearance of the transparent object.

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0010]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0011] However, in the conventional video display methods described in Patent Documents 1 and 2, the video display surface is limited to the inside of a hollow or the surface of an object, and there is a problem that it is difficult to embed a video inside a transparent object. That is, when imaging a video inside a non-hollow transparent object, since the change in the optical path due to refraction occurring at the boundary surface of the object cannot be ignored, even if an attempt is made to display a video inside the transparent object, the video cannot be correctly displayed at the desired position.

[0012] In addition, in the optical system described in Non-Patent Document 1, only the refractive distortion generated on the light source side is the subject of compensation for the sense of reality of the video. For this reason, there is a problem that the refractive distortion generated in the viewing direction cannot be compensated. In addition, in the optical system described in Non-Patent Document 1, a mirror image of the transparent object targeted on the light source side or a transparent object obtained by cutting the transparent object at the video display position is required, but it is difficult to prepare a compensation transparent object for each shape and arrangement method of the transparent object.

[0013] An object of the present invention is to provide a virtual image generation device, a distortion correction function generation method, and a virtual image generation method capable of generating a distortion-free virtual image at a preset spatial position even when there is a transparent object or a reflective object in the optical path from a light source.

Means for Solving the Problems

[0014] To solve the above problems, a virtual image generation device of the present invention includes an image processing unit that generates a distortion correction image that distorts an input image signal, a distortion correction image storage unit that stores the distortion correction image generated by the image processing unit, a distortion correction image display light source that uses the distortion correction image stored in the distortion correction image storage unit as a light source, In the optical path between the distortion correction image display light source and the aerial image, there is at least one of a transparent object that transmits light and a reflective object that reflects light, and a recursive transmission optical element. a virtual image optical system that receives the light emitted from the distortion correction image display light source and generates a virtual image as a real image at a predetermined spatial position, A light source distortion correction function generation unit that generates a distortion correction function based on a calculation image obtained by recording the pixel values of the light rays passing through the transparent object and / or the reflective object installed in the aerial image optical system and the recursive transmission optical element at the light source installation position of the actual machine. and is provided with. Then, the image processing unit generates a distortion correction image so that the distortion of the virtual image given by the virtual image optical system is corrected.

[0015] In addition, the method for generating a distortion correction function of the present invention includes steps (1) to (3) in the CG space, and the method for generating an aerial image of the present invention includes steps (4) and (5) in the actual machine in addition to steps (1) to (3) in the CG space. (1) Step of designing an aerial image optical system including the arrangement relationship of a transparent object or a reflective object and a recursive transmission optical element. (2) Step of outputting an ideal camera image by rendering a portion of an ideal image arranged at the aerial image display position in the CG space within the viewing angle of a CG camera corresponding to the viewpoint. (3) Step of generating a distortion correction function from a calculated image generated by emitting light rays having the pixel values of each pixel of the ideal camera image from the CG camera and recording the pixel values of all pixels of the ideal camera image when the light rays reach the light source installation position of the actual machine through the aerial image optical system. (4) Step of generating a distortion-corrected image that distorts the input image signal using the distortion correction function, and generating a distortion-corrected image display light source based on this distortion-corrected image. (5) Step of arranging the aerial image optical system designed in step (1) to be designed and the distortion-corrected image display light source in the real space, receiving the light emitted from the distortion-corrected image display light source, and generating an aerial image as a real image at a predetermined spatial position.

Advantages of the Invention

[0016] According to the present invention, it becomes possible to generate an aerial image in which video distortion due to a transparent object or a reflective object is compensated in a predetermined space. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0018] Hereinafter, the basic configuration and embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, elements having substantially the same configuration or function are denoted by the same reference numerals, and redundant explanations are omitted.

[0019] <Overall Configuration of the Aerial Image Generation Device> FIG. 1 is a diagram showing the overall configuration of the aerial image generation device 100 according to an embodiment of the present invention. Generally, when generating an aerial image at a position inside a transparent object or at a position passing through a transparent object, since the light rays transmitted through the transparent object are refracted, distortion occurs in the aerial image. Therefore, it is necessary to prevent distortion from occurring in the shape and color of the video generated in the air.

[0020] As shown in FIG. 1, the aerial image generation device 100 of the present embodiment includes a distortion correction function generation unit 101 for a light source. As will be described later, this distortion correction function generation unit 101 is a functional block that calculates the distortion of the video when passing through each optical system and derives a correction function to prevent distortion from occurring in the aerial image. It is prepared in advance outside the actual machine during aerial image formation.

[0021] In addition, the aerial image generation device 100 of the present embodiment includes an image processing unit 102 that performs distortion processing, a distortion correction image storage unit 103, a distortion correction image display light source 104, and an aerial image optical system 105. The distortion correction image storage unit 103, the distortion correction image display light source 104, and the aerial image optical system 105 together constitute an aerial image display unit.

[0022] An image signal serving as a basis for the aerial image and a distortion correction function generated by the distortion correction function generation unit 101 for the light source are supplied to the image processing unit 102, and distortion corresponding to a predetermined optical system described later is added to the supplied image signal by the distortion correction function. The corrected image (distortion-corrected image) with distortion added by the image processing unit 102 is recorded in the distortion correction image storage unit 103. Note that the image signal serving as a basis for the aerial image is an image signal in which an undistorted image is displayed by inputting it to a general display device.

[0023] Then, the distortion-corrected image stored in the distortion correction image storage unit 103 becomes the distortion correction image display light source 104, and an aerial image as a real image is displayed at a predetermined spatial position via the aerial image optical system 105. Although the details of the aerial image optical system 105 will be described later, it includes at least MMAPs, which are recursive transmission optical elements, and a transparent object (or a reflective object).

[0024] <Steps of the aerial image generation method> FIG. 2 is a process diagram of an aerial image generation method for generating an aerial image using the aerial image generation apparatus of the present embodiment. As shown in FIG. 2, in the aerial image generation method of the present embodiment, there are a process A of designing the arrangement of the aerial image optical system 105 and the distortion correction image display light source 104 shown in FIG. 1, a process B of rendering an ideal camera image by simulation using CG, a process C of generating a distortion correction function from a calculation image created using the ideal camera image, and a process D of applying distortion to an image signal using the generated distortion correction function and displaying the image at a predetermined spatial position. Note that process A is a design in the CG space, and process D is a process that is carried out in the actual machine by assembling the actual machine corresponding to the aerial image optical system designed in process A in the real space.

[0025] Here, processes A to C are processes in the CG space. The shooting by a camera in this CG space is called rendering in CG, and one method for this rendering is ray tracing. Hereinafter, the procedure for generating an aerial image in the aerial image generation apparatus and the aerial image generation method of the present embodiment will be described in the order of processes A to D.

[0026] <Design of the aerial image optical system 105> The aerial image optical system 105 shown in FIG. 1 is an optical system for generating a real image in a predetermined space by special optical elements such as a recursive transmission optical element MMAPs for the light of the light source. Thereby, it is possible to enable a video expression in which the video floats in the air.

[0027] The aerial image optical system 105 in the present embodiment is composed of an MMAPs 10 for generating an inclined aerial image 40 as shown in FIGS. 3(a) and 3(b), and an arbitrary-shaped transparent object 30 arranged at the position where the aerial image 40 is generated, or between the position where the aerial image 40 is generated and the MMAPs 10.

[0028] Fig. 3(a) shows an example of an optical system in which the virtual image 40 is generated inside the transparent object 30, and Fig. 3(b) shows an example of an optical system in which the virtual image 40 is generated outside the transparent object 30. Here, as the transparent object 30, a material with a smooth surface and high transparency is used. The distance between the transparent object 30 and the MMAPs 10 is designed to be the same as the distance L between the display serving as the light source 20 and the MMAPs 10.

[0029] Also, in Fig. 3(a), in order to display the virtual image 40 inside the transparent object 30, the size of the virtual image 40 needs to be smaller than the cross-section of the transparent object 30 at the imaging position. Although not shown in Fig. 3, the case where the transparent object 30 is arranged between the light source 20 and the MMAPs 10 is also conceivable.

[0030] First, the structure and function of the recursive transmission optical element MMAPs 10 used in this virtual image optical system will be described. The MMAPs 10 is an optical element that reflects and transmits the light incident from the light source 20 twice and forms a real image (virtual image 40) at a position symmetric with respect to the surface. Specifically, the MMAPs 10 has a configuration in which a plurality of micromirrors are arranged in a plane, deflects the light incident at 45° with respect to the plane by 90°, and forms a real image at a position symmetric with respect to the surface. By configuring the imaging positions of the light source 20 and the virtual image 40 as described above, the luminance of the virtual image by the MMAPs 10 becomes the highest.

[0031] Since the MMAPs 10 is easy to install and a virtual image 40 can be formed at a position symmetric with respect to the light source 20, it is used in digital signage, amusement facilities, and entertainment in general. However, in the MMAPs 10, due to two reflections, light attenuation occurs and the generation of stray light and the like is also considerable, so not all of the incident light can be used as the virtual image 40. It is said that among the incident light, the amount used as the virtual image 40 is 50% or less of the incident light.

[0032] <Rendering of Ideal Camera Images> As shown in FIGS. 3(a) and 3(b), when generating the aerial image 40, distortion occurs in the aerial image 40 due to the transparent object 30 intervening between the MMAPs 10 and the aerial image generation position, or between the light source 20 and the MMAPs 10. Therefore, in this embodiment, as shown in step B of FIG. 2, rendering of the ideal camera image by CG is performed. Here, the ideal camera image refers to an image obtained by rendering a portion within the viewing angle of the CG camera in a space defined in the computer (hereinafter referred to as the "CG space") that includes the ideal image. The CG camera is a camera that defines the origin of the light ray and the light ray emission range on the CG space in order to perform ray tracing in rendering.

[0033] FIGS. 4 and 5 are diagrams for explaining the rendering of the ideal camera image shown in step B of FIG. 2. As shown in FIG. 4, first, in the CG space, an ideal image is arranged at the position where the aerial image 40 shown in FIG. 3 is to be displayed. This ideal image is the image to be displayed inside and outside the transparent object. Here, an image with a color gradation in which the colors of all pixels are different is generated as the ideal image in the CG space.

[0034] Also, the ideal camera image shown in FIG. 4 is an image after the rendering of the CG space including the ideal image is completed. That is, the initial state of the ideal camera image is a state where no color is painted on all pixels. It can be said that the entire area within the viewing angle of the CG camera 50 is in a colorless state. From this state, by the method described later in FIG. 5, the image obtained by rendering the portion within the viewing angle of the CG camera 50 in the CG space including the ideal image becomes the ideal camera image shown in FIG. 4.

[0035] Therefore, the ideal camera image includes, in addition to the ideal image shown in FIG. 4, the outer frame portion outside the ideal image. Hereinafter, in this specification, an image obtained by shooting using an actual camera or rendering a portion within the viewing angle of the CG camera 50 in the CG space will be referred to as an "image", and an object displayed as a light source and an object generated by a process different from rendering will be referred to as an "image".

[0036] <Procedure for generating an ideal camera image> FIG. 5 is a flowchart showing a procedure for generating the ideal camera image described in FIG. 4. First, as shown in FIG. 4, an ideal image is arranged at the aerial image display position in the CG space (step S10). The ideal image in this CG space is an image light source displayed in a planar shape. Next, processing pixels for rendering a portion within the viewing angle of the CG camera 50 in the CG space including the ideal image arranged in step S10 are selected (step S11).

[0037] Here, as already described, a portion within the viewing angle of the CG camera 50 in the CG space including the ideal image is to be rendered. However, within the viewing angle of the CG camera 50, there are the ideal image and the outer region other than it. The selection of the processing pixels in step S11 means selecting pixels within the viewing angle of the CG camera 50 one by one. Naturally, this selection includes pixels in the outer region other than the ideal image.

[0038] Next, a light ray is emitted from the CG camera 50 toward the pixel selected in step S11 (hereinafter referred to as "selected pixel") (step S12). The process of this step S12 is performed for all pixels within the viewing angle of the CG camera 50 with respect to the CG space including the ideal image.

[0039] Then, it is determined whether or not the light ray emitted from the CG camera 50 intersects the ideal image (step S13). In step S13, when the light ray emitted from the CG camera 50 intersects the ideal image (YES in step S13), the pixel value of the pixel of the ideal image at the position where the light ray intersects, that is, the pixel value is calculated (step S14).

[0040] Also, when the light ray emitted from the CG camera 50 does not intersect the ideal image (NO in step S13), that is, when the intersection point is a portion of the outer frame other than the ideal image within the viewing angle of the CG camera 50, since there is no ideal image on the extension line of the light ray from the CG camera 50, the pixel value is not calculated (step S15). When the processes of step S14 and step S15 are completed, subsequently, it is determined whether the processing of all pixels of the CG camera 50 has been completed (step S16). If, in step S16, the processing of all pixels of the CG camera 50 has not been completed (NO in step S16), the processes of steps S11 to S16 are repeated until the processing of all pixels is completed.

[0041] If, in step S16, the processing of all pixels of the CG camera 50 has been completed (YES in step S16), all pixels of the CG camera 50 including the pixel values of the ideal image calculated in step S14 are output as the ideal camera video (step S17). Thereby, the generation process of the ideal camera video is completed.

[0042] <Method for Generating Distortion Correction Function> FIG. 6 is a flowchart showing the generation process of the distortion correction function which is the process of step C in FIG. 2. First, in the CG space, the transparent object 30, the MMAPs 10, and the light source installation position of the actual machine are set based on step A in FIG. 2 (step S20). Next, the CG camera 50 is arranged at the camera position of the aerial image optical system, and a processing pixel in the ideal camera video is selected (step S21). Then, a light ray having the pixel value of the selected pixel in the ideal camera video is emitted from the camera position (corresponding to the viewpoint position) (step S22).

[0043] Next, the next light ray path when the light ray emitted from the CG camera 50 in step S22 intersects the transparent object 30 is calculated (step S23). Here, although the transparent object 30 is used, there may be a case where the light ray path is changed by reflection on a reflective object (reflective surface) instead of the transparent object 30.

[0044] When the light ray transmitted through the transparent object 30 intersects the MMAPs 10, the next light ray path is calculated (step S24). Since the transparent object 30 may be between the MMAPs 10 and the light source position, in that case, the processes of step S23 and step S24 are interchanged.

[0045] Next, when a light ray having the pixel value of the processing pixel in the ideal camera image intersects the light source installation position of the actual machine, the pixel value at that time is associated with the intersecting position and recorded in a storage unit (not shown) (step S25). Then, for all the processing pixels in the ideal camera image, it is determined whether or not the processing from step S21 to step S25 has been completed (step S26).

[0046] In step S26, if the pixel values for all the processing pixels in the ideal camera image have not been recorded (NO in step S26), the processing in steps S21 to S25 is repeated until the pixel values for all the processing pixels in the ideal camera image are recorded.

[0047] Then, in step S26, when the pixel values for all the processing pixels in the ideal camera image have been recorded (YES in step S26), a calculation image having the stored pixel values is output (step S27). Finally, a distortion correction function is generated from the calculation image generated in step S27 (step S28). Note that the distortion correction function generated in step S28 is a function indicating to which position in the calculation image calculated in step S27 each pixel in the ideal image shown in the ideal camera image has moved, and represents a function for distorting the input image signal obtained from the calculation image. The generation process of the distortion correction function ends with the processing in steps S20 to S28 described above.

[0048] As described above, in step C of FIG. 2 and FIG. 6, a distortion correction function for determining how to distort the image signal from the calculation image is obtained. That is, the distortion correction function is a function for determining how to distort the input image signal, and when this distortion correction function is applied to the image signal, a distortion correction image display light source 104 (see FIG. 1) for displaying the aerial image 40 is obtained. Therefore, it can be said that the distortion correction function is a function indicating the correspondence relationship of the positions to which the pixel values of the input image signal should be moved in order to be converted into the distortion correction image display light source 104.

[0049] As described above, the distortion correction function is obtained from the correspondence between each pixel of the ideal camera image and each pixel of the calculation image. By recording not only the positional relationship of each pixel but also the amount of change in the pixel value of each corresponding pixel as the correspondence, it is possible to correct the color change of the image due to a transparent object or a reflective object using the distortion correction function. Therefore, in this specification, "distortion" includes not only a change (deformation) in shape but also a change in color. By using the distortion correction function of the present invention, it is possible to perform not only correction of the deformation of the aerial image but also color correction of the aerial image.

[0050] Next, with reference to FIG. 7, the procedure for generating the calculation image from the ideal camera image described in the flowchart of FIG. 6 will be described based on the optical path change along the optical system. As shown in FIG. 7, first, consider an optical system in which MMAPs 10 and a transparent object 30 are interposed between the light source position P4 and the viewpoint position P0, and arrange a CG camera 50 at the viewpoint position P0. Therefore, the viewpoint position P0 becomes the CG camera position P0.

[0051] A ray having a pixel value c of a specific point of the ideal camera image exits from the CG camera 50 in the emission direction v0 from the CG camera position P0. This ray enters the origin P1 of the transparent object 30 and is refracted, changing the optical path, and becomes a ray that exits in the emission direction v1. Further, when this ray exits from the point P2 of the transparent object 30, the optical path is changed, and it becomes a ray that exits in the emission direction v2 and enters the point P3 of the MMAPs 10.

[0052] Then, at the point P3 of the MMAPs 10, it exits in the emission direction v3 with the changed optical path, and when it reaches the light source position P4, the pixel value of the ray is recorded. The set of pixel values recorded at the position where it reaches the light source position P4 becomes the calculation image. Note that the pixel value of the ray that reaches the light source position P4 corresponds to the pixel value c of a specific point of the ideal camera image emitted from the CG camera 50. By repeating the above steps for all the pixels of the ideal camera image, a distorted calculation image corresponding to the optical system such as the transparent object 30 and the MMAPs 10 arranged in the middle path is formed at the light source position P4.

[0053] Note that a distortion correction function is generated from the calculation image obtained in FIG. 7, and this distortion correction function is used to generate a light source image for displaying an ideal image without distortion in the space inside or through the transparent object 30. Then, by placing the generated light source image with distortion at the light source position P4, a virtual image without distortion is generated at the position of the virtual image 40 in FIGS. 3(a) and 3(b).

[0054] FIG. 8 is a diagram for explaining step D. In the real space, a light source image with distortion generated by the procedure shown in the flowchart of FIG. 6 and the method described in the schematic diagram of FIG. 7 is placed at the actual light source position, and a specific example of displaying a virtual image 40 without distortion inside the spherical transparent object 30 is shown. As shown in FIG. 8, an image of a goldfish is prepared as a light source image including distortion. This light source image corresponds to the distortion correction image display light source 104 shown in FIG. 1.

[0055] The distortion correction image display light source 104 having the image of the goldfish is bent by 90° at the MMAPs 10 at a distance L from its center, and is imaged at the center position of the transparent object 30 also at a distance L from the MMAPs 10. Note that the goldfish image in the transparent object 30 shown in FIG. 8 is an image close to the actual goldfish image. This is because distortion is added to the image used as the light source using the distortion correction function obtained in FIG. 6 of this embodiment. It has been verified that if there is no distortion correction for the light source image by this distortion correction function, the image will be different from the actual goldfish image.

[0056] <Reduction of noise by MMAPsBRDF> Next, in order to improve the efficiency of light source image generation and reduce the noise of the generated light source image, among the functions of the MMAPs, MMAPsBRDF (Bidirectional Reflection Distribution Function), which has only the functions necessary for imaging a virtual image, will be described.

[0057] As shown in Fig. 9(a), MMAPs10 is composed of a two-layer mirror array and the glass covering it. The incident light is reflected once by each layer, and the light from the light source forms an aerial image at a position that is plane-symmetrical with respect to MMAPs10. However, depending on the incident angle of the light, there is also light that is only reflected by one layer, generating stray light that forms an obstructive image for the observer, or light that is transmitted without being reflected at all.

[0058] In addition to the two-layer mirror array, reflection by the glass also occurs, so not all light forms an aerial image. In fact, when the light from the light source forms an aerial image, the luminance is attenuated to less than half. Therefore, if MMAPs10 is directly used for generating the calculation image, in the tracing of the optical path, there is a probability of more than 50% of tracing the optical paths that generate stray light, transmitted light, and reflected light, resulting in poor efficiency in generating the calculation image.

[0059] Therefore, in the aerial image generation device of this embodiment, as shown in Fig. 9(b), a BRDF (hereinafter referred to as "MMAPsBRDF") having only the function of emitting the incident light in a plane-symmetrical direction is used in the CG space. MMAPsBRDF is a function that mathematically describes the recursive transmission of light rays for simulation in the CG space, and is realized by calculation processing in the CG space, and does not physically exist. Note that the BRDF itself is used as a general term in CG, but the term MMAPsBRDF is a name given by the inventors.

[0060] MMAPsBRDF is, so to speak, a function that represents only the property that the incident light is emitted in a plane-symmetrical direction. That is, even without using a large number of mirror arrays like MMAPs10, if there is a single plane to which MMAPsBRDF is applied, that plane can emit light in a plane-symmetrical direction and form an aerial image 40 in the same way as MMAPs10.

[0061] Hereinafter, the principle and effect of MMAPsBRDF will be described. As shown in Fig. 9(b), the MMAPsBRDF only has the function of emitting the light incident in direction i in the mirror-symmetric direction o. Therefore, when using the MMAPsBRDF, it is necessary to obtain the outgoing vector o from the incident vector i and the surface normal n.

[0062] Here, if the vector h shown in Fig. 9(b) is defined as the vector when a perpendicular line is drawn from the origin of the incident vector i to the surface, Equation (1) holds by simple vector calculation. Here, the vector h is obtained by multiplying the unit vector n by the inner product of the vector i and the unit vector n, and can be expressed by Equation (2). Therefore, the required outgoing vector o becomes as shown in Equation (3) by substituting Equation (2) into Equation (1).

[0063] (Equation (1)) o = -i + 2h (Equation (2)) h = (i·n)n (Equation (3)) o = -i + 2(i·n)n

[0064] Fig. 10 shows the difference between the calculated image (a) generated using the conventional MMAPs10 and the calculated image (b) generated by applying the MMAPsBRDF to a planar object. Note that a CPU calculation rendering program created by the inventors in the C++ language was used to generate the light source image.

[0065] As can be seen from Figs. 10(a) and (b), it can be seen that the calculated image (b) generated using the MMAPsBRDF has significantly less noise than the calculated image (a) generated by the MMAPs10.

[0066] Also, it was found that the calculation time for generating the calculated image (b) using the MMAPsBRDF is significantly reduced compared to the time for generating the calculated image (a) by the MMAPs10. That is, when calculated with the sampling number per pixel being 100 and the resolution of the ideal image being 512×512 pixels, it was 2552.8 seconds when using the MMAPs10, whereas it was 5.1 seconds when using the MMAPsBRDF.

[0067] This is because the MMAPs10 has a structure with two layers of mirror arrays each having 1380 mirrors per layer. As a result, the number of objects in the CG space is large, while the MMAPsBRDF can form an aerial image on a single plane.

[0068] <Implementation Test> To confirm the effects of the aerial image generation device 100 of this embodiment, the following experiments (1) to (4) were conducted. (1) When directly displaying an aerial image at the video display position without placing a transparent object between the aerial image and the MMAPsBRDF (hereinafter referred to as the "desired video"). (2) When the image of the light source of the aerial image is set to the same as the image used for the above "desired video" and the video is displayed inside the transparent object (hereinafter referred to as the "video without compensation"). (3) When the video is displayed inside the transparent object using the optical system of InFloasion described in Non-Patent Document 1 (hereinafter referred to as the "video with compensation (InFloasion)"). (4) When the image of the light source is set to the image of the light source that generates a light source image for correcting distortion and the video is displayed inside the transparent object (hereinafter referred to as the "video with compensation (this embodiment)").

[0069] FIG. 11 is a diagram showing the results of photographing the desired video and the video displayed inside the transparent object and measuring the imaging position of the video. For the "desired video", "video without compensation", and "video with compensation (InFloasion)" as the videos to be displayed, the light source image shown in FIG. 11(a) was used, and for the "video with compensation (this embodiment)", the light source image shown in FIG. 11(b) was used.

[0070] Here, in the light source image shown in FIG. 11(a), one side of the grid point to be displayed was 30 mm, but in the light source image for the "video with compensation (this embodiment)" in FIG. 11(b), in the generation of the calculation image shown in step S27 of the flowchart in FIG. 6, the display surface of the light source image was set to a larger square of 50 mm so as to surely record the pixel values propagated from the light rays.

[0071] Figure 11(c) shows the "desired video", Figure 11(d) shows the "video without compensation", Figure 11(e) shows the "video with compensation (InFloasion)", and Figure 11(f) shows the "video with compensation (this embodiment)". Therefore, in Figures 11(c) to (e), the image presentation part in the light source is 30 mm square, while in Figure 11(f), it is 50 mm square.

[0072] As shown in Figures 11(a) to (f), the video displayed in the transparent object is tilted. However, the experiment was conducted on a horizontal optical bench, and since the light source, MMAPs, and transparent object are not tilted, it is considered that this tilt of the video may be caused by the thickness and deflection of the MMAPs.

[0073] The positions of the videos displayed in the images taken under each of the above conditions (1) to (4) were compared. In this comparison, the differences in the positions of the grid points in the formed video were examined. That is, taking the x - direction with the right - hand direction as the positive direction and the y - direction with the downward (vertical direction) as the positive direction, the positions of 49 grid points of the formed video were recorded. At this time, the taken image is a digital image with 4000 pixels in the vertical direction and 6000 pixels in the horizontal direction. As shown in Figure 11(c), the width of the grid points in the x - direction in the desired video is 1165 pixels.

[0074] After recording the positions of the 49 grid points, for the following three combinations, the differences in the coordinates of the corresponding grid points of each image were obtained. Case A: The "desired video" in (1) and the "video without compensation" in (2) Case B: The "desired video" in (1) and the "video with compensation (InFloasion)" in (3) Case C: The "desired video" in (1) and the "video with compensation (this embodiment)" in (4)

[0075] Figure 12 shows this result. As can be seen from Figure 12, a difference in the number of pixels from the desired video was observed, with an average of 276.1 pixels in Case A, 155.4 pixels in Case B, and 62.8 pixels in Case C. Thus, it can be confirmed that in Case C, which is the method of this embodiment, the compensation for video distortion in the transparent object is significantly improved compared to the other methods shown in Case A and Case B. However, considering that the width in the x-direction of the grid points of the desired video in (1) is 1156 pixels, the width of one grid is about 191 pixels. Therefore, in this embodiment as well, it has to be said that about one-third of one grid is shifted from the desired video.

[0076] <Modification example> As described above, the aerial image generation apparatus and the aerial image generation method of this embodiment have been described by giving an example in which the transparent object 30 is arranged between the viewpoint and the MMAPs 10. However, the present invention is not limited to this embodiment. Figure 13 shows an example in which the arrangement relationship of the optical system is different from the optical system shown in Figure 3. Figure 13(a) shows an optical system in which the light emitted from the light source LS is reflected by the reflective object R and then enters the MMAPs. Figure 13(b) shows an optical system in which the light emitted from the light source LS passes through the transparent object T and then enters the MMAPs. Further, Figure 13(c) shows an optical system in which the light from the light source LS enters the MMAPs, exits the MMAPs, and then is reflected by the reflective object R.

[0077] The optical systems shown in these Figures 13(a) to 13(c) are merely examples that can be considered. The example of this embodiment is not limited to these optical systems, and of course, various application examples and modification examples are included as long as they do not deviate from the description in the claims. An optical system including both a transparent object and a reflective object in the optical path is also assumed, and an optical system including a plurality of transparent objects and a plurality of reflective objects in the optical path is also assumed.

Description of reference numerals

[0078] 100…Aerial image generation device, 101…Function generation unit for correcting distortion of light source, 102…Image processing unit (distortion processing unit), 103…Distortion-corrected image storage unit, 104…Distortion-corrected image display light source, 105…Aerial image optical system, 10…Recursive transmission optical element (MMAPs), 20, LS…Light source, 30, T…Transparent object, 40, MI…Aerial image, 50…CG camera, R…Reflective object

Claims

1. An image processing unit that generates a distortion correction image that distorts an input image signal; A distortion correction image storage unit that stores the distortion correction image generated by the image processing unit; A distortion correction image display light source that uses the distortion correction image stored in the distortion correction image storage unit as a light source; In the optical path between the distortion correction image display light source and the aerial image, it has at least one of a transparent object that transmits light and a reflective object that reflects light, and a recursive transmission optical element, and receives the light emitted from the distortion correction image display light source, An aerial image optical system that generates an aerial image as a real image at a predetermined spatial position; A light source distortion correction function generation unit that generates a distortion correction function based on a calculation image obtained by recording the pixel values of the light rays passing through the transparent object and / or the reflective object installed in the aerial image optical system and the recursive transmission optical element at the actual light source installation position; and The image processing unit generates the distortion correction image so that the distortion of the aerial image provided by the aerial image optical system is corrected. An aerial image generation device.

2. In the CG space corresponding to the aerial image optical system, An ideal camera video obtained by rendering an ideal image arranged at the aerial image display position within the viewing angle of a CG camera corresponding to the viewpoint; After selecting each pixel within the viewing angle of the CG camera and emitting a light ray having the pixel value of the ideal camera video corresponding to the position of the pixel from the CG camera in the direction of the pixel, the image processing unit uses the distortion correction function generated by the light source distortion correction function generation unit to generate the distortion correction image that distorts the input image signal. The aerial image generation device according to claim 1.

3. The spatial position where the aerial image is displayed is a position inside or outside the transparent object. The aerial image generation device according to claim 1 or 2.

4. The recursive transmission optical element used in generating the distortion correction function is a single planar object having an MMAPsBRDF that mathematically describes the recursive transmission of light rays for simulation in the CG space. The aerial image generation device according to any one of claims 1 to 3.

5. In the CG space, Designing an aerial image optical system including the arrangement relationship of a transparent object or a reflective object and a recursive transmission optical element; Outputting an ideal camera video by rendering an ideal image arranged at the aerial image display position within the viewing angle of a CG camera corresponding to the viewpoint; Emitting light rays having the pixel values of each pixel of the ideal camera video from the CG camera, and generating a distortion correction function from a calculated image generated by recording the pixel values of all pixels of the ideal camera video when the light rays reach the light source installation position of the actual machine through the aerial image optical system. A distortion correction function generation method.

6. In the CG space, Designing an aerial image optical system including the arrangement relationship of a transparent object or a reflective object and a recursive transmission optical element; Outputting an ideal camera video by rendering an ideal image arranged at the aerial image display position within the viewing angle of a CG camera corresponding to the viewpoint; Emitting light rays having the pixel values of each pixel of the ideal camera video from the CG camera, and generating a distortion correction function from a calculated image generated by recording the pixel values of all pixels of the ideal camera video when the light rays reach the light source installation position of the actual machine through the aerial image optical system; In the actual machine, Generating a distortion correction image that distorts an input image signal using the distortion correction function, and generating a distortion correction image display light source based on the distortion correction image. Arranging the aerial image optical system designed in the designing step and the distortion correction image display light source in real space, receiving the light emitted from the distortion correction image display light source, and generating an aerial image as a real image at a predetermined spatial position; and An aerial image generation method.

Images