Image processing system, control method for image processing system, and program

Abstract

To generate a CG image with improved optical consistency with the real world.SOLUTION: An image processing system has: first acquisition means for acquiring information on the position and attitude of an imaging apparatus that images the real space that constitutes the user's field of view; second acquisition means for acquiring parameter information corresponding to the position attitude of the imaging apparatus from an environment map in which position attitude information and image processing parameter information are mutually correlated; and generation means for generating an image of an object to be displayed in the real space based on the position attitude information of the imaging apparatus acquired by the first acquisition means and the parameter information acquired by the second acquisition means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method for the image processing apparatus, and a program.

Background Art

[0002] In recent years, research has been conducted on mixed reality (MR) aimed at seamlessly combining the real space and the virtual space. As one of the MR technologies, a technique using a video see-through type head-mounted display (HMD) is known. The video see-through type HMD superimposes an image of computer graphics (CG) generated according to the position and orientation of an imaging device on a real space image captured by an imaging device such as a video camera. A user can observe, through the HMD, an image in which the CG image is superimposed on the real space image.

[0003] In processing such as estimation of the position and orientation of the imaging device and rendering of the CG image, high-load calculations are performed. The video see-through type HMD displays an image after these calculation processes. Therefore, a delay may occur in the display of the CG image. On the other hand, a method has been proposed in which the captured real space image is displayed without waiting for the estimation of the position and orientation and the rendering process of the CG image, and the CG image is separately synthesized after a series of processes are performed. When this method is used, the delay in the display of the real space image is reduced. However, since a delay occurs in the display of the CG image, a delay occurs in the display of the CG image with respect to the real space image, and a deviation occurs in the position and orientation when observing the real space image and the CG image. Patent Document 1 discloses a method of performing reprojection processing on a CG image based on a time stamp to reduce the deviation in the position and orientation of observing the real space image and the CG image.

[0004] In addition, in a video see-through type HMD, if a CG image is directly superimposed on a real space image, the resulting image may look unnatural. This is because when the image characteristics such as tonality, white balance, noise, resolution, and gamma between the real space image and the CG image are different, the composite image of the real space image and the CG image will lack optical consistency.

[0005] The real space image is an image that compresses the dynamic range of the real space into the dynamic range of the imaging device, which is narrower than the real space. For this reason, a subject in front of a brightly lit white wall under sunlight is imaged darker than it appears to the eye. On the other hand, since CG can adjust its brightness to be optimally displayed on a computer display without depending on the surrounding environment, it can be reproduced brightly even in front of a white wall. Therefore, a composite image with a subject in front of a white wall and a CG image side by side will lack optical consistency and look unnatural because of the difference in brightness between the subject and the CG image. Patent Documents 2 and 3 disclose methods for reducing the unnaturalness caused by the inconsistency in optical consistency.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, if there is a shift in the timing of displaying the real space image and the CG image, there will also be a shift in the timing when the parameters of the image processing for adjusting the image characteristics are applied. That is, the image processing parameters are immediately applied to the real-space image, whereas for the CG image, there is a time lag corresponding to the delay time until it is displayed, so it appears to be applied with a delay.

[0008] An object of the present invention is to provide an image processing apparatus capable of generating a CG image with improved optical consistency with the real space.

Means for Solving the Problems

[0009] A first aspect of the present invention includes: a first acquisition means for acquiring information on the position and orientation of an imaging device that images the real space that is the user's field of view; a second acquisition means for acquiring information on the parameters related to the position and orientation and image processing from an environmental map in which the information on the position and orientation and the information on the parameters related to the image processing are associated with each other, and acquiring the information on the parameters corresponding to the position and orientation of the imaging device; and a generation means for generating an image of an object to be displayed in the real space based on the information on the position and orientation of the imaging device acquired by the first acquisition means and the information on the parameters acquired by the second acquisition means. The image processing apparatus is characterized by having these components.

[0010] A second aspect of the present invention is a control method for an image processing apparatus, including: a first acquisition step of acquiring information on the position and orientation of an imaging device that images the real space that is the user's field of view; a second acquisition step of acquiring information on the parameters corresponding to the position and orientation of the imaging device from an environmental map in which the information on the position and orientation and the information on the parameters related to the image processing are associated with each other; and a generation step of generating an image of an object to be displayed in the real space based on the information on the position and orientation of the imaging device acquired in the first acquisition step and the information on the parameters acquired in the second acquisition step. The control method is characterized by having these steps.

[0011] A third aspect of the present invention is a program for causing a computer to function as each means of the above-described image processing apparatus.

Effects of the Invention

[0012] According to the present invention, it is possible to generate a CG image with improved optical consistency with the real space.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Modes for Carrying Out the Invention

[0014] <Embodiment 1> Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram of an image processing apparatus 100 and an HMD 110 (head-mounted display 110) according to Embodiment 1.

[0015] In the following description, an example is shown in which the image processing apparatus 100 is connected to and used with the HMD 110. Note that part of the functions of the image processing apparatus 100 may be executed by the HMD 110, and part of the functions of the HMD 110 may be executed by the image processing apparatus 100. Also, the image processing apparatus 100 may be a configuration included in the HMD 110.

[0016] The imaging unit 111 of the HMD 110 is an imaging device that images the real space that is the user's field of view. The captured image (real space image) of the real space captured by the imaging unit 111 is sent to the image processing unit 112. Note that the imaging unit 111 may include a plurality of imaging devices. The imaging unit 111 may include, for example, a pair of imaging devices that image an image to be displayed as a background image, and a pair of imaging devices that image an image for acquiring the position and orientation of the HMD 110.

[0017] The image processing unit 112 acquires information on parameters related to image processing (hereinafter referred to as image processing parameters) from the captured image. The information on the image processing parameters includes an exposure correction value and a white balance correction value, etc. The exposure correction value includes a gain, a shutter speed, and an aperture value. The white balance correction value includes values such as R / G and B / G. The information on the image processing parameters may also include a noise reduction value that is a parameter of NR (Noise Reduction), a contour enhancement value that is a parameter of edge enhancement processing, and a gamma correction value, etc.

[0018] In the following description, the image processing parameters are, for example, the gain and the shutter speed that are exposure correction values, but they may be other parameters. First, the image processing unit 112 evaluates the brightness of the captured image in order to appropriately correct the exposure. When the captured image is excessively bright, the image processing unit 112 calculates an exposure correction value so that the captured image becomes darker by lowering the gain or increasing the shutter speed. Conversely, when the captured image is dark, the image processing unit 112 calculates an exposure correction value so that the captured image becomes brighter by increasing the gain or lowering the shutter speed.

[0019] The imaging unit 111 captures the real space using the exposure correction value calculated by the image processing unit 112. The image processing unit 112 calculates the exposure correction value again from the image captured by applying the exposure correction value. By repeating the calculation of the exposure correction value and the imaging using the exposure correction value, the image processing unit 112 can appropriately control the exposure of the captured image. The image processing unit 112 transmits the captured image subjected to image processing such as exposure correction to the CG synthesis unit 113.

[0020] The CG synthesis unit 113 generates a composite image of the captured image received from the image processing unit 112 and the CG image generated by the CG generation unit 106. The display unit 114 (display control means) displays the composite image on an image display element such as an OLED (Organic light emitting diode) mounted on the HMD 110. The HMD 110 can reduce the delay in the display of the real space image by quickly completing the process from imaging to display within the device.

[0021] A method for generating the CG image synthesized by the CG synthesis unit 113 with the real space image will be described. The image processing unit 112 outputs the captured image subjected to various image processes to the position and orientation estimation unit 103 of the image processing apparatus 100. The communication between the image processing apparatus 100 and the HMD 110 may be wired or wireless.

[0022] The position and orientation estimation unit 103 estimates the position and orientation of the HMD 110 from the real space image received from the image processing unit 112. The information on the position and orientation can be estimated, for example, by a method using SLAM (Simultaneous Localization and Mapping). In SLAM, an environmental map used for estimating the position and orientation is generated in advance.

[0023] The environment map generation unit 101 generates an environment map using the real space image captured by the imaging unit 111. The environment map is information that associates the captured image, the position and orientation information of the imaging unit 111 when the captured image was taken, and the parameter information regarding the image processing of the captured image with each other. The environment map enables efficient estimation of the position and orientation of the imaging unit 111.

[0024] Specifically, the environment map is constructed by generating key frames including the following information. The key frame includes the captured image and the following information corresponding to the captured image. · ID of the key frame · Position and orientation of the imaging device (imaging unit 111) associated with the key frame · Captured image associated with the key frame · Image coordinates (x, y) of the feature points extracted from the captured image of the key frame · Number of feature points extracted from the captured image of the key frame · Variance value of the feature points on the image of the captured image of the key frame (for example, a value obtained by dividing the number of regions with feature points in a block by the total number of blocks after dividing the captured image of the key frame into 10×10 image blocks and multiplying the result by 100) · Reliability of the key frame (an index representing the possibility of alignment error obtained from the number of feature points and the variance value) · Feature point ID uniquely assigned by associating the extracted feature points between the captured images of multiple key frames · 3D coordinates (X, Y, Z) of the feature points obtained by triangulation based on the position and orientation of the imaging device of each key frame for the feature points having the same associated feature point ID · ID of the neighboring key frames

[0025] The environmental map includes a plurality of generated key frames and is stored in the environmental map storage unit 102. The position and orientation estimation unit 103 estimates the position and orientation of the imaging unit 111 using the environmental map stored in the environmental map storage unit 102. The estimated position and orientation of the imaging unit 111 can be the position and orientation of the HMD 110. Note that the imaging device used for generating the environmental map may be a device different from the imaging device that captures a real-space image to be synthesized with the CG image.

[0026] The CG generation unit 106 generates (renders) a CG image based on the position and orientation of the HMD 110 estimated by the position and orientation estimation unit 103. The real-space image used when synthesizing the CG image is an image captured with a delay of the rendering processing time compared to the captured image used for estimating the position and orientation of the HMD 110. The position and orientation of the HMD 110 can change while the CG image is being rendered. For this reason, a deviation in the display position or orientation due to a change in the position and orientation of the HMD 110 occurs between the CG image and the real-space image with which the CG image is synthesized.

[0027] FIG. 2 is a diagram for explaining how a deviation in the display position of the CG image occurs. The real-space images 201 to 205 show time-series captured images when the position and orientation of the HMD 110 are changed while capturing a stationary ball.

[0028] The CG images 210 to 212 show images obtained by rendering a triangular object (polygon) so as to be in contact with the ball. The CG images 210 to 212 are images obtained by rendering the object using the position and orientation of the HMD 110 estimated from the real-space images 201 to 203, respectively.

[0029] The composite images 221 to 225 show images obtained by synthesizing the CG images 210 to 212 with the real-space images 201 to 205. In the example of FIG. 2, a delay of approximately two frames occurs from the estimation of the position and orientation until the rendering of the CG image is completed. For this reason, the CG image 210 rendered using the position and orientation estimated from the real-space image 201 is synthesized with the real-space image 203.

[0030] In the composite image 223 in which the CG image 210 and the real-space image 203 are combined, the triangle that was supposed to be displayed in contact with the ball is displayed at a position shifted from the ball. Similarly, in the composite images 224 and 225 as well, the triangle that was supposed to be displayed in contact with the ball is displayed at a position shifted from the ball. Thus, the shift in the display position continuously occurs.

[0031] For the same reason as the shift in the display position, even when the optical characteristics such as the brightness of the real-space image change, a shift in brightness or the like occurs between the CG image and the real-space image. Therefore, the optical consistency between the CG image and the real-space image deteriorates.

[0032] FIG. 3 is a diagram for explaining how a brightness shift of the CG image occurs. The real-space images 301 to 305 show time-series captured images when the brightness of the surroundings is gradually changed while imaging a stationary ball. Also in the example of FIG. 3, the position and orientation of the HMD 110 change in the same manner as in FIG. 2.

[0033] The CG images 310 to 312 show images in which a triangular object (polygon) is rendered reflecting the change in the brightness of the surroundings. The CG images 310 to 312 are images in which the object is rendered using the exposure correction values of the real-space images 301 to 303, respectively.

[0034] The composite images 321 to 325 show images in which the CG images 310 to 312 are combined with the real-space images 301 to 305. In the example of FIG. 3, a delay of approximately two frames occurs from the estimation of the position and orientation until the rendering of the CG image is completed. Therefore, the CG image 310 rendered using the exposure correction value of the real-space image 301 is combined with the real-space image 303.

[0035] In the composite image 323 where the CG image 310 and the real-space image 303 are combined, the image of the superimposed triangular object is displayed brighter than the surroundings. That is, the optical consistency between the CG image and the real-space image is impaired. Similarly, in the composite images 324 and 325 as well, the brightness of the surroundings and the object are displayed in a deviated state. Thus, the deviation in brightness continues to occur, and the state where the optical consistency is impaired continues.

[0036] As described with reference to FIGS. 2 and 3, a deviation occurs between the position and orientation of the HMD 110 when the generation of the CG image is started and the position and orientation of the HMD 110 when the CG image is displayed. Due to the delay in the display timing of the CG image, a deviation in the position and orientation of the HMD 110 until the CG image is displayed causes a deviation in the display position or brightness or the like between the real-space image and the CG image. A method for improving the deviation in the display position or brightness or the like between the real-space image and the CG image will be described.

[0037] First, a method for reducing the deviation in the display position of the CG image described with reference to FIG. 2 will be described. The position and orientation prediction unit 104 in FIG. 1 predicts the position and orientation of the HMD 110 when the CG image is displayed based on the information on the current position and orientation of the HMD 110 and the information on the acceleration acquired by the acceleration acquisition unit 115 of the HMD 110. The information on the current position and orientation of the HMD 110 used for predicting the position and orientation may be the position and orientation estimated by the position and orientation estimation unit 103 or the position and orientation acquired by the acceleration acquisition unit 115.

[0038] The acceleration acquisition unit 115 includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like. Acceleration is a value indicating changes in three rotational motions (so-called postures) around the X-axis, Y-axis, and Z-axis of the HMD 110, and is also called 3DoF (Degree of Freedom). The acceleration acquisition unit 115 may be a sensor capable of acquiring 6DoF instead of 3DoF. 6DoF is a value corresponding to six motions obtained by adding the translational motions of the HMD 110 along the X-axis, Y-axis, and Z-axis to the 3DoF motions (so-called position and orientation).

[0039] Since the acceleration acquisition unit 115 can acquire the position and orientation of the HMD 110 at a high frame rate in a shorter time than the position and orientation estimation unit 103, it can acquire the most recent position and orientation compared to the position and orientation estimated by the position and orientation estimation unit 103. Therefore, the acceleration acquisition unit 115 can acquire the position and orientation of the HMD 110 immediately before rendering. Since the acceleration acquisition unit 115 can acquire the history of changes in the position and orientation up to the present at a high frame rate, the position and orientation prediction unit 104 can predict the position and orientation several frames ahead.

[0040] The CG generation unit 106 generates a CG image based on the position and orientation of the HMD 110 predicted by the position and orientation prediction unit 104. Thereby, the deviation in the display position between the real-space image and the CG image when the CG image is displayed is improved. The position and orientation prediction unit 104 can suppress the deviation in the display position of the CG image with respect to the real-space image to an inconspicuous level by improving the prediction accuracy.

[0041] Next, a method for reducing the brightness deviation of the CG image described in FIG. 3 and improving the optical consistency will be described. The parameter acquisition unit 105 acquires information on image processing parameters corresponding to the position and orientation of the HMD 110. The parameter acquisition unit 105 can acquire this from an environment map in which information on image processing parameters is associated with each key frame.

[0042] When generating the environment map, the environment map generation unit 101 associates information on the image processing parameters at the time when the imaging image of the key frame is captured with each key frame, and stores it in the environment map storage unit 102. The information on the image processing parameters associated with each key frame includes exposure correction values such as gain and shutter speed, for example. The environment map generation unit 101 can generate an environment map in which the information on the position and orientation of the imaging device (HMD 110) and the information on the image processing parameters are associated with each other using the key frames.

[0043] The parameter acquisition unit 105 can acquire the exposure correction value corresponding to the position and orientation of the HMD 110 predicted by the position and orientation prediction unit 104 from the environment map. The parameter acquisition unit 105 obtains the key frame closest to the predicted position and orientation of the HMD 110, and acquires the exposure correction value of the obtained key frame as the predicted value. The closest key frame can be a key frame with a smaller difference in position and orientation from the predicted position and orientation of the HMD 110.

[0044] The parameter acquisition unit 105 may obtain a plurality of key frames in the vicinity of the predicted position and orientation of the HMD 110, perform weighting according to the difference from the position and orientation of each key frame, and perform interpolation calculation of the exposure correction value. The parameter acquisition unit 105 can obtain a more accurate value by predicting the exposure correction value based on a plurality of key frames.

[0045] The environment map generation unit 101 can generate an environment map including information on various image processing parameters, not limited to the exposure correction value. Thereby, the parameter acquisition unit 105 can acquire information on various image processing parameters corresponding to the position and orientation of the HMD 110. The information on the image processing parameters associated with each key frame of the environment map includes, for example, at least any one of the exposure correction value, white balance correction value, gamma correction value, noise reduction value, and edge enhancement value. The information on the image processing parameters may be any one of the exposure correction value, white balance correction value, gamma correction value, noise reduction value, and edge enhancement value, or may be information obtained by arbitrarily combining these.

[0046] The CG synthesis unit 113 generates a CG image based on the position and orientation information of the HMD 110 acquired (predicted) by the position and orientation prediction unit 104 and the image processing parameter information acquired by the parameter acquisition unit 105. Thereby, the optical consistency between the real space image and the CG image is improved. The accuracy of the image processing parameters acquired by the parameter acquisition unit 105 depends on the prediction accuracy of the position and orientation prediction unit 104. Therefore, if the prediction accuracy of the position and orientation prediction unit 104 increases, the accuracy of the image processing parameters acquired by the parameter acquisition unit 105 will improve.

[0047] When allowing a delay in the display timing of the CG image, the parameter acquisition unit 105 may acquire the image processing parameter information based on the position and orientation information used to generate the CG image instead of the position and orientation information predicted by the position and orientation prediction unit 104. Even in this case, the optical consistency between the real space image and the CG image is improved as compared with the case of generating the CG image without adjusting the image processing parameters.

[0048] FIG. 4 is a flowchart illustrating the CG image display process. In step S401, the imaging unit 111 images the real space that is the user's field of view. In step S402, the image processing unit 112 obtains an exposure correction value and performs exposure correction. In step S403, the position and orientation estimation unit 103 estimates the position and orientation of the HMD 110 from the captured image that has been subjected to image processing such as exposure correction in step S402 using the environmental map stored in the environmental map storage unit 102.

[0049] In step S404, the position and orientation prediction unit 104 predicts the position and orientation of the HMD 110 after the delay time using the 3DoF or 6DoF information acquired by the acceleration acquisition unit 115. The delay time may be a predetermined time, or may be a time corresponding to a predetermined number of frames. Further, the delay time may be set according to the size of the CG to be displayed and the number of CGs, etc.

[0050] In step S405, the parameter acquisition unit 105 acquires an exposure correction value corresponding to the position and orientation of the HMD 110 predicted in step S404 from the environmental map. In step S406, the CG generation unit 106 generates (renders) a CG image using the information on the position and orientation of the HMD 110 predicted in step S404 and the exposure correction value acquired in step S405.

[0051] In step S407, the CG composition unit 113 composes the real space image and the CG image generated in step S406. The display unit 114 displays the composite image composed by the CG composition unit 113 on the display. The real space image to be composed with the CG image (the captured image of the real space when the CG image is displayed) is the real space image captured immediately before the composition.

[0052] In step S408, the imaging unit 111 determines whether to stop imaging. When the imaging unit 111 receives an instruction to stop imaging from the user, the process shown in FIG. 4 ends. When the imaging unit 111 does not receive an instruction to stop imaging from the user, it returns to step S401 and continues the display process of the CG image.

[0053] FIG. 5 is a flowchart illustrating the process of acquiring an exposure correction value in step S405 of FIG. 4. In step S501, the parameter acquisition unit 105 acquires information on the position and orientation of the HMD 110 after the delay time predicted by the position and orientation prediction unit 104. In step S502, the parameter acquisition unit 105 refers to the environmental map stored in the environmental map storage unit 102, which includes information on image processing parameters such as exposure correction values. In step S503 , the parameter acquisition unit 105 acquires an exposure correction value corresponding to the position and orientation of the HMD 110 after the delay time acquired in step S501 using the environmental map referred to in step S502. In step S504, the parameter acquisition unit 105 outputs the exposure correction value acquired in step S503 to the CG generation unit 106.

[0054] In the above-described Embodiment 1, the image processing apparatus 100 acquires image processing parameters corresponding to the position and orientation of the HMD 110 from an environmental map in which the information on the position and orientation of the HMD 110 and the information on the image processing parameters are associated with each other, and generates a CG image. Therefore, the image processing apparatus 100 can generate a composite image with improved optical consistency between the real space image and the CG image.

[0055] Further, the image processing apparatus 100 may predict the position and orientation of the HMD 110 when the CG image is displayed, and generate a CG image based on the predicted position and orientation. In this case, the image processing apparatus 100 can reduce the deviation in the display position and the deviation in optical characteristics such as brightness between the CG image and the real space image when the CG image is displayed.

[0056] Note that the HMD 110 according to Embodiment 1 has been described as a video see-through type HMD including a CG synthesis unit 113, but it may be an optical see-through type HMD. FIG. 6 is a block diagram of an image processing apparatus and an optical see-through type HMD to which Embodiment 1 is applied. The optical see-through type HMD 610 does not include a CG synthesis unit 113. Since the other configurations of the HMD 610 and the configuration of the image processing apparatus 100 are the same as those in FIG. 1, the description thereof is omitted. The optical see-through type HMD 610 displays the CG image generated by the CG generation unit 106 of the image processing apparatus 100 on the real space viewed by the user through the display.

[0057] <Embodiment 2> Embodiment 1 is an example in which information on image processing parameters such as an exposure correction value is used to acquire information on image processing parameters corresponding to the position and orientation of the HMD 110 after a delay time. Embodiment 1 is premised on the fact that the brightness of the real space does not change between when the environmental map is constructed and when the user actually wears the HMD 110 and superimposes and displays the CG image.

[0058] Embodiment 2 is an example assuming that the brightness of the real space changes when the environmental map is constructed and when the user actually wears the HMD and superimposes and displays the CG image. FIG. 7 is a block diagram of the image processing apparatus 700 and the HMD 710 according to Embodiment 2. The image processing apparatus 700 has a parameter evaluation unit 107 in addition to the configuration of the image processing apparatus 100 in FIG. 1. For the same configuration as that in Embodiment 1, the same reference numerals are given and detailed description thereof is omitted.

[0059] When the brightness of the real space changes after the environmental map generation unit 101 generates the environmental map, the image processing parameters acquired by the parameter acquisition unit 105 from the environmental map may not be suitable for the brightness of the real space when displaying the CG image. Therefore, the parameter evaluation unit 107 acquires and compares the information of the image processing parameters acquired from the environmental map with the information of the image processing parameters determined by the image processing unit 112 for the real space when the CG image is displayed. Note that the real space when the CG image is displayed may be the real space immediately before the CG image is synthesized.

[0060] When the information of the image processing parameter is the exposure correction value, the parameter evaluation unit 107 compares the exposure correction value acquired from the environmental map with the exposure correction value determined by the image processing unit 112 for the real space when the CG image is displayed. When the two acquired exposure correction values are different, the parameter evaluation unit 107 determines that the brightness of the real space has changed. When the two acquired exposure correction values are different, the parameter evaluation unit 107 updates the exposure correction value of the environmental map stored in the environmental map storage unit 102 based on the exposure correction value determined by the image processing unit 112.

[0061] Among the key frames of the environmental map, the key frame to be updated for the exposure correction value is the key frame in which the parameter evaluation unit 107 acquires the exposure correction value. Note that the parameter evaluation unit 107 may include the key frame to be updated and the key frame in which the difference in the position and orientation is equal to or less than a predetermined threshold value in the update target of the exposure correction value.

[0062] Also, when the exposure correction value obtained from the environmental map is different from the exposure correction value determined by the image processing unit 112 with respect to the real space when the CG image is displayed, the CG generation unit 106 generates a CG image using the exposure correction value determined by the image processing unit 112. That is, the CG generation unit 106 generates a CG image based on the position and orientation of the HMD 110 predicted by the position and orientation prediction unit 104 and the exposure correction value determined by the image processing unit 112.

[0063] In the above-described Embodiment 2, when the information on the image processing parameters obtained from the environmental map is different from the information on the image processing parameters determined by the imaging device (image processing unit 112), the image processing apparatus 700 updates the information on the image processing parameters of the environmental map. Therefore, even when the brightness of the real space changes after the environmental map is generated, the image processing apparatus 700 can update the environmental map to obtain an exposure correction value corresponding to the brightness of the real space after the change.

[0064] <Embodiment 3> Embodiment 1 is an example in which the position and orientation of the HMD 110 when the CG image is displayed are predicted in advance before the rendering of the CG image. In contrast, Embodiment 3 is an example in which the information on the position and orientation of the HMD 110 is corrected based on the information on the acceleration of the HMD 110 when the CG image is displayed after the rendering of the CG image. Note that the information on the acceleration when the CG image is displayed may be the information on the acceleration immediately before the CG image is synthesized with the real space image.

[0065] FIG. 8 is a block diagram of the image processing apparatus 800 and the HMD 110 according to Embodiment 3. The HMD 110 has the same configuration as in Embodiment 1. The image processing apparatus 800 has a position and orientation correction unit 108 and a CG correction unit 109 instead of the position and orientation prediction unit 104 in the configuration of the image processing apparatus 100 in FIG. 1. The same components as those in Embodiment 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0066] When the CG image is displayed (immediately before the CG image is synthesized with the real - space image), the image processing apparatus 800 corrects the CG image based on the acceleration information acquired by the acceleration acquisition unit 115. This method is generally called post - processing. By using post - processing, the image processing apparatus 800 can generate a synthesized image without deviation between the real - space image and the CG image without predicting the position and orientation of the HMD110 (imaging unit 111). Embodiment 3 is applied to the image processing apparatus 800 that uses such post - processing.

[0067] The real - space image captured by the imaging unit 111 is subjected to exposure correction by the image processing unit 112. The real - space image whose exposure has been corrected by the image processing unit 112 is input to the position and orientation estimation unit 103. The parameter acquisition unit 105 acquires an exposure correction value corresponding to the position and orientation estimated by the position and orientation estimation unit 103 from the environmental map. Note that when using post - processing, the image processing apparatus 800 may not include the parameter acquisition unit 105. In this case, the image processing apparatus 800 can acquire the exposure correction value determined by the image processing unit 112.

[0068] The CG generation unit 106 generates (renders) a CG image using the position and orientation of the HMD110 estimated by the position and orientation estimation unit 103 and the exposure correction value acquired by the parameter acquisition unit 105. However, if the rendered CG image is directly synthesized with the real - space image by the CG synthesis unit 113, a deviation in the display position will occur due to a delay in the display timing.

[0069] Therefore, the position and orientation correction unit 108 corrects the position and orientation of the HMD110 estimated by the position and orientation estimation unit 103 based on the acceleration information acquired by the acceleration acquisition unit 115. Further, the CG correction unit 109 also corrects the brightness of the CG image. The CG correction unit 109 acquires an exposure correction value corresponding to the position and orientation of the HMD110 corrected by the position and orientation correction unit 108 from the environmental map. The CG synthesis unit 113 synthesizes the CG image corrected by the CG correction unit 109 with the captured image of the real - space immediately before synthesis.

[0070] In the above-described Embodiment 3, the position and orientation correction unit 108 corrects the position and orientation information of the HMD 110 based on the acceleration information of the HMD 110. Further, the CG correction unit 109 acquires information on image processing parameters corresponding to the corrected position and orientation of the HMD 110 from the environmental map, and corrects the CG image based on the acquired image processing parameters. The image processing apparatus 800 includes the position and orientation correction unit 108 and the CG correction unit 109, and can improve the optical consistency of the composite image of the CG image and the real space image by correcting the CG image using post-processing.

[0071] <Embodiment 4> Embodiment 4 is an example in which the HMD includes two types of imaging units: a dedicated CV (Computer Vision) imaging unit for calculating the position and orientation, and a background imaging unit for imaging the real space.

[0072] FIG. 9 is a block diagram of the HMD 910 according to Embodiment 4. The HMD 910 includes a part of the functions of the image processing apparatus 100 according to Embodiment 1. Embodiment 4 is applied to the HMD 910 as an image processing apparatus. For the same configurations as those of the image processing apparatus 100 and the HMD 110 according to Embodiment 1, the same reference numerals are given and detailed descriptions are omitted.

[0073] The CV imaging unit 901 is a low-resolution and high-frame-rate imaging device, and can calculate the position and orientation more frequently. The CV imaging unit 901 captures images at a frame rate of, for example, 90 FPS.

[0074] The background imaging unit 921 is a high-resolution and low-frame-rate imaging device, and improves the apparent resolution of the real space image. The background imaging unit 921 captures images at a frame rate of, for example, 30 FPS.

[0075] In Embodiment 4, the CG image is corrected by post - processing in the same manner as in Embodiment 3. The HMD 910 has a position and orientation correction unit 904 and a CG correction unit 905. Similar to Embodiment 3, the position and orientation correction unit 904 corrects the position and orientation of the HMD 910 estimated by the position and orientation estimation unit 103 based on the acceleration information acquired by the acceleration acquisition unit 115. Also, the CG correction unit 905 corrects the brightness of the CG image.

[0076] FIG. 10 is a graph showing the changes in the brightness of the real space, the background image, and the CG image. The vertical axis represents the brightness level. The horizontal axis represents the frame number. Note that the brightness of the real space is the brightness of the captured image of the real space captured by the imaging unit 901 for CV. The brightness of the background image is the brightness of the captured image of the real space captured by the imaging unit 921 for the background. FIG. 10 shows a case where the real space becomes darker as the frame number advances over time.

[0077] If the brightness of the real space is applied to the CG image without considering the delay time required for generating the CG image with respect to the change in the brightness of the real space captured by the imaging unit 901 for CV, the brightness of the CG image changes with a delay of the delay time. In the example of FIG. 10, the delay time is the time corresponding to 3 frames. On the other hand, since the frame rate of the imaging unit 921 for the background is 30 FPS, which is 1 / 3 of 90 FPS of the imaging unit 901 for CV, the brightness of the background image captured by the imaging unit 921 for the background does not change during 3 frames. That is, the brightness of the background image changes following the ambient brightness every 3 frames. Since the brightness of the background image (real - space image) and the CG image are different, the optical consistency of the composite image is impaired.

[0078]

[0079] ​Therefore, the position and orientation correction unit 904 and the CG correction unit 905 correct the CG image according to the brightness of the real space when the CG image is displayed by post - processing. FIG. 11 is a graph showing the changes in the brightness of the real space, the background image, and the CG image when the CG image is corrected. In FIG. 11, the graph of the brightness of the CG image overlaps with the graph of the brightness of the real space. By correcting the CG image according to the captured image of the real space captured by the imaging unit 901 for CV, the brightness of the CG image changes following the brightness of the real space.

[0080] However, since the brightness of the background image (real - space image) captured by the background imaging unit 921 changes following the ambient brightness every 3 frames, there are frames in which it is different from the brightness of the CG image. Even in the situation of FIG. 11, the optical consistency of the composite image of the real - space image and the CG image is impaired.

[0081] Therefore, the image processing unit 922 corrects the 30 - FPS real - space image using projective transformation or the like and performs frame interpolation so that it becomes 90 FPS. The image processing unit 922 can perform frame interpolation based on the position and orientation corrected by the position and orientation correction unit 904.

[0082] The image processing unit 922 further acquires an exposure correction value corresponding to the position and orientation of the interpolated frame from the environment map and corrects the brightness of the frame. FIG. 12 is a graph showing the change in the brightness of the background image after frame interpolation. In FIG. 12, the graph of the brightness of the background image overlaps with the graphs of the brightness of the real space and the CG image. That is, the brightness of the background image (real - space image) follows the change in the brightness of the CG image. The CG synthesis unit 113 synthesizes the real - space image frame - interpolated by the image processing unit 922 and the CG image corrected by the CG correction unit 905 to generate a composite image.

[0083] In the above-described Embodiment 4, when the frame rate of the captured image by the background imaging unit 921 is different from the frame rate of the captured image by the imaging unit 901 for CV, the HMD 910 corrects the background image (real space image) captured by the background imaging unit 921. The HMD 910 corrects the real space image to be combined with the CG image by using the information on the position and orientation of the HMD 910 corrected by the position and orientation correction unit 904 and the image processing parameters acquired by the image processing unit 922 from the environmental map.

[0084] Specifically, when the frame rate of the captured image by the background imaging unit 921 is smaller than the frame rate of the captured image by the imaging unit 901 for CV, the HMD 910 performs frame interpolation on the real space image to be combined with the CG image. The HMD 910 interpolates the frames of the real space image so that the frame rate of the real space image becomes the frame rate of the captured image by the background imaging unit 921 by using the information on the corrected position and orientation of the HMD 910 and the image processing parameters corresponding to the position and orientation.

[0085] Therefore, even when the frame rate of the real space image to be combined with the CG image is different from the frame rate of the captured image by the imaging unit 901 for CV, the HMD 910 can improve the optical consistency between the real space image and the CG image.

[0086] <Embodiment 5> Embodiment 5 is an example in which the information on the image processing parameters includes the information on the light source in the real space (light source information). Embodiment 5 is applicable to each of the HMDs and image processing apparatuses of Embodiments 1 to 4. In the following description, the case of applying to the HMD 110 and the image processing apparatus 100 will be described, but Embodiment 5 is also applicable to the HMDs and image processing apparatuses of Embodiments 2 to 4.

[0087] As a method for achieving optical consistency between a real-space image and a CG image, in addition to a method of adjusting the brightness of the CG image to match the brightness of the real-space image, a method of rendering shadows and reflections on the CG based on light source information is known. The light source information includes, for example, information such as the position and intensity of the light source. By rendering shadows and reflections according to the light source information, the reality of the CG image is improved.

[0088] As a method for simply realizing a realistic reflection, IBL (Image Based Lighting) etc. are known. In IBL, a 360-degree real-space image around is acquired in advance, and from the position and orientation of the HMD110, which part of the 360-degree image is used as the reflection image is determined. For this reason, as described in Embodiment 1, when there is a delay in the display of the CG image with respect to the real-space image, a delay also occurs in the reflection image applied by IBL.

[0089] Therefore, the environment map generation unit 101 associates the light source information as image processing parameter information with the key frame and stores it in the environment map storage unit 102. By generating an environment map including the light source information, the parameter acquisition unit 105 or the CG correction unit 905 can acquire the light source information corresponding to the real-space image to be synthesized with the CG image from the environment map.

[0090] According to the above Embodiment 5, the HMD110 can improve the optical consistency between the real-space image and the CG image by using the light source information included in the environment map. In addition, the position and orientation prediction unit 104 of the image processing apparatus 100 predicts the position and orientation of the HMD110 when the CG image is displayed, thereby improving the delay in the application of the light source information to the CG image.

[0091] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof. Also, these embodiments can be appropriately combined within the scope of the gist of the present invention.

[0092] Note that each functional unit of the image processing apparatus and the HMD according to Embodiments 1 to 3 (FIGS. 1, 6, 7, and 8), and each functional unit of the HMD according to Embodiment 4 (FIG. 9) may be individual hardware or not. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Also, each functional unit may be realized by hardware or not. For example, the apparatus may have a processor and a memory storing a control program. And the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0093] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and having one or more processors in the computer of the system or apparatus read and execute the program. Also it can be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0094] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) First acquisition means for acquiring information on the position and orientation of an imaging device that images a real space that is the user's field of view, Second acquisition means for acquiring information on the parameter corresponding to the position and orientation of the imaging device from an environmental map in which the information on the position and orientation and the information on the parameter related to image processing are associated with each other, Generation means for generating an image of an object to be displayed in the real space based on the information on the position and orientation of the imaging device acquired by the first acquisition means and the information on the parameter acquired by the second acquisition means An image processing apparatus characterized by comprising. (Configuration 2) The information on the position and orientation in the environmental map and the information on the parameters related to image processing are further associated with the captured image. The first acquisition means estimates the position and orientation of the imaging device from the captured image of the real space using the environmental map. The image processing apparatus according to Configuration 1, characterized by the above. (Configuration 3) Based on the information on the position and orientation of the imaging device and the information on the acceleration of the imaging device, the first acquisition means predicts the position and orientation of the imaging device when the image of the object is displayed. The second acquisition means acquires the information on the parameters corresponding to the position and orientation of the imaging device predicted by the first acquisition means from the environmental map. The image processing apparatus according to Configuration 1 or 2, characterized by the above. (Configuration 4) When the information on the parameters acquired by the second acquisition means from the environmental map is different from the information on the parameters determined by the imaging device with respect to the real space when the image of the object is displayed, there is further an update means for updating the information on the parameters of the environmental map based on the information on the parameters determined by the imaging device. The image processing apparatus according to Configuration 3, characterized by the above. (Configuration 5) When the information on the parameters acquired by the second acquisition means from the environmental map is different from the information on the parameters determined by the imaging device with respect to the real space when the image of the object is displayed, the generation means generates the image of the object based on the position and orientation of the imaging device predicted by the first acquisition means and the information on the parameters determined by the imaging device. The image processing apparatus according to Configuration 3 or 4, characterized by the above. (Configuration 6) Based on the information on the acceleration of the imaging device when the image of the object is displayed, the first acquisition means corrects the information on the position and orientation of the imaging device. The second acquisition means acquires information on the parameters corresponding to the position and orientation of the imaging device corrected by the first acquisition means from the environment map. The image processing apparatus according to Configuration 1 or 2, characterized by the above. (Configuration 7) It further has display control means for controlling to display a composite image obtained by synthesizing the captured image of the real space and the image of the object. When the first frame rate of the captured image of the real space to be synthesized with the image of the object is different from the second frame rate of the captured image of the real space used for estimating the position and orientation of the imaging device using the environment map, the position and orientation information of the imaging device corrected by the first acquisition means and the parameter information acquired by the second acquisition means are used to correct the captured image of the real space to be synthesized with the image of the object. The image processing apparatus according to Configuration 6, characterized by the above. (Configuration 8) (Configuration 8) When the first frame rate is smaller than the second frame rate, the display control means interpolates frames so that the frame rate of the captured image of the real space to be synthesized with the image of the object becomes the second frame rate using the position and orientation information of the imaging device corrected by the first acquisition means and the parameter information acquired by the second acquisition means. The image processing apparatus according to Configuration 7, characterized by the above. (Configuration 9) It further has display control means for controlling to display a composite image obtained by synthesizing the captured image of the real space and the image of the object. The image processing apparatus according to any one of Configurations 1 to 6, characterized by the above. (Configuration 10) It further has display control means for controlling to display the image of the object with respect to the real space. The image processing apparatus according to any one of Configurations 1 to 6, characterized by the above. (Configuration 11) The information of the parameters includes at least any one of an exposure correction value, a white balance correction value, a gamma correction value, a noise reduction value, and an edge enhancement value. The image processing apparatus according to any one of Configurations 1 to 10, characterized in that. (Configuration 12) The information of the parameters includes the information of the light source in the real space. The image processing apparatus according to any one of Configurations 1 to 11, characterized in that. (Method) A first acquisition step of acquiring information on the position and orientation of an imaging device that images the real space that is the user's field of view; A second acquisition step of acquiring the information of the parameters corresponding to the position and orientation of the imaging device from an environment map in which the information of the position and orientation and the information of the parameters related to image processing are associated with each other; A generation step of generating an image of an object to be displayed in the real space based on the information of the position and orientation of the imaging device acquired in the first acquisition step and the information of the parameters acquired in the second acquisition step A control method for an image processing apparatus, characterized by having. (Program) A program for causing a computer to function as each means of the image processing apparatus according to any one of Configurations 1 to 12.

Explanation of Signs

[0095] 100: Image processing apparatus, 101: Environment map generation unit, 102: Environment map storage unit, 103: Position and orientation estimation unit, 105: Parameter acquisition unit, 106: CG generation unit

Claims

1. first acquisition means for acquiring information on the position and orientation of an imaging device that images the real space that is within the user's field of view; second acquisition means for acquiring information on the parameters corresponding to the position and orientation of the imaging device from an environmental map in which information on the position and orientation and information on the parameters related to image processing are associated with each other; generation means for generating an image of an object to be displayed in the real space based on the information on the position and orientation of the imaging device acquired by the first acquisition means and the information on the parameters acquired by the second acquisition means An image processing apparatus, characterized by comprising the above.

2. In the environmental map, the information on the position and orientation and the information on the parameters related to image processing are further associated with the captured image, The first acquisition means estimates the position and orientation of the imaging device from the captured image of the real space using the environmental map The image processing apparatus according to claim 1, characterized by the above.

3. The first acquisition means predicts the position and orientation of the imaging device when the image of the object is displayed based on the information on the position and orientation of the imaging device and the information on the acceleration of the imaging device, The second acquisition means acquires information on the parameters corresponding to the position and orientation of the imaging device predicted by the first acquisition means from the environmental map The image processing apparatus according to claim 1, characterized by the above.

4. When the information on the parameters acquired by the second acquisition means from the environmental map is different from the information on the parameters determined by the imaging device with respect to the real space when the image of the object is displayed, the image processing apparatus further includes update means for updating the information on the parameters in the environmental map based on the information on the parameters determined by the imaging device The image processing apparatus according to claim 3, characterized by the above.

5. When the information of the parameters acquired by the second acquisition means from the environmental map is different from the information of the parameters determined by the imaging device with respect to the real space when the image of the object is displayed, the generation means generates an image of the object based on the position and orientation of the imaging device predicted by the first acquisition means and the information of the parameters determined by the imaging device. The image processing apparatus according to claim 3, characterized in that.

6. The first acquisition means corrects the information on the position and orientation of the imaging device based on the information on the acceleration of the imaging device when the image of the object is displayed. The second acquisition means acquires information on the parameters corresponding to the position and orientation of the imaging device corrected by the first acquisition means from the environmental map. The image processing apparatus according to claim 1, characterized in that.

7. The image processing apparatus further includes display control means for controlling to display a composite image obtained by synthesizing the captured image of the real space and the image of the object. When the first frame rate of the captured image of the real space to be synthesized with the image of the object is different from the second frame rate of the captured image of the real space used to estimate the position and orientation of the imaging device using the environmental map, the display control means uses the information on the position and orientation of the imaging device corrected by the first acquisition means and the information on the parameters acquired by the second acquisition means. To correct the captured image of the real space to be synthesized with the image of the object. The image processing apparatus according to claim 6, characterized in that.

8. When the first frame rate is smaller than the second frame rate, the display control means uses the information on the position and orientation of the imaging device corrected by the first acquisition means and the information on the parameters acquired by the second acquisition means to interpolate frames so that the frame rate of the captured image of the real space to be synthesized with the image of the object becomes the second frame rate. The image processing apparatus according to claim 7, characterized in that

9. further comprising display control means for controlling to display a composite image obtained by compositing the captured image of the real space and the image of the object The image processing apparatus according to claim 1, characterized in that

10. further comprising display control means for controlling to display the image of the object in the real space The image processing apparatus according to claim 1, characterized in that

11. The information of the parameter includes at least any one of an exposure correction value, a white balance correction value, a gamma correction value, a noise reduction value, and an edge enhancement value The image processing apparatus according to claim 1, characterized in that

12. The information of the parameter includes information on a light source in the real space The image processing apparatus according to claim 1, characterized in that

13. a first acquisition step of acquiring information on the position and orientation of an imaging device that images the real space in the user's field of view; a second acquisition step of acquiring the information of the parameter corresponding to the position and orientation of the imaging device from an environmental map in which the information of the position and orientation and the information of the parameter related to image processing are associated with each other; a generation step of generating an image of an object to be displayed in the real space based on the information on the position and orientation of the imaging device acquired in the first acquisition step and the information of the parameter acquired in the second acquisition step A control method for an image processing apparatus, characterized by comprising

14. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images