Image processing apparatus, control method of image processing apparatus, computer program product, and computer readable storage medium

Abstract

The invention discloses an image processing apparatus, a control method of the image processing apparatus, a computer program product, and a computer readable storage medium. The image processing apparatus includes: a first acquisition unit configured to acquire information related to a position and a posture of an imaging apparatus for capturing an image of a real space; a second acquisition unit configured to acquire, from the environment map, information related to parameters associated with a position and a posture of the image pickup apparatus; and a generation unit configured to generate an image of an object to be displayed in a real space based on the information relating to the position and posture of the imaging apparatus acquired by the first acquisition unit and the information relating to the parameter acquired by the second acquisition unit, in which the first acquisition unit acquires the image of the object to be displayed in the real space by using the environment map. A position and a posture of an imaging apparatus are estimated from a captured image of a real space.

Description

Technical Field

[0001] The present invention relates to an image processing device, a control method of an image processing device, a computer program product and a computer-readable storage medium. Background Art

[0002] In recent years, research on mixed reality (MR) aimed at seamless integration of real space and virtual space is underway. As MR technology, a technology using a video see-through head mounted display (HMD) is known. In a video see-through HMD, a computer graphics (CG) image generated according to the position and posture of a camera (e.g., a camera) is superimposed on a real space image captured by the camera and displayed. The user can observe the image generated by superimposing and displaying the CG image on the real space image via the HMD.

[0003] In processes such as estimating the position and posture of a camera device and rendering a CG image, high-load operations are performed. The video perspective HMD displays the image after performing this arithmetic processing. Therefore, in some cases, a delay is generated in the display of the CG image. On the other hand, another proposed method displays a captured real-space image without waiting for the completion of the position and posture estimation and the completion of the CG image rendering, and synthesizes the CG image separately after performing a series of processes. If this method is used, the delay in displaying the real-space image is reduced. However, a delay in displaying the CG image is generated, so the display of the CG image is delayed relative to the real-space image, and the position and posture of the real-space image and the position and posture of the CG image deviate when observed. Japanese Patent Laid-Open No. 2019-95916 discloses the following method: re-projecting a CG image based on a timestamp to reduce the deviation in position and posture between the real-space image and the CG image when observed.

[0004] In some cases, if a CG image is superimposed on a real-space image without modification in a video see-through HMD, a visually unnatural image may be generated. This is because when image characteristics such as grayscale, white balance, noise, resolution, and gamma value differ between the real-space image and the CG image, the composite image of the real-space image and the CG image cannot establish optical consistency.

[0005] A real-space image is an image generated by limiting the dynamic range of the real space to the dynamic range of the camera device that is narrower than the dynamic range of the real space. Therefore, a subject that exists in front of a bright white wall exposed to the sun is photographed darker than it actually appears. On the other hand, CG, whose brightness can be adjusted regardless of the surrounding environment to be optimally displayed on a computer monitor, can be reproduced well even in a bright state in front of a white wall. This means that a composite image that simply displays a subject in front of a white wall and a CG image side by side becomes an unnatural image without optical consistency due to the brightness difference between the subject and the CG image.

[0006] Japanese Patent Laid-Open No. 2014-203326 and Japanese Patent Laid-Open No. 2018-49614 disclose methods of reducing unnaturalness generated due to optical inconsistency.

[0007] However, if there is a deviation between the timing of displaying the real space image and the timing of displaying the CG image, there is also a deviation between the timing of applying the parameters of the image processing to adjust the image characteristics. The parameters of the image processing are applied immediately to the real space time, but because there is a time lag until the CG image is displayed, the parameters appear to be applied to the CG image with a delay. Summary of the invention

[0008] The present invention provides an image processing device that can generate a CG image with improved optical consistency with a real space.

[0009] A first aspect of the present invention is an image processing device, comprising: a first acquisition unit, which is configured to acquire information related to the position and posture of a camera device used to capture an image of a real space as a user's field of view; a second acquisition unit, which is configured to acquire information related to parameters regarding image processing from an environment map, wherein the parameters are associated with the position and posture of the camera device, and in the environment map, the captured image, information related to the position and posture, and information related to the parameters regarding image processing are associated with each other; and a generation unit, which is configured to generate an image of an object to be displayed in the real space based on the information related to the position and posture of the camera device acquired by the first acquisition unit and the information related to the parameters acquired by the second acquisition unit, wherein the first acquisition unit estimates the position and posture of the camera device based on the captured image of the real space by using the environment map.

[0010] A second aspect of the present invention is a control method for an image processing device, the control method comprising: a first acquisition step for acquiring information related to the position and posture of a camera device for capturing an image of a real space as a user's field of view; a second acquisition step for acquiring information related to parameters regarding image processing from an environment map, wherein the parameters are associated with the position and posture of the camera device, and in the environment map, the captured image, information related to the position and posture, and information related to the parameters regarding image processing are associated with each other; and a generation step for generating an image of an object to be displayed in the real space based on the information related to the position and posture of the camera device acquired in the first acquisition step and the information related to the parameters acquired in the second acquisition step, wherein, in the first acquisition step, the position and posture of the camera device are estimated based on the captured image of the real space by using the environment map.

[0011] The third aspect of the present invention is a computer program product, which includes a program for causing a computer to execute the various steps of the control method of the above-mentioned image processing device. The fourth aspect of the present invention is a computer-readable storage medium, which stores a program for causing a computer to execute the various steps of the control method of the above-mentioned image processing device.

[0012] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an image processing apparatus and an HMD according to Embodiment 1;

[0014] Figure 2 It is a diagram for describing how to generate the deviation of the display position of the CG image;

[0015] Figure 3 It is a graph used to describe how to generate the brightness deviation of a CG image;

[0016] Figure 4 is a flowchart depicting an example of CG image display processing;

[0017] Figure 5 is a flowchart depicting an example of a process for acquiring exposure correction values;

[0018] Figure 6 is a block diagram of an image processing device and an optical see-through type HMD;

[0019] Figure 7 is a block diagram of an image processing apparatus and an HMD according to Embodiment 2;

[0020] Figure 8is a block diagram of an image processing apparatus and an HMD according to Embodiment 3;

[0021] Fig. 9 is a block diagram of an HMD according to Embodiment 4;

[0022] Fig.10 is a graph indicating changes in brightness of the real space, background image, and CG image;

[0023] Fig.11 is a graph indicating changes in brightness in the case of correcting a CG image; and

[0024] Fig.12 is a graph indicating brightness changes after frame interpolation of a background image. DETAILED DESCRIPTION

[0025] Embodiments of the present invention will be described with reference to the accompanying drawings. Figure 1 1 is a block diagram of an image processing apparatus 100 and a head mounted display (HMD) 110 according to Embodiment 1.

[0026] In the example described below, the image processing apparatus 100 is connected to the HMD 110 and used in this state. However, part of the functions of the image processing apparatus 100 may be performed by the HMD 110, or part of the functions of the HMD 110 may be performed by the image processing apparatus 100. In addition, the image processing apparatus 100 may be included in the HMD 110.

[0027] The camera unit 111 of the HMD 110 is a camera that captures an image of a real space that is a field of view of the user. The captured image of the real space captured by the camera unit 111 (real space image) is sent to the image processing unit 112. The camera unit 111 may include a plurality of camera devices. For example, the camera unit 111 may include a pair of camera devices that capture an image to be displayed as a background image, and a pair of camera devices that capture an image to acquire the position and posture of the HMD 110.

[0028] The image processing unit 112 acquires information related to parameters related to image processing (hereinafter referred to as "image processing parameters") from the captured image. The information related to the image processing parameters includes exposure correction values, white balance correction values, etc. The exposure correction values ​​include gain, shutter speed, and aperture value. The white balance correction value includes values ​​such as R / G and B / G. In addition, the information related to the image processing parameters may include a noise reduction value as a noise reduction (NR) parameter, an edge enhancement value as an edge enhancement processing parameter, a gamma correction value, etc.

[0029] In the following description, for example, the image processing parameters are gain and shutter speed as exposure correction values, but may be other parameters. In order to correct the exposure in an optimal manner, the image processing unit 112 first evaluates the brightness of the captured image. If the captured image is too bright, the image processing unit 112 reduces the gain or increases the shutter speed so as to calculate the exposure correction value to darken the captured image. On the other hand, if the captured image is dark, the image processing unit 112 increases the gain or reduces the shutter speed so as to calculate the exposure correction value to brighten the captured image.

[0030] The imaging unit 111 captures an image of a real space using the exposure correction value calculated by the image processing unit 112. Based on the image captured using the exposure correction value, the image processing unit 112 calculates the exposure correction value again. By repeating the calculation of the exposure correction value and the imaging using the exposure correction value, the image processing unit 112 can control the exposure of the captured image in an optimal manner. The image processing unit 112 sends the captured image to the CG synthesis unit 113 after performing image processing such as exposure correction.

[0031] The CG synthesis unit 113 generates a synthesized image of the captured image received from the image processing unit 112 and the CG image generated by the CG generation unit 106. The display unit 114 (display control unit) displays the synthesized image on an image display element such as an organic light emitting diode (OLED) installed in the HMD 110. By quickly completing the steps from imaging to display within the device, the HMD 110 can reduce the display delay of the real space image.

[0032] The generation method of the CG image to be synthesized with the real space image by the CG synthesis unit 113 will be described. The image processing unit 112 outputs the captured image to the position and posture estimation unit 103 of the image processing device 100 after performing various image processing. The communication between the image processing device 100 and the HMD 110 may be via a cable or wirelessly.

[0033] The position and posture estimation unit 103 estimates the position and posture of the HMD 110 based on the real space image received from the image processing unit 112. For example, information related to the position and posture can be estimated by using a simultaneous localization and mapping (SLAM) method. In SLAM, an environment map for estimating the position and posture is generated in advance.

[0034] The environment map generation unit 101 generates an environment map using the real space image captured by the imaging unit 111. The environment map is information that associates the captured image, information about the position and posture of the imaging unit 111 when the image is captured, and information about parameters of image processing of the captured image with each other. The environment map enables efficient estimation of the position and posture of the imaging unit 111.

[0035] Specifically, the environment map is created by generating a key frame including the following information: The key frame includes a captured image and the following information associated with the captured image.

[0036] Keyframe ID

[0037] The position and orientation of the imaging device (imaging unit 111) associated with the key frame

[0038] The captured image associated with the keyframe

[0039] Image coordinates (x, y) of feature points extracted from the captured image of the key frame

[0040] The number of feature points extracted from the captured image of the key frame

[0041] Dispersion value of feature points of the image on the captured image of the key frame (for example, a value determined by dividing the image of the key frame into 10×10 image blocks and dividing the number of regions having feature points in the blocks by the total number of regions and multiplying the result by 100)

[0042] The reliability of the keyframe (an indicator indicating the probability of alignment error determined by the number of feature points and the dispersion value lock)

[0043] When each extracted feature point is associated between captured images of multiple key frames, the unique feature point ID assigned

[0044] The three-dimensional coordinates (X, Y, Z) of the feature points determined by triangulation based on the position and posture of the camera device in each key frame (where the feature points are determined for the above-mentioned associated feature points with the same feature point ID)

[0045] The ID of the nearby keyframe

[0046] The environment map includes a plurality of key frames generated and stored in the environment map storage unit 102. The position and posture estimation unit 103 estimates the position and posture of the camera unit 111 using the environment map stored in the environment map storage unit 102. The estimated position and posture of the camera unit 111 can be regarded as the position and posture of the HMD 110. The camera device used to generate the environment map may be a device different from the camera device that captures the real space image to be synthesized with the CG image.

[0047] The CG generation unit 106 generates (renders) a CG image based on the position and posture of the HMD 110 estimated by the position and posture estimation unit 103. The real-space image used for synthesizing the CG image is an image captured by delaying the time required for the rendering process compared with the captured image used for estimating the position and posture of the HMD 110. While the CG image is being rendered, the position and posture of the HMD 110 may change. Changes in the position and posture of the HMD 110 may cause a deviation in the display position or orientation between the CG image and the real-space image synthesized with the CG image.

[0048] Figure 2 201 to 205 are time-series images of a stationary ball captured while the position and posture of the HMD 110 are changed.

[0049] The CG images 210 to 212 are images in which a triangular object (polygon) is rendered to contact the ball. The CG images 210 to 212 are images generated by rendering the object using the position and posture of the HMD 110 estimated from the real space images 201 to 203, respectively.

[0050] The synthesized images 221 to 225 are images generated by synthesizing the CG images 210 to 212 and the real-world images 201 to 205. Figure 2 In the example of , a delay of about 2 frames is generated from the estimation of the position and posture to the completion of the rendering of the CG image. Therefore, the CG image 210 rendered using the position and posture estimated from the real space image 201 is synthesized with the real space image 203.

[0051] In the synthesized image 223 generated by synthesizing the CG image 210 and the real space image 203, the triangle to be displayed in contact with the ball is displayed at a position deviated from the ball. In the same manner, in the synthesized images 224 and 225, the triangle to be displayed in contact with the ball is also displayed at a position deviated from the ball. In this manner, deviations in the display positions are continuously generated.

[0052] For the same reason as the deviation of the display position, when the optical characteristics such as the brightness of the real-space image vary, a brightness deviation etc. is generated between the CG image and the real-space image. This reduces the optical consistency between the CG image and the real-space image.

[0053] Figure 3 is a diagram for describing how to generate brightness deviation of a CG image. The real space images 301 to 305 are time-series images of a ball that is stationary while changing the surrounding brightness in a gradually darkening manner. Figure 3 In the example of Figure 2 The same way changes.

[0054] The CG images 310 to 312 are images in which triangular objects (polygons) reflecting changes in surrounding brightness are rendered. The CG images 310 to 312 are images generated by rendering the objects using exposure correction values ​​of the real-space images 301 to 303, respectively.

[0055] The synthesized images 321 to 325 are images generated by synthesizing the CG images 310 to 312 with the real-world images 301 to 305. Figure 3 In the example of , a delay of about 2 frames has been generated from the estimation of the position and posture to the completion of the rendering of the CG image. Therefore, the CG image 310 rendered using the exposure correction value of the real-space image 301 is synthesized with the real-space image 303.

[0056] In the synthesized image 323 generated by synthesizing the CG image 310 and the real-space image 303, the image of the superimposed triangular object is displayed brighter than the surrounding area. In other words, the optical consistency between the CG image and the real-space image is weakened. In the same way, the synthesized images 324 and 325 are also displayed in a state where the brightness of the surrounding area and the brightness of the object deviate. In this way, the deviation of brightness is continuously generated while the optical consistency is continuously weakened.

[0057] like Figure 2 and Figure 3 As shown, a deviation is generated between the position and posture of the HMD 110 when the CG image is started to be generated and the position and posture of the HMD 110 when the CG image is displayed. If the display timing of the CG image is delayed, the display position or brightness, etc. deviates between the real space image and the CG image due to the deviation of the position and posture of the HMD 110 until the CG image is displayed. A method for reducing the deviation of the display position or brightness, etc. between the real space image and the CG image will be described.

[0058] First, the description is used to reduce Figure 2 A method of indicating the deviation of the display position of the CG image. Figure 1 The position and posture prediction unit 104 in the HMD 110 predicts the position and posture of the HMD 110 when the CG image is displayed, based on the information about the current position and posture of the HMD 110 and the information about the acceleration acquired by the acceleration acquisition unit 115 of the HMD 110. The information about the current position and posture of the HMD 110 used for predicting the position and posture may be the position and posture estimated by the position and posture estimation unit 103, or may be the position and posture acquired by the acceleration acquisition unit 115.

[0059] The acceleration acquisition unit 115 includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like. Acceleration is a value indicating a change in three rotational motions (i.e., posture) about the X-axis, Y-axis, and Z-axis of the HMD 110, and is also referred to as "3 degrees of freedom" (3DoF). The acceleration acquisition unit 115 may be a sensor capable of acquiring 6DoF instead of 3DoF. 6DoF is a value corresponding to six motions (i.e., position and posture), which are the motions of 3DoF and the translational motions of the HMD 110 on the X-axis, Y-axis, and Z-axis.

[0060] The acceleration acquisition unit 115 can acquire the position and posture of the HMD 110 in a shorter time and at a higher frame rate than the position and posture estimation unit 103, and thus can acquire a position and posture closer than the position and posture estimated by the position and posture estimation unit 103. This means that the acceleration acquisition unit 115 can acquire the position and posture of the HMD 110 immediately before rendering. The acceleration acquisition unit 115 can acquire the change history of the position and posture until the current time at a high frame rate, and thus the position and posture prediction unit 104 can predict the position and posture several frames later.

[0061] The CG generation unit 106 generates a CG image based on the position and posture of the HMD 110 predicted by the position and posture prediction unit 104. Thus, the deviation of the display position between the real space image and the CG image when the CG image is displayed is reduced. By improving the prediction accuracy, the position and posture prediction unit 104 can reduce the deviation of the display position of the CG image relative to the real space image to a level where the deviation is not noticeable.

[0062] Next, we will describe the method for reducing Figure 3 The parameter acquisition unit 105 acquires information related to image processing parameters corresponding to the position and posture of the HMD 110. The parameter acquisition unit 105 may acquire information from an environment map associated with each key frame and information related to the image processing parameters.

[0063] When generating an environment map, the environment map generation unit 101 stores information about image processing parameters used when capturing images of key frames in association with each key frame in the environment map storage unit 102. The information about image processing parameters associated with each key frame includes, for example, exposure correction values ​​such as gain and shutter speed. The environment map generation unit 101 can generate an environment map in which information about the position and posture of the camera (HMD 110) and information about image processing parameters are associated with each other using the key frames.

[0064] The parameter acquisition unit 105 may acquire, from the environment map, an exposure correction value associated with the position and posture of the HMD 110 predicted by the position and posture prediction unit 104. The parameter acquisition unit 105 determines a key frame closest to the predicted position and posture of the HMD 110, and acquires the exposure correction value of the determined key frame as a prediction value. The closest key frame may be a key frame whose position and posture have the smallest difference with respect to the predicted position and posture of the HMD 110.

[0065] The parameter acquisition unit 105 may determine a plurality of key frames of the predicted position and posture close to the HMD 110, and interpolate the exposure correction value by weighting according to the difference in the position and posture relative to each key frame. By predicting the exposure correction value based on a plurality of key frames, the parameter acquisition unit 105 may acquire a more accurate value.

[0066] The environment map generation unit 101 may pre-generate an environment map including not only information about the exposure correction value but also information about various image processing parameters. Thus, the parameter acquisition unit 105 may acquire information about various image processing parameters associated with the position and posture of the HMD 110. The information about the image processing parameters associated with each key frame of the environment map includes, for example, at least one of an exposure correction value, a white balance correction value, a gamma correction value, a noise reduction value, and an edge enhancement value. The information about the image processing parameters may be any one of the exposure correction value, the white balance correction value, the gamma correction value, the noise reduction value, and the edge enhancement value, or information that arbitrarily combines these values.

[0067] The CG synthesis unit 113 generates a CG image based on the information about the position and posture of the HMD 110 acquired (predicted) by the position and posture prediction unit 104 and the information about the image processing parameters acquired by the parameter acquisition unit 105. As a result, the optical consistency between the real space image and the CG image is improved. The accuracy of the image processing parameters acquired by the parameter acquisition unit 105 depends on the prediction accuracy of the position and posture prediction unit 104. Therefore, if the prediction accuracy of the position and posture prediction unit 104 increases, the accuracy of the image processing parameters acquired by the parameter acquisition unit 105 increases.

[0068] In the case where the display timing of the CG image is allowed to be delayed, the parameter acquisition unit 105 may acquire information related to the image processing parameters based on information related to the position and posture used to generate the CG image, not based on information related to the position and posture predicted by the position and posture prediction unit 104. In this case as well, the optical consistency between the real space image and the CG image is improved compared to the case where the CG image is generated without adjusting the image processing parameters.

[0069] Figure 4 4 is a flowchart depicting an example of CG image display processing. In step S401, the imaging unit 111 captures an image of a real space as a user's field of view. In step S402, the image processing unit 112 determines an exposure correction value and thereby corrects the exposure. In step S403, the position and posture estimation unit 103 estimates the position and posture of the HMD 110 using the environment map stored in the environment map storage unit 102 based on the captured image subjected to exposure correction and other image processing in step S402.

[0070] In step S404, the position and posture prediction unit 104 predicts the position and posture of the HMD 110 after the delay time using the 3DoF or 6DoF information acquired by the acceleration acquisition unit 115. The delay time may be a predetermined time, or may be a time corresponding to a predetermined number of frames. The delay time may be set according to the size of the CG to be displayed or the number of CG images, etc.

[0071] In step S405, the parameter acquisition unit 105 acquires the exposure correction value associated with the position and posture of the HMD 110 predicted in step S404 from the environment map. In step S406, the CG generation unit 106 generates (renders) a CG image using the information about the position and posture of the HMD 110 predicted in step S404 and the exposure correction value acquired in step S405.

[0072] In step S407, the CG synthesis unit 113 synthesizes the real space image and the CG image generated in step S406. The display unit 114 displays on the display the synthesized image generated by the CG synthesis unit 113. The real space image to be synthesized with the CG image (the captured image of the real space when the CG image is displayed) is the real space image captured immediately before synthesis.

[0073] In step S408, the imaging unit 111 determines whether the imaging is stopped. If the user receives an instruction to stop the imaging, the imaging unit 111 stops the imaging. Figure 4 If no instruction to stop imaging is received from the user, the imaging unit 111 returns to step S401 and continues processing to display a CG image.

[0074] Figure 5 is depicted in Figure 4Flowchart of a processing example of acquiring the exposure correction value in step S405 of . In step S501, the parameter acquisition unit 105 acquires information about the position and posture of the HMD 110 after the delay time predicted by the position and posture prediction unit 104. In step S502, the parameter acquisition unit 105 refers to an environment map including information about image processing parameters such as the exposure correction value. The environment map is stored in the environment map storage unit 102. In step S503, using the environment map referred to in step S502, the parameter acquisition unit 105 acquires the exposure correction value associated with the position and posture of the HMD 110 after the delay time acquired in step S501. In step S504, the parameter acquisition unit 105 outputs the exposure correction value acquired in step S503 to the CG generation unit 106.

[0075] In the above-described Embodiment 1, the image processing apparatus 100 acquires image processing parameters associated with the position and posture of the HMD 110 from the environment map in which information about the position and posture of the HMD 110 and information about the image processing parameters are associated with each other, and thereby generates a CG image. Therefore, the image processing apparatus 100 can generate a synthesized image in which the optical consistency between the real-world image and the CG image is improved.

[0076] The image processing device 100 can also predict the position and posture of the HMD 110 when displaying a CG image, and generate a CG image based on the predicted position and posture. In this case, the image processing device 100 can reduce the deviation of the display position and the deviation of the optical characteristics (e.g., brightness) between the CG image and the real space image when the CG image is displayed.

[0077] The HMD 110 according to Embodiment 1 is a video see-through type HMD including the CG synthesis unit 113 in the above description, but may also be an optical see-through type HMD. Figure 6 610 is a block diagram of an image processing apparatus and an optical see-through type HMD to which Embodiment 1 is applied. The optical see-through type HMD 610 does not include the CG synthesis unit 113. Other configurations of the HMD 610 and the configuration of the image processing apparatus 100 are similar to those of Figure 1 The optical see-through type HMD 610 displays a CG image generated by the CG generation unit 106 of the image processing apparatus 100 on a real space that the user is viewing via a display.

[0078] Example 2

[0079] Embodiment 1 is an example for acquiring information about image processing parameters (such as exposure correction values) associated with the position and posture of the HMD 110 after the delay time using an environment map including information about imaging parameters. Embodiment 1 is based on the assumption that the brightness of the real space does not change between the time when the environment map is constructed and the time when the user actually wears the HMD 110 and a CG image is superimposed and displayed.

[0080] Embodiment 2 is an example based on the assumption that the brightness of the real space changes between the time when the environment map is constructed and the time when the user actually wears the HMD and the CG image is superimposed and displayed. Figure 7 7 is a block diagram of an image processing apparatus 700 and an HMD 710 according to Embodiment 2. Figure 1 In addition to the configuration of the image processing apparatus 100 of the embodiment 1, the image processing apparatus 700 further includes a parameter evaluation unit 107. The same constituent elements as those of Embodiment 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0081] In the case where the brightness of the real space has changed since the environment map generation unit 101 generated the environment map, the image processing parameters acquired from the environment map by the parameter acquisition unit 105 may not be suitable for the brightness of the real space when the CG image is displayed. Therefore, the parameter evaluation unit 107 acquires and compares the information on the image processing parameters acquired from the environment map and the information on the image processing parameters determined by the image processing unit 112 for the real space when the CG image is displayed. The real space when the CG image is displayed may be the real space immediately before the synthesized CG image.

[0082] In the case where the information related to the image processing parameter is an exposure correction value, the parameter evaluation unit 107 compares the exposure correction value acquired from the environment map with the exposure correction value determined by the image processing unit 112 for the real space when the CG image is displayed. If the two acquired correction values ​​are different, the parameter evaluation unit 107 considers that this is because the brightness of the real space has changed. In the case where the two acquired exposure correction values ​​are different, the parameter evaluation unit 107 updates the exposure correction value of the environment map stored in the environment map storage unit 102 based on the exposure correction value determined by the image processing unit 112.

[0083] Among the key frames of the environment map, the key frame that becomes the target of updating the exposure correction value is the key frame whose exposure correction value is acquired by the parameter evaluation unit 107. The parameter evaluation unit 107 may include, among the targets of updating the exposure correction value, a key frame whose position and posture differ from the update target key frame by a predetermined threshold or less.

[0084] In a case where the exposure correction value acquired from the environment map is different from the exposure correction value determined by the image processing unit 112 for the real space when the CG image is displayed, the CG generation unit 106 generates the CG image using the exposure correction value determined by the image processing unit 112. In other words, the CG generation unit 106 generates the CG image based on the position and posture of the HMD 110 predicted by the position and posture prediction unit 104 and the exposure correction value determined by the image processing unit 112.

[0085] In the above-described embodiment 2, when the information related to the image processing parameters acquired from the environment map and the information related to the image processing parameters determined by the camera (image processing unit 112) are different, the image processing device 700 updates the information related to the image processing parameters of the environment map. Therefore, even if the brightness of the real space changes after the environment map is generated, the image processing device 700 can update the environment map and thereby acquire an exposure correction value corresponding to the brightness of the real space after the change.

[0086] Example 3

[0087] Embodiment 1 is an example of predicting in advance the position and posture of the HMD 110 when displaying a CG image before rendering the CG image. On the other hand, Embodiment 3 is an example of correcting information related to the position and posture of the HMD 110 based on information related to the acceleration of the HMD 110 when displaying the CG image after rendering the CG image. The information related to the acceleration when displaying the CG image may be information related to the acceleration immediately before the CG image and the real space image are synthesized.

[0088] Figure 8 is a block diagram of an image processing apparatus 800 and an HMD 110 according to Embodiment 3. The HMD 110 has the same configuration as that of Embodiment 1. The image processing apparatus 800 has Figure 1 The configuration of the image processing apparatus 100 in the embodiment includes a position and posture correction unit 108 and a CG correction unit 109 instead of the position and posture prediction unit 104. The same constituent elements as those in Embodiment 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0089] The image processing device 800 corrects the CG image based on the information about the acceleration acquired by the acceleration acquisition unit 115 when the CG image is displayed (immediately before the CG image is synthesized with the real-world image). This method is so-called "post-processing". By using post-processing, the image processing device 800 can generate a synthesized image that does not generate a deviation between the real-world image and the CG image without predicting the position and posture of the HMD 110 (camera unit 111). Embodiment 3 is applied to the image processing device 800 using post-processing like this.

[0090] The exposure of the real-space image captured by the camera unit 111 has been corrected by the image processing unit 112. The real-space image whose exposure is corrected by the image processing unit 112 is input to the position and posture estimation unit 103. The parameter acquisition unit 105 acquires an exposure correction value associated with the position and posture estimated by the position and posture estimation unit 103 based on the environment map. However, in the case of using post-processing, the image processing device 800 does not need to include the parameter acquisition unit 105. In this case, the image processing device 800 can acquire the exposure correction value determined by the image processing unit 112.

[0091] The CG generation unit 106 generates (renders) a CG image using the position and posture of the HMD 110 estimated by the position and posture estimation unit 103 and the exposure correction value acquired by the parameter acquisition unit 105. However, if the CG synthesis unit 113 synthesizes the rendered CG image with the real space image without any correction, its display position deviates due to a delay in display timing.

[0092] Therefore, the position and posture correction unit 108 corrects the position and posture of the HMD 110 estimated by the position and posture estimation unit 103 based on the information about the acceleration acquired by the acceleration acquisition unit 115. In addition, the CG correction unit 109 also corrects the brightness of the CG image. The CG correction unit 109 acquires, from the environment map, an exposure correction value associated with the position and posture of the HMD 110 corrected by the position and posture correction unit 108. The CG synthesis unit 113 synthesizes the CG image corrected by the CG correction unit 109 with the captured image of the real space immediately before the synthesis.

[0093] In the above-described embodiment 3, the position and posture correction unit 108 corrects the information related to the position and posture of the HMD 110 based on the information related to the acceleration of the HMD 110. The CG correction unit 109 acquires information related to the image processing parameters associated with the corrected position and posture of the HMD 110 from the environment map, and corrects the CG image based on the acquired image processing parameters. The image processing apparatus 800 includes the position and posture correction unit 108 and the CG correction unit 109, and corrects the CG image using post-processing, whereby the optical consistency between the CG image and the real-space image in the synthesized image can be improved.

[0094] Example 4

[0095] Embodiment 4 is an example in which the HMD includes two kinds of imaging units: a dedicated computer vision (CV) imaging unit for calculating the position and orientation, and a background imaging unit for capturing an image of a real space.

[0096] Fig. 9 1 is a block diagram of an HMD 910 according to Embodiment 4. The HMD 910 includes a part of the functions of the image processing apparatus 100 according to Embodiment 1. Embodiment 4 is applied to the HMD 910 used as an image processing apparatus. The same constituent elements as those of the image processing apparatus 100 and the HMD 110 according to Embodiment 1 are denoted by the same reference numerals, and a detailed description thereof will be omitted.

[0097] The CV camera unit 901 is an image capturing device that captures images at a low resolution and a high frame rate, and can calculate positions and postures more frequently. For example, the CV camera unit 901 captures images at a frame rate of 90 FPS.

[0098] The background camera unit 921 is a camera device that captures images at a high resolution and a low frame rate, and improves the resolution of the appearance of the real space image. For example, the background camera unit 921 captures images at a frame rate of 30 FPS.

[0099] In Embodiment 4, the CG image is corrected by post-processing as in Embodiment 3. The HMD 910 includes a position and posture correction unit 904 and a CG correction unit 905. In the same manner as in Embodiment 3, the position and posture correction unit 904 corrects the position and posture of the HMD 910 estimated by the position and posture estimation unit 103 based on the information about the acceleration acquired by the acceleration acquisition unit 115. In addition, the CG correction unit 905 corrects the brightness of the CG image.

[0100] Fig.10 It is a graph indicating the brightness changes of the real space, the background image, and the CG image. The ordinate indicates the brightness level. The abscissa indicates the frame number. The brightness of the real space is the brightness of the captured image of the real space captured by the CV camera unit 901. The brightness of the background image is the brightness of the captured image of the real space captured by the background camera unit 921. Fig.10 Indicates that as the frame number advances over time, the real space becomes darker.

[0101] When the brightness of the image of the real space captured by the CV camera unit 901 changes, if the brightness of the real space is applied to the CG image without considering the delay time required to generate the CG image, the brightness of the CG image changes with the delay time. Fig.10 In the example shown in FIG. 1 , the delay time is a time equivalent to 3 frames.

[0102] On the other hand, since the frame rate of the background camera unit 921 is 30 FPS (which is 1 / 3 of the frame rate of the CV camera unit 901 (i.e., 90 FPS)), the brightness of the background image captured by the background camera unit 921 does not change within 3 frames. In other words, the brightness of the background image changes every 3 frames according to the surrounding brightness. Since the brightness of the background image (real space image) and the brightness of the CG image are different, the optical consistency of the synthesized image is weakened.

[0103] To prevent this, the position and posture correction unit 904 and the CG correction unit 905 correct the CG image using post-processing so as to match the brightness of the real space when the CG image is displayed. Fig.11 is a graph indicating changes in brightness of a real space, a background image, and a CG image when a CG image is corrected. Fig.11 In FIG. 9 , the graph of the brightness of the CG image overlaps with the graph of the brightness of the real space. By correcting the CG image based on the image of the real space captured by the CV imaging unit 901, the brightness of the CG image changes based on the brightness of the real space.

[0104] However, the brightness of the background image (real space image) captured by the background imaging unit 921 changes every 3 frames according to the surrounding brightness, so the brightness of the background image becomes different from the brightness of the CG image according to the frame. Fig.11 In this case, the optical consistency in the composite image of the real space image and the CG image is weakened.

[0105] To prevent this, the image processing unit 922 corrects the real-world image of 30 FPS using projection transformation or the like, and performs frame interpolation to make it 90 FPS. The image processing unit 922 may perform frame interpolation based on the position and orientation corrected by the position and orientation correction unit 904 .

[0106] The image processing unit 922 also acquires an exposure correction value corresponding to the position and posture of the interpolated frame from the environment map, and corrects the brightness of the frame. Fig.12 is a graph indicating the brightness change of the background image after frame interpolation. Fig.12 , the curve graph of the brightness of the background image overlaps with the curve graphs of the brightness of the real space and the CG image. In other words, the brightness of the background image (real space image) changes according to the change in the brightness of the CG image. The CG synthesis unit 113 generates a synthesized image by synthesizing the real space image after frame interpolation by the image processing unit 922 and the CG image corrected by the CG correction unit 905.

[0107] In the above-described Embodiment 4, when the frame rate of the image captured by the background imaging unit 921 is different from the frame rate of the image captured by the CV imaging unit 901, the HMD 910 corrects the background image (real space image) captured by the background imaging unit 921. The HMD 910 uses the information on the position and posture of the HMD 910 corrected by the position and posture correction unit 904 and the image processing parameters acquired from the environment map by the image processing unit 922 to correct the real space image to be synthesized with the CG image.

[0108] Specifically, when the frame rate of the image captured by the background camera unit 921 is lower than the frame rate of the image captured by the CV camera unit 901, the HMD 910 performs frame interpolation on the real space image to be synthesized with the CG image. Using the corrected information about the position and posture of the HMD 910 and the image processing parameters associated with the position and posture, the HMD 910 interpolates the frames of the real space image so that the frame rate of the real space image becomes the frame rate of the image captured by the background camera unit 921.

[0109] Therefore, even when the frame rate of the real-world image to be synthesized with the CG image is different from the frame rate of the image captured by the CV imaging unit 901, the HMD 910 can improve the optical consistency between the real-world image and the CG image.

[0110] Example 5

[0111] Embodiment 5 is an example in which information related to image processing parameters includes information related to a light source in a real space (light source information). Embodiment 5 is applicable to the HMD and image processing apparatus of Embodiments 1 to 4, respectively. Hereinafter, a case where Embodiment 5 is applied to the HMD 110 and the image processing apparatus 100 will be described, but Embodiment 5 can also be applied to the HMD and the image processing apparatus of Embodiments 2 to 4.

[0112] In addition to the method of matching the brightness of a CG image with that of a real-space image, a known method for establishing optical consistency between a real-space image and a CG image is a method of drawing shadows and reflections, etc. on CG based on light source information. The light source information includes information on the position and intensity of the light source, etc., and the realism of the CG image is improved by drawing shadows and reflections corresponding to the light source information.

[0113] As a method of simply realizing real reflection, image-based lighting (IBL) is known. In IBL, a 360° real-space image is acquired in advance, and which part of the 360° image is used as a reflected image is determined based on the position and posture of the HMD 110. Therefore, as described in Embodiment 1, if the display of the CG image is delayed with respect to the real-space image, a delay is also generated in the reflected image to which IBL is applied.

[0114] Therefore, the environment map generation unit 101 stores the light source information as information related to the image processing parameters in association with the key frames in the environment map storage unit 102. By generating an environment map including the light source information, the parameter acquisition unit 105 or the CG correction unit 905 can acquire the light source information associated with the real space image to be synthesized with the CG image from the environment map.

[0115] According to the above-mentioned Embodiment 5, the HMD 110 can use the light source information included in the environment map to improve the optical consistency between the real space image and the CG image. In addition, the position and posture prediction unit 104 of the image processing device 100 predicts the position and posture of the HMD 110 when the CG image is displayed, thereby improving the delay when the light source information is applied to the CG image.

[0116] Although the preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and can be modified or altered in various ways within the scope of the spirit thereof. These embodiments can be combined as needed within the scope of the spirit of the present invention.

[0117] The image processing apparatus and the respective functional units of the HMD according to the embodiments 1 to 3 (see Figure 1 , Figure 6 , Figure 7 and Figure 8 ) and the respective functional units of the HMD according to Embodiment 4 (see Fig. 9 ) may or may not be an independent hardware unit. The functions of at least two functional units may be implemented by a common hardware unit. Each of the multiple functions of a functional unit may be implemented by an independent hardware unit. At least two functions of a functional unit may be implemented by a common hardware unit. In addition, each functional unit may or may not be implemented by a hardware unit. For example, the device may include a processor and a memory storing a control program, and the functions of at least some of the functional units of the device may be implemented by the processor reading the control program from the memory and executing the program.

[0118] According to the present invention, it is possible to generate a CG image whose optical consistency with the real space is improved.

[0119] Note that the various types of control described above may be processes performed by one hardware (e.g., a processor or a circuit) or in other ways. Processing may be shared among multiple hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) to perform control of the entire device.

[0120] In addition, the above-mentioned processor is a processor in a broad sense, and includes a general-purpose processor and a special-purpose processor. Examples of general-purpose processors include a central processing unit (CPU), a microprocessing unit (MPU), a digital signal processor (DSP), etc. Examples of special-purpose processors include a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. Examples of PLDs include a field programmable gate array (FPGA), a complex programmable logic device (CPLD), etc.

[0121] Other embodiments

[0122] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0123] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: a first acquisition unit configured to acquire information about a position and a posture of an image pickup device for capturing an image of a real space as a user's field of view; a second acquisition unit configured to acquire information related to a parameter regarding image processing, wherein the parameter is associated with a position and a posture of the camera device, from an environment map in which a captured image, information related to the position and the posture, and information related to the parameter regarding image processing are associated with each other; and a generating unit configured to generate an image of the object to be displayed in the real space based on the information about the position and posture of the camera device acquired by the first acquiring unit and the information about the parameter acquired by the second acquiring unit, The first acquisition unit estimates the position and posture of the camera device according to the captured image of the real space by using the environment map.

2. The image processing apparatus according to claim 1, wherein: The first acquisition unit predicts the position and posture of the imaging device when the image of the object is displayed based on the information about the position and posture of the imaging device and the information about the acceleration of the imaging device, and The second acquisition unit acquires, from the environment map, information about the parameter associated with the position and posture of the imaging device predicted by the first acquisition unit.

3. The image processing device according to claim 2, further comprising: An updating unit is used to update the information related to the parameter of the environment map based on the information related to the parameter determined by the camera device when the information related to the parameter acquired by the second acquisition unit from the environment map is different from the information related to the parameter determined by the camera device for the real space when the image of the object is displayed.

4. The image processing device according to claim 2 or 3, wherein: When the information related to the parameters acquired by the second acquisition unit from the environment map is different from the information related to the parameters determined by the camera device for the real space when displaying the image of the object, the generation unit generates an image of the object based on the position and posture of the camera device predicted by the first acquisition unit and the information related to the parameters determined by the camera device.

5. The image processing apparatus according to claim 1, wherein: the first acquisition unit corrects the information about the position and posture of the imaging device based on the information about the acceleration of the imaging device when the image of the object is displayed, and The second acquisition unit acquires, from the environment map, information about the parameter associated with the position and posture of the imaging device corrected by the first acquisition unit.

6. The image processing device according to claim 5, further comprising: a display control unit configured to perform control so that a synthesized image in which the captured image of the real space and the image of the object are synthesized is displayed, In which, when a first frame rate of the captured image of the real space to be synthesized with the image of the object is different from a second frame rate of the captured image of the real space used to estimate the position and posture of the camera device based on the environment map, the display control unit corrects the captured image of the real space to be synthesized with the image of the object by using the information related to the position and posture of the camera device corrected by the first acquisition unit and the information related to the parameters acquired by the second acquisition unit.

7. The image processing apparatus according to claim 6, wherein: When the first frame rate is lower than the second frame rate, the display control unit uses the information related to the position and posture of the imaging device corrected by the first acquisition unit and the information related to the parameters acquired by the second acquisition unit to interpolate frames so that the frame rate of the captured image of the real space to be synthesized with the image of the object becomes the second frame rate.

8. The image processing device according to any one of claims 1 to 3, further comprising: A display control unit is configured to perform control so that a synthesized image in which the captured image of the real space and the image of the object are synthesized is displayed.

9. The image processing device according to any one of claims 1 to 3, further comprising: A display control unit is configured to perform control so that the image of the object is displayed in the real space.

10. The image processing apparatus according to any one of claims 1 to 3, wherein: The information related to the parameters includes at least any one of an exposure correction value, a white balance correction value, a gamma correction value, a noise reduction value, and an edge enhancement value.

11. The image processing apparatus according to any one of claims 1 to 3, wherein: The information related to the parameters includes information related to light sources in the real space.

12. A control method for an image processing device, the control method comprising: a first acquisition step for acquiring information about a position and a posture of an image pickup device for capturing an image of a real space serving as a field of view of a user; a second acquisition step for acquiring information related to parameters related to image processing from an environment map in which captured images, information related to the position and posture, and information related to parameters related to image processing are associated with each other; and a generating step for generating an image of the object to be displayed in the real space based on the information related to the position and posture of the camera device acquired in the first acquiring step and the information related to the parameter acquired in the second acquiring step, Wherein, in the first acquisition step, the position and posture of the camera device are estimated based on the captured image of the real space by using the environment map.

13. A computer program product comprising a program for causing a computer to execute each step of the control method of the image processing apparatus according to claim 12.

14. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method of the image processing apparatus according to claim 12.

Images