Information processing apparatus, method for controlling information processing apparatus, and program

Abstract

To provide a technique that can create images with a feel of noise suited to images of a real space as images of a virtual space, and in turn can obtain images with a feel of noise with no discomfort as images in which the images of the virtual space are superimposed on the images of the real space.SOLUTION: An information processing apparatus of the present invention has: creation means that creates images of a virtual space to be superimposed on images of a real space; and addition means that adds noise to the images of the virtual space with characteristics based on at least one of characteristics unique to an imaging unit that picks up the images of the real space and image processing executed on the images of the real space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] Mixed Reality (MR) aims to seamlessly combine real and virtual spaces. One of the technologies related to Mixed Reality (MR) is the mixed reality (MR) system, which uses a video see-through head-mounted display (HMD). This system superimposes computer graphics (CG) images generated according to the position and orientation of an imaging device such as a video camera onto real-world images, and the images are viewed through a display device such as the HMD.

[0003] In an MR system using a video see-through HMD, when real-world images and CG images are simply superimposed on each other, the images often look strange to the eye. This is due to differences in image quality characteristics such as gradation characteristics, white balance, noise, and resolution between the real-world images and the CG images.

[0004] For example, suppose a real space image and a CG image with different levels of noise (for example, a real space image with noise and a CG image without noise) are superimposed and displayed as they are. In this case, as shown in FIG. 9, the levels of noise differ between the CG image and the surrounding real space image, resulting in an image with an unnatural feeling of noise being displayed. A technology for reducing this unnatural feeling (unnatural feeling due to the noise when a real space image and a CG image are superimposed and displayed) is disclosed in Patent Document 1. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-203326 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in conventional technologies such as the technology disclosed in Patent Document 1, noise is added to a virtual space image (image of virtual space; CG image) according to a predetermined fixed standard. Therefore, in conventional technologies, a virtual space image with a different sense of noise than a real space image (image of real space) is generated, and an image obtained by superimposing the virtual space image on the real space image may have an unnatural sense of noise.

[0007] The present invention aims to provide a technology that can generate a virtual space image with a noisy feel that matches the real space image, and ultimately can obtain an image in which the virtual space image is superimposed on the real space image without any sense of incongruity in the noisy feel. [Means for solving the problem]

[0008] A first aspect of the present invention is an information processing device comprising: a generating means for generating an image of a virtual space to be superimposed on an image of a real space; and an adding means for adding noise to the image of the virtual space with characteristics based on characteristics specific to an imaging unit that captures the image of the real space. A second aspect of the present invention is an information processing device comprising: a generating means for generating an image of a virtual space to be superimposed on an image of the real space; and an adding means for adding noise to the image of the virtual space with characteristics based on image processing to be performed on the image of the real space.

[0009] A third aspect of the present invention provides a method for generating an image of a virtual space to be superimposed on an image of a real space. and adding noise to the image of the virtual space with characteristics based on characteristics specific to an imaging unit that captures the image of the real space.A fourth aspect of the present invention is a control method of an information processing device, comprising the steps of generating an image of a virtual space to be superimposed on an image of the real space, and adding noise to the image of the virtual space with characteristics based on image processing to be performed on the image of the real space.

[0010] A fifth aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device described above.A sixth aspect of the present invention is a head-mounted display comprising the information processing device described above and display means for superimposing an image of a virtual space to which noise has been added by the adding means on an image of a real space. [Effects of the Invention]

[0011] According to the present invention, it is possible to generate a virtual space image with a noisy feel that matches the real space image, and ultimately to obtain an image in which the virtual space image is superimposed on the real space image, without any sense of incongruity in the noisy feel. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an MR system according to a first embodiment; [Figure 2] FIG. 1 is a diagram showing an example of the configuration of an imaging unit according to a first embodiment; [Figure 3] FIG. 1 is a diagram showing an example of a marker according to a first embodiment; [Figure 4] FIG. 10 is a diagram showing an example of the configuration of a CG image correction unit in the first and second embodiments. [Figure 5] FIG. 10 is a diagram showing an example of the operation of a noise characteristic evaluation unit in the first and second embodiments. [Figure 6] FIG. 10 is a diagram showing an example of the correlation between pixel values ​​and the amount of noise added in the first embodiment; [Figure 7] FIG. 10 is a diagram showing a configuration example of an MR system according to a second embodiment. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of an imaging unit in a second embodiment; [Figure 9] A diagram showing an example of an unnatural mixed reality image. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0014] When a virtual space image (a virtual space image) is superimposed on a real space image (a real space image) with a different sense of noise, an image with an unnatural sense of noise is obtained. To reduce this sense of unnaturalness, a technique has been proposed that adds noise to the virtual space image according to a predetermined fixed standard.

[0015] However, since there are individual differences in the noise characteristics of the imaging units that capture real-space images, the sense of noise (noise characteristics) of real-space images varies among multiple imaging devices. Furthermore, although the sense of noise of real-space images changes when image processing is applied to the real-space images, the image processing applied to the real-space images is not necessarily the same among multiple imaging devices. For this reason, the sense of noise of real-space images varies among multiple imaging devices. Therefore, with conventional technology, virtual-space images with a different sense of noise than real-space images (images of real space) are generated, and images obtained by superimposing the virtual-space images on real-space images may have an unnatural sense of noise.

[0016] Therefore, in the following embodiment, noise is added to the virtual space image, taking into consideration the noise characteristics specific to the imaging unit, the image processing applied to the real space image, etc. In this way, it is possible to generate a virtual space image with a noisy appearance that matches the real space image, and ultimately to obtain an image in which the virtual space image is superimposed on the real space image, without any sense of incongruity due to the noisy appearance.

[0017] (First embodiment) A first embodiment of the present invention will be described. FIG. 1 is a block diagram showing an example of the configuration of a mixed reality (MR) system as an information processing system in the first embodiment. The MR system shown in FIG. 1 is an MR system using a video see-through head-mounted display (HMD). The MR system shown in FIG. 1 displays, on the HMD, a mixed reality image in which a computer graphics (CG) image serving as a virtual space image generated in accordance with the position and orientation of an imaging device is superimposed on a real space image captured by an imaging device such as a video camera, and provides the mixed reality image to a user. The display of the mixed reality image can also be said to be a superimposed display of a real space image and a CG image.

[0018] 1, the MR system in this embodiment includes an information processing device 100 and an HMD 110. Note that the configuration shown in Fig. 1 is an example, and for example, the HMD 110 may include some or all of the functions of the information processing device 100.

[0019] The HMD 110 is a video see-through type HMD and includes an imaging unit 111 and a display unit 112. The imaging unit 111 captures an image of real space to acquire a real space image and transmits it to the information processing device 100. The real space image may be transmitted from the imaging unit 111 to the information processing device 100 via wired communication or wireless communication. The display unit 112 is a display or the like that displays an image, and displays, for example, a display image received from the information processing device 100.

[0020] FIG. 2 is a block diagram showing an example of the configuration of the imaging unit 111.

[0021] The optical system 201 is an optical system including a plurality of lenses and an aperture mechanism, and specifically includes a focus lens used for focusing, a zoom lens used for zooming, and an aperture mechanism.

[0022] The image sensor 202 is an image sensor such as a CCD or CMOS, and includes an A / D converter that converts analog signals into digital signals. An RGB color filter, for example, in a Bayer array, is formed on the surface of the image sensor 202, enabling color photography. When an image of a subject is formed on the imaging surface of the image sensor 202, the image sensor 202 generates and outputs image data (image signals). The image data output from the image sensor 202 is stored in memory 203.

[0023] In this embodiment, the imaging surface of the image sensor 202 includes an area for capturing a real space image (image area; area where the above-mentioned subject image is formed) and an optically shielded area (light-shielded area; optical black (OB) area). Image data of the image area (data of a real space image) and image data of the OB area are output from the image sensor 202. The image data of the OB area is so-called black image data, and is output as image data indicating pixel values ​​with the lowest brightness.

[0024] Image data of the OB region contains noise with characteristics specific to the imaging unit 111 (for example, noise generated in the image sensor 202 or its surrounding electrical circuits, etc.). Therefore, in reality, image data of the OB region does not indicate the lowest brightness pixel value (fixed value), but rather pixel values ​​that fluctuate due to noise. The variation (for example, standard deviation) in pixel values ​​among all pixels in a predetermined region set in at least a part of the OB region can be measured as the amount of noise (so-called dark noise) in the frame image of the real space video.

[0025] Therefore, in this embodiment, noise (noise in each frame image of the real space image) with characteristics specific to the imaging unit 111 is estimated from the imaging result of the OB region, and noise based on the estimation result is added to the CG image. Specifically, the OB region is used as a noise measurement region to measure the amount of noise. Then, noise based on the noise amount measurement result (noise amount measurement result) is added to the CG image. In this way, noise equivalent to that in the real space image can be added to the CG image with characteristics based on characteristics unique to the imaging unit 111. The noise amount may be measured constantly (for example, for each frame of the real space image) or periodically (for example, for each multiple frames of the real space image).

[0026] The memory 203 can store image data output from the image sensor 202 and data necessary for the overall processing of the image capturing unit 111. The memory 203 can also store the noise amount measurement results described above and image processing setting information described below.

[0027] The image processing circuit 204 performs image processing such as predetermined pixel interpolation processing and color conversion processing on the image data (real space image) stored in the memory 203. The image processing circuit 204 can also perform predetermined arithmetic processing using the captured image data, determine shooting conditions based on the obtained arithmetic results, and notify the system control unit 205 of the shooting conditions. The image processing circuit 204 can also perform image analysis, filtering, noise reduction, edge enhancement, contrast adjustment, synthesis, and the like. For example, the image processing circuit 204 reads out the noise amount measurement results for the OB region from the memory 203, and then performs noise reduction processing that matches the noise characteristics of the real space image based on the noise amount measurement results.

[0028] The user can select and set the image processing to be performed by the image processing circuit 204 and its intensity, thereby obtaining a real space image with the desired image quality. As described above, the noise characteristics of the real space image change depending on the image processing performed by the image processing circuit 204.

[0029] Therefore, in this embodiment, noise is added to the CG image with characteristics based on the image processing performed by the image processing circuit 204. The characteristics based on the image processing are characteristics based on image processing settings such as noise reduction settings, edge emphasis settings, and contrast adjustment settings. In this way, noise equivalent to that in the real-space image after image processing can be added to the CG image.

[0030] The image processing circuit 204 can also perform digital zoom. Unlike optical zoom, which controls the zoom lens of the optical system 201, digital zoom is image processing that obtains an image that appears to be zoomed without controlling the zoom lens, such as bicubic enlargement processing or image cropping. Therefore, the size of noise particles changes depending on the digital zoom magnification (digital zoom magnification). It is also possible to select a function and change the digital zoom magnification as needed.

[0031] Therefore, in this embodiment, noise is generated taking into account the digital zoom magnification and added to the CG image. For example, the noise is enlarged by the same magnification as the digital zoom magnification and added to the CG image.

[0032] The system control unit 205 controls the entire imaging unit 111. The system control unit 205 outputs instructions to control the shutter speed, aperture mechanism, focus lens, and zoom lens in order to capture images under the shooting conditions determined by the image processing circuit 204. Specifically, the system control unit 205 outputs instructions to an exposure amount control unit 206, a focus lens control unit 207, and a focal length control unit 208.

[0033] The system control unit 205 can also read image data of the OB area from the memory 203, measure the amount of noise in the OB area, and store the noise amount measurement result in the memory 203. The system control unit 205 can also change the image processing settings (including the digital zoom magnification setting) of the image processing circuit 204 based on a user operation. The system control unit 205 reads the noise amount measurement result from the memory 203 and stores the image processing setting information from the image processing circuit 204. It is also possible to read out information (information relating to image processing settings) and notify the same to a CG image corrector 106 (described later).

[0034] An exposure amount control unit 206 adjusts the aperture mechanism of the optical system 201, the exposure time of the image sensor 202, and the image pickup sensitivity (so-called ISO sensitivity), thereby performing appropriate exposure amount control.

[0035] The focus lens control unit 207 controls the focus lens of the optical system 201 .

[0036] The focal length control unit 208 controls the zoom lens of the optical system 201 in accordance with instructions from the system control unit 205, thereby changing the focal length.

[0037] The operation member 209 is a button, a touch panel, or the like, and receives user operations such as instructions to shoot and instructions to change image processing settings. The system control unit 205 determines or changes the operation of the imaging unit 111 in accordance with the user operations received by the operation member 209. The touch panel may be provided integrally with the display unit 112.

[0038] 1, an information processing device 100 includes a captured image acquisition unit 101, a marker detection unit 102, a position and orientation estimation unit 103, a CG data storage unit 104, a CG image generation unit 105, a CG image correction unit 106, and a display image generation unit 107.

[0039] The captured image acquisition unit 101 acquires a real space image captured by the imaging unit 111 of the HMD 110 , and transmits the acquired real space image to the marker detection unit 102 and the display image generation unit 107 .

[0040] The marker detection unit 102 identifies a location in the acquired real space image where a marker (two-dimensional code) used for estimating the position and orientation of the HMD 110 as an imaging device is reflected. A marker is a code written in black on a white background, as shown by 301 in FIG. 3. The marker detection unit 102 transmits information indicating the position, orientation, etc. of the detected marker 301 (so-called marker location information) to the position and orientation estimation unit 103 together with the real space image.

[0041] The position and orientation estimation unit 103 obtains the position and orientation of the HMD 110 from the marker location information detected by the marker detection unit 102 and the real space image. Various research reports have been published on methods for estimating the position and orientation, and any method may be used. In addition, to improve the accuracy of the position and orientation information, a magnetic sensor, an optical sensor, or the like may be used in combination. The position and orientation estimation unit 103 transmits the obtained position and orientation information to the CG image generation unit 105.

[0042] The CG image generation unit 105 acquires the CG data from the CG data storage unit 104, and performs processing such as movement, rotation, and scaling to generate a CG image to be superimposed on the real space image, which is an image of the virtual space as seen from the HMD 110. In the CG image generation process, the CG image generation unit 105 reads and uses pre-stored principal point position, focal length information, etc. from the memory 203 via the system control unit 205 of the imaging unit 111, in addition to the position and orientation information acquired from the position and orientation estimation unit 103. The CG image generation unit 105 then transmits the generated CG image to the CG image correction unit 106.

[0043] The CG image correction unit 106 generates noise based on information related to noise in the real space image notified by the system control unit 205 of the imaging unit 111, and adds the noise to the CG image. The information related to noise includes, for example, the results of measuring the amount of noise in the OB region and information on image processing settings based on user operations (user instructions).

[0044] In the MR system of this embodiment, the CG image correction unit 106 adds noise to the CG image so as to align the CG image with the real space image, thereby reducing the sense of incongruity that occurs when the real space image and the CG image serving as a virtual space image are superimposed and displayed. The CG image correction unit 106 transmits the corrected CG image to the display image generation unit 107.

[0045] The display image generation unit 107 superimposes the CG image (CG image with noise added) acquired from the CG image correction unit 106 on the real space image acquired from the captured image acquisition unit 101, and generates a mixed reality image consisting of the real space image and the CG image as a display image. The display image generation unit 107 transmits the generated mixed reality image to the display unit 112 of the HMD 110.

[0046] 4 is a block diagram showing an example of the configuration of the CG image corrector 106. The CG image corrector 106 includes a noise characteristic evaluator 401, a noise addition amount calculator 402, and a noise addition processor 403.

[0047] The noise characteristic evaluation unit 401 acquires information related to noise from the system control unit 205 of the imaging unit 111, and determines, based on the information, the correlation (correspondence) between the pixel values ​​of the CG image and the amount of noise to be added to the pixel values. Fig. 5 is a flowchart showing an example of the operation of the noise characteristic evaluation unit 401 (operation to determine the correlation).

[0048] 5, the noise characteristic evaluation unit 401 acquires information about the imaging sensitivity (ISO sensitivity) set in the image sensor 202 from the memory 203 via the system control unit 205. The ISO sensitivity information is also a type of noise-related information. Generally, as the ISO sensitivity value increases, the light-receiving sensitivity of the image sensor 202 improves, enabling it to detect weak light, but the amount of noise superimposed on the captured image data increases (deteriorates).

[0049] In S502, the noise characteristic evaluation unit 401 acquires information about the current image processing settings of the image processing circuit 204 from the image processing circuit 204 via the system control unit 205. For example, the image processing setting information includes setting information about the strength of noise reduction. The image processing setting information may be any information related to noise characteristics, and may include setting information about the strength of edge enhancement or contrast adjustment, for example. The image processing setting information may include only one piece of information or multiple pieces of information. The image processing setting information may include setting information about a digital zoom magnification. The image processing setting information may also include information other than information indicating the image processing settings. The temperature of the image capturing unit 111 (the temperature of the image capturing element 202) changes depending on the image processing performed by the image processing circuit 204, and the noise characteristics specific to the image capturing unit 111 (the noise characteristics of the image capturing element 202) also change. Therefore, the image processing setting information may include temperature information of the image capturing unit 111. Using the temperature information of the image capturing unit 111, for example, the current amount of noise can be estimated from the amount of noise in the OB region measured in the past.

[0050] In S503, the noise characteristic evaluation unit 401 acquires the noise amount measurement result for the OB region from the memory 203 via the system control unit 205.

[0051] In S504, the noise characteristic evaluation unit 401 determines the correlation between the pixel value of the CG image and the amount of noise to be added to the pixel value (amount of added noise) based on the three types of information acquired in S501 to S503.

[0052] In this embodiment, the correlation between pixel values ​​and the amount of noise addition is determined based on the above three types of information, but this is not limiting. For example, the correlation may be determined without using either the image processing setting information acquired in S502 or the noise amount measurement result acquired in S503. If there is little individual difference in noise characteristics of the image sensor 202 within a predetermined ISO sensitivity range, the correlation may be determined by acquiring only the image processing setting information. Also, the correlation between the ISO sensitivity setting and the image processing If the ISO sensitivity setting and image processing setting are predetermined settings, such as when the settings are fixed (unchangeable), only the noise amount measurement result may be acquired to determine the correlation.

[0053] 6(a) and 6(b) show examples of the correlation determined in S504. FIG. 6(a) is a correlation graph showing the correlation between pixel values ​​of a CG image and the amount of noise added when the ISO sensitivity is "X" and the noise reduction setting is "Weak." FIG. 6(b) is a correlation graph showing the correlation between pixel values ​​of a CG image and the amount of noise added when the ISO sensitivity is "X" and the noise reduction setting is "Strong." In FIGS. 6(a) and 6(b), the solid line shows the correlation for "Noise Amount Measurement Result 1," and the dashed line shows the correlation for "Noise Amount Measurement Result 2."

[0054] 5 is performed for a certain shooting scene, information on ISO sensitivity "X" is acquired in S501, information on noise reduction setting "weak" is acquired in S502, and information on "noise amount measurement result 1" is acquired in S503. In this case, in S504, the correlation shown by the solid line in FIG. 6(a) is determined, and information on this correlation is transmitted to the noise addition amount calculation unit 402.

[0055] In this embodiment, it is assumed that a plurality of pieces of correlation information corresponding to a plurality of combinations of ISO sensitivity, image processing setting information, and noise amount measurement results are stored in advance in a memory (not shown) of the noise characteristic evaluation unit 401. It is assumed that the correlation information is information (table) showing the correlation in tabular form. The correlation information can be generated based on imaging results such as a predetermined chart image. In S504, the noise characteristic evaluation unit 401 selects and acquires correlation information corresponding to the combination of the three types of information acquired in S501 to S503 from the plurality of pieces of correlation information stored in the memory.

[0056] Instead of determining the correlation, the amount of added noise may be calculated using a function f(N, ISO, X, Y). The function f(N, ISO, X, Y) is a predetermined function that takes "N", "ISO", "X", and "Y" as inputs and outputs the amount of added noise. One or two of "N", "ISO", and "X" may be excluded from the inputs. The function f(N, ISO, X, Y) is expressed, for example, as follows: f(N,ISO,X,Y)=a(N)×b(ISO)×c(X)×f0(Y) N: Noise amount measurement result ISO: ISO sensitivity X: Image processing setting information (parameters related to image processing) Y: CG image pixel value f0(Y): Reference function for calculating the amount of noise added a(N): Function for calculating the weight (weighting coefficient for f0(Y)) according to the noise amount measurement result b(ISO): Function for calculating the weighting factor for f0(Y) according to the ISO sensitivity c(X): Function for calculating the weight (weighting coefficient for f0(Y)) according to the image processing setting information

[0057] The reference function f0(Y) indicates the correlation corresponding to the combination of a predetermined ISO sensitivity (e.g., ISO 100), predetermined image processing setting information (e.g., noise reduction setting "weak"), and a predetermined noise amount (e.g., the noise amount in the OB area of ​​a typical image sensor). The functions a(N), b(ISO), and c(X) are all determined so that a large weight is assigned when the noise amount is large, and a small weight is assigned when the noise amount is small.

[0058] If we can use the function f(N,ISO,X,Y), we can create a table of multiple correlations. There is no need to store the information in advance in the memory of the noise characteristic evaluation unit 401. This reduces the memory usage of the noise characteristic evaluation unit 401, and is expected to have the effect of reducing the load on the hardware. By inputting the information acquired in S501 to S503 in Fig. 5 into the function f(N,ISO,X,Y), a function that indicates the correlation between the pixel values ​​of the CG image and the amount of noise added is uniquely determined.

[0059] The noise addition amount calculation unit 402 in FIG. 4 determines the amount of noise (noise addition amount) to be added to each pixel of each frame image of the CG video in accordance with the correlation determined by the noise characteristic evaluation unit 401. The noise amount varies over time. In this embodiment, the noise amount indicated on the vertical axis of the graphs in FIGS. 6(a) and 6(b), determined for each pixel value Y indicated on the horizontal axis of the graphs, is the standard deviation σ(Y) of the time-varying noise amount. For example, the noise addition amount calculated using the above-mentioned function f(N,ISO,X,Y) is also the standard deviation σ(Y). For each pixel of the CG video output from the CG video generation unit 105, the noise addition amount calculation unit 402 calculates the noise amount σ'(Y) from the noise amount σ(Y) corresponding to the pixel value Y of the CG video and a random function RAND using the following equation 1. The random function RAND is a function that randomly determines a value according to a normal distribution. By using Equation 1, it is possible to determine a random noise amount for each pixel of the CG image that reflects the correlation determined by the noise characteristic evaluation unit 401. Note that the random function RAND may be a function that randomly determines a value according to a distribution different from a normal distribution. It is preferable to determine the function used as the random function RAND by taking into consideration the noise characteristics of the real space image. σ'(Y)=σ(Y)×RAND (Formula 1)

[0060] The noise addition processing unit 403 uses the following equation 2 to add the noise amount σ'(Y) determined by the noise addition amount calculation unit 402 to the pixel value Y of each pixel of the CG image output from the CG image generation unit 105, thereby obtaining the pixel value Y' of each pixel of the CG image after noise has been added. Y'=Y+σ'(Y)=Y+σ(Y)×RAND (Formula 2)

[0061] In this way, noise equivalent to the real space image can be added to the CG image based on various information related to noise in the real space image notified by the system control unit 205 of the imaging unit 111. The CG image with the added noise is then transmitted from the CG image correction unit 106 to the display image generation unit 107, which generates a mixed reality image using the CG image with added noise similar to that of the real space image, and displays it on the display unit 112 of the HMD 110.

[0062] The following describes a case where the image processing setting information acquired in S502 of FIG. 5 includes setting information for the digital zoom magnification. The noise addition amount calculation unit 402 generates a noise image made of random noise σ'(Y) and enlarges the noise image using the digital zoom magnification. The noise addition amount calculation unit 402 then trims the enlarged noise image to fit the image size of the CG image and transmits the trimmed noise image to the noise addition processing unit 403. The noise addition processing unit 403 adds noise based on the noise image from the noise addition amount calculation unit 402 to the CG image from the CG image generation unit 105, thereby generating a CG image with enlarged noise added. This makes it possible to add noise that looks similar to the noise in the real-space image enlarged by digital zoom to the CG image superimposed on the real-space image, thereby producing a mixed reality image that does not look unnatural even when digitally zoomed.

[0063] The above-mentioned CG image correction process (a series of processes for adding noise to CG images) requires a certain amount of processing time, and also requires hardware resources such as a memory with sufficient capacity to realize this process, which increases the cost of the device. To suppress these disadvantages (increased cost of the device), the CG image correction unit 106 may determine whether or not to perform the CG image correction process based on the result of the noise amount measurement of the OB region acquired in S503. When the imaging unit 111 captures images at low sensitivity, the real space image captured by the imaging unit 111 contains noise. Since there is almost no noise, even if a noise-free CG image is directly superimposed, a mixed reality image with almost no sense of incongruity can be obtained. Therefore, if the amount of noise in the OB region acquired in S503 is smaller than a predetermined threshold, the CG image correction unit 106 does not perform CG image correction processing, and instead acquires a mixed reality image by directly combining the CG image with the real space image. On the other hand, if the amount of noise in the OB region is greater than the predetermined threshold, the CG image correction unit 106 performs CG image correction processing to acquire a mixed reality image. This minimizes the above-mentioned disadvantage (increased device costs). If the amount of noise in the OB region is equal to the predetermined threshold, the CG image correction processing may or may not be performed.

[0064] (Second embodiment) A second embodiment of the present invention will be described. Fig. 7 is a block diagram showing a configuration example of an MR system as an information processing system in the second embodiment. The MR system in this embodiment does not have the display image generation unit 107 of the first embodiment. Unlike the first embodiment, an imaging unit 111 generates a mixed reality image by superimposing a CG image on a real space image, and transmits the generated mixed reality image to a display unit 112. The other modules, which are assigned the same reference numerals as those in the first embodiment, operate in the same manner as in the first embodiment.

[0065] 8 is a block diagram showing an example of the configuration of the imaging unit 111 in this embodiment. Unlike the first embodiment, the imaging unit 111 in this embodiment includes a composite image generation unit 210 that generates a mixed reality image by superimposing a CG image on a real space image. Furthermore, in this embodiment, the mixed reality image generated by the composite image generation unit 210 is transmitted to the system control unit 205 via the image processing circuit 204, and then transmitted to the display unit 112. The other modules that are assigned the same reference numerals as in the first embodiment operate in the same manner as in the first embodiment.

[0066] An example of the operation of the MR system in this embodiment will be described.

[0067] First, the image sensor 202 captures an image. As a result, image data of the real space image and image data of the OB region acquired by the image sensor 202 via the optical system 201 are stored in the memory 203. Next, the system control unit 205 measures the amount of noise in the OB region from the image data of the OB region, and also stores the measurement result (noise amount measurement result) in the memory 203. Thereafter, the system control unit 205 transmits the image data of the real space image, ISO sensitivity information, and noise amount measurement result to the information processing device 100 without going through the image processing circuit 204 or without causing the image processing circuit 204 to perform image processing. Because image processing by the image processing circuit 204 is not performed, image processing setting information is not transmitted to the information processing device 100.

[0068] In the information processing device 100, similarly to the first embodiment, each module operates, and the CG image with noise added is output from the CG image correction unit 106. However, the CG image with noise added is transmitted from the CG image correction unit 106 to the imaging unit 111. Furthermore, in the information processing device 100, noise is added to the CC image without taking into consideration the image processing setting information of the imaging unit 111. The noise added to the CG image is noise with characteristics specific to the imaging unit 111, and is noise that matches the real space image before image processing by the image processing circuit 204 is applied.

[0069] The CG image (CG image with added noise) transmitted from the information processing device 100 to the imaging unit 111 is transmitted to the composite image generation unit 210 of the imaging unit 111. The composite image generation unit 210 generates a mixed reality image by superimposing this CG image on the real space image stored in the memory 203. Here, a mixed reality image is generated in which the CG image is superimposed on the real space image before image processing by the image processing circuit 204 is performed.

[0070] The image data of the mixed reality image generated by the mixed image generation unit 210 is sent to the image processing circuit 204, where image processing is performed according to a user instruction. The processed image data is transmitted to the display unit 112, and the mixed reality video is displayed on the display unit 112.

[0071] As described above, according to the second embodiment, a mixed reality image is generated using a CG image to which noise similar to that of the real-space image is added before image processing, and image processing is then performed on this mixed reality image. This allows a natural-looking mixed reality image to be obtained without using image processing setting information when adding noise to the CG image.

[0072] The above-described embodiment (including modifications) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the above-described configuration within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the above-described configurations.

[0073] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0074] 100: Information processing device 105: CG image generation unit 106: CG image correction unit

Claims

1. A generating means for generating an image of a virtual space to be superimposed on an image of a real space; an adding means for adding noise to the image of the virtual space with a characteristic based on a characteristic specific to an imaging unit that captures the image of the real space; having the imaging unit images an area for imaging an image of the real space and a light-shielded area that is shaded regardless of a change in a position of the imaging unit; The adding means adds the noise to the image of the virtual space based on an image acquired in the light-shielded area.

23. An information processing apparatus comprising:

2. The adding means adds the noise to the image of the virtual space based on a variation in pixel values ​​in a predetermined area that is at least a part of the light-shielded area.

2. The information processing apparatus according to claim 1,

3. A generating means for generating an image of a virtual space to be superimposed on an image of a real space; adding means for adding noise to the image of the virtual space with characteristics based on image processing to be applied to the image of the real space; 13. An information processing device comprising:

4. The characteristic based on the image processing is a characteristic based on at least one of a noise reduction setting, an edge enhancement setting, a contrast adjustment setting, and a temperature of an imaging unit that captures an image of the real space.

4. The information processing apparatus according to claim 3.

5. The adding means adds noise to the image of the virtual space with a characteristic based on a characteristic specific to an imaging unit that captures the image of the real space and an image processing to be performed on the image of the real space.

5. The information processing apparatus according to claim 3, wherein the information processing apparatus is a computer.

6. Noise with characteristics specific to an imaging unit that captures the image of the real space and a noise generated by the imaging unit that captures the image of the real space determining a correspondence relationship between a pixel value of an image of the virtual space and an amount of noise to be added to the pixel value based on at least one of image processing and The adding means adds noise to the image of the virtual space in accordance with the correspondence relationship.

6. The information processing device according to claim 1,

7. A plurality of correspondence relationships between pixel values ​​and noise amounts are determined in advance; The determining means selects one of the plurality of correspondence relationships based on at least one of noise of a characteristic specific to an imaging unit that captures the image of the real space and image processing to be performed on the image of the real space.

7. The information processing apparatus according to claim 6,

8. a calculation means for calculating an amount of noise to be added to the image of the virtual space by using a predetermined function that receives as input at least one of a noise amount specific to an imaging unit that captures the image of the real space and a parameter related to image processing to be performed on the image of the real space, and a pixel value of the image of the virtual space, and outputs an amount of noise to be added to the pixel value; Further having 6. The information processing device according to claim 1,

9. The adding means, when an enlargement process is applied to the image in the real space, enlarges the noise by the same enlargement factor as that of the enlargement process and adds the noise to the image in the virtual space.

9. The information processing device according to claim 1,

10. The adding means is When the amount of noise characteristic of an imaging unit that captures the image of the real space is smaller than a threshold value, no noise is added to the image of the virtual space; When an amount of noise characteristic specific to an imaging unit that captures the image of the real space is greater than a threshold, noise is added to the image of the virtual space.

10. The information processing device according to claim 1,

11. The adding means adds the noise to the image of the virtual space based on an image of the light-blocking area before image processing is performed, The information processing device includes: an image processing means for performing image processing on a composite image obtained by superimposing the image of the virtual space after the noise has been added onto the image of the real space; 3. The information processing apparatus according to claim 1, further comprising:

12. The light-shielding area is an optical black area.

12. The information processing apparatus according to claim 1, 2 or 11.

13. generating an image of a virtual space to be superimposed on an image of a real space; adding noise to the image of the virtual space with a characteristic based on a characteristic specific to an imaging unit that captures the image of the real space; having the imaging unit images an area for imaging an image of the real space and a light-shielded area that is shaded regardless of a change in a position of the imaging unit; In the adding step, the noise is added to the image of the virtual space based on an image acquired in the light-shielded area.

23. A method for controlling an information processing apparatus comprising:

14. generating an image of a virtual space to be superimposed on an image of a real space; adding noise to the image of the virtual space with characteristics based on image processing to be applied to the image of the real space; 13. A method for controlling an information processing apparatus comprising the steps of:

15. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

16. An information processing device according to any one of claims 1 to 12, a display means for displaying the image of the virtual space to which noise has been added by the adding means, superimposed on the image of the real space; A head mounted display comprising:

Images