Information processing device

Abstract

To provide a technology that can reduce discomfort in composite reality and augmented reality (make the composite reality and the augmented reality closer to reality) while suppressing a significant increase in an amount of computer throughput.SOLUTION: An information processing device of the present invention includes: first acquisition means for acquiring a real image, which is an image of real space; second acquisition means for acquiring a virtual image, which is an image of a virtual object, to be composited with the real image; analysis means for analyzing the virtual image; and correction means for correcting, on the basis of a result of the analysis by the analysis means, the real image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, and more particularly to a technique for synthesizing an image of a virtual object with an image of a real space. [Background technology]

[0002] Technologies have been proposed that provide mixed reality (MR) and augmented reality (AR) by displaying an image in which an image of a real space (real image) is combined with an image of a virtual object (virtual image). Furthermore, in relation to such technologies, technologies have been proposed that correct the virtual image in order to reduce the sense of incongruity felt by the mixed reality or augmented reality (to make the mixed reality or augmented reality closer to reality).

[0003] Patent Document 1 discloses a technique for generating an additional virtual image. Patent Document 2 discloses a technique for correcting the luminance and chromaticity of a virtual image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-352960 [Patent Document 2] Patent Publication No. 2021-056963 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the techniques disclosed in Patent Documents 1 and 2 require a large amount of computer processing because a virtual image is added or corrected, which may cause a delay in compositing the virtual image with the real image. This delay causes a misalignment between the real image and the virtual image in the composite image obtained by compositing. Furthermore, attempts to suppress the misalignment result in a delay in displaying the real image.

[0006] The present invention aims to provide a technology that makes it possible to reduce the sense of incongruity felt in mixed reality and augmented reality (to make mixed reality and augmented reality closer to reality) while suppressing a significant increase in the amount of processing required by a computer. [Means for solving the problem]

[0007] A first aspect of the present invention is an information processing device characterized by having a first acquisition means for acquiring a real image which is an image of real space, a second acquisition means for acquiring a virtual image which is an image of a virtual object to be composited with the real image, an analysis means for analyzing the virtual image, and a correction means for correcting the real image based on the results of the analysis by the analysis means.

[0008] A second aspect of the present invention is a control method for an information processing device, characterized by having a first acquisition step of acquiring a real image which is an image of real space, a second acquisition step of acquiring a virtual image which is an image of a virtual object to be composited with the real image, an analysis step of analyzing the virtual image, and a correction step of correcting the real image based on the results of the analysis in the analysis step.

[0009] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device. [Effects of the Invention]

[0010] According to the present invention, it is possible to reduce the sense of incongruity felt in mixed reality or augmented reality (to make mixed reality or augmented reality closer to reality) while suppressing a significant increase in the amount of processing on a computer. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of a display device according to a first embodiment. [Figure 2] 3A to 3C are schematic diagrams showing examples of various images according to the first embodiment. [Figure 3]FIG. 10 is a block diagram showing an example of the configuration of a display device according to a second embodiment. [Figure 4] 10A to 10C are schematic diagrams showing examples of various images according to the second embodiment. [Figure 5] FIG. 10 is a block diagram showing an example of the configuration of a display device according to a third embodiment. [Figure 6] 10A to 10C are schematic diagrams showing examples of various images according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] (First embodiment) A first embodiment of the present invention will be described. In the following, an example in which the present invention is applied to a display device will be described. The display device is, for example, a video see-through type HMD (Head Mounted Display).

[0013] FIG. 1 is a block diagram showing an example of the configuration of a display device according to the first embodiment.

[0014] The real image capturing unit 11 captures an image of real space to acquire image data of the real image (image of the real space). The real image capturing unit 11 has, for example, a group of lenses that is an objective optical system and a CMOS image sensor that converts light into an electrical signal. Note that it is sufficient to be able to acquire the real image (image data of the real image). For example, the real image capturing unit 11 may be an external device of an information processing device to which the present invention is applied, and the information processing device may have an input interface that acquires the real image from outside.

[0015] The virtual image generating unit 12 generates a virtual image, which is an image of a virtual object (virtual space), to be composited with a real image. For example, the virtual image generating unit 12 generates image data of a 2D virtual image from 3D model data by performing rendering. The virtual image generating unit 12 is, for example, a computing device such as a CPU or GPU. Note that it is only necessary to acquire a virtual image (image data of the virtual image). For example, the virtual image generating unit 12 may be an external device of an information processing device to which the present invention is applied, and the information processing device may have an input interface for acquiring a virtual image from outside. The real image capturing unit 11 and the virtual image generating unit 12 may be provided in the same device or in different devices.

[0016] When generating a virtual image, the virtual image generation unit 12 estimates the position and orientation of the display device (real image capture unit 11). The estimation method is not particularly limited, and for example, the position and orientation may be estimated by analyzing the real image acquired by the real image capture unit 11, or the position and orientation may be estimated by calculation using signals output from another capture unit. The position and orientation may also be estimated by calculation using signals output from a sensor other than the image sensor (for example, an acceleration sensor, an angular velocity sensor, or a geomagnetic sensor). By using the position and orientation estimation results, a virtual image that matches the real image can be generated.

[0017] The virtual video analysis unit 13 analyzes the virtual video generated by the virtual video generation unit 12. For example, the virtual video analysis unit 13 analyzes metadata of the virtual video (data accompanying the virtual video). The virtual video analysis unit 13 may analyze pre-rendering data of the virtual video (for example, data of a 3D model). The virtual video analysis unit 13 may analyze post-rendering data of the virtual video (for example, video data of a 2D virtual video). The virtual video analysis unit 13 is, for example, a computing device such as a CPU or a GPU, similar to the virtual video generation unit 12.

[0018] The real image correction unit 14 corrects the real image captured by the real image capturing unit 11 based on the results of the analysis by the virtual image analysis unit 13. For example, the real image correction unit 14 corrects at least one of the brightness, chromaticity, gamma characteristics, noise, distortion, position, and resolution of the real image. The real image correction unit 14 is, for example, a semiconductor device such as a DSP, FPGA, ISP, or ASIC. The real image correction unit 14 may also be a computing device such as a CPU or GPU.

[0019] The image synthesizing unit 15 synthesizes the virtual image generated by the virtual image generating unit 12 with the real image corrected by the real image correcting unit 14. This process generates a composite (or augmented) real image in which the real image and the virtual image can be viewed simultaneously. The image synthesizing unit 15 is, for example, a semiconductor device such as a DSP, FPGA, ISP, or ASIC, similar to the real image correcting unit 14. The image synthesizing unit 15 may be an external device to the information processing device to which the present invention is applied.

[0020] The image display unit 16 displays the image synthesized by the image synthesis unit 15, i.e., the mixed (or augmented) reality image generated by the image synthesis unit 15. The image display unit 16 has a display device such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED). The image display unit 16 may have an eyepiece optical system. The image display unit 16 may be an external device of the information processing device to which the present invention is applied.

[0021] FIG. 2 is a schematic diagram showing an example of various images according to the first embodiment.

[0022] Image 21 is a real image captured by real image capturing unit 11. Real image 21 is an image of a dark night, and includes a distant moon 210 and clouds 211, and a nearby face 212.

[0023] Image 22 is a virtual image generated by virtual image generation unit 12. In virtual image 22, a bright flash of light 221, which is a virtual object (CG), is drawn at a position corresponding to the vicinity of face 212 in real image 21. Virtual image analysis unit 13 analyzes virtual image 22.

[0024] Image 23 is a composite image (composite (or augmented) reality image) generated by image synthesis unit 15. Real image correction unit 14 corrects real image 21 based on the results of analyzing virtual image 22, and image synthesis unit 15 generates composite image 23 by synthesizing virtual image 22 with the corrected real image.

[0025] Because virtual image 22 includes a bright flash of light 221, real image corrector 14 increases the brightness of the entire real image 21. Then, image synthesizer 15 synthesizes virtual image 22 with a real image including a moon 230, clouds 231, and face 232, the brightness of which has been increased. This makes it possible to express the appearance of the surrounding brightness being increased by flash of light 221, thereby reducing the sense of incongruity felt by the mixed (or augmented) reality (making the mixed (or augmented) reality closer to reality).

[0026] As described above, according to the first embodiment, a virtual image is analyzed, and a real image is corrected based on the analysis results. Generally, the amount of processing required to correct a real image is less than the amount of processing required to correct a virtual image. Therefore, by correcting a real image based on the analysis results of a virtual image, it is possible to reduce the sense of incongruity felt by mixed (or augmented) reality (make mixed reality or augmented reality closer to reality) while suppressing a significant increase in the amount of processing required by the computer.

[0027] Furthermore, by suppressing a significant increase in the amount of computer processing, it is possible to suppress a significant delay in combining a virtual image with a real image. As a result, it is possible to suppress a large positional deviation between the real image and the virtual image in the combined image, and a significant delay in displaying the real image. This effect is particularly pronounced when processing the real image and the virtual image is performed by different processors (or circuits).

[0028] (Second embodiment) A second embodiment of the present invention will be described. Note that, hereinafter, explanations of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described. In the second embodiment, not only a virtual image but also a real image is analyzed, and the real image is corrected based on the results of analyzing the virtual image and the results of analyzing the real image.

[0029] FIG. 3 is a block diagram showing an example of the configuration of a display device according to the second embodiment.

[0030] The real image analysis unit 31 analyzes the real image captured by the real image capturing unit 11. For example, the real image analysis unit 31 not only acquires image information such as the brightness, chromaticity, gamma characteristics, noise, distortion, position, and resolution of the real image, but also acquires depth information of the real image by depth calculation and acquires segmentation information of the real image by segmentation. The real image analysis unit 31 is, for example, a semiconductor device such as a DSP, FPGA, ISP, or ASIC, similar to the real image correction unit 14.

[0031] The real image correcting section 14 corrects the real image acquired by the real image capturing section 11 based on not only the result of the analysis by the virtual image analyzing section 13 but also the result of the analysis by the real image analyzing section 31 .

[0032] FIG. 4 is a schematic diagram showing an example of various images according to the second embodiment.

[0033] Image 41 is a composite image generated by image synthesis unit 15. Real image correction unit 14 corrects real image 21 based on the results of analyzing real image 21 and the results of analyzing virtual image 22, and image synthesis unit 15 generates composite image 41 by synthesizing virtual image 22 with the corrected real image.

[0034] The real image analysis unit 31 acquires, through segmentation, division information that divides (indicates) the regions of the moon 210, clouds 211, and face 212, and acquires, through depth calculation, depth information that indicates the depth of each of the moon 210, clouds 211, and face 212. Therefore, the real image correction unit 14 slightly increases the brightness of the distant moon 210 and clouds 211 and greatly increases the brightness of the nearby face 213. The image synthesis unit 15 then synthesizes the virtual image 22 with a real image including the moon 410 and clouds 411, whose brightness has been slightly increased, and the face 412, whose brightness has been greatly increased. This makes it possible to express how the brightness of objects far from the flash 221 is not significantly affected by the flash 221, while the brightness of objects closer to the flash 221 is significantly affected by the flash 221. As a result, the sense of incongruity felt in mixed (or augmented) reality can be further reduced (mixed (or augmented) reality can be made to more closely resemble reality).

[0035] As described above, according to the second embodiment, not only the virtual image but also the real image is analyzed, and the real image is corrected based on the results of analyzing the virtual image and the results of analyzing the real image. This makes it possible to further reduce the sense of incongruity felt by mixed (or augmented) reality (to make the mixed reality or augmented reality closer to reality) while suppressing a significant increase in the amount of processing by the computer.

[0036] (Third embodiment) A third embodiment of the present invention will be described. Note that, hereinafter, a description of the same points as in the second embodiment (for example, the same configuration and processing as in the second embodiment) will be omitted, and only points different from the second embodiment will be described. In the third embodiment, not only the real image but also the virtual image is corrected based on the results of analyzing the virtual image and the results of analyzing the real image.

[0037] FIG. 5 is a block diagram showing an example of the configuration of a display device according to the third embodiment.

[0038] The analysis result control unit 51 performs control to balance the results of the analysis by the real video analysis unit 31 and the results of the analysis by the virtual video analysis unit 13 (control to reconcile the results of these analyses). For example, the analysis result control unit 51 determines whether to correct the real video or the virtual video based on the results of the analysis by the real video analysis unit 31 and the results of the analysis by the virtual video analysis unit 13, and performs control to balance the results of these analyses. The control to balance the results of the analysis by the real video analysis unit 31 and the results of the analysis by the virtual video analysis unit 13 (control to reconcile the results of these analyses) may also be interpreted as control to balance the correction of the real video and the correction of the virtual video (control to reconcile the corrections). The analysis result control unit 51, like the real video correction unit 14, is, for example, a semiconductor device such as a DSP, FPGA, ISP, or ASIC.

[0039] The real image correcting section 14 corrects the real image acquired by the real image capturing section 11 based on the result of control by the analysis result control section 51.

[0040] The virtual image correction unit 52 corrects the virtual image generated by the virtual image generation unit 12 based on the control results of the analysis result control unit 51. For example, the virtual image correction unit 52 corrects at least one of the luminance, chromaticity, gamma characteristics, noise, distortion, position, resolution, and transparency of the virtual image. The virtual image correction unit 52 can also correct the virtual image by taking into account depth information and section information of the real image. The virtual image correction unit 52 can also perform correction to add a virtual object as a correction to the virtual image. Like the real image correction unit 14, the virtual image correction unit 52 is, for example, a semiconductor device such as a DSP, FPGA, ISP, or ASIC. The virtual image correction unit 52 may also be a computing device such as a CPU or GPU.

[0041] FIG. 6 is a schematic diagram showing an example of various images according to the third embodiment.

[0042] Image 61 is a composite image generated by image synthesis unit 15. Real image correction unit 14 corrects real image 21 based on the control result of analysis result control unit 51, and virtual image correction unit 52 corrects virtual image 22 based on the control result of analysis result control unit 51. Then, image synthesis unit 15 generates composite image 61 by synthesizing the corrected virtual image with the corrected real image.

[0043] The analysis result control unit 51 determines that the moon 210, clouds 211, and face 212 in the real image should be corrected, and the real image correction unit 14 slightly increases the brightness of the distant moon 210 and clouds 211 and greatly increases the brightness of the nearby face 213, as in the second embodiment. The analysis result control unit 51 also determines that the virtual image should be corrected, and the virtual image correction unit 52 adds a shadow 611 of the face 213 and light 612 leaking from the clouds 211 and reduces the brightness of the flash of light 221. The image synthesis unit 15 then synthesizes a virtual image including the shadow 611, light 612, and flash of light 613 with reduced brightness with a real image including the moon 410, clouds 411, and face 412 with increased brightness. This makes it possible to express how shadows and light are created by the flash of light 221, and how the brightness of the flash of light 221 changes depending on the situation (scene) in real space. As a result, it is possible to further reduce the sense of strangeness felt in mixed (or augmented) reality (make the mixed (or augmented) reality closer to reality).

[0044] As described above, according to the third embodiment, not only the real image but also the virtual image is corrected based on the results of analyzing the virtual image and the results of analyzing the real image. This makes it possible to further reduce the sense of incongruity felt in mixed (or augmented) reality (make the mixed reality or augmented reality closer to reality) while suppressing a significant increase in the amount of processing by the computer.

[0045] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0046] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0047] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0048] At least one of real image corrector 14 and virtual image corrector 52 may change the amount of correction depending on the angle of view of the display image displayed on image display unit 16. For example, if the angle of view of the display image is wide, the brightness of the display image may be gradually increased or decreased from the center to the edges so that the brightness of the edges of the display image approaches the brightness of the surrounding environment. This can further reduce the sense of incongruity in the mixed (or augmented) reality (make the mixed (or augmented) reality closer to reality). The angle of view of the display image may be equal to or narrower than the angle of view of the composite image. In other words, the display image may be a composite image or a part of a composite image.

[0049] At least one of the real image correction unit 14 and the virtual image correction unit 52 may change the amount of correction depending on the gaze position of the user viewing the displayed image. For example, fine correction may be performed in areas close to the gaze position, and coarse correction may be performed in areas far from the gaze position. This can reduce the amount of computer processing while reducing the sense of incongruity in the mixed (or augmented) reality (while making the mixed (or augmented) reality closer to reality). The method for detecting the gaze position is not particularly limited. The gaze sensor that detects the gaze position may or may not be provided in the information processing device to which the present invention is applied. It is sufficient to be able to acquire gaze information related to the gaze position; for example, the gaze sensor may be an external device to the information processing device, and the information processing device may have an input interface that acquires gaze information from outside.

[0050] The virtual object is not limited to a flash of light, and the correction of the real image and the virtual image (the effect of the virtual object on the real image) is not limited to the correction described above. For example, a virtual object of fire may be used. In that case, the real image may be corrected by changing the area around the fire to express a heat haze. Blurring correction may be performed. A virtual car object may be used. In this case, correction of the real image may be performed by distorting the real image to express the sense of speed of the car. The virtual object is not particularly limited, and correction of the real image and virtual image may vary depending on the virtual image (such as the type of virtual object) and the real image.

[0051] Furthermore, in the above-described embodiment, the present invention has been described as being applied to a display device, but this is not limited to this example and can also be applied to other information processing devices such as personal computers and server devices.

[0052] (Other embodiments) The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0053] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first acquisition means for acquiring a real image that is an image of real space; a second acquisition means for acquiring a virtual image, which is an image of a virtual object, to be composited with the real image; analysis means for analyzing the virtual image; a correcting means for correcting the real image based on the result of the analysis by the analyzing means; An information processing device comprising: (Configuration 2) The first acquisition means is an imaging means for imaging the real space. 2. The information processing device according to configuration 1, (Configuration 3) A synthesis means for synthesizing the virtual image with the image corrected by the correction means. Further having 3. The information processing device according to configuration 1 or 2. (Configuration 4) a display means for displaying the image after the synthesis by the synthesis means; Further having 4. The information processing device according to configuration 3. (Configuration 5) The correction means corrects at least one of the luminance, chromaticity, gamma characteristics, noise, distortion, position, and resolution of the real image. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) a second analysis means for analyzing the real image; and The correcting means corrects the real image based on the result of the analysis by the analyzing means and the result of the analysis by the second analyzing means. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The result of the analysis by the second analysis means includes at least one of depth information and division information of the real image. 7. The information processing device according to configuration 6. (Configuration 8) The correction means corrects the result of the analysis by the analysis means and the result of the analysis by the second analysis means. and correcting the real image and the virtual image based on the 8. The information processing device according to configuration 6 or 7. (Configuration 9) The correction of the virtual image includes a correction of adding a virtual object. 9. The information processing device according to configuration 8. (Configuration 10) The analysis means analyzes metadata of the virtual image. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The analysis means analyzes pre-rendering data of the virtual image. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) The analysis means analyzes the rendered data of the virtual image. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) a display image obtained by combining the virtual image with the image corrected by the correction means is displayed on a display means; The correction means changes the amount of correction depending on the angle of view of the displayed image. 13. The information processing device according to any one of configurations 1 to 12. (Configuration 14) a display image obtained by combining the virtual image with the image corrected by the correction means is displayed on a display means; The correction means changes the amount of correction depending on the line of sight of the user viewing the displayed image. 14. The information processing device according to any one of configurations 1 to 13. (method) a first acquisition step of acquiring a real image which is an image of real space; a second acquisition step of acquiring a virtual image, which is an image of a virtual object, to be composited with the real image; an analysis step of analyzing the virtual image; a correcting step of correcting the real image based on the result of the analysis in the analyzing step; 1. A method for controlling an information processing device, comprising: (program) A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0054] 11: Real image capturing unit 12: Virtual image generating unit 13: Virtual image analysis unit 14: Real image correction unit

Claims

1. a first acquisition means for acquiring a real image which is an image of real space; a second acquisition means for acquiring a virtual image, which is an image of a virtual object, to be composited with the real image; analysis means for analyzing the virtual image; a correcting means for correcting the real image based on the result of the analysis by the analyzing means; An information processing device comprising:

2. The first acquisition means is an imaging means for imaging the real space.

2. The information processing apparatus according to claim 1, wherein:

3. A synthesis means for synthesizing the virtual image with the image corrected by the correction means. Further having 2. The information processing apparatus according to claim 1, wherein:

4. a display means for displaying the image after the synthesis by the synthesis means; Further having 4. The information processing apparatus according to claim 3,

5. The correction means corrects at least one of the luminance, chromaticity, gamma characteristics, noise, distortion, position, and resolution of the real image.

2. The information processing apparatus according to claim 1, wherein:

6. a second analysis means for analyzing the real image; and The correcting means corrects the real image based on the result of the analysis by the analyzing means and the result of the analysis by the second analyzing means.

2. The information processing apparatus according to claim 1, wherein:

7. The result of the analysis by the second analysis means includes at least one of depth information and division information of the real image.

7. The information processing apparatus according to claim 6,

8. The correction means corrects the real image and the virtual image based on the result of the analysis by the analysis means and the result of the analysis by the second analysis means.

7. The information processing apparatus according to claim 6,

9. The correction of the virtual image includes a correction of adding a virtual object.

9. The information processing apparatus according to claim 8,

10. The analysis means analyzes metadata of the virtual image.

2. The information processing apparatus according to claim 1, wherein:

11. The analysis means analyzes pre-rendering data of the virtual image.

2. The information processing apparatus according to claim 1, wherein:

12. The analysis means analyzes the rendered data of the virtual image.

2. The information processing apparatus according to claim 1, wherein:

13. a display image obtained by combining the virtual image with the image corrected by the correction means is displayed on a display means; The correction means changes the amount of correction depending on the angle of view of the displayed image.

2. The information processing apparatus according to claim 1, wherein:

14. a display image obtained by combining the virtual image with the image corrected by the correction means is displayed on a display means; The correction means changes the amount of correction depending on the line of sight of the user viewing the displayed image.

2. The information processing apparatus according to claim 1, wherein:

15. a first acquisition step of acquiring a real image which is an image of real space; a second acquisition step of acquiring a virtual image, which is an image of a virtual object, to be composited with the real image; an analysis step of analyzing the virtual image; a correcting step of correcting the real image based on the result of the analysis in the analyzing step; 1. A method for controlling an information processing device, comprising:

16. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 14.

Images