Information processor, information processing method, and program

Abstract

To provide an information processor that can provide a user with higher quality and higher resolution, while enabling a compact, lightweight, and a power-saving configuration, and further enabling reduction in an amount of transmission and an amount of processing.SOLUTION: An information processor acquires a first image imaging a real world at a narrower angle of view than an image presented by a presentation device, and a second image imaging the real world at a wider angle of view than the first image, and generates, on the basis of the first and second images, a background image. Then, the information processor generates a composite image by combining the background image with a computer graphic image, and presents the composite image on the presentation device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for generating an image to be presented to a user based on a captured image. [Background technology]

[0002] In recent years, so-called Augmented Reality (AR) technology and Mixed Reality (MR) technology have been known as technologies for seamlessly fusing real space and virtual space in real time. In the following description, MR will be taken as an example. As one of these technologies, a technology that uses a video see-through head mounted display (HMD) is known. In a video see-through HMD, a real world that approximately matches what is observed from the pupil position of the HMD user is captured by a video camera or the like, and computer graphics (CG) are superimposed on the captured image and displayed on the HMD.

[0003] In order to provide HMD users with a higher quality MR experience, the angle of view and number of pixels of the images displayed on the HMD are steadily increasing, and the angle of view and number of pixels of CG and captured images are also increasing accordingly. However, for example, increasing the angle of view often requires larger configurations of the imaging unit and display unit, which makes it difficult to make the HMD smaller and lighter. Although an increase in the number of pixels contributes to an improvement in the sense of resolution when HMD users experience MR, it increases the transmission and processing load in the system.

[0004] In response to this, Patent Document 1 discloses a technology that can reduce the amount of data transmitted as a system by dividing an area of ​​an image captured by an imaging unit, reducing the resolution before transmitting the image, and then combining the images again after transmission. However, although the technology described in Patent Document 1 can reduce the amount of data transmitted as a system, the imaging unit has the same angle of view and resolution as the display unit. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-96772 Summary of the Invention [Problem to be solved by the invention]

[0006] As mentioned above, in order to provide HMD users with an MR experience that provides higher quality and a higher resolution, it is necessary for not only the images displayed but also the captured images to have a wide angle of view and high resolution. However, imaging units capable of capturing images with a wide angle of view tend to have large, complex, and heavy optical systems, and image sensors capable of capturing high-resolution images tend to be large in size and consume large amounts of power. Furthermore, when dealing with wide-angle, high-resolution images, i.e. images with a large number of pixels, the amount of transmission and processing also increases.

[0007] Therefore, an object of the present invention is to provide a user with a higher quality and higher resolution image while enabling a compact, lightweight, and power-saving configuration, and further enabling a reduction in the amount of transmission and processing. [Means for solving the problem]

[0008] The information processing device of the present invention has an acquisition means for acquiring a first image capturing the real world at a narrower angle of view than an image presented by a presentation device, and a second image capturing the real world at a wider angle of view than the first image, a background generation means for generating a background image based on the first image and the second image, and a CG synthesis means for generating a composite image by synthesizing the background image with a computer graphic image, and is characterized in that the composite image is presented on the presentation device. Effect of the Invention

[0009] According to the present invention, it is possible to provide a user with a higher quality and a higher resolution image, while enabling a compact, lightweight configuration and reduced power consumption, and further reducing the amount of transmission and processing. [Brief description of the drawings]

[0010] [Figure 1] 2 is a diagram illustrating an example of the configuration of an imaging unit and a processing unit according to the first embodiment. FIG. [Diagram 2] 3A to 3C are diagrams illustrating a flowchart and example images according to the first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of the configuration of an imaging unit and a processing unit according to a general configuration. [Figure 4] FIG. 1 shows a flow chart and example images of a general configuration. [Diagram 5] 13 is a diagram illustrating an example of the configuration of an imaging unit and a processing unit according to a second embodiment. FIG. [Figure 6] 13A to 13C are diagrams illustrating a flowchart and example images according to the second embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of the configuration of an imaging unit and a processing unit according to a third embodiment. [Figure 8] 13A to 13C are diagrams illustrating a flowchart and example images according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the embodiment may be configured by appropriately combining parts of each embodiment described below. In the following embodiments, the same configurations are described with the same reference symbols.

[0012] FIG. 1 is a diagram showing a schematic configuration of information processing and the like according to the first embodiment, and is a system configuration including an HMD (Head Mounted Display) equipped with an imaging unit 11, an imaging unit 12, and a display unit 14, and a processing unit 13. As an example, the system according to this embodiment is an MR system for providing a user with mixed reality (MR). In this embodiment, an example is described in which an HMD worn by a user on the head is used to provide MR to the user (hereinafter referred to as an HMD user). In this embodiment, the HMD is a so-called video see-through type HMD. Note that only the main parts are shown in FIG. 1, and illustrations of other components that are generally provided in an MR system are omitted. Also, FIG. 2 is a diagram showing a flowchart showing a processing flow in the processing unit 13 according to the first embodiment and an example of an image.

[0013] Fig. 3 is a diagram showing a configuration example of a general MR system for comparison with the MR system of this embodiment shown in Fig. 1. Fig. 4 is a flowchart showing the flow of processing in the processing unit 33 of the MR system shown in Fig. 3 and a diagram showing an example of an image. Before describing the configuration of the MR system according to the first embodiment in FIG. 1 and the flowchart in FIG. 2, the configuration and processing of a general MR system will be described below with reference to FIGS. 3 and 4. FIG.

[0014] 3, the imaging unit 31 is an imaging device having an objective optical system 311 and an image sensor 312. The objective optical system 311 of the imaging unit 31 takes in light of the external real world and forms an optical image of the real world on the image sensor 312. The image sensor 312 converts the optical image of the real world formed by the objective optical system 311 into an electrical signal to generate an imaging signal (RAW data) for each frame of a moving image. The imaging signal obtained by the imaging unit 31 is sent to a processing unit 33.

[0015] The imaging unit 32 is an imaging device having an objective optical system 321 and an image sensor 322. The objective optical system 321 of the imaging unit 32, like the imaging unit 31, captures light from the external real world and forms an optical image of the real world on the image sensor 322. The image sensor 322 converts the optical image of the real world formed by the objective optical system 321 into an electrical signal to generate an imaging signal (RAW data) for each frame of a video. The imaging signal obtained by this imaging unit 32 is sent to the processing unit 33.

[0016] The processing unit 33 uses an imaging signal (captured image) captured from the real world by the imaging unit 31 for image information calculations that estimate at least the position and orientation of the HMD, that is, the position and orientation of the head of a user wearing the HMD. The captured image captured from the real world by the imaging unit 31 is also used for image information calculations such as calculations to detect the skeleton of the hand. Here, it is desirable that the image used for position and orientation estimation calculations and the like be a wide-angle image captured with as wide an angle of view as possible, so the objective optical system 311 of the imaging unit 31 is a wide-angle optical system.

[0017] Furthermore, the processing unit 33 generates a display image in the video see-through type HMD using an imaging signal (captured image) obtained by capturing an image of the real world by the imaging unit 32. That is, the processing unit 33 generates an image observed by the HMD user from the captured image obtained by the imaging unit 32. Here, it is desirable that the image observed by the HMD user be a high-resolution image so that an MR experience with higher quality and resolution can be provided to the HMD user. Therefore, the image sensor 322 of the imaging unit 32 is capable of capturing images with a higher resolution than the image sensor 322 of the imaging unit 31.

[0018] The processing section 33 in FIG. 3 is configured to include an image processing section 331 , an image processing section 332 , a calculation section 333 , a CG generation section 334 , and a CG synthesis section 335 . The image processing unit 331 performs development processing, image quality adjustment processing, etc. of RAW data of a captured image acquired by the image sensor 312 of the imaging unit 31. Similarly, the image processing unit 332 performs development processing, image quality adjustment processing, etc. of RAW data of a captured image acquired by the image sensor 322 of the imaging unit 32. These image processes are existing processes, so detailed description thereof will be omitted.

[0019] The calculation unit 333 performs image information calculations called computer vision calculations, such as calculations to estimate the position and orientation of the HMD and detection processing of bones, based on the images obtained by the imaging unit 31 and processed by the image processing unit 331. The CG generating unit 334 performs rendering processing of a CG image (computer graphic image) based on the calculation result by the calculation unit 333. The CG synthesis unit 335 synthesizes the CG image generated by the CG generation unit 334 with the image captured by the imaging unit 32 and processed by the image processing unit 332. In other words, the image of the real world captured by the imaging unit 32 and processed by the image processing unit 332 is used as a background image on which the CG image generated by the CG generation unit 334 is superimposed. The synthesized image in which the CG image is superimposed on the background image of the real world by the CG synthesis unit 135 is sent to the display unit 34.

[0020] The display unit 34 includes an eyepiece optical system 341 and a display 342, and is a presentation device for presenting an image to the HMD user. The display 342 of the display unit 34 is a display device that converts the electrical signal of the composite image into image light. The eyepiece optical system 341 is an optical system that delivers image light output from the display 342 to the eyes of the HMD user.

[0021] 3, the processing unit 33 generates a CG image corresponding to the position and orientation of the HMD based on the captured image obtained by the imaging unit 31, and generates a composite image by combining the CG image with the captured image (image of the outside world seen by the HMD user) obtained by the imaging unit 32. The composite image is then sent to the display unit 34 and displayed. The HMD user can experience mixed reality by viewing an image based on the composite image, that is, an MR image on which a CG image that does not exist in the real world is superimposed.

[0022] Next, the processing of the processing unit 33 in Fig. 3 will be described with reference to the flowchart in Fig. 4. In each of the subsequent flowcharts, the reference character S indicates a processing step (process). In the MR system of Fig. 3, an image 21 in Fig. 4 is a schematic representation of a wide-angle image captured of the real world by an imaging unit 31 equipped with a wide-angle objective optical system 311. As the process of S401, the processing unit 33 acquires the wide-angle image 21 captured by the imaging unit 31. Then, the wide-angle image 21 is sent to an image processing unit 331. 4 is a schematic representation of an image captured by the imaging unit 32 equipped with a high-resolution image sensor 322 of the real world, that is, a high-resolution image having a higher resolution than the wide-angle image 21. The processing unit 33 acquires the high-resolution image 42 captured by the imaging unit 32 as the process of S402. The high-resolution image 42 is then sent to the image processing unit 332.

[0023] The image processing unit 331 performs development processing, image quality adjustment processing, and the like on the RAW data of the wide-angle image 21 as the processing of S403. Similarly, the image processing unit 332 performs development processing, image quality adjustment processing, and the like on the RAW data of the high resolution image 42 as the processing of S404.

[0024] Next, in step S405, the calculation unit 333 performs image information calculations such as estimating the position and orientation of the HMD and detecting the skeleton, based on the wide-angle image 21 processed by the image processing unit 331. Next, the CG generation unit 334 performs rendering processing of the CG image based on the calculation result in the calculation unit 333 as processing in S406.

[0025] Then, in the process of S408, the CG synthesis unit 335 uses the high-resolution image after processing by the image processing unit 332 as a background image, and generates a synthesized image by synthesizing the background image with the CG image generated by the CG generation unit 334. Image 43 in Fig. 4 represents the background image made of the high-resolution image after processing by the image processing unit 332, and the CG synthesis unit 335 generates the synthesized image by superimposing the CG image on this background image 43. Thereafter, the processing unit 33 outputs the composite image to the display unit 34 as processing in S409.

[0026] As described above, in the MR system shown in FIG. 3, a CG image is generated based on the position and orientation of the HMD calculated from the wide-angle image 21 captured by the imaging unit 31, and the CG image is synthesized with a background image based on the high-resolution image 42 captured by the imaging unit 32.

[0027] In the MR system, in order to provide the HMD user with a higher quality MR experience, the angle of view and the number of pixels of the image displayed on the HMD are steadily increasing, as described above, and the angle of view and the number of pixels of the CG and captured images are also increasing accordingly. In other words, the image displayed on the display unit 34 based on the composite image is required to have a wide angle of view and high resolution, and therefore the high-resolution image 42 captured by the imaging unit 32 is also designed to have a wide angle of view and high resolution.

[0028] However, as described above, a relatively large wide-angle objective optical system is required to capture a wide-angle captured image, and an image sensor is also required to capture a high-resolution image. In particular, the imaging unit 32 capturing a wide-angle high-resolution image 42 has a large, complex, and heavy objective optical system 321, and the image sensor 322 has a large size and consumes large power, making it difficult to make the HMD small and lightweight and energy-efficient. In addition, the high-resolution image 42 captured by the imaging unit 32 has a large number of pixels, which increases the transmission and processing load in the MR system.

[0029] <First embodiment> Therefore, the MR system of this embodiment has the configuration shown in Figure 1 and performs the processing shown in Figure 2, making it possible to provide HMD users with a high-quality, high-resolution MR experience, while also making the HMD smaller, lighter, and more energy-efficient, and further reducing the amount of transmission and processing. The configuration of the MR system in FIG. 1 according to the first embodiment and the process in the flowchart in FIG. 2 will be described below. As shown in Fig. 1, the information processing device of the first embodiment is an MR system including an HMD having an imaging unit 11, an imaging unit 12 and a display unit 14, and a processing unit 13. Note that Fig. 1 shows only the main parts, and does not show other components that are usually included in an MR system.

[0030] In the MR system of the present embodiment, the imaging unit 11, the imaging unit 12, and the display unit 14 are mounted in the HMD, and the processing unit 13 is realized by a personal computer or a tablet terminal connected to the HMD so as to be able to communicate with the HMD. When the imaging unit 11, the imaging unit 12, and the display unit 14 are mounted in the HMD and the processing unit 13 is configured separately, as in the MR system of the present embodiment, communication between the HMD and the processing unit 13 is performed, for example, wirelessly. For wireless communication, a small-scale network such as a WLAN (Wireless Local Area Network) or a WPAN (Wireless Personal Area Network) is used. Note that communication between the HMD and the processing unit 13 is not limited to wireless communication, and may be wired communication. In addition, in the present embodiment, an example is given in which the HMD and the processing unit 13 are configured as separate hardware, but all of the functions of the processing unit 13 may be implemented and integrated in the HMD.

[0031] The imaging unit 12 is an imaging device having an objective optical system 121 and an image sensor 122. The objective optical system 121 of the imaging unit 12 takes in light of the external real world and forms an optical image of the real world on the image sensor 122. The image sensor 122 converts the optical image of the real world formed by the objective optical system 121 into an electrical signal to generate a first image (imaging signal) for each frame of a video. Similarly, the imaging unit 11 is an imaging device having an objective optical system 111 and an image sensor 112. The objective optical system 111 of the imaging unit 11 captures light from the external real world and forms an optical image of the real world on the image sensor 112. The image sensor 112 converts the optical image of the real world formed by the objective optical system 111 into an electrical signal to generate a second image (imaging signal) for each frame of the video.

[0032] In this embodiment, the second image acquired by the imaging unit 11 is used in a processing unit 13 described later for image information calculations such as estimating the position and orientation of the HMD, that is, the position and orientation of the head of the HMD user. Also, in this embodiment, the first image acquired by the imaging unit 12 and the second image acquired by the imaging unit 12 are used in the processing unit 13 described later for generating a background image observed by the user of the video see-through HMD.

[0033] Here, it is desirable that the image used for image information calculation such as estimating the position and orientation of the HMD is a wide-angle image captured in the real world with as wide an angle of view as possible, but the resolution does not need to be very high. In the case of this embodiment, since the second image obtained by the imaging unit 11 is used for calculation such as estimating the position and orientation of the HMD, the angle of view of the imaging unit 11 is wider than that of the image displayed on the display unit 14, but the resolution is not so high. Note that, since it is desirable for the image used for calculation such as estimating the position and orientation of the HMD to have little motion distortion, the image sensor 112 of the imaging unit 11 is a so-called global shutter type image sensor.

[0034] On the other hand, it is desirable for the background image observed by the user of a video see-through HMD to have a high resolution in order to provide the HMD user with a high-quality, high-resolution MR experience. However, the human viewing angle characteristics provide high resolution for the part being gazed at, but not so high resolution for the part not being gazed at. Also, the human visual characteristics provide high discrimination ability for brightness and color for the part being gazed at, but not so high discrimination ability for the part not being gazed at. In other words, it is desirable for the background image observed by the user of a video see-through HMD to have a high resolution in the center where the HMD user is likely to gaze, while it is considered that the resolution does not need to be so high in the peripheral parts.

[0035] For this reason, in the present embodiment, the second image, which is a wide-angle image but has a low resolution, is used in the peripheral portion of the background image. In the present embodiment, the imaging unit 12 is capable of acquiring an image with a narrower angle of view and fewer pixels than the image displayed on the display unit 14, and with a higher resolution than the imaging unit 11, and the first image acquired by the imaging unit 12 is used in the center of the background image. In this manner, in the present embodiment, the first image with a narrow angle of view and high resolution captured by the imaging unit 12 of the real world is used in the central portion of the background image, and the second image with a wide angle of view and low resolution captured by the imaging unit 11 of the real world is used in the peripheral portion of the background image.

[0036] In addition to the objective optical system 111 and the image sensor 112, the imaging unit 11 also includes various components that are included in a general imaging device, but illustration and description of these components are omitted. The imaging unit 11 may have the objective optical system 111 and the image sensor 112 arranged as separate components, or may be configured as a camera module in which they are integrated. Furthermore, in the case of FIG. 1, only one imaging unit 11 is shown, but for example, two imaging units may be used to configure a stereo camera, and more imaging units may be prepared to widen the angle of view. A signal (RAW data) of the second image obtained by this imaging unit 11 is sent to the processing unit 13. Similarly, the imaging unit 12 includes various components that are included in a general imaging device in addition to the objective optical system 121 and the image sensor 122, but illustration and description of those components are omitted. Furthermore, the imaging unit 12 may include the objective optical system 121 and the image sensor 122 as separate components, or may be configured as a camera module in which they are integrated. Furthermore, although only one imaging unit 12 is shown in FIG. 1, the imaging unit 12 may be configured as two cameras corresponding to the left and right eyes of a human, for example. A signal (RAW data) of the first image obtained by this imaging unit 12 is sent to the processing unit 13.

[0037] The processing unit 13 estimates the position and orientation of the head of the HMD user using the second image, and generates a background image observed by the user of the video see-through HMD using the first and second images. The configuration and processing of the processing unit 13 will be described in detail later.

[0038] The display unit 14 includes an eyepiece optical system 141 and a display 142, and is a presentation device for visually presenting an image to an HMD user. The display 142 of the display unit 14 is a display device that converts an electrical signal of a composite image generated by the processing unit 13 described later into image light. The display 142 is assumed to be a flat-panel image display element such as an organic EL display or a liquid crystal display. The eyepiece optical system 141 is an optical system that delivers image light output from the display 142 to the eyes of the HMD user. Although only one display unit 14 is shown in Fig. 1, the display unit 14 may be configured with two display units corresponding to the left and right eyes of a human, for example. Alternatively, the display unit 14 may be, for example, a presentation device of a retinal scan type using MEMS (Micro Electro Mechanical Systems).

[0039] The configuration and processing of the processing unit 13 of this embodiment will be described below. The processing unit 13 is configured to include an image processing unit 131 , an image processing unit 132 , a calculation unit 133 , a CG generation unit 134 , a background synthesis unit 136 , and a CG synthesis unit 135 . The image processing unit 131 performs development processing and image quality adjustment processing of the RAW data of the second image obtained by capturing the real world with the image sensor 112 of the imaging unit 11. Similarly, the image processing unit 132 performs development processing and image quality adjustment processing of the RAW data of the first image captured by the image sensor 122 of the imaging unit 12. Since the images captured by the imaging units 11 and 12 have partially different purposes of use, the image processing of the image processing units 131 and 132 also differs appropriately. Since the image processing according to the purpose of use of these images is existing processing, detailed description thereof will be omitted. The second image after image processing by the image processing unit 131 is sent to the calculation unit 133 and the background synthesis unit 136, and the first image after image processing by the image processing unit 132 is sent to the background synthesis unit 136.

[0040] The calculation unit 133 performs image information calculation, also called computer vision calculation, which is not directly observed by the user, based on the second image after image processing by the image processing unit 131. That is, the calculation unit 133 performs image information calculation such as estimating the position and orientation of the HMD, that is, the position and orientation of the head of the HMD user, and detecting the skeleton of the hand, based on the second image after image processing by the image processing unit 131. Note that, in addition to or in addition to a method using an image captured from the real world, there is also a method of calculating using various sensor outputs such as acceleration and angular velocity for the estimation of the position and orientation of the HMD (the position and orientation of the head of the HMD user). In the case of this embodiment, any of these methods may be used. Since these methods are also existing methods, detailed explanations thereof will be omitted.

[0041] The background synthesis unit 136 is a background generation unit that generates a background image based on a first image captured by the imaging unit 12 and processed by the image processing unit 132, and a second image captured by the imaging unit 11 and processed by the image processing unit 131. As described above, in this embodiment, the second image acquired by the imaging unit 11 is a low-resolution wide-angle image, while the first image acquired by the imaging unit 12 is a narrow-angle high-resolution image. The background synthesis unit 136 of this embodiment generates a background image using the first image, which is a high-resolution image with a narrow angle of view, in the center, and the second image, which is a low-resolution wide-angle image, in the periphery. The background image generated by the background synthesis unit 136 is then sent to the CG synthesis unit 135.

[0042] In this embodiment, the background synthesis unit 136 generates a background image from a first image captured by the imaging unit 12 and a second image captured by the imaging unit 11, but the imaging units 11 and 12 are located at positions that are physically separated on the HMD. When using two images captured by the imaging units 11 and 12 that are located at positions that are physically separated in this way, a process is required to align the positions (e.g., center positions) of the two images. For this reason, the background synthesis unit 136 also performs a process to align the center positions of the images obtained by the imaging units 11 and 12.

[0043] In addition, the background synthesis unit 136 generates a background image by synthesizing the second image captured by the imaging unit 11 for the peripheral portion and the first image captured by the imaging unit 12 for the central portion. In this way, when synthesizing the images obtained by the respective imaging units, a process for joining the images and a process for making the seams between the images less noticeable are required. For this reason, the background synthesis unit 136 also performs a stitching process for joining the images, a deformation process for appropriately deforming the shapes of the subjects, etc., so that the subjects, etc., shown in the images are joined, and a correction process such as a smoothing process for the seams between the images.

[0044] Furthermore, the imaging units 11 and 12 capture light images that have passed through objective optical systems 111 and 112, which have different angles of view, to obtain images. When using images obtained at different angles of view in this way, processing is required to match the angles of view of the two images. For this reason, the background synthesis unit 136 performs angle of view correction processing according to the difference in magnification (difference in focal length) between the two objective optical systems 111 and 112 of the imaging units 11 and 12, to match the two angles of view.

[0045] Furthermore, the imaging units 11 and 12 acquire images from light images that have passed through the objective optical systems 111 and 112, which have different optical performances such as distortion aberration. When using images acquired through the objective optical systems 111 and 112 with different optical performances, a process for correcting the difference in the two optical performances is required. For this reason, the background synthesis unit 136 also performs an optical performance correction process to correct the difference in the different optical performances.

[0046] Moreover, the imaging unit 11 includes an image sensor 112 made of a semiconductor, and similarly, the imaging unit 12 includes an image sensor 122. There are often differences in performance between the semiconductors used in these image sensors 112 and 122, and the differences in performance between these semiconductors often result in differences in image quality, such as brightness, color, and noise, in the captured images. Therefore, when using images obtained by these image sensors 112 and 122, an image quality difference correction process is required to correct the difference in image quality due to the difference in performance between these image sensors 112 and 122. For this reason, the background synthesis unit 136 also performs a process to correct the difference in image quality due to these image sensors 112 and 122, that is, an image quality difference correction process to correct the difference in brightness, color, and noise. The angle of view correction process, the optical performance correction process, and the image quality difference correction process may be performed by either one or both of the image processing units 131 and 132.

[0047] The CG generating unit 134 performs rendering processing of the CG image based on the calculation result by the calculation unit 133. The CG synthesis unit 135 synthesizes the CG image generated by the CG generation unit 134 with the background image generated as described above by the background synthesis unit 136. That is, the CG synthesis unit 135 generates a synthetic image by superimposing the CG image rendered by the CG generation unit 134 on the background image of the real world generated by the background synthesis unit 136. The synthetic image generated by the CG generation unit 134 is then sent to the display unit 14.

[0048] Next, the processing of the processing unit 13 according to the first embodiment will be described with reference to FIG. In the MR system of the first embodiment shown in Fig. 1, an image 21 in Fig. 2 is a schematic representation of a low-resolution wide-angle image that is a second image captured by the imaging unit 11. The processing unit 13 acquires the wide-angle image 21 captured by the imaging unit 11 as processing in S201. The wide-angle image 21 is then sent to the image processing unit 131. 2 is a schematic representation of a high-resolution image 22 with a narrow angle of view, which is a first image captured by the imaging unit 12. As the process of S202, the processing unit 13 acquires the high-resolution image 22 with a narrow angle of view captured by the imaging unit 12. Then, the high-resolution image 22 is sent to the image processing unit 132.

[0049] As the process of S203, the image processing unit 131 performs development processing, image quality adjustment processing, and the like on the RAW data of the low-resolution wide-angle image 21. Similarly, the image processing unit 132 performs development processing, image quality adjustment processing, and the like on the RAW data of the narrow-angle high-resolution image 22 as the processing of S204.

[0050] Next, in step S205, the calculation unit 133 performs image information calculations such as estimation of the position and orientation of the HMD and detection of the skeleton, based on the wide-angle image 21 processed by the image processing unit 131. Next, in step S206, the CG generating unit 134 performs rendering processing of the CG image based on the image information calculation result from the calculation unit 133.

[0051] Furthermore, in the process of S207, the background synthesis unit 136 synthesizes a background image in which the wide-angle image 21 after the process of S203 is used as the peripheral part and the high-resolution image 22 after the process of S204 is used as the central part.

[0052] Then, in the process of S208, the CG synthesis unit 135 generates a synthetic image by synthesizing the background image generated by the background synthesis unit 136 in S207 and the CG image generated by the CG generation unit 334 in S206. The image 23 in Fig. 2 represents the background image generated by the background synthesis unit 136, and the CG synthesis unit 135 generates the synthetic image by superimposing the CG image on this background image 23. Thereafter, the processing unit 13 outputs the composite image to the display unit 14 as processing in S209.

[0053] As described above, the processing unit 13 of the first embodiment generates a background image based on the second image, which is a low-resolution wide-angle image captured by the imaging unit 11 of the real world, and the first image, which is a high-resolution image with a narrow angle of view captured by the imaging unit 12 of the real world. Then, the processing unit 13 generates a composite image by superimposing a CG image on the background image. In particular, in the case of this embodiment, the high-resolution image with a narrow angle of view acquired by the imaging unit 12 is used in the center, and the low-resolution wide-angle image acquired by the imaging unit 11 is used in the peripheral part to generate a background image. That is, in this embodiment, since the high-resolution image with a narrow angle of view is used in the center of the background image, the HMD user can view MR images with high visual quality and a high resolution. Furthermore, according to this embodiment, since the imaging unit 12, which captures particularly high-resolution images, has a narrow angle of view, it is possible to realize a small, lightweight, and power-saving HMD, and further reduce the transmission amount and processing amount.

[0054] <Second embodiment> Fig. 5 shows a configuration example of an MR system according to the second embodiment, and Fig. 6 shows a flowchart showing the flow of processing in a processing unit 53 of the second embodiment and example images. In Fig. 5 and Fig. 6, the same functional units as those in Fig. 1 of the first embodiment and the same processing steps as those in Fig. 2 are given the same reference numerals as those in Fig. 1 and Fig. 2, and redundant explanations will be omitted.

[0055] As shown in FIG. 5, the MR system of the second embodiment includes an HMD including the image capturing unit 11, the image capturing unit 12, and the display unit 14 similar to those in the example of FIG. As shown in FIG. 5, a processing unit 53 of the second embodiment does not include the background synthesis unit 136 of the processing unit 13 of FIG. 1, and also includes a CG / surrounding background generation unit 534 instead of the CG generation unit 134 of FIG.

[0056] In the second embodiment, the first image (a high-resolution image with a narrow angle of view) obtained by capturing an image of the real world by the imaging unit 12 and processing by the image processing unit 132 is sent to the CG synthesis unit 135. In the second embodiment, the second image (low-resolution wide-angle image) obtained after the imaging unit 11 captures an image of the real world and the image processing unit 131 processes the image is sent only to the calculation unit 133. The calculation unit 133 performs image information calculations such as estimating the position and orientation of the HMD and detecting the skeleton, as described above. In the second embodiment, the calculation unit 133 also sends the wide-angle image, which is the second image after processing in the image processing unit 131, to the CG and surrounding background generation unit 534 together with the image information calculation results.

[0057] The CG / surrounding background generating unit 534 generates a CG image in the same manner as the CG generating unit 134 described above, based on the image information calculation result by the calculation unit 133. Furthermore, the CG / surrounding background generating unit 534 cuts out the wide-angle image sent via the calculation unit 133 to fit the angle of view of the display unit 14, and generates an image with the center of the cut-out image made transparent. In the second embodiment, the wide-angle image cut out to fit the angle of view of the display unit 14 and with the center made transparent is used as a background image of the periphery (herein referred to as a peripheral background image). The CG / surrounding background generating unit 534 then sends the generated CG image and the peripheral background image cut out to fit the angle of view of the display unit 14 and with the center made transparent to the CG synthesis unit 135.

[0058] In the case of the second embodiment, the CG synthesis unit 135 synthesizes a background image in which the first image (high-resolution image with a narrow angle of view) after processing by the image processing unit 132 is the central part, and the peripheral background image from the CG and surrounding background generation unit 534 is the peripheral part. Furthermore, the CG synthesis unit 135 synthesizes a CG image with the background image to generate a synthesized image. Note that the CG image generated by the CG and surrounding background generation unit 534 is synthesized with the background image by the CG synthesis unit 135, and at that time, synthesis is often performed using a chromakey or an alpha channel. In this case, since the CG image is synthesized as a layer above the high-resolution image by the imaging unit 12, the CG synthesis unit 135 outputs the peripheral background image together with the CG image at the same time, so that the image displayed on the display unit 14 becomes the same image as that of the first embodiment described above.

[0059] Next, the flow of processing in the processing unit 53 of the second embodiment will be described with reference to FIG. In the second embodiment MR system shown in Figure 5, image 21 in Figure 6 is a wide-angle image 21 (second image) similar to that in Figure 2, and image 22 in Figure 6 is a high-resolution image 22 (first image) similar to that in Figure 2.

[0060] In the second embodiment, after the image information calculation in S205, a CG image and a surrounding background image are generated by the CG / surrounding background generation unit 534 as processing in S606. Image 54 in Fig. 6 is a schematic representation of a surrounding background image obtained by cutting out wide-angle image 21 to fit the angle of view of display unit 14 and making the central portion of the cut-out image transparent. Then, in the second embodiment, after the processing in S606, the process proceeds to the processing in S208.

[0061] In S208, the CG synthesis unit 135 generates a background image such as image 23 from the surrounding background image generated in S606 and the high-resolution image after image processing in S204, and generates a composite image by combining the background image with the CG image. Then, the processing unit 53 outputs the composite image to the display unit 14 as processing in S209.

[0062] In the second embodiment, the peripheral background image is an image that has been processed by the calculation unit 133 and the CG / peripheral background generation unit 534, so there is a possibility that a processing delay will occur more than in the example of the first embodiment. That is, in the second embodiment, compared with the background image in the center of the screen of the display unit 14, there is a possibility that a time difference due to delay will occur in the peripheral background image. For this reason, the CG synthesis unit 135 corrects the time difference due to delay between the background image in the center and the peripheral background image. Also, in the first embodiment described above, the background synthesis unit 136 performs image positioning, seam correction, angle of view correction, optical performance correction, image quality difference correction, etc. In the second embodiment, the CG synthesis unit 135 performs image positioning and seam correction, and the angle of view correction, optical performance correction, and image quality difference correction may be performed by either one or both of the image processing unit 131 and the image processing unit 132.

[0063] As described above, in the second embodiment, a background image that matches the viewing angle characteristics and visual characteristics of humans, that is, a background image with high resolution in the center and low resolution in the periphery, can also be generated. That is, in the second embodiment, as in the first embodiment, the HMD user can view MR images with high visual quality and a high resolution. Also, in the second embodiment, as in the first embodiment, the image acquired by the imaging unit 12 is a high-resolution image with a narrow viewing angle, so that the HMD can be made small and lightweight and consume less power, and the amount of transmission and processing can be reduced.

[0064] <Third embodiment> Fig. 7 shows a configuration example of an MR system according to the third embodiment, and Fig. 8 shows a flowchart showing the flow of processing in a processing unit 73 of the third embodiment and example images. In Fig. 7 and Fig. 8, the same functional units as those in Fig. 1 and the same processing steps as those in Fig. 2 are given the same reference numerals as those in Fig. 1 and Fig. 2, and redundant explanations will be omitted.

[0065] In the third embodiment, an example will be described in which a display 742 of a display unit 74 mounted on a video see-through type HMD is a display that allows images to be input separately to the central and peripheral parts of the screen. The HMD of the MR system according to the third embodiment includes the imaging units 11 and 12 similar to those in the example of Fig. 1, and a display unit 74 having a display 742 that can input images separately for the central and peripheral parts of the screen. As shown in Fig. 7, the processing unit 73 of the MR system of this embodiment does not include the background synthesis unit 136 of the processing unit 13 in Fig. 1, and includes a central CG synthesis unit 737 and a peripheral CG synthesis unit 738 instead of the CG generation unit 134 in Fig. 1.

[0066] In the third embodiment, the first image (a high-resolution image with a narrow angle of view) obtained by capturing an image of the real world with the imaging unit 12 and processing by the image processing unit 132 is sent to a central CG synthesis unit 737 . In the third embodiment, the second image (low-resolution wide-angle image) obtained by capturing an image of the real world by the imaging unit 11 and processing by the image processing unit 131 is sent to the calculation unit 133.

[0067] The calculation unit 133 performs image information calculations such as estimating the position and orientation of the HMD and detecting the skeleton as described above. In the third embodiment, the calculation unit 133 sends the image information calculation results to the central CG synthesis unit 737. In the third embodiment, the calculation unit 133 also sends the low-resolution wide-angle image after processing in the image processing unit 131 to the peripheral CG synthesis unit 738 together with the image information calculation results.

[0068] The central CG synthesis unit 737 generates a CG image in the same manner as the CG generation unit 134 based on the image information calculation result of the calculation unit 133, and synthesizes the CG image with a high-resolution image with a narrow angle of view that has been captured by the imaging unit 12 and processed by the image processing unit 132 to generate a synthetic image. Then, the central CG synthesis unit 737 outputs the generated synthetic image as an image to be input to the center of the display 742 of the display unit 74. In other words, the central CG synthesis unit 737 performs a first CG synthesis process to synthesize the CG image with the first image captured by the imaging unit 12 to generate a synthetic image to be displayed in the center of the display 742 of the display unit 74.

[0069] The peripheral CG synthesis unit 738 generates a CG image in the same manner as described above, based on the image information calculation result of the calculation unit 133. Furthermore, the peripheral CG synthesis unit 738 generates a composite image by synthesizing a CG image with an image that has been cut out from the low-resolution wide-angle image via the calculation unit 133 to match the angle of view of the display unit 14. Then, the peripheral CG synthesis unit 738 outputs the generated composite image as an image to be input to the peripheral portion of the display 742 of the display unit 74. In other words, the peripheral CG synthesis unit 738 performs a second CG synthesis process in which the CG image is synthesized with the second image by the imaging unit 11 to generate a composite image to be displayed in the peripheral portion of the display 742 of the display unit 74.

[0070] In the third embodiment, there is a possibility that delay time difference, angle of view difference, optical performance difference, image quality difference, etc. may occur between the two composite images produced by the central CG synthesis unit 737 and the peripheral CG synthesis unit 738, which are generally similar to those described in the second embodiment. In the case of the third embodiment, the time difference may be corrected by either or both of the central CG synthesis unit 737 and the peripheral CG synthesis unit 738, and the angle of view correction, optical performance correction, and image quality difference correction may be performed by either or both of the image processing unit 131 and the image processing unit 132.

[0071] Next, the flow of processing in the processing unit 73 of the third embodiment will be described with reference to FIG. An image 21 in FIG. 8 is a wide-angle image 21 similar to that in FIG. 2, and an image 22 in FIG. 8 is a high-resolution image 22 similar to that in FIG.

[0072] In the third embodiment, the peripheral CG synthesis unit 738 generates a CG image based on the result of the image information calculation by the calculation unit 133 in S205 as the process of S806. Furthermore, the peripheral CG synthesis unit 738 generates a synthetic image by synthesizing the CG image with an image that has been cut out from the low-resolution wide-angle image sent from the calculation unit 133 to fit the angle of view of the display unit 14. Image 84 in Fig. 8 is a schematic representation of an image cut out from the wide-angle image 21 to fit the angle of view of the display unit 74. Then, the processing unit 73 outputs the synthetic image generated by the peripheral CG synthesis unit 738 as an image to be input to the peripheral portion of the display 742 as the process of S809.

[0073] Here, the peripheral CG synthesis unit 738 converts the synthesized image obtained by synthesizing the image 84 with the CG image into a resolution corresponding to the number of pixels of the display 742 and outputs the converted image, but as described above, the wide-angle image 21 captured by the imaging unit 11 has a low resolution, and the resolution seen by the HMD user is not very high. For this reason, the CG image generated by the peripheral CG synthesis unit 738 may have a resolution equivalent to that of the wide-angle image 21 captured by the imaging unit 11, or may have a resolution equivalent to that of the display 742. In other words, since the CG image generated by the peripheral CG synthesis unit 738 is displayed in the peripheral part of the display unit 74, the resolution may be somewhat low according to the viewing angle characteristics of humans, whose resolution is low except for the part being watched, which is also preferable from the viewpoint of system resources.

[0074] As processing in S807, the central CG synthesis unit 737 generates a CG image based on the result of the image information calculation by the calculation unit 133 in S205, and further generates a synthesized image by synthesizing the CG image with the high-resolution image with a narrow angle of view sent from the calculation unit 133. Then, as processing in S808, the processing unit 73 outputs the synthesized image generated by the central CG synthesis unit 737 as an image to be input to the center of the display 742.

[0075] An image 83 in Fig. 8 is a schematic diagram showing an example of an image when a composite image 84 produced by a peripheral CG synthesis unit 738 and a composite image produced by a central CG synthesis unit 737 are displayed on a display 742 that allows separate image input for the central and peripheral parts of the screen. Note that the CG image is omitted from the image 83 shown in Fig. 8.

[0076] As described above, according to the third embodiment, even when a display 742 capable of inputting images separately into the center and periphery of the screen is used, it is possible to provide the HMD user with visually high quality and high resolution images similar to those of the above-mentioned embodiments. Also, in the third embodiment, as in the above-mentioned embodiments, it is possible to make the HMD small and light, and to reduce power consumption, and it is also possible to reduce the amount of transmission and processing.

[0077] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics.

[0078] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a background generating means for generating a background image based on the first image and the second image; a CG synthesis means for generating a synthetic image by synthesizing the background image with a computer graphic image; having The information processing apparatus is characterized in that the composite image is presented on the presentation device. (Configuration 2) A calculation means for calculating a position and orientation of a user observing an image presented by the presentation device based on the second image; a CG generating means for generating the computer graphic image based on the result of the calculation of the position and orientation; 2. The information processing device according to configuration 1, comprising: (Configuration 3) The information processing device according to configuration 1 or 2, wherein the background generating means generates the background image by synthesizing the first image and the second image based on at least one of human viewing angle characteristics and visual characteristics. (Configuration 4) the first image has a higher resolution than the second image; the second image is an image with a wider angle of view than the image presented by the presentation device; The information processing device according to any one of configurations 1 to 3, wherein the background generating means generates the background image using the first image in a central portion and the second image in a peripheral portion. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the background generating means aligns the positions of the first image and the second image when generating the background image. (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, wherein the background generating means, when generating the background image, matches an angle of view between the first image and the second image. (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, wherein the background generating means corrects a difference in image quality between the first image and the second image when generating the background image. (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, wherein the background generating means corrects a seam between the first image and the second image when generating the background image. (Configuration 9) an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a generating means for generating a background image of a peripheral portion and a computer graphic image based on the second image; a CG synthesis means for generating a synthetic image by using the first image as a background image of a central portion and synthesizing the background image of the central portion with the background image of the peripheral portion generated by the generation means and the computer graphic image; having The information processing apparatus is characterized in that the composite image is presented on the presentation device. (Configuration 10) a calculation means for calculating a position and orientation of a user observing an image presented by the presentation device based on the second image, 10. The information processing device according to configuration 9, wherein the generating means generates the computer graphic image based on a result of the position and orientation calculation. (Configuration 11) the first image has a higher resolution than the second image; 11. The information processing device according to configuration 9 or 10, wherein the second image is an image with a wider angle of view than the image presented by the presentation device. (Configuration 12) The information processing device described in configuration 11, characterized in that the generation means generates an image in which the central part of an image cut out from the second image is made transparent according to the angle of view of the image presented by the presentation device, as a background image of the peripheral part. (Configuration 13) an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a first CG synthesis means for synthesizing the first image with a computer graphic image to generate an image of the central portion to be presented by the presentation device; a second CG synthesis means for synthesizing the second image with a computer graphic image to generate an image of the peripheral area presented by the presentation device; having 2. An information processing apparatus comprising: an image generated by said first CG synthesis means and an image generated by said second CG synthesis means, the image being displayed on said display device. (Configuration 14) a calculation means for calculating a position and orientation of a user observing an image presented by the presentation device based on the second image, 14. The information processing apparatus according to configuration 13, wherein the first CG synthesis means and the second CG synthesis means synthesize the computer graphic image according to a result of the calculation of the position and orientation. (Configuration 15) the first image has a higher resolution than the second image; 15. The information processing device according to configuration 13 or 14, wherein the second image is an image with a wider angle of view than the image presented by the presentation device. (Method 1) an acquisition step of acquiring a first image captured of the real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a background generating step of generating a background image based on the first image and the second image; a CG compositing step of generating a composite image by compositing the background image with a computer graphic image; having The information processing method further comprises presenting the composite image on the presentation device. (Method 2) an acquisition step of acquiring a first image captured of the real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a generating step of generating a background image of a peripheral portion and a computer graphic image based on the second image; a CG compositing step of using the first image as a background image of a central portion and generating a composite image by compositing the background image of the central portion with the background image of the peripheral portion generated in the generating step and the computer graphic image; having The information processing method further comprises presenting the composite image on the presentation device. (Method 3) an acquisition step of acquiring a first image captured of the real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a first CG synthesis step of synthesizing the first image with a computer graphic image to generate an image of the central portion presented by the presentation device; a second CG synthesis step of synthesizing the second image with a computer graphic image to generate an image of the peripheral area presented by the presentation device; having An information processing method, comprising: presenting, on the presentation device, the image generated in the first CG compositing step and the image generated in the second CG compositing step. (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0079] 11, 12: imaging unit, 13: processing unit, 14: display unit, 131, 132: image processing unit, 133: calculation unit, 134: CG generation unit, 135: CG synthesis unit, 136: background synthesis unit

Claims

1. an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a background generating means for generating a background image based on the first image and the second image; a CG compositing means for generating a composite image by compositing the background image with a computer graphic image; having The information processing apparatus is characterized in that the composite image is presented on the presentation device.

2. A calculation means for calculating a position and orientation of a user observing an image presented by the presentation device based on the second image; a CG generating means for generating the computer graphic image based on the result of the calculation of the position and orientation; 2. The information processing apparatus according to claim 1, further comprising:

3. 2. The information processing apparatus according to claim 1, wherein the background generating means generates the background image by synthesizing the first image and the second image based on at least one of human viewing angle characteristics and visual characteristics.

4. the first image has a higher resolution than the second image; the second image is an image having a wider angle of view than the image presented by the presentation device; 4. The information processing apparatus according to claim 3, wherein the background generating means generates the background image by using the first image in a central portion and the second image in a peripheral portion.

5. 5. The information processing apparatus according to claim 1, wherein the background generating means aligns the positions of the first image and the second image when generating the background image.

6. 5 . The information processing apparatus according to claim 1 , wherein the background generating means adjusts an angle of view between the first image and the second image when generating the background image.

7. 5. The information processing apparatus according to claim 1, wherein the background generating means corrects a difference in image quality between the first image and the second image when generating the background image.

8. 5. The information processing apparatus according to claim 1, wherein the background generating means corrects a seam between the first image and the second image when generating the background image.

9. an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a generating means for generating a background image of a peripheral portion and a computer graphic image based on the second image; a CG compositing means for using the first image as a background image of a central portion and generating a composite image by compositing the background image of the central portion with the background image of the peripheral portion generated by the generating means and the computer graphic image; having The information processing apparatus is characterized in that the composite image is presented on the presentation device.

10. a calculation means for calculating a position and orientation of a user observing an image presented by the presentation device based on the second image, 10. The information processing apparatus according to claim 9, wherein said generating means generates said computer graphic image based on a result of the calculation of said position and orientation.

11. the first image has a higher resolution than the second image; 11. The information processing apparatus according to claim 9, wherein the second image has a wider angle of view than the image presented by the presentation device.

12. The information processing device according to claim 11, characterized in that the generating means generates an image in which a central portion of an image cut out from the second image is made transparent according to an angle of view of the image presented by the presentation device, as a background image of the peripheral portion.

13. an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a first CG synthesis means for synthesizing the first image with a computer graphic image to generate an image of the central portion to be presented by the presentation device; a second CG synthesis means for synthesizing the second image with a computer graphic image to generate an image of the peripheral area presented by the presentation device; having An information processing apparatus comprising: an image generated by said first CG synthesis means and an image generated by said second CG synthesis means, the image being displayed on said display device.

14. a calculation means for calculating a position and orientation of a user observing an image presented by the presentation device based on the second image, 14. The information processing apparatus according to claim 13, wherein the first CG synthesis means and the second CG synthesis means synthesize the computer graphic image according to a result of the calculation of the position and orientation.

15. the first image has a higher resolution than the second image; 15. The information processing apparatus according to claim 13, wherein the second image has a wider angle of view than the image presented by the presentation device.

16. an acquisition step of acquiring a first image captured of the real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a background generating step of generating a background image based on the first image and the second image; a CG compositing step of generating a composite image by compositing the background image with a computer graphic image; having The information processing method further comprises presenting the composite image on the presentation device.

17. an acquisition step of acquiring a first image captured of the real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a generating step of generating a background image of a peripheral portion and a computer graphic image based on the second image; a CG compositing step of using the first image as a background image of a central portion and generating a composite image by compositing the background image of the central portion with the background image of the peripheral portion generated in the generating step and the computer graphic image; having The information processing method further comprises presenting the composite image on the presentation device.

18. an acquisition step of acquiring a first image captured of the real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a first CG synthesis step of synthesizing the first image with a computer graphic image to generate an image of the central portion to be presented by the presentation device; a second CG synthesis step of synthesizing the second image with a computer graphic image to generate an image of the peripheral area presented by the presentation device; having An information processing method comprising: presenting, on the presentation device, the image generated in the first CG compositing step and the image generated in the second CG compositing step.

19. Computer, an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a background generating means for generating a background image based on the first image and the second image; a CG compositing means for generating a composite image by compositing the background image with a computer graphic image; having A program that causes the information processing device to function as an information processing device that presents the composite image on the presentation device.

20. Computer, an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a generating means for generating a background image of a peripheral portion and a computer graphic image based on the second image; a CG compositing means for using the first image as a background image of a central portion and generating a composite image by compositing the background image of the central portion with the background image of the peripheral portion generated by the generating means and the computer graphic image; having A program that causes the information processing device to function as an information processing device that presents the composite image on the presentation device.

21. Computer, an acquisition means for acquiring a first image captured of a real world at a narrower angle of view than an image presented by a presentation device, and a second image captured of the real world at a wider angle of view than the first image; a first CG synthesis means for synthesizing the first image with a computer graphic image to generate an image of the central portion to be presented by the presentation device; a second CG synthesis means for synthesizing the second image with a computer graphic image to generate an image of the peripheral area presented by the presentation device; having a program that causes the information processing device to function as an information processing device that presents the image generated by the first CG synthesis means and the image generated by the second CG synthesis means on the presentation device;

Images