Information processing device and information processing method

Abstract

To provide a technology for generating an image by changing a previously photographed image according to a reproduced condition but not photographing again the image.SOLUTION: An information processing device includes: an image acquisition unit which acquires a first image containing a first foreground and a first background photographed by a first photographing device, and acquires a second image containing a second foreground and a second background photographed by a second photographing device; an information acquisition unit which acquires first position and attitude information indicating the position and attitude of the first photographing device when photographing the first image, or second position and attitude information indicating the position and attitude of the second photographing device when photographing the second image; and an image generation unit which generates a synthesized image obtained by synthesizing the first foreground and the second background or by synthesizing the second foreground and the first background on the basis of the acquired first position and attitude information or the second position and attitude information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, mixed reality (MR) has become known as a technology that combines real space and virtual space in real time. In MR, a composite image is displayed in which a virtual space image corresponding to the position and orientation of an imaging device mounted on an HMD (Head Mounted Display) is superimposed on a real image captured by the imaging device.

[0003] In the past, when developing MR content and applications, there were issues such as the time required to recreate interaction situations and check operation, and the time required to recreate situations when multiple people are experiencing them. To address this issue, a technology has been proposed that uses pre-recorded videos to generate synthetic images that recreate the desired situation.

[0004] In Patent Document 1, 3D data included in a 3D video is synthesized onto an MR image based on a user's instructions. Patent Document 1 also discloses that a recorded image of an avatar or virtual object is synthesized onto the MR image viewed by the user. Furthermore, in Patent Document 1, the position and timing of image playback are determined according to the user's movements.

[0005] In Patent Document 2, when playing a 360-degree video from a single viewpoint, an image is generated that matches the viewpoint position of the HMD. In Patent Document 2, the background and foreground (objects) of the images that make up the video are separated, and the position of the foreground is changed to generate an image from a new viewpoint. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 1,113,885 [Patent Document 2] US Patent Application Publication No. 2019 / 0371030 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in Patent Document 1, a video that reproduces a desired situation is generated using a pre-recorded video, but since the content of the stored video cannot be changed, it may become necessary to re-take the video depending on changes in the situation. Furthermore, Patent Document 2 does not anticipate using an image captured by an HMD when generating the image of the new viewpoint.

[0008] Therefore, with the conventional technology disclosed in the above patent document, if it becomes necessary to reproduce a video under conditions different from those at the time of shooting, the video must be shot again, which poses a problem in that it incurs costs associated with video shooting.

[0009] The present invention has been made in consideration of the above, and aims to provide a technology for generating an image by modifying an image captured in advance to suit the situation to be reproduced, without capturing the image again. [Means for solving the problem]

[0010] The information processing device according to the present invention includes an image acquisition unit that acquires a first image including a first foreground and a first background captured by a first imaging device, and acquires a second image including a second foreground and a second background captured by a second imaging device; The information processing device is characterized by comprising: an information acquisition unit that acquires first position and orientation information indicating the position and orientation of a first imaging device, or second position and orientation information indicating the position and orientation of the second imaging device at the time of capturing the second image; and an image generation unit that generates a composite image in which the first foreground and the second background are combined, or in which the second foreground and the first background are combined, based on the acquired first position and orientation information or the acquired second position and orientation information.

[0011] Furthermore, an information processing method according to the present invention is an information processing method characterized by comprising the steps of acquiring a first image including a first foreground and a first background captured by a first imaging device, and acquiring a second image including a second foreground and a second background captured by a second imaging device; acquiring first position and orientation information indicating the position and orientation of the first imaging device when capturing the first image, or second position and orientation information indicating the position and orientation of the second imaging device when capturing the second image; and generating a composite image by combining the first foreground with the second background or the second foreground with the first background, based on the acquired first position and orientation information or the acquired second position and orientation information. [Effects of the Invention]

[0012] According to the present invention, a video captured in advance can be modified to suit the situation to be reproduced, and a video captured in a situation different from that at the time of capture can be generated without re-capturing the video. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a functional block diagram showing a schematic configuration of an information processing device according to a first embodiment. [Figure 2] 1 is a hardware configuration diagram of an information processing device according to a first embodiment. [Figure 3] 4 is a flowchart of a process executed by the information processing device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an image processed by the information processing device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] <Embodiment 1> An information processing device 100 according to a first embodiment of the present invention will be described below with reference to Fig. 1. Fig. 1 shows a functional block configuration diagram of an information processing system 1 including the information processing device 100 according to this embodiment. As shown in Fig. 1, the information processing system 1 has the information processing device 100 and an HMD (Head Mounted Display) 105. In this embodiment, the information processing device 100 is externally connected to the HMD 105 via a wired or wireless connection.

[0015] The HMD 105 is a head-mounted device that includes an imaging device 110 and a display device 120. However, any device may be used in place of the HMD 105 as long as it includes the imaging device 110 and the display device 120 or is connected to the imaging device 110 and the display device 120. For example, a device that includes a USB camera and a display, or a device that includes a camera built into a laptop PC or tablet PC and the display of the laptop PC or tablet PC may be used in place of the HMD 105.

[0016] 1, the information processing device 100 includes a first image acquisition unit 1010, a first position and orientation acquisition unit 1020, a first foreground acquisition unit 1030, a second image acquisition unit 1040, a second position and orientation acquisition unit 1050, and a second foreground acquisition unit 1060. The information processing device 100 also includes a foreground selection unit 1070, a background selection unit 1080, and a position and orientation selection unit 1090. The information processing device 100 also includes an image generation unit 1110 and a video storage unit 1120.

[0017] In this embodiment, the video storage unit 1120 stores moving images captured in advance by the first image acquisition unit 1010 at a different time and / or viewpoint from the images acquired from the imaging device 110, which is a stereo camera mounted on the HMD 105. The moving images stored in the video storage unit 1120 are also captured by a second imaging device different from the HMD 105. The details of the processes executed by the above-mentioned units of the information processing device 100 will be described later.

[0018] 2 is a diagram showing an example of the hardware configuration of the information processing device 100. The information processing device 100 has a CPU 1210, a RAM (Random Access Memory) 1220, and a ROM (Read-Only Memory) 1230. The information processing device 100 also has a keyboard 1240, a mouse 1250, a display unit 1260, a storage unit 1270, and an I / F (Interface) 1280. These units of the information processing device 100 are connected to each other via a bus 1300.

[0019] The CPU 1210 controls each unit of the information processing device 100. The ROM 1230 stores in advance various types of information used in the processes executed by the information processing device 100, which will be described below. For example, the ROM 1230 stores an OS (Operating System) program, a device driver program, and a program for executing the processes according to this embodiment. The RAM 1220 temporarily stores various types of information used in the processes executed by the information processing device 100. The CPU 1210 loads the programs stored in the ROM 1230 into the RAM 1220 and executes them.

[0020] The keyboard 1240 and the mouse 1250 are operation units (input I / F (Interface)) that accept operations (instructions) from the user of the information processing device 100. The user uses at least one of the keyboard 1240 and the mouse 1250 to perform various operations (for example, operations on virtual objects) for processes executed by the information processing device 100. The keyboard 1240 and the mouse 1250 output operation signals in response to operations (instructions) from the user, and the CPU 1210 executes processes in response to the output operation signals. The keyboard 1240 and the mouse 1250 may be provided detachably with respect to the information processing device 100, or may be provided integrally with the information processing device 100.

[0021] The display unit 1260 displays various images generated by the processing of the information processing device 100. The display unit 1260 is, for example, a display. The display unit 1260 may be provided detachably with respect to the information processing device 100, or may be provided integrally with the information processing device 100. The storage unit 1270 stores various information generated by the processing of the information processing device 100, and is built into the information processing device 100.

[0022] The I / F 1280 is used to output image data to an external device (e.g., the HMD 105). The I / F 1280 can also be connected to an external storage device that stores information generated by processing by the information processing device 100. The I / F 1280 may also be used to acquire information from the external device.

[0023] Fig. 3 is a flowchart showing an example of processing for realizing an information processing method executed by the information processing device 100 according to this embodiment. The processing of this embodiment will be described in detail below with reference to Fig. 3. As an example, when a user of the information processing device 100 operates the keyboard 1240 or the mouse 1250 to issue an instruction to execute the processing of the flowchart in Fig. 3, the CPU 1210 starts the following processing.

[0024] In step S2010, the first image acquisition unit 1010 of the information processing device 100 acquires images constituting a real-time video captured by the imaging device 110 of the HMD 105. The image captured by the imaging device 110 of the HMD 105 is a first image including a first foreground and a first background. FIG. 4A shows an example of an image 401 acquired by the first image acquisition unit 1010 in step S2010. In the image 401, a human hand 411 in the foreground and a background 431 are depicted. In addition, an index 421, which will be described later, is also depicted. The first image acquisition unit 1010 outputs the image acquired from the image sensing device 110 to the first position and orientation acquisition unit 1020, the first foreground acquisition unit 1030, and the background selection unit 1080, respectively.

[0025] In step S2020, the first position and orientation acquisition unit 1020 acquires first position and orientation information indicating the position and orientation of the image capture device 110 of the HMD 105 at the time of capturing the image acquired by the first image acquisition unit. The first position and orientation acquisition unit 1020, which is an information acquisition unit that acquires the position and orientation information, outputs the acquired first position and orientation information to the position and orientation selection unit 1090.

[0026] Here, the method for acquiring the position and orientation information of the image capturing device 110 when capturing an image is not limited to a specific method. For example, if a known index (e.g., index 421 in FIG. 4A ) located in real space is present within the imaging range of the image capturing device 110 when capturing the image, the first image acquiring unit 1010 acquires the index's location information along with the image. For example, the index's location information may be attached to the image as image metadata. The first position and orientation acquiring unit 1020 can acquire the position and orientation information of the image capturing device 110 by specifying the position and orientation of the image capturing device 110 using the image coordinates of the index in the image and the location information of the index. Note that the index may be a feature point or an edge extracted from the image. Since the technique for specifying the position and orientation of the image capturing device 110 using these indexes is a well-known technique, detailed description thereof will be omitted here.

[0027] Furthermore, for example, if the imaging device 110 is equipped with a position and orientation sensor (not shown), the first image acquisition unit 1010 acquires measurement values ​​output from the position and orientation sensor when the image is captured. Note that information about the measurement values ​​may be attached to the image as metadata for the image. Then, the first position and orientation acquisition unit 1020 identifies the position and orientation of the imaging device 110 based on the information about the measurement values ​​acquired by the first image acquisition unit 1010.

[0028] In step S2030, the first foreground acquisition unit 1030 extracts and acquires at least a part of a human body (such as a hand or a body) and / or an object depicted in the image acquired in step S2010. Fig. 4B shows an example of a foreground acquired by the first foreground acquisition unit 1030 from the first image 401 of Fig. 4A. The first foreground acquisition unit 1030 acquires a hand 411 depicted in the first image 401 as the foreground 402.

[0029] For example, when extracting a hand depicted in an image, the first foreground acquisition unit 1030 can extract the foreground using color information stored in advance in the storage unit 1270. The method of extracting the foreground from an image is not limited to this. For example, the first foreground acquisition unit 1030 may extract the foreground from the image acquired in step S2010 using a machine learning model that has been trained by machine learning to extract at least a part of a human body (such as a hand or a body) and / or an object as the foreground from the image. Note that the technology for acquiring the foreground from an image using a machine learning model is well known, and therefore a detailed description thereof will be omitted here.

[0030] The imaging device 110 mounted on the HMD 105 may be a stereo camera. In this case, the imaging device 110 is configured with two cameras arranged on the left and right. The information processing device 100 associates the image coordinates of the human hand or body and / or object extracted from each image captured by the left and right cameras of the imaging device 110. Then, the information processing device 100 acquires depth information of the human hand or body and / or object from the imaging device by stereo measurement using the associated image coordinates and the relative positions and orientations of the left and right cameras acquired in advance. Here, stereo measurement is a well-known technique, so a detailed description will be omitted. Furthermore, in step S2030, the first foreground acquisition unit 1030 may acquire the depth information acquired above in addition to the image information of the human hand or body as the foreground. The first foreground acquisition unit 1030 outputs the acquired foreground and / or depth information to the foreground selection unit 1070.

[0031] In step S2040, the second image acquisition unit 1040 acquires a second image from a 3D video captured by a second imaging device different from the imaging device 110 mounted on the HMD 105. Fig. 4C shows an example of an image 403 acquired by the second image acquisition unit 1040 in step S2040. In the image 403, a human body 413 in the foreground and a background 423 are depicted.

[0032] In this embodiment, the 3D video is stored in the video storage unit 1120, and the second image acquisition unit 1040 acquires the 3D video from the video storage unit 1120. The video storage unit 1120 also stores position and orientation information of the imaging device that captured the 3D video at the time the 3D video was captured. As an example, the position and orientation of the imaging device can be acquired by the method described in step S2020, and the video storage unit 1120 stores pairs of images that make up the 3D video and their position and orientation at the time of capturing the video.

[0033] The second image acquisition unit 1040 outputs the acquired second image to the second foreground acquisition unit 1060 and the background selection unit 1080. The second image acquisition unit 1040 also outputs to the second position and orientation acquisition unit 1050 the position and orientation information of the second imaging device at the time of capturing the second image.

[0034] In step S2050, the second position and orientation acquisition unit 1050 acquires second position and orientation information indicating the position and orientation of the second image capture device at the time of capturing the image acquired by the second image acquisition unit 1040. Note that the second position and orientation acquisition unit 1050, which is an information acquisition unit that acquires the position and orientation information, may acquire, from the video storage unit 1120, the position and orientation information at the time of capturing that is paired with the image acquired by the second image acquisition unit 1040.

[0035] Note that the method for acquiring the position and orientation of the second image capturing device when capturing an image acquired by the second image acquisition unit 1040 is not limited to this. For example, an index for specifying the position and orientation of the image capturing device may be depicted in the second image, and the second position and orientation acquisition unit 1050 may calculate the position and orientation of the second image capturing device using the image coordinates of the index detected from the second image and the placement information of the index. Any known method may be used as long as it can acquire the position and orientation information of the second image capturing device when capturing an image based on the image. The second position and orientation acquisition unit 1050 outputs the acquired position and orientation information of the second image capturing device to the position and orientation selection unit 1090.

[0036] In step S2060, second foreground acquisition unit 1060 acquires the foreground from the second image acquired from second image acquisition unit 1040. Second foreground acquisition unit 1060 acquires the foreground from the second image by the method described in step S2030. FIG. 4D shows an example of the foreground acquired by second foreground acquisition unit 1060 from second image 403 of FIG. 4C. Second foreground acquisition unit 1060 acquires human body 413 depicted in second image 403 as foreground 404. Note that video storage unit 1120 may store pairs of images constituting the 3D video and the foreground depicted in the images. In this case, second foreground acquisition unit 1060 may acquire the foreground of the second image from video storage unit 1120.

[0037] Alternatively, any method other than the above may be used as long as it can acquire the foreground from the second image. For example, the second foreground acquisition unit 1060 can extract the foreground from the image input from the second image acquisition unit 1040 by using color information for foreground extraction stored in advance in the storage unit 1270. Alternatively, any known method for extracting the foreground from an image may be used.

[0038] In step S2070, the background selection unit 1080 selects the background from either the first image input from the first image acquisition unit 1010 or the second image input from the second image acquisition unit 1040. FIG. 4E shows the background 405 of the first image 401 selected by the background selection unit 1080. FIG. 4F shows the background 405 of the second image 401 selected by the background selection unit 1080. 4 shows a second background 406 of the image 403.

[0039] For example, a first image background 405 and a second image background 406 may be displayed on the display unit 1260 via a user interface (not shown), and the user may select which background to use in the composite image by operating the keyboard 1240 and the mouse 1250. The background selection unit 1080 outputs the selected background to the image generation unit 1110.

[0040] In step S2080, the position and orientation selection unit 1090 selects either the position and orientation of the first image capture device input from the first position and orientation acquisition unit 1020 or the position and orientation of the second image capture device input from the second position and orientation acquisition unit 1050. For example, options for the first position and orientation and the second position and orientation may be displayed on the display unit 1260 via a user interface (not shown), and the user may select which position and orientation to use in the composite image by operating the keyboard 1240 and mouse 1250. The position and orientation selection unit 1090 outputs position and orientation information indicating the selected position and orientation to the image generation unit 1110.

[0041] In step S2090, foreground selection unit 1070 selects either the first foreground input from first foreground acquisition unit 1030 or the second foreground input from second foreground acquisition unit. For example, the first foreground and the second foreground may be displayed on display unit 1260 via a user interface (not shown), and the user may operate keyboard 1240 and mouse 1250 to select which of the images to use as the foreground in the composite image. Foreground selection unit 1070 outputs the selected foreground to image generation unit 1110.

[0042] In step S2100, the image generation unit 1110 generates a composite image using the background output by the background selection unit 1080, the position and orientation of the image capture device output by the position and orientation selection unit 1090, and the foreground output by the foreground selection unit 1070. Here, it is assumed that parameters such as the focal length and resolution of the image capture device at the time of capturing the images used for composition are known information. FIG. 4G shows an example of a composite image generated by the image generation unit 1110. The composite image 407 shown in FIG. 4G is an image synthesized using a foreground 402 and a background 406. Furthermore, a virtual object 427, which will be described later, is depicted in the composite image 407. In FIG. 4G, as an example, the image generation unit 1110 processes the foreground 402 and the background 406, respectively, using a viewpoint position based on the position and orientation of the image capture device 110 of the HMD 105, to generate a foreground 417 and a background 437 viewed from the viewpoint position. The image generation unit 1110 then generates a composite image 407 by combining the foreground 417, the virtual object 427, and the background 437.

[0043] The image generation unit 1110 may also acquire information about a virtual object to be superimposed on the composite image. For example, the information about the virtual object may be stored in the video storage unit 1120. In this case, the image generation unit 1110 generates an image of a virtual space using the information about the virtual object acquired from the video storage unit 1120 and superimposes the image on a background used in the composite image, thereby realizing mixed reality. Note that, since the technology for generating an image of a virtual object based on a virtual viewpoint and superimposing it on a background is well known, a detailed description of the process by which the image generation unit 1110 generates a composite image using information about the virtual object will be omitted. The image generation unit 1110 outputs the generated composite image to the display device 120 of the HMD 105.

[0044] In step S2110, the display device 120 of the HMD 105 displays the composite image input from the image generation unit 1110.

[0045] In step S2120, the CPU 1210 of the information processing device 100 determines whether or not to end the processing of the flowchart in Fig. 3. For example, when the user operates the keyboard 1240 and the mouse 1250 to instruct the information processing device 100 to hide the virtual object (to end the display of the virtual object), the CPU 1210 determines to end the processing. If the CPU 1210 has not received such an instruction from the user, the CPU 1210 determines not to end the process. If the CPU 1210 determines to end the process, it ends the process of the flowchart in Figure 3, and if it determines not to end the process, it returns to step S2010.

[0046] As described above, according to this embodiment, a composite image is generated by modifying images constituting a 3D image captured in advance in accordance with the real-time situation of the image displayed on the HMD 105, and the generated composite image can be displayed on the HMD 105. This makes it possible to test different interactions on the HMD 105 without recapturing the 3D image. For example, by replacing the foreground hand region with a hand region extracted in real time, it is possible to check an application on the HMD 105 using hand interaction operations that are different from those used when capturing the 3D image.

[0047] Also, consider a case where multiple users using an HMD similar to the HMD 105 check content using a 3D image in which each user is reflected while displaying different backgrounds. In this case, the information processing device 100 can display a composite image without recapturing the 3D image by selecting a background different from the background of the 3D image and combining it with the images of the multiple users in the foreground.

[0048] Therefore, according to the information processing device 100 of this embodiment, even in an environment where repeated imaging is not easy and a composite image under a different situation from that at the time of imaging is to be displayed on the HMD 105, the effort, time, and cost of re-capturing the 3D image can be reduced.

[0049] In the above description, the first image acquisition unit 1010 and the second image acquisition unit 1040 may be realized by a single image acquisition unit. Similarly, the first position and orientation acquisition unit 1020 and the second position and orientation acquisition unit 1050 may be realized by a single position and orientation acquisition unit. Furthermore, the first foreground acquisition unit 1030 and the second foreground acquisition unit 1060 may be realized by a single foreground acquisition unit.

[0050] Next, a description will be given of a modified example of embodiment 1. In the following description, the same configurations and processes as those of the information processing device 100 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0051] <Variation 1-1> A first modification of the first embodiment will be described. In the first embodiment, it is assumed that the three-dimensional video stored in the video storage unit 1120 is a video captured in advance at a different time and / or viewpoint from the first image captured by the imaging device 110 (such as a stereo camera) of the HMD 105. However, in this modification, the three-dimensional video is not limited to this, and may be, for example, a 360° video captured by a second imaging device or a video or still image captured with a wide-angle field of view.

[0052] Information processing device 100 according to this modification executes the process of the flowchart shown in Fig. 3. Then, in step S2040, second image acquisition unit 1040 acquires a panoramic video or a video or still image captured with a wide-angle field of view from video storage unit 1120, and acquires a second image from the acquired panoramic video, video, or still image. Furthermore, in step S2070, background selection unit 1080 selects the second image acquired by second image acquisition unit 1040 and outputs it to image generation unit 1110.

[0053] As an example, in step S2080, the position and orientation selection unit 1090 selects the position and orientation of the image capture device 110 of the HMD 105, and in step S2090, the foreground selection unit 1070 Suppose that selects the first foreground.

[0054] In this case, the image generation unit 1110 generates a background for the composite image using the background of the omnidirectional video or video or still image captured with a wide-angle field of view acquired from the background selection unit 1080, based on the camera parameters specific to the camera, such as the position and orientation of the image capture device 110 and the focal length. Note that the method by which the image generation unit 1110 generates a background based on the camera parameters specific to the camera, such as the orientation and focal length of the image capture device 110, from the omnidirectional video or video or still image captured with a wide-angle field of view can be realized by known technology, and therefore a detailed description thereof will be omitted.

[0055] The image generation unit 1110 generates a background for the composite image from a video or still image captured with a panoramic video or a wide-angle field of view, captured by an imaging device different from the imaging device 110 at a time and / or viewpoint different from the time and / or viewpoint when the first image was captured. As a result, the image generation unit 1110 generates a composite image by combining the first foreground depicted in the first image with a background generated using an image captured by the second imaging device, based on the position and orientation of the imaging device 110 of the HMD 105. Therefore, even in this modification, a composite image can be generated using a background generated based on a 3D image captured by the second imaging device, without re-capturing the 3D image.

[0056] <Variation 1-2> Next, a second modified example of the first embodiment will be described. In this modified example, it is assumed that the second imaging device is a virtual camera. Therefore, the 3D video stored in the video storage unit 1120 is, for example, a 3D video captured by a virtual camera in a 3D reconstructed virtual space. Furthermore, the video storage unit 1120 stores a pair of images constituting the 3D video and the position and orientation of the virtual camera at the time of capturing the images. The second image captured by the virtual camera in this modified example can be replaced with the image 403 captured by the second imaging device in the first embodiment.

[0057] The information processing device 100 according to this modification executes the process of the flowchart shown in Fig. 3. In step S2040, the second image acquisition unit 1040 acquires a 3D image captured by a virtual camera from the image storage unit 1120, and acquires a second image from the 3D image. The second image acquisition unit 1040 also acquires, from the image storage unit 1120, information on the position and orientation of the virtual camera paired with the second image. The second image acquisition unit 1040 outputs the acquired information on the position and orientation of the virtual camera to the second position and orientation acquisition unit 1050. Then, in step S2050, the second position and orientation acquisition unit 1050 acquires position and orientation information indicating the position and orientation of the virtual camera when the second image was captured. Then, in step S2070, the background selection unit 1080 selects the second image acquired by the second image acquisition unit 1040 as a background, and outputs the selected image to the image generation unit 1110.

[0058] Here, it is assumed that in step S2080, the position and orientation selection unit 1090 selects the position and orientation of the virtual camera, and in step S2090, the foreground selection unit 1070 selects the first foreground.

[0059] In this case, the image generation unit 1110 generates a background for the composite image using the background of the 3D video captured by the virtual camera acquired from the background selection unit 1080, based on the camera parameters specific to the camera, such as the position and orientation of the image capture device 110 and the focal length. Note that a method for generating a background from the video captured by the virtual camera, based on the camera parameters specific to the camera, such as the orientation and focal length of the image capture device 110, can be realized by known technology, and therefore a detailed description thereof will be omitted.

[0060] As a result, the image generating unit 1110 captures an image using a virtual camera different from the image capturing device 110. The background of the composite image is generated from the 3D video of the virtual space captured by the virtual camera. As a result, the image generation unit 1110 generates a composite image by combining the first foreground depicted in the first image with the background generated using the image captured by the virtual camera, based on the position and orientation of the image capture device 110 of the HMD 105. Therefore, even in this modification, it is possible to reduce the effort, time, and cost associated with re-capturing the 3D video.

[0061] <Variation 1-3> Next, a third modified example of the first embodiment will be described. In this modified example, the position and orientation selection unit 1090 selects a third position and orientation that is different from the position and orientation of the first image capture device and the position and orientation of the second image capture device as the position and orientation to be used for generating a composite image. For example, the position and orientation selection unit 1090 can select, as the third position and orientation, a position and orientation selected by a user of the information processing device 100 by operating an input unit such as the keyboard 1240 or the mouse 1250. Then, the position and orientation selection unit 1090 receives third position and orientation information indicating the third position and orientation specified by the user.

[0062] The information processing device 100 according to this modification executes the processing of the flowchart shown in Fig. 3. In step S2080, the position and orientation selection unit 1090 selects third position and orientation information indicating the third position and orientation received as described above, and outputs the selected information to the image generation unit 1110. Then, in step S2100, the image generation unit 1110 generates a composite image based on the third position and orientation information instead of the first position and orientation information or the second position and orientation information. That is, the image generation unit 1110 generates a composite image using the background selected in step S2070 and the foreground selected in step S2090, based on the third position and orientation selected by the position and orientation selection unit 1090.

[0063] As a result, according to the information processing device 100 of this modified example, it is possible to generate a mixed reality synthetic image that corresponds to more flexible changes in the position and orientation of the imaging device, thereby reducing the effort, time, and cost associated with re-capturing 3D images.

[0064] Note that each functional unit of the information processing device 100 according to the above embodiment and modified examples may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware. For example, the information processing device may have a processor and a memory in which a control program is stored. Then, the functions of at least some of the functional units of the information processing device may be realized by the processor reading and executing the control program from the memory.

[0065] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configuration of the above-described embodiment within the scope of the gist of the present invention.Furthermore, the present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0066] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0067] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image acquisition unit that acquires a first image including a first foreground and a first background captured by a first imaging device, and acquires a second image including a second foreground and a second background captured by a second imaging device; an information acquisition unit that acquires first position and orientation information indicating a position and orientation of the first image capturing device when the first image is captured, or second position and orientation information indicating a position and orientation of the second image capturing device when the second image is captured; an image generation unit that generates a composite image by combining the first foreground and the second background or by combining the second foreground and the first background based on the acquired first position and orientation information or the acquired second position and orientation information; An information processing device comprising: (Configuration 2) The information processing device according to configuration 1, wherein the second image is an image captured by the second imaging device at a time or viewpoint position different from the time or viewpoint position at which the first image was captured by the first imaging device. (Configuration 3) 3. The information processing device according to configuration 2, wherein the first image is a real-time video image, and the second image is an image captured in advance by the second imaging device. (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the second image is an all-around image captured by the second imaging device, or a video image or a still image captured by the second imaging device with a wide-angle field of view. (Configuration 5) the information acquisition unit accepts designation of third position and orientation information different from the first position and orientation information and the second position and orientation information; The image generation unit generates the composite image based on the third position and orientation information instead of the acquired first position and orientation information or the acquired second position and orientation information. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, wherein the first imaging device is a stereo camera. (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, wherein the second imaging device is a virtual camera. (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, wherein the first foreground and the second foreground are images including at least a part of a human body or an object. (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, wherein the image generation unit generates a virtual object based on the first position and orientation information or the second position and orientation information, and generates the composite image by superimposing the virtual object on the first background or the second background. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, wherein the image generation unit generates the composite image viewed from a viewpoint position specified by the first position and orientation information or the second position and orientation information. (method) acquiring a first image including a first foreground and a first background captured by a first imaging device, and acquiring a second image including a second foreground and a second background captured by a second imaging device; acquiring first position and orientation information indicating a position and orientation of the first image capturing device when the first image is captured, and second position and orientation information indicating a position and orientation of the second image capturing device when the second image is captured; generating a composite image by combining the first foreground and the second background or by combining the second foreground and the first background based on the first position and orientation information or the second position and orientation information; An information processing method comprising: (program) A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0068] 100 Information processing device, 105 HMD, 1010 First image acquisition unit, 1040 Second image acquisition unit, 1020 First position and orientation acquisition unit, 1050 Second position and orientation acquisition unit, 1110 Image generation unit

Claims

1. an image acquisition unit that acquires a first image including a first foreground and a first background captured by a first imaging device, and acquires a second image including a second foreground and a second background captured by a second imaging device; an information acquisition unit that acquires first position and orientation information indicating a position and orientation of the first image capturing device when capturing the first image, or second position and orientation information indicating a position and orientation of the second image capturing device when capturing the second image; an image generation unit that generates a composite image by combining the first foreground and the second background or by combining the second foreground and the first background based on the acquired first position and orientation information or the acquired second position and orientation information; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the second image is an image captured by the second imaging device at a time or viewpoint position different from the time or viewpoint position at which the first image was captured by the first imaging device.

3. 3. The information processing apparatus according to claim 2, wherein the first image is a real-time video image, and the second image is an image captured in advance by the second imaging device.

4. 2. The information processing device according to claim 1, wherein the second image is a panoramic video image captured by the second imaging device, or a video image or a still image captured with a wide-angle field of view by the second imaging device.

5. the information acquisition unit accepts designation of third position and orientation information different from the first position and orientation information and the second position and orientation information; The image generation unit generates the composite image based on the third position and orientation information instead of the acquired first position and orientation information or the acquired second position and orientation information.

2. The information processing apparatus according to claim 1, wherein:

6. 2. The information processing apparatus according to claim 1, wherein the first imaging device is a stereo camera.

7. 2. The information processing apparatus according to claim 1, wherein the second image capturing device is a virtual camera.

8. The information processing apparatus according to claim 1 , wherein the first foreground and the second foreground are images including at least a part of a human body or an object.

9. 2. The information processing device according to claim 1, wherein the image generation unit generates a virtual object based on the first position and orientation information or the second position and orientation information, and generates the composite image by superimposing the virtual object on the first background or the second background.

10. The information processing apparatus according to claim 1 , wherein the image generating unit generates the composite image viewed from a viewpoint position specified by the first position and orientation information or the second position and orientation information.

11. A first image including a first foreground and a first background is acquired by a first image capturing device, and a second image including a second foreground and a second background is acquired by a second image capturing device. and acquiring first position and orientation information indicating a position and orientation of the first image capturing device when capturing the first image, or second position and orientation information indicating a position and orientation of the second image capturing device when capturing the second image; generating a composite image by combining the first foreground and the second background or by combining the second foreground and the first background based on the acquired first position and orientation information or the acquired second position and orientation information; An information processing method comprising:

12. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

Images