Image synthesis device, image synthesis method, and storage medium

Abstract

The present application provides an image synthesizing apparatus capable of appropriately synthesizing a stereoscopic image and an image of a three-dimensional object. The image synthesizing apparatus (1) includes: a storage section (11) that stores a first stereoscopic image having a first right-eye image and a first left-eye image; a reception section (12) that receives an operation of a three-dimensional object; a disposition section (13) that disposes a three-dimensional object corresponding to the operation in a three-dimensional virtual space in such a manner that a straight line connecting a right-eye viewpoint and a point of the first right-eye image corresponding to infinity and a straight line connecting a left-eye viewpoint and a point of the first left-eye image corresponding to infinity intersect each other; a generation section (14) that generates a second stereoscopic image including a second right-eye image and a second left-eye image in the three-dimensional virtual space in which the three-dimensional object is disposed, the second right-eye image being an image observed from the right-eye viewpoint in a state in which the first right-eye image is displayed, and the second left-eye image being an image observed from the left-eye viewpoint in a state in which the first left-eye image is displayed; and an output section (15) that outputs the second stereoscopic image.

Description

Technical Field

[0001] This invention relates to an image synthesis apparatus for synthesizing stereoscopic images and three-dimensional objects. Background Technology

[0002] Previously, techniques have been employed to provide stereoscopic images to users wearing HMDs by displaying stereoscopic images on HMDs (see, for example, Japanese Patent Application Publication No. 2020-071718). Additionally, three-dimensional computer graphics (3DCG) are known, including rendering processes that configure three-dimensional objects in a three-dimensional virtual space and generate two-dimensional images viewed from any viewpoint. Summary of the Invention

[0003] The technical problem to be solved by the present invention

[0004] However, methods for properly compositing 3D objects into stereoscopic images are not well known. For example, when compositing a 2D image generated by 3DCG with the right-eye and left-eye images contained in a stereoscopic image, there is a problem that the 3D object is displayed in a planar form.

[0005] The present invention was made to solve the above-mentioned technical problems, and its purpose is to provide an image synthesis apparatus for appropriately compositing three-dimensional objects into a stereoscopic image.

[0006] Technical means for solving technical problems

[0007] To achieve the above objectives, an image synthesis apparatus according to one aspect of the present invention includes: a storage unit for storing a first stereoscopic image having a first right-eye image and a first left-eye image; a receiving unit for receiving operations of a three-dimensional object disposed in a three-dimensional virtual space; a placement unit for placing the three-dimensional virtual space of the first right-eye image and the first left-eye image in such a way that a first straight line intersects with a second straight line, and placing a three-dimensional object corresponding to the operation received by the receiving unit, wherein the first straight line is a straight line connecting a right-eye viewpoint and a point corresponding to infinity in the first right-eye image, and the second straight line is a straight line connecting a left-eye viewpoint and a point corresponding to infinity in the first left-eye image; a generation unit for generating a second stereoscopic image including the second right-eye image and the second left-eye image in the three-dimensional virtual space in which the three-dimensional object is disposed, wherein the second right-eye image is an image viewed from a right-eye viewpoint while the first right-eye image is displayed, and the second left-eye image is an image viewed from a left-eye viewpoint while the first left-eye image is displayed; and an output unit for outputting the second stereoscopic image.

[0008] Based on this structure, for example, it is possible to prevent the 3D object being manipulated from being located behind the two images by adjusting the configuration of the first right-eye image and the first left-eye image in the 3D virtual space. Therefore, it is possible to appropriately synthesize the 3D object and the first stereoscopic image. Furthermore, for example, when the 3D object is located behind the first stereoscopic image, it is also possible to prevent the 3D object from being located behind the first stereoscopic image by adjusting the movement of the first right-eye image and the first left-eye image by a shorter distance.

[0009] Alternatively, in one aspect of the image synthesis apparatus of the present invention, a first right-eye image and a first left-eye image may be configured in a three-dimensional virtual space such that points corresponding to a predetermined distance overlap.

[0010] According to this structure, when a three-dimensional object is located closer to the front than a specified distance, the three-dimensional object is not located on the back side of the first stereoscopic image.

[0011] Alternatively, in one aspect of the image synthesis apparatus of the present invention, a specifying part may be further provided, which specifies the distance of a predetermined object contained in the first stereoscopic image, the predetermined distance being greater than the longest distance specified by the specifying part.

[0012] According to this structure, when the three-dimensional object is located closer to the front of a specified object at a distance specified by a specific part, the three-dimensional object is not located on the back side of the first stereoscopic image.

[0013] In addition, in one aspect of the image synthesis apparatus of the present invention, the defined object may also be all objects included in the first stereoscopic image.

[0014] According to this structure, when a 3D object is located closer to the front side than the object contained in the deepest side of the first stereoscopic image, the 3D object is not located on the back side of the first stereoscopic image.

[0015] In addition, in one aspect of the image synthesis apparatus of the present invention, the receiving unit may also receive information specifying a defined object, wherein the defined object is the object specified by the information received by the receiving unit.

[0016] According to this structure, if the three-dimensional object is located closer to the front than the object specified by the information received by the receiving department, the three-dimensional object will not be located on the back side of the first stereoscopic image.

[0017] In addition, in one aspect of the image synthesis apparatus of the present invention, the configuration unit may also configure the first stereoscopic image at a predetermined distance when the distance to the three-dimensional object is close to a predetermined distance.

[0018] Based on this structure, it is possible to avoid the three-dimensional object being located on the back side of the first stereo image by reconfiguring the first stereo image.

[0019] Alternatively, in one aspect of the image synthesis apparatus of the present invention, a first right-eye image and a first left-eye image may be configured in a three-dimensional virtual space in such a way that points corresponding to infinity overlap.

[0020] According to this structure, a three-dimensional object located at the same distance as any object contained in the first stereoscopic image is not located on the back side of the first stereoscopic image.

[0021] Alternatively, in one aspect of the image synthesis apparatus of the present invention, the first stereoscopic image may also be captured by a pair of cameras arranged in such a way that the optical axes are parallel.

[0022] Alternatively, in one aspect of the image synthesis apparatus of the present invention, the first stereoscopic image may also be captured by a pair of cameras arranged such that their optical axes intersect.

[0023] Alternatively, in one aspect of the image synthesis apparatus of the present invention, the first stereoscopic image and the second stereoscopic image may also be videos.

[0024] Based on this structure, for example, when the first stereoscopic image is a surgical image, it is possible to overlap the forceps or other instruments that are three-dimensional objects with the forceps or other instruments contained in the first stereoscopic image, so that novice surgeons can imitate the operation of experienced surgeons in virtual space and can carry out surgical training.

[0025] In addition, in one aspect of the image synthesis apparatus of the present invention, a determination unit may be further provided, which determines the position of an object specifically included in the first stereoscopic image in the three-dimensional virtual space based on the first right-eye image and the first left-eye image arranged in the three-dimensional virtual space, and makes a determination related to the relationship between the specific position and the position of the three-dimensional object, and the output unit outputs the determination result related to the determination unit.

[0026] Based on this structure, for example, a user who operates a 3D object can know whether the 3D object is being operated appropriately.

[0027] Furthermore, one aspect of the image synthesis method of the present invention includes: a step of accepting an operation of a three-dimensional object configured in a three-dimensional virtual space; a step of configuring a three-dimensional object corresponding to the accepted operation in a three-dimensional virtual space having a first right-eye image and a first left-eye image, configured such that a first straight line intersects a second straight line, wherein the first straight line is a straight line connecting a right-eye viewpoint and a point corresponding to infinity in the first right-eye image, and the second straight line is a straight line connecting a left-eye viewpoint and a point corresponding to infinity in the first left-eye image; a step of generating a second stereoscopic image including a second right-eye image and a second left-eye image in the three-dimensional virtual space having the three-dimensional object configured, wherein the second right-eye image is an image viewed from a right-eye viewpoint while the first right-eye image is displayed, and the second left-eye image is an image viewed from a left-eye viewpoint while the first left-eye image is displayed; and a step of outputting the second stereoscopic image.

[0028] The effects of the invention

[0029] An image synthesis apparatus or the like according to one aspect of the present invention can appropriately synthesize stereoscopic images and three-dimensional objects. Attached Figure Description

[0030] Figure 1 This is a block diagram illustrating the structure of an image synthesis apparatus according to an embodiment of the present invention.

[0031] Figure 2 This is a flowchart illustrating the operation of the image synthesis apparatus according to this embodiment.

[0032] Figure 3 This is a diagram used to illustrate the capture of a stereoscopic image in this embodiment.

[0033] Figure 4A This is a diagram used to illustrate the generation of the stereoscopic image in this embodiment.

[0034] Figure 4B This is a diagram illustrating an example of the display of the synthesized stereoscopic image in this embodiment.

[0035] Figure 4C This is a diagram illustrating an example of the display of the synthesized stereoscopic image in this embodiment.

[0036] Figure 5 This is a diagram used to illustrate the point corresponding to infinity in this embodiment.

[0037] Figure 6A This is a diagram used to illustrate the generation of the stereoscopic image in this embodiment.

[0038] Figure 6B This is a diagram illustrating an example of the display of the synthesized stereoscopic image in this embodiment.

[0039] Figure 6C This is a diagram used to illustrate the generation of the stereoscopic image in this embodiment.

[0040] Figure 7A This is a diagram used to illustrate the point corresponding to infinity in this embodiment.

[0041] Figure 7B This is a diagram used to illustrate the point corresponding to infinity in this embodiment.

[0042] Figure 8A This is a diagram illustrating an example of a stereoscopic image and a three-dimensional object configured in a three-dimensional virtual space in this embodiment.

[0043] Figure 8B This is a diagram illustrating an example of a stereoscopic image and a three-dimensional object configured in a three-dimensional virtual space in this embodiment.

[0044] Figure 9 This is a block diagram illustrating other structures of the image synthesis apparatus according to this embodiment.

[0045] Figure 10 This is a diagram illustrating an example of a stereoscopic image configured in a three-dimensional virtual space in this embodiment.

[0046] Figure 11 This is a diagram illustrating an example of a stereoscopic image configured in a three-dimensional virtual space in this embodiment.

[0047] Figure 12 This is a diagram illustrating an example of a stereoscopic image configured in a three-dimensional virtual space in this embodiment.

[0048] Figure 13 This is a block diagram illustrating other structures of the image synthesis apparatus according to this embodiment.

[0049] Figure 14 This is a schematic diagram illustrating an example of the appearance of the computer system in this embodiment.

[0050] Figure 15 This is a diagram illustrating an example of the structure of the computer system in this embodiment. Detailed Implementation

[0051] Hereinafter, the image compositing apparatus and image compositing method of the present invention will be described using embodiments. Furthermore, in the following embodiments, constituent elements and steps that are given the same reference numerals are the same or corresponding, and sometimes further descriptions are omitted. The image compositing apparatus and image compositing method of this embodiment generate a second stereoscopic image that combines a first stereoscopic image disposed in a three-dimensional virtual space and a three-dimensional object.

[0052] Figure 1 This is a block diagram showing the structure of the image compositing apparatus 1 according to this embodiment. The image compositing apparatus 1 of this embodiment includes a storage unit 11, a receiving unit 12, a configuration unit 13, a generation unit 14, and an output unit 15. The image compositing apparatus 1 can be a general-purpose device such as a personal computer, smartphone, or tablet terminal, or it can be a dedicated device for compositing stereoscopic images and three-dimensional objects. Furthermore, the image compositing apparatus 1 can be a standalone device, or it can be a server device in a server / client system. In the latter case, the receiving unit 12 or the output unit 15 can also receive input or output information via a communication line such as the Internet. In this embodiment, the case where the image compositing apparatus 1 is a standalone device will be mainly described.

[0053] The storage unit 11 stores a first stereoscopic image having a first right-eye image and a first left-eye image. The first stereoscopic image can be, for example, a video, or a still image. In this embodiment, the case where the first stereoscopic image is a video will be primarily described. The first stereoscopic image can be, for example, a captured image, or an image generated by 3DCG, etc. In this embodiment, the case where the first stereoscopic image is a captured stereoscopic image will be primarily described.

[0054] Furthermore, in this embodiment, the description mainly focuses on the case where all the data of the first stereoscopic image as a video is stored in the storage unit 11, but this is not always the case. For example, the first stereoscopic image as a video may be received in real time in the image synthesis apparatus 1, and only a portion of the data to be processed may be stored in the storage unit 11. Even in this case, each frame constituting the first stereoscopic image as a video is at least temporarily stored in the storage unit 11, so it can be said that the first stereoscopic image is stored in the storage unit 11.

[0055] In addition, in this embodiment, the case where the first stereoscopic image is captured by a pair of cameras arranged in a way that makes the optical axes parallel will be mainly described, and the case where a pair of cameras arranged in a way that makes the optical axes intersect will be described later.

[0056] The process of storing the first stereoscopic image in the storage unit 11 is not limited. For example, the first stereoscopic image can be stored in the storage unit 11 using a storage medium, or it can be stored in the storage unit 11 using a first stereoscopic image transmitted via a communication line, or it can be stored in the storage unit 11 using a first stereoscopic image input from an imaging device, etc. The storage unit 11 is preferably implemented using a non-volatile storage medium, but it can also be implemented using a volatile storage medium. The storage medium can be, for example, a semiconductor memory, a magnetic disk, an optical disk, etc.

[0057] The receiving unit 12 accepts operations on three-dimensional objects configured in a three-dimensional virtual space. These operations can be, for example, operations related to the pose, position, or shape of the three-dimensional object. Operations accepted by the receiving unit 12 can be arbitrary operations as long as they cause changes in the orientation, position, or shape of the three-dimensional object configured in the three-dimensional virtual space. Operations related to the shape of the three-dimensional object could be, for example, opening or closing the tip of forceps used in surgery. The receiving unit 12 can accept operations on three-dimensional objects, for example, from the controller of a VR (Virtual Reality) headset (e.g., Oculus Quest 2, etc.) that outputs a second stereoscopic image (described later) from the image synthesis device 1, or from other input devices such as a joystick or mouse. Furthermore, the acceptance of operations on three-dimensional objects can be done using sensors such as accelerometers or gyroscopes to accept the results of detecting changes in position or angle in the actual three-dimensional space, or it can accept input from input devices such as buttons. The receiving unit 12 typically accepts operations on three-dimensional objects in real time from input devices such as the controller of an HMD2 or VR headset. The receiving unit 12 can also accept operations on three-dimensional objects using input devices or communication lines. Furthermore, the receiving unit 12 may or may not include devices for accepting data (e.g., input devices or communication devices). Additionally, the receiving unit 12 can be implemented in hardware or in software such as drivers for specified devices.

[0058] The configuration unit 13 configures three-dimensional objects corresponding to the operations handled by the receiving unit 12 in a three-dimensional virtual space that configures the first right-eye image and the first left-eye image. Since the position, posture, shape, etc., of the three-dimensional objects change according to the handled operations, the configuration unit 13 changes the three-dimensional objects in the three-dimensional virtual space according to the operations performed. Because the configuration of the three-dimensional objects based on the operations handled by the receiving unit 12 is performed by the configuration unit 13, for example, the configuration of the three-dimensional objects in the three-dimensional virtual space continuously changes during continuous handling of operations. The method of configuring three-dimensional objects in the three-dimensional virtual space according to operations is known, and its detailed description is omitted. Furthermore, the information of the three-dimensional objects configured in the three-dimensional virtual space may, for example, be stored in the storage unit 11.

[0059] Furthermore, the placement of the first right-eye image and the first left-eye image of the first stereoscopic image in the three-dimensional virtual space, and their relationship with the right-eye viewpoint and the left-eye viewpoint, are usually predetermined. The specific relationship between the two will be explained later, but for example, (A) the first right-eye image and the first left-eye image may also be positioned in front of each other along a parallel line of sight. Additionally, for example, (B) the first right-eye image and the first left-eye image may also be positioned so that they overlap with points corresponding to infinity.

[0060] Typically, 3D objects are positioned between the first stereo image and the viewpoint. This is because when a first stereo image exists between a 3D object and the viewpoint, the 3D object will not be included in the rendered second stereo image.

[0061] The generation unit 14 generates a second stereoscopic image in a three-dimensional virtual space where three-dimensional objects are configured. This second stereoscopic image includes a second right-eye image as an image viewed from a right-eye viewpoint while displaying a first right-eye image, and a second left-eye image as an image viewed from a left-eye viewpoint while displaying a first left-eye image. Specifically, when generating the second right-eye image, it is preferable to display the first right-eye image in the three-dimensional virtual space, but not the first left-eye image. Similarly, when generating the second left-eye image, it is preferable to display the second right-eye image in the three-dimensional virtual space, but not the second left-eye image. Furthermore, as long as the first left-eye image does not enter the second right-eye image, the first left-eye image can also be displayed during rendering using a right-eye viewpoint. The same applies during rendering using a left-eye viewpoint. The second stereoscopic image is preferably generated such that objects displayed stereoscopically in the first stereoscopic image, as well as three-dimensional objects, are also displayed stereoscopically in the second stereoscopic image. Additionally, the rendering perspective can be, for example, the same as, or different from, the perspective of the camera that captured the first stereoscopic image.

[0062] The positions of the right-eye and left-eye viewpoints in the 3D virtual space can be predetermined or changed in real time. In the latter case, for example, the positions of the right-eye and left-eye viewpoints can be changed according to the position or orientation of HMD2. Furthermore, the arrangement of the first stereoscopic image in the 3D virtual space can also be changed according to the viewpoint changes. This is because, generally, the position of the viewpoint and its relative position to the positions of the first right-eye and first left-eye images are predetermined. Moreover, changes to viewpoints corresponding to the position or orientation of HMD2 are well known, and detailed descriptions are omitted. Additionally, the gaze direction is preferably directed towards the center of the corresponding image. For example, the gaze direction of the right-eye viewpoint is preferably directed towards the center of the right-eye image. In this embodiment, for ease of explanation, the case where the gaze directions of the right-eye and left-eye viewpoints are horizontal is mainly described; however, naturally, the gaze directions can also be other than horizontal.

[0063] When the first stereoscopic image is a video, a second stereoscopic image is generated by combining the individual frames of the video with the three-dimensional object; therefore, the second stereoscopic image is also a video. In this case, for example, the first stereoscopic image can be reproduced in a three-dimensional virtual space, or the configuration of the three-dimensional object can be changed according to the received operation to generate a second stereoscopic image containing the reproduced first stereoscopic image and the three-dimensional object changed according to the operation. Furthermore, in the second right-eye image and the second left-eye image of the second stereoscopic image, the frame (i.e., the outer edge) of the first stereoscopic image may or may not be displayed. In the latter case, the second stereoscopic image can be generated by making the frame of the first right-eye image (or the first left-eye image) the frame of the second right-eye image (or the second left-eye image).

[0064] Alternatively, configuration unit 13 and generation unit 14 can be implemented using a real-time 3D development platform such as Unity (Unity Technologies). Figure 8A , Figure 8B This is an illustration showing a first stereoscopic image configured in Unity's 3D virtual space, along with surgical forceps as a 3D object. Figure 8A In this configuration, the positions of the first right-eye image and the first left-eye image are staggered. For example, in case (A) above, this is the configuration. Furthermore, while the two images would normally exist in the same plane, they are slightly offset for ease of explanation. Figure 8BIn this configuration, a first right-eye image and a first left-eye image are overlapped. For example, in the case described in (B) above, this configuration is used. Furthermore, the generation of images based on the generation unit 14 as a rendering process in 3DCG is well-known, and its detailed explanation is omitted. Moreover, it is natural that texture mapping, etc., can be performed during this rendering process.

[0065] Output unit 15 outputs the second stereoscopic image generated by generation unit 14. This output can be, for example, a display device (e.g., HMD, VR headset, etc.) that displays the stereoscopic image, a transmission to a designated device via a communication line, storage on a storage medium, or delivery to other components. In this embodiment, as described above, the case where output unit 15 outputs the second stereoscopic image to HMD2 will be primarily explained. Furthermore, output unit 15 may or may not include an output device (e.g., a display device or a communication device). Additionally, output unit 15 can be implemented in hardware or in software such as drivers for these devices.

[0066] Next, using Figure 2 The flowchart describes the operation of the image synthesis apparatus 1 of this embodiment, i.e., the image synthesis method. In this flowchart, the case of generating a second stereoscopic image by compositing a three-dimensional object corresponding to the operation received by the receiving unit 12 into a first stereoscopic image of a video stored in the storage unit 11 is explained.

[0067] (Step S101) The generation unit 14 determines whether to generate a second stereoscopic image. Then, if a second stereoscopic image is generated, the process proceeds to step S102; otherwise, the process of step S101 is repeated until it is determined that a second stereoscopic image will be generated. Furthermore, the generation unit 14 may, for example, periodically determine whether to generate a second stereoscopic image. More specifically, the generation unit 14 may also determine whether to generate a second stereoscopic image based on the time interval between frames of the second stereoscopic image.

[0068] (Step S102) The receiving unit 12 determines whether the operation on the three-dimensional object has been accepted. Then, if the operation has been accepted, proceed to step S103; otherwise, proceed to step S104.

[0069] (Step S103) The configuration unit 13 configures the three-dimensional object in the three-dimensional virtual space according to the operation received by the receiving unit 12. The configuration of the three-dimensional object may also be a change to the configuration of the three-dimensional object configured in the three-dimensional virtual space, for example, changing the position, angle, shape, etc. of the three-dimensional object.

[0070] (Step S104) In a state where the first right-eye image is displayed but the first left-eye image is not displayed in the three-dimensional virtual space, the generation unit 14 generates a second right-eye image; in a state where the first left-eye image is displayed but the first right-eye image is not displayed, the generation unit 14 generates a second left-eye image, thereby generating a second stereoscopic image. Furthermore, the generation of the second right-eye image and the second left-eye image can also be achieved by generating one frame for the right eye and one frame for the left eye. In this case, the configuration of the first right-eye image and the first left-eye image in the three-dimensional virtual space can be changed frame by frame each time step S104 is executed. The generated pair of frames can also be stored, for example, in a storage medium or storage unit 11 (not shown).

[0071] (Step S105) Output unit 15 outputs the second stereoscopic image generated in step S104. This output may be, for example, the output of one frame for the right eye and one frame for the left eye. Then, return to step S101. In this way, by repeating the processing of steps S101 to S105, the first stereoscopic image and the second stereoscopic image of the three-dimensional object configured according to the operation change are displayed on HMD2.

[0072] In addition, Figure 2 In the flowchart, for example, the processing of steps S101, S104, and S105 can be performed in parallel with the processing of steps S102 and S103. Additionally, in Figure 2 In the flowchart, the process ends by powering off or by an interrupt indicating the end of the process.

[0073] Next, the process of capturing the first stereoscopic image and synthesizing the first stereoscopic image with the three-dimensional object will be explained in detail.

[0074] Figure 3 This is a diagram used to illustrate the taking of the first stereoscopic image. In Figure 3 In this process, a first stereoscopic image is captured using a right-eye camera 21a and a left-eye camera 21b. Specifically, a first right-eye image 22a is captured using the right-eye camera 21a, and a first left-eye image 22b is captured using the left-eye camera 21b. The right-eye camera 21a and the left-eye camera 21b are typically of the same specifications, resulting in the first right-eye image 22a and the first left-eye image 22b being images with the same number of pixels. Figure 3In this example, it is assumed that the first stereoscopic image was captured by a pair of cameras 21a and 21b arranged with their optical axes parallel. Furthermore, it is assumed that the optical axes of both cameras 21a and 21b are in the horizontal plane. The distance between the optical axes of the right-eye camera 21a and the left-eye camera 21b could be, for example, approximately the average distance between human pupils. In this example, it is assumed that the object 20 being photographed is a cone arranged with its central axis in a plumb line. Moreover, it is assumed that the object 20 is included in both the first right-eye image 22a and the first left-eye image 22b. Figure 3 The figures show the view of cameras 21a and 21b from a plumb line perpendicular to the horizontal plane. Therefore, strictly speaking, object 20 should be circular, but for ease of illustration, it is shown as triangular. The same applies to the other figures.

[0075] Next, we will explain the cases in which the first right eye image 22a and the first left eye image 22b are respectively arranged in front of each other in the parallel line of sight in the three-dimensional virtual space (A) and (B) in a way that overlaps with the point corresponding to infinity.

[0076] (A) Cases where they are respectively positioned in front of each other along a parallel line of sight.

[0077] Figure 4A This diagram illustrates the arrangement of a first right-eye image 22a and a first left-eye image 22b in a three-dimensional virtual space, positioned in front of the viewpoint along a parallel line of sight. Figure 4A In the diagram, the right eye uses viewpoint 23a and the left eye uses viewpoint 23b to separate the interpupillary distance, or the distance between the optical axes of stereo cameras 21a and 21b. Furthermore, the lines of sight indicated by arrows are parallel and point in the same direction (in...). Figure 4A (Upper-middle direction). Furthermore, in Figure 4A In this case, it is assumed that the line of sight exists within the horizontal plane. That is, Figure 4A and Figure 3 Similarly, assume that viewpoints 23a and 23b, or images 22a and 22b, are viewed from a plumb line perpendicular to the horizontal plane. Furthermore, the first right-eye image 22a and the first left-eye image 22b are positioned in front of the right-eye viewpoint 23a and the left-eye viewpoint 23b, respectively, with the line of sight orthogonal to the image and pointing towards the center of the image. Additionally, assume that the positional relationship between the right-eye viewpoint 23a and the first right-eye image 22a is the same as the positional relationship between the left-eye viewpoint 23b and the first left-eye image 22b. Furthermore, the first right-eye image 22a and the first left-eye image 22b are arranged to be contained within the same plane. The distance from the right-eye viewpoint 23a and the left-eye viewpoint 23b to the plane containing the first right-eye image 22a and the first left-eye image 22b can, for example, be U / (2tan( / 2)). Here, U is the horizontal length of the first right-eye image 22a and the first left-eye image 22b. These are the horizontal viewing angles of the right eye using viewpoint 23a and the left eye using viewpoint 23b. Furthermore, in Figure 4A The image shows a first right eye image 22a and a first left eye image 22b arranged separately to the right and left, but the two images can also be arranged to partially overlap (for example, see reference). Figure 8A ).

[0078] exist Figure 4A In this context, assuming a first right-eye image 22a is displayed, a second right-eye image observed from right-eye viewpoint 23a is generated; and a first left-eye image 22b is displayed, a second left-eye image observed from left-eye viewpoint 23b is generated; these second stereoscopic images are output to HMD2. Thus, as... Figure 4A As shown by the dashed lines, the user observing the second stereoscopic image appears as if the object 20 exists on the depth side of the configured first right-eye image 22a and first left-eye image 22b.

[0079] exist Figure 4A In this scenario, assume a user wearing an HMD2 manipulates a 3D object, acting as a robotic arm, via a controller and positions it as shown in the 3D object 25a. Thus, as... Figure 4B As shown, from the user's perspective, the robotic arm appears to be in front, and object 20 appears to be on the depth side. Therefore, if the user attempts to touch object 20 with the robotic arm, the 3D object needs to be moved towards the depth side, i.e., away from the viewpoint. Then, when the user operates the controller to move the 3D object towards the depth side and position it as shown for 3D object 25b, the top side of 3D object 25b is located on the back side of the first right-eye image 22a and cannot be seen from the viewpoint. As a result, as... Figure 4C As shown, the top side of the robotic arm cannot be observed by the user. This is because the first right eye image 22a is typically opaque. Furthermore, in Figure 4C In the image, dashed lines indicate the parts of the robotic arm that cannot be observed. Thus, with the first right-eye image 22a and the first left-eye image 22b positioned on the front side in parallel viewing directions, it is difficult to adjust the orientation of the object displayed in the first stereoscopic image (e.g., object 20) with the three-dimensional object. That is, it is difficult to achieve overlap between the two in the second stereoscopic image. Furthermore, Figure 4B , Figure 4C This is a diagram showing a stereoscopic image on a two-dimensional plane, which differs from the actual stereoscopic image. The same applies to other diagrams showing stereoscopic images.

[0080] Furthermore, while the drawbacks of placing the right-eye image and left-eye image separately on the front side in parallel viewing directions have been explained, there are no drawbacks even when compositing a stereoscopic image and a three-dimensional object as described above, where the object included in the first stereoscopic image is independent of the three-dimensional object, i.e., they are unrelated. For example, if the first stereoscopic image is a landscape image and the three-dimensional object is a character dancing according to the user's operation, the compositing method can be used as long as the character dances on the foreground side of the landscape.

[0081] (B) The case of arranging the points in a way that overlaps with the points corresponding to infinity.

[0082] Figure 5 It is a diagram used to illustrate points in a three-dimensional image that correspond to infinity. For example... Figure 5 As shown, when assuming the shooting position for the right eye, the shooting position for the left eye, the viewing angle, and the shooting direction indicated by the arrow are set respectively, the right-eye image and the left-eye image are assumed to be images of objects, etc., projected onto a virtual plane in the attached figure, within the range of the viewing angle observed from the right-eye shooting position in the shooting direction. Furthermore, the virtual plane is a plane perpendicular to the shooting direction. In this case, points existing at infinity in front of the shooting direction are projected onto the virtual plane as points α and β. That is, it is assumed that point α in the right-eye image and point β in the left-eye image are points corresponding to infinity. Furthermore, points α and β are respectively the points where the right-eye image intersects with the shooting direction extending from the right-eye shooting position and the left-eye image intersects with the shooting direction extending from the left-eye shooting position. Therefore, when the first right-eye image and the first left-eye image are arranged in a three-dimensional virtual space such that the points corresponding to infinity overlap, they are arranged in the same position. That is, the two images are simply overlapped.

[0083] Figure 6A This diagram illustrates the arrangement of a first right-eye image 22a and a first left-eye image 22b captured in a three-dimensional virtual space, such that points corresponding to infinity are overlapped. Figure 6A In, also with Figure 4A Similarly, the right-eye viewpoint 23a and the left-eye viewpoint 23b are configured to be separated by the interpupillary distance, or the distance between the optical axes of the stereo cameras 21a and 21b. Furthermore, in this case, the line of sight indicated by the arrow points towards the center of the first right-eye image 22a and the first left-eye image 22b. Additionally, in Figure 6A In this context, it is also assumed that the line of sight exists within the horizontal plane. Furthermore, Figure 6A This is a diagram viewed from the direction of the plumb bob. Figure 6A The first right eye image 22a and the first left eye image 22b shown are respectively from the... Figure 4AThe positions shown are moved by the same distance until they overlap in the plane direction of the image.

[0084] exist Figure 6A In this example, assuming a first right-eye image 22a is displayed but a first left-eye image 22b is not, a second right-eye image observed from the right-eye viewpoint 23a is generated; similarly, a second left-eye image observed from the left-eye viewpoint 23b is generated while the first left-eye image 22b is displayed but the first right-eye image 22a is not, and their second stereoscopic images are output to HMD2. Thus, if using... Figure 6A As shown by the dashed lines, the user observing the second stereoscopic image appears as if the object 20 is present in front of the configured first right-eye image 22a and first left-eye image 22b.

[0085] exist Figure 6A In this scenario, assume a user wearing an HMD2 manipulates a 3D object, acting as a robotic arm, via a controller and positions it at the location shown in 3D object 25c. Thus, as... Figure 6B As shown, the robotic arm appears to be near object 20 to the user. Therefore, unlike the situation described in (A) above, the user can operate the controller to make the tip of the robotic arm overlap with object 20. In this way, by arranging the first right-eye image 22a and the first left-eye image 22b in a way that makes the points corresponding to infinity overlap, the positional relationship between the object (e.g., object 20) displayed in the first stereoscopic image and the three-dimensional object can be easily adjusted. That is, the object displayed in the stereoscopic image can overlap with the three-dimensional object. This is because the points on the first right-eye image 22a corresponding to infinity and the points on the first left-eye image 22b corresponding to infinity overlap, so there is a position on the image corresponding to infinity, and the three-dimensional object can be arranged at any position within the range from the viewpoint to the image, or at any position from the position closest to the viewpoint to infinity.

[0086] Next, the case of capturing a first stereoscopic image using a pair of cameras arranged in a manner that intersects the optical axes will be described. For example, in the case of capturing stereoscopic images using the da Vinci surgical robot, there is a situation where the cameras are arranged in a manner that intersects the optical axes. In this case, as... Figure 7AAs shown, the shooting direction (i.e., the camera's optical axis) indicated by the arrow from the right-eye shooting position intersects with the shooting direction (i.e., the front side of the camera) from the left-eye shooting position on the front side of the line of sight. The positions of the right-eye and left-eye images taken in this way, corresponding to infinity, are points α1 and β1 in the attached diagram. These points are the intersections of two parallel straight lines extending from the right-eye and left-eye shooting positions at an angle equal to the shooting direction, and the right-eye and left-eye images. Furthermore, since the situation on the right side contrasts with that on the left side, if point α1 can be specified in the right-eye image, point β1 can be specified in the left-eye image. Therefore, the specific location of point α1 in the right-eye image will be explained below.

[0087] like Figure 7B As shown, let X be the distance from the intersection of the shooting direction and the right-eye image to point α1, let L be the distance from the right-eye shooting position to the captured image, let U be the length of the right-eye image in the horizontal direction, and let the horizontal viewing angle be... Let θ be the angle between the shooting direction and the straight line connecting the shooting position and point α1 of the right eye. Then we can obtain the following formula.

[0088] tan ( / 2)=U / (2L)

[0089] tanθ=X / L

[0090] Therefore, X is as shown in the following formula.

[0091] X = U × tanθ / (2tan( / 2))

[0092] Here, It is determined by the camera used in the shooting, and therefore is known. Additionally, since 2θ is... Figure 7A The angle between the lines of sight on the right and left sides of the eye is known, therefore θ is also known. Thus, the arrangement is such that the point corresponding to infinity overlaps... Figure 7A When capturing the first right-eye image 22a and the first left-eye image 22b, U, θ, and θ are used as the width of the image in the horizontal direction. Calculate X as described above, and then arrange the two images such that the point α1, which is X apart horizontally from the center in the first right-eye image 22a, overlaps with the point β1, which is X apart horizontally from the center in the first left-eye image 22b. Furthermore, when arranging the first right-eye image 22a and the first left-eye image 22b in a three-dimensional virtual space, it is also possible to... Figure 7BThe shooting position for the right eye is the right eye viewpoint. That is, the first right eye image 22a and the first left eye image 22b can also be positioned on a plane at a distance L from the right eye viewpoint and left eye viewpoint positioned in the three-dimensional virtual space, such that point α1 overlaps with point β1. Here, L is obtained from the above formula as L = U / (2tan( / 2)). Furthermore, the first right-eye image 22a and the first left-eye image 22b are preferably configured to be symmetrical with respect to a plane perpendicular to the line segment passing through the midpoint of the line segment connecting the right-eye viewpoint 23a and the left-eye viewpoint 23b. That is, when viewed from the front side... Figure 6A In the case of the configured image, the first right eye is moved X to the left in the plane containing the image using image 22a, and the first left eye is moved X to the right in the plane containing the image using image 22b. The result is that, as with... Figure 6C As shown, the first right eye image 22a and the first left eye image 22b are configured accordingly. Furthermore, in Figure 6C For ease of explanation, the first right-eye image 22a and the first left-eye image 22b are shown separately, but in reality, they exist on the same plane. In this case, the viewing direction is also set to face the center of the corresponding image. Thus, even when the first stereoscopic image is captured by a pair of cameras arranged with their optical axes intersecting, the first right-eye image and the first left-eye image can be arranged to overlap points corresponding to infinity. As a result, the positional relationship between objects contained in the first stereoscopic image and three-dimensional objects can be easily adjusted.

[0093] As described above, according to the image synthesis apparatus 1 and image synthesis method of this embodiment, a first right-eye image and a first left-eye image, along with a three-dimensional object, are arranged in a three-dimensional virtual space. A new right-eye image is generated while the first right-eye image is displayed, and a new left-eye image is generated while the first left-eye image is displayed. This allows the generation of a second stereoscopic image that synthesizes the first stereoscopic image and the three-dimensional object. By synthesizing the two in this way, a second stereoscopic image that can stereoscopically display both the object contained in the stereoscopic image and the three-dimensional object can be generated. Furthermore, by arranging the first right-eye image and the first left-eye image in a way that overlaps points corresponding to infinity in the three-dimensional virtual space, the positional relationship between the object contained in the stereoscopic image and the three-dimensional object can be appropriately adjusted. For example, the three-dimensional object can be displayed overlapping the object contained in the stereoscopic image. Additionally, when the three-dimensional object overlaps the object contained in the stereoscopic image, situations where a portion of the three-dimensional object is located on the back side of the stereoscopic image and cannot be observed can be avoided.

[0094] Furthermore, this embodiment primarily describes the case where the first and second stereoscopic images are videos, but they may not be. The first and second stereoscopic images may also be still images. In this case, a first stereoscopic image and a second stereoscopic image, which are still images, are generated by compositing the three-dimensional objects configured according to the operation.

[0095] Next, an example of using the image synthesis apparatus 1 of this embodiment will be described.

[0096] The first stereoscopic image is a video recording of a skilled person's actions. The three-dimensional object corresponds to the tools or similar implements used by the skilled person, and the user operating the three-dimensional object can be a beginner imitating the skilled person's actions. In this case, a second stereoscopic image is generated that composites the three-dimensional object operated by the beginner with the first stereoscopic image representing the skilled person's actions. More specifically, the skilled person's actions can be surgical procedures, sports competitions, or musical instrument performances. Furthermore, the three-dimensional object can be, for example, surgical instruments such as forceps, sports equipment such as baseball bats, golf clubs, tennis or badminton rackets, or ping-pong paddles, or musical instruments. In this way, beginners can imitate the skilled person's actions and practice surgery, sports competitions, or musical instrument performances. In this case, since it is considered important to overlap the three-dimensional object with the object contained in the first stereoscopic image, it is preferable to composite the first stereoscopic image with the three-dimensional object as described in (B) above.

[0097] Alternatively, the first stereoscopic image could be a video of an actor, actress, singer, or similar person, and the three-dimensional object could be the user's hand, allowing the user to shake hands with the actor or similar person in virtual space by manipulating the hand, which is the three-dimensional object. In this case, the second stereoscopic image is a composite image of the actor or similar person and the hand, which is the three-dimensional object manipulated by the user. In this way, the user could, for example, shake hands with a celebrity. In this case, it is also considered important to overlap the position of the three-dimensional object corresponding to the user's hand with the hand of the celebrity or similar person included in the first stereoscopic image; therefore, it is preferable to perform the composite of the first stereoscopic image and the three-dimensional object as described in (B) above.

[0098] Alternatively, the first stereoscopic image can be a still image of the interior of a building, such as a photograph or CG image, and the three-dimensional object can be furniture. The user manipulates the furniture to change its arrangement, generating a second stereoscopic image showing the state of the furniture arranged inside the building. The user can then observe the arrangement of the furniture in three dimensions through this second stereoscopic image. In this case, it is also considered important to overlap the three-dimensional object with the object contained in the first stereoscopic image; therefore, it is preferable to composite the first stereoscopic image and the three-dimensional object as described in (B) above.

[0099] Alternatively, the first stereoscopic image can be a video captured by a stereoscopic camera worn by a user in an actual space such as an engineering site or repair site. The three-dimensional object can be an object that serves as a pointer for a skilled worker to indicate a position on the first stereoscopic image or an object used in the actual space. The second stereoscopic image is an image composited with the three-dimensional object into the first stereoscopic image in the actual space. The second stereoscopic image can be viewed by both a skilled worker and a user in the actual space. The skilled worker can instruct the user in the actual space by manipulating the three-dimensional object, or demonstrate the procedures that the user in the actual space should perform using the three-dimensional object. Furthermore, the user in the actual space can perform their work appropriately in the actual space by observing the second stereoscopic image containing the three-dimensional object manipulated by the skilled worker. In this case, the image compositing device 1 can also receive the first stereoscopic image in real time and output the second stereoscopic image in real time. In this case, it is also considered important to overlap the three-dimensional object with the object contained in the first stereoscopic image, so it is preferable to perform the compositing of the first stereoscopic image and the three-dimensional object as described in (B) above.

[0100] Furthermore, several usage examples are shown here, but it is obvious that the image synthesis device 1 can be used for other examples of synthesizing the first stereoscopic image and the three-dimensional object, and naturally, the image synthesis device 1 can also be used in situations other than those described above.

[0101] Alternatively, the image synthesis apparatus 1 can also perform a determination of the positional relationship between an object contained in the first stereoscopic image and a three-dimensional object configured accordingly to the operation received by the receiving unit 12. This determination of positional relationship could be, for example, a touch determination to determine whether the object in the first stereoscopic image and the three-dimensional object touch, an overlap determination to determine whether they overlap, or other determinations regarding their positional relationship. In this case, such as in... Figure 9 As shown in the diagram, the image synthesis apparatus 1 may further include a determination unit 16. Furthermore, in this case, adjusting the positional relationship between the object contained in the first stereoscopic image and the three-dimensional object becomes important; therefore, it is preferable to synthesize the first stereoscopic image and the three-dimensional object as described in (B) above.

[0102] The determination unit 16 determines the position of an object contained in the first stereoscopic image in the three-dimensional virtual space based on the first right-eye image and the first left-eye image configured in the three-dimensional virtual space. This position determination can, for example, be based on corresponding points in the first right-eye image and the first left-eye image, using the positions of these corresponding points in the three-dimensional virtual space and their corresponding positions in the right-eye and left-eye viewpoints. For example, the position in the three-dimensional virtual space corresponding to a pair of corresponding points in the first right-eye image and the first left-eye image can be the intersection point of the line connecting the corresponding point in the first right-eye image to the right-eye viewpoint and the line connecting the corresponding point in the first left-eye image to the left-eye viewpoint. Furthermore, if two lines do not intersect due to errors, the midpoint of the positions on the two lines closest to each other can be used as the position in the three-dimensional virtual space corresponding to a pair of corresponding points in the first right-eye image and the first left-eye image. Since this method of determining points in the three-dimensional space corresponding to corresponding points in the stereoscopic image is well-known, its detailed explanation is omitted. The determination unit 16 can specify a position in a three-dimensional virtual space corresponding to each pixel contained in the first right-eye image and the first left-eye image by performing this processing on each corresponding point of the first stereoscopic image. Furthermore, such specified positions typically form surfaces in the three-dimensional virtual space. This is because the structure of the back side is not known in the stereoscopic image. Additionally, the determination unit 16 can also specify objects contained in the image by performing contour extraction or the like in the first stereoscopic image, specifying the position for each specified object.

[0103] When the position of an object contained in the first stereoscopic image in the three-dimensional virtual space is specified, the determination unit 16 determines the relationship between that specific position and the position of the three-dimensional object in the three-dimensional virtual space. This determination may be at least one of touch determination and overlap determination. For example, the determination unit 16 may determine that the two objects touch if a surface formed by the position of the object contained in the first stereoscopic image in the three-dimensional virtual space overlaps with at least a portion of the three-dimensional object in the three-dimensional virtual space. Alternatively, the determination unit 16 may determine that the two objects overlap if a surface formed by at least a portion of the position of the object contained in the first stereoscopic image in the three-dimensional virtual space overlaps with at least a portion of the surface of the three-dimensional object in the three-dimensional virtual space. In this case, for example, an index indicating the degree of overlap may be obtained. This index may be, for example, an index indicating the degree of overlap where the overlap is 100% when the surface formed by the position of the object contained in the first stereoscopic image in the three-dimensional virtual space overlaps most with the surface of the three-dimensional object in the three-dimensional virtual space.

[0104] The output unit 15 can also output a judgment result from the judgment unit 16 regarding the positional relationship between the object included in the first stereoscopic image and the three-dimensional object. This output can be, for example, the result of a touch judgment, the result of an overlap judgment, or, if the touch judgment result indicates that the two objects have touched, an output indicating that they have touched; or, if the overlap judgment result indicates that they have overlapped, an output indicating that they have overlapped, or an index indicating the degree of overlap. More specifically, the output unit 15 can overlay the display corresponding to the judgment result with the second stereoscopic image, vibrate the controller operated by the user based on the judgment result, output a sound corresponding to the judgment result, or perform other outputs corresponding to the judgment result. For example, such as... Figure 6B As shown in the diagram, when a user operates a robotic arm as a three-dimensional object, and the robotic arm encounters a cone-shaped object contained in the first stereoscopic image, it can also display an indication of the collision (e.g., a flickering of the entire screen), or output vibration or sound indicating the collision. Additionally, for example, as... Figure 8A , Figure 8B As shown, when a user manipulates forceps, which is a three-dimensional object, and these forceps overlap with forceps included in a first stereoscopic image, which is an image of a simulated surgery, the system displays an indication of the overlap (e.g., a display of an overlapping graphic) or outputs an index indicating the degree of overlap. In this way, by determining the relationship between the position of the object included in the first stereoscopic image and the position of the three-dimensional object, the user manipulating the three-dimensional object can, for example, confirm the position of the three-dimensional object and know whether the three-dimensional object is being manipulated appropriately.

[0105] Furthermore, in section (B) above, the case where the first right-eye image 22a and the first left-eye image 22b are configured in a three-dimensional virtual space such that points corresponding to infinity overlap, was described. However, this is not always the case. For example, if a three-dimensional object is located in front of the viewpoint (i.e., on the viewpoint side) compared to a position separated from the viewpoint by a predetermined distance, the first right-eye image 22a and the first left-eye image 22b can also be configured in the three-dimensional virtual space such that points corresponding to that predetermined distance overlap. For example, in Figure 10 In the context of a three-dimensional virtual space, when the first right-eye image 22a-1 and the first left-eye image 22b-1, shown by dashed lines, are configured such that points corresponding to infinity overlap, as shown in the example... Figure 10As shown in the first right-eye image 22a-2 and the first left-eye image 22b-2, the first right-eye image 22a-1 and the first left-eye image 22b-1 can also be configured by moving the same distance in the direction indicated by the arrow in their respective planar directions. More specifically, the first right-eye image 22a-2 can be moved to the right from the first right-eye image 22a-1, and the first left-eye image 22b-2 can be moved to the left from the first left-eye image 22b-1. Moreover, it is assumed that in the first left-eye image 22b-2 and the first left-eye image 22b-2, points corresponding to positions N meters apart from viewpoints 23a and 23b are arranged in a three-dimensional virtual space, overlapping each other. Here, N is a positive real number. In addition, in Figure 10 In this context, assuming the line of sight lies within the horizontal plane, it represents the view from the direction of the plumb bob. Figure 11 , Figure 12 Similarly, different reference numerals are used for each configured position, such as first right eye images 22a-1, 22a-2, etc., but without specifically distinguishing the position, it can also be called first right eye image 22a. The same applies to first left eye image 22b.

[0106] Furthermore, the first stereoscopic image can be configured to overlap with points corresponding to positions N meters away from viewpoints 23a and 23b. For example, the first left-eye image 22b-2 and the first left-eye image 22b-2 can be configured to overlap with objects existing N meters away from the camera when the first stereoscopic image is captured. Additionally, for example, parallax, the focal length of the stereoscopic camera, or the distance between the stereoscopic cameras can be used to specify points in the first right-eye image 22a and the first left-eye image 22b that correspond to positions N meters apart. This distance can, for example, be the distance to the straight line connecting viewpoints 23a and 23b. Furthermore, N meters can, for example, be specified by the user. For example, the receiving unit 12 can receive information representing N meters and, based on this, configure the first stereoscopic image in the three-dimensional virtual space to overlap with points corresponding to positions N meters apart. For example, this configuration can also be performed by the configuration unit 13. In this case, the points in the first right-eye image 22a and the first left-eye image 22b corresponding to positions separated by N meters can be specified using parallax, focal distance, distance between viewpoints, or other methods. Furthermore, for example, if information relating the degree of overlap between the two images (the distance between them and the points corresponding to positions separated by that distance) is stored in the storage unit 11, this information is used to obtain the degree of overlap between the two images corresponding to N meters, and the images are configured such that the first right-eye image 22a and the first left-eye image 22b overlap to that degree. Additionally, in Figure 10In the first right-eye image 22a and the first left-eye image 22b, the position of the point corresponding to infinity (i.e., the center point of the image) is indicated by a black dot. Additionally, while the first left-eye image 22b-2 and the first left-eye image 22b-2 originally existed on the same plane (i.e., on the same straight line in the attached diagram), they are slightly offset vertically for ease of explanation. These are in... Figure 11 , Figure 12 The same applies to China.

[0107] In such Figure 10 When the first stereoscopic image is configured as shown in the first right-eye image 22a-2 and the first left-eye image 22b-2, in the three-dimensional virtual space, all objects contained in the first stereoscopic image up to N meters from the viewpoint can be observed stereoscopically in front of the first left-eye image 22b-2 and the first left-eye image 22b-2. Therefore, when attempting to touch an object contained in the first stereoscopic image up to N meters away with a three-dimensional object, it is possible to avoid situations where at least a portion of the three-dimensional object is located on the depth side of the first stereoscopic image and cannot be observed. Thus, instead of arranging the first right-eye image 22a and the first left-eye image 22b in a way that overlaps points corresponding to infinity in the three-dimensional virtual space, the first right-eye image 22a and the first left-eye image 22b can be arranged in a way that overlaps points corresponding to a predetermined distance. The predetermined distance can be, for example, a finite distance.

[0108] In addition, such as Figure 11 As shown, images 22a-2, 22a-3, and 22a-4 for the first right eye are configured such that the viewing angle observed from viewpoint 23a by the right eye remains unchanged. Furthermore, from... Figure 11 It can be seen that the first right-eye image 22a-3, positioned at a greater depth of vision based on the viewpoint, is larger than the first right-eye image 22a-2 positioned at the anterior end of vision. Similarly, the first right-eye image 22a-4 positioned at greater depth of vision is larger than the first right-eye image 22a-3 positioned at the anterior end of vision. Furthermore, it is preferable that the size of each image is changed in a manner that maintains the aspect ratio. The same applies to the first left-eye images 22b-2, 22b-3, and 22b-4. Additionally, in Figure 11In the image, dashed lines 26a and 26b are used to connect viewpoints 23a and 23b with points corresponding to infinity in each image. Furthermore, in the first right-eye image 22a-2 and the first left-eye image 22b-2, points corresponding to a predetermined distance (e.g., N meters) overlap; in the first right-eye image 22a-3 and the first left-eye image 22b-3, points corresponding to infinity overlap; and in the first right-eye image 22a-4 and the first left-eye image 22b-4, there are no overlapping points. Thus, when the first right-eye image 22a and the first left-eye image 22b are configured in a three-dimensional virtual space such that the first straight line 26a connecting the right-eye viewpoint 23a to the point corresponding to infinity in the first right-eye image 22a, and the second straight line 26b connecting the left-eye viewpoint 23b to the point corresponding to infinity in the first left-eye image 22b, are arranged in a different manner than when the first right-eye image 22a and the first left-eye image 22b are configured to overlap with the point corresponding to infinity or a point corresponding to a predetermined distance, it is also possible to achieve the same situation in the three-dimensional virtual space by arranging the first right-eye image 22a and the first left-eye image 22b in a manner that is similar to arranging the first right-eye image 22a and the first left-eye image 22b in a manner that is similar to arranging the first right-eye image 22a to overlap with the point corresponding to infinity or a point corresponding to a predetermined distance. Therefore, the first right-eye image 22a and the first left-eye image 22b can also be arranged in a three-dimensional virtual space such that the first straight line 26a intersects with the second straight line 26b. In this case, for example, it is preferable to arrange the first right-eye image 22a and the first left-eye image 22b in a three-dimensional virtual space, so that the object that is touched or overlapped by the three-dimensional object arranged according to the operation received by the receiving unit 12, which is an object included in the first stereoscopic image, is displayed stereoscopically at a position closer to the front of the first right-eye image 22a and the first left-eye image 22b. For example, if the distance to the object touched or overlapped by the three-dimensional object is less than N meters, the first stereoscopic image can be arranged in the three-dimensional virtual space in such a way that the point corresponding to N meters or the point corresponding to a distance greater than N meters overlaps with the object. Alternatively, the first stereoscopic image can be arranged at a position closer to the depth side (i.e., farther from the viewpoint side) than the intersection of the first and second straight lines 26a and 26b, as in the first right-eye image 22a-4 and the first left-eye image 22b-4.

[0109] Next, the effect of moving the first right-eye image 22a and the first left-eye image 22b in the plane direction will be explained. Figure 12The image shows a first right-eye image 22a-2 and a first left-eye image 22b-2, and a first right-eye image 22a-5 and a first left-eye image 22b-5 that are moved by the same distance in the direction of a right-hand arrow and a left-hand arrow, respectively. Point 31a is the intersection of lines 26a and 26b connecting the points of the first right-eye images 22a-2 and 22b-2 at infinity and the viewpoints 23a and 23b, respectively. Point 31b is the intersection of lines 26a and 26b connecting the points of the first right-eye images 22a-5 and 22b-5 at infinity and the viewpoints 23a and 23b, respectively. Points 31a and 31b are respectively located at infinity in the three-dimensional virtual space corresponding to the first right-eye image 22a-2 and the first left-eye image 22b-2, and at infinity corresponding to the first right-eye image 22a-5 and the first left-eye image 22b-5. Therefore, when the first stereoscopic image is configured as shown in the first right-eye image 22a-2 and the first left-eye image 22b-2, the user perceives as if all objects contained in the first stereoscopic image exist between viewpoints 23a and 23b and point 31a. Similarly, when the first stereoscopic image is configured as shown in the first right-eye image 22a-5 and the first left-eye image 22b-5, the user perceives as if all objects contained in the first stereoscopic image exist between viewpoints 23a and 23b and point 31b. Therefore, by moving the first right-eye image 22a and the first left-eye image 22b in a direction of reduced overlap or separation, the depth of the display range of objects contained in the first stereoscopic image becomes longer. On the other hand, by moving the first right-eye image 22a and the first left-eye image 22b in a direction that increases overlap or in a direction that brings the two images closer together, the depth of the display range of objects contained in the first stereoscopic image becomes shorter. Therefore, for example, when attempting to overlap a three-dimensional object that is the object of manipulation with various objects contained in the first stereoscopic image, the first right-eye image 22a and the first left-eye image 22b can be configured to make the depth of the display range of objects contained in the first stereoscopic image shorter. According to this configuration, the amount of movement of the three-dimensional object when overlapping the three-dimensional object with various objects contained in the first stereoscopic image can be reduced. For example, the receiving unit 12 can also receive information from the user indicating a change in the distance from viewpoints 23a, 23b to the intersection of lines 26a, 26b. Moreover, the configuration unit 13 can also change the configuration of the first right-eye image 22a and the first left-eye image according to the received information, so that the position of the intersection of lines 26a, 26b changes. In this way, for example, the depth of the first stereoscopic image can also be changed according to input from the user. The depth perception of the first stereoscopic image can also be adjusted by configuring the first right-eye image 22a and the first left-eye image 22b in such a way that the straight lines 26a and 26b intersect.

[0110] Next, the case where the first right-eye image 22a and the first left-eye image 22b are configured in a three-dimensional virtual space based on the distance of the object contained in the first stereoscopic image will be described. In this case, as Figure 13 As shown, the image synthesis apparatus 1 may further include a specific part 17. Furthermore, if using... Figure 9 As explained, in Figure 13 The image synthesis apparatus 1 shown may further include a determination unit 16, and an output unit 15 outputs an output related to the determination result based on the determination unit 16.

[0111] The specific unit 17 specifies the distance to a defined object contained in the first stereoscopic image. This distance can be calculated, for example, using the parallax in the first right-eye image 22a and the first left-eye image 22b associated with the defined object, the focal length of cameras 21a and 21b, the distance between cameras 21a and 21b, etc. In this way, as long as the object whose distance is measured in the first stereoscopic image is determined, the distance to that object can be calculated. Furthermore, this calculated distance can be converted into a distance in three-dimensional virtual space for use in later stages of processing.

[0112] The object used for distance measurement could be any object contained in the first stereoscopic image. In this case, for example, the targeting section 17 could also target individual objects contained in the first stereoscopic image, and assign a distance to each of these targets. Object targeting could be performed, for example, by contour extraction or image segmentation. Furthermore, the targeting section 17 could also be used as a background such as the sky or road contained in the first stereoscopic image for object targeting.

[0113] Furthermore, the object used for distance measurement can be an object specified by the user. In this case, the receiving unit 12 can also receive information specifying the object. Moreover, the object specified by the information received by the receiving unit 12 can also be the object used for distance measurement. The information specifying the object can be, for example, information specifying the position of the object in the image, or information specifying the type of object (e.g., "hand" or "ball"). For example, the position of the object can be specified by specifying a point or a rectangular area. In addition, the acceptance of information specifying the type of object can be, for example, acceptance from an input device such as a keyboard, reading information from a storage medium, acceptance of speech recognition results related to speech input from a microphone, etc., or acceptance of other information specifying the type of object. In this case, the receiving unit 12 can, for example, accept multiple types of acceptance, such as acceptance of operations on three-dimensional objects and acceptance of information specifying the object.

[0114] When the information specifying the object is the location information of the specified object, the specifying unit 17 can, for example, specify the object displayed at the specified location and specify the distance of the specified object, or it can specify the location in the first right-eye image 22a and the first left-eye image 22b corresponding to the specified location and specify the distance associated with the specified location. Alternatively, when the information specifying the object is the type of a specific object, the specifying unit 17 can, for example, specify objects of that type in the first right-eye image 22a and the first left-eye image 22b and specify the distance of the specific object. Methods for specifying the type of object to be received can include, for example, specifying the location of an object in an image through pattern matching, specifying a region of the type of object to be received based on the type corresponding to each pixel of the image specified through image segmentation, or specifying a region in an image associated with an object of the type of object to be received through object detection, etc.

[0115] When the information specifying the object is the location of the specified object, for example, distance determination can be performed in the frame where that location is specified. Alternatively, when the information specifying the object is the type of the specified object, the determination of the object as the distance measurement object can be performed, for example, when the first stereoscopic image is video, either on a specific frame or repeatedly on multiple frames. In the former case, for example, when the video output as a second stereoscopic image is received by the receiving unit 12, the object can be determined at that time point in the frame. In the latter case, when the object is determined on multiple frames, for example, the object can be determined at predetermined frame intervals. Furthermore, when the first stereoscopic image is received in real time and the second stereoscopic image is generated in real time using the received first stereoscopic image, object determination can be performed in real time, for example. On the other hand, when the first stereoscopic image is pre-stored in the storage unit 11, for example, object determination can be performed in advance until the second stereoscopic image is generated.

[0116] In the three-dimensional virtual space, the first right-eye image 22a and the first left-eye image 22b can also be configured to overlap points corresponding to distances greater than the longest distance specified by the specifying unit 17. In this way, the object specified by the specifying unit 17 is displayed in the three-dimensional virtual space at a position closer to the front of the first stereoscopic image, for example, allowing the three-dimensional object to overlap with that object. Furthermore, when specifying the distances of multiple objects, it is preferable to configure the first stereoscopic image to overlap points corresponding to distances greater than the longest of the multiple distances. Additionally, when specifying the distances of objects for multiple frames, it is preferable, for example, to configure the first stereoscopic image to overlap points corresponding to distances greater than the longest of the specified distances. Furthermore, the distance greater than the specified longest distance can be, for example, the distance by adding a predetermined distance to the specified longest distance. The predetermined distance as the addition object can be, for example, a length greater than the degree of positional misalignment that can occur when performing the operation to overlap the three-dimensional object with an object included in the first stereoscopic image. In this way, even if such positional misalignment occurs, it is possible to prevent the three-dimensional object from being located on the back side of the first stereoscopic image. Alternatively, the configuration of this first stereoscopic image can also be performed by the configuration unit 13.

[0117] Furthermore, the specific unit 17 can be omitted to avoid placing the three-dimensional object behind the first right-eye image 22a and the first left-eye image 22b. In this case, when the first stereoscopic image is configured in the three-dimensional virtual space such that points corresponding to a predetermined distance overlap, the configuration unit 13 may increase the predetermined distance and configure the first stereoscopic image when the distance to the three-dimensional object being operated on approaches the predetermined distance (i.e., when the three-dimensional object approaches the first stereoscopic image). For example, if the distance obtained by adding a predetermined distance to the distance from the viewpoints 23a and 23b to the three-dimensional object reaches the predetermined distance, the configuration unit 13 may reconfigure the first stereoscopic image in the three-dimensional virtual space such that points corresponding to a distance longer than the predetermined distance overlap. The distance longer than the predetermined distance may be, for example, the distance obtained by adding a predetermined distance to a predetermined distance. Furthermore, the configuration unit 13 may also reconfigure the first stereoscopic image in real time when generating or outputting the second stereoscopic image. Therefore, the first stereoscopic image may be reconfigured repeatedly according to the operation of the three-dimensional object.

[0118] As a method for reconfiguring the first stereo image, for example, there exist methods such as: Figure 11 As shown, a method for moving the first right eye image 22a and the first left eye image 22b towards the depth side without changing the viewing angle from viewpoints 23a and 23b; and as shown Figure 12As shown, this method moves the first right-eye image 22a and the first left-eye image 22b in the plane direction of the image without changing the distance from viewpoints 23a and 23b. In the latter case, by changing the arrangement of the first stereoscopic image, the depth perception of the objects contained in the first stereoscopic image changes, and consequently, the composite position of the three-dimensional objects arranged in the three-dimensional virtual space also changes. Therefore, from the viewpoint of not producing these changes, it is preferable to use a method such as... Figure 11 The method shown reconfigures the first stereoscopic image by moving its position towards the depth side without changing the viewing angle. Furthermore, by moving the first stereoscopic image a shorter distance, the 3D object can be positioned away from the back side of the first stereoscopic image because the first stereoscopic image is configured such that lines 26a and 26b intersect. Therefore, in this case, it is also preferable to reconfigure the first stereoscopic image so that lines 26a and 26b intersect.

[0119] Furthermore, when the three-dimensional object is prevented from being located on the back side of the first stereoscopic image by reconfiguring the first stereoscopic image, for example, when the first right eye image 22a and the first left eye image 22b are configured in the three-dimensional virtual space in such a way that points corresponding to a predetermined distance overlap, the first stereoscopic image is only reconfigured to make the predetermined distance longer, depending on the position of the three-dimensional object. Even if the position of the three-dimensional object is close to the viewpoints 23a and 23b, the first stereoscopic image is not reconfigured to make the predetermined distance shorter.

[0120] In addition, naturally, for example, the first stereoscopic image captured by a pair of cameras arranged in a manner that makes the optical axes intersect can be operated as follows: the first stereoscopic image can be arranged in a three-dimensional virtual space in a manner that makes the first straight line 26a intersect with the second straight line 26b; the stereoscopic image can be arranged in a three-dimensional virtual space in a manner that makes the points corresponding to a specified distance overlap; the specified distance can be made greater than the longest distance specified by the specific part 17; or the first stereoscopic image can be reconfigured when a three-dimensional object is close to the first stereoscopic image; etc.

[0121] Furthermore, in the above embodiments, each process or function can be implemented either by centralized processing by a single device or a single system, or by decentralized processing by multiple devices or multiple systems.

[0122] Furthermore, in the above embodiments, the information exchange between the constituent elements can be performed in various ways. For example, if the two constituent elements exchanging the information are physically different, it can be done by outputting information from one constituent element and receiving information from the other constituent element. Alternatively, if the two constituent elements exchanging the information are physically the same, it can be done by moving from the processing stage corresponding to one constituent element to the processing stage corresponding to the other constituent element.

[0123] Furthermore, in the above embodiments, even if not explicitly stated in the above description, information related to the processing performed by each component can be temporarily or permanently stored in a storage medium (not shown). This includes information received, acquired, selected, generated, sent, or received by each component, as well as information such as thresholds, mathematical formulas, and addresses used by each component in the processing. Alternatively, information can be stored in the storage medium (not shown) by each component or by an accumulation unit (not shown). Furthermore, information can be read from the storage medium (not shown) by each component or by a read unit (not shown).

[0124] Furthermore, in the above embodiments, where the user can also change the information used in each component, such as the thresholds or addresses used in processing, various setting values, etc., the user may appropriately change this information even if it is not explicitly described in the above description, or it may not be. When the user can change this information, such change can be achieved, for example, through a receiving unit (not shown) that receives a change instruction from the user and a change unit (not shown) that changes the information according to the change instruction. The acceptance of the change instruction based on the receiving unit (not shown) can be, for example, acceptance from an input device, reception of information transmitted via a communication line, or acceptance of information read from a specified storage medium.

[0125] Furthermore, in the above embodiments, when two or more constituent elements included in the image synthesis apparatus 1 have communication devices or input devices, the two or more constituent elements may have a single device physically, or they may have separate devices.

[0126] Furthermore, in the above embodiments, each component can be constructed using dedicated hardware, or, for components that can be implemented using software, can be implemented by executing a program. For example, each component can be implemented by reading and executing a software program stored in a storage medium such as a hard disk or semiconductor memory by a program execution unit such as a CPU. During execution, the program execution unit can also execute the program while accessing the storage unit or storage medium. Moreover, the software implementing the image synthesis apparatus 1 in the above embodiments is a program as follows. In other words, this program is used to enable a computer capable of accessing a storage unit containing a first stereoscopic image having a first right-eye image and a first left-eye image to perform the following steps: accepting an operation to configure a three-dimensional object in a three-dimensional virtual space; configuring the three-dimensional virtual space containing the first right-eye image and the first left-eye image in such a way that a first straight line connecting the right-eye viewpoint and a point corresponding to infinity in the first right-eye image, and a second straight line connecting the left-eye viewpoint and a point corresponding to infinity in the first left-eye image, and configuring a three-dimensional object corresponding to the accepted operation; generating a second stereoscopic image in the three-dimensional virtual space containing the three-dimensional object, which includes a second right-eye image as an image viewed from the right-eye viewpoint while displaying the first right-eye image, and a second left-eye image as an image viewed from the left-eye viewpoint while displaying the first left-eye image; and outputting the second stereoscopic image.

[0127] Furthermore, the functions implemented in the above-described procedure do not include those that can only be implemented through hardware. For example, functions that can only be implemented through hardware such as modems or interface cards in information receiving units or information output units are not included in the functions implemented by the above-described procedure.

[0128] Furthermore, the program can be executed either by downloading it from a server or by reading it from a program stored on a specified storage medium (e.g., an optical disc or disk such as a CD-ROM, a semiconductor memory, etc.). Additionally, the program can also be used as a component of a program product.

[0129] Furthermore, the computer executing the program can be a single computer or multiple computers. That is, it can be processed centrally or distributed.

[0130] Figure 14 This is a schematic diagram showing an example of the appearance of the computer that executes the above-described program and implements the image synthesis apparatus 1 of the above-described embodiments. The above-described embodiments can also be implemented by computer hardware and a computer program executed thereon.

[0131] exist Figure 14In the computer system 900, there is a computer 901 including a CD-ROM drive 905, a keyboard 902, a mouse 903, and a monitor 904.

[0132] Figure 15 This is a diagram showing the internal structure of computer system 900. Figure 15 In addition to the CD-ROM drive 905, the computer 901 also includes: an MPU (Micro Processing Unit) 911, a ROM 912 for storing programs such as boot programs, a RAM 913 connected to the MPU 911 that temporarily stores application program commands and provides temporary storage space, a hard disk 914 for storing application programs, system programs, and data, and a bus 915 that interconnects the MPU 911, ROM 912, etc. Furthermore, the computer 901 may also include a network card (not shown) that provides connectivity to a LAN or WAN.

[0133] The program that enables the computer system 900 to perform the functions of the image synthesis apparatus 1 described above can also be stored in the CD-ROM 921, inserted into the CD-ROM drive 905, and forwarded to the hard disk 914. Alternatively, the program can be sent to the computer 901 via a network (not shown) and stored in the hard disk 914. The program is loaded into the RAM 913 during execution. Furthermore, the program can be loaded directly from the CD-ROM 921 or via a network. Alternatively, the program can be read into the computer system 900 using other storage media (e.g., DVD) instead of the CD-ROM 921.

[0134] The program may not necessarily include an operating system (OS) or third-party programs that enable the computer 901 to perform the functions of the image synthesis apparatus 1 described above. The program may also simply contain a portion of commands that call appropriate functions or modules in a controlled manner to obtain the desired results. How the computer system 900 operates is well-known, and detailed descriptions are omitted.

[0135] Furthermore, the present invention is not limited to the above-described embodiments, and various modifications can be made; naturally, these modifications are also included within the scope of the present invention.

Claims

1. An image synthesis apparatus, comprising: The storage unit stores a first stereoscopic image having a first right-eye image and a first left-eye image; The processing department handles operations on three-dimensional objects configured in the three-dimensional virtual space. The configuration unit configures a three-dimensional virtual space of the first right-eye image and the first left-eye image in such a way that the first straight line intersects with the second straight line, and configures a three-dimensional object corresponding to the operation received by the receiving unit. The first straight line is a straight line connecting the right-eye viewpoint and the point of the first right-eye image corresponding to infinity, and the second straight line is a straight line connecting the left-eye viewpoint and the point of the first left-eye image corresponding to infinity. The generation unit generates a second stereoscopic image in a three-dimensional virtual space in which the three-dimensional object is configured, which includes a second right-eye image and a second left-eye image. The second right-eye image is an image viewed from a right-eye viewpoint while the first right-eye image is displayed, and the second left-eye image is an image viewed from a left-eye viewpoint while the first left-eye image is displayed. as well as The output unit outputs the second stereoscopic image.

2. The image synthesis apparatus according to claim 1, wherein, In the three-dimensional virtual space, the first right-eye image and the first left-eye image are configured in such a way that points corresponding to a specified distance overlap.

3. The image synthesis apparatus according to claim 2, wherein, Furthermore, it includes a specific portion, which specifies the distance of a defined object contained in the first stereoscopic image. The specified distance is greater than the longest distance specified by the specific part.

4. The image synthesis apparatus according to claim 3, wherein, The defined object is all objects contained in the first stereoscopic image.

5. The image synthesis apparatus according to claim 3, wherein, The receiving department also accepts information from the designated objects. The specified object is the object designated by the information received by the receiving department.

6. The image synthesis apparatus according to claim 2, wherein, When the distance from the viewpoint to the three-dimensional object is close to the predetermined distance, the configuration unit increases the predetermined distance to configure the first stereoscopic image.

7. The image synthesis apparatus according to claim 1, wherein, In the three-dimensional virtual space, the first right-eye image and the first left-eye image are configured in such a way that points corresponding to infinity overlap.

8. The image synthesis apparatus according to any one of claims 1 to 7, wherein, The first stereoscopic image was captured by a pair of cameras configured such that the optical axes are parallel.

9. The image synthesis apparatus according to any one of claims 1 to 7, wherein, The first stereoscopic image was captured by a pair of cameras configured such that their optical axes intersect.

10. The image synthesis apparatus according to any one of claims 1 to 7, wherein, The first stereoscopic image and the second stereoscopic image are videos.

11. The image synthesis apparatus according to any one of claims 1 to 7, wherein, The system further includes a determination unit that, based on the first right-eye image and the first left-eye image disposed in the three-dimensional virtual space, determines the position of an object specifically included in the first stereoscopic image within the three-dimensional virtual space, and performs a determination related to the relationship between that specific position and the position of the three-dimensional object. The output unit also outputs information related to the judgment result of the judgment unit.

12. An image synthesis method, comprising: The steps for handling operations on three-dimensional objects configured in a three-dimensional virtual space; In the step of configuring a three-dimensional virtual space of a first right-eye image and a first left-eye image in a manner that makes a first straight line intersect with a second straight line, and configuring a three-dimensional object corresponding to the operation being performed, the first straight line is a straight line connecting the right-eye viewpoint and a point in the first right-eye image corresponding to infinity, and the second straight line is a straight line connecting the left-eye viewpoint and a point in the first left-eye image corresponding to infinity. In a three-dimensional virtual space configured with the three-dimensional object, the step of generating a second stereoscopic image including a second right-eye image and a second left-eye image, wherein the second right-eye image is an image viewed from a right-eye viewpoint while the first right-eye image is displayed, and the second left-eye image is an image viewed from a left-eye viewpoint while the first left-eye image is displayed; and The step of outputting the second stereoscopic image.

13. A storage medium storing a program that enables a computer capable of accessing a storage unit storing a first stereoscopic image having a first right-eye image and a first left-eye image to perform the following steps: The steps for handling operations on three-dimensional objects configured in a three-dimensional virtual space; In the step of configuring a three-dimensional virtual space for the first right-eye image and the first left-eye image in such a way that the first straight line intersects with the second straight line, and configuring a three-dimensional object corresponding to the operation being performed, the first straight line is a straight line connecting the right-eye viewpoint and the point of the first right-eye image corresponding to infinity, and the second straight line is a straight line connecting the left-eye viewpoint and the point of the first left-eye image corresponding to infinity. In a three-dimensional virtual space configured with the three-dimensional object, the step of generating a second stereoscopic image including a second right-eye image and a second left-eye image, wherein the second right-eye image is an image viewed from a right-eye viewpoint while the first right-eye image is displayed, and the second left-eye image is an image viewed from a left-eye viewpoint while the first left-eye image is displayed; and The step of outputting the second stereoscopic image.

Images