Information processing device, information processing method, and program

Abstract

To obtain images in which there is no inconsistency in positional relations between virtual objects and real objects, and the virtual objects and the real objects do not overlap.SOLUTION: An information processing device includes: acquisition means for acquiring a real image in which real space is image-captured; generation means for generating, based on data of a virtual object placed in virtual space, a virtual image representing the virtual space in which the virtual object is placed; combining means for combining the real image and the virtual image and generating a combined image; and control means for performing framing control to adjust positions of a real object and the virtual object in the combined image based on whether or not the real object included in the real image and the virtual object included in the virtual image overlap. The combining means combines the real image that is image-captured based on the framing control and the virtual image that is generated based on the framing control.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, technologies such as Augmented Reality (AR) and Mixed Reality (MR) have been used in various devices. For example, AR technology and MR technology are used in devices equipped with a camera such as a Head Mounted Display (HMD), a smartphone, and a tablet terminal. These devices can synthesize a real subject photographed by a camera (hereinafter referred to as a real object) and computer graphics representing a virtual object that does not actually exist (hereinafter referred to as a virtual object) into a single image and display or record it. AR technology and MR technology are used, for example, in games. Devices that realize AR and MR can photograph a person as a subject with a virtual character that does not actually exist as if they were in the same space.

[0003] In addition, an imaging device (hereinafter referred to as an automatic shooting camera) that can automatically control shooting-related operations such as framing and releasing and perform an autonomous shooting operation without depending on a user's operation has become widespread. The automatic shooting camera has become widespread as a product for photographing a desired location for security purposes, such as a security camera. In recent years, the uses of the automatic shooting camera have diversified, and an automatic shooting camera that photographs images of a person moving around for individual users has emerged.

[0004] The information processing apparatus described in Patent Document 1 and the image display apparatus described in Patent Document 2 are assumed to be mounted on an HMD capable of MR display, and present a synthesized virtual object with respect to the real scenery. The information processing apparatus of Patent Document 1 realizes interaction by expressing the contact between a real object and a virtual object. The information processing apparatus acquires two types of models, a contour model that represents the visual contour of an object and a surface model that represents the three-dimensional surface shape of the object, as models representing the shape of the real object. The information processing apparatus performs contact determination between the real object and the virtual object using the surface model, and performs image synthesis using the contour model.

[0005] The image display apparatus of Patent Document 2 adjusts at least one of the focal length of the imaging optical system or the digital zoom ratio with respect to the virtual object in order to maintain the geometric size consistency between the real object and the virtual object, and synthesizes the object in the virtual world into the image of the real world.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] With the spread of AR technology and MR technology, it is desirable for an automatic shooting camera to be able to automatically shoot an image synthesized with a subject existing in the real space and a subject existing in the virtual space. However, when synthesizing a real image and a virtual image, an inconsistency occurs in the positional relationship between the real object and the virtual object, and a synthesized image in which the real object and the virtual object overlap or one of the real object and the virtual object intrudes into the other may be generated.

[0008] The technology described in Patent Document 1 is a technology for realizing interaction from a virtual object in an MR system, and does not assume a state where most of the real object and the virtual object overlap. Further, the technology described in Patent Document 2 is a technology for eliminating the inconsistency regarding the apparent size of the real object and the virtual object, and does not eliminate the inconsistency in the positional relationship and the overlap of the objects. Thus, when a real object and a virtual object are synthesized with an automatic photographing camera, the synthesized image may become an unnatural image due to the real object and the virtual object in the space overlapping due to the inconsistency in the positional relationship.

[0009] Therefore, an object of the present invention is to obtain an image in which there is no inconsistency in the positional relationship between the virtual object and the real object and the virtual object and the real object do not overlap.

Means for Solving the Problem

[0010] A first aspect of the present invention includes an acquisition unit that acquires a real image obtained by imaging a real space, a generation unit that generates a virtual image representing the virtual space in which the virtual object is arranged based on data of the virtual object arranged in the virtual space, a synthesis unit that synthesizes the real image and the virtual image to generate a synthesized image, and a control unit that performs framing control for adjusting positions of the real object and the virtual object in the synthesized image based on whether or not the real object included in the real image and the virtual object included in the virtual image overlap. The synthesis unit synthesizes the real image imaged based on the framing control and the virtual image generated based on the framing control. The information processing apparatus is characterized by this.

[0011] A second aspect of the present invention includes an acquisition unit that acquires a real image obtained by imaging a real space, a generation unit that generates a virtual image representing the virtual space in which the virtual object is arranged based on data of the virtual object arranged in the virtual space, a control unit that controls a first timing for acquiring the real image and a second timing for generating the virtual image based on whether a real object included in the real image overlaps with the virtual object included in the virtual image, and a synthesis unit that synthesizes the real image acquired at the first timing and the virtual image generated at the second timing to generate a synthesized image. The information processing apparatus is characterized by having these components.

Effects of the Invention

[0012] According to the present invention, an image in which there is no inconsistency in the positional relationship between the virtual object and the real object and the virtual object and the real object do not overlap can be obtained.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Referring to FIG. 1, the configurations of the imaging device 1 and the arithmetic device 2 common to each embodiment of the present invention will be described. The user uses the imaging device 1 as the information processing device according to the present invention to obtain a composite image obtained by synthesizing a real image obtained by imaging the real space by the imaging device 1 and a virtual image representing a virtual space in which virtual objects are arranged. The data of the virtual objects arranged in the virtual space is transferred to the imaging device 1 from, for example, an external arithmetic device 2 or the like.

[0015] FIG. 1 is a diagram showing the hardware configuration of the imaging device 1 and the arithmetic device 2. FIG. 2 is an external view of the imaging device 1. In FIGS. 1 and 2, the same reference numerals indicate the same components. The imaging device 1 includes an arithmetic unit 101, an information processing unit 102, a communication unit 103, a primary storage unit 104, a secondary storage unit 105, an imaging unit 106, and a drive unit 107. Each component of the imaging device 1 transmits and receives data via a bus 108.

[0016] The arithmetic unit 101 is a processor such as a CPU, which controls other components. The information processing unit 102 performs arithmetic processing on the image data acquired by the imaging unit 106, arithmetic processing on various evaluation values acquired by the imaging unit 106, arithmetic processing on the data of virtual objects acquired by the communication unit 103, and arithmetic processing on the data used for controlling the drive unit 107.

[0017] The communication unit 103 is a communication interface for transmitting and receiving data of virtual objects and the like to and from the arithmetic device 2. The primary storage unit 104 is, for example, a DRAM (Dynamic Random Access Memory), which temporarily stores data used by the arithmetic unit 101 and the information processing unit 102. The secondary storage unit 105 is, for example, a flash memory, which stores data used by the arithmetic unit 101 and the recorded images processed and encoded by the information processing unit 102.

[0018] The imaging unit 106 collects light from the subject to form an image and converts it into digital data. The drive unit 107 changes the optical system of the imaging unit 106. The drive unit 107 drives the zoom and iris, and may also rotate the angle of view with respect to at least one of the pan axis, tilt axis, and roll axis. The rotation of the pan axis and tilt axis is performed by adjusting the angle of the entire optical system including the optical lens and the imaging element. The rotation of the roll axis is performed by adjusting the angle of the imaging element. Further, the imaging device 1 may be movable in at least one of the up-down, left-right, and front-back directions.

[0019] When the imaging device 1 is an automatic shooting camera, the drive unit 107 performs framing by adjusting the angle of the imaging element, but is not limited thereto. When the imaging device 1 is a device that can move and rotate, such as a drone, the drive unit 107 can perform framing by moving and rotating the body itself. When the imaging device 1 is equipped with an external device such as a gimbal operating pan-tilt head, the drive unit 107 can perform framing by controlling the movement of the operating pan-tilt head. Further, framing may also be performed by trimming the captured image.

[0020] Note that the information processing apparatus according to the present invention is not limited to an apparatus having an imaging unit 106 like the imaging apparatus 1. The information processing apparatus may acquire a real image obtained by imaging the real space from an external apparatus.

[0021] The arithmetic unit 2 has an arithmetic unit 201, a communication unit 202, a primary storage unit 203, and a secondary storage unit 204. Each component of the arithmetic unit 2 transmits and receives data via a bus 205. The arithmetic unit 201 is a processor such as a CPU, for example, and controls other components. The communication unit 202 is a communication interface for transmitting and receiving data such as virtual object data to and from the imaging apparatus 1. The primary storage unit 203 is a DRAM, for example, and temporarily stores data used by the arithmetic unit 201. The secondary storage unit 204 is a flash memory, for example, and stores data used by the arithmetic unit 201.

[0022] The imaging apparatus 1 and the arithmetic unit 2 operate in cooperation to generate a composite image by synthesizing a real image obtained by imaging the real space and a virtual image representing a virtual space in which virtual objects are arranged. The imaging apparatus 1 images the real space and acquires a real image including real objects. Further, the imaging apparatus 1 receives virtual object data from the arithmetic unit 2 and generates a virtual image representing a virtual space in which virtual objects are arranged based on the received virtual object data.

[0023] The imaging apparatus 1 determines whether a real object included in the real image and a virtual object included in the virtual image overlap. Whether a real object and a virtual object overlap includes, for example, a case where the real object and the virtual object are superimposed and displayed, and a case where one of the real object and the virtual object intrudes into the other. The fact that a real object and a virtual object overlap is hereinafter also referred to as "collision".

[0024] The imaging device 1 determines whether a real object and a virtual object collide, and performs framing control based on the determination result. The framing control includes processing for adjusting the positions of the real object and the virtual object in the composite image and controlling them to fit within the composite image. The framing control may include processing for adjusting the position of the real object in the real space and the position of the virtual object in the virtual space.

[0025] The imaging device 1 captures a real image or generates a virtual image based on the framing control. The imaging device 1 synthesizes the real image captured based on the framing control and the virtual image generated based on the framing control to generate a composite image. In this way, the imaging device 1 can generate a composite image in a state where a collision is avoided.

[0026] Note that the imaging device 1 may control the release timing based on whether the real object and the virtual object collide. The release timing includes the timing for capturing a real image and the timing for generating a virtual image.

[0027] <Embodiment 1> Embodiment 1 is an embodiment in which a collision between a real object and a virtual object is determined, and a composite image is generated using the real image and the virtual image at a timing (release timing) when the real object and the virtual object do not collide (do not overlap).

[0028] FIG. 3 is a diagram showing the configuration of the information processing unit 102 according to Embodiment 1. The information processing unit 102 includes an image processing unit 301, a virtual image generation unit 302, an image synthesis unit 303, a collision determination unit 304, a shooting determination unit 305, and a framing adjustment unit 306. Each component of the information processing unit 102 transmits and receives data via the bus 108 of the imaging device 1.

[0029] The image processing unit 301 acquires data of a real image (hereinafter referred to as a real image) that captures a real object and is input from the imaging unit 106 or the primary storage unit 104. The image processing unit 301 performs known image processing such as development processing on the acquired real image. The image-processed real image is output to the primary storage unit 104 or the image composition unit 303.

[0030] The virtual image generation unit 302 generates a virtual image in which a virtual object is arranged based on the data of the virtual object input from the communication unit 103. The data of the virtual object includes, for example, the coordinate information of the representative points of the virtual object (3D global coordinate data) and the coordinate information of the points constituting the virtual object (local coordinate data based on the representative points). Further, the data of the virtual object includes physical characteristic information such as the color, texture, transparency, contact / transmission characteristic information, and mass of the virtual object. The contact / transmission characteristic information is information indicating whether it affects the virtual object, such as causing movement or deformation, or passes through without affecting it when contacting the virtual object. The virtual image generation unit 302 generates (renders) a virtual image that is the result of photographing the virtual object with a virtual camera (hereinafter referred to as a virtual camera) in the virtual space using the data of the virtual object by a known method. Further, the virtual image generation unit 302 makes the shooting parameters such as the shooting angle of view of the virtual camera the same as those of the real image in consideration of the composition with the real image.

[0031] It is preferable that the time in the virtual space is synchronized with the time in the real space. Also, the timing of transmitting the data of the virtual object from the communication unit 202 to the communication unit 103 is preferably synchronized with the timing of acquiring the real image, but it may be asynchronous.

[0032]

[0033] ​The image compositing unit 303 superimposes the virtual image input from the virtual image generation unit 302 on the real image input from the image processing unit 301 by a known method to generate composite image data (hereinafter referred to as the composite image). The generated composite image is output to the primary storage unit 104 or the secondary storage unit 105. The composite image may be output to the primary storage unit 104 or the secondary storage unit 105 after being subjected to known encoding processing such as JPEG for recording.

[0034] The collision determination unit 304 determines the collision between the subject (real object) in the real image input from the image processing unit 301 and the subject (virtual object) in the virtual image input from the virtual image generation unit 302. The collision determination unit 304 outputs the determination result as collision information to the shooting determination unit 305. The collision determination process will be described in detail with reference to the flowchart of FIG. 4 described later.

[0035] The shooting determination unit 305 determines the shooting timing based on the collision information, the position (two-dimensional image coordinates) of the real object in the real image input from the image processing unit 301, and the position (two-dimensional image coordinates) of the virtual object in the virtual image input from the virtual image generation unit 302. The shooting determination unit 305 gives a shooting instruction to the imaging unit 106 based on the determined shooting timing. The shooting determination process will be described in detail with reference to the flowchart of FIG. 4 described later.

[0036] The framing adjustment unit 306 determines the adjustment amount of the shooting angle based on the position of the real object in the real image input from the image processing unit 301, the position of the virtual object in the virtual image input from the virtual image generation unit 302, and the shooting determination result by the shooting determination unit 305. The framing adjustment unit 306 instructs the drive unit 107 to adjust the shooting angle based on the determined adjustment amount of the shooting angle. The framing adjustment process will be described in detail with reference to the flowchart of FIG. 4 described later.

[0037] FIG. 4 is a flowchart illustrating the image synthesis process of Embodiment 1. The image synthesis process is a process of synthesizing a real image obtained by imaging the real space and a virtual image representing a virtual space in which virtual objects are arranged to generate a synthesized image. In the image synthesis process, each component shown in FIGS. 1 and 3 is controlled by parameter setting and operation instructions from the arithmetic unit 101.

[0038] In step S401, the imaging unit 106 performs an imaging process for acquiring a real image for collision determination. The imaging unit 106 performs the imaging process using a frame synchronization signal used in a moving image. Note that the imaging unit 106 may perform the imaging process under direct instruction from the arithmetic unit 101. Also, in order to increase the speed of the collision determination process cycle, the real image may be read out after pixel addition or pixel decimation.

[0039] In step S402, the image processing unit 301 of the information processing unit 102 performs image processing for collision determination on the real image for collision determination acquired in the imaging process of step S401. The image processing unit 301 may simplify the image processing compared to the case of recording the real image in order to increase the speed of the collision determination process cycle.

[0040] In parallel with steps S401 and S402, in step S403, the virtual image generation unit 302 acquires virtual object data from the communication unit 202 of the arithmetic device 2 via the communication unit 103. In step S404, the virtual image generation unit 302 generates a virtual image for collision determination based on the acquired virtual object data.

[0041] The communication unit 103 issues a request to the communication unit 202 at a predetermined period, and the communication unit 202 transmits data of virtual objects existing in the virtual space to the communication unit 103 for each request. The virtual image generation unit 302 extracts data of virtual objects existing within the shooting angle of the real image acquired in step S401 from the data of the virtual objects received by the communication unit 103, and generates a virtual image based on the extracted data of the virtual objects. The virtual image generation unit 302 may simplify the virtual image generation process rather than recording it in order to increase the period of the collision determination process.

[0042] In step S405, the collision determination unit 304 determines whether a real object on the real image and a virtual object on the virtual image are in collision using the real image and the virtual image for collision determination acquired from step S401 to step S404. That is, the collision determination unit 304 determines whether a real object included in the real image and a virtual object included in the virtual image overlap in the two-dimensional plane of the composite image obtained by combining the real image and the virtual image.

[0043] The collision determination between the real object and the virtual object will be described with reference to FIG. 5. In the first embodiment, the collision determination unit 304 determines whether the real object and the virtual object are in collision based on conditions regarding two-dimensional coordinates. FIG. 5(A) shows the real image, FIG. 5(B) shows the virtual image, and FIG. 5(C) shows the composite image obtained by combining the real image and the virtual image.

[0044] The real image in FIG. 5(A) includes a real object 502 within the shooting angle 501. The real object region 504 is a rectangular region surrounding the real object 502 and can be obtained by known subject detection techniques. The real object region 504 is represented by the coordinates (x0, y0) and the coordinates (x1, y1) of two vertices on the diagonal.

[0045] The virtual image in Fig. 5(B) includes a virtual object 503 within the viewing angle 501. The virtual object region 505 is a rectangular region surrounding the virtual object 503 and can be obtained by known subject detection techniques. The virtual object region 505 is represented by the coordinates (x2, y2) and coordinates (x3, y3) of two vertices on the diagonal.

[0046] In the composite image of Fig. 5(C), the real object 502 and the virtual object 503 overlap. The collision determination unit 304 determines that the real object and the virtual object are in collision (overlapping) when there exists a region 506 where the real object region 504 and the virtual object region 505 overlap. The presence or absence of the region 506 where the real object region 504 and the virtual object region 505 overlap can be determined by known conditional expressions.

[0047] In step S405, if it is determined that the real object and the virtual object are in collision, the process returns to steps S401 and S403. When the real object and the virtual object overlap, the imaging device 1 does not synthesize the real image and the virtual image, and acquires a real image for collision determination and generates a virtual image for collision determination until it is determined that the real object and the virtual object are not in collision. When it is determined that the real object and the virtual object are not in collision, the process proceeds to step S406.

[0048] In step S406, the shooting determination unit 305 determines whether it is possible to shoot a real image for generating a composite image for recording. The shooting determination unit 305 can determine whether shooting is possible, for example, based on whether the shooting is appropriate and whether the shooting composition is appropriate. With reference to Figs. 6(A) to 6(D), the determination of whether shooting is possible and the framing control will be described.

[0049] Figure 6(A) shows a real image, Figure 6(B) shows a virtual image, Figure 6(C) shows a composite image obtained by synthesizing the real image and the virtual image, and Figure 6(D) shows a composite image obtained by synthesizing the real image and the virtual image with the shooting angle changed. The real image in Figure 6(A) includes a real object 602 within the shooting angle 601. The virtual image in Figure 6(B) includes a virtual object 603 within the shooting angle 601. Figure 6(C) shows the composite image at the shooting angle 601. Figure 6(D) shows the composite image at the shooting angle 604 adjusted by the framing control.

[0050] In step S406, the shooting determination unit 305 determines, for example, (a) whether the shooting is appropriate and (b) whether the shooting composition is appropriate. If both of these determination results are appropriate and it is determined that shooting is possible, the process proceeds to steps S408 and S410. If the determination result of either of the determination conditions (a) and (b) is inappropriate and it is determined that shooting is not possible, the process proceeds to step S407.

[0051] (a) Whether the shooting is appropriate is determined, for example, by whether the following determination conditions are satisfied. (a1) The real object and the virtual object are facing forward. (a2) It is in focus on the face of the real object (person). (a3) The real object (person) is not closing their eyes. The determination of whether the determination conditions (a1) to (a3) are satisfied can be realized by a known technique.

[0052] The shooting determination unit 305 determines that the shooting is appropriate when all of the determination conditions (a1), (a2), and (a3) are satisfied. Note that the determination conditions for determining whether the shooting is appropriate may include other conditions realized by known techniques such as smile detection processing where the face of the real object (person) is a smile. Further, the shooting determination unit 305 may determine that the shooting is appropriate when at least any one of the determination conditions (a1) to (a3) is satisfied.

[0053] (b) Whether the shooting composition is appropriate is determined, for example, by whether the following determination conditions are satisfied. (b1) The entirety of the real object and the virtual object is detected. (b2) The sizes (areas of the regions) of the real object and the virtual object are larger than a predetermined threshold value. (b3) The real object and the virtual object (the region including them) are present at the center of the viewing angle. Determination of whether the determination conditions (b1) to (b3) are satisfied can be realized by known techniques.

[0054] When all of the determination conditions (b1), (b2), and (b3) are satisfied, the photographing determination unit 305 determines that the photographing composition is appropriate. Note that the determination conditions for determining whether the photographing is appropriate may include other conditions other than the determination conditions (b1) to (b3). Further, the photographing determination unit 305 may determine that the photographing composition is appropriate when at least any one of the determination conditions (b1) to (b3) is satisfied.

[0055] For example, although the entire real object 602 shown in FIG. 6(A) is detected, a part (the leg of the dog) of the virtual object 603 shown in FIG. 6(B) is outside the viewing angle 601 and the whole is not detected. Further, in FIG. 6(C), the region including the real object 602 and the virtual object 603 is not present at the center of the viewing angle 601. Therefore, in the examples of FIGS. 6(A) to 6(C), it is determined that the photographing compositions of the real image and the virtual image are not appropriate.

[0056] In step S407, the framing adjustment unit 306 adjusts the framing. Specifically, the framing adjustment unit 306 generates drive information for adjusting the framing and transmits the generated drive information to the drive unit 107. The drive information includes, for example, zoom drive parameters, iris (aperture) drive parameters, and rotation parameters of the pan axis, tilt axis, and roll axis. These drive information are obtained by known calculation methods.

[0057] The framing adjustment unit 306 generates drive information so that the following control conditions (c1), (c2), and (c3) are satisfied. (c1) The entirety of the real object and the virtual object is detected. (c2) The size (area of the region) of the real object and the virtual object becomes larger than a predetermined threshold value. (c3) The real object and the virtual object (the region including them) exist at the center of the viewing angle.

[0058] In the example of FIG. 6(C), the framing adjustment unit 306 adjusts the zoom to the wide-angle side so that the entire virtual object 603 is detected within the viewing angle 601. Also, the framing adjustment unit 306 rotates the pan axis counterclockwise so that the real object 602 and the virtual object 603 are located at the center of the viewing angle 601 and fit within the composite image. The framing adjustment unit 306 generates drive information so that the control conditions (c1) to (c3) are satisfied and transmits it to the drive unit 107. The drive unit 107 can perform framing control so as to obtain an appropriate shooting composition shown in FIG. 6(D) by adjusting the framing according to the drive information. Note that the framing adjustment unit 306 may gradually adjust the framing by repeating the processes of steps S401 to S407 for a plurality of frames.

[0059] In step S408, the imaging unit 106 performs imaging processing for acquiring a real image for recording. In step S409, the image processing unit 301 of the information processing unit 102 performs image processing on the real image for recording acquired in the imaging processing of step S408.

[0060] In parallel with steps S408 and S409, in step S410, the virtual image generation unit 302 acquires data of the virtual object from the communication unit 202 of the arithmetic unit 2 via the communication unit 103. In step S411, the virtual image generation unit 302 generates a virtual image for recording based on the acquired data of the virtual object.

[0061] The communication unit 103 issues a request to the communication unit 202 at the timing of step S408, and the communication unit 202 transmits data of virtual objects existing in the virtual space to the communication unit 103. The virtual image generation unit 302 extracts data of virtual objects existing within the shooting angle of view of the real image acquired in step S408 from the data of virtual objects received by the communication unit 103, and generates a virtual image based on the extracted data of virtual objects.

[0062] In step S412, the image synthesis unit 303 synthesizes the real image obtained in step S409 and the virtual image obtained in step S411 to generate a composite image for recording. The image synthesis unit 303 can synthesize the real image and the virtual image by, for example, alpha blending. In the example of FIG. 6(D), since there is no overlap between the real object and the virtual object, the blending ratio of the virtual object may be set to 1.0 in the region of the virtual object and 0.0 in the regions other than the virtual object.

[0063] In step S413, the arithmetic unit 101 determines whether to end the shooting. The arithmetic unit 101 ends the shooting, for example, by receiving an instruction to end the shooting from the user. If the shooting is not ended, the process returns to steps S401 and S403. The processes of steps S401 to S412 are repeatedly executed until an instruction to end the shooting is given.

[0064] According to the above-described first embodiment, the imaging device 1 can generate a composite image in a state where the real object and the virtual object do not collide and the framing is adjusted. Therefore, the imaging device 1 can generate a composite image with the subject (real object and virtual object) in a suitable state and at an appropriate angle of view.

[0065] <Embodiment 2> Embodiment 2 is different from Embodiment 1 in the method for determining the collision between a real object and a virtual object. In Embodiment 1, the collision determination unit 304 determines a collision based on the planar overlap between the real object and the virtual object in the two-dimensional plane of the composite image. In contrast, in Embodiment 2, the collision determination unit 304 determines whether the real object and the virtual object collide (overlap) in a three-dimensional space that fuses the real space and the virtual space.

[0066] FIG. 7 is a diagram showing the hardware configuration of the imaging device 1 according to Embodiment 2. The imaging device 1 according to Embodiment 2 has a distance acquisition unit 110 and a position and orientation acquisition unit 111 in addition to the configuration shown in FIG. 1. The same components as those in FIG. 1 are denoted by the same reference numerals, and detailed descriptions thereof are omitted. Also, since the arithmetic device 2 according to Embodiment 2 has the same configuration as that shown in FIG. 1, detailed descriptions thereof are omitted.

[0067] The distance acquisition unit 110 is a sensor that acquires the distance from the imaging device 1 to the subject, which is a real object. The distance acquisition unit 110 is, for example, a phase difference sensor that measures the distance by detecting the phase difference of light rays incident from the subject, a distance sensor that emits light rays toward the subject and measures the distance by measuring the time until the reflected light returns, or a similar sensor. The distance acquisition unit 110 may acquire distance information using the same sensor as the imaging unit 106.

[0068] The position and orientation acquisition unit 111 is a sensor that acquires information on the position and orientation of the imaging device 1. The position and orientation acquisition unit 111 can acquire position information, for example, by a positioning system sensor using a satellite, and acquire orientation information by an acceleration sensor.

[0069] FIG. 8 is a diagram showing the configuration of the information processing unit 102 according to Embodiment 2. In the information processing unit 102 according to Embodiment 2, the processing of the collision determination unit 304 is different from the processing described with reference to FIG. 3. In Embodiment 2, the collision determination unit 304 acquires distance information from the distance acquisition unit 110, and acquires the position information and attitude information of the imaging device 1 from the position and attitude acquisition unit 111. The distance information is the distance from the imaging device 1 to the subject. The collision determination unit 304 executes a collision determination process in a method different from that of Embodiment 1 using the distance information and the position and attitude information.

[0070] The image synthesis process of Embodiment 2 is the same as that of Embodiment 1 except for the process of step S405 shown in FIG. 4. Hereinafter, in step S405 of Embodiment 2, a method for determining whether a real object and a virtual object are in collision (overlapping) will be described.

[0071] In the collision determination of Embodiment 2, the collision determination unit 304 uses the distance information of the real object and the virtual object, obtains the coordinates of the points constituting the real object and the virtual object in the three-dimensional space, and determines whether these objects are in collision.

[0072] FIGS. 9(A) and 9(B) are diagrams for explaining the coordinate systems of the real object and the virtual object used for collision determination. In the collision determination process of Embodiment 2, three coordinate systems, namely, a real space global coordinate system, a virtual space global coordinate system, and a virtual space local coordinate system, are defined.

[0073] The coordinates of the virtual object in the three-dimensional space in which the real space and the virtual space are fused can be obtained based on the coordinates of the real object in the three-dimensional space. That is, the coordinates of the virtual object expressed in the virtual space local coordinate system can be converted into coordinates in the real space global coordinate system based on the coordinates of the real object expressed in the real space global coordinate system.

[0074] ​First, the collision determination unit 304 obtains the three-dimensional coordinates of the points that make up the real object 903. The real space global coordinate system 901 is a coordinate system for representing the coordinates of the real object 903. The real space global coordinate system 901 is defined by the origin 904 and the xr-axis, yr-axis, and zr-axis. The origin 904 can be the optical center of the imaging device 1 (imaging device 902) existing in the real space. The real object 903 is a subject existing in the real space. The point 905 is a point that makes up the real object 903.

[0075] Based on the position and orientation of the imaging device 1 that images the real space and the distance from the imaging device 1 to the real object 903, the collision determination unit 304 can obtain the coordinates of the real object 903 in the three-dimensional space. Specifically, the position and orientation acquisition unit 111 of the imaging device 1 acquires the coordinates of the optical center (origin 904) in the coordinate system 901 and the subject orientation θ. Also, the distance acquisition unit 110 of the imaging device 1 acquires the subject distance d. The collision determination unit 304 obtains the three-dimensional coordinates of the point 905 in the coordinate system 901 based on the coordinates of the optical center (coordinates of the origin 904), the subject orientation θ of the point 905 with respect to the origin 904, and the subject distance d from the origin 904 to the point 905.

[0076] For the points that make up the real object 903 or some of these points, the collision determination unit 304 obtains the coordinates in the same way as the point 905. The points that make up the real object 903 are, for example, points in the three-dimensional space imaged by the imaging device 1 and are a plurality of representative points on the surface of the real object 903. The collision determination unit 304 stores the obtained coordinates in the primary storage unit 104 for the plurality of points that make up the real object 903.

[0077] Next, the collision determination unit 304 obtains the coordinates of the points that make up the virtual object 908. The virtual space global coordinate system 906 is defined by the origin 909 and the xvg-axis, yvg-axis, and zvg-axis. The virtual space global coordinate system 906 is a coordinate system uniquely determined in the virtual space that the arithmetic device 2 has. The virtual space global coordinate system 906 may be the same as the real space global coordinate or may be a different coordinate system that corresponds one-to-one.

[0078] The virtual space local coordinate system 907 is a coordinate system defined by the origin 910 and the xvl axis, yvl axis, and zvl axis, and is included in the virtual space global coordinate system 906. The virtual object 908 is the subject of the virtual object existing in the virtual space global coordinate system 906. The virtual space local coordinate system 907 is used to represent the state of each individual virtual object existing in the virtual space. The virtual space local coordinate system 907 moves in accordance with the movement of the virtual object 908. There may be a plurality of virtual space local coordinate systems 907 in the virtual space global coordinate system 906. The point 911 is a point that constitutes the virtual object 908.

[0079] The imaging device 1 obtains the global coordinates of the point 911 in the real space based on the relationship between the real space global coordinate system 901 and the virtual space global coordinate system 906, the real space global coordinates of the origin 910 of the virtual space local coordinate system 907, and the local coordinates of the point 911 in the virtual space. In this way, the imaging device 1 can acquire the coordinates of the virtual object in the three-dimensional space based on the coordinates of the real object in the three-dimensional space.

[0080] The collision determination unit 304 obtains coordinates for the points that constitute the virtual object 908, or some of these points, in the same manner as the point 911. The points that constitute the virtual object 908 can be the vertices of the polygon when the virtual object is composed of a polygon, or the representative points within each voxel when the virtual object is composed of voxels. The collision determination unit 304 stores the obtained coordinates in the primary storage unit 104 for the plurality of points that constitute the virtual object 908.

[0081] The collision determination unit 304 determines whether the surface formed by each point and adjacent points that constitute the virtual object 908 intersects with the surface formed by each point and adjacent points that constitute the real object in the three-dimensional space. Whether two surfaces intersect in the three-dimensional space can be determined by a known method.

[0082] In step S405 of FIG. 4, the collision determination unit 304 determines that the virtual object and the real object are in collision when the plane formed by the points constituting the virtual object intersects the plane formed by the points constituting the real object. When the plane formed by the points constituting the virtual object does not intersect the plane formed by the points constituting the real object, the collision determination unit 304 determines that the virtual object and the real object are not in collision. According to the collision determination result by the collision determination unit 304, the image composition process after step S406 of FIG. 4 is continued.

[0083] According to the above-described Embodiment 2, the imaging device 1 can determine the collision between the real object and the virtual object in the three-dimensional space in which the real space and the virtual space are fused, rather than the planar collision determination in the composite image. The imaging device 1 can more accurately determine the inconsistency in the positional relationship of the objects in the three-dimensional space and generate a composite image in which there is no collision between the real object and the virtual object in the three-dimensional space. In the following Embodiments 3 to 6, the collision determination unit 304 may determine whether the real object and the virtual object are in collision using the configuration and method of either Embodiment 1 or Embodiment 2.

[0084] <Embodiment 3> Embodiment 3 is an embodiment in which a composite image is generated at a timing when the real object and the virtual object do not collide (do not overlap) by predicting the position of the future virtual object. The arithmetic device 2 holds the data of the virtual space and the data of the virtual object for each time. How the virtual object will behave in the future is predetermined. That is, the data of the virtual object includes movement information such as the position, speed, orientation, shape, and size of the virtual object for each time. The imaging device 1 can predict the position of the virtual object after a predetermined time by acquiring the data of the virtual object up to a predetermined time in the future from the arithmetic device 2.

[0085] FIG. 10 is a diagram showing the configuration of the information processing unit 102 according to Embodiment 3. The information processing unit 102 according to Embodiment 3 has a virtual object position prediction unit 1001 and a timing control unit 1002 in addition to the configuration shown in FIG. 3. For the same configuration as in FIG. 3, the same reference numerals are given and redundant explanations are omitted.

[0086] The virtual object position prediction unit 1001 predicts the position of a future virtual object and outputs information on the predicted position of the virtual object to the collision determination unit 304. Therefore, the collision determination unit 304 can determine whether a real object and a virtual object collide in a composite image of the current real image and the future virtual image.

[0087] The timing control unit 1002 designates, to the imaging unit 106, the timing for acquiring (imaging) a real image based on whether a real object and a virtual object collide (overlap). Further, the timing control unit 1002 designates, via the communication unit 103, to the arithmetic device 2 the timing for acquiring (generating) a virtual image based on whether a real object and a virtual object collide (overlap). Therefore, the timing control unit 1002 can acquire the real image and the virtual image at different timings.

[0088] FIG. 11 is a flowchart illustrating the image composition process of Embodiment 3. The processes of steps S401 to S404, S408, S409, and S410 to S413 are the same as the processes of Embodiment 1 shown in FIG. 4, so redundant explanations are omitted. Hereinafter, the processes of steps S1101 to S1109 and S1110, which are different from Embodiment 1, will be described. Since the processes of steps S1101 to S1109 and S1110, which are different from Embodiment 1, will be described.

[0089] In step S1101, the collision determination unit 304 determines whether a real object included in the real image and a virtual object included in the virtual image overlap in a composite image obtained by combining the real image and the virtual image.

[0090] The collision determination process in step S1101 will be described with reference to FIGS. 12(A) to 12(D). FIG. 12(A) is a diagram showing a composite image obtained by synthesizing a real image and a virtual image at the timing of step S1101. The composite image in FIG. 12(A) includes a real object 1202 and a virtual object 1203 in a virtual image generated at substantially the same timing as the real image captured in step S401 within an angle of view 1201. A real object region 1204 is a rectangular region surrounding the real object 1202. A virtual object region 1205 is a rectangular region surrounding the virtual object 1203. A region 1206 indicates the overlap between the real object region 1204 and the virtual object region 1205.

[0091] In the composite image of FIG. 12(A), the real object 1202 and the virtual object 1203 overlap. When there is a region 1206 where the real object region 1204 and the virtual object region 1205 overlap, the collision determination unit 304 determines that the real object and the virtual object are in collision (overlap). When the real object and the virtual object are in collision, the process proceeds to step S1102. When the real object and the virtual object are not in collision, the process proceeds to step S1107.

[0092] In step S1102, the virtual object position prediction unit 1001 acquires movement information of the virtual object from the arithmetic device 2 via the communication unit 103. The arithmetic device 2 holds position information of the virtual object and controls the position of the virtual object in the virtual space to be moved based on the position information of the virtual object. The virtual object position prediction unit 1001 can acquire the movement information of the virtual object determined to be in collision with the real object from the arithmetic device 2 and predict the future movement of the virtual object. The movement information includes information such as the position, velocity, orientation, shape, and size of the virtual object (points and surfaces constituting the virtual object) in the three-dimensional virtual space at each time. The virtual object position prediction unit 1001 can acquire virtual object data after a predetermined time using the movement information. The predetermined time may be determined based on, for example, the velocity of the virtual object.

[0093] In step S1103, the virtual object position prediction unit 1001 predicts the position of the virtual object after a predetermined time using the data of the virtual object after the predetermined time acquired in step S1102. The virtual object position prediction unit 1001 predicts the position of the virtual object after a predetermined time when the real object included in the real image and the virtual object included in the virtual image overlap.

[0094] In step S1104, the collision determination unit 304 determines whether or not a real object included in the real image and a virtual object after a predetermined time arranged at the position predicted by the virtual object position prediction unit 1001 in the virtual image collide (overlap). If it is determined that they collide, the process proceeds to step S1105. If it is determined that they do not collide, the process proceeds to step S1106.

[0095] A specific example of the process of step S1104 will be described with reference to FIG. 12(B). FIG. 12(B) is a diagram showing a composite image obtained by combining a real image captured in step S401 and a virtual image showing a virtual object region 1207 which is the position of the virtual object after a predetermined time. The real object region 1204 shown within the viewing angle 1201 of the composite image in FIG. 12(B) is a rectangular region surrounding the real object 1202. The virtual object region 1207 is a rectangular region indicating the position of the virtual object after a predetermined time. When the real object region 1204 and the virtual object region 1207 do not overlap, the collision determination unit 304 determines that the real object and the virtual object after a predetermined time do not collide (do not overlap).

[0096] In step S1105, the collision determination unit 304 determines whether or not the time (timing) after a predetermined time is within the prediction period. The prediction period is a preset elapsed period from the time when the real image for determination was acquired in step S401. When the real object and the virtual object do not collide (do not overlap) during the prediction period, a composite image for recording is generated. On the other hand, if the collision between the real object and the virtual object is not resolved (the real object and the virtual object overlap) even after the prediction period has elapsed from the time when the real image was acquired, a composite image for recording is not generated.

[0097] If the time after the predetermined time is within the prediction period, the process returns to step S1102. The virtual object position prediction unit 1001 newly acquires virtual object data after the next predetermined time and predicts the position of the virtual object after the next predetermined time. The next predetermined time is set to a time longer than the current predetermined time. The virtual object position prediction unit 1001 repeats the processes of step S1102 and step S1103 within the prediction period until there is no collision between the real object and the predicted virtual object.

[0098] If the time after the predetermined time is not within the prediction period, the process returns to step S401 and step S403. Since a collision cannot be avoided between the real object included in the real image captured in step S401 and the virtual object within the prediction period, the imaging unit 106 newly acquires a real image for collision determination, and the virtual image generation unit 302 newly generates a virtual image for collision determination.

[0099] In step S1106, based on the determination in step S1104, the collision determination unit 304 designates the time when the virtual object stops colliding with the real object as the timing for generating the virtual image and records it in the primary storage unit 104 or the like.

[0100] In step S1107, based on whether the shooting composition is appropriate, the shooting determination unit 305 determines whether it is possible to shoot a real image for generating a composite image for recording. Since the determination process of whether the shooting composition in step S1107 is appropriate is the same as the determination process of whether the determination conditions (b1) to (b3) described in step S406 of FIG. 4 are satisfied, the description is omitted. If the shooting composition is appropriate, the process proceeds to step S1109. If the shooting composition is not appropriate, the process proceeds to step S1108.

[0101] In step S1108, the framing adjustment unit 306 adjusts the framing. Specifically, the framing adjustment unit 306 generates drive information for adjusting the framing and transmits the generated drive information to the drive unit 107. The drive unit 107 can adjust the framing based on the received drive information.

[0102] Using FIGS. 12(B) and 12(C), a specific example of the process of step S1108 will be described. In FIG. 12(B), the real object area 1204 is included within the viewing angle 1201, but the virtual object area 1207 after a predetermined time is outside the viewing angle 1201. Also, the real object area 1204 surrounding the real object and the virtual object area 1207 surrounding the virtual object do not exist at the center of the viewing angle 1201 but are closer to the edge. The shooting determination unit 305 determines that the shooting composition of the composite image in FIG. 12(B) is not appropriate.

[0103] In the example of FIG. 12(C), the framing adjustment unit 306 adjusts the zoom to the wide-angle side so that the entire virtual object area 1207 is detected within the viewing angle 1210. Also, the framing adjustment unit 306 rotates the pan axis counterclockwise to adjust so that the real object and the virtual object area 1207 are located at the center of the viewing angle 1210 and fit within the composite image. The framing adjustment unit 306 generates drive information for adjusting the framing and transmits it to the drive unit 107. The drive unit 107 adjusts the framing according to the drive information, so that framing control can be performed to obtain an appropriate shooting composition as shown in FIG. 12(C).

[0104] In step S1109, the shooting determination unit 305 determines whether it is possible to shoot a real image for generating a composite image for recording based on whether the shooting is appropriate. The determination process of whether the shooting in step S1109 is appropriate is the same as the determination process of whether the determination conditions (a1) to (a3) described in step S406 of FIG. 4 are satisfied, so the description is omitted. If the shooting is appropriate, the process proceeds to step S408. If the shooting is not appropriate, the process returns to steps S401 and S403.

[0105] Note that if it is possible to avoid a collision between the real object and the virtual object by shifting the timing of acquiring the real object and the timing of generating the virtual object, the framing adjustment process in steps S1107 and S1108 may be omitted.

[0106] In step S408, the timing control unit 1002 designates, as the acquisition timing of the real image, the timing determined to be appropriate for shooting in step S1109 to the imaging unit 106. The imaging unit 106 performs imaging processing for acquiring a real image for recording at the timing designated by the timing control unit 1002. In step S409, the image processing unit 301 of the information processing unit 102 performs image processing on the real image for recording acquired in the imaging processing of step S408.

[0107] In step S1110, the timing control unit 1102 waits until the time of the generation timing of the virtual image recorded in step S1106. When the time recorded in step S1106 arrives, the timing control unit 1102 designates the acquisition timing of the virtual image to the arithmetic unit 2 via the communication unit 103.

[0108] In step S410, the virtual image generation unit 302 acquires data of the virtual object from the communication unit 202 of the arithmetic unit 2 via the communication unit 103. The virtual image generation unit 302 generates a virtual image for recording based on the acquired data of the virtual object.

[0109] The communication unit 103 issues a request to the communication unit 202 at the timing of step S408, and the communication unit 202 of the arithmetic unit 2 transmits data of the virtual object existing in the virtual space to the communication unit 103. The virtual image generation unit 302 extracts data of the virtual object existing within the shooting angle of view of the real image acquired in step S408 from the data of the virtual object received by the communication unit 103, and generates a virtual image based on the extracted data of the virtual object.

[0110] In step S412, the image synthesis unit 303 can generate a composite image for recording as shown in FIG. 12(D) by synthesizing the real image obtained in step S409 and the virtual image obtained in step S411. The image synthesis unit 303 can synthesize the real image and the virtual image by, for example, alpha blending.

[0111] In step S413, the arithmetic unit 101 determines whether to end the shooting. If the shooting is not ended, the process returns to steps S401 and S403. The process in FIG. 11 is repeatedly executed until an instruction to end the shooting is given.

[0112] According to the above-described Embodiment 3, the imaging device 1 can predict the position of a future virtual object, and capture a real object at an optimal timing regardless of the position of the current virtual image, and generate a composite image. Further, the imaging device 1 can generate a suitable composite image based on the shooting angle of view and the states of the subjects (real object and virtual object).

[0113] <Embodiment 4> Embodiment 4 is an embodiment in which, when a real object different from the main subject collides (overlaps) with a virtual object, framing control is performed so that the real object and the virtual object are out of the angle of view of the composite image. Since the configuration of the information processing unit 102 according to Embodiment 4 is the same as the configuration of Embodiment 1 shown in FIG. 3, redundant descriptions are omitted. The imaging device 1 according to Embodiment 4 can obtain a suitable image by removing the subjects (real object and virtual object) in which a collision has occurred from the angle of view.

[0114] FIG. 13 is a flowchart illustrating the image synthesis process of Embodiment 4. The image synthesis process of Embodiment 4 is different from the image synthesis process of Embodiment 1 shown in FIG. 4 in the process of step S405 and the added process of step S1301. The collision determination in step S405 and the framing control in step S1301 in Embodiment 4 will be described with reference to FIGS. 14(A) and 14(B).

[0115] In step S405, the collision determination unit 304 determines whether a real object, which is a subject other than the main subject of the imaging target, has collided with the virtual object. FIG. 14(A) shows an example of a composite image obtained by combining a real image and a virtual image. The composite image shown in FIG. 14(A) includes a real object 1402 of the imaging target, which is the main subject, a real object 1403 other than the main subject, and a virtual object 1404 within the angle of view 1401. Region 1405 indicates the overlap between the real object 1403 and the virtual object 1404. Region 1405 is obtained in the same manner as in the first embodiment from a rectangular region indicating the real object region (not shown) and a rectangular region indicating the virtual object region.

[0116] The collision determination unit 304 detects the main subject, which is a real object of the imaging target, from the real image by a known method. The collision determination unit 304 can detect the main subject, for example, by performing feature point matching using an image indicating the main subject registered in the primary storage unit 104 in advance. Note that the detection of the main subject may be executed by the image processing unit 301.

[0117] In step S1301, the framing adjustment unit 306 generates drive information for setting the region 1405 where the real object 1403 and the virtual object 1404 overlap outside the angle of view. The drive information includes, for example, zoom drive parameters, iris (aperture) drive parameters, and rotation parameters of the pan axis, tilt axis, and roll axis. These drive information are obtained by a known calculation method. The framing adjustment unit 306 transmits the generated drive information to the drive unit 107. The framing adjustment unit 306 can adjust the framing by controlling the drive unit 107 based on the drive information.

[0118] The framing adjustment unit 306 generates drive information so that the following control conditions (d1), (d2), and (d3) are satisfied. (d1) The entire real object of the imaging target is detected (d2) The size (area of the region) of the real object of the imaging target becomes larger than a predetermined threshold value (d3) The difference in position between the center of the region of the real object of the imaging target and the center of the angle of view is smaller than a predetermined threshold value

[0119] In the example of FIG. 14(B), the framing adjustment unit 306 adjusts the zoom to the wide-angle side so that the entire real object 1402 of the subject to be photographed is detected. Further, the framing adjustment unit 306 adjusts by rotating the pan axis counterclockwise and the tilt axis downward so as to remove the overlapping area 1405 from the angle of view 1401. The framing adjustment unit 306 generates drive information so that the control conditions (d1) to (d3) are satisfied and the overlapping area 1405 is removed from the angle of view 1401, and transmits it to the drive unit 107. The drive unit 107 can perform framing control so as to obtain an appropriate shooting composition shown in FIG. 14(B) by adjusting the framing according to the drive information.

[0120] Note that the framing adjustment unit 306 may gradually adjust the framing by repeating the processes of steps S401 to S407 and S1301 for a plurality of frames. Further, when the real object of the subject to be photographed is not detected, the framing adjustment unit 306 may obtain drive information for removing the center coordinates of the overlapping area 1405 from the angle of view 1401 by a known method.

[0121] According to the above-described Embodiment 4, the imaging device 1 can generate a composite image in a state where a real object other than the main subject and a virtual object do not collide, not limited to the real object that is the main subject. Further, the imaging device 1 can generate a suitable composite image based on the shooting angle of view and the states of the subjects (real object and virtual object).

[0122] Note that the composite processing of the real image and the virtual image in Embodiment 4 has been described for the purpose of generating a composite image for recording, but is not limited to the case of recording. The imaging device 1 may have an image display unit such as a display and display the composite image as a live view image.

[0123] <Embodiment 5> Embodiment 5 is an embodiment in which a real object and a virtual object different from the main subject are predicted as to whether they will move and collide, and when it is predicted that they will collide, framing control is performed so that the position where the real object and the virtual object collide is outside the angle of view of the composite image. Since the configuration of the information processing unit 102 according to Embodiment 5 is the same as the configuration of Embodiment 1 shown in FIG. 3, duplicate explanations are omitted.

[0124] The image composition process of Embodiment 5 is the same as the flow of the image composition process of Embodiment 4 shown in FIG. 13, except for the processes of steps S405 and S1301, so duplicate explanations are omitted. With reference to FIGS. 15(A) to 15(C), the collision determination in step S405 and the framing control in step S1301 in Embodiment 5 will be described.

[0125] In step S405, the collision determination unit 304 uses the position information and movement amount information of the real object and the virtual object to determine whether the two objects will collide in the future. The collision determination unit 304 predicts the coordinates of the collision position.

[0126] FIG. 15(A) shows an example of a composite image obtained by combining a real image and a virtual image. The composite image shown in FIG. 15(A) includes a real object 1502 of a shooting target that is the main subject, a real object 1503 other than the main subject, a virtual object 1504 of the shooting target, and a virtual object 1505 different from the virtual object 1504 within the angle of view 1501. Each object is detected by a rectangular region (not shown) surrounding each object, in the same manner as FIGS. 5(A) and 5(B).

[0127] The collision determination unit 304 obtains the respective movement amounts v1 and v2 of the real object 1503 other than the main subject and the virtual object 1505 by a known method. The collision determination unit 304 can obtain the movement amount from the difference in position between frames, for example. Also, for the virtual object, the collision determination unit 304 may obtain the movement amount based on the data of the virtual object acquired from the arithmetic device 2.

[0128] The collision determination unit 304 determines whether the following determination conditions (e1) and (e2) are satisfied. (e1) The difference in the coordinates of the rectangular regions of the real object 1503 and the virtual object 1505 becomes smaller than a predetermined threshold after a predetermined number of frames. (e2) The relative movement amount between the real object 1503 for which collision prediction has been performed and the virtual object 1505 is smaller than a predetermined threshold.

[0129] In the state of FIG. 15(A), when both of the determination conditions (e1) and (e2) are satisfied, the collision determination unit 304 determines that the real object 1503 and the virtual object 1505 collide. The coordinates (x4, y4) of the position where the collision is predicted are calculated, for example, as the midpoint of the center coordinates of the rectangular region indicating the real object 1503 and the rectangular region indicating the virtual object 1505 at the time when the determination conditions (e1) and (e2) are satisfied. When at least one of the determination conditions (e1) and (e2) is not satisfied, the collision determination unit 304 determines that the real object 1503 and the virtual object 1505 do not collide.

[0130] In step S1301, when it is determined that the real object 1503 and the virtual object 1505 collide, the framing adjustment unit 306 performs framing control so as to remove the coordinates (x4, y4) from the viewing angle 1501 as shown in FIG. 15(B). The framing adjustment unit 306 calculates, as drive information for removing the coordinates (x4, y4) from the viewing angle 1501, the drive parameters of zoom, the drive parameters of the iris (diaphragm), and the rotation parameters of the pan axis, tilt axis, and roll axis. The framing adjustment unit 306 can adjust the framing by controlling the drive unit 107 according to the drive information.

[0131] Note that the framing adjustment unit 306 may gradually adjust the framing by repeating the processes of steps S401 to S407 and S1301 for a plurality of frames. Further, when the real object as the shooting target is not detected, the framing adjustment unit 306 may obtain the drive amount for removing the coordinates (x4, y4) of the position where the collision is predicted from the viewing angle 1501 by a known method.

[0132] As shown in FIG. 15(C), when it is determined that the real object 1503 and the virtual object 1505 do not collide, the framing adjustment unit 306 does not perform framing control. When it is determined that the real object 1503 and the virtual object 1505 do not collide, the imaging device 1 may wait until the real object 1503 and the virtual object 1505 go out of the viewing angle 1501 and then generate a composite image.

[0133] According to the above-described Embodiment 5, the imaging device 1 can generate a composite image in which no positional relationship inconsistency occurs by predicting in advance whether the real object and the virtual object collide. Note that the composite processing of the real image and the virtual image in Embodiment 5 has been described for the purpose of generating a composite image for recording, but is not limited to the case of recording. The imaging device 1 may have an image display unit such as a display and display the composite image as a live view image.

[0134] <Embodiment 6> Embodiment 6 is an embodiment in which framing control is performed for each of the real object and the virtual object. The imaging device 1 can avoid a collision between the real object and the virtual object by performing different framing controls for the real object in the real space and the virtual object in the virtual space.

[0135] FIG. 16 is a diagram showing the configuration of the information processing unit 102 according to Embodiment 6. The information processing unit 102 according to Embodiment 6 includes a real space framing adjustment unit 1606 and a virtual space framing adjustment unit 1607 instead of the framing adjustment unit 306. The same components as those in FIG. 3 are denoted by the same reference numerals, and redundant descriptions are omitted.

[0136] The real-space framing adjustment unit 1606 adjusts the framing of the real space. The virtual-space framing adjustment unit 1607 adjusts the framing of the virtual space. In Embodiment 6, the imaging device 1 performs different framing controls for the real space and the virtual space by respective framing adjustment units. The imaging device 1 adjusts the framing so that, for example, a real object included in the real space and a virtual object included in the virtual space do not collide. The framing is adjusted so that a virtual object included in the virtual space does not collide with a real object included in the real space.

[0137] FIG. 17 is a flowchart exemplifying the image synthesis process of Embodiment 6. For the processes of steps S401 to S404 and S408 to S413 in the image synthesis process of Embodiment 6, since they are the same as the processes of Embodiment 1 shown in FIG. 4, duplicate explanations are omitted. Hereinafter, the processes of steps S1701 to S1705 different from Embodiment 1 will be described.

[0138] The information processing unit 102 according to Embodiment 6, similarly to Embodiment 1, acquires a real image and a virtual image for collision determination in steps S401 to S404.

[0139] In step S1701, the collision determination unit 304 uses the real image and the virtual image for collision determination acquired in steps S401 to S404 to determine whether a real object included in the real image and a virtual object included in the virtual image are colliding (overlapping). The method of collision determination is the same as the process of step S405 in Embodiment 1. If it is determined that the real object and the virtual object are colliding, the process proceeds to step S1702. If it is determined that the real object and the virtual object are not colliding, the process proceeds to step S1703.

[0140] In step S1702, the real-space framing adjustment unit 1606 and the virtual-space framing adjustment unit 1607 perform framing control of the real space and the virtual space with different adjustment amounts. Here, the framing for the real space and the framing for the virtual space are processes of adjusting the framing by controlling the movement with respect to at least any one of the six axes of the pan axis, tilt axis, roll axis, zoom axis, up-down axis, left-right axis, and front-back axis.

[0141] In step S1703, the shooting determination unit 305 determines whether the shooting composition is appropriate. That is, the shooting determination unit 305 determines whether appropriate framing control is being performed with respect to the position of the subject, etc. The shooting determination unit 305 can determine whether the shooting composition is appropriate by using the determination conditions (b1), (b2), and (b3) described in Embodiment 1. If it is determined that the shooting composition is not appropriate, the process proceeds to step S1704. If it is determined that the shooting composition is appropriate, the process proceeds to step S1705.

[0142] In step S1704, the real-space framing adjustment unit 1606 and the virtual-space framing adjustment unit 1607 perform framing control of the real space and the virtual space with the same adjustment amount.

[0143] In step S1705, the shooting determination unit 305 determines whether the shooting is appropriate. That is, the shooting determination unit 305 determines whether it is the best timing for the subject (the state of the subject is suitable for shooting). The shooting determination unit 305 can determine whether the shooting is appropriate by using the determination conditions (a1), (a2), and (a3) described in Embodiment 1.

[0144] If it is determined that the shooting is not appropriate, the process returns to step S401 and step S403. The information processing unit 102 repeats the processes of steps S401 to S404 and S1701 to S1705 until it is determined that the shooting is appropriate. If it is determined that the shooting is appropriate, the process proceeds to step S408 and step S410. The processes of steps S408 to S413 are the same as the processes described in FIG. 4.

[0145] According to the above-described Embodiment 6, when the real object and the virtual object are in collision, the imaging device 1 can synthesize the real image and the virtual image without waiting for the collision to be resolved. The imaging device 1 can capture a real image, generate a virtual image, and generate a composite image at the best timing of the subject that existed during the time when the real object and the virtual object were in collision. Thus the imaging device 1 can image the subject (real object) in a more appropriate state and generate a composite image without collision with the virtual object.

[0146] Note that although the composite processing of the real image and the virtual image in Embodiment 6 has been described for the purpose of generating a composite image for recording, it is not limited to the case of recording. The imaging device 1 may have an image display unit such as a display and display the composite image as a live view image.

[0147] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine the embodiments.

[0148] In addition, in each of the above-described embodiments, the case where the present invention is applied to a general automatic photographing camera has been described as an example. However, the present invention is not limited to an automatic photographing camera, and the present invention is applicable to any imaging device capable of framing. That is, the present invention is applicable to a personal computer, a tablet terminal, a mobile phone terminal, a portable image viewer, and a digital photo frame. Further, the present invention is also applicable to a music player, a game machine, various robots, a security camera, a network camera, a drone, a gimbal, and the like.

[0149] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0150] The disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) An acquisition unit that acquires a real image obtained by imaging a real space, A generation unit that generates a virtual image representing the virtual space in which the virtual object is arranged based on data of the virtual object arranged in the virtual space, A composition unit that composes the real image and the virtual image to generate a composite image, A control unit that performs framing control to adjust positions of the real object and the virtual object in the composite image based on whether or not the real object included in the real image overlaps with the virtual object included in the virtual image having The composition unit composes the real image imaged based on the framing control and the virtual image generated based on the framing control An information processing apparatus characterized by having. (Configuration 2) When the real object and the virtual object do not overlap, the control means performs the framing control so that the real object and the virtual object are included in the composite image. The information processing apparatus according to Configuration 1, characterized by the above. (Configuration 3) When a second real object different from the real object and a second virtual object different from the virtual object overlap, the control means performs the framing control so that the second real object and the second virtual object are outside the viewing angle of the composite image. The information processing apparatus according to Configuration 1 or 2, characterized by the above. (Configuration 4) The control means predicts whether a second real object different from the real object and a second virtual object different from the virtual object will move and collide, and when it is predicted that they will collide, the control means performs the framing control so that the position where the second real object and the second virtual object collide is outside the viewing angle of the composite image. The information processing apparatus according to Configuration 1 or 2, characterized by the above. (Configuration 5) The control means performs the framing control for each of the real object and the virtual object. The information processing apparatus according to any one of Configurations 1 to 4, characterized by the above. (Configuration 6) The control means further designates a first timing for acquiring the real image and a second timing for generating the virtual image based on whether the real object and the virtual object overlap, and the synthesizing means synthesizes the real image acquired at the first timing based on the framing control and the virtual image generated at the second timing based on the framing control. The information processing apparatus according to any one of Configurations 1 to 5, characterized by the above. (Configuration 7) An acquisition means for acquiring a real image obtained by imaging the real space, Generating means for generating a virtual image representing the virtual space in which the virtual object is arranged, based on data of the virtual object arranged in the virtual space; Control means for controlling a first timing for acquiring the real image and a second timing for generating the virtual image, based on whether or not a real object included in the real image overlaps with the virtual object included in the virtual image; Combining means for combining the real image acquired at the first timing and the virtual image generated at the second timing to generate a combined image An information processing apparatus comprising the same. (Configuration 8) The control means predicts the position of the virtual object after a predetermined time based on the movement information of the virtual object included in the data of the virtual object, and determines whether or not the real object overlaps with the virtual object arranged at the predicted position The information processing apparatus according to Configuration 6 or 7, characterized by the above. (Configuration 9) The control means predicts the position of the virtual object after the predetermined time when the real object and the virtual object overlap The information processing apparatus according to Configuration 8, characterized by the above. (Configuration 10) When the real object does not overlap with the virtual object arranged at the predicted position, the control means designates the timing after the predetermined time as the second timing The information processing apparatus according to Configuration 8 or 9, characterized by the above. (Configuration 11) Even if a preset prediction period has elapsed since the time when the real image was acquired, when the real object and the virtual object overlap, the combining means does not combine the real image and the virtual image The information processing apparatus according to Configuration 8, characterized by the above. (Configuration 12) When the real object and the virtual object overlap, the combining means does not combine the real image and the virtual image The information processing apparatus according to any one of Configurations 1 to 6, characterized by the above. (Configuration 13) The control means determines whether or not the real object and the virtual object overlap in the two-dimensional plane of the composite image. The information processing apparatus according to any one of Configurations 1 to 12, characterized in that. (Configuration 14) The control means determines whether or not the real object and the virtual object overlap in a three-dimensional space in which the real space and the virtual space are fused. The information processing apparatus according to any one of Configurations 1 to 12, characterized in that. (Configuration 15) The control means acquires the coordinates of the virtual object in the three-dimensional space based on the coordinates of the real object in the three-dimensional space. The information processing apparatus according to Configuration 14, characterized in that. (Configuration 16) The control means acquires the coordinates of the real object in the three-dimensional space based on the position and orientation of the imaging device that images the real space and the distance from the imaging device to the real object. The information processing apparatus according to Configuration 15, characterized in that. (Configuration 17) The time in the virtual space is synchronized with the time in the real space. The information processing apparatus according to any one of Configurations 1 to 16, characterized in that. (Method 1) An acquisition step of acquiring a real image obtained by imaging a real space; A generation step of generating a virtual image representing the virtual space in which the virtual object is arranged based on data of the virtual object arranged in the virtual space; A synthesis step of synthesizing the real image and the virtual image to generate a composite image; A control step of performing framing control to adjust the positions of the real object and the virtual object in the composite image based on whether or not the real object included in the real image and the virtual object included in the virtual image overlap. And having In the synthesis step, synthesize the real image captured based on the framing control and the virtual image generated based on the framing control. An information processing method characterized by the above. (Method 2) An acquisition step of acquiring a real image obtained by imaging a real space; A generation step of generating a virtual image representing the virtual space in which the virtual object is arranged based on data of the virtual object arranged in the virtual space; A control step of controlling a first timing for acquiring the real image and a second timing for generating the virtual image based on whether a real object included in the real image overlaps with the virtual object included in the virtual image; A synthesis step of synthesizing the real image acquired at the first timing and the virtual image generated at the second timing to generate a synthesized image An information processing method characterized by including the above. (Program) A program for causing a computer to function as each means of the information processing apparatus according to any one of Configurations 1 to 17.

Explanation of Signs

[0151] 1: Imaging device (information processing device), 101: Arithmetic unit, 102: Information processing unit

Claims

1. An acquisition means for acquiring a real image obtained by imaging a real space; A generation means for generating a virtual image representing the virtual space in which the virtual object is arranged, based on data of the virtual object arranged in the virtual space; A composition means for composing the real image and the virtual image to generate a composite image; A control means for performing framing control to adjust the positions of the real object and the virtual object in the composite image based on whether or not the real object included in the real image overlaps with the virtual object included in the virtual image and having the composition means composes the real image imaged based on the framing control and the virtual image generated based on the framing control An information processing apparatus characterized by the above.

2. The control means performs the framing control so that the real object and the virtual object are contained within the composite image when the real object and the virtual object do not overlap. The information processing apparatus according to claim 1, characterized by the above.

3. The control means performs the framing control so that the second real object different from the real object and the second virtual object different from the virtual object are outside the viewing angle of the composite image when the second real object and the second virtual object overlap. The information processing apparatus according to claim 1, characterized by the above.

4. The control means predicts whether or not a second real object different from the real object and a second virtual object different from the virtual object move and collide, and when it is predicted that they collide, performs the framing control so that the position where the second real object and the second virtual object collide is outside the viewing angle of the composite image. The information processing apparatus according to claim 1, characterized by the above.

5. The control means performs the framing control for each of the real object and the virtual object. The information processing apparatus according to claim 1, characterized by the above.

6. The control means further designates a first timing for acquiring the real image and a second timing for generating the virtual image based on whether or not the real object and the virtual object overlap, and the composition means composes the real image acquired at the first timing based on the framing control and the virtual image generated at the second timing based on the framing control. The information processing apparatus according to claim 1, characterized by the above.

7. An acquisition means for acquiring a real image obtained by imaging a real space; A generation means for generating a virtual image representing the virtual space in which the virtual object is arranged, based on data of the virtual object arranged in the virtual space; A control means for controlling a first timing for acquiring the real image and a second timing for generating the virtual image based on whether or not a real object included in the real image overlaps with the virtual object included in the virtual image; A composition means for composing the real image acquired at the first timing and the virtual image generated at the second timing to generate a composite image An information processing apparatus characterized by comprising the same.

8. The control means predicts the position of the virtual object after a predetermined time based on the movement information of the virtual object included in the data of the virtual object, and determines whether or not the real object overlaps with the virtual object arranged at the predicted position The information processing apparatus according to claim 6, characterized by the above.

9. The control means predicts the position of the virtual object after the predetermined time when the real object and the virtual object overlap The information processing apparatus according to claim 8, characterized by the above.

10. When the real object does not overlap with the virtual object arranged at the predicted position, the control means designates the timing after the predetermined time as the second timing The information processing apparatus according to claim 8, characterized by the above.

11. Even if a preset prediction period has elapsed from the time when the real image is acquired, when the real object and the virtual object overlap, the composition means does not compose the real image and the virtual image The information processing apparatus according to claim 8, characterized by the above.

12. When the real object and the virtual object overlap, the composition means does not compose the real image and the virtual image The information processing apparatus according to claim 1, characterized by the above.

13. The control means determines whether or not the real object and the virtual object overlap in the two-dimensional plane of the composite image The information processing apparatus according to claim 1, characterized by the above.

14. The control means determines whether or not the real object and the virtual object overlap in a three-dimensional space in which the real space and the virtual space are fused The information processing apparatus according to claim 1, characterized by the above.

15. The control means acquires the coordinates of the virtual object in the three-dimensional space based on the coordinates of the real object in the three-dimensional space. The information processing apparatus according to claim 14, characterized in that.

16. The control means acquires the coordinates of the real object in the three-dimensional space based on the position and orientation of the imaging device that images the real space and the distance from the imaging device to the real object. The information processing apparatus according to claim 15, characterized in that.

17. The time in the virtual space is synchronized with the time in the real space. The information processing apparatus according to claim 1, characterized in that.

18. An acquisition step of acquiring a real image obtained by imaging the real space; A generation step of generating a virtual image representing the virtual space in which the virtual object is arranged based on the data of the virtual object arranged in the virtual space; A synthesis step of synthesizing the real image and the virtual image to generate a synthesized image; Based on whether the real object included in the real image and the virtual object included in the virtual image overlap, adjusting the positions of the real object and the virtual object in the synthesized image A control step of performing framing control And having In the synthesis step, the real image captured based on the framing control and the virtual image generated based on the framing control are synthesized. An information processing method, characterized in that.

19. An acquisition step of acquiring a real image obtained by imaging the real space; A generation step of generating a virtual image representing the virtual space in which the virtual object is arranged based on the data of the virtual object arranged in the virtual space; Based on whether the real object included in the real image and the virtual object included in the virtual image overlap, a control step of controlling the first timing for acquiring the real image and the second timing for generating the virtual image; A synthesis step of synthesizing the real image acquired at the first timing and the virtual image generated at the second timing to generate a synthesized image An information processing method, characterized by comprising.

20. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 17.

Images