IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, CONTROL METHOD, AND PROGRAM

Abstract

To provide an image processing device capable of generating a mixed reality image that reflects the movement of a virtual object and gives little sense of incongruity.SOLUTION: A CPU 117 of an imaging device 100 acquires the amount of motion of a target including a virtual object, and performs exposure control on the basis of the acquired amount of motion. The CPU 117 captures an image of a real space through the exposure control. Then, the CPU 117 synthesizes the captured image of the real space and the image of the virtual object to generate a mixed reality image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, a control method, and a program. [Background technology]

[0002] An imaging device capable of changing the exposure time during shooting has been proposed. Blurring of an image occurs due to the movement of the imaging device or the subject during the exposure period. Although blurring of an image can be a cause of deterioration in image quality due to a decrease in resolution, it can also express a sense of speed and dynamism. Therefore, for example, as a shooting technique, shooting that intentionally generates blur in an image, such as panning, may be performed. Patent Document 1 discloses an electronic camera that determines the exposure time to be used for shooting based on the amount of movement of the subject in order to obtain a panning image. On the other hand, MR (Mixed Reality) presents mixed reality images by overlaying virtual objects on real space. Patent Document 2 discloses an image processing device that changes the characteristics of a blur filter applied to a virtual object according to a motion vector and imaging parameters when a real image is captured. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-245774 A [Patent Document 2] JP 2018-28572 A Summary of the Invention [Problem to be solved by the invention]

[0004] The electronic camera disclosed in Patent Document 1 cannot perform exposure control that takes into account the movement of virtual objects that do not exist in real space. In addition, the image processing device disclosed in Patent Document 2 does not perform processing that takes into account the movement of virtual objects, and therefore cannot generate a mixed reality image that reflects the movement of virtual objects and that creates a natural appearance. The present invention aims to provide an image processing device that can generate a mixed reality image that reflects the movement of virtual objects and that creates a natural appearance. [Means for solving the problem]

[0005] An image processing device according to one embodiment of the present invention has an acquisition means for acquiring an amount of movement of a target including a virtual object, a control means for performing exposure control based on a photometry result and the acquired amount of movement, and a generation means for generating a composite image by combining an image of real space captured by the exposure control with an image of the virtual object. Effect of the Invention

[0006] According to the present invention, it is possible to generate a mixed reality image that reflects the movement of a virtual object and that creates a less awkward feeling. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of the configuration of an image processing device. [Diagram 2] 11 is a flowchart illustrating an example of an image capturing operation in a mixed reality space. [Diagram 3] 11 is a flowchart illustrating an example of a calculation process of an amount of motion. [Figure 4] 11 is a flowchart illustrating an example of a display process of a mixed reality image. [Diagram 5] 11 is a flowchart illustrating an example of a process for generating a mixed reality image. [Figure 6] 11 is a flowchart illustrating an example of a recording process of a mixed reality image. [Figure 7] 11 is a flowchart illustrating an example of a recording process of a mixed reality image. [Figure 8]11 is a flowchart illustrating an example of a recording process of a mixed reality image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Example 1 FIG. 1 is a diagram showing an example of the configuration of an image processing device according to this embodiment. 1, the configuration of an image capturing apparatus 100 as an example of an image processing apparatus will be described. The image capturing apparatus 100 includes an imaging optical system 101 through a position and orientation detection unit 119. The imaging optical system 101 guides subject light to an image sensor 106. The imaging optical system 101 has a focusing lens 102, an aperture 103, and a lens group (not shown). A lens driving unit 104 drives and controls the focusing lens 102 and the lens group (not shown) of the imaging optical system 101. The focusing lens 102 drives in a direction along the optical axis according to a drive control signal from the lens driving unit 104.

[0009] Aperture driving unit 105 drives and controls aperture 103. Aperture 103 is driven to a predetermined aperture diameter in accordance with a drive control signal from aperture driving unit 105, and adjusts the amount of light. Image sensor 106 photoelectrically converts the received light flux and outputs a signal related to a captured image. Image sensor 106 has a pixel array in which unit pixels are arranged two-dimensionally. A group of signals output from image sensor 106 is stored in a storage area (not shown) and transferred to various processing units via CPU (Central Processing Unit) 117.

[0010] The CPU 117 controls the overall operation of the imaging device 100. For example, the CPU 117 performs flow control of the overall shooting by the imaging device 100, issues commands to the lens driving unit 104, the aperture driving unit 105, etc. The focus adjustment unit 112 performs a defocus calculation based on a signal related to the captured image, converts the result of the defocus calculation into a lens driving amount, and issues a lens driving signal.

[0011] The instruction unit 113 is an operation unit used by the user for inputting operations. The instruction unit 113 is, for example, a physical switch configured on the body of the imaging device 100, and is used to switch the shooting mode, specify the focus detection position during autofocus, give a shooting preparation instruction, and give a shooting start instruction. For example, when a release switch included in the instruction unit 113 is half-pressed (SW1 is on), a signal indicating that a shooting preparation instruction has been issued is output. Also, when the release switch is fully pressed (SW2 is on), a signal indicating that a shooting start instruction has been issued is output. Note that the instruction unit 113 may have a touch panel built into the display unit 110.

[0012] The photometry unit 115 measures the brightness of a subject in real space. The position and orientation detection unit 119 detects position and orientation information indicating the position and orientation of the imaging device 100. The motion amount calculation unit 114 calculates (acquires) the amount of motion of a target including a virtual object. The target including a virtual object is the subject and the virtual object. The exposure control unit 116 executes exposure control based on the photometry result by the photometry unit 115 and the amount of motion calculated by the motion amount calculation unit 114. In this example, the exposure control unit 116 sets exposure conditions such as a shutter speed and an aperture value based on the amount of motion of a target including a virtual object. As a result, an exposure time according to the set exposure conditions is determined.

[0013] The real image generating unit 107 performs a development process on the group of signals output from the imaging element 106 to generate an image of the real space. The virtual object generating unit 108 generates a virtual object model (a virtual object) to be superimposed on the image of the real space based on a virtual object model previously stored in the memory 118. The mixed reality image generating unit 109 synthesizes the image of the real space with an image related to the virtual object model (an image of the virtual object) to generate a mixed reality image, which is a composite image. In this example, the mixed reality image generating unit 109 renders the image of the real space by superimposing the virtual object model generated by the virtual object generating unit 108 on the image of the real space. The virtual object model is stored in advance in the memory 118. The virtual object model is four-dimensional data having three-dimensional spatial information and one-dimensional time information. In addition, the memory 118 stores in advance a control program executed by the CPU 117.

[0014] The display unit 110 displays a live view of a captured image, displays a shooting setting screen, displays a playback image, etc. The display unit 110 is, for example, a display device having a liquid crystal display, etc. The recording unit 111 is, for example, a storage medium such as an SD card, and records the mixed reality image generated by the mixed reality image generating unit 109.

[0015] FIG. 2 is a flowchart illustrating an example of an image capturing operation of the mixed reality space by the imaging device. Each process according to this flowchart is realized by CPU 117 executing a control program stored in memory 118. S in Fig. 2 indicates a step number corresponding to each process according to this flowchart.

[0016] When shooting starts, the process proceeds to S201. In S201, CPU 117 determines whether the power is OFF. If the power is not OFF, the process proceeds to S202. If the power is OFF, the shooting process ends. In S202, CPU 117 performs display processing of a mixed reality image (live view display). The display processing of a mixed reality image will be described in detail later.

[0017] Next, in S203, the CPU 117 determines whether SW1 is ON. If SW1 is not ON but OFF, the process proceeds to S207. If SW1 is ON, the process proceeds to S204. Next, the CPU 117 controls the focus adjustment unit 112 to perform focus adjustment processing. Based on the output of the focus adjustment unit 112, the CPU 117 drives the focusing lens 102 so as to focus on any subject, including a virtual object, that exists in a preset focus detection area. Next, in S205, the CPU 117 controls the motion amount calculation unit 114 to calculate the motion amount of a target, including a virtual object.

[0018] FIG. 3 is a flowchart illustrating an example of the motion amount calculation process in S205 of FIG. Each process according to this flowchart is realized by CPU 117 executing a control program stored in memory 118. S in Fig. 3 indicates a step number corresponding to each process according to this flowchart.

[0019] In S301, motion amount calculation unit 114 judges whether the main object of shooting in mixed reality space (for convenience, referred to as "main subject") is a virtual object. Specifically, the area focused on in focus adjustment processing is set as the main subject area, and if a virtual object is superimposed on the main subject area, it is judged that the main subject is a virtual object. If no virtual object is superimposed on the main subject area, it is judged that the main subject is not a virtual object but an object in real space.

[0020] If motion amount calculation unit 114 determines that the main subject is not a virtual object, the process proceeds to S303. If motion amount calculation unit 114 determines that the main subject is a virtual object, the process proceeds to S302. In S303, motion amount calculation unit 114 calculates the amount of motion of the subject in real space. Specifically, based on a signal related to the captured image output by image sensor 106, motion amount calculation unit 114 calculates a motion vector as the amount of motion from the displacement of feature points of the subject between two images.

[0021] In addition, in S302, the motion amount calculation unit 114 calculates the motion amount of the virtual object. The virtual object at each position in the time axis direction indicated by the virtual object model stored in the memory 118 can be defined in different spatial coordinates, and the movement amount on the coordinates of the virtual object between each position in the time axis direction can be treated as the motion amount. When the period in which the motion is detected and the superimposition position of the virtual object in the mixed reality space are known, a motion vector can be calculated by projecting the motion amount of the virtual object in a predetermined period onto the two-dimensional coordinates of the mixed reality image. Therefore, the motion amount calculation unit 114 refers to the virtual object model stored in the memory 118, acquires information of the virtual object model superimposed on the main subject region, and calculates the motion amount of the virtual object based on the acquired information of the virtual object model and the period in which the motion is detected. When the motion vector is calculated in S302 and S303, the motion amount calculation process ends.

[0022] Returning to FIG. 2, the description will be continued. In S206, the exposure control unit 116 performs exposure control based on the photometry result by the photometry unit 115 and the amount of movement of the target including the virtual object calculated by the motion amount calculation unit 114. Specifically, the exposure control unit 116 determines an appropriate EV value based on the photometry result, and determines the shutter speed and the aperture value at the appropriate EV value. Based on the amount of movement output by the motion amount calculation unit 114, the exposure control unit 116 sets the exposure conditions so that image blur caused by the movement of the target including the virtual object is minimized. For example, the exposure control unit 116 increases the shutter speed and decreases the aperture value (increases the aperture diameter) as the amount of movement increases. In this way, the exposure time according to the set exposure conditions is determined.

[0023] Next, in S207, CPU 117 determines whether SW2 is ON. If SW2 is not ON but OFF, the process returns to S201. If SW2 is ON, the process proceeds to S208. Next, in S208, CPU 117 performs a recording process of the mixed reality image. The details of the recording process of the mixed reality image will be described later. After the recording process of the mixed reality image is completed, the process returns to S201.

[0024] FIG. 4 is a flowchart illustrating an example of the display process of the mixed reality image in S202 of FIG. Each process according to this flowchart is realized by CPU 117 executing a control program stored in memory 118. S in Fig. 4 indicates a step number corresponding to each process according to this flowchart.

[0025] In S401, the CPU 117 performs exposure processing for display at a period corresponding to a predetermined display rate, and acquires a signal related to a captured image from the image sensor 106. Next, in S402, the position and orientation detection unit 119 detects the position and orientation of the imaging device 100, and outputs position and orientation information. The CPU 117 stores the position and orientation information output by the position and orientation detection unit 119 in a storage unit (for example, memory 118) in association with the signal acquired in S401.

[0026] Next, in S403, CPU 117 detects the depth of the space within the shooting angle of view and acquires depth information. CPU 117 associates the acquired depth information with the signal acquired in S401 and stores it in a storage unit (for example, memory 118). Next, in S404, CPU 117 controls the reality image generation unit 107 to execute a process of generating a mixed reality image. Then, in S405, CPU 117 shapes the mixed reality image generated in S404 to a size suitable for display, and controls the display unit 110 to display the shaped mixed reality image as a live view.

[0027] FIG. 5 is a flowchart illustrating an example of the mixed reality image generation process in S404 of FIG. Each process according to this flowchart is realized by CPU 117 executing a control program stored in memory 118. S in Fig. 5 indicates a step number corresponding to each process according to this flowchart.

[0028] In S501, the real image generating unit 107 performs a development process on the signal related to the captured image obtained by the exposure for display in S401 of FIG. 4 to generate an image of the real space. Subsequently, in S502, the mixed reality image generating unit 109 acquires information (arrangement position information) related to the arrangement position of the virtual object model in the real space. For example, in a case where a method of constructing a mixed reality space by recognizing a marker that is a reference for the arrangement of a virtual object model in the real space is adopted, the mixed reality image generating unit 109 acquires the arrangement position information of the virtual object model in the real space based on the detection information of the marker within the shooting angle of view. The mixed reality image generating unit 109 associates the acquired arrangement position information with the signal related to the captured image and stores it in a storage unit (for example, memory 118). Note that when a virtual object is not arranged in the real space, there is no arrangement position information of the virtual object. Therefore, in this case, the mixed reality image generating unit 109 stores information indicating that a virtual object is not arranged in the real space in the storage unit.

[0029] Next, in S503, the mixed reality image generation unit 109 determines the shape of the virtual object model as seen by the imaging device 100, i.e., the orientation of the virtual object model, using position and orientation information associated with a signal related to the captured image and stored in the memory 118. Then, in S504, the mixed reality image generation unit 109 determines the size of the virtual object model as seen by the imaging device 100, based on depth information corresponding to the arrangement position of the virtual object model. Then, in S505, the mixed reality image generation unit 109 superimposes the virtual object model as seen by the imaging device 100 onto the real image and projects (renders) it. As a result, the image of real space and the image of the virtual object are synthesized to generate a mixed reality image.

[0030] FIG. 6 is a flowchart illustrating an example of the mixed reality image recording process in S208 of FIG. Each process according to this flowchart is realized by CPU 117 executing a control program stored in memory 118. S in Fig. 6 indicates a step number corresponding to each process according to this flowchart.

[0031] In S601, the CPU 117 performs exposure for recording based on the aperture value and the shutter speed set by the exposure control in S206 of FIG. 2. The processes of S602 to S604 are the same as the processes of S402 to S404 of FIG. 4. By the processes of S602 to S604, the image of the real space captured by the exposure for recording and the image of the virtual object are synthesized to generate a mixed reality image. Then, in S605, the mixed reality image generating unit 109 records the mixed reality image generated in S604 in the recording unit 111. The image processing device of the above-mentioned embodiment 1 synthesizes the image of the real space obtained by imaging by exposure control based on the amount of movement of the target including the virtual object and the image of the virtual object. This makes it possible to obtain a mixed reality image that reflects the movement of the virtual object and is less unnatural.

[0032] Example 2 The image processing device of the second embodiment performs exposure multiple times based on the amount of movement of the target including the virtual object described in the first embodiment, and generates a mixed reality image based on image information obtained by imaging with each exposure (split exposure). The number of split exposures is obtained by dividing the exposure time corresponding to a predetermined shutter speed by the exposure time determined by exposure control based on the amount of movement of the target including the virtual object described in the first embodiment. This allows a recorded image corresponding to a desired shutter speed to be obtained. Hereinafter, an image of real space obtained by imaging with split exposure is also referred to as a split exposure image.

[0033] The overall operation of capturing an image of a mixed reality space in the second embodiment is the same as the overall operation of capturing an image of a mixed reality space in the first embodiment described with reference to Fig. 2, except for the processes in S206 and S208. Only the parts different from the first embodiment will be described below. The exposure control in the second embodiment will be described. In S206 in Fig. 2, the exposure control unit 116 first sets exposure conditions based on the amount of movement of the target including the virtual object output by the movement amount calculation unit 114 so as to minimize the blurring of the image caused by the movement of the target, as in the first embodiment. As a result, an exposure time (first exposure time) according to the set exposure conditions is determined.

[0034] In the second embodiment, the exposure control unit 116 further acquires an exposure time (second exposure time) according to a preset shutter speed. Then, the exposure control unit 116 calculates the number of divided exposures by dividing the second exposure time by the first exposure time. In a recording process of a mixed reality image described later, a signal related to a captured image corresponding to the second exposure time can be obtained by performing the calculated number of divided exposures. When the first exposure time is shorter than the second exposure time, the number of divided exposures is multiple. Note that when the first exposure time is longer than the second exposure time, the exposure control unit 116 sets the first exposure time to the second exposure time and sets the number of divided exposures to 1. The calculated number of divided exposures (number of exposure repetitions) is referred to in the recording process of a mixed reality image.

[0035] FIG. 7 is a flowchart illustrating an example of a recording process of a mixed reality image in S208 of FIG. 2 according to the second embodiment. Each process according to this flowchart is realized by CPU 117 executing a control program stored in memory 118. In Fig. 7, S indicates a step number corresponding to each process according to this flowchart. Note that processes similar to those described with reference to Fig. 6 are given the same step numbers as in Fig. 6, and descriptions thereof will be omitted.

[0036] When the recording process of the mixed reality image starts, the process proceeds to S701. In S701, the CPU 117 initializes the number of exposures to 0. Then, in S702, the CPU 117 performs exposure (split exposure) with a first exposure time. After that, the split exposure image and the image of the virtual object are combined by a series of processes from S602 to S604, and a mixed reality image corresponding to the first exposure time is generated. Then, in S703, the CPU 117 increments the number of exposures. Then, in S704, the CPU 117 determines whether the number of exposures has reached the number of exposure repetitions. If the number of exposures has not reached the number of exposure repetitions, the process proceeds to S702. Then, the next split exposure and mixed reality image generation process are executed. If the number of exposures has reached the number of exposure repetitions, mixed reality images for the number of exposure repetitions have already been generated. Therefore, in this case, the process proceeds to S705.

[0037] In S705, the mixed reality image generating unit 109 uses an image processing circuit (not shown) to synthesize the mixed reality images for the number of exposure repetitions to generate one synthetic mixed reality image. The generated synthetic mixed reality image includes an image signal of a second exposure time, that is, an exposure time corresponding to a shutter speed for recording. In addition, since information of a virtual object on a similar time axis is superimposed in each exposure time, image blurring due to the movement of the virtual object is reproduced in the synthesized image. Next, in S706, the mixed reality image generating unit 109 records the synthetic mixed reality image in the recording unit 111. Then, the processing ends. The image processing device of the second embodiment described above performs a divided exposure process based on the first exposure time, which causes less image blurring due to the movement of the target, when shooting with a fixed shutter speed. Therefore, according to the image processing device of the second embodiment, it is possible to suppress deterioration of the positioning accuracy of the marker in the generation process of the mixed reality image, and to reproduce image blurring due to the movement of the virtual object in the recorded image.

[0038] Example 3 The image processing device of the third embodiment synthesizes a plurality of real space images obtained by multiple divided exposures, and renders the synthesized real space image by superimposing a virtual object model corrected according to the amount of movement of the virtual object.

[0039] FIG. 8 is a flowchart illustrating an example of a recording process of a mixed reality image by the image processing device according to the third embodiment. Each process according to this flowchart is realized by the CPU 117 executing a control program stored in the memory 118. In Fig. 8, S indicates a step number corresponding to each process according to this flowchart. Note that the overall operation related to shooting the mixed reality space in the third embodiment is similar to the operation described with reference to Fig. 2.

[0040] In S801, the CPU 117 acquires an image (divided exposure image) in real space by divided exposure, as in the second embodiment. Next, the CPU 117 determines whether a virtual object model is arranged for the divided exposure image, that is, whether a virtual object is included in the subject corresponding to the divided exposure. If a virtual object model is not arranged for the divided exposure image, the process returns to S801. If a virtual object model is arranged for the divided exposure image, the process proceeds to S803. The processes of S801 and S802 are performed the same number of times as the number of exposure repetitions.

[0041] Next, in S803, CPU 117 acquires the amount of movement of a virtual object included in a target corresponding to each divided exposure. CPU 117 then integrates the acquired amount of movement of the virtual object to acquire the amount of movement of the virtual object corresponding to multiple exposures (exposure repetition count), that is, the amount of movement of the virtual object corresponding to the second exposure time.

[0042] Next, in S804, the CPU 117 synthesizes the divided exposure images obtained by the exposure repetition count. As a result, an image in real space corresponding to the second exposure time is obtained. Then, in S805, the CPU 117 determines the position, orientation, and depth of the virtual object model to be placed in the image in real space.

[0043] Next, in S806, the CPU 117 executes a correction process on the image of the virtual object to be placed according to the amount of movement of the virtual object corresponding to the exposure for the number of exposure repetitions acquired in S803. Specifically, the CPU 117 applies motion blur to the image of the virtual object.

[0044] Next, in S807, the CPU 117 synthesizes the image of the real space obtained by the synthesis process in S804 and the image of the virtual object corrected in S806. As a result, a mixed reality image is generated. Then, in S808, the CPU 117 records the generated mixed reality image in the recording unit 111. According to the image processing device of the third embodiment, it is possible to generate a mixed reality image by synthesizing the image of the real space and the image of the virtual object once while suppressing image blur by divided exposure.

[0045] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0046] 100 Imaging device 109 Mixed reality image generation unit 117 CPU

Claims

1. An acquisition means for acquiring an amount of movement of a target including a virtual object; a control means for performing exposure control based on a photometry result and the acquired amount of motion; a generating means for generating a composite image by synthesizing an image of a real space captured by the exposure control and an image of the virtual object.

13. An image processing device comprising:

2. the control means executes a plurality of exposures based on an exposure time determined by the exposure control; The generating means generates a plurality of composite images by combining images of the real space obtained by imaging with each exposure and an image of the virtual object, and combines the generated plurality of composite images.

2. The image processing device according to claim 1,

3. the control means performs a plurality of exposures based on an exposure time determined by exposure control performed based on the amount of motion; the acquiring means acquires an amount of movement of the virtual object corresponding to the plurality of exposures based on an amount of movement of the target corresponding to each exposure; The generating means synthesizes images of real space obtained by imaging with each exposure, and synthesizes the image of real space obtained by the synthesis with an image of the virtual object that has been corrected according to the amount of movement of the acquired virtual object.

2. The image processing device according to claim 1,

4. The acquisition means acquires the amount of movement of the virtual object corresponding to the multiple exposures by integrating the amount of movement of the virtual object included in the target for each of the exposures.

4. The image processing device according to claim 3.

5. The control means executes exposure a number of times obtained by dividing an exposure time corresponding to a preset shutter speed by an exposure time determined by exposure control performed based on the amount of motion.

5. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.

6. the acquiring means acquires an amount of movement of the virtual object included in the target based on a virtual object model including position information of the virtual object in a time axis direction stored in a storage means; The control means determines the exposure time based on the acquired amount of movement of the virtual object.

6. The image processing device according to claim 2, 3 or 5.

7. The generating means synthesizes an image of the real space and an image of the virtual object based on information on a placement position of a virtual object model in a real space, the information being acquired based on detection information of a marker within a shooting angle of view.

7. The image processing device according to claim 1,

8. 8. An image processing device according to claim 1, further comprising: an image pickup means for photoelectrically converting subject light and outputting a signal related to a captured image.

1. An imaging device comprising:

9. An acquisition step of acquiring an amount of movement of a target including a virtual object; a control step of performing exposure control based on a photometry result and the acquired amount of motion; a generating step of generating a composite image by combining an image of the real space captured by the exposure control with an image of the virtual object.

23. A method for controlling an image processing apparatus comprising the steps of:

10. A program for causing a computer to function as each of the means included in the image processing apparatus according to any one of claims 1 to 7.

Images