Image processing apparatus, imaging apparatus, image processing method, program, and storage medium

Abstract

To provide an image processing apparatus that can correct out-of-focus between two images having parallax after imaging.SOLUTION: An image processing apparatus (900) has: image acquisition means (1000) that acquires first image data obtained by performing imaging through a first optical system (301R), and second image data obtained by performing imaging through a second optical system (301L); refocus means (1004) that can perform refocus processing independently on the first image data and the second image data; and out-of-focus correcting means (1003) that, in accordance with the refocus means performing the refocus processing on at least either one of the first image data or the second image data, corrects the difference in the degree of focus between the first image data and the second image data.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, a program, and a storage medium. [Background technology]

[0002] There is a known technology that captures two images with a parallax between them through two optical systems, and displays the two images so that they can be viewed stereoscopically. If there is a focus shift between the two images used for stereoscopic viewing, the user will feel uncomfortable viewing the stereoscopic view.

[0003] Patent Documents 1 and 2 disclose a method for adjusting the focus of two images (parallax images) having parallax with each other when performing automatic focus adjustment (autofocus: AF) during imaging. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 08-194274 [Patent Document 2] JP 2012-133232 A Summary of the Invention [Problem to be solved by the invention]

[0005] The methods disclosed in Patent Documents 1 and 2 cannot adjust the focus of two captured images.

[0006] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an image processing device capable of adjusting the focus shift between two images having parallax after capture. [Means for solving the problem]

[0007] An image processing device as one aspect of the present invention has an image acquisition means for acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system, a refocusing means capable of performing refocusing processing independently on each of the first image data and the second image data, and a focus adjustment means for adjusting the difference in focus degree between the first image data and the second image data by performing the refocusing processing by the refocusing means on at least one of the first image data or the second image data.

[0008] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image processing device capable of adjusting the focus deviation between two images having parallax after capturing the images. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an overall configuration of a system according to each embodiment. [Diagram 2] FIG. 1 is an external view of a camera according to each embodiment. [Diagram 3] FIG. 2 is a block diagram of a camera according to each embodiment. [Figure 4] FIG. 2 is a configuration diagram of a lens unit according to each embodiment. [Diagram 5] 5A to 5C are schematic diagrams showing features of captured images according to each embodiment. [Figure 6] FIG. 2 is a pixel array diagram according to each embodiment. [Figure 7] 1A to 1C are schematic diagrams showing images included in a captured image in a multi-viewpoint shooting mode according to each embodiment. [Figure 8] FIG. 2 is a block diagram of a PC according to each embodiment. [Figure 9] FIG. 2 is a block diagram of a processing unit of a PC according to the first embodiment. [Figure 10]4 is a flowchart showing an operation procedure according to the first embodiment. [Figure 11] FIG. 11 is a block diagram of a processing unit of a PC according to a second embodiment. [Figure 12] 13 is an example of a UI screen according to the second embodiment. [Figure 13] 10 is a flowchart showing an operation procedure according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] <System configuration> First, a system 10 according to this embodiment will be described with reference to Figs. 1(a) and (b). Figs. 1(a) and (b) are schematic diagrams showing an example of the overall configuration of the system 10. The system 10 includes a camera (imaging device) 100 and a personal computer (PC) 600. A lens unit (lens device) 300 is attached (connected) to the camera 100. The lens unit 300 will be described in detail later, but by attaching the lens unit 300, the camera 100 can capture two images (still images or video) having a predetermined parallax at once. The PC 600 is an information processing device that processes images captured by an imaging device such as the camera 100.

[0013] FIG. 1(a) shows a configuration in which a camera 100 and a PC 600 are connected to be able to communicate with each other wirelessly or by wire. FIG. 1(b) shows a configuration in which images captured by the camera 100 and the like are input to the PC 600 via an external storage device 700 on a file basis. The external storage device 700 is connected to one or both of the camera 100 and the PC 600. For example, the external storage device 700 may be connected to the camera 100, and files of images captured by the camera 100 may be stored in the external storage device 700. Thereafter, the external storage device 700 may be removed from the camera 100 and connected to the PC 600, and the files stored in the external storage device 700 may be imported by the PC 600.

[0014] Next, the external configuration of the camera 100 will be described with reference to Figures 2(a) and (b). Figures 2(a) and (b) are external views of the camera 100. Figure 2(a) shows a perspective view of the camera 100 seen from the front side, and Figure 2(b) shows a perspective view of the camera 100 seen from the rear side.

[0015] The camera 100 has on its top surface a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an outside viewfinder display unit 107. The shutter button 101 is an operating member for issuing a shooting preparation instruction or a shooting instruction. The power switch 102 is an operating member for switching the power of the camera 100 on and off.

[0016] The mode change switch 103 is an operating member for switching between various modes. The main electronic dial 104 is a rotary operating member for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operating member for moving a selection frame (cursor) and forwarding images. The video button 106 is an operating member for issuing instructions to start and stop video shooting (recording). The outside viewfinder display unit 107 displays various settings such as shutter speed and aperture.

[0017] Camera 100 has, on the rear surface, a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119. Display unit 108 displays images and various information. Touch panel 109 is an operation member that detects touch operations on the display surface (touch operation surface) of display unit 108.

[0018] The directional key 110 is an operation section made up of keys that can be pressed up, down, left and right (four-way key). Processing can be performed according to the position of the directional key 110 that is pressed. The SET button 111 is an operation member that is pressed mainly to confirm a selected item. The AE lock button 112 is an operation member that is pressed to fix the exposure state in a shooting standby state.

[0019] The enlargement button 113 is an operating member for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge the playback image or increase the magnification ratio in the playback mode.

[0020] The playback button 114 is an operating member for switching between a shooting mode and a playback mode. When the playback button 114 is pressed in the shooting mode, the mode changes to the playback mode, and the latest image among the images recorded in the recording medium 227 described later can be displayed on the display unit 108.

[0021] Menu button 115 is an operating member that is pressed to display a menu screen on which various settings can be made on display unit 108. A user can intuitively make various settings using the menu screen displayed on display unit 108, direction key 110, and SET button 111.

[0022] Eyepiece unit 116 is a part where the user places his / her eye close to eyepiece finder (eye-peek type finder) 117 and looks into it. Through eyepiece unit 116, the user can view an image displayed on an EVF (Electronic View Finder) 217 ​​(described below) inside camera 100. Eyepiece detection unit 118 is a sensor that detects whether or not the user places his / her eye close to eyepiece unit 116 (eyepiece finder 117).

[0023] Touch bar 119 is a line-shaped touch operation member (line touch sensor) capable of receiving a touch operation. Touch bar 119 is disposed at a position where it can be touched by the thumb of the right hand when grip portion 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that shutter button 101 can be pressed with the index finger of the right hand. In other words, touch bar 119 can be operated in a state (shooting posture) where the eye is placed close to eyepiece viewfinder 117, the eye is looked into eyepiece portion 116, and the camera is held so that shutter button 101 can be pressed at any time.

[0024] Touch bar 119 can accept tap operations (operation of touching and then releasing the touched position within a predetermined period of time without moving it) and left and right slide operations (operation of touching and then moving the touched position while still touching) on ​​touch bar 119. Touch bar 119 is an operating member different from touch panel 109 and does not have a display function. Touch bar 119 functions as, for example, a multi-function bar (M-Fn bar) to which various functions can be assigned.

[0025] The camera 100 also has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section formed in a shape that is easy to hold with the right hand when the user holds the camera 100. In a state in which the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are disposed in positions that can be operated with the index finger of the right hand. In a similar state, the sub electronic dial 105 and the touch bar 119 are disposed in positions that can be operated with the thumb of the right hand.

[0026] Thumb rest section 121 (thumb standby position) is a grip section provided at a position on the rear side of camera 100 where the thumb of the right hand gripping grip section 120 can be easily placed when none of the operation members are being operated. Thumb rest section 121 is made of a rubber member or the like to enhance holding power (grip feeling).

[0027] Terminal cover 122 protects connectors such as a connection cable that connects camera 100 to an external device (external apparatus). Lid 123 protects recording medium 227 and the slot by closing the slot for storing recording medium 227 (described later). Communication terminal 124 is a terminal for communicating with a lens unit (such as lens unit 200 (described later) or lens unit 300 shown in FIGS. 1(a) and (b)) that is detachable from camera 100.

[0028] Next, the internal configuration of the camera 100 will be described with reference to Fig. 3. Fig. 3 is a block diagram of the camera 100. In Fig. 3, the same components as those in Figs. 2(a) and (b) are given the same reference numerals as those in Figs. 2(a) and (b), and descriptions of those components will be omitted as appropriate. In Fig. 3, a lens unit (lens device) 200 is attached to the camera 100.

[0029] First, the lens unit 200 will be described. The lens unit 200 is an interchangeable lens that is detachable from the camera 100. The lens unit 200 is a single lens, and is an example of a normal lens. The lens unit 200 has an aperture (aperture diaphragm) 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0030] The aperture 201 is configured so that the aperture diameter can be adjusted. The lens 202 is made up of a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus.

[0031] The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 50, which will be described later. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203, and adjusts the focus by changing the position of the lens 202 via the AF drive circuit 204.

[0032] The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, communication is performed via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100.

[0033] Next, a description will be given of the camera 100. The camera 100 has a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0034] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on an instruction from the system control unit 50. The imaging unit 211 is an imaging element (image sensor) composed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 211 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts an analog signal output from the imaging unit 211 into a digital signal. The A / D converter 212 may be built into the imaging unit 211.

[0035] The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic results. This processing allows TTL (through-the-lens) AF processing, AE (auto exposure) processing, EF (flash pre-flash) processing, etc. to be performed. Furthermore, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs TTL AWB (auto white balance) processing based on the obtained arithmetic results.

[0036] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, and image data to be displayed on the display unit 108 and the EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0037] The D / A converter 216 converts the display image data stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216.

[0038] The display unit 108 and the EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and the EVF 217 are, for example, displays such as an LCD or an organic EL. A digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216, and the analog signal is sequentially transferred to and displayed on the display unit 108 or the EVF 217, thereby performing a live view display.

[0039] The system control unit 50 is a control unit that includes at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100.

[0040] The system control unit 50 realizes each process of the flowchart described below by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, and the like.

[0041] The camera 100 also includes a system memory 218 , a non-volatile memory 219 , a system timer 220 , a communication unit 221 , an attitude detection unit 222 , and an eye proximity detection unit 118 .

[0042] For example, a RAM is used as the system memory 218. Constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are expanded in the system memory 218. The nonvolatile memory 219 is an electrically erasable and recordable memory, and for example, an EEPROM is used as the nonvolatile memory 219.

[0043] The non-volatile memory 219 stores constants, programs, etc. for the operation of the system control unit 50. The programs here are programs for executing the flowcharts described below. The system timer 220 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0044] The communication unit 221 transmits and receives video signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 221 can also connect to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth (registered trademark) Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded in a recording medium 227, and can receive images and various other information from an external device.

[0045] The attitude detection unit 222 detects the attitude of the camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 is an image captured by holding the camera 100 horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image according to the detected attitude. For example, an acceleration sensor or a gyro sensor can be used for the attitude detection unit 222. It is also possible to detect the movement of the camera 100 (panning, tilting, lifting, whether it is stationary, etc.) using the attitude detection unit 222.

[0046] Eyepiece detection unit 118 can detect the approach of some object to eyepiece unit 116 (eyepiece finder 117). Eyepiece detection unit 118 can be, for example, an infrared proximity sensor. When an object approaches, infrared light projected from a light-projecting unit of eyepiece detection unit 118 is reflected by the object and received by a light-receiving unit of the infrared proximity sensor. The distance from eyepiece unit 116 to the object can be determined based on the amount of infrared light received.

[0047] In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity of an object to the eyepiece unit 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eyepiece) and separation (eye separation) of an eye (object) to the eyepiece unit 116. When an object is detected approaching within a predetermined distance from the non-eyepiece state (non-approaching state) to the eyepiece unit 116, the eyepiece detection unit detects that the object has been placed near the eyepiece unit 116. On the other hand, when an object that was detected to be approaching from the eyepiece state (approaching state) moves away from the eyepiece unit 116 by a predetermined distance or more, the eyepiece detection unit detects that the object has been removed from the eyepiece unit 116. The threshold value for detecting the eyepiece and the threshold value for detecting the removal of the eye may be different, for example, by providing a hysteresis. In addition, after the eyepiece is detected, the eyepiece state is assumed to remain until the removal of the eyepiece is detected. After the removal of the eyepiece is detected, the non-eyepiece state is assumed to remain until the eyepiece is detected.

[0048] The system control unit 50 switches between display (display state) / non-display (non-display state) of the display unit 108 and the EVF 217 depending on the state detected by the eyepiece detection unit 118. Specifically, when at least the shooting standby state is present and the display destination switching setting is automatic switching, the display destination is set to the display unit 108 when the eye is not placed in contact with the subject, the display is turned on, and the EVF 217 is hidden. Also, when the eye is placed in contact with the subject, the display destination is set to the EVF 217, the display is turned on, and the display unit 108 is hidden. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor, and other sensors may be used for the eyepiece detection unit 118 as long as they can detect a state that can be regarded as being in contact with the eyepiece.

[0049] The camera 100 also has an outside-finder display unit 107, an outside-finder display drive circuit 223, a power supply control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and the like.

[0050] The outside viewfinder display unit 107 is driven by an outside viewfinder display drive circuit 223, and displays various settings of the camera 100 such as the shutter speed and aperture. The power supply control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery level, etc. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage to each unit including the recording medium 227 for the required period of time.

[0051] Power supply unit 225 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery or a Li battery, an AC adapter, etc. Recording medium I / F 226 is an interface with recording medium 227 such as a memory card or a hard disk. Recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, etc. Recording medium 227 may be detachable from camera 100, or may be built into camera 100.

[0052] The operation unit 228 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode changeover switch 103, the touch panel 109, and other operation units 229. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, and the like.

[0053] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231 .

[0054] The first shutter switch 230 is turned on by half-pressing (shooting preparation instruction) during operation of the shutter button 101, and outputs a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing.

[0055] The second shutter switch 231 is turned on when the operation of the shutter button 101 is completed, that is, when the shutter button 101 is fully pressed (shooting instruction), and outputs a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processes, from reading out a signal from the imaging unit 211 to generating an image file including a shot image and writing it to the recording medium 227.

[0056] The mode changeover switch 103 switches the operation mode of the system control unit 50 to one of a still image shooting mode, a video shooting mode, a playback mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode).

[0057] There are also various scene modes, custom modes, etc., which are shooting settings for different shooting scenes. The user can directly switch to any of the above-mentioned shooting modes using the mode change switch 103. Alternatively, the user can once switch to a list screen of shooting modes using the mode change switch 103, and then selectively switch to any of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0058] Touch panel 109 is a touch sensor that detects various touch operations on the display surface of display unit 108 (operation surface of touch panel 109). Touch panel 109 and display unit 108 can be configured as one unit. For example, touch panel 109 is attached to the upper layer of the display surface of display unit 108 so that the light transmittance does not interfere with the display of display unit 108. Then, by associating input coordinates on touch panel 109 with display coordinates on the display surface of display unit 108, a GUI (Graphical User Interface) can be configured as if the user could directly operate the screen displayed on display unit 108.

[0059] The touch panel 109 may be of any of various types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, an optical sensor type, etc. Depending on the type, there are types that detect a touch by contact with the touch panel 109, and types that detect a touch by the approach of a finger or a pen to the touch panel 109, but any type may be used.

[0060] The system control unit 50 can detect the following operations or states on the touch panel 109.

[0061] A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down).

[0062] A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).

[0063] The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move).

[0064] The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the touch ends (hereinafter, referred to as "touch-up").

[0065] A state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0066] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0067] These operations and states, as well as the position coordinates of the touch panel 109 touched by a finger or pen, are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch-move, the moving direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical component and horizontal component on the touch panel 109 based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed.

[0068] An operation of touching the touch panel 109 with a finger, quickly moving the finger for a certain distance, and then releasing the finger is called a flick. In other words, a flick is an operation of quickly tracing the touch panel 109 with a finger as if flicking it. When a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is detected after that, it is determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation).

[0069] Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved farther apart is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply pinch).

[0070] Next, lens unit 300 will be described with reference to Fig. 4. Fig. 4 is a configuration diagram of lens unit 300. Fig. 4 shows a state in which lens unit 300 is attached to camera 100. Note that, among the camera 100 shown in Fig. 4, the same components as those described with reference to Fig. 3 are denoted by the same reference numerals as in Fig. 3, and descriptions of those components will be omitted as appropriate.

[0071] The lens unit 300 is an interchangeable lens that can be attached to and detached from the camera 100. The lens unit 300 is a twin lens that can capture right and left images with parallax. In this embodiment, the lens unit 300 has two optical systems, and each of the two optical systems can capture a wide viewing angle range of approximately 180 degrees. Specifically, each of the two optical systems of the lens unit 300 can capture an object in a viewing field (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle). In other words, each of the two optical systems can capture an image of the range of the front hemisphere.

[0072] The lens unit 300 includes a right-eye optical system 301R having a plurality of lenses and a reflecting mirror, a left-eye optical system 301L having a plurality of lenses and a reflecting mirror, a lens system control circuit 303, and an AF drive circuit 304. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. The right-eye optical system 301R includes a lens 302R arranged on the subject side, and the left-eye optical system 301L includes a lens 302L arranged on the subject side. The lenses 302R and 302L face in the same direction, and their optical axes are approximately parallel.

[0073] The lens system control circuit 303 controls the AF drive circuit 304 and the like. The lens system control circuit 303 focuses on the subject by changing the positions of the right eye optical system 301R and the left eye optical system 301L via the AF drive circuit 304. In each embodiment, the AF drive circuit 304 controls the right eye optical system 301R and the left eye optical system 301L so that they operate in conjunction with each other. That is, the focus adjustment is adjusted for the entire lens unit 300, and the focus adjustment does not cause a defocus between the right image and the left image. Note that the defocus between the right image and the left image can be finely adjusted by an adjustment unit not shown in FIG. 4, and can be adjusted in advance before shooting begins.

[0074] The lens unit 300 is a twin lens (VR180 lens) for obtaining a VR180 image, which is one of the formats of VR (Virtual Reality) images that enable twin-eye stereoscopic vision. In this embodiment, the lens unit 300 has fisheye lenses capable of capturing a range of approximately 180 degrees in each of the right-eye optical system 301R and the left-eye optical system 301L. Note that the range that can be captured by the lenses of each of the right-eye optical system 301R and the left-eye optical system 301L may be approximately 160 degrees, which is narrower than the range of 180 degrees.

[0075] The lens unit 300 can form a right image (first image) formed via the right eye optical system 301R and a left image (second image) formed via the left eye optical system 301L onto one or two imaging elements of a camera to which the lens unit 300 is attached.

[0076] The lens unit 300 is attached to the camera 100 via the lens mount portion 305 and the camera mount portion 306 of the camera 100. In this manner, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 307 of the lens unit 300.

[0077] Next, the characteristics of an image (captured image) 500 captured by the camera 100 of each embodiment will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing the characteristics of the image 500. In this embodiment, a right image formed via the right eye optical system 301R and a left image formed via the left eye optical system 301L are simultaneously (as a set) formed on the imaging section 211 of the camera 100. That is, two optical images (first image) 501R and (second image) 501L formed by the right eye optical system 301R and the left eye optical system 301L are formed on one imaging element 502.

[0078] The imaging unit 211 converts the formed subject image (optical signal) into an analog electric signal. By using the lens unit 300 in this way, two images (first image and second image) with parallax can be simultaneously (as a set) acquired from two locations (optical systems) of the right eye optical system 301R and the left eye optical system 301L. By dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image in a range of approximately 180 degrees.

[0079] Next, an example of pixel arrangement of the image sensor 502 will be described with reference to Fig. 6. Fig. 6 is a pixel arrangement diagram of the image sensor 502, representatively showing an area in which pixels are arranged in a 4×4 pixel array. The image sensor 502 is configured to divide the pupil area of ​​the imaging optical system in the pupil division direction, and to be able to generate a plurality of image signals (parallax images) from signals based on light beams that have passed through different pupil areas (pupil partial areas). Specifically, the photoelectric conversion area of ​​each pixel is divided into two in the horizontal direction (pupil division direction), and each photoelectric conversion area functions as a sub-pixel. Therefore, Fig. 6 can also be said to be an area in which sub-pixels are arranged in a 8×4 pixel array.

[0080] 6 corresponds to a repeating unit of color filters in a primary color Bayer array provided on the image sensor 502. Therefore, a pixel 510R having a spectral sensitivity of R (red) is arranged at the upper left, pixels 510G having a spectral sensitivity of G (green) are arranged at the upper right and lower left, and a pixel 510B having a spectral sensitivity of B (blue) is arranged at the lower right.

[0081] 6, each pixel has a photoelectric conversion unit equally divided into 2 horizontally and 1 vertically, with the left half of the photoelectric conversion unit being usable as a first subpixel 511 and the right half being usable as a second subpixel 512. One image obtained by acquiring the output of subpixel 511 and one image obtained by acquiring the output of subpixel 512 constitute one pair of viewpoint images. Thus, camera 100 can generate two viewpoint images by capturing an image once.

[0082] Furthermore, by adding (combining) the signals obtained from the first subpixel 511 and the second subpixel 512 of each pixel, the signals can be used as a pixel signal of one normal pixel without pupil division. That is, a first image described below is an image obtained by combining the first parallax image and the second parallax image, and a second image described below is an image obtained by combining the third parallax image and the fourth parallax image.

[0083] In this embodiment, the circuit of each pixel corresponding to each microlens is configured to have a charge storage section (floating diffusion section: FD section) common to multiple pupil-divided photoelectric conversion sections. Therefore, by controlling the transfer of charge to the FD section and the resetting of the charge in the FD section, it is possible to output a pixel signal based on the charge from each sub-pixel and a pixel signal that is a mixture of the charges from each sub-pixel and is output.

[0084] Here, for convenience, the optical images based on the light beams that pass through different pupil partial regions and are incident on each sub-pixel are called the A image and the B image, the signals (charges) based on each optical image are called the A image signal and the B image signal, and the signal obtained by mixing the signals from each sub-pixel is called the (A+B) image signal.

[0085] In this embodiment, in normal shooting mode, only the (A+B) image signal is read out when capturing an image, and in multi-view shooting mode, the A image signal is read out in addition to the (A+B) image signal, and when recording in each shooting mode, image data corresponding to each of the read-out image signals is recorded.

[0086] Next, an image that is additionally included in image 500 when photographed in multi-viewpoint shooting mode will be described with reference to Fig. 7. Fig. 7 is a schematic diagram of an image that is additionally included in image 500 when photographed in multi-viewpoint shooting mode. Image 500 has image A 500A and image B 500B in addition to the image A+B (image 500) shown in Fig. 5.

[0087] The A image 500A includes an image (first parallax image) 500AR by the right-eye optical system and an image (third parallax image) 500AL by the left-eye optical system. Similarly, the B image 500B includes an image (second parallax image) 500BR by the right-eye optical system and an image (fourth parallax image) 500BL by the left-eye optical system. The images 500AR and 500BR have a mutual parallax, and the images 500AL and 500BL have a mutual parallax. In the first and second embodiments described below, the description will be made on the premise that the images are captured in a multi-viewpoint capturing mode.

[0088] Next, the internal configuration of the PC 600 will be described with reference to Fig. 8. Fig. 8 is a block diagram of the PC 600. The control unit 601 is, for example, a central processing unit (CPU), and controls the entire PC 600. The read only memory (ROM) 602 non-temporarily stores programs and parameters. The random access memory (RAM) 603 temporarily stores programs and data supplied from an external device or the like. The recording medium 604 is a hard disk or flash memory fixedly installed in the PC 600, or an optical disk, magnetic card, optical card, IC card, memory card, or the like that is detachable from the PC 600. The file of the image captured by the camera 100 is read from the recording medium 604.

[0089] The operation unit 605 accepts user operations on the PC 600. The operation members used by the user to perform operations may be buttons or a touch panel provided on the PC 600, or may be a keyboard or a mouse detachable from the PC 600. The display unit 606 displays data held by the PC 600 or data supplied from an external source. The display unit 606 may be a part of the PC 600, or may be a display device separate from the PC 600. The communication unit 607 communicates with external devices such as the camera 100. The system bus 608 connects the components of the PC 600 to enable communication between them.

[0090] The processing means (image processing device) of the PC 600 in each embodiment described below is realized by the control unit 601 expanding a predetermined program stored in the ROM 602 into the work RAM 603, executing it, and each unit performing a predetermined operation based on an instruction from the control unit 601. Note that each embodiment will be described assuming that the processing is performed in the PC 600, but it is also possible that the camera 100 has the same components as the PC 600 and performs the same processing in the camera 100. Each embodiment will be described below.

[0091] First Embodiment First, the processing means (image processing device) 900 included in the PC 600 according to the first embodiment of the present invention will be described with reference to Fig. 9. Fig. 9 is a block diagram of the processing means (image processing device) 900.

[0092] The image acquisition means 1000 acquires an image file stored in the recording medium 604. That is, the image acquisition means 1000 acquires first image data obtained by imaging via a first optical system (e.g., a right-eye optical system) and second image data obtained by imaging via a second optical system (e.g., a left-eye optical system). In this embodiment, the image acquisition means 1000 can acquire the following six images from the image files (first image data and second image data) captured in the multi-viewpoint shooting mode.

[0093] - Three images captured by the right eye optical system: (A+B) image (first image), image A (first parallax image), and image B (second parallax image) - Three images captured by the left eye optical system: (A+B) image (second image), image A (third parallax image), and image B (fourth parallax image) The incidental information acquisition means 1001 acquires incidental information recorded in association with an image file. The incidental information includes shooting information such as the shutter speed and aperture at the time of shooting, camera attitude information at the time of shooting, and autofocus information (AF information). In this embodiment, the AF information includes AF optical system information indicating whether AF was performed by the right-eye optical system or the left-eye optical system, and coordinate position information where AF was performed. Note that, although an example in which incidental information is recorded in the same image file has been described, the incidental information may be recorded in a different file associated with the image file.

[0094] The reference image determination means 1002 determines an image (reference image, reference image data) that is to be used as a reference in the focus adjustment means 1003 described below. In this embodiment, the reference image determination unit 1002 determines an image of an optical system that has undergone AF as the reference image based on the AF optical system information included in the supplementary information. Since an image of an optical system that has undergone AF is more likely to be in focus than an optical system that has not undergone AF, it is preferable to select an image of an optical system that has undergone AF as the reference image.

[0095] The method of determining the reference image is not limited to the above-mentioned method. For example, the degree of focus (degree of focus) of the (A+B) image of the right eye optical system and the (A+B) image of the left eye optical system at the AF coordinate (predetermined position) may be calculated by a known method, and the image of the optical system that is more in focus may be determined as the reference image. Hereinafter, the image of the optical system (the other of the first image data or the second image data) that is not selected as the reference image (one of the first image data or the second image data) is called an adjustment image.

[0096] The focus adjustment means 1003 performs a refocus process on the adjusted image using the refocus means 1004 described later. That is, the focus adjustment means 1003 adjusts the difference in the degree of focus between the first image data and the second image data by performing a refocus process on at least one of the first image data and the second image data.

[0097] More specifically, the focus adjustment means 1003 calculates the degree of focus of each of the reference image and the adjusted image at the AF coordinates. Then, the focus adjustment means 1003 performs a refocus process so that the degree of focus of the adjusted image falls within a predetermined range with respect to the reference image (so as to adjust the difference in the degree of focus between the reference image and the adjusted image). The refocus means 1004 performs a refocus process using an image A (first or third parallax image) and an image B (second or fourth parallax image), which are parallax images in the optical system (first or second optical system) of the adjusted image.

[0098] The refocusing means 1004 can perform a refocusing process on the first image using the first parallax image and the second parallax image, and can perform a refocusing process on the second image using the third parallax image and the fourth parallax image. Note that the first parallax image and the second parallax image are images corresponding to light beams passing through different pupil regions of the first optical system, and the third parallax image and the fourth parallax image are images corresponding to light beams passing through different pupil regions of the second optical system.

[0099] In this way, the refocusing means 1004 can perform refocusing processing independently for each of the first image data and the second image data. The method of the refocusing processing is not particularly limited, and any known method can be used. The warning means 1005 displays a warning on the display unit 606.

[0100] Next, the operation (image processing method) of the processing means 900 according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the operation procedure of the processing means 900.

[0101] In step S1001, the image acquisition means 1000 acquires an image from, for example, the recording medium 604 (reads the image).

[0102] In step S1002, the additional information acquisition unit 1001 acquires the additional information.

[0103] In step S1003, the reference image determining means 1002 determines a reference image and an adjustment image.

[0104] In step S1004, the focus adjustment means 1003 adjusts the focus of the adjusted image so that the focus (degree of focus) of the reference image and the adjusted image match.

[0105] In step S1005, the focus adjustment unit 1003 determines whether the focus of the reference image and the adjusted image falls within a predetermined range and the adjustment is successful.

[0106] If it is determined that the adjustment has failed (if the focus adjustment unit 1003 has determined that the difference in the degree of focus between the first image data and the second image data cannot be adjusted), the process proceeds to step S1006. On the other hand, if it is determined that the adjustment has been successful, the process of this flowchart ends.

[0107] In step S1006, the warning means 1005 displays (displays a warning) on ​​the display unit 600 to the effect that the adjustment has failed, and then ends the processing of this flowchart.

[0108] In this embodiment, a warning is displayed in step S1006, but the present invention is not limited to this, and for example, the process may be switched to the process in the second embodiment described below. In addition, in this embodiment, the refocus process is performed using one of the first image data or the second image data as the adjusted image data, but the present invention is not limited to this, and both the first image data and the second image data may be refocused.

[0109] According to this embodiment, it is possible to reduce the defocus between the image of the right eye optical system and the image of the left eye optical system in an image captured in the multi-viewpoint shooting mode.

[0110] <Second embodiment> Next, a processing unit (image processing device) 1100 included in a PC 600 according to a second embodiment of the present invention will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a block diagram of the processing unit (image processing device) 1100. Fig. 12 is an example of a UI screen (screen 1200) displayed to a user. Note that in this embodiment, explanations common to the first embodiment will be omitted.

[0111] The image acquiring means 2000 acquires an image file stored in the recording medium 604. That is, the image acquiring means 2000 acquires first image data obtained by imaging via a first optical system (e.g., a right-eye optical system) and second image data obtained by imaging via a second optical system (e.g., a left-eye optical system).

[0112] In this embodiment, the image acquisition means 2000 can acquire the following six images from the image file (first image data and second image data) captured in the multi-viewpoint shooting mode.

[0113] - Three images captured by the right eye optical system: (A+B) image (first image), image A (first parallax image), and image B (second parallax image) - Three images captured by the left eye optical system: (A+B) image (second image), image A (third parallax image), and image B (fourth parallax image) The additional information acquisition unit 2001 acquires additional information recorded in association with an image file. The additional information includes shooting information such as the shutter speed and aperture at the time of shooting, information about the camera's attitude at the time of shooting, and autofocus information (AF information).

[0114] In this embodiment, the AF information includes AF optical system information indicating whether AF was performed in the right eye optical system or the left eye optical system, and coordinate position information where AF was performed. Note that, in this embodiment, an example in which the incidental information is recorded in the same image file has been described, but the incidental information may be recorded in a separate file associated with the image file.

[0115] The reference image determination means 2002 determines an image to be used as a reference in the focus adjustment means 2003 described below. In this embodiment, the reference image determination means 2002 determines an image of an optical system that has undergone AF as the reference image based on the AF optical system information included in the supplementary information. Since an image of an optical system that has undergone AF is more likely to be in focus than an optical system that has not undergone AF, it is preferable to select an image of an optical system that has undergone AF.

[0116] The method of determining the reference image is not limited to the above-mentioned method. For example, the degree of focus (degree of focusing) of the (A+B) image of the right eye optical system and the (A+B) image of the left eye optical system at the AF coordinate (AF frame 1204 in FIG. 12) may be calculated by a known method, and the image of the optical system that is more in focus may be determined as the reference image. Hereinafter, the image of the optical system that is not selected as the reference image is called the adjustment image.

[0117] The focus adjustment means 2003 performs a refocusing process using the refocusing means 2004 according to the adjustment value of the adjustment slider 1201 in Fig. 12. The refocusing means 2004 performs a refocusing process using an image A (first parallax image or third parallax image) and an image B (second parallax image or fourth parallax image), which are parallax images of each optical system. The method of the refocusing process is not particularly limited, and a known method can be used.

[0118] The focus index display means 2005 calculates the degree of focus (degree of focus) in the image adjusted by the focus adjustment means 2003 using a known method, and displays on the display unit 606 according to the degree of focus. That is, the focus index display means 2005 displays an index indicating the degree of focus between the first image data and the second image data. In this embodiment, for example, the focus index display 1202 in Fig. 12 is displayed. In the focus index display 1202, the closer the two left and right bars are to the center bar, the better the focus is.

[0119] A linked adjustment switching means 2006 switches whether or not to move the left and right images in a linked manner when adjusting the adjustment slider 1201 in Fig. 12. In this embodiment, whether or not to adjust the left and right images in a linked manner can be set by the link setting 1203 in Fig. 12. Also, in this embodiment, a UI is displayed and the user manually switches whether or not to adjust the left and right images in a linked manner, but the switching may be automatic.

[0120] For example, in the case of automatic switching, adjustments can be made as follows. That is, when the reference image is adjusted, the focus is considered to be poor overall, so adjustments are made in tandem. On the other hand, when the adjusted image is adjusted, the focus is considered to be out of focus with respect to the reference image, so adjustments are made individually without tandem. The image display means 2007 displays an image of the right eye optical system and an image of the left eye optical system on the display unit 606.

[0121] In this manner, in this embodiment, the processing means 1100 has a refocus adjustment means capable of adjusting the refocus process by the refocus means 2004 for at least one of the first image data and the second image data. The refocus adjustment means corresponds to at least one of the focus index display means 2005, the linked adjustment switching means 2006, and the image constant means 2007.

[0122] Next, the operation (image processing method) of the processing means 1100 according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the operation procedure of the processing means 1100.

[0123] In step S2001, the image acquisition means 2000 acquires an image from, for example, the recording medium 604 (reads the image).

[0124] In step S2002, the additional information acquisition unit 2001 acquires the additional information.

[0125] In step S2003, the reference image determining means 2002 determines a reference image and an adjustment image.

[0126] In step S2004, the image display unit 2007 displays the image of the right eye optical system and the image of the left eye optical system on the display unit 606.

[0127] In step S2005, the focus adjustment unit 2003 adjusts the focus of the image of the right eye optical system and the image of the left eye optical system, and updates the images.

[0128] In step S2006, if the user decides to continue the adjustment, the process returns to step S2004, the display image is updated, and focus adjustment is continued, whereas if the user decides to end the adjustment, this flowchart ends.

[0129] According to this embodiment, in an image captured in multi-view shooting mode, the focus of the image of the right eye optical system and the image of the left eye optical system can be adjusted separately, and focus deviation of these images can be reduced.

[0130] <Other embodiments> The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0131] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, and a program capable of adjusting the focus shift between two images having parallax after capturing the images.

[0132] The disclosure of each embodiment includes the following configurations and methods. (Configuration 1) an image acquisition means for acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; a refocusing unit capable of performing a refocusing process independently on each of the first image data and the second image data; and a focus adjustment means for adjusting a difference in focus between the first image data and the second image data by performing the refocusing process on at least one of the first image data or the second image data using the refocusing means. (Configuration 2) The image processing device further includes a reference image determination unit for determining one of the first image data and the second image data as reference image data, 2. The image processing device according to configuration 1, wherein the focus adjustment means performs the refocusing process on the other of the first image data or the second image data by the refocusing means. (Configuration 3) 3. The image processing apparatus according to configuration 2, further comprising an additional information acquisition unit for acquiring additional information corresponding to each of the first image data and the second image data. (Configuration 4) 4. The image processing apparatus according to claim 3, wherein the reference image determination means determines the reference image data by using the additional information. (Configuration 5) The image processing device according to configuration 3 or 4, wherein the focus adjustment means adjusts the difference in the degree of focus between the first image data and the second image data at a predetermined position determined by the additional information. (Configuration 6) 6. The image processing device according to any one of configurations 2 to 5, wherein the reference image determination means determines the reference image data based on the degree of focus of each of the first image data and the second image data. (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, wherein the focus adjustment means displays a warning when it determines that the difference in the degree of focus cannot be adjusted. (Configuration 8) the first image data includes a first image, and a first parallax image and a second parallax image having a parallax with respect to each other; the first image is an image obtained by combining the first parallax image and the second parallax image, the second image data includes a second image having a parallax with respect to the first image, and a third parallax image and a fourth parallax image having a parallax with respect to each other; the second image is an image obtained by combining the third parallax image and the fourth parallax image, The refocusing means is The refocusing process can be performed on the first image by using the first parallax image and the second parallax image, 8. The image processing device according to any one of configurations 1 to 7, wherein the refocusing process for the second image is performed by using the third parallax image and the fourth parallax image. (Configuration 9) the first parallax image and the second parallax image are images corresponding to light beams passing through different pupil regions of the first optical system, 9. The image processing device according to configuration 8, wherein the third parallax image and the fourth parallax image are images corresponding to light beams passing through different pupil regions of the second optical system. (Configuration 10) an image acquisition means for acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; a refocusing unit capable of performing a refocusing process independently on each of the first image data and the second image data; and a refocus adjustment unit capable of adjusting the refocus process by the refocus unit for at least one of the first image data and the second image data. (Configuration 11) 11. The image processing device according to configuration 10, wherein the refocus adjustment means adjusts the refocus processing independently for each of the first image data and the second image data. (Configuration 12) 11. The image processing device according to configuration 10, wherein the refocus adjustment means adjusts the refocus processing in conjunction with the first image data and the second image data. (Configuration 13) The image processing device described in configuration 10, characterized in that the refocus adjustment means has a switching means for switching between adjusting the refocus processing independently for each of the first image data and the second image data, or adjusting the refocus processing in conjunction with the first image data and the second image data. (Configuration 14) 14. The image processing device according to any one of configurations 10 to 13, wherein the refocus adjustment means has a focus index display means for displaying an index indicating a degree of focus between the first image data and the second image data. (Configuration 15) An imaging element; 15. An imaging device comprising the image processing device according to any one of configurations 1 to 14. (Method 1) acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; and adjusting a difference in focus between the first image data and the second image data by performing a refocus process on at least one of the first image data or the second image data. (Method 2) acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; and performing a display for adjusting a refocus process for at least one of the first image data or the second image data. (Configuration 16) A program for causing a computer to execute the image processing method according to the method 1 or 2. (Configuration 17) A computer-readable storage medium storing the program according to configuration 16.

[0133] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined. [Explanation of symbols]

[0134] 301R Right eye optical system (1st optical system) 301L Left eye optical system (second optical system) 900, 1100 Processing means (image processing device) 1000 Image Acquisition Means 1003 Focus adjustment means 1004 Refocusing means

Claims

1. an image acquisition means for acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; a refocusing unit capable of performing a refocusing process independently on each of the first image data and the second image data; An image processing device characterized by having a focus adjustment means that adjusts the difference in focus between the first image data and the second image data by performing the refocus processing on at least one of the first image data or the second image data using the refocus means.

2. The image processing device further includes a reference image determination unit for determining one of the first image data and the second image data as reference image data, 2 . The image processing apparatus according to claim 1 , wherein the focus adjustment means performs the refocusing process on the other of the first image data and the second image data by the refocusing means.

3. 3. The image processing apparatus according to claim 2, further comprising: an additional information acquisition unit for acquiring additional information corresponding to each of the first image data and the second image data.

4. 4. The image processing apparatus according to claim 3, wherein said reference image determining means determines said reference image data by using said supplementary information.

5. 4. The image processing apparatus according to claim 3, wherein the focus adjustment means adjusts the difference in the degree of focus between the first image data and the second image data at a predetermined position determined by the additional information.

6. 3. The image processing apparatus according to claim 2, wherein the reference image determination means determines the reference image data based on the degree of focus of each of the first image data and the second image data.

7. 2. The image processing apparatus according to claim 1, wherein the focus adjustment means displays a warning when it determines that the difference in the degree of focus cannot be adjusted.

8. the first image data includes a first image, and a first parallax image and a second parallax image having a parallax with respect to each other; the first image is an image obtained by combining the first parallax image and the second parallax image, the second image data includes a second image having a parallax with respect to the first image, and a third parallax image and a fourth parallax image having a parallax with respect to each other; the second image is an image obtained by combining the third parallax image and the fourth parallax image, The refocusing means is The refocusing process can be performed on the first image by using the first parallax image and the second parallax image, The image processing device according to claim 1 , wherein the refocusing process for the second image is performed by using the third parallax image and the fourth parallax image.

9. the first parallax image and the second parallax image are images corresponding to light beams passing through different pupil regions of the first optical system, 9. The image processing apparatus according to claim 8, wherein the third parallax image and the fourth parallax image are images corresponding to light beams passing through different pupil regions of the second optical system.

10. an image acquisition means for acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; a refocusing unit capable of performing a refocusing process independently on each of the first image data and the second image data; and a refocus adjustment unit capable of adjusting the refocus process by the refocus unit for at least one of the first image data and the second image data.

11. 11. The image processing apparatus according to claim 10, wherein the refocus adjustment means adjusts the refocus processing independently for each of the first image data and the second image data.

12. 11. The image processing apparatus according to claim 10, wherein the refocus adjustment means adjusts the refocus processing in conjunction with the first image data and the second image data.

13. The image processing device according to claim 10, characterized in that the refocus adjustment means has a switching means for switching between adjusting the refocus processing independently for each of the first image data and the second image data, or adjusting the refocus processing in conjunction with the first image data and the second image data.

14. 11. The image processing apparatus according to claim 10, wherein the refocus adjustment means comprises a focus index display means for displaying an index indicating a degree of focus between the first image data and the second image data.

15. An imaging element; An imaging device comprising the image processing device according to claim 1 .

16. acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; An image processing method comprising a step of adjusting a difference in focus degree between the first image data and the second image data by performing a refocus process on at least one of the first image data or the second image data.

17. acquiring first image data obtained by imaging via a first optical system and second image data obtained by imaging via a second optical system; and performing a display for adjusting a refocus process for at least one of the first image data or the second image data.

18. A program for causing a computer to execute the image processing method according to claim 16 or 17.

19. 20. A computer-readable storage medium storing the program according to claim 18.

Images