Image processing device, control method for image processing device, imaging device, and program

Abstract

The present invention provides an image processing device that allows the photographer to easily visually identify the occurrence of occlusion in the subject. [Solution] The image processing apparatus according to the present disclosure includes: an image acquisition means for acquiring a plurality of images that include a common subject and have parallax; a determination means for determining a specific subject in which occlusion occurs among the plurality of images; and a generation means for generating information that makes it visible that occlusion has occurred in the specific subject in a display image based on at least one of the plurality of images.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method for an image processing apparatus, an imaging apparatus, and a program.

Background Art

[0002] In recent years, imaging apparatuses capable of capturing stereoscopic video (hereinafter referred to as 3D video or 3D image) have been known. Among such imaging apparatuses, in a lens-exchangeable camera, there has been proposed an apparatus that creates parallax image data by imaging left and right images on either side of the center using a single imaging element via a special binocular lens (Patent Document 1).

[0003] In addition, techniques have also been proposed for processing a plurality of images captured with compound eyes having different viewpoints, synthesizing them into a single display image, and displaying the image (Patent Documents 2 and 3).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, stereoscopic vision is achieved by perceiving depth through the parallax of binocular images. When occlusion occurs, a situation may arise where the subject is visible to one eye but hidden to the other. In such cases, the correspondence between the subjects in both eyes cannot be established, making it impossible to acquire parallax information of the subject. In other words, if 3D images shot under such conditions are used, viewers may not be able to experience normal stereoscopic vision and may feel a sense of unease. On the other hand, if the photographer can appropriately grasp the occlusion of the subject at the time of shooting, it becomes easier to shoot the 3D image intended by the photographer. Patent documents 1-3 mentioned above did not take into account the photographer's ability to easily grasp occlusion.

[0006] This invention has been made in view of the above problems, and its purpose is to realize a technology that allows the photographer to easily visually identify the occurrence of occlusion in the subject. [Means for solving the problem]

[0007] To solve this problem, for example, the image processing apparatus of the present invention has the following configuration: an image acquisition means for acquiring a plurality of images that include a common subject and have parallax; a determination means for determining a specific subject in which occlusion occurs among the plurality of images; and a generation means for generating information that makes it visible that the occlusion occurs in the specific subject in a display image based on at least one of the plurality of images. [Effects of the Invention]

[0008] According to the present invention, the photographer will be able to easily visually identify the occurrence of occlusion in the subject. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing an example of the external configuration of the camera in Embodiment 1. [Figure 2] A diagram showing an example of the internal configuration of the camera in Embodiment 1. [Figure 3] A schematic diagram showing an example of the configuration of the lens unit of Embodiment 1. [Figure 4] Block diagram showing an example of the functional configuration of Embodiment 1 [Figure 5] Flowchart showing the operation of the occlusion information generation process in Embodiment 1 [Figure 6] A diagram illustrating an example of a shooting scene in Embodiment 1. [Figure 7] A diagram illustrating an example of an image captured in Embodiment 1. [Figure 8] A diagram illustrating the selection of the main subject in Embodiment 1. [Figure 9] A diagram illustrating the base image of Embodiment 1. [Figure 10] A diagram illustrating an example of an occlusion display image in Embodiment 1. [Figure 11] A diagram illustrating the base image of Embodiment 2. [Figure 12] A diagram illustrating an example of an occlusion display image in Embodiment 2. [Figure 13] A figure showing one form of the occlusion display image of Embodiment 3. [Figure 14] A figure showing another form of the occlusion display image of Embodiment 3. [Modes for carrying out the invention]

[0010] (Embodiment 1) The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] Hereinafter, as an example of an image processing apparatus, an example using a digital camera capable of processing a plurality of images with parallax will be described. However, this embodiment is not limited to a digital camera, and is also applicable to other devices capable of processing a plurality of images with parallax. These devices may include, for example, smartphones, game machines, tablet terminals, wearable information terminals, medical devices, and the like.

[0012] <Overall Hardware Configuration> FIG. 1 is a diagram showing an example of the external configuration of a digital camera 100 (hereinafter referred to as a camera). FIG. 1(a) is a perspective view of the camera 100 seen from the front, and FIG. 1(b) is a perspective view of the camera 100 seen from the back.

[0013] On the upper surface of the camera 100, there are a shutter button 101, a power switch 102, a mode switch 103, a main electronic dial 104, a sub - electronic dial 105, a video button 106, and an external - finder display unit 107. The shutter button 101 is an operation unit for performing shooting preparation or a shooting instruction. The power switch 102 is an operation unit for switching on and off the power of the camera 100. The mode switch 103 is an operation unit for switching various modes. The main electronic dial 104 is a rotary operation unit for changing set values such as shutter speed and aperture. The sub - electronic dial 105 is a rotary operation unit for moving a selection frame (cursor) or scrolling images. The video button 106 is an operation unit for instructing the start and stop of video shooting (recording). The external - finder display unit 107 displays various set values such as shutter speed and aperture.

[0014] Camera 100 has a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back. The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 is an operation unit consisting of keys that can be pressed up, down, left, and right (4-way keys). Operations can be performed according to the position in which the directional keys 110 are pressed. The SET button 111 is an operation unit that is mainly pressed when confirming a selection item. The AE lock button 112 is an operation unit that is pressed when fixing the exposure state in shooting standby mode. The zoom button 113 is an operation unit that switches the zoom mode on and off in the live view display (LV display) of the shooting mode. When the magnification mode is on, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 104. The magnification button 113 is used in playback mode to enlarge the playback image or increase the magnification ratio. The playback button 114 is an operation unit for switching between shooting mode and playback mode. By pressing the playback button 114 in shooting mode, the camera switches to playback mode, and the latest image recorded on the recording medium 228 (described later) can be displayed on the display unit 108.

[0015] The menu button 115 is an operation button that is pressed to display a menu screen on the display unit 108 that allows for various settings. Users can intuitively make various settings using the menu screen displayed on the display unit 108 and the directional keys 110 and SET button 111. The eyepiece section 116 is the part for placing the eye into the eyepiece viewfinder (peep-in type viewfinder) 117. Users can view the image displayed on the internal EVF 217 (Electronic View Finder), which will be described later, through the eyepiece section 116. The eyepiece detection section 118 is a sensor that detects whether or not the user is placing their eye into the eyepiece section 116.

[0016] The touch bar 119 is a line-shaped touch operation area (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched by the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger) so that the shutter button 101 can be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece viewfinder 117 with their eyepiece in the eyepiece section 116 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (touching and releasing without moving within a predetermined period), left and right sliding operations (touching and then moving the touch position while keeping the touch on the surface), etc. The touch bar 119 is a different operation area from the touch panel 109 and does not have a display function. The touch bar 119 in this embodiment is a multifunction bar and functions as, for example, an M-Fn bar.

[0017] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. With the camera 100 held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section located on the back of the camera 100, in a place where the thumb of the right hand holding the grip section 120 can be easily rested when no other controls are being operated. The thumb rest section 121 is made of a rubber material or the like to enhance the holding force (grip feel). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The cover 123 protects the recording medium 228 and the slot for storing the recording medium 228, which will be described later, by blocking the slot. The communication terminal 124 is a terminal for the camera 100 to communicate with the lens unit 200, which will be described later and can be attached to and detached.

[0018] <Internal configuration of the camera> Figure 2 shows an example of the internal configuration of camera 100. Components identical to those in Figure 1 are denoted by the same reference numerals, and their descriptions are omitted as appropriate. A lens unit 200 is attached to camera 100.

[0019] First, let me explain the lens unit 200.

[0020] The lens unit 200 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 200 is a single-lens reflex lens and is an example of a typical lens. Here, a single-lens reflex lens example is described for the sake of simplifying the explanation of the hardware configuration, but the twin-lens reflex lens of this embodiment, which will be described later, can be attached as shown in Figure 3.

[0021] The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0022] The aperture 201 is configured to have an adjustable aperture diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc., based on instructions from the system control unit 218, which will be described later. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203 and focuses by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication occurs via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100.

[0023] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 218.

[0024] The shutter 210 is a focal-plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 218. The imaging unit 211 is an image sensor composed of a CCD or CMOS element that converts an optical image into an electrical signal. It is also possible to achieve the exposure control performed using the shutter 210 by controlling the imaging unit 211 (electronic shutter). In that case, it is also possible to have a configuration in which the shutter 210 is removed from the imaging device (mechanical shutterless). The imaging unit 211 may have an image plane phase difference sensor that outputs defocus amount information to the system control unit 218. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined calculation processing using the captured image data, and the system control unit 218 performs exposure control and distance measurement control based on the obtained calculation results. This process performs TTL (Through-the-Lens) AF processing, AE (Automatic Exposure) processing, EF (Flash Pre-flash) processing, etc. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs TTL AWB (Automatic White Balance) processing based on the obtained calculation results.

[0025] Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data for display on the display unit 108 and EVF 217. The memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. The memory 215 also serves as a memory for image display (video memory). Furthermore, the memory 215 may store data of occlusion information (occlusion display images) generated by the occlusion information generation process described later.

[0026] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and EVF 217 are displays such as LCDs and OLEDs. The digital signal, which has been A / D converted by the A / D converter 212 and stored in the memory 215, is converted into an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and EVF 217 for display, thereby performing live view display. In addition, the display unit 108 and EVF 217 can display occlusion information, which will be described later.

[0027] The system control unit 218 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 218 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 218 controls the entire camera 100. The system control unit 218 implements each process in the flowchart described later by executing a program recorded in the non-volatile memory 220. The system control unit 218 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0028] The camera 100 also includes a system memory 219, a non-volatile memory 220, a system timer 221, a communication unit 222, a posture detection unit 223, and an eyepiece detection unit 118.

[0029] System memory 219 may be RAM, for example. System memory 219 stores constants and variables for the operation of the system control unit 218, programs read from non-volatile memory 220, and the like.

[0030] The non-volatile memory 220 is an electrically erasable and recordable memory, such as an EEPROM. The non-volatile memory 220 stores constants for the operation of the system control unit 218, programs, and the like. The program referred to here is a program for executing the flowchart described later.

[0031] The system timer 221 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 222 transmits and receives video and audio signals to and from external devices connected wirelessly or via wired cable. The communication unit 222 can also connect to a wireless LAN (Local Area Network) or the internet. Furthermore, the communication unit 222 can communicate with external devices using Bluetooth® or Bluetooth Low Energy.

[0032] The communication unit 222 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 228, and can receive image data and other various information from external devices.

[0033] The attitude detection unit 223 detects the attitude of the camera 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 223, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 218 can also add orientation information corresponding to the attitude detected by the attitude detection unit 223 to the image file of the image captured by the imaging unit 211, or rotate the image before recording. The attitude detection unit 223 can use, for example, an acceleration sensor or a gyroscope. It is also possible to use the attitude detection unit 223 to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.).

[0034] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the eyepiece viewfinder 117, which incorporates the EVF 217. The eyepiece detection unit 118 can use, for example, an infrared proximity sensor. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece section 116 to the object can be determined by the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece section 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eye-catching) and departure (eye-moving) of an eye (object) to the eyepiece section 116 of the eyepiece viewfinder 117. When an object is detected approaching within a predetermined distance from the eyepiece section 116 from a non-eye-catching state (non-approaching state), it is detected that the eye has been caught in the eye. On the other hand, if the object that was detected as approaching moves beyond a predetermined distance from the eyepiece state (close-up state), it is detected that the eye has been removed. The threshold for detecting eyepiece contact and the threshold for detecting eye removal may be different, for example, by providing hysteresis. After eyepiece contact is detected, the eyepiece state is maintained until eye removal is detected. After eye removal is detected, the non-eyepiece state is maintained until eyepiece contact is detected again. The system control unit 218 switches the display (display state) / non-display (non-display state) of the display unit 108 and EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state and when the display destination switching setting is set to automatic switching, when the eyepiece is not in use, the display destination is set to the display unit 108 and the display is turned on, and the EVF 217 is hidden. When eyepiece contact is made, the display destination is set to the EVF 217 and the display is turned on, and the display unit 108 is hidden. Furthermore, the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as an eyepiece may be used.

[0035] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 224, a power control unit 225, a power supply unit 226, a recording medium interface 227, an operation unit 229, and the like. The external viewfinder display unit 107 displays various settings of the camera 100, such as shutter speed and aperture, via the external viewfinder display drive circuit 224.

[0036] The power control unit 225 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 225 also controls the DC-DC converter based on the detection results and instructions from the system control unit 218, supplying the necessary voltage to each part, including the recording medium 228, for the required period. The power supply unit 226 consists of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and an AC adapter. The recording medium I / F 227 is an interface with the recording medium 228, such as a memory card or hard disk. The recording medium 228 is a memory card or the like for recording captured images, and consists of semiconductor memory or a magnetic disk. The recording medium 228 may be removable or built-in.

[0037] The operation unit 229 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 218. The operation unit 229 includes the shutter button 101, power switch 102, mode switch 103, touch panel 109, and other operation units 230. The other operation units 230 include the main electronic dial 104, sub electronic dial 105, video button 106, directional keys 110, SET button 111, AE lock button 112, zoom button 113, playback button 114, menu button 115, touch bar 119, and the like.

[0038] The shutter button 101 has a first shutter switch 231 and a second shutter switch 232. The first shutter switch 231 turns on during the operation of the shutter button 101, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. The system control unit 218 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 232 turns on when the operation of the shutter button 101 is completed, so-called full-press (shooting instruction), and generates a second shutter switch signal SW2. The system control unit 218 starts a series of shooting processes from reading the signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 228 in response to the second shutter switch signal SW2.

[0039] The mode switch 103 switches the operating mode of the system control unit 218 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. Modes included in still image shooting mode include, but are not limited to, auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode).

[0040] Furthermore, various scene modes, custom modes, etc., which are shooting settings for different shooting scenes, may be set via the mode switch 103. For example, the user can directly switch to any of the above-mentioned shooting modes using the mode switch 103. Alternatively, the user can switch to a list screen of shooting modes using the mode switch 103, and then selectively switch to any of the displayed modes using the operation unit 229. Similarly, the video shooting mode may also include multiple modes.

[0041] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is mounted on the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109; either method is acceptable.

[0042] The system control unit 218 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). The finger or pen that was touching the touch panel 109 is lifted (released), meaning the touch action ends (hereinafter referred to as "Touch-Up"). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0043] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0044] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 218 via the internal bus. Based on the notified information, the system control unit 218 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation was performed. An operation in which a finger is touched on the touch panel 109 and then quickly moved a certain distance and then released is called a flick. In other words, a flick is an operation in which the finger is quickly traced across the touch panel 109 as if flicking it. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected, and a touch-up is then detected, it is determined that a flick was performed (it can be determined that a flick followed a slide operation). Furthermore, touching multiple points (for example, two points) simultaneously (multitouch) to bring them closer together is called pinch-in, and touching them further apart is called pinch-out. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch).

[0045] <Lens Unit Configuration> Figure 3 is a schematic diagram showing an example of the configuration of the lens unit 300. Figure 3 shows the lens unit 300 attached to the camera 100. Note that components of the camera 100 shown in Figure 3 that are the same as those described in Figure 2 are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0046] The lens unit 300 is an example of a detachable interchangeable lens for the camera 100. The lens unit 300 is a twin-lens system capable of capturing two images with parallax between the left and right images. For example, the lens unit 300 has two optical systems, each with a wide field of view of approximately 180 degrees, and can capture an area of ​​the front hemisphere. Specifically, by using the two optical systems of the lens unit 300, the camera 100 can capture subjects with a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle).

[0047] The lens unit 300 includes a right-eye optical system 301R having multiple lenses and a reflective mirror, a left-eye optical system 301L having multiple lenses and a reflective mirror, and a lens system control circuit 303. The right-eye optical system 301R corresponds to an example of a first optical system, and the left-eye optical system 301L corresponds to an example of a second optical system. In the right-eye optical system 301R and the left-eye optical system 301L, the lenses 302R and 302L located on the subject side are oriented in the same direction, and their optical axes are approximately parallel. The lens unit 300 of this embodiment is a VR180 lens for capturing images for so-called VR180, which is a VR image format that enables binocular stereoscopic viewing. The VR180 lens has a fisheye lens in the right-eye optical system 301R and the left-eye optical system 301L, each capable of capturing a range of approximately 180 degrees. The VR180 lens only needs to be capable of acquiring images that enable bi-eye VR display as VR180, with the right eye optical system 301R and the left eye optical system 301L each capable of capturing a wide field of view of about 160 degrees, which is narrower than the 180-degree range. The VR180 lens can form a right image (first image) formed via the right eye optical system 301R and a left image (second image) formed via the left eye optical system 301L, which has parallax with the right image, on one or more image sensors of the camera to which it is mounted. The lens unit 300 is equipped with a focus ring for adjusting the focus. Although not shown in Figure 3, the lens unit 300 is equipped with a focus ring for adjusting the focus of the right image formed via the right eye optical system 301R and a focus ring for adjusting the focus of the left image formed via the left eye optical system 301L. Alternatively, the lens unit 300 includes a focus ring that simultaneously adjusts the focus of both the right image formed via the right eye optical system 301R and the left image formed via the left eye optical system 301L, and a focus ring that adjusts the focus of either one of the images.

[0048] Furthermore, the lens unit 300 is attached to the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. When the lens unit 300 is attached to the camera 100, the system control unit 218 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300. Communication terminal 206 corresponds to communication terminal 306.

[0049] In this embodiment, the right image formed via the right-eye optical system 301R and the left image formed via the left-eye optical system 301L, which has parallax with respect to the right image, are imaged side by side on the imaging unit 211 of the camera 100. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single image sensor. The imaging unit 211 converts the imaged subject (optical signal) into an analog electrical signal. In this way, by using the lens unit 300, two images with parallax can be acquired simultaneously (as a set) from two locations (optical systems) - the right-eye optical system 301R and the left-eye optical system 301L. Furthermore, by separating the acquired images into an image for the left eye and an image for the right eye and performing VR display, the user can view a three-dimensional VR image with a range of approximately 180 degrees, so-called VR180.

[0050] VR images are images that enable VR display, as described later. For example, VR images include omnidirectional images (spherical images) taken with an omnidirectional camera (spherical camera), and panoramic images with an effective range (also called the effective image range) that is wider than the display range that can be displayed on the display unit at once. Furthermore, VR images are not limited to still images, but also include videos and live images (images acquired from the camera in near real time). For example, a VR image has an effective image range of up to 360 degrees horizontally and 360 degrees vertically. In addition, VR images also include images that have a wider field of view than that that can be captured by a normal camera, or an effective range that is wider than the display range that can be displayed on the display unit at once, even if the field of view is less than 360 degrees horizontally or vertically. Images captured by camera 100 using the lens unit 300 described above are a type of VR image. VR images can be displayed in VR, for example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR View". By displaying VR images with a 360-degree field of view in VR, users can change the orientation of the display device left or right (horizontal rotation direction) to view seamless, all-around images in the left-right direction.

[0051] Here, VR display (VR view) refers to a display method (display mode) in which the display range can be changed, displaying an image from among VR images that corresponds to the orientation of the display device. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by performing a transformation (a transformation that applies distortion correction) that maps the VR image to a virtual sphere. VR display also includes "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side-by-side in the left and right regions by performing transformations that map them to virtual spheres. Stereoscopic viewing is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax relative to each other. In any VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image (e.g., video) corresponding to the user's face orientation will be displayed. For example, suppose that among the VR images, at a certain point in time, an image is displayed with a field of view centered at 0 degrees horizontally (a specific direction, e.g., north) and 90 degrees vertically (90 degrees from the zenith, i.e., horizontal). If the orientation of the display device is reversed from this state (for example, changing the display surface from facing south to facing north), the display range of the same VR image will change to an image with a field of view centered at 180 degrees horizontally (opposite direction, e.g., south) and 90 degrees vertically. In other words, if the user, while wearing the HMD, turns their face from north to south (i.e., turns their back), the image displayed on the HMD will also change from a northern image to a southern image. Note that the VR image captured using the lens unit 300 of this embodiment is a VR180 image that captures a range of approximately 180 degrees in front, and there is no image of a range of approximately 180 degrees behind. If such a VR180 image is displayed in VR and the orientation of the display device is changed to the side where there is no image, a blank area will be displayed, for example.

[0052] By displaying VR images in this way, users can visually experience the sensation of being inside the VR image (in a VR space). Note that the method of displaying VR images is not limited to changing the orientation of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel or directional buttons. Furthermore, during VR display (in "VR View" display mode), in addition to changing the display range due to changes in orientation, the display range may also be changed in response to touch movements on the touch panel, drag operations with a mouse, or pressing directional buttons. Note that a smartphone mounted on VR goggles (head-mounted adapter) is a type of HMD (Head-Mounted Display).

[0053] <Explanation of each processing method> The following description will explain, using as an example the case in which the generation and display of an image that clearly indicates the presence of occlusion (occlusion display image) in this embodiment is performed by a camera 100 and a two-lens unit 300. The display in this embodiment differs in its purpose of image generation and display from two-lens VR display, which generates a left-eye image and a right-eye image mainly for stereoscopic viewing and displays these images side by side in the left and right regions. This embodiment provides a technology that allows the photographer to easily see the state of occlusion (hiddenness) of the subject in real time during shooting.

[0054] Figure 4 shows an example of the functional configuration of the camera 100 in Embodiment 1. The first image acquisition unit 401 and the second image acquisition unit 402 sequentially acquire the captured images acquired by the imaging unit 211 of the camera 100 as image signals in real time via the lens unit 300. If the optical system corresponding to the first image acquisition unit 401 is the right eye optical system 301R, then the optical system corresponding to the second image acquisition unit 402 is the left eye optical system 301L. That is, the first image acquisition unit 401 and the second image acquisition unit 402 constitute an image acquisition means that acquires multiple (in this case, two) images from a single image signal output from the imaging unit 211. The first image acquisition unit 401 and the second image acquisition unit 402 can acquire multiple images that include a common subject and have parallax. Each of the multiple images corresponds to an image obtained through a different optical system from the right eye optical system 301R and the left eye optical system 301L. In this embodiment, an example of acquiring two images using a lens unit 300 having two optical systems is described, but this embodiment is also applicable when acquiring (more) multiple images using a lens unit with more optical systems. Furthermore, in this embodiment, an example is described in which the first image acquisition unit 401 and the second image acquisition unit each acquire images, but multiple images may be acquired by a single image acquisition unit.

[0055] The subject detection unit 403 performs subject detection processing on each of the images acquired by the first image acquisition unit 401 and the second image acquisition unit 402. It detects subjects such as people, animals, and vehicles that have been set in advance as targets for detection, and stores the position, size, and type of the detected subjects as a list of subjects in the system memory 219 or non-volatile memory 220. The subject detection unit 403 may perform subject detection using known pattern matching or using a machine learning model trained with known Deep Learning.

[0056] The main subject selection unit 404 selects (determines) the subject to be displayed if occlusion occurs in the subject detected by the subject detection unit 403. This subject selection process is performed according to the main subject selection conditions set in advance by the photographer. The selection conditions include, for example, at least one of the size of the subject relative to the shooting angle of view and the type of subject. However, the selection conditions are not limited to these and may include other conditions as well. In other words, the main subject selection unit 404 selects (determines) a specific subject in which occlusion is occurring from among the detected subjects that satisfy the selection conditions corresponding to being a main subject.

[0057] When selecting a primary subject based on its size relative to the field of view, for example, the photographer may set a threshold Lmin for the minimum size of the primary subject. In this case, the primary subject selection unit 404 selects one or more subjects as primary subjects whose detected size (e.g., the proportion of the subject's field of view) is greater than the threshold Lmin. The threshold Lmin can be set, for example, using a percentage of the field of view for each eye. In this case, the subject size can be determined, for example, using the length of the longer side of the rectangular circumscribing frame surrounding the detected subject.

[0058] Furthermore, when selecting a primary subject based on the type of subject, the type of primary subject to be selected may be set by the photographer. In this case, the primary subject selection unit 404 may, for example, select a subject determined to be a person as the primary subject, based on a pre-set setting of a person as the primary subject. If the photographer sets multiple selection conditions, such as the size of the subject relative to the shooting angle of view and the type of subject, the primary subject selection unit 404 may select a subject that satisfies any of the set selection conditions as the primary subject. Alternatively, the primary subject selection unit 404 may select a subject that satisfies all of the set selection conditions as the primary subject. The primary subject selection unit 404 stores the detection results of one or more primary subjects as a subject list in the system memory 219 or non-volatile memory 220.

[0059] The image synthesis unit 405 synthesizes the images from both eyes to generate a composite image (also called a display image or base image) that serves as the base for displaying occlusion. Since the images from the two eyes are taken from different viewpoints, simply superimposing the two images would result in a blurry image of subjects with large parallax, as if they were blurred. Therefore, the image synthesis unit 405 generates a composite image with the viewpoints of both eyes aligned to create an image that is easy for the photographer to view. First, the image synthesis unit 405 pre-sets a virtual viewpoint position for synthesizing the two images. Here, the virtual viewpoint position is set to an intermediate position between the right eye optical system 301R and the left eye optical system 301L. For each of the two images (right image and left image), the image synthesis unit 405 performs a viewpoint transformation process to the set virtual viewpoint using a known perspective projection transformation for the main subject selected by the main subject selection unit 404. For example, in the viewpoint transformation process, the image synthesis unit 405 estimates the subject distance for the main subject selected by the main subject selection unit 404 and performs viewpoint alignment processing for the subject in the left and right images according to the subject distance. The image synthesis unit 405 uses the viewpoint alignment processing result to superimpose the image area of ​​the main subject after viewpoint transformation in the left image and the image area of ​​the main subject after viewpoint transformation in the right image and performs averaging processing to generate a composite image with the viewpoints of both eyes aligned (i.e., with reduced parallax). Here, the image synthesis unit 405 also performs processing to determine whether occlusion has occurred for each subject. For example, the image synthesis unit 405 compares the list of subjects detected by the right eye optical system 301R and the left eye optical system 301L, and can determine if occlusion has occurred from the results of the subject distance estimation for the main subject detected by only one eye. If the image synthesis unit 405 determines that occlusion has occurred for a particular main subject, it adds a flag to the subject list indicating that occlusion has occurred.

[0060] The determination of whether the main subject in the left image and the main subject in the right image are the same corresponding subject, and the estimation of the subject distance, can be performed by calculating the correlation between the left and right images in the direction along the epipolar line for a rectangular region surrounding the main subject. This calculation of the correlation between the left and right images can be performed using known techniques, such as SAD (Sum of Absolute Difference) calculation in the direction along the epipolar line. It is assumed that the viewpoint distance between the two eyes and the camera's internal parameters in perspective projection transformation are known from the optical design values ​​of the lens unit 300 or from prior camera calibration. If, through such processing, the main subject in the left image and the main subject in the right image are the same corresponding subject and the subject distance can be estimated, the image synthesis unit 405 can determine that no occlusion has occurred in that main subject. Also, if the subject distance of a certain main subject (for example, a main subject detected by only one eye) cannot be estimated, the image synthesis unit 405 can determine that occlusion has occurred in that main subject. In this way, the image synthesis unit 405 can determine subjects where occlusion has occurred between multiple images.

[0061] Furthermore, these processes described above with reference to Figure 4 can be implemented by the system control unit 218 or the image processing unit 214 executing a computer program stored, for example, in the non-volatile memory 220. Also, the processes described above with reference to Figure 4 are not limited to those described above. For example, the image synthesis unit 405 may perform a process to pass through the image of one eye from among the images acquired by the first image acquisition unit 401 or the second image acquisition unit 402.

[0062] The occlusion information generation unit 406 generates occlusion information for a subject that has been determined to be occluded in the previous step, making it visible that occlusion is occurring in that subject. The occlusion information includes, for example, an image (also called an occlusion display image) obtained by processing the base image generated by the image synthesis unit 405 so that the subject experiencing occlusion is visible to the photographer. The occlusion information generation unit 406 can, for example, perform contour enhancement display that emphasizes the edges of the outline of the subject experiencing occlusion as occlusion information. These processes can also be realized by the system control unit 218 or the image processing unit 214 executing a computer program stored in, for example, the non-volatile memory 220.

[0063] The display control unit 407 controls and processes the display of the occlusion display image generated by the occlusion information generation unit 406 on the display unit 108 of the camera 100.

[0064] Next, with reference to Figure 5, a series of operations for occlusion information generation will be described. The occlusion information generation process is performed by each part of the camera 100 shown in Figure 4. That is, the occlusion information generation process can be realized by the system control unit 218 or the image processing unit 214 executing a computer program stored, for example, in the non-volatile memory 220. The occlusion information generation process according to this embodiment can be applied, for example, when capturing a shooting scene as illustrated in Figure 6. Specifically, Figure 6 schematically shows a shooting scene captured by a camera 100 equipped with a lens unit 300 having a right-eye optical system 301R and a left-eye optical system 301L. As described above, the first image acquisition unit 401 corresponds to the right-eye optical system 301R, and the second image acquisition unit 402 corresponds to the left-eye optical system 301L. The right-eye optical system 301R and the left-eye optical system 301L are optical systems that have parallax but share a common field of view. In the shooting scene illustrated in Figure 6, the person 601 is obscured by the tree 602 in the right eye optical system 301R, resulting in occlusion. The other subjects, the animal 603 and the flower 604, are within the field of view of both eyes (i.e., the right eye optical system 301R and the left eye optical system 301L). As will be discussed later, for the sake of explanation, the four subjects shown here are limited to those detected in the subject detection step S504.

[0065] In step S501, the image synthesis unit 405 sets a virtual viewpoint position (display viewpoint setting) for the occlusion display image. In subsequent processing, the image synthesis unit 405 uses the display viewpoint setting to convert multiple images acquired by the binoculars from different viewpoints into an image at a single virtual viewpoint position, and generates an image by synthesizing the images at that virtual viewpoint position. In this embodiment, the midpoint of the viewpoints of the two eyes is used as an example to explain the case where the virtual viewpoint position is set, but the photographer may set the virtual viewpoint position by operating the touch panel 109 or the operation unit 229, etc.

[0066] In step S502, the first image acquisition unit 401 and the second image acquisition unit 402 acquire binocular images from the binocular images captured. Figure 7 shows an example of captured images when the shooting scene shown in Figure 6 is shot with binoculars. This image shows a configuration in which the right eye image 600R and the left eye image 600L are captured side by side on a single identical imaging sensor as the imaging unit 211. However, in the example shown in Figure 7, for the sake of explanation, the distortion of the binocular optical system is corrected in step S503. The left eye image 600L shows images 601L to 604L of the four subjects 601 to 604 shown in Figure 6, but the right eye image 600R shows images 602R to 604R of 602 to 604. In other words, occlusion occurs, and the image of person 601 is not shown in the right eye image 600R.

[0067] In step S503, the image processing unit 214 performs image distortion correction. Since the lens unit 300 in this embodiment is a VR180 lens, the image captured in step S502 is a distorted image. Performing distortion correction improves the viewer's visibility, so executing the processing in this step is a more preferable configuration, but this step may be omitted.

[0068] In step S504, the subject detection unit 403 performs subject detection for each of the two images. For example, in this step, the four subjects shown in Figure 7 are detected.

[0069] In step S505, the main subject selection unit 404 selects a main subject for each of the two-eye images. Figure 8 shows the selection of a main subject from the detected subjects shown in Figure 7. In the figure, the subject selected as the main subject is indicated by a dashed rectangle surrounding it. For example, if the subject size is the size of the longer side of the circumscribing frame of the rectangle surrounding the detected subject, then a subject whose size is 20% or more of the field of view width of the right eye image 600R and the left eye image 600L can be selected as the main subject. In the example shown in Figure 8, the flower 604 does not meet the selection criteria and is not selected as the main subject.

[0070] In step S506, the image synthesis unit 405 performs occlusion determination of the subject. At this time, the image synthesis unit 405 calculates the parallax of each subject. For the person 601, where occlusion occurs, the image synthesis unit 405 cannot find the image region of the right eye corresponding to the image 601L of the person 601 in the left eye by calculating the correlation between the left and right images of the image synthesis unit 405. That is, for example, in the image region of the right eye corresponding to the image 601L of the person 601 in the left eye, a correlation value exceeding the threshold indicating a high correlation does not appear. In such a case, the image synthesis unit 405 cannot calculate the parallax for the image 601L of the person 601 in the left eye (i.e., it cannot estimate the subject distance) and determines that occlusion has occurred. For other subjects, there are image regions with high correlation (correlation value exceeding the threshold) in both eyes, and the amount of parallax can be calculated. The tree 602 has a short subject distance, and a large amount of parallax is calculated. On the other hand, the animal 603 is a distant subject, and a small amount of parallax is calculated. The image synthesis unit 405 can determine that there is no occlusion for subjects for which the amount of parallax can be calculated.

[0071] In step S507, the image synthesis unit 405 generates a base image that explicitly indicates occlusion according to the calculated parallax. Figure 9 shows an example of a base image 600C created by synthesizing images of two eyes at a set virtual viewpoint position. For the person 601, which was determined to have occlusion in S506, the image of the eye 601L, which contains an image, is displayed in the base image 600C. For the tree 602 and animal 603, which were determined not to have occlusion, the image synthesis unit 405 performs viewpoint transformation processing to the virtual viewpoint position using the amount of parallax calculated from each of the two eye images. The image synthesis unit 405 also generates an average image of the viewpoint-transformed images of both eyes (i.e., an image of the subject area with reduced parallax) by superimposing the left and right images after the image viewpoint transformation, and adds this composite image to the base image 600C.

[0072] In step S508, the occlusion information generation unit 406 generates an occlusion display image. For example, as shown in Figure 10, the occlusion information generation unit 406 can indicate an object that has been determined to have occlusion by highlighting its outline. The display control unit 407 displays the image in Figure 10 on the display unit 108. In this embodiment, the case in which the occlusion information generation unit 406 generates an occlusion display image indicating the occurrence of occlusion has been described as an example, but this embodiment is not limited to the generation of an image as long as it can indicate the occurrence of occlusion. That is, the occlusion information generation unit 406 only needs to be able to generate information in the base image that makes it visible that occlusion has occurred in the object. Furthermore, in the above embodiment, an example of making an object with occlusion visible has been described. However, the embodiment is not limited to this example, and information that makes it visible that occlusion has occurred in the object may be generated using, for example, text or symbols. For example, the occlusion information generation unit 406 may generate information that makes it visible that occlusion is occurring in a subject, for example, by using text or the like related to the subject in which occlusion has occurred (for example, by adding text or the like to the base image).

[0073] As described above, in this embodiment, for subjects with occlusion determined based on the correlation between the left and right images, an occlusion display image is generated that includes information that makes the subject visible to the photographer, and this image is displayed on the display unit 108. In this way, the photographer can easily and in real time see the status of occlusion occurring in the current shooting.

[0074] (Embodiment 2) Embodiment 2 describes a configuration example that further reduces the processing load of the image synthesis unit 405. This embodiment is more suitable when there are limitations on the processing performance of the system control unit 218 of the camera 100, or when high-speed processing is required, such as for shooting at a high frame rate.

[0075] The configuration of Embodiment 2 is the same as that of the camera 100, etc., described in Embodiment 1. Therefore, the same configurations and processes as in Embodiment 1 will be used and their descriptions will be omitted, with the differences being explained in detail.

[0076] The occlusion information generation process of this embodiment will now be described. In this embodiment as well, the occlusion information generation process is performed by each part of the camera 100 shown in Figure 4. That is, the series of operations of the occlusion information generation process can be realized by the system control unit 218 or the image processing unit 214 executing a computer program stored in, for example, the non-volatile memory 220.

[0077] Steps S501 to S506 are performed by the image synthesis unit 405 and the like, in the same way as in Embodiment 1.

[0078] In step S507, the image synthesis unit 405 compares the list of subjects selected as the main subject and generates a base image for the occlusion display image by performing a pass-through process on the image of the eye with a larger number of main subjects. This is because the image of the eye with a larger number of main subjects is considered to have a higher probability of showing subjects where occlusion is occurring compared to the image of the other eye. For example, as shown in Figure 11, the image synthesis unit 405 performs a pass-through process to make 600L of the left eye, which has a larger number of detected subjects, the base image 600C for the occlusion display image.

[0079] In step S508, similar to Embodiment 1, the occlusion information generation unit 406 generates an occlusion display image, and then the display control unit 407 displays the image on the display unit 108. Figure 12 shows an example of an image in which occlusion information is displayed by highlighting the contours of subjects that have been determined to have occlusion in this embodiment. If no subjects with occlusion are visible in the pass-through processed image, the occlusion information generation unit 406 obtains the subject's position from the list of main subjects recorded by the main subject selection unit 404. Then, the subject with occlusion can be clearly indicated by adding a frame or the like to the approximate position of the subject on the image.

[0080] In this way, the image synthesis unit 405 performs a pass-through process on the image with a larger number of detected subjects, using it as the base image for the occlusion display image. This reduces the processing load for generating the occlusion display image and allows the occlusion display image to be displayed on the display unit 108 at high speed. That is, by displaying the occlusion display image shown in Figure 12 on the display unit 108 using the display control unit 407, the photographer can easily and in real time see the current occlusion situation during shooting. (Embodiment 3) In this embodiment, the display mode of the occlusion display image differs from that of the embodiments described above. For example, the occlusion display image generated in this embodiment differs from the occlusion display mode shown in Figure 10 of Embodiment 1 and Figure 12 of Embodiment 2. Therefore, although the processing in the occlusion information generation unit 406 according to this embodiment differs from that of the embodiments described above, other configurations and processing contents are the same as those of the embodiments described above. For this reason, the same reference numerals are used for configurations that are the same as those of the embodiments described above, and their descriptions are omitted, with the differences being explained in detail.

[0081] The occlusion information generation process of this embodiment will now be described. In this embodiment as well, the occlusion information generation process is performed by each part of the camera 100 shown in Figure 4. That is, the series of operations of the occlusion information generation process can be realized by the system control unit 218 or the image processing unit 214 executing a computer program stored in, for example, the non-volatile memory 220.

[0082] Steps S501 to S507 are performed by the image synthesis unit 405 and the like, in the same way as in the embodiment described above. Then, in step S508, the image synthesis unit 405 generates and displays an occlusion display image from the base image 600C of the generated occlusion display image.

[0083] Furthermore, in order to generate and display images for the photographer in real time during shooting, it is preferable to be able to select or change the occlusion display mode that is suitable depending on the brightness of the shooting location and the type of subject being photographed.

[0084] Figure 13 shows an example of an occlusion display image according to this embodiment. For example, the occlusion information generation unit 406 displays occlusion for a person 601 where occlusion is occurring by displaying a rectangular frame surrounding the subject in a "circumscribing frame display". This display method is suitable when the size of the subject in the image is relatively small or when the shape of the subject is complex, and even in these cases, it makes it easier for the photographer to see the subject with occlusion.

[0085] Figure 14 shows another example of an occlusion display image according to this embodiment. For example, the occlusion information generation unit 406 displays occlusion for a person 601 where occlusion is occurring by "color / brightness / transmittance change display". That is, the occlusion information generation unit 406 generates an occlusion display image by changing the color, brightness, or transmittance of the area of ​​the person 601 (specific subject) in the captured original image. This display mode is suitable for shooting indoors or in relatively dark environments where the color and brightness of the live view screen of the display unit 108 can be clearly seen, and even in these shooting environments, it is possible to make it easier for the photographer to see subjects with occlusion. Similar to the "contour enhancement display" shown in Embodiment 1, this display mode allows the photographer to intuitively recognize the type of object, so the photographer can easily see which subject is experiencing occlusion.

[0086] The occlusion information generation unit 406 may combine multiple of the above-described occlusion display forms. Furthermore, the display forms of the occlusion display image according to the present invention are representative display forms and are not limited to the display forms shown in the above-described embodiments; other similar display forms may also be used.

[0087] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0088] (Disclosure of this Specified Use) The disclosures herein include the following image processing apparatus, control method for the image processing apparatus, imaging apparatus, and program. (Item 1) Image acquisition means for acquiring multiple images that include a common subject and have parallax, A determination means for determining a specific subject in which occlusion occurs among the aforementioned multiple images, An image processing apparatus characterized by having a generation means for generating information that makes it visible that the occlusion is occurring on a specific subject in a display image based on at least one of the plurality of images. (Item 2) The system further includes an image generation means for generating the display image based on at least one of the plurality of images, The image processing apparatus according to item 1, characterized in that the image generation means generates an image captured from a virtual viewpoint different from the viewpoints used when each of the plurality of images is captured, and uses this as the image for display. (Item 3) The image processing apparatus according to item 2, characterized in that the image generation means performs a viewpoint transformation on each of the plurality of images to become an image of the virtual viewpoint, and generates the display image using each of the images transformed to become the virtual viewpoint. (Item 4) The image processing apparatus according to item 3, further comprising the image generation means estimating the subject distance to a subject included in the image, and adding an image of the subject region in which the parallax to the subject in the plurality of images has been reduced according to the subject distance to the display image. (Item 5) The system further includes an image generation means for generating the display image based on at least one of the plurality of images, The image processing apparatus according to item 1, characterized in that the image generation means uses the image with the most subjects among the plurality of images as the image for display. (Item 6) The image processing apparatus according to any one of items 1 to 5, characterized in that the determination means determines the specific subject in which occlusion is occurring based on whether or not it is possible to estimate the subject distance of the subject detected in the first image among the plurality of images. (Item 7) The image processing apparatus according to any one of items 1 to 6, characterized in that the determination means determines the specific subject in which occlusion occurs from among the subjects detected in each of the plurality of images that satisfy predetermined conditions corresponding to being the main subject. (Item 8) The image processing apparatus according to item 7, characterized in that a subject that satisfies the predetermined conditions is a subject that satisfies at least one of the set conditions of the proportion of the subject that occupies the field of view and the type of subject. (Item 9) The image processing apparatus according to any one of items 1 to 8, characterized in that the information that makes it possible to visually confirm that the occlusion is occurring in the specific subject includes information added to the display image, which includes at least one of contour enhancement of the region of the specific subject and a frame that circumsects the region of the specific subject. (Item 10) The image processing apparatus according to any one of items 1 to 8, characterized in that the information that makes it possible to visually confirm that the occlusion is occurring in the specific subject includes information obtained by changing the color of the region of the specific subject, the brightness of the region of the specific subject, or the transmittance of the region of the specific subject on the captured image. (Item 11) The image processing apparatus according to any one of items 1 to 10, further comprising a display control means for displaying on a display means the information generated by the generation means that makes it visible that the occlusion is occurring on the specific subject. (Item 12) An imaging means for capturing an image, The image processing apparatus described in any one of items 1 to 11, and includes, The image acquisition means is characterized by acquiring the plurality of images from a single image signal output from the imaging means. (Item 13) The imaging apparatus according to item 12, characterized in that each of the plurality of images corresponds to an image obtained through a different optical system among the plurality of optical systems. (Item 14) An image acquisition process that acquires multiple images that include a common subject and have parallax, A determination step to determine a specific subject in which occlusion occurs among the aforementioned multiple images, A control method for an image processing apparatus, characterized by comprising: a generation step of generating information that makes it visible that the occlusion is occurring in a specific subject in a display image based on at least one of the plurality of images. (Item 15) A program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 11.

[0089] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0090] 100 Camera, 218 System Control Unit, 401 First Image Acquisition Unit, 402 Second Image Acquisition Unit, 405 Image Synthesis Unit, 406 Occlusion Information Generation Unit, 407 Display Control Unit

Claims

1. Image acquisition means for acquiring multiple images that include a common subject and have parallax, A determination means for determining a specific subject in which occlusion occurs among the aforementioned multiple images, An image processing apparatus characterized by having a generation means for generating information that makes it visible that the occlusion is occurring on a specific subject in a display image based on at least one of the plurality of images.

2. The system further includes an image generation means for generating the display image based on at least one of the plurality of images, The image processing apparatus according to claim 1, characterized in that the image generation means generates an image captured from a virtual viewpoint different from the viewpoints used when each of the plurality of images is captured, and uses this as the display image.

3. The image processing apparatus according to claim 2, characterized in that the image generation means performs a viewpoint transformation on each of the plurality of images to become an image of the virtual viewpoint, and generates the display image using each of the images transformed to become the virtual viewpoint.

4. The image processing apparatus according to claim 3, further comprising the image generation means estimating the subject distance to a subject included in the image, and adding to the display image an image of a subject region in which the parallax to the subject in the plurality of images has been reduced according to the subject distance.

5. The system further includes an image generation means for generating the display image based on at least one of the plurality of images, The image processing apparatus according to claim 1, characterized in that the image generation means uses the image with the most subjects among the plurality of images as the image for display.

6. The image processing apparatus according to claim 1, characterized in that the determination means determines the specific subject in which occlusion is occurring based on whether or not it is possible to estimate the subject distance of the subject detected in the first image among the plurality of images.

7. The image processing apparatus according to claim 1, characterized in that the determination means determines a specific subject in which occlusion occurs from among subjects that satisfy predetermined conditions corresponding to being a main subject, among the subjects detected in each of the plurality of images.

8. The image processing apparatus according to claim 7, characterized in that the subject that satisfies the predetermined conditions is a subject that satisfies at least one of the set conditions of the proportion of the subject that occupies the angle of view and the type of subject.

9. The image processing apparatus according to claim 1, wherein the information that makes it possible to visually confirm that the occlusion is occurring in the specific subject includes information added to the display image, which includes at least one of contour enhancement of the region of the specific subject and a frame that circumsects the region of the specific subject.

10. The image processing apparatus according to claim 1, wherein the information that makes it possible to visually confirm that the occlusion is occurring in the specific subject includes information obtained by changing any of the color of the region of the specific subject, the brightness of the region of the specific subject, or the transmittance of the region of the specific subject in the captured image.

11. The image processing apparatus according to claim 1, further comprising a display control means for displaying on a display means the information generated by the generation means that makes it visible that the occlusion is occurring on the specific subject.

12. An imaging means for capturing an image, The image processing apparatus according to any one of claims 1 to 11, The image acquisition means is characterized by acquiring the plurality of images from a single image signal output from the imaging means.

13. The imaging apparatus according to claim 12, characterized in that each of the plurality of images corresponds to an image obtained through a different optical system among the plurality of optical systems.

14. An image acquisition process that acquires multiple images that include a common subject and have parallax, A determination step to determine a specific subject in which occlusion occurs among the aforementioned multiple images, A control method for an image processing apparatus, characterized by comprising: a generation step of generating information that makes it visible that the occlusion is occurring in a specific subject in a display image based on at least one of the plurality of images.

15. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 11.

Images