Information processing device, imaging device, information processing method, program

Abstract

To provide a technique capable of obtaining a state where foreign matter can be removed from a captured image with high accuracy, even after the replacement of an area in the captured image is performed.SOLUTION: An information processing apparatus includes image acquisition means for acquiring the captured image, lens information acquisition means for acquiring lens information relating to a first optical system and a second optical system when the captured image includes a first area corresponding to a first optical image inputted through the first optical system and a second area corresponding to a second optical image which is an optical image having a predetermined parallax with respect to the first optical image and is inputted through the second optical system, foreign matter information acquisition means for acquiring foreign matter information indicating one or a plurality of foreign matter areas in the captured image, correction means for correcting the position of the first area and the position of the second area on the basis of the lens information, and adjustment means for adjusting the foreign matter information on the basis of the lens information.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an imaging apparatus, an information processing method, and a program.

Background Art

[0002] There is known a camera that can acquire an image including two regions having parallax with respect to each other by mounting a lens unit having two optical systems. In an image (one captured image including two regions having parallax with respect to each other) in which a subject is imaged by mounting a lens unit having two optical systems, the positional relationship between the two regions and the positional relationship between the two optical systems are reversed. Therefore, when generating VR content or the like, the left and right regions of the captured image are swapped so that the positional relationship between the two optical systems and the positional relationship between the two regions are the same.

[0003] Also, in a lens interchangeable camera or the like, foreign matter (dust) may adhere to an imaging element or the like, and the quality of the captured image may deteriorate. Patent Documents 1 and 2 disclose a processing apparatus that adds foreign matter information at the time of shooting to a captured image and corrects the captured image according to the foreign matter information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the foreign object information includes the coordinates of the foreign object, swapping the left and right regions will cause a discrepancy between the coordinates of the foreign object indicated by the foreign object information and the actual coordinates of the foreign object. This leads to a problem where the foreign object removal process cannot be executed correctly. To avoid this problem, it is conceivable to perform the foreign object removal process before swapping the left and right regions, but depending on the constraints of the processing order and the processing procedure set by the user, it may not be possible to perform the foreign object removal process first.

[0006] Therefore, the present invention aims to provide a technology that enables the removal of foreign matter from an captured image with high precision, even after regions in the captured image have been rearranged. [Means for solving the problem]

[0007] One aspect of the present invention is, The optical system includes a first region corresponding to a first optical image input via a first optical system, and a second region corresponding to a second optical image input via a second optical system, which has a predetermined parallax with respect to the first optical image. Image acquisition means for acquiring captured images and 、 before Lens information acquisition means for acquiring lens information relating to the first optical system and the second optical system, Foreign object information indicating the region of one or more foreign objects in the aforementioned captured image The foreign object information includes region data for each foreign object in the captured image, which includes information on the center coordinates and radius of the circular region indicating the area of ​​the foreign object. A means for acquiring foreign object information, Based on the aforementioned lens information, In the aforementioned captured image The position of the first region and the position of the second region Swap the two Correction means, Based on the aforementioned lens information, the aforementioned foreign matter information The information of the aforementioned center coordinates An adjustment means for adjusting, It has, If the adjustment means has a region of a first foreign object that spans both the first region and the second region in the image before correction, it adds new region data to the foreign object information that corresponds to a region in the first region and the second region that does not correspond to the center coordinates of the circular region indicating the region of the first foreign object. This is an information processing device characterized by the following features.

[0008] One aspect of the present invention is, The optical system includes a first region corresponding to a first optical image input via a first optical system, and a second region corresponding to a second optical image input via a second optical system, which has a predetermined parallax with respect to the first optical image. Image acquisition step to acquire captured image and 、 before A lens information acquisition step for acquiring lens information relating to the first optical system and the second optical system, Foreign object information indicating the area of one or more foreign objects in the captured image The foreign object information includes region data for each foreign object in the captured image, which includes information on the center coordinates and radius of the circular region indicating the area of ​​the foreign object. A foreign object information acquisition step for acquiring Based on the lens information, In the aforementioned captured image The position of the first area and the position of the second area Swap the two A correction step; Based on the lens information, the foreign object information The information of the aforementioned center coordinates An adjustment step for adjusting; It has, In the adjustment step, if there is a region of a first foreign object that spans both the first region and the second region in the captured image before correction, new region data corresponding to the region of the first region and the second region that does not correspond to the center coordinates of the circular region representing the region of the first foreign object is added to the foreign object information. It is an information processing method characterized by the above.

Effect of the Invention

[0009] According to the present invention, even after the areas in the captured image are swapped, it is possible to bring the captured image into a state where foreign objects can be removed with high precision.

Brief Description of the Drawings

[0010] [Figure 1] It is a diagram showing the system. [Figure 2] It is an external view of the camera. [Figure 3] It is a configuration diagram of the camera. [Figure 4] It is a diagram showing the configuration of the lens unit. [Figure 5] It is a hardware configuration diagram of the PC. [Figure 6] It is a software configuration diagram of the PC. [Figure 7] It is a diagram for explaining the image file. [Figure 8] It is a diagram for explaining the lens information. [Figure 9] It is a diagram for explaining the foreign object information. [Figure 10] It is a diagram for explaining the correction of the positions of two images. [Figure 11] It is a diagram for explaining the exposure area. [Figure 12] It is a flowchart of the foreign object removal process. [Figure 13] It is a flowchart of the adjustment process. [Figure 14] This diagram illustrates the addition of foreign object region data. [Modes for carrying out the invention]

[0011] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The embodiments described below are merely examples of means for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. Furthermore, the embodiments can be combined as appropriate.

[0012] <Embodiment> Figures 1A and 1B are schematic diagrams showing an example of the overall configuration of the system according to this embodiment. The system according to this embodiment includes a digital camera (camera) 100 and a personal computer (PC) 500. A lens unit 300 is attached (connected) to the camera 100. Details of the lens unit 300 will be described later, but by attaching the lens unit 300, the camera 100 can capture two images (still images or moving images) with a predetermined parallax at the same time. The PC 500 is an information processing device (electronic device) that handles images captured by imaging devices such as the camera 100.

[0013] Figure 1A shows a configuration in which the camera 100 and PC 500 are connected to each other via wireless or wired connections. Figure 1B shows a configuration in which images captured by the camera 100 are input to the PC 500 in a file-based manner via an external storage device. The external storage device is connected to the camera 100 and PC 5 Both 00s may or may not be connected. For example, an external storage device may be connected to the camera 100, and the image files taken by the camera 100 may be stored in the external storage device. After that, the external storage device may be removed from the camera 100 and connected to the PC 500, and the PC 500 may retrieve the files stored in the external storage device.

[0014] Figures 2A and 2B are external views showing an example of the camera 100's appearance. Figure 2A is a perspective view of the camera 100 from the front, and Figure 2B is a perspective view of the camera 100 from the rear.

[0015] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface.

[0016] The shutter button 101 is an operating component for issuing a shooting preparation or shooting command. The power switch 102 is an operating component for switching the power of the camera 100 on and off. The mode switch 103 is an operating component for switching between various modes. The main electronic dial 104 is a rotary operating component for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operating component for moving the selection frame (cursor) and advancing images. The video button 106 is an operating component for issuing a command to start and stop video recording. The viewfinder external display 107 displays various settings such as shutter speed and aperture.

[0017] The camera 100 has a display unit 108, a touch panel 109, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back.

[0018] The display unit 108 displays images and various information. The touch panel 109 is an operating member that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 is an operating unit consisting of keys (four directional keys) that can be pressed up, down, left, and right. The camera 100 can perform processing according to the pressed position (position of the directional keys 110).

[0019] The SET button 111 is an operating element that is mainly pressed when confirming a selection item. The AE lock button 112 is an operating element that is pressed when fixing the exposure state in shooting standby mode. The zoom button 113 is an operating element that switches the zoom mode on and off in the live view display (LV display) of the shooting mode. When the zoom mode is on, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 104. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio.

[0020] The playback button 114 is an operating component for switching between shooting mode and playback mode. By pressing the playback button 114 in shooting mode, the camera switches to playback mode, and the most recent image recorded on the recording medium 227 (described later) can be displayed on the display unit 108.

[0021] The menu button 115 is an operating element that is pressed to display a menu screen on the display unit 108, which allows for various settings. The user can intuitively make various settings using the menu screen displayed on the display unit 108 and the directional keys 110 and SET button 111. The eyepiece section 116 is the part that the user looks through when looking into the eyepiece viewfinder (peep-in type viewfinder) 117. The user views the image displayed on the EVF 217 (Electronic View Finder), which will be described later, inside the camera 100, through the eyepiece section 116. This is possible. The eyepiece detection unit 118 is a sensor that detects whether or not the user is looking into the eyepiece unit 116 (eyepiece finder 117).

[0022] The touch bar 119 is a line-shaped touch operation component (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched by the right thumb when the grip portion 120 is held with the right hand (with the little finger, ring finger, and middle finger) so that the shutter button 101 can be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece 116 with their eyepiece in the eyepiece viewfinder 117 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (an operation where the user touches the touch bar and releases it within a predetermined period without moving the touch position), and left and right sliding operations (an operation where the user touches the touch bar and then moves the touch position while keeping the touch position). The touch bar 119 is a different operation component from the touch panel 109 and does not have a display function. The Touch Bar 119 functions, for example, as a multifunction bar (M-Fn bar) to which various functions can be assigned.

[0023] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, and a communication terminal 124. The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. With the camera 100 held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand. The thumb rest section 121 (thumb waiting position) is a grip section located on the back of the camera 100, in a place where the thumb of the right hand holding the grip section 120 can be easily rested when no operating members are being operated. The thumb rest section 121 is made of a rubber material or the like to enhance the holding force (grip feel). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external equipment (external devices). The cover 123 protects the recording medium 227 and the slot for storing the recording medium 227, which will be described later, by closing the slot. The communication terminal 124 is a terminal for communicating with a lens unit (such as the lens unit 200 or lens unit 300, which will be described later) that can be attached to or detached from the camera 100.

[0024] Figure 3 is a block diagram showing an example of the configuration of camera 100. Components identical to those in Figures 2A and 2B are denoted by the same reference numerals, and their descriptions are omitted as appropriate. In Figure 3, a lens unit 200 is attached to camera 100. First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that can be attached to camera 100. The lens unit 200 is a single-lens reflex lens and is an example of a typical lens. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0025] The aperture 201 is configured to have an adjustable aperture diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc., based on instructions from the system control unit 50, which will be described later. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203 and focuses by changing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication takes place via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100.

[0026] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0027] The shutter 210 is a focal-plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an image sensor composed of a CCD or CMOS element that converts an optical image into an electrical signal. The imaging unit 211 may have an image plane phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing, and color conversion) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This processing enables TTL (through-the-lens) AF processing, AE (automatic exposure) processing, EF (flash pre-flash) processing, etc. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data, and the system control unit 50 performs TTL-type AWB (auto white balance) processing based on the obtained calculation results.

[0028] Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data for display on the display unit 108 and EVF 217. The memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. The memory 215 also serves as a memory for image display (video memory).

[0029] The D / A converter 216 converts the display image data stored in the memory 215 into an analog signal and supplies it to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and EVF 217 are displays such as LCDs and OLEDs. The digital signal, which has been A / D converted by the A / D converter 212 and stored in the memory 215, is converted into an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and EVF 217 for display, thereby performing live view display.

[0030] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0031] The camera 100 also includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0032] For example, RAM is used as the system memory 218. The system control unit 50's operating constants, variables, and programs read from the non-volatile memory 219 are then displayed. The non-volatile memory 219 is an electrically erasable and recordable memory, and an EEPROM, for example, is used as the non-volatile memory 219. The non-volatile memory 219 stores the operating constants and programs of the system control unit 50. The program here refers to a program for executing the flowchart described later.

[0033] The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from external devices connected by wireless or wired cables.

[0034] The communication unit 221 can connect to a wireless LAN (Local Area Network) and the internet. Furthermore, the communication unit 221 can communicate with external devices using Bluetooth® and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive images and other various information from external devices.

[0035] The attitude detection unit 222 detects the attitude of the camera 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image according to the detected attitude. For example, an acceleration sensor or a gyroscope sensor can be used for the attitude detection unit 222. It is also possible to use the attitude detection unit 222 to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.).

[0036] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece unit 116 (eyepiece finder 117). For example, an infrared proximity sensor can be used in the eyepiece detection unit 118. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece unit 116 to the object can be determined by the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece unit 116.

[0037] The eyepiece detection unit 118 is an eyepiece detection sensor that detects when an eye (object) approaches (eyepiece) and moves away (eye-away) from the eyepiece unit 116. When an object is detected approaching the eyepiece unit 116 within a predetermined distance from the non-eyepiece state (non-approach state), it is detected that the eye has been placed on the eyepiece. On the other hand, when the object that was detected approaching moves away from the eyepiece state (approach state) beyond a predetermined distance, it is detected that the eye has been moved away. The threshold for detecting eyepiece placement and the threshold for detecting eye-away may be different, for example, by providing hysteresis. After eyepiece placement is detected, the eyepiece remains in the eyepiece state until eye-away is detected. After eye-away is detected, the eyepiece remains in the non-eyepiece state until eyepiece placement is detected again. The system control unit 50 switches the display (display state) / non-display (non-display state) of the display unit 108 and EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, when the camera is in at least the shooting standby state and the display destination switching setting is set to automatic switching, the display destination is set to the display unit 108 and the display is turned on, and the EVF 217 is hidden when the eyepiece is not being used. When the eyepiece is being used, the display destination is set to the EVF 217 and the display is turned on, and the display unit 108 is hidden. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as eyepiece use may be used for the eyepiece detection unit 118.

[0038] The camera 100 also includes a viewfinder external display unit 107, a viewfinder external display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, and the like. .

[0039] The external viewfinder display unit 107 is driven by the external viewfinder display drive circuit 223 and displays various settings of the camera 100, such as shutter speed and aperture. The power control unit 224 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are powered, etc. The power control unit 224 detects whether a battery is installed, the type of battery, the remaining battery level, etc. Also, the power control unit 224 controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage to each part, including the recording medium 227, for the required period of time. The power supply unit 225 is a primary battery (alkaline battery or lithium battery, etc.), a secondary battery (NiCd battery, NiMH battery or Li battery, etc.), or an AC adapter, etc. The recording medium I / F 226 is an interface with the recording medium 227, such as a memory card or hard disk. The recording medium 227 is a memory card for recording captured images, and consists of semiconductor memory or magnetic disk, etc. The recording medium 227 may be detachable from the camera 100, or it may be built into the camera 100.

[0040] The operation unit 228 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operation units 229. The other operation units 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, and a touch bar 119.

[0041] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 turns on during the operation of the shutter button 101, so-called half-press (shooting preparation instruction), and outputs a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the output of the first shutter switch signal SW1. The second shutter switch 231 turns on when the operation of the shutter button 101 is completed, so-called full-press (shooting instruction), and outputs a second shutter switch signal SW2. The system control unit 50 starts a series of shooting processes (from reading the signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227) in response to the output of the second shutter switch signal SW2.

[0042] The mode switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide shooting settings for different shooting scenes. The user can directly switch to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0043] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integral unit. For example, the touch panel 109 is mounted on the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109; either method is acceptable.

[0044] The system control unit 50 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). The finger or pen that was touching the touch panel 109 is lifted (released), meaning the touch action ends (hereinafter referred to as "Touch-Up"). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0045] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0046] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation was performed. An operation in which a finger is touched on the touch panel 109 and then quickly moved a certain distance and then released is called a flick. In other words, a flick is an operation in which the finger is quickly swiped across the touch panel 109. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected, and a touch-up is then detected, it is determined that a flick was performed (it can be determined that a flick followed a slide operation). Furthermore, touching multiple points (for example, two points) simultaneously (multitouch) to bring them closer together is called pinch-in, and touching them further apart is called pinch-out. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch).

[0047] Figure 4 is a schematic diagram showing an example of the configuration of the lens unit 300. Figure 4 shows the lens unit 300 attached to the camera 100. Note that, among the components of the camera 100 shown in Figure 4, those identical to those described in Figure 3 are denoted by the same reference numerals as in Figure 3, and their descriptions are omitted as appropriate.

[0048] The lens unit 300 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 300 is a twin-lens system capable of capturing right and left images with a predetermined parallax between them. In this embodiment, the lens unit 300 has two optical systems, and each of the two optical systems can capture an image within a wide field of view of approximately 180 degrees. Specifically, each of the two optical systems of the lens unit 300 can capture a subject within a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle). In other words, each of the two optical systems can capture an image within the front hemisphere.

[0049] The lens unit 300 includes a right-eye optical system 301R having multiple lenses and a reflective mirror, a left-eye optical system 301L having multiple lenses and a reflective mirror, and a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. The right-eye optical system 301R has a lens 302R positioned on the subject side, and the left-eye optical system 301L has a lens 302L positioned on the subject side. Lenses 302R and 302L face the same direction, and their optical axes are approximately parallel.

[0050] The lens unit 300 is a binocular lens (VR180 lens) for obtaining VR180 images, which are one of the VR (Virtual Reality) image formats that enable binocular stereoscopic viewing. In this embodiment, the lens unit 300 has a fisheye lens in each of the right eye optical system 301R and the left eye optical system 301L that can capture a range of approximately 180 degrees. The range that can be captured by the lenses of each of the right eye optical system 301R and the left eye optical system 301L may be narrower than 180 degrees, around 160 degrees. The lens unit 300 can image the right image (first image) formed via the right eye optical system 301R and the left image (second image) formed via the left eye optical system 301L onto one or two image sensors of a camera to which the lens unit 300 is attached.

[0051] The lens unit 300 is attached to the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. In this way, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0052] In this embodiment, the right image formed via the right-eye optical system 301R and the left image formed via the left-eye optical system 301L are simultaneously (as a set) imaged on the imaging unit 211 of the camera 100. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single image sensor. The imaging unit 211 converts the imaged subject (optical signal) into an analog electrical signal. By using the lens unit 300 in this way, two images with parallax can be simultaneously (as a set) acquired from two locations (optical systems) - the right-eye optical system 301R and the left-eye optical system 301L. By separating the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a three-dimensional VR image with a range of approximately 180 degrees. In other words, the user can view a VR180 image in 3D.

[0053] Here, a VR image is an image that can be displayed in VR, as described later. VR images include omnidirectional images (spherical images) captured by omnidirectional cameras (spherical cameras), and panoramic images with a wider field of view (effective field of view) than the display area that can be shown on the display unit at once. Furthermore, VR images are not limited to still images, but also include videos and live images (images acquired from the camera in near real-time). A VR image has a field of view (effective field of view) of up to 360 degrees horizontally and vertically. In addition, even if a VR image has a field of view of less than 360 degrees horizontally or vertically, it may still have a wider field of view than that of a normal camera. This also includes images with a wide field of view, or an image range wider than the display area that can be displayed on the display unit at once. Images captured by the camera 100 using the lens unit 300 described above are a type of VR image. VR images can be displayed in VR, for example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR view". By displaying a VR image with a 360-degree field of view in VR, the user can change the orientation of the display device left or right (horizontal rotation direction) to view seamless, omnidirectional images in the left and right directions.

[0054] VR display (VR view) is a display method (display mode) that allows the display range to be changed, displaying images within the field of view range according to the orientation of the display device, from among VR images. One type of VR display is "single-eye VR display (single-eye VR view)," which displays a single image by performing deformation (distortion correction) that maps the VR image to a virtual sphere. Another type of VR display is "two-eye VR display (two-eye VR view)," which displays the VR image for the left eye and the VR image for the right eye side by side by performing deformation that maps each to a virtual sphere. By performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax with each other, it is possible to view these VR images in 3D.

[0055] Regardless of the VR display method, when a user wears a display device such as an HMD (Head-Mounted Display), the image displayed will correspond to the user's face orientation and field of view. For example, suppose a VR image is currently displaying an image with a field of view centered at 0 degrees horizontally (a specific direction, e.g., north) and 90 degrees vertically (90 degrees from the zenith, i.e., horizontal) at a given point in time. If the orientation of the display device is reversed (for example, changing the display surface from facing south to facing north), the display range of the same VR image will change to an image with a field of view centered at 180 degrees horizontally (the opposite direction, e.g., south) and 90 degrees vertically. In other words, when a user wearing an HMD turns their face from north to south (i.e., turns their back), the image displayed on the HMD will also change from a north image to a south image. Note that the VR image captured using the lens unit 300 of this embodiment is an image capturing a range of approximately 180 degrees in front (180° image), and there is no image of a range of approximately 180 degrees behind. When displaying such an image in VR, if the orientation of the display device is changed to the side where there is no image, a blank area will be displayed.

[0056] By displaying VR images in VR in this way, users can visually experience the sensation (immersion) of being inside the VR image (in VR space). Note that the method of displaying VR images is not limited to changing the orientation of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel or directional buttons. Furthermore, during VR display (in "VR View" display mode), in addition to changing the display range due to changes in orientation, the display range may also be changed in response to touch movements on the touch panel, drag operations with a mouse, or pressing directional buttons. Note that a smartphone mounted on VR goggles (head-mounted adapter) is a type of HMD (Head-Mounted Display).

[0057] Figure 5 is a block diagram showing an example of the configuration of the PC500. The control unit 501 is, for example, a Central Processing Unit (CPU) and controls the entire PC500. The Read Only Memory (ROM) 502 stores programs and parameters non-temporarily. The Random Access Memory (RAM) 503 temporarily stores programs and data supplied from external devices. The recording medium 504 is a hard disk or flash memory fixed to the PC500, or an optical disk, magnetic card, optical card, IC card, memory card, etc. that can be attached to and removed from the PC500. The image files captured by the camera 100 are read from the recording medium 504.

[0058] The operation unit 505 receives user input for the PC 500. The operating device used by the user may be a button or touch panel provided on the PC 500, or a keyboard or mouse that can be attached to or detached from the PC 500. The display unit 506 displays data held by the PC 500 or data supplied from an external source. The display unit 506 may be part of the PC 500, or it may be a separate display device. The communication unit 507 communicates with external devices such as the camera 100. The system bus 508 connects the components of the PC 500 in a way that allows communication between them.

[0059] Here, we will explain the characteristics of the images captured with the lens unit 300 (two-lens system). In the case of the lens unit 200 (a normal single-lens system), an image that is inverted vertically and horizontally (a 180-degree rotated image) relative to the actual view is formed on the imaging unit 211. Therefore, the entire formed image is rotated 180 degrees to acquire (image) an image that matches the actual view. On the other hand, in the case of the lens unit 300 (two-lens system), the right image and the left image are each rotated 180 degrees relative to the actual view and formed on the imaging unit 211. The arrangement of the right and left images is not particularly limited, but in this embodiment, it is assumed that the right image is formed on the right side and the left image is formed on the left side on the imaging unit 211. Then, as with the lens unit 200 (a normal single-lens system), if the entire formed image (the image including the right and left images) is rotated 180 degrees, the right and left images can be adjusted to match the actual view, but the positions of the right and left images are swapped. In other words, the left-right positional relationship is reversed, resulting in an image where the right image is positioned on the left and the left image is positioned on the right.

[0060] Figure 6 shows the software configuration (logical configuration) of the PC500. Each configuration shown in Figure 6 is realized by the control unit 501 loading predetermined programs stored in the ROM 502 into the RAM 503 and executing those predetermined programs. Specifically, the control unit 501 operates as each configuration according to the predetermined program. In other words, the control unit 501 operates as the image acquisition unit 600, the lens information acquisition unit 601, the foreign object information acquisition unit 602, the position correction unit 603, the adjustment unit 604, the foreign object removal unit 605, and the exposure area calculation unit 606.

[0061] (Regarding the image acquisition unit 600) The image acquisition unit 600 reads the image file stored in the recording medium 504. Figure 7 is a schematic diagram showing an example of the structure of an image file. The image file shown in Figure 7 has a header 701, a camera information area 702, a lens information area 703, an other information area 704, and an image data area 705.

[0062] Header 701 records information such as the type of image (captured image) of the subject (for example, whether it is a video or a still image).

[0063] The camera information area 702 records information about the camera used for imaging (shooting) as metadata for the captured image. For example, the camera information area 702 records imaging information (such as shutter speed or aperture at the time of imaging) and camera orientation information at the time of imaging.

[0064] The lens information area 703 records information about the lens used for imaging (photography) as metadata for the captured image. For example, the design values ​​and individual values ​​(deviations from the design values) of the twin lenses used for imaging are recorded in the lens information area 703.

[0065] Other information area 704 records other information as metadata of the captured image. Other information area 704 records, for example, foreign object information indicating the coordinates of foreign objects attached to the image sensor (image sensor; lens). The control unit 501 or system control unit 50 can, for example, determine the coordinates of foreign objects based on an image of a white piece of paper previously captured by the imaging unit 211 (i.e., determine the coordinates of areas other than the white area as the coordinates of foreign objects). If the captured image is a video, the information area 704 records information that changes with each frame. If the captured image is a RAW image, the information area 704 also records data necessary for development.

[0066] The image data area 705 records the image data of the captured image. If a video is recorded as the captured image in the image data area 705, not only the image data but also the audio data is recorded in the image data area 705.

[0067] The above example shows how camera information, twin-lens information, and other information are recorded within a single image file. However, this information may also be recorded in a separate file (a file associated with the image file).

[0068] (Regarding the lens information acquisition unit 601) The lens information acquisition unit 601 acquires lens information from the lens information area in the image file. Figure 8A is a schematic diagram showing an example of lens information. The lens information includes lens design values, individual lens values, lens flags, lens focal length, and lens temperature.

[0069] Lens design values ​​are designed to correct for aberrations. During the manufacturing process of a binocular lens, errors (aberrations) such as lens eccentricity and tilt occur in each of the two optical systems (left eye optical system 301L and right eye optical system 301R). If the left and right regions are swapped or equirectangular conversion is performed without considering these errors, the quality (precision) of the binocular VR display will deteriorate. In this case, it will be difficult for the user to experience good stereoscopic vision.

[0070] Lens individual values ​​are the measurement results of deviations (errors) from the design values ​​detected during the manufacturing process of the binocular lens. Details of lens design values ​​and lens individual values ​​will be described later using Figure 8B.

[0071] The lens flag indicates whether or not a twin-lens system was used for imaging. The lens flag can be used to determine whether or not a twin-lens system was used when capturing an image.

[0072] The lens focal length is the distance between the "principal point," which is the center of the lens, and the image sensor (image formation position). The lens focal length may be a common parameter for the two optical systems of a twin-lens system (left eye optical system 301L and right eye optical system 301R), or it may not be. High-precision lens focal lengths are necessary because high-precision operations such as left-right reversal and equirectangular transformation are required to achieve high-quality twin-lens VR display.

[0073] Lens temperature refers to the temperature of the two lenses and is used to understand the ambient temperature during imaging (shooting).

[0074] Figure 8B is a schematic diagram showing the details of the lens design values ​​and individual lens values. In this embodiment, the lens design values ​​and individual lens values ​​are used for swapping the left and right image regions in the captured image, adjusting for foreign object information, etc.

[0075] Lens design values ​​include image circle position, image circle diameter, angle of view, and distortion correction coefficient.

[0076] The image circle position is the coordinate of the optical axis center of the optical system in the captured image, and is provided for each of the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R). In other words, the image circle position is the coordinate of the image circle (circumferential fisheye image) formed on the image sensor. The image circle coordinates are the center coordinates of the image, and are prepared for both the right and left images. The origin of the coordinates is, for example, the center of the image sensor (center of the captured image). The image circle position includes horizontal and vertical coordinates. Various information regarding the optical axis center of the optical system in the captured image can be used as the image circle position. For example, the distance from a predetermined position in the captured image (such as the center or upper left corner) to the optical axis center can be used as the image circle position.

[0077] The image circle diameter is the diameter of the image circle (circumferential fisheye image) formed on the image sensor. The field of view is the field of view (range) of the image circle (circumferential fisheye image) formed on the image sensor. The distortion correction coefficient is the ratio of the design image height to the ideal image height of the lens. A distortion correction coefficient may be set for each image height, and for image heights for which a distortion correction coefficient has not been set, the distortion correction coefficient may be calculated by interpolation using multiple distortion correction coefficients. A polynomial that approximates the relationship between image height and distortion correction coefficient may also be set.

[0078] The image circle diameter, field of view, and distortion correction coefficient may or may not be common parameters for the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R).

[0079] Lens individual values ​​include image circle misalignment, optical axis tilt, and image magnification misalignment. This information is prepared by measuring each of the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R).

[0080] Image circle misalignment is the deviation of the center coordinates of the image circle (circumferential fisheye image) formed on the image sensor from the design value. For example, image circle misalignment includes both horizontal and vertical misalignment. Using the design value coordinates (a two-dimensional coordinate system including horizontal and vertical coordinates) as the origin, the horizontal misalignment is indicated by the horizontal coordinates, and the vertical misalignment is indicated by the vertical coordinates. Optical axis tilt is the deviation of the direction of the optical axis on the subject side from the design value. For example, optical axis tilt includes both horizontal and vertical misalignment. The misalignment in each direction is indicated by an angle. Image magnification misalignment is the deviation of the size of the image circle (circumferential fisheye image) formed on the image sensor from the design value. This misalignment is indicated, for example, by a ratio to the design value.

[0081] (Regarding the foreign object information acquisition unit 602) The foreign object information acquisition unit 602 acquires foreign object information from the other information area. Figure 9A is a schematic diagram showing an example of foreign object information acquired by the foreign object information acquisition unit 602. The foreign object information includes a foreign object data count and foreign object area data for each foreign object.

[0082] The foreign object data count indicates the number of foreign object region data points included in the foreign object information. Each foreign object region data point contains information such as the center coordinates and radius of the circular region containing the foreign object, as shown in Figure 9B. The circular region determined by the center coordinates and radius is the foreign object region, which is the area containing the foreign object.

[0083] (Regarding the position correction unit 603) The position correction unit 603 corrects the positions of the right image region and the left image region (the positions of the two optical images) in the captured image captured by the two-lens system, based on lens information, to correspond to the two optical systems (right eye optical system 301R and left eye optical system 301L).

[0084] The following describes the position correction process for the right image region and the left image region with reference to Figure 10. First, the position correction unit 603 acquires the width WA of the uncorrected captured image (uncorrected image), and also acquires the width WL of the left image and the width WR of the right image based on the field of view recorded in the lens information. Let's assume that WA = WL + WR holds true. Then, if the upper left corner of the captured image is taken as the origin of the coordinate system (0,0), the X coordinate of the boundary line B between the right image region and the left image region is X B Regarding X B =WR is true.

[0085] Regarding the pixel coordinates (X,Y) in the uncorrected image, X <X B In this case, the pixel is included in the right image of the uncorrected image. Therefore, the position correction unit 603 adjusts the pixel that was located at coordinate (X,Y) in the uncorrected image to coordinate (X',Y')=(X+X B Move to ,Y).

[0086] On the other hand, X ≥ X B In this case, the pixel is included in the left image of the uncorrected image. Therefore, the position correction unit 603 adjusts the pixel that was located at coordinate (X,Y) in the uncorrected image to coordinate (X',Y')=(XX) in the corrected image (corrected image). B Move to ,Y).

[0087] Through this correction process, the regions of the right and left images are swapped, and the positions of each image region are corrected to correspond to the optical system. In this embodiment, the above-described correction process (swapping process) is assumed to be performed, but any process may be performed as long as the positions of each image region are corrected to correspond to the optical system.

[0088] (Regarding adjustment unit 604) The adjustment unit 604 performs adjustments such as adjusting the center coordinates of the foreign object region data in the foreign object information. The processing of the adjustment unit 604 will be described later using the flowchart in Figure 13.

[0089] (Regarding the foreign matter removal section 605) The foreign object removal unit 605 removes foreign objects from the captured image corrected by the position correction unit 603 using the foreign object information adjusted by the adjustment unit 604. In this embodiment, the foreign object removal unit 605 removes foreign objects by copying the surrounding pixels above, below, left, and right of the pixels in the foreign object region of the captured image to the pixels in the foreign object region. However, the method of removing foreign objects is not particularly limited, and various known methods may be used, such as a method of removal based on other captured images that are temporally close. Furthermore, the process of removing foreign objects may be performed by another information processing device that receives the corrected captured image and the adjusted foreign object information from the PC 500.

[0090] (Regarding the exposure area calculation unit 606) The exposure area calculation unit 606 (exposure determination unit) calculates the exposure area (exposed area; the area in which the captured subject is visible) based on the image circle position and image circle diameter recorded in the lens information. In the captured image (uncorrected image) captured by a twin-lens system, two exposure areas are calculated based on the information of each optical system. Specifically, as shown in Figure 11, in the captured image, the area of ​​a circle with the image circle position as the center and a radius of half the image circle diameter (image circle radius) corresponds to the exposure area. At this time, the exposure area calculation unit 606 may calculate the exposure area taking into account the image circle position shift.

[0091] (Regarding the removal of foreign objects) The foreign matter removal process performed by the PC500 according to this embodiment will be explained with reference to the flowchart in Figure 12. The process shown in the flowchart in Figure 12 is realized when the control unit 501 (processor) loads the program stored in the ROM 502 into the RAM 503 and executes it. Specifically, the process shown in the flowchart in Figure 12 is realized when the control unit 501 operates in the configurations shown in Figure 6.

[0092] In step S1201, the image acquisition unit 600 creates an image file containing the captured image (Figure (See 7) is obtained from the recording medium 504.

[0093] In step S1202, the lens information acquisition unit 601 acquires lens information from the lens information area in the image file.

[0094] In step S1203, the foreign object information acquisition unit 602 acquires foreign object information from the other information area of ​​the image file.

[0095] In step S1204, the lens information acquisition unit 601 determines, based on the lens flag included in the lens information, whether the lens used to capture the image is a binocular lens (i.e., whether it includes multiple optical systems). If it is determined that the lens used for capturing is a binocular lens (i.e., the captured image includes two regions: the region of the right image and the region of the left image), the process proceeds to step S1205. If it is determined that the lens used for capturing is not a binocular lens (i.e., the captured image does not include two regions: the region of the right image and the region of the left image), the process proceeds to step S1207.

[0096] In step S1205, the position correction unit 603 performs a correction process (swapping process) that swaps the right image region and the left image region in the captured image based on the lens information.

[0097] In step S1206, the adjustment unit 604 adjusts the center coordinates of the foreign object information (performs adjustment processing). Details of the adjustment processing in step S1206 will be described later with reference to Figure 13.

[0098] In step S1207, the foreign matter removal unit 605 removes foreign matter from the corrected image based on the foreign matter information. In other words, if the image includes the regions of the right image and the left image (if YES in step S1204), the foreign matter removal unit 605 removes foreign matter based on the foreign matter information adjusted in step S1206. On the other hand, if the image does not include the regions of the right image and the left image (if NO in step S1204), the foreign matter removal unit 605 removes foreign matter based on unadjusted foreign matter information.

[0099] Note that between step S1204 (or step S1206) and step S1207, image adjustments that do not involve coordinate transformations (for example, adjustment of brightness values ​​or color adjustments) may be performed.

[0100] (Regarding adjustment procedures; S1206) The adjustment process performed in step S1206 will be explained with reference to the flowchart in Figure 13.

[0101] In step S1301, the exposure area calculation unit 606 calculates the exposure area (the area in which the captured subject is visible; the area other than the black area) in the captured image before correction processing, based on the lens information (lens setting value and individual lens value).

[0102] In step S1302, the adjustment unit 604 determines the right image region and the left image region in the captured image before correction processing based on the angle of view indicated by the lens information.

[0103] In step S1303, the adjustment unit 604 determines whether or not unadjusted foreign object region data (foreign object region data whose coordinates have not been adjusted in step S1308) is included in the foreign object information. If it is determined that unadjusted foreign object region data is included in the foreign object information, the process proceeds to step S1304. If it is determined that all foreign object region data in the foreign object information has been adjusted, the process shown in the flowchart of Figure 13 is completed.

[0104] In step S1304, the adjustment unit 604 selects one foreign object region data from the unadjusted foreign object region data. The adjustment unit 604 then determines whether the foreign object region indicated by the selected foreign object region data (hereinafter referred to as the "selected foreign object region") is included in the exposure region (the exposure region in the captured image before correction processing) (i.e., overlaps with the exposure region). Specifically, the adjustment unit 604 determines that the selected foreign object region is included in the exposure region if even one of the pixels included in the selected foreign object region is included in the exposure region (i.e., overlaps with the exposure region). If it is determined that the selected foreign object region is not included in the exposure region, the process proceeds to step S1305. If it is determined that the selected foreign object region is included in the exposure region, the process proceeds to step S1306.

[0105] In step S1305, the adjustment unit 604 determines that the foreign object region data indicating the selected foreign object region is unnecessary in the foreign object information because the selected foreign object region is not included in the exposure region. This is because the foreign object does not affect the display of the subject in the captured image. The adjustment unit 604 then deletes the foreign object region data indicating the selected foreign object region from the foreign object information. At the same time, the adjustment unit 604 updates the foreign object information by decreasing the foreign object data count in the foreign object information by 1. According to step S1305, the possibility of unnecessary processing in the foreign object removal process (step S1207) can be reduced, and the foreign object removal process can be performed efficiently.

[0106] In step S1306, the adjustment unit 604 determines whether the selected foreign object region spans the right image region and the left image region. The selection foreign object region spans the right image region and the left image region, meaning that both the right image region and the left image region in the captured image before correction processing contain at least one pixel of the selected foreign object region. If it is determined that the selected foreign object region spans two regions (the right image region and the left image region), the process proceeds to step S1307. If it is determined that the selected foreign object region does not span two regions, the process proceeds to step S1308.

[0107] In step S1307, the adjustment unit 604 adds foreign object region data corresponding to (belonging to) a region that does not include the center coordinates of the foreign object region data of the selected foreign object region (either the region of the right image or the region of the left image in the captured image before correction processing). Figure 14 is a diagram illustrating an example of the foreign object region data to be added.

[0108] For example, as shown in Figure 14, the adjustment unit 604 determines the coordinate closest to the center coordinate of the selected foreign object region in the region of the right image and the left image that does not include the center coordinate of the selected foreign object region, as the center coordinate of the foreign object region data to be added (referred to as the "additional center coordinate"). Note that the center coordinate of the additional foreign object region data may be a coordinate at a position symmetric to the center coordinate of the selected foreign object region with respect to the boundary line B (the boundary line B between the right image region and the left image region).

[0109] The adjustment unit 604 then determines the radius of the foreign object region to be added by determining the distance between the pixel furthest from the additional center coordinates and the additional center coordinates in the circular region determined from the foreign object region data, which does not include the center coordinates of the selected foreign object region (either the right image region or the left image region). Note that the radius of the additional foreign object region may be the same value as the radius of the foreign object region data of the selected foreign object region.

[0110] Furthermore, the adjustment unit 604 increases the foreign object data count in the foreign object information by 1 to match the foreign object data count in the foreign object information with the actual number of foreign object data.

[0111] In step S1308, the adjustment unit 604 determines whether the center coordinates of the foreign object region data belong to the right image region or the left image region of the image before correction. Then, the same coordinate calculation (swapping process) as the position correction unit 603 is performed (see Figure 10). As a result, the adjustment unit 604 adjusts the position (center position) of the foreign object information so that the center coordinates of the foreign object area data match the coordinates of the foreign object in the captured image. In other words, if a certain foreign object area (center coordinates of the foreign object area) in the captured image belongs to the right image area, the adjustment unit 604 converts the center coordinates of the foreign object area from the coordinates of the right image area before correction to the coordinates of the right image area after correction. On the other hand, if a certain foreign object area (center coordinates of the foreign object area) in the captured image belongs to the left image area, the adjustment unit 604 converts the center coordinates of the foreign object area from the coordinates of the left image area before correction to the coordinates of the left image area after correction.

[0112] Based on the above, since the position of the foreign matter area is adjusted based on the lens information, the foreign matter removal process can be performed appropriately even after swapping the left and right regions so that the positional relationship between the two optical systems and the positional relationship between the two left and right regions are the same.

[0113] Furthermore, in the above, the statement "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" can be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, the statement "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" can be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, the expression "greater than or equal to A" can be replaced with "A or greater than (higher; longer; more)" or rephrased as "greater than (higher; longer; more)." On the other hand, the expression "less than or equal to A" can be replaced with "A or less than (lower; shorter; fewer)" or rephrased as "less than (lower; shorter; fewer)." Furthermore, "larger than A (higher; longer; more)" can be rephrased as "greater than or equal to A," and "smaller than A (lower; shorter; fewer)" can be rephrased as "less than or equal to A."

[0114] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0115] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0116] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0117] The above-disclosed embodiments include the following configurations, methods, and programs. [Configuration 1] Image acquisition means for acquiring captured images, If the captured image includes a first region corresponding to a first optical image input via a first optical system, and a second region corresponding to a second optical image input via a second optical system, which is an optical image having a predetermined parallax with respect to the first optical image, then the lens information acquisition means acquires lens information relating to the first optical system and the second optical system. A means for acquiring foreign object information that acquires foreign object information indicating the region of one or more foreign objects in the captured image, Correction means for correcting the position of the first region and the position of the second region based on the lens information, An adjustment means for adjusting the foreign matter information based on the lens information, An information processing device characterized by having the following features. [Configuration 2] The lens information includes information relating to the optical axis center of the first optical system and the optical axis center of the second optical system in the captured image. The correction means swaps the positions of the first region and the second region in the captured image based on the optical axis center of the first optical system and the optical axis center of the second optical system. The information processing device according to configuration 1, characterized by the above. [Configuration 3] The correction means corrects the positions of the first region and the second region by applying correction processing based on the lens information to each of the first region and the second region, The adjustment means adjusts the foreign matter information by applying the correction process to each of the one or more foreign matter regions indicated by the foreign matter information. An information processing device according to configuration 1 or 2, characterized by the above. [Structure 4] The aforementioned foreign object information includes region data for each foreign object in the captured image, which includes information on the center coordinates and radius of a circular region indicating the area of ​​the foreign object. An information processing device according to any one of configurations 1 to 3. [Composition 5] The adjustment means adjusts the information of the center coordinates included in the foreign object information based on the lens information. The information processing apparatus according to configuration 4, characterized by the features described above. [Composition 6] If the adjustment means has a region of a first foreign object that spans both the first region and the second region in the image before correction, it adds new region data to the foreign object information that corresponds to a region in the first region and the second region that does not correspond to the center coordinates of the circular region indicating the region of the first foreign object. The information processing apparatus according to configuration 5, characterized by the features described herein. [Composition 7] The system further includes exposure determination means for determining the exposed region among the first region and the second region based on the lens information, The adjustment means deletes the information relating to the second foreign object from the foreign object information if the information relating to the second foreign object is included in the foreign object information and the region of the second foreign object does not overlap with the exposure region. An information processing device according to any one of configurations 1 to 6. [Structure 8] The system further includes a removal means for removing foreign matter from the captured image based on the adjusted foreign matter information. An information processing device according to any one of configurations 1 to 7, characterized by the above. [Composition 9] If the captured image does not include the first region and the second region, the adjustment means does not adjust the foreign matter information. An information processing device according to any one of configurations 1 to 8. [Configuration 10] An imaging means for capturing the image of the captured image, which includes one or more optical images corresponding to one or more optical systems, An information processing device described in any one of items 1 to 9, An imaging device characterized by having the following features. [method] Image acquisition step to acquire captured images, If the captured image includes a first region corresponding to a first optical image input via a first optical system, and a second region corresponding to a second optical image input via a second optical system, which is an optical image having a predetermined parallax with respect to the first optical image, then the lens information acquisition step involves acquiring lens information relating to the first optical system and the second optical system. A foreign object information acquisition step, which acquires foreign object information indicating the regions of one or more foreign objects in the captured image, A correction step in which the positions of the first region and the second region are corrected based on the lens information, An adjustment step to adjust the foreign matter information based on the lens information, An information processing method characterized by having the following features. [program] A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 9 of the configuration. [Explanation of Symbols]

[0118] 500: PC, 600: Image acquisition unit, 601: Lens information acquisition unit, 602: Foreign object information acquisition section, 603: Position correction section, 604: Adjustment section

Claims

1. Image acquisition means for acquiring an image image including a first region corresponding to a first optical image input via a first optical system, and a second region corresponding to a second optical image input via a second optical system, which is an optical image having a predetermined parallax with respect to the first optical image. Lens information acquisition means for acquiring lens information relating to the first optical system and the second optical system, Foreign object information acquisition means for acquiring foreign object information which includes, for each foreign object in the captured image, region data that indicates the region of one or more foreign objects in the captured image, and which includes information on the center coordinates and radius of the circular region indicating the region of the foreign object. Correction means for swapping the positions of the first region and the second region in the captured image based on the lens information, The system includes an adjustment means for adjusting the information of the central coordinates included in the foreign object information based on the lens information, The adjustment means is characterized in that, if a region of a first foreign object exists in the captured image before correction that spans both the first region and the second region, it adds new region data corresponding to a region in the first region and the second region that does not correspond to the center coordinates of the circular region indicating the region of the first foreign object to the foreign object information.

2. The lens information includes information relating to the optical axis center of the first optical system and the optical axis center of the second optical system in the captured image. The correction means swaps the positions of the first region and the second region in the captured image based on the optical axis center of the first optical system and the optical axis center of the second optical system. The information processing apparatus according to feature 1.

3. The correction means applies correction processing based on the lens information to each of the first and second regions, thereby swapping the positions of the first and second regions. The adjustment means applies to each of the one or more foreign matter regions indicated by the foreign matter information. By performing the correction process described above, the information of the center coordinates is adjusted. The information processing apparatus according to claim 1 or 2.

4. The system further includes exposure determination means for determining the exposed region among the first region and the second region based on the lens information, The adjustment means deletes the information relating to the second foreign object from the foreign object information if the information relating to the second foreign object is included in the foreign object information and the region of the second foreign object does not overlap with the exposure region. The information processing apparatus according to claim 1 or 2.

5. The system further includes a removal means for removing foreign matter from the captured image based on the adjusted foreign matter information. The information processing apparatus according to claim 1 or 2.

6. If the image acquisition means acquires a second image which is different from the captured image and does not include the first region and the second region, the adjustment means does not adjust the foreign matter information of the second image. The information processing apparatus according to claim 1 or 2.

7. An imaging means for capturing the image of the captured image including the first region and the second region, An information processing apparatus according to claim 1 or 2, An imaging device characterized by having the following features.

8. An image acquisition step of acquiring an image image including a first region corresponding to a first optical image input via a first optical system, and a second region corresponding to a second optical image input via a second optical system, which is an optical image having a predetermined parallax with respect to the first optical image; A lens information acquisition step for acquiring lens information relating to the first optical system and the second optical system, A foreign object information acquisition step involves acquiring foreign object information that includes, for each foreign object in the captured image, region data that indicates the region of one or more foreign objects, and which includes information on the center coordinates and radius of the circular region indicating the region of the foreign object. A correction step in which the positions of the first region and the second region in the captured image are swapped based on the lens information, The adjustment step includes adjusting the information of the central coordinates included in the foreign object information based on the lens information, In the adjustment step, if there is a region of a first foreign object that spans both the first region and the second region in the captured image before correction, new region data corresponding to the region of the first region and the second region that does not correspond to the center coordinates of the circular region indicating the region of the first foreign object is added to the foreign object information. An information processing method characterized by the following:

9. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1 or 2.

Images