Information processing device, imaging device, control method, program, and storage medium

Abstract

To provide a technique capable of displaying two images captured with a camera attached with a lens unit including two optical systems such that positions of the two images correspond to the two optical systems.SOLUTION: An information processing apparatus includes: obtaining means for obtaining one image including a first image region corresponding to a first optical image input via a first optical system and a second image region corresponding to a second optical image input via a second optical system, the second optical system having predetermined parallax with respect to the first optical system, and correction information on the first optical system and the second optical system; correction means for executing correction processing to correct the position of a pixel included in the first image region and a pixel included in the second image region in the image, based on the correction information; and generation means for executing processing to convert the corrected first and the second image regions to generate a processed image.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an imaging apparatus, a control method, a program, and a storage medium.

Background Art

[0002] There is known a technique of capturing two images with a parallax using two cameras and displaying the two captured images in a stereoscopic manner. Patent Document 1 discloses a camera that can capture two images with a parallax at once by attaching a lens unit having two optical systems.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when an image captured by attaching a lens unit having two optical systems (one image including two images with a parallax) is displayed like a conventional image, the positional relationship between the two optical systems and the positional relationship between the two images in the one image are reversed.

[0005] An object of the present invention is to provide a technique that enables a display in which the positions of two images captured by attaching a lens unit having two optical systems correspond to the two optical systems.

Means for Solving the Problems

[0006] The information processing apparatus of the present invention includes a first image area corresponding to a first optical image input via a first optical system on the right side of a lens unit, and a second image area corresponding to a second optical image input via a second optical system on the left side of the lens unit, which has a predetermined parallax with respect to the first optical system, in one image,A first parameter which is a design parameter of the lens unit, and a parameter specific to the lens unit. A means of obtaining and An adjustment means for adjusting the first parameter to a second parameter based on the parameters specific to the lens unit, The aforementioned Second parameter Based on ite , The correspondence between pixels before and after the transformation of the first image region and the second image region is generated, and the first Image area and Second A generation means that performs a process to transform the image region and generates a processed image, The process for transforming the first image region and the second image region includes, A swapping process is performed to swap the positions of the first image region and the second image region in the aforementioned image. implied In the image before swapping the positions of the first image region and the second image region, the first image region is positioned on the left and the second image region on the right, when the upper part of the imaging range is considered the upper part of the image. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to display two images captured with a lens unit having two optical systems, such that the positions of the two images correspond to the two optical systems. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing the overall system configuration. [Figure 2] This is an external view of the camera. [Figure 3] This is a block diagram showing the camera configuration. [Figure 4] This is a schematic diagram showing the configuration of the lens unit. [Figure 5] This is a block diagram showing the PC configuration. [Figure 6] This is a flowchart showing the camera's operation. [Figure 7] This is a flowchart showing how the PC works. [Figure 8] This is a schematic diagram showing the left and right sides swapped. [Figure 9] This is a schematic diagram showing the structure of an image file. [Figure 10] This is a schematic diagram showing lens information and camera information. [Figure 11] This is a schematic diagram of the display screen. [Figure 12]It is a schematic diagram of positive-distance cylindrical conversion. [Figure 13] It is a flowchart showing the operation of the camera. [Figure 14] It is a flowchart showing the operation of the PC. [Figure 15] It is a schematic diagram of the display screen. [Figure 16] It is a diagram showing the difference between the left and right lenses. [Figure 17] It is a diagram showing the deviation caused by the lens error. [Figure 18] It is a diagram showing the correction of the image position on the spatial coordinates.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] <The First Embodiment> A first embodiment of the present invention will be described. FIGS. 1(a) and 1(b) are schematic diagrams showing an example of the overall configuration of the system according to this embodiment. The system according to this embodiment includes a digital camera (camera) 100 and a personal computer (PC) 500. A lens unit 300 is attached (connected) to the camera 100. Although details of the lens unit 300 will be described later, by attaching the lens unit 300, the camera 100 can capture two images (still images or moving images) having a predetermined parallax at once. The PC 500 is an information processing device that handles images captured by an imaging device such as the camera 100. FIG. 1(a) shows a configuration in which the camera 100 and the PC 500 are communicably connected to each other by wireless or wired means. FIG. 1(b) shows a configuration in which an image captured by the camera 100 is input to the PC 500 via an external storage device on a file basis. The external storage device may or may not be connected to both the camera 100 and the PC 500. For example, the external storage device may be connected to the camera 100, and a file of an image captured by the camera 100 may be stored in the external storage device. Thereafter, the external storage device may be removed from the camera 100 and connected to the PC 500, and the PC 500 may take in the file stored in the external storage device.

[0011] FIGS. 2(a) and 2(b) are external views showing an example of the appearance of the camera 100. FIG. 2(a) is a perspective view of the camera 100 seen from the front side, and FIG. 2(b) is a perspective view of the camera 100 seen from the back side.

[0012] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface. The shutter button 101 is an operating element for giving a shooting preparation instruction or a shooting instruction. The power switch 102 is an operating element for switching the power of camera 100 on and off. The mode selector switch 103 is an operating element for switching between various modes. The main electronic dial 104 is a rotary operating element for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operating element for moving the selection frame (cursor) and advancing images. The video button 106 is an operating element for giving instructions to start and stop video recording. The external viewfinder display 107 displays various settings such as shutter speed and aperture.

[0013] The camera 100 has a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back. The display unit 108 displays images and various information. The touch panel 109 is an operating member that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 is an operating unit consisting of keys that can be pressed up, down, left, and right (4-way keys). Processing can be performed according to the position where the directional keys 110 are pressed. The SET button 111 is mainly for selecting items. The AE lock button 112 is an operating member that is pressed when determining the exposure state. The AE lock button 112 is an operating member that is pressed when fixing the exposure state in shooting standby mode. The zoom button 113 is an operating member that switches the zoom mode on and off in the live view display (LV display) of shooting mode. When the zoom mode is on, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 104. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio. The playback button 114 is an operating member that switches between shooting mode and playback mode. By pressing the playback button 114 in shooting mode, the camera switches to playback mode, and the latest image among the images recorded on the recording medium 227 (described later) can be displayed on the display unit 108.

[0014] The menu button 115 is an operating element that is pressed to display a menu screen on the display unit 108 that allows for various settings. The user can intuitively make various settings using the menu screen displayed on the display unit 108 and the directional keys 110 and SET button 111. The eyepiece section 116 is the part that the user looks through when they place their eye on the eyepiece viewfinder (peep-in type viewfinder) 117. The user can view the image displayed on the EVF 217 (Electronic View Finder), which will be described later, inside the camera 100, through the eyepiece section 116. The eyepiece detection section 118 is a sensor that detects whether or not the user is looking through the eyepiece section 116 (eyepiece viewfinder 117).

[0015] The touch bar 119 is a line-shaped touch operation component (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched by the right thumb when the grip portion 120 is held with the right hand (with the little finger, ring finger, and middle finger) so that the shutter button 101 can be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece 116 with their eyepiece in the eyepiece viewfinder 117 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (touching and releasing without moving the touch position within a predetermined period), left and right sliding operations (touching and then moving the touch position while keeping the touch on the surface), etc. The touch bar 119 is a different operation component from the touch panel 109 and does not have a display function. The Touch Bar 119 functions, for example, as a multifunction bar (M-Fn bar) to which various functions can be assigned.

[0016] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. With the camera 100 held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section located on the back of the camera 100, in a place where it is easy to rest the thumb of the right hand holding the grip section 120 when no operating members are being operated. The thumb rest section 121 is made of a rubber material or the like to enhance the holding force (grip feel). The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external equipment (external devices). The cover 123 protects the recording medium 227 and the slot for storing the recording medium 227, which will be described later, by closing the slot. The communication terminal 124 is a terminal for communicating with the lens unit (such as the lens unit 200 or lens unit 300, which will be described later) that can be attached to or detached from the camera 100.

[0017] Figure 3 is a block diagram showing an example of the configuration of camera 100. Note that components identical to those in Figure 2 are denoted by the same reference numerals as in Figure 2, and their descriptions are omitted as appropriate. In Figure 3, the camera... The lens unit 200 is attached to the Mera 100.

[0018] First, let's describe the lens unit 200. The lens unit 200 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 200 is a single-lens reflex lens and is an example of a normal lens. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, etc.

[0019] The aperture 201 is configured to have an adjustable aperture diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc., based on instructions from the system control unit 50, which will be described later. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203 and focuses by changing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication takes place via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100.

[0020] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0021] The shutter 210 is a focal-plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an image sensor composed of a CCD or CMOS element, etc., which converts an optical image into an electrical signal. The imaging unit 211 may have an image plane phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. The image processing unit 214 also performs predetermined calculation processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. Through this processing, TTL (through-the-lens) AF processing, AE (automatic exposure) processing, EF (flash pre-flash) processing, etc. are performed. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data, and the system control unit 50 performs TTL-type AWB (auto white balance) processing based on the obtained calculation results.

[0022] Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data for display on the display unit 108 and EVF 217. The memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. The memory 215 also serves as a memory for image display (video memory).

[0023] The D / A converter 216 converts the display image data stored in the memory 215 into an analog signal and supplies it to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 receive the analog signal from the D / A converter 216. The system displays information according to the signal. The display unit 108 and EVF 217 are, for example, displays such as LCDs or OLEDs. The digital signal, which has been converted from digital to digital by the A / D converter 212 and stored in the memory 215, is converted to an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and EVF 217 for display, thereby enabling live view display.

[0024] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0025] The camera 100 also includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0026] For example, RAM is used as the system memory 218. The system memory 218 stores constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 219, etc. The non-volatile memory 219 is an electrically erasable and recordable memory, and for example, EEPROM is used as the non-volatile memory 219. The non-volatile memory 219 stores constants and programs for the operation of the system control unit 50, etc. The program here refers to a program for executing the flowchart described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from external devices connected by wireless or wired cables. The communication unit 221 can also connect to wireless LAN (Local Area Network) and the internet. In addition, the communication unit 221 can communicate with external devices using Bluetooth® and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive images and other various information from external devices. The attitude detection unit 222 detects the attitude of the camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image taken by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image taken by the imaging unit 211, or rotate the image according to the detected attitude. For example, an acceleration sensor or a gyro sensor can be used for the attitude detection unit 222. It is also possible to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.) using the attitude detection unit 222.

[0027] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece unit 116 (eyepiece finder 117). For example, an infrared proximity sensor can be used for the eyepiece detection unit 118. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece unit 116 to the object can be determined by the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece unit 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eye-catching) and departure (eye-moving) of an eye (object) to the eyepiece unit 116. When an object is detected approaching the eyepiece unit 116 within a predetermined distance from a non-eyepiece state (non-approach state), it is detected that the eye has been caught in the eye. On the other hand, if the object that was detected as approaching moves beyond a predetermined distance from the eye-contact state (close-up state), it is detected as having been moved away from the eye. The threshold for detecting eye contact and the threshold for detecting eye separation are different, for example, by providing hysteresis. It is also acceptable to do so. Furthermore, after detecting eye contact, the eye will remain in eye contact state until eye separation is detected. After detecting eye separation, the eye will remain in non-eye contact state until eye contact is detected. The system control unit 50 switches the display (display state) / non-display (non-display state) of the display unit 108 and EVF 217 according to the state detected by the eye contact detection unit 118. Specifically, at least in the shooting standby state and when the display destination switching setting is set to automatic switching, the display destination is set to the display unit 108 and the display is turned on, and the EVF 217 is turned off when the eye is not in eye contact. Also, when the eye is in eye contact, the display destination is set to the EVF 217 and the display is turned on, and the display unit 108 is turned off. Note that the eye contact detection unit 118 is not limited to an infrared proximity sensor, and other sensors that can detect a state that can be considered as eye contact may be used for the eye contact detection unit 118.

[0028] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, and the like.

[0029] The external viewfinder display unit 107 is driven by the external viewfinder display drive circuit 223 and displays various settings of the camera 100, such as shutter speed and aperture. The power control unit 224 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, supplying the necessary voltage to each part, including the recording medium 227, for the required period. The power supply unit 225 includes primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and an AC adapter. The recording medium I / F 226 is an interface with the recording medium 227, such as a memory card or hard disk. The recording medium 227 is a memory card or the like for recording captured images and consists of semiconductor memory or a magnetic disk. The recording medium 227 may be detachable from the camera 100 or may be built into the camera 100.

[0030] The operation unit 228 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, power switch 102, mode switch 103, touch panel 109, and other operation units 229. Other operation units 229 include the main electronic dial 104, sub electronic dial 105, video button 106, directional keys 110, SET button 111, AE lock button 112, zoom button 113, playback button 114, menu button 115, touch bar 119, and the like.

[0031] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 turns on during the operation of the shutter button 101, so-called half-press (shooting preparation instruction), and outputs a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 231 turns on when the operation of the shutter button 101 is completed, so-called full-press (shooting instruction), and outputs a second shutter switch signal SW2. The system control unit 50 starts a series of shooting processes, from reading the signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227, in response to the second shutter switch signal SW2.

[0032] The mode switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, playback mode, etc. The modes included in still image shooting mode are auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). In addition, there are various scene modes and custom modes for shooting settings specific to different shooting scenes. There are modes such as [code]. The user can directly switch to any of the above-mentioned shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to any of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0033] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is mounted on the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109; either method is acceptable.

[0034] The system control unit 50 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). The finger or pen that was touching the touch panel 109 is lifted (released), meaning the touch action ends (hereinafter referred to as "Touch-Up"). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0035] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0036] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If it is detected that a touch move of a predetermined distance or more has been performed, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109 and then quickly moved a certain distance and then released is called a flick. In other words, a flick is an operation in which the finger is quickly traced across the touch panel 109 as if flicking it. If it is detected that a touch move of a predetermined distance or more has been performed at a predetermined speed or faster, and a touch-up is then detected, it is determined that a flick operation has been performed. A click is detected (it can be determined that a flick occurred following a slide operation). Furthermore, touching multiple locations (for example, two points) together (multitouch) and bringing the touch positions closer together is called a pinch-in, and touching them further apart is called a pinch-out. Pinch-out and pinch-in are collectively referred to as a pinch operation (or simply a pinch).

[0037] Figure 4 is a schematic diagram showing an example of the configuration of the lens unit 300. Figure 4 shows the lens unit 300 attached to the camera 100. Note that, among the components of the camera 100 shown in Figure 4, those identical to those described in Figure 3 are denoted by the same reference numerals as in Figure 3, and their descriptions are omitted as appropriate.

[0038] The lens unit 300 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 300 is a twin-lens system capable of capturing parallax right and left images. In this embodiment, the lens unit 300 has two optical systems, and each of the two optical systems can capture a wide field of view of approximately 180 degrees. Specifically, each of the two optical systems of the lens unit 300 can capture a subject with a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle). In other words, each of the two optical systems can capture an area of ​​the front hemisphere.

[0039] The lens unit 300 includes a right-eye optical system 301R having multiple lenses and a reflective mirror, a left-eye optical system 301L having multiple lenses and a reflective mirror, and a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. The right-eye optical system 301R has a lens 302R positioned on the subject side, and the left-eye optical system 301L has a lens 302L positioned on the subject side. Lenses 302R and 302L face the same direction, and their optical axes are approximately parallel.

[0040] The lens unit 300 is a binocular lens (VR180 lens) for obtaining VR180 images, which are one of the VR (Virtual Reality) image formats that enable binocular stereoscopic viewing. In this embodiment, the lens unit 300 has a fisheye lens capable of capturing a range of approximately 180 degrees in each of the right eye optical system 301R and the left eye optical system 301L. The range that can be captured by the lenses of each of the right eye optical system 301R and the left eye optical system 301L may be narrower than 180 degrees, around 160 degrees. The lens unit 300 can image the right image (first image) formed via the right eye optical system 301R and the left image (second image) formed via the left eye optical system 301L onto one or two image sensors of the camera to which the lens unit 300 is attached.

[0041] The lens unit 300 is attached to the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. In this way, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0042] In this embodiment, the right image formed via the right-eye optical system 301R and the left image formed via the left-eye optical system 301L are simultaneously (as a set) imaged on the imaging unit 211 of the camera 100. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single image sensor. The imaging unit 211 converts the imaged subject image (optical signal) into an analog electrical signal. By using the lens unit 300 in this way, two images with parallax can be simultaneously (as a set) acquired from two locations (optical systems) - the right-eye optical system 301R and the left-eye optical system 301L. By separating the acquired images into a left-eye image and a right-eye image and displaying them in VR, the user can view a three-dimensional VR image in a range of approximately 180 degrees. The image can be viewed. In other words, the user can view VR180 images in 3D.

[0043] Here, a VR image is an image that can be displayed in VR, as described later. VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera), and panoramic images with a wider field of view (effective field of view) than the display area that can be displayed on the display unit at once. Furthermore, VR images are not limited to still images, but also include videos and live images (images acquired from the camera in near real-time). A VR image has a field of view (effective field of view) of up to 360 degrees horizontally and vertically. In addition, VR images also include images with a wider field of view than that that can be captured by a normal camera, or images with a wider field of view than the display area that can be displayed on the display unit at once, even if the field of view is less than 360 degrees horizontally or vertically. Images captured by camera 100 using the lens unit 300 described above are a type of VR image. VR images can be displayed in VR, for example, by setting the display mode of a display device (a display device that can display VR images) to "VR view". By displaying VR images with a 360-degree field of view in VR, users can change the orientation of the display device left or right (horizontal rotation direction) to view seamless, omnidirectional images in the left and right directions.

[0044] VR display (VR view) is a display method (display mode) that allows the display range to be changed, displaying images within a field of view that corresponds to the orientation of the display device. One type of VR display is "single-eye VR display (single-eye VR view)," which displays a single image by performing a transformation (distortion correction) that maps the VR image to a virtual sphere. Another type of VR display is "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side by side by performing a transformation that maps each to a virtual sphere. By performing "two-eye VR display" using VR images for the left eye and the right eye that have parallax with each other, it is possible to view these VR images in 3D. In any type of VR display, for example, when a user wears a display device such as an HMD (head-mounted display), the image displayed will correspond to the field of view that corresponds to the orientation of the user's face. For example, suppose that in a VR image, at a certain point in time, the image displayed will correspond to a field of view that is centered around 0 degrees horizontally (a specific direction, e.g., north) and 90 degrees vertically (90 degrees from the zenith, i.e., horizontal). If the orientation of the display device is reversed from this state (for example, changing the display surface from facing south to facing north), the display range of the same VR image will change to an image with a field of view centered on 180 degrees horizontally (opposite direction, e.g., south) and 90 degrees vertically. In other words, if the user, while wearing the HMD, turns their face from north to south (i.e., turns their back), the image displayed on the HMD will also change from a northern image to a southern image. Note that the VR image captured using the lens unit 300 of this embodiment is an image capturing a range of approximately 180 degrees in front (180° image), and there is no image of a range of approximately 180 degrees behind. If such an image is displayed in VR and the orientation of the display device is changed to the side where there is no image, a blank area will be displayed.

[0045] By displaying VR images in VR in this way, users can visually experience the sensation (immersion) of being inside the VR image (in VR space). Note that the method of displaying VR images is not limited to changing the orientation of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel or directional buttons. Furthermore, during VR display (in "VR View" display mode), in addition to changing the display range due to changes in orientation, the display range may also be changed in response to touch movements on the touch panel, drag operations with a mouse, or pressing directional buttons. Note that a smartphone mounted on VR goggles (head-mounted adapter) is a type of HMD (Head-Mounted Display).

[0046] Figure 5 is a block diagram showing an example of the configuration of the PC500. The control unit 501 is, for example, a Central Processing Unit (CPU) and controls the entire PC500. The Read Only Memory (ROM) 502 stores programs and parameters. Random Access Memory (RAM) 503 temporarily stores data. The recording medium 504 is a hard disk or flash memory fixed to the PC 500, or a removable optical disk, magnetic card, optical card, IC card, memory card, etc. The image file captured by the camera 100 is read from the recording medium 504. The operation unit 505 accepts user operations on the PC 500. The operating device used by the user when performing operations may be a button or touch panel provided on the PC 500, or a removable keyboard or mouse. The display unit 506 displays data held by the PC 500 or data supplied from an external source. The display unit 506 may be part of the PC 500, or it may be a separate display device. The communication unit 507 communicates with external devices such as the camera 100. The system bus 508 connects the components of the PC 500 in a communicative manner.

[0047] Here, we will explain the characteristics of the images captured with the lens unit 300 (two-lens system). In the case of the lens unit 200 (a normal single-lens system), an image that is inverted vertically and horizontally (a 180-degree rotated image) relative to the actual view is formed on the imaging unit 211. Therefore, the entire formed image is rotated 180 degrees to acquire (image) an image that matches the actual view. On the other hand, in the case of the lens unit 300 (two-lens system), the right image and the left image are each rotated 180 degrees relative to the actual view and formed on the imaging unit 211. The arrangement of the right and left images is not particularly limited, but in this embodiment, it is assumed that the right image is formed on the right side and the left image is formed on the left side on the imaging unit 211. Then, as with the lens unit 200 (a normal single-lens system), if the entire formed image (an image including the image area of ​​the right image and the image area of ​​the left image) is rotated 180 degrees, the right image and the left image can be adjusted to match the actual view, but the positions of the right and left images are swapped. In other words, the left-right positional relationship is reversed, resulting in an image where the right image is positioned on the left and the left image is positioned on the right. In this embodiment, the positions of the right and left images can be displayed in a way that corresponds to the two optical systems (right eye optical system 301R and left eye optical system 301L).

[0048] Figure 6 is a flowchart showing an example of the operation of the camera 100 in shooting mode (shooting mode processing). This operation is achieved by the system control unit 50 loading a program recorded in the non-volatile memory 219 into the system memory 218 and executing it. For example, when the camera 100 is started in shooting mode, or when the mode of the camera 100 is switched to shooting mode, the operation shown in Figure 6 begins.

[0049] In step S601, the system control unit 50 determines whether the camera 100 supports a dual-lens system (e.g., lens unit 300). For example, the system control unit 50 determines whether the firmware version of the system control unit 50 supports a dual-lens system. If it is determined that the system supports a dual-lens system, the process proceeds to step S602; otherwise, the process proceeds to step S615. In this embodiment, unlike the case of a normal single-lens system, in the case of a dual-lens system, it is necessary to acquire and record information about the dual-lens system (lens information; information about the two optical systems of the dual-lens system) for post-processing. Therefore, the processing in step S601 is necessary.

[0050] In step S602, the system control unit 50 determines whether or not the twin-lens system is attached to the camera 100. If it determines that the twin-lens system is attached, the system proceeds to step S603; otherwise, it proceeds to step S615. Note that if the twin-lens system is attached when it was not previously attached, the system also proceeds to step S603. If the twin-lens system is removed and a single lens is attached, the system proceeds to step S615.

[0051] In step S603, the system control unit 50 obtains the design values ​​of the twin lenses from the attached (connected) twin lenses. The design values ​​are design parameters and are used for left-right swapping and equirectangular conversion described later. For example, the image circle position, image circle diameter, field of view, and distortion correction coefficient shown in Figure 10(b) are obtained. The image of the subject captured by the lens is projected as a circle on the surface of the image sensor. The image circle is this circular image of the subject (image circle).

[0052] In step S604, the system control unit 50 acquires individual values ​​of the attached (connected) twin-lens system. These individual values ​​are parameters specific to the lens unit, such as manufacturing tolerances. For example, the image circle position shift, optical axis tilt, and image magnification shift shown in Figure 10(b) are acquired. By utilizing these individual values, image processing can be performed with higher accuracy than when only design values ​​are used.

[0053] In step S605, the system control unit 50 acquires an image from the imaging unit 211.

[0054] In step S606, the system control unit 50 displays the image acquired in step S605 on the EVF 217 or the display unit 108 (live view display).

[0055] In step S607, the system control unit 50 determines whether or not the user of the camera 100 has issued a recording start command. If it determines that a recording start command has been issued, the system proceeds to step S608; otherwise, it proceeds to step S614. The live view display on the EVF 217 or the display unit 108 continues until a recording start command or a recording termination command (described later in step S614) is issued.

[0056] The instruction to start recording is to fully press the shutter button 101, for example. The instruction to start recording may be to take a still image or to start recording a video. The captured image may be a JPEG still image, MP4 video, RAW still image, or RAW video. During video recording, the recorded video may be displayed on the EVF 217 or the display unit 108 in a live view display.

[0057] In step S608, the system control unit 50 acquires an image from the imaging unit 211.

[0058] In step S609, the system control unit 50 acquires shooting information such as shutter speed and aperture during shooting, as well as attitude information detected by the attitude detection unit 222 during shooting. If a RAW image is being captured, it also acquires data (parameters) necessary for development.

[0059] In step S610, the system control unit 50 stores the image data (image data) acquired in step S608 in file format on the recording medium 227 (storage medium).

[0060] In step S611, the system control unit 50 stores the information acquired in step S609 (information from camera 100) in the image file stored in step S610. As a result, the information acquired in step S609 is added as metadata to the image data acquired in step S608 within the image file.

[0061] In step S612, the system control unit 50 stores the information acquired in steps S603 and S604 (information on the two lenses) into the image file stored in step S610. As a result, the information acquired in steps S603 and S604 is added as metadata to the image data acquired in step S608 within the image file.

[0062] In step S613, the system control unit 50 tells the user of the camera 100 to end recording. The system determines whether a recording completion instruction has been given. If it is determined that a recording completion instruction has been given, the system proceeds to step S614; otherwise, it proceeds to step S608. By repeating steps S608 through S613, it is possible to record frames of moving images one after another into a video file, or to take continuous still images.

[0063] For still image shooting, the instruction to end recording is to fully press and then release the shutter button 101. If the shutter button 101 is fully pressed and then released within a predetermined time, one still image is taken. If the shutter button 101 is held fully pressed for longer than the predetermined time, continuous shooting of still images will occur. When taking a single still image, a single full press of the shutter button 101 may serve as both the instruction to start recording and the instruction to end recording. For video recording, the instruction to end recording is to fully press the shutter button 101. For example, video recording starts when the shutter button 101 is fully pressed, and ends when the shutter button 101 is fully pressed again.

[0064] In step S614, the system control unit 50 determines whether or not a termination command has been issued by the user of the camera 100. If it determines that a termination command has been issued, it terminates the operation shown in Figure 6; otherwise, it proceeds to step S605. A termination command is an instruction to turn off the power of the camera 100 or an instruction to switch the mode of the camera 100 from shooting mode to another mode. In other words, a termination command is an instruction to press the power switch 102 or the mode switching switch 103.

[0065] If a single-lens reflex lens is attached to the camera 100, the process in step S615 is performed. In step S615, the system control unit 50 performs imaging (shooting) using the single-lens reflex lens (single-lens reflex imaging process). The operation of the single-lens reflex imaging process is the same as the conventional imaging (shooting) operation in a camera with a single-lens reflex lens attached, so a detailed explanation is omitted. In this embodiment, when the system control unit 50 records the image file of the image captured by the single-lens reflex lens onto the recording medium 227, it acquires information about the attached single-lens reflex lens (such as design values ​​and individual values) from the single-lens reflex lens and stores it in the image file.

[0066] Figure 7 is a flowchart showing an example of the operation (display control) in which the PC 500 displays an image based on an image file. This operation is realized by the control unit 501 loading a program (application program) recorded in the ROM 502 into the RAM 503 and executing it. For example, when a user of the PC 500 operates the operation unit 505 (operating member) to select an image file taken by the camera from the files stored in the recording medium 504, the operation shown in Figure 7 begins. At this time, the recording medium 504 may be a recording medium 227 removed from the camera 100, etc. In this embodiment, the right image and the left image are each circular fisheye images (equidistance projection images). Below, we will explain the operation of correcting the positions of the right image and the left image while keeping them as circular fisheye images, using the information of the two lenses as correction information.

[0067] In step S701, the control unit 501 reads the image file selected by the PC 500 user from the recording medium 504. Here, not only the captured image data but also the headers and metadata attached to the image data are read. For example, the image file shown in Figure 9 is read.

[0068] In step S702, the control unit 501 determines, based on the image file read in step S701, whether or not the image file is an image file taken with a twin-lens system (using a twin-lens system). If it determines that the image file is an image file taken with a twin-lens system, the unit proceeds to step S703; otherwise, it proceeds to step S719. In step S702, for example, the unit determines whether or not the image file is an image file taken with a twin-lens system based on whether or not the image file contains information about the twin-lens system (such as design values ​​or individual values). This determination can be made based on whether or not an image file was taken with a twin-lens system, rather than on detailed information about the twin-lens system itself.

[0069] In step S703, the control unit 501 obtains (extracts) information about the two lenses (design values ​​and individual values) from the image file read in step S701.

[0070] In step S704, the control unit 501 obtains the center coordinates of the right and left images in the captured image from the design values ​​obtained in step S703. The center coordinates of the right and left images correspond to the optical axis center coordinates of the two optical systems of the binocular lens (left eye optical system 301L and right eye optical system 301R). For example, the center coordinates 804 and 808 shown in Figures 8(a) and (b) are obtained. Center coordinate 804 is the center coordinate of the right image in the left-right direction, and center coordinate 808 is the center coordinate of the left image in the left-right direction.

[0071] In step S705, the control unit 501 acquires the captured image from the image file read in step S701. If the acquired image is displayed as is, the image will be displayed with the positions of the right and left images swapped. Therefore, a process is then performed to swap the positions of the right and left images.

[0072] In step S706, the control unit 501 determines whether or not to perform a circular fisheye display. If it determines to perform a circular fisheye display, it proceeds to step S707; otherwise (to perform an equirectangular display), it proceeds to step S710. In step S706, for example, it determines whether or not to perform a circular fisheye display based on whether the radio button 1102 in Figures 11(a) and (b) is selected or deselected. In Figure 11(a), the radio button 1102 is selected, and in Figure 11(b), the radio button 1102 is deselected. If the radio button 1102 is selected, it determines to perform a circular fisheye display and proceeds to step S707. If the radio button 1102 is deselected, it proceeds to step S710. By default (for example, before displaying a screen like those shown in Figures 11(a) and (b)), it determines to perform a circular fisheye display and proceeds to step S707, so that the display starts from a circular fisheye display. The actions taken after step S710 will be described later.

[0073] In step S707, the control unit 501 swaps the positions of the right and left images in the captured image based on the center coordinates (the optical axis centers of the left eye optical system 301L and the right eye optical system 301R) obtained in step S704, and generates a processed image (left-right swap). For example, the control unit 501 identifies the region of the right image in the captured image based on the center coordinates of the right image, and identifies the region of the left image in the captured image based on the center coordinates of the left image. Then, the control unit 501 swaps the positions of the two identified regions. In this embodiment, the right and left images are arranged side by side in the captured image, and the left-right swap reverses the left-right positional relationship between the right and left images. To identify the regions of the right and left images with higher accuracy, the diameter (diameter or radius) of the right and left images may be obtained from the binocular lens information.

[0074] Note that the method of swapping left and right images is not limited to the method described above. For example, the displacement amounts 805, 806, 809, and 810 in Figure 8(a) can be obtained from the information of the two-lens system, and when swapping the positions of the right and left images, the right and left images can be positioned so that the obtained displacement amounts are maintained, and the remaining area can be filled with black or other colors. Displacement amount 805 is the distance from the left edge of the captured image to the left edge of the right image, and displacement amount 806 is the distance from the center of the captured image to the right edge of the right image. When swapping left and right images, displacement amount 805 will be the distance from the left edge of the captured image to the left edge of the left image, and displacement amount 806 will be the distance from the center of the captured image to the right edge of the left image. Similarly, displacement amount 809 is the distance from the right edge of the captured image to the right edge of the left image, and displacement amount 810 is the distance from the center of the captured image to the left edge of the left image. When swapping left and right images, Amount 809 is the distance from the right edge of the captured image to the right edge of the right image, and displacement amount 810 is the distance from the center of the captured image to the left edge of the right image.

[0075] In step S708, the control unit 501 displays the processed image generated in step S707 on the display unit 506. For example, the processed image 1101 in Figure 11(a) is displayed on the display unit 506.

[0076] In step S709, the control unit 501 determines whether or not a termination command has been issued by the user of the PC 500. If it determines that a termination command has been issued, the operation shown in Figure 7 is terminated; otherwise, the process proceeds to step S706. This allows the display on the display unit 506 (display of the captured image) to be switched between multiple displays, such as a circular fisheye display or an equirectangular display. The termination command is given using the operation unit 505 (operating member). For example, the termination command is given by pressing the termination button 1106 shown in Figures 11(a) and (b).

[0077] If an image file captured with a single-lens reflex camera is read in step S701, the process in step S719 is performed. In step S719, the control unit 501 displays the image based on the image file captured with the single-lens reflex camera on the display unit 506. The process in step S719 is the same as the conventional process for displaying an image captured with a single-lens reflex camera, so a detailed explanation is omitted.

[0078] Figures 8(a) and 8(b) are schematic diagrams of left-right reversal. Figure 8(a) shows a conventional left-right reversal that does not utilize information from the two lenses. Figure 8(b) shows the left-right reversal of this embodiment that utilizes information from the two lenses as correction information.

[0079] As shown in Figures 8(a) and (b), in image 801 before the left and right sides were swapped, the right image 803, which is a circular fisheye image, is positioned on the left, and the left image 807, which is a circular fisheye image, is positioned on the right.

[0080] In Figure 8(a), image 801 is divided into a left half and a right half at its center coordinate 802, and the left half and the right half are swapped. In other words, the left half is moved to the right of the right half. Image 811 is the image after this left-right swap.

[0081] In Figure 8(a), the displacement 806 is smaller than the displacement 805. This means that in image 801, the right image 803 is shifted from the center of the left half of image 801 towards the center of image 801. Similarly, the displacement 810 is smaller than the displacement 809. This means that in image 801, the left image 807 is shifted from the center of the right half of image 801 towards the center of image 801. Therefore, in image 811, the center coordinate 813 of the left image 807 is shifted by a distance of 814 from the center coordinate 804 in the left-right direction, and the center coordinate 816 of the right image 803 is shifted by a distance of 817 from the center coordinate 808 in the left-right direction.

[0082] In this embodiment, by using lens information, in the image 837 (Figure 8(b)) after left-right swapping, the center coordinates of the left image in the left-right direction can be made to coincide with the center coordinates 804, and the center coordinates of the right image in the left-right direction can be made to coincide with the center coordinates 808.

[0083] Figure 9 is a schematic diagram showing an example of the structure of an image file captured with a twin-lens camera. The image file in Figure 9 has a header 901, a camera information section 902, a lens information section 903, an other information section 904, and an image data section 905. The header 901 records information such as the type of image captured. The camera information section 902 records information about the camera used for shooting as metadata. For example, information such as the shutter speed and aperture at the time of shooting. Shadow information and camera orientation information during shooting are recorded. Lens information section 903 records information about the twin-lens system used for shooting as metadata. For example, the design values ​​and individual values ​​of the twin-lens system are recorded. Other information section 904 records other information as metadata. For example, in the case of video, information that changes from frame to frame is recorded. In the case of RAW images, data necessary for development is recorded. Image data section 905 records image data. In the case of video, not only image data but also audio data is recorded. Although an example has been shown in which camera information, twin-lens system information, and other information are recorded in the same image file, they may also be recorded in separate files associated with the image file.

[0084] Figure 10(a) is a schematic diagram showing an example of lens information obtained from a binocular lens. The lens information is: 1. Lens design values 2. Lens Individual Values 3. Lens Flag 4. Lens focal length 5. Includes lens temperature, etc.

[0085] Lens design values ​​are the design values ​​used for aberration correction. During the manufacturing process of a binocular lens, errors such as lens eccentricity and tilt occur in each of the two optical systems (left eye optical system 301L and right eye optical system 301R). If operations such as left / right swapping or equirectangular conversion are performed without considering these errors, the quality of the binocular VR display will deteriorate, making good stereoscopic vision difficult. Lens individual values ​​are the measurement results of errors detected during the manufacturing process of the binocular lens. Details of lens design values ​​and lens individual values ​​will be described later using Figure 10(b).

[0086] The reasons why the quality of binocular VR display deteriorates and good stereoscopic viewing becomes difficult when operations such as swapping left and right lenses or performing equirectangular conversion are performed without considering the individual characteristics (manufacturing tolerances) of the lenses are explained with reference to Figures 16(a) and 16(b). Figure 16(a) shows an ideal optical system. In Figure 16(a), the optical axes of the left and right lenses are parallel and point in the same direction. Therefore, the closer the subject is to the image sensor, the larger the parallax becomes, and good stereoscopic viewing can be achieved. Figure 16(b) shows an example of an optical system with errors from the design values. In Figure 16(b), the optical axes of the left and right lenses are not parallel and point in different directions. Therefore, parallax occurs at infinity, and good stereoscopic viewing cannot be achieved. In addition to this, good stereoscopic viewing may not be possible due to parallax occurring vertically or differences in image magnification caused by the individual characteristics of each of the left and right lenses. Therefore, in this embodiment, by correcting the captured image using the errors of the left and right lenses actually used for shooting, the image is brought closer to the image that would have been captured with an ideal optical system, thereby achieving good stereoscopic vision.

[0087] The lens flag indicates that a twin-lens system is in use and can be used to determine whether or not a twin-lens system is being used. The lens focal length is the distance from the "principal point," which is the center of the lens, to the image sensor (image formation position). The lens focal length may or may not be a common parameter for the two optical systems of a twin-lens system (left-eye optical system 301L and right-eye optical system 301R). Detailed (high-precision) lens focal lengths are necessary to perform high-quality twin-lens VR display by performing high-precision operations such as left-right swapping and equirectangular conversion. The lens temperature is the temperature of the twin-lens system and is used to understand the ambient temperature during shooting.

[0088] Figure 10(b) is a schematic diagram showing the details of the lens design values ​​and individual lens values. In this embodiment, the lens design values ​​and individual lens values ​​are used as correction information when performing left-right swapping or equirectangular transformation.

[0089] The lens design values ​​are, 1. Image circle position 2. Image circle diameter 3. Angle of view 4. Includes distortion correction coefficients, etc.

[0090] The image circle position is the coordinate of the optical axis center of the optical system in the captured image, and is provided for each of the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R). In other words, the image circle position is the coordinate of the center of the image circle (circular fisheye image) formed on the image sensor, and is provided for both the right and left images. The origin of the coordinates is, for example, the center of the image sensor (the center of the captured image). The image circle position includes horizontal and vertical coordinates. Various information regarding the optical axis center of the optical system in the captured image can be used as the image circle position. For example, the distance from a predetermined position in the image (such as the center or upper left corner) to the optical axis center can be used.

[0091] The image circle diameter is the diameter of the image circle (circular fisheye image) formed on the image sensor. The field of view is the field of view of the image circle (circular fisheye image) formed on the image sensor. The distortion correction coefficient is the ratio of the design image height to the ideal image height of the lens. A distortion correction coefficient may be set for each image height, and for image heights for which a distortion correction coefficient has not been set, the distortion correction coefficient may be calculated by interpolation using multiple distortion correction coefficients. A polynomial approximating the relationship between image height and distortion correction coefficient may also be set. The image circle diameter, field of view, and distortion correction coefficient may or may not be common parameters for the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R).

[0092] The PC500 may display a magic window on the circular fisheye image when displaying a circular fisheye image. The magic window is a display item that indicates the area to be (first) cropped for VR display. For example, the magic window is displayed based on the image circle position, image circle diameter, and field of view. This improves the display quality of the magic window. To properly display the magic window, the PC500 edits and uses the image circle position, image circle diameter, and field of view as appropriate. For example, the PC500 multiplies the image circle position and image circle diameter by a coefficient.

[0093] Lens individual values ​​are, 5. Image circle misalignment 6. Optical axis tilt 7. This includes image magnification shift, etc. This information is prepared by taking measurements for each of the two optical systems of the binocular lens (left eye optical system 301L and right eye optical system 301R).

[0094] Image circle misalignment is the deviation of the center coordinates of the image circle (circular fisheye image) formed on the image sensor from the design value. For example, image circle misalignment includes both horizontal and vertical misalignment. With the design value coordinates (a two-dimensional coordinate system including horizontal and vertical coordinates) as the origin, the horizontal misalignment is shown by the horizontal coordinates, and the vertical misalignment is shown by the vertical coordinates. Figure 17(a) shows an example of image circle misalignment. Region 1701 represents the right or left half of the sensor size screen (imaging surface). Image circle 1702 is the actual image circle of the right or left lens. Image circle 1703 is the image circle of the ideal optical system (design value). A misalignment occurs between the position of the actual image circle 1702 and the ideal image circle 1703.

[0095] Optical axis tilt is the deviation of the orientation of the optical axis on the subject side from the design value. For example, optical axis tilt includes horizontal and vertical deviations. The deviation in each direction is expressed as an angle. Figure 17(b) shows an example of horizontal deviation due to optical tilt. Line 1712 shows the actual orientation (tilt) of the horizontal optical axis of the right or left lens. Line 1713 shows the orientation of the horizontal optical axis of the ideal optical system (design value). A deviation occurs between the actual horizontal optical axis orientation 1712 and the ideal horizontal optical axis orientation 1713. Also, Figure 17(c) shows an example of vertical deviation due to optical tilt. Line 1722 shows the actual right or left lens vertical This indicates the direction of the optical axis. Line 1723 shows the direction of the horizontal optical axis of the ideal optical system (design value). There is a discrepancy between the actual horizontal optical axis direction 1722 and the ideal horizontal optical axis direction 1723.

[0096] Image magnification shift is the deviation of the size of the image circle (circular fisheye image) formed on the image sensor from the design value. This deviation is expressed, for example, as a ratio to the design value. Figure 17(d) shows an example of image magnification shift. Image circle 1732 is the image circle formed on the image sensor through the actual right or left lens. Image circle 1733 is the image circle formed on the image sensor through the ideal optical system (design value). A deviation occurs between the actual image circle 1732 and the image circle 1733 of the ideal optical system.

[0097] The information included in the lens information is not limited to the information described above. For example, the lens information may include the boundary positions of the right and left images in the captured image (the position of the edge of the circular fisheye image; the position indicated by the displacement amounts 805, 806, 809, 810, etc.). The lens information may also include the midpoint coordinates between the right and left images in the captured image. In many cases, the midpoint coordinates coincide with the center coordinates of the captured image. The lens information may also include information indicating the area of ​​the magic window (for example, the coordinates of the upper left corner of the magic window, the width of the magic window, and the height of the magic window). The lens information may also include correction values ​​to improve the accuracy of left-right swapping and equirectangular transformation (for example, correction values ​​obtained by calibration of a binocular lens).

[0098] Figure 10(c) is a schematic diagram showing an example of camera information generated within the camera. For example, camera information is used to perform high-quality VR display. Camera information is, 1. Camera recording area information 2. Camera-internal accelerometer information 3. Includes right exposure compensation information, etc.

[0099] Camera recording area information is information about the effective image area. The displayable effective image area differs depending on the camera's sensor and recording mode. The PC500 uses camera recording area information to provide a more accurate display. Camera accelerometer information is attitude information obtained using the camera's accelerometer (level), and represents the camera's attitude in the roll and pitch directions. The PC500 uses camera accelerometer information to understand the camera's attitude during shooting. Based on the understood attitude, the PC500 performs electronic image stabilization and horizontal correction (zenith correction to bring the vertical direction of the display closer to the vertical direction of real space). Right exposure correction information is the exposure setting value that brings the exposure of the right image closer to the exposure of the left image. The PC500 uses right exposure correction information to provide a natural (or reduced-discomfort) two-lens VR display.

[0100] Figure 11(a) is a schematic diagram showing an example of the display after left-right reversal (display in step S708) of the application screen displayed on the display unit 506 by the control unit 501. Screen 1100 is the application screen. Screen 1100 in Figure 11(a) consists of a processing image 1101, radio buttons 1102, 1103, a checkbox 1104, and a save button 1105. , including the Exit button 1106. The processed image 1101 is the image after left-right swapping, i.e., the processed image generated in step S707. Radio button 1102 is the radio button selected when performing circular fisheye display, and radio button 1103 is the radio button selected when performing equirectangular display. When radio button 1102 is selected, radio button 1103 is deselected, and when radio button 1102 is deselected, radio button 1103 is selected. Check box 1104 and save button 1105 are display items used when performing equirectangular display. When radio button 1102 is selected, check box 1104 and save button 1105 are displayed grayed out or otherwise rendered inoperable. The uses of check box 1104 and save button 1105 will be described later. The Exit button 1106 is the button to exit the application on screen 1100.

[0101] Next, we will explain the case of performing equirectangular display. If it is determined in step S706 of Figure 7 that the circular fisheye display is not selected and the equirectangular display is selected, the process proceeds to step S710, and the image is displayed in equirectangular display mode. For example, suppose that among the radio buttons 1102 and 1103 shown in Figures 11(a) and (b), radio button 1103 is selected, and it is determined that the circular fisheye display is not selected and the equirectangular display is selected.

[0102] In step S710, the control unit 501 of the PC500 determines whether or not to perform adjustment using the individual values ​​of the two lenses (individual value adjustment). If it is determined that individual value adjustment should be performed, the process proceeds to step S716; otherwise, it proceeds to step S711. In step S710, for example, the decision to perform individual value adjustment is made based on whether or not checkbox 1104 in Figure 11(b) is checked. If checkbox 1104 is checked, it is determined that individual value adjustment should be performed, and the process proceeds to step S716. If checkbox 1104 is not checked, the process proceeds to step S711. The operation after proceeding to step S716 will be described later.

[0103] In step S711, the control unit 501 generates a map for equirectangular transformation based on the center coordinates (the optical axis centers of the left eye optical system 301L and the right eye optical system 301R) obtained in step S704. Equirectangular transformation is a transformation process that treats a circular fisheye image as a sphere, similar to an equirectangular projection in maps, and transforms it so that the lines of latitude (horizontal lines) and lines of longitude (vertical lines) intersect at right angles. Through equirectangular transformation, the circular circular fisheye image is transformed into a rectangular equirectangular image. The map shows which position in the original image corresponds to each pixel after transformation. In this embodiment, a map for equirectangular transformation is generated not only to transform a circular fisheye image into an equirectangular image, but also to correct the positions of the right and left images. In step S711, the regions of the right and left images in the captured image are identified in the same way as in step S707. Then, a map is generated based on the two identified regions.

[0104] In step S712, the control unit 501 performs an equirectangular transformation using the map generated in step S711 to generate a processed image. Although left-right swapping is included as part of the equirectangular transformation, left-right swapping may be performed separately from the equirectangular transformation.

[0105] In step S713, the control unit 501 displays the processed image generated in step S712 or step S718 (described later in another embodiment) on the display unit 506. For example, the processed image 1111 shown in Figure 11(b) is displayed on the display unit 506.

[0106] In step S714, the control unit 501 determines whether or not to save the processed image (image after equirectangular transformation) displayed in step S713. If it determines to save the processed image, it proceeds to step S715; otherwise, it proceeds to step S709. In step S714, for example, based on whether or not the save button 1105 in Figure 11(b) was pressed, Determine whether or not to save the processed image. If the save button 1105 is pressed, determine to save the processed image and proceed to step S715. If the save button 1105 is not pressed, proceed to step S709.

[0107] In step S715, the control unit 501 saves the image file of the processed image (image after equirectangular transformation) displayed in step S713 to the recording medium 504.

[0108] Figure 12 is a schematic diagram showing the equirectangular transformation of this embodiment. As shown in Figure 12, in image 1201 before equirectangular transformation, the right image 1202, which is a circular fisheye image, is located on the left side, and the left image 1205, which is a circular fisheye image, is located on the right side. Image 1208 is the image after equirectangular transformation and includes equirectangular images 1209 and 1210. In this embodiment, an equirectangular transformation map is generated such that the correspondences shown by arrows 1211 and 1212 are made. In the map of this embodiment, each pixel of the equirectangular image 1209 located on the left side is associated with each position in the left image 1205 located on the right side, and each pixel of the equirectangular image 1210 located on the right side is associated with a position in the right image 1202 located on the left side. By using such a map, the left image 1205 located on the right side is transformed into the equirectangular image 1209 located on the left side, and the right image 1202 located on the left side is transformed into the equirectangular image 1210 located on the right side. In other words, not only is the circular fisheye image converted to an equirectangular image, but the positions of the right and left images are also swapped. This allows the relative positions of the right and left images to match the relative positions of the two optical systems (right eye optical system 301R and left eye optical system 301L).

[0109] Figure 11(b) is a schematic diagram showing an example of the display after equirectangular transformation (display in step S713) on the application screen displayed by the control unit 501 on the display unit 506. In Figure 11(a), the processed image 1101, which is the image after left-right swapping, was displayed on screen 1100 (application screen). In Figure 11(b), the processed image 1111 (the processed image generated in step S712 or step S718), which is the image after equirectangular transformation, is displayed on screen 1100. Also, in Figure 11(b), the grayed-out checkbox 1104 and the save button 1105 have been removed, and the checkbox 1104 and the save button 1105 are now operable. Checkbox 1104 is checked when performing individual value adjustment. The save button 1105 is pressed when saving the image file of processed image 1111. Since checkbox 1104 is not checked, the processed image generated in step S712 is displayed as processed image 1111.

[0110] Next, we will explain an example of adjusting individual values ​​when performing equirectangular display. In other words, we will explain the operation from step S716 in Figure 7. Although a detailed explanation will be omitted, the individual values ​​of the two lenses may also be used when performing circular fisheye display.

[0111] In step S716, the control unit 501 adjusts the design value acquired in step S703 based on the individual value acquired in step S703. For example, it adjusts the image circle position (the center coordinates of the right and left images in the captured image) based on the image circle position shift in Figure 10(b). If the individual value is the difference from the design value, the individual value is added to the design value. If the individual value is the same absolute value as the design value, the design value is replaced with the individual value.

[0112] In step S717, the control unit 501 generates a map for equirectangular transformation based on the adjusted center coordinates (the optical axis centers of the left eye optical system 301L and the right eye optical system 301R) from step S716. Figure 18 shows an example of adjusting the spatial image position for each pixel based on the adjusted center coordinates (correction process to correct the pixel position). For the left image, the spatial image position is adjusted from image position 1801 to image position 1802 for each pixel. The image is adjusted. For the right image, the image position is adjusted pixel by pixel so that the spatial coordinate image position is adjusted from 1803 to 1804. The method for generating the map is the same as in step S711. By using the adjusted center coordinates, a more accurate equirectangular transformation becomes possible.

[0113] In step S718, the control unit 501 performs an equirectangular transformation using the map generated in step S717 to generate a processed image. The method for generating the processed image is the same as in step S712. The operation from step S713 onwards is as described above. As described above, by correcting the design values ​​of the lens information based on the individual values ​​of the two lenses, it becomes possible to convert a circular fisheye image into an equirectangular image with greater accuracy.

[0114] According to the information processing device of this embodiment, when displaying an image based on an image file captured by a binocular lens, it is possible to correct the centers of the left and right images obtained through each optical system of the binocular lens and then appropriately rearrange the left and right images. This makes it possible to display a binocular image that is less likely to cause discomfort.

[0115] <Second Embodiment> A second embodiment of the present invention will now be described. In this embodiment, the camera 100 and the PC 500 are connected to each other in a way that allows them to communicate with each other, and a live view image captured by the camera 100 is transmitted to the PC 500, and the PC 500 displays the live view image on the display unit 506.

[0116] Figure 13 is a flowchart illustrating an example of the operation of camera 100. This operation is achieved by the system control unit 50 loading a program recorded in the non-volatile memory 219 into the system memory 218 and executing it. For example, when camera 100 is started up, the operation shown in Figure 13 begins. The operation shown in Figure 13 is for the function (PC live view) that displays the live view image captured by the camera on the PC's display unit. The operation shown in Figure 13 is performed when camera 100 is in a shooting standby state. If a recording start command is input from PC 500 during PC live view operation, still image shooting or video shooting is performed. At this time, PC live view may continue. Control of shooting using the twin-lens system is performed by executing the processes from steps S608 to S613 in Figure 6, and the explanation is omitted.

[0117] In step S1301, the system control unit 50 determines whether the camera 100 is compatible with a dual-lens system (e.g., lens unit 300). If it determines that it is compatible with a dual-lens system, the system proceeds to step S1302; otherwise, it proceeds to step S1311. The process in step S1301 is the same as the process in step S601 in Figure 6.

[0118] In step S1302, the system control unit 50 determines whether or not the twin-lens system is attached to the camera 100. If it determines that the twin-lens system is attached, it proceeds to step S1303; otherwise, it proceeds to step S1311. The process in step S1302 is the same as the process in step S602 in Figure 6.

[0119] In step S1303, the system control unit 50 obtains the design value of the twin lenses from the attached (connected) twin lenses. In step S1304, the system control unit 50 obtains the individual value of the twin lenses from the attached (connected) twin lenses. The processing in steps S1303 and S1304 is the same as the processing in steps S603 and S604.

[0120] In step S1305, the camera 100 is connected to the PC 500, and the system control unit 50 detects that the camera 100 has been connected to the PC 500. In step S1306, The system control unit 50 receives a PC live view start request from the PC 500. In step S1307, the system control unit 50 receives a live view image request from the PC 500. The live view image request includes information (resolution information) that specifies the resolution of the live view image, as will be described later. The system control unit 50 executes the process in step S1309 to send a live view image of the specified resolution to the PC 500.

[0121] In step S1308, the system control unit 50 converts the information acquired in steps S1303 and S1304 (lens information of the twin-lens system) to match the coordinate system of the live view image to be transmitted. Because the resolution and other properties of the captured image (the image recorded in the image file) and the live view image are different, the information acquired in steps S1303 and S1304 cannot be used directly for image processing of the live view image. Therefore, in this embodiment, the lens information is converted into information that matches the coordinate system of the live view image.

[0122] In step S1309, the system control unit 50 transmits the lens information converted in step S1308 and the live view image to the PC 500. Based on the resolution information acquired in step S1307, the system control unit 50 converts the resolution of the live view image and transmits it to the PC 500. In this embodiment, the system control unit 50 of the camera 100 performs the conversion of the lens information, but the control unit 501 of the PC 500 may also perform the conversion of the lens information. In that case, the lens information before conversion and the parameters necessary for the conversion of the lens information are transmitted to the PC 500.

[0123] In step S1310, the system control unit 50 determines whether or not to terminate the PC live view. For example, it determines to terminate the PC live view if the connection between camera 100 and PC 500 is disconnected, or if the user instructs camera 100 or PC 500 to terminate the PC live view. If it determines to terminate the PC live view, it terminates the operation shown in Figure 13; otherwise, it proceeds to step S1307.

[0124] If a single-lens reflex lens is attached to the camera 100, the process in step S1311 is performed. In step S1311, the system control unit 50 transmits the live view image captured by the single-lens reflex lens to the PC 500. The process in step S1311 is the same as the conventional process of transmitting a live view image captured by a single-lens reflex lens to an external device, so a detailed explanation is omitted. In this embodiment, when the system control unit 50 transmits the live view image captured by the single-lens reflex lens to the PC 500, it does not acquire information about the attached single-lens reflex lens (such as design values ​​or individual values) from the lens, nor transmit it to the PC 500.

[0125] Figure 14 is a flowchart illustrating an example of PC500 operation. This operation is achieved by the control unit 501 loading a program (application program) recorded in ROM 502 into RAM 503 and executing it. For example, when a user instructs PC500 to launch a specific application, the operation shown in Figure 14 begins. The operation shown in Figure 14 is for the function (PC Live View) that displays live view images captured by the camera on the PC's display unit.

[0126] In step S1401, a camera (for example, camera 100) is connected to the PC 500, and the control unit 501 detects that a camera has been connected to the PC 500.

[0127] In step S1402, the control unit 501 determines whether the camera connected in step S1401 is a camera compatible with a dual-lens system (e.g., lens unit 300). For example, the control unit 501 obtains the camera model information from the connected camera and determines whether the camera is compatible with a dual-lens system based on the obtained model information. If it is determined that the camera is compatible with a dual-lens system, the process proceeds to step S1403; otherwise, it proceeds to step S Proceed to 1421. A camera compatible with dual lenses is, for example, a camera that can be fitted with two lenses.

[0128] In step S1403, the control unit 501 determines whether the firmware of the camera connected in step S1401 is compatible with dual-lens systems. For example, the control unit 501 obtains information about the firmware version of the connected camera from the connected camera and, based on the obtained information, determines whether the firmware version of the connected camera is compatible with dual-lens systems. If it is determined that it is compatible with dual-lens systems, the process proceeds to step S1404; otherwise, the process proceeds to step S1421.

[0129] Even if a camera capable of supporting dual lenses is connected to the PC500, the connected camera may not be able to support dual lenses due to reasons such as an outdated firmware version. For this reason, the process in step S1403 is necessary. In addition, various cameras can be connected to the PC500, and cameras that do not support dual lenses may be connected regardless of the firmware version. For this reason, the process in step S1402 is necessary before the process in step S1403.

[0130] In step S1404, the control unit 501 determines whether or not a twin-lens system is attached to the camera connected in step S1401. If it determines that a twin-lens system is attached, the process proceeds to step S1405; otherwise, it proceeds to step S1421.

[0131] In step S1405, the control unit 501 sends a PC live view start request to the camera connected in step S1401.

[0132] In step S1406, the control unit 501 determines whether or not to perform a circular fisheye display. If it determines to perform a circular fisheye display, it proceeds to step S1407; otherwise (to perform an equirectangular display), it proceeds to step S1414. In step S1406, for example, it determines whether or not to perform a circular fisheye display based on whether the radio button 1505 in Figures 15(a) to (d) is selected or not. In Figures 15(a) and (c), the radio button 1505 is selected, and in Figures 15(b) and (d), the radio button 1505 is not selected. If the radio button 1505 is selected, it is determined to perform a circular fisheye display and the process proceeds to step S1407. If the radio button 1505 is not selected, the process proceeds to step S1414.

[0133] In step S1407, the control unit 501 sends a live view image request to the camera connected in step S1401. In this embodiment, the live view image request in step S1407 is a request for a live view image at normal resolution. The normal resolution is, for example, 4K resolution.

[0134] In step S1408, the control unit 501 receives a live view image captured by the camera connected in step S1401, along with lens information of the twin-lens system mounted on the camera. The resolution of the live view image received in step S1408 is the normal resolution. The lens information received in step S1408 is information that has been converted to match the received live view image (for example, the lens information converted in step S1308 in Figure 13).

[0135] In step S1409, the control unit 501 determines whether or not to swap the left and right sides. If it determines to swap the left and right sides, it proceeds to step S1410; otherwise, it proceeds to step S1412. In step S1409, for example, it determines whether or not to swap the left and right sides based on whether or not checkbox 1507 in Figures 15(a) and (c) is checked. If checkbox 1507 is checked, the left and right sides are swapped. If it is determined that this is the case, proceed to step S1410. If checkbox 1507 is not checked, proceed to step S1412.

[0136] In step S1410, the control unit 501 swaps the positions of the right and left images in the live view image acquired in step S1408 based on the lens information acquired in step S1408, and generates a processed live view image (left-right swap). The method of left-right swapping is the same as in step S707 in Figure 7. Based on the center coordinates (the optical axis centers of the left eye optical system 301L and the right eye optical system 301R) included in the lens information received along with the live view image, the control unit 501 swaps the positions of the right and left images in the live view image and generates a processed live view image.

[0137] In step S1411, the control unit 501 displays the processed live view image generated in step S1410 on the display unit 506.

[0138] In step S1412, the control unit 501 displays the live view image acquired in step S1408 on the display unit 506.

[0139] In step S1413, the control unit 501 determines whether or not to terminate the PC live view. For example, it determines to terminate the PC live view if the connection between the camera 100 and the PC 500 is disconnected, or if the user instructs the camera 100 or PC 500 to terminate the PC live view. The instruction to terminate the PC live view is, for example, by pressing the termination button 1508 shown in Figures 15(a) to (d). If it determines to terminate the PC live view, the operation in Figure 14 is terminated; otherwise, the process proceeds to step S1406.

[0140] As described above, when performing equirectangular display, the process proceeds from step S1406 to step S1414. In step S1414, the control unit 501 sends a live view image request to the camera connected in step S1401. In this embodiment, the live view image request in step S1414 is a request for a low-resolution live view image (a resolution lower than the normal resolution). When performing equirectangular display, equirectangular conversion (conversion from a circular fisheye image to an equirectangular image) is required, and the higher the resolution of the image to which equirectangular conversion is applied, the longer the time required for equirectangular conversion and the greater the delay caused by the equirectangular conversion. In this embodiment, a low-resolution live view image is requested in order to speed up the equirectangular conversion (reduce the time required for equirectangular conversion). However, if the delay caused by equirectangular conversion is within an acceptable range, a normal-resolution live view image may be requested even when performing equirectangular display.

[0141] In step S1415, the control unit 501 receives a live view image captured by the camera and lens information of the twin-lens system attached to the camera from the camera connected in step S1401. The resolution of the live view image received in step S1415 is low resolution. The lens information received in step S1415 is information that has been converted to match the received live view image (for example, the lens information converted in step S1308 in Figure 13).

[0142] In step S1416, the control unit 501 determines whether or not to swap the left and right sides. If it determines to swap the left and right sides, it proceeds to step S1417; otherwise, it proceeds to step S1419. In step S1416, for example, it determines whether or not to swap the left and right sides based on whether or not checkbox 1507 in Figures 15(b) and (d) is checked. If checkbox 1507 is checked, it determines to swap the left and right sides and proceeds to step S1417. If checkbox 1507 is not checked, it proceeds to step S1419.

[0143] In step S1417, the control unit 501 swaps the positions of the right and left images in the live view image acquired in step S1415 based on the lens information acquired in step S1415, and converts the right and left images into equirectangular images. In other words, the control unit 501 performs left-right swapping and equirectangular conversion to generate a processed live view image. The conversion to equirectangular display including left-right swapping is the same as in step S712. The conversion to equirectangular display including left-right swapping may also be the same as in steps S716 to S718. In other words, the control unit 501 may generate a map based on lens design values ​​corrected using individual values ​​included in the lens information received together with the live view image, and perform the conversion to equirectangular display including left-right swapping based on this map. Similar to Embodiment 1, left-right swapping may be performed as part of the equirectangular conversion, or they may be performed as separate processes.

[0144] In step S1418, the control unit 501 displays the processed live view image generated in step S1417 on the display unit 506.

[0145] In step S1419, the control unit 501 converts the right and left images into equirectangular images without swapping their positions in the live view image acquired in step S1415. In other words, the control unit 501 generates a processed live view image by performing equirectangular conversion without swapping the left and right sides.

[0146] In step S1420, the control unit 501 displays the processed live view image generated in step S1419 on the display unit 506.

[0147] If camera 100 is not compatible with a dual-lens system, or if camera 100 is equipped with a single-lens system, the process in step S1421 is performed. In step S1421, the control unit 501 displays the live view image captured by the single-lens system on the display unit 506. The process in step S1421 is the same as the conventional process of displaying a live view image captured by a single-lens system on a PC or the like, so a detailed explanation is omitted.

[0148] In steps S1410, S1417, and S1419, the control unit 501 performs image processing on the live view image acquired from the connected camera. In step S1413, which follows steps S1410, S1417, and S1419, the control unit 501 determines whether or not to terminate the PC live view. If the PC live view is to continue, processing returns to step S1406, which precedes steps S1410, S1417, and S1419. Therefore, in the operation shown in Figure 14, one of the image processing steps S1410, S1417, or S1419 may be executed repeatedly.

[0149] Therefore, in order to speed up image processing, the control unit 501 may record information related to the executed image processing in the RAM 503 and use it for subsequent image processing. For example, the control unit 501 records the correspondence between pixels before image processing and pixels after image processing (image processing map). The image processing map can be used continuously as long as there are no changes in the resolution of the live view image or lens information. When the control unit 501 executes any of the image processing steps S1410, S1417, or S1419, it records the image processing map for that image processing. Then, when the control unit 501 executes the same image processing again, it uses the recorded image processing map to perform the image processing. In this way, image processing can be sped up.

[0150] Figures 15(a) to (d) are schematic diagrams showing an example of the display (PC Live View display) of the application screen displayed on the display unit 506 by the control unit 501. Screen 1500 is This is the application screen (remote live view screen). Screen 1500 includes the live view display area 1501, the guide display area 1502, the guide display area 1503, the operation area 1504, and the exit button 1508.

[0151] The live view display area 1501 is the area that displays the live view image. The live view display area 1501 consists of the left display area 1501A and the right display area 1501B. The guide display area 1502 is the area that displays a string indicating which of the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R) the image displayed in the left display area 1501A is the image. The guide display area 1503 is the area that displays a string indicating which of the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R) the image displayed in the right display area 1501B is the image. The operation area 1504 is the area for receiving operations related to PC live view, and the operation area 1504 displays radio buttons 1505, 1506 and a checkbox 1507. Radio button 1505 is selected when performing circular fisheye display, and radio button 1506 is selected when performing equirectangular display. When radio button 1505 is selected, radio button 1506 is deselected, and when radio button 1505 is deselected, radio button 1506 is selected. Check box 1507 is checked when swapping left and right images. When check box 1507 is operated, the positions of the right image (right eye image) and left image (left eye image) in the live view image are swapped, and the text displayed in the guide display areas 1502 and 1503 is also swapped. Exit button 1508 is a button to exit PC live view.

[0152] In Figure 15(a), radio button 1505 for circular fisheye display is selected. The checkbox 1507 for swapping left and right is not checked. Therefore, the live view image acquired from the camera is displayed as is in the live view display area 1501. Specifically, the right eye image, which is a circular fisheye image, is displayed in the left display area 1501A, and the left eye image, which is a circular fisheye image, is displayed in the right display area 1501B.

[0153] In Figure 15(b), radio button 1506 for equirectangular display is selected. Checkbox 1507 for swapping left and right is not checked. Therefore, the right eye image and left eye image in the live view image acquired from the camera are not displayed. Remo Each of the circular fisheye images is converted to an equirectangular image (without left-right swapping). Then, the live view image after the equirectangular conversion is displayed in the live view display area 1501. Specifically, The right-eye image, which is an equirectangular image, is displayed in the left display area 1501A, and the left-eye image, which is an equirectangular image, is displayed in the right display area 1501B.

[0154] In Figure 15(c), radio button 1505 for circular fisheye display is selected, and checkbox 1507 for swapping left and right images is checked. As a result, the positions of the right-eye and left-eye images in the live view image acquired from the camera are swapped. The swapped live view image is then displayed in the live view display area 1501. Specifically, the left-eye image, which is a circular fisheye image, is displayed in the left display area 1501A, and the right-eye image, which is a circular fisheye image, is displayed in the right display area 1501B.

[0155] In Figure 15(d), radio button 1506 for equirectangular display is selected, and checkbox 1507 for swapping left and right is checked. As a result, the positions of the right eye image and the left eye image in the live view image acquired from the camera are swapped, and the right eye image and the left eye image (izumi Remo Each of the circular fisheye images is converted into an equirectangular image. The live view image after the left-right swap and equirectangular conversion is then displayed in the live view display area 1501. Specifically, the left eye image, which is an equirectangular image, is displayed in the left display area 1501A, and the right eye image, which is an equirectangular image, is displayed in the right display area 1501B.

[0156] Furthermore, the various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device. Similarly, the various controls described above as being performed by the control unit 501 may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0157] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0158] Furthermore, the present invention is not limited to cameras and PCs, but can be applied to any electronic device capable of handling two images with parallax. For example, the present invention can be applied to PDAs, mobile phone terminals and portable image viewers, printers, digital photo frames, music players, game consoles, e-book readers, etc. The present invention can also be applied to video players, display devices (including projection devices), tablet terminals, smartphones, AI speakers, home appliances, and in-vehicle devices.

[0159] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0160] 100: Digital camera (camera) 50: System control unit 500: Personal computer (PC) 501: Control unit

Claims

1. An acquisition means for acquiring an image comprising: a first image region corresponding to a first optical image input via a first optical system on the right side of the lens unit; a second image region corresponding to a second optical image input via a second optical system on the left side of the lens unit having a predetermined parallax with respect to the first optical system; a first parameter which is a design parameter of the lens unit; and a parameter specific to the lens unit. An adjustment means for adjusting the first parameter to a second parameter based on the parameters specific to the lens unit, A generation means generates a correspondence between pixels before and after the transformation of the first image region and the second image region based on the second parameter, and uses the correspondence to perform a process to transform the first image region and the second image region to generate a processed image. It has, The process for transforming the first image region and the second image region includes a swapping process for swapping the positions of the first image region and the second image region in the image. In the image before swapping the positions of the first and second image regions, when the top of the imaging range is considered the top of the image, the first image region is positioned on the left and the second image region is positioned on the right. An information processing device characterized by the following:

2. The first parameter includes information regarding the optical axis center of the first optical system in the image, and information regarding the optical axis center of the second optical system in the image. The information processing apparatus according to feature 1.

3. In the swapping process, a first image region is identified in the image based on the optical axis center of the first optical system in the image, a second image region is identified in the image based on the optical axis center of the second optical system in the image, and the positions of the two identified regions are swapped. The information processing apparatus according to claim 1 or 2.

4. The correspondence relationship generated by the generation means includes: In the processed image, the left-right positional relationship between the first image region and the second image region in the image is reversed. In the processed image, the region of the circular fisheye image is transformed into the region of the equirectangular image, Includes The information processing apparatus according to claim 1 or 2.

5. The aforementioned image is a side-by-side image in which the first image region and the second image region are arranged side by side, In the swapping process, the left-right positional relationship between the first image region and the second image region in the image is reversed. The information processing apparatus according to any one of claims 1 to 4.

6. The first optical system and the second optical system are, respectively, fisheye lenses. The information processing apparatus according to any one of claims 1 to 5, characterized in that the first image region and the second image region are each regions of a circular fisheye image.

7. The generation means converts the first image region and the second image region, respectively, from the region of a circular fisheye image to the region of an equirectangular image. The information processing apparatus according to feature 6.

8. The acquisition means acquires an image file in which the first parameter and the lens unit-specific parameter are added as metadata to the image data. The information processing apparatus according to any one of claims 1 to 7.

9. The aforementioned image is a live view image output from an imaging device connectable to a lens unit including the first optical system and the second optical system, The first parameter and the lens unit-specific parameter are included in the lens information acquired by the imaging device from the lens unit. The information processing apparatus according to any one of claims 1 to 8.

10. The aforementioned image is an image captured by an imaging device that can be connected to a lens unit including the first optical system and the second optical system, The first parameter and the lens unit-specific parameter are included in the lens information acquired by the imaging device from the lens unit and added as metadata to the image data. The information processing apparatus according to any one of claims 1 to 9.

11. Steps to obtain: an image including a first image region corresponding to a first optical image input via a first optical system on the right side of the lens unit, and a second image region corresponding to a second optical image input via a second optical system on the left side of the lens unit having a predetermined parallax with respect to the first optical system; a first parameter which is a design parameter of the lens unit; and a parameter specific to the lens unit. The steps include adjusting the first parameter to the second parameter based on the parameters specific to the lens unit, Based on the second parameter, a correspondence between pixels before and after the transformation of the first image region and the second image region is generated, and using the correspondence, a process is executed to transform the first image region and the second image region to generate a processed image. It has, The process of transforming the first image region and the second image region is performed in the image This includes a swapping process that swaps the positions of the first image region and the second image region. In the image before swapping the positions of the first and second image regions, when the top of the imaging range is considered the top of the image, the first image region is positioned on the left and the second image region is positioned on the right. A control method for an information processing device characterized by the following features.

12. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 10.

Images