Information Processing Apparatus, Control Method, and Computer-Readable Storage Medium

Abstract

The present invention provides an information processing apparatus, a control method, and a computer-readable storage medium. The information processing apparatus according to the present disclosure includes: an acquisition unit configured to acquire an image and correction information regarding a first optical system and a second optical system, the image including a first image region corresponding to a first optical image input via the first optical system and a second image region corresponding to a second optical image input via the second optical system, the second optical system having a predetermined parallax with respect to the first optical system; a correction unit configured to perform correction processing that corrects positions of pixels included in the first image region and positions of pixels included in the second image region in the image based on the correction information; and a generation unit configured to generate a processed image by performing processing that transforms the corrected first image region and the corrected second image region.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a control method, and a computer-readable storage medium. Background Art

[0002] In known techniques, two images with parallax are captured by two cameras and are stereoscopically displayed. Japanese Unexamined Patent Application Publication No. 2013-141052 discloses a camera that includes a lens unit having two optical systems and can capture two images with parallax at once.

[0003] However, when an image (including an image of two images with parallax) captured using a lens unit having two optical systems is displayed as a conventional image, the positional relationship between the two optical systems and the positional relationship between the two images in the image are inverted with respect to each other. Summary of the Invention

[0004] The present disclosure provides a technique that enables a display in which the positions of two images captured by a lens unit having two optical systems are aligned with the two optical systems.

[0005] An information processing apparatus according to the present disclosure includes: an acquisition unit configured to acquire an image and correction information regarding a first optical system and a second optical system, the image including a first image region and a second image region, the first image region corresponding to a first optical image input via the first optical system, the second image region corresponding to a second optical image input via the second optical system, the second optical system having a predetermined parallax with respect to the first optical system; a correction unit configured to perform correction processing that corrects the positions of pixels included in the first image region and the positions of pixels included in the second image region in the image based on the correction information; and a generation unit configured to generate a processed image by performing transformation processing on the corrected first image region and the corrected second image region.

[0006] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0007] Figure 1A and Figure 1B are schematic diagrams showing the overall configuration of the system;

[0008] Figure 2A and Figure 2B are external views of the camera;

[0009] Figure 3 is a block diagram showing the configuration of the camera;

[0010] Figure 4 is a schematic diagram showing the structure of the lens unit;

[0011] Figure 5 is a block diagram showing the structure of the PC;

[0012] Figure 6 is a flowchart showing the operation of the camera;

[0013] Figure 7 is a flowchart showing the operation of the PC;

[0014] Figure 8A and Figure 8B is a schematic diagram showing the position exchange of the left and right images;

[0015] Figure 9 is a schematic diagram showing the structure of the image file;

[0016] Figure 10A is a schematic diagram representing lens information, Figure 10B is a schematic diagram representing the lens design value and the individual lens value, while Figure 10C is a schematic diagram representing camera information;

[0017] Figure 11A and Figure 11B is a schematic diagram of the display screen;

[0018] Figure 12 is a schematic diagram showing the equidistant cylindrical transformation;

[0019] Figure 13 is a flowchart showing the operation of the camera;

[0020] Figure 14 is a flowchart showing the operation of the PC;

[0021] Figures 15A to 15D is a schematic diagram of the display screen;

[0022] Figure 16A and Figure 16B is a diagram showing the difference between the left and right lenses;

[0023] Figures 17A to 17D is a diagram showing the displacement caused by the error of the lens; and

[0024] Figure 18 is a diagram showing the correction of the image position in the spatial coordinates. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] Embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0026] First Embodiment

[0027] The first embodiment of the present disclosure will be described below. Figure 1A and Figure 1B are schematic views showing an example of the overall configuration of the system according to the present embodiment. The system according to the present embodiment includes a digital camera (camera) 100 and a personal computer (PC) 500. A lens unit 300 is attached (connected) to the camera 100. The lens unit 300 will be specifically described later. Attaching the lens unit 300 enables the camera 100 to capture two images (still images or moving images) at a time with a predetermined parallax. The PC 500 is an information processing device that processes information of images captured by an imaging device such as the camera 100. Figure 1A Shows a configuration in which the camera 100 and the PC 500 are connected to communicate with each other via wireless communication or wired communication. Figure 1B Shows a configuration in which images and the like captured by the camera 100 are basically input to the PC 500 via an external storage device. The external storage device may or may not be connected to both the camera 100 and the PC 500. For example, the external storage device may be connected to the camera 100, and image files captured by the camera 100 may be stored in the external storage device. Thereafter, the external storage device can be detached from the camera 100 and connected to the PC 500, and the files stored in the external storage device can be imported into the PC 500.

[0028] Figure 2A and Figure 2B are external views showing an example of the appearance of the camera 100. Figure 2A is a perspective view of the camera 100 observed from the front. Figure 2B is a perspective view of the camera 100 observed from the back.

[0029] The camera 100 includes a shutter button 101, a power switch 102, a mode change switch 103, a main electronic dial 104, a sub-electronic dial 105, a moving image button 106, and a viewfinder external display unit 107 on the top surface. The shutter button 101 is an operation member for providing a photography preparation instruction or a photography instruction. The power switch 102 is an operation member for turning on or off the power of the camera 100. The mode change switch 103 is an operation member for switching various modes. The main electronic dial 104 is a rotary operation member for changing setting values such as shutter speed and aperture. The sub-electronic dial 105 is a rotary operation member for moving a selection box (cursor) or switching images. The moving image button 106 is an operation member for providing an instruction to start or stop photographing (recording) a moving image. The viewfinder external display unit 107 displays various setting values such as shutter speed and aperture.

[0030] The camera 100 includes a display unit 108, a touch panel 109, arrow keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a playback button 114, and a menu button 115, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar 119 on the back surface. The display unit 108 displays images and various information. The touch panel 109 is an operation member for detecting touches on the display screen (touch screen) of the display unit 108. The arrow keys 110 are operation members including keys (cross keys) that can be pressed up, down, left, and right. Processing can be performed according to the pressed position of the arrow keys 110. The SET button 111 is an operation member to be pressed mainly when determining a selected item. The AE lock button 112 is an operation member to be pressed when fixing the exposure state in the shooting standby state. The zoom button 113 is an operation member for turning on or off the zoom mode in the live view display (LV display) in the shooting mode. When the zoom mode is turned on, the main electronic dial 104 is operated to zoom in or out the live view image (LV image). The zoom button 113 is used when zooming in on a played-back image or increasing the magnification in the playback mode. The playback button 114 is an operation member for switching between the shooting mode and the playback mode. By pressing the playback button 114 in the shooting mode, the mode is switched to the playback mode, and thus the latest image recorded in a recording medium 227 described later can be displayed on the display unit 108.

[0031] The menu button 115 is an operation member to be pressed to display a menu screen on the display unit 108 that enables various settings. The user can intuitively perform various settings by using the arrow keys 110, the SET button 111, and the menu screen displayed on the display unit 108. The eyepiece unit 116 is a component for viewing at eye level through an eyepiece viewfinder (peephole viewfinder) 117. The user can visually confirm the video displayed on an electronic viewfinder (EVF) 217 (described later) in the camera 100 through the eyepiece unit 116. The eyepiece detection unit 118 is a sensor for detecting whether the user's eye is placed near the eyepiece unit 116 (eyepiece viewfinder 117).

[0032] The touch bar 119 is a linear touch member (linear touch sensor) that can receive touches. The touch bar 119 is positioned so that it can be touched by the right thumb while holding the grip member 120 with the right hand (grasping with the little finger, ring finger, and middle finger of the right hand), in order to press the shutter button 101 with the index finger of the right hand. In other words, when the user places the eyepiece viewfinder 117 at the same height as the eyes, gazes at the eyepiece member 116, and holds the camera in a position (photographing position) that allows the shutter button 101 to be pressed at any time, the touch bar 119 can be operated. For example, the touch bar 119 can receive tap operations (touching and releasing within a predetermined time period without moving the touch position) and left / right swipe operations (touching and moving the touch position while maintaining the touch) on the touch bar 119. The touch bar 119 is an operation member different from the touch panel 109 and does not have a display function. The touch bar 119 is used, for example, as a multi-functional bar (M-Fn bar) that can assign various functions.

[0033] The camera 100 also includes a grip member 120, a thumb rest member 121, an end cap 122, a lid 123, and a communication terminal 124. The grip member 120 is a gripping member having a shape that is easy to hold with the right hand of a user holding the camera 100. When holding the camera 100 by gripping the grip member 120 with the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned so that they can be operated with the index finger of the right hand. In the same state, the sub-electronic dial 105 and the touch bar 119 are positioned so that they can be operated with the index finger of the right hand. The thumb rest member 121 (thumb standby position) is a gripping member provided at a point where it is easy to place the right thumb that is holding the grip member 120 without operating any operation members on the back of the camera 100. The thumb rest member 121 includes a rubber member for increasing the gripping force (grip feeling). The end cap 122 protects connectors, such as a connection cable that connects the camera 100 to external equipment (external device). The lid 123 closes a slot for storing a recording medium 227, which will be described later, thereby protecting the recording medium 227 and the slot. The communication terminal 124 is a terminal for communicating with a lens unit (e.g., the lens unit 200 or the lens unit 300, which will be described later) that can be detached from the camera 100.

[0034] Figure 3 is a block diagram showing a configuration example of the camera 100. The same Figure 2A and Figure 2B component elements are denoted by the same reference numerals as in Figure 2A and Figure 2B , and their descriptions are optionally omitted. In Figure 3 , the lens unit 200 is attached to the camera 100.

[0035] The lens unit 200 will be described first below. The lens unit 200 is a replaceable lens that can be detached from the camera 100. The lens unit 200 is a single lens (an example of a typical lens). The lens unit 200 includes a diaphragm 201, a lens 202, a diaphragm driving circuit 203, an autofocus (AF) driving circuit 204, a lens system control circuit 205, and a communication terminal 206.

[0036] The diaphragm 201 is configured to have an adjustable aperture. The lens 202 includes a plurality of lenses. The diaphragm driving circuit 203 adjusts the amount of light by controlling the aperture of the diaphragm 201. The AF driving circuit 204 adjusts the focus by driving the lens 202. The lens system control circuit 205 controls the diaphragm driving circuit 203 and the AF driving circuit 204 based on instructions from the system control unit 50, which will be described later. The lens system control circuit 205 controls the diaphragm 201 via the diaphragm driving circuit 203 and adjusts the focus by changing the position of the lens 202 via the AF driving circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication is performed through the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal that allows the lens unit 200 to communicate with the camera 100.

[0037] The camera 100 will be described below. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0038] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on an instruction from the system control unit 50. The imaging unit 211 is an imaging device (image sensor) including a CCD and a CMOS device that convert an optical image into an electrical signal. The imaging unit 211 may include an imaging area phase difference sensor for outputting defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (including pixel interpolation, color conversion, and size adjustment such as size reduction) on the data from the A / D converter 212 or the data from the memory control unit 213. In addition, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and then the system control unit 50 performs exposure control and distance measurement control based on the obtained processing results. For example, this processing can implement through-the-lens (TTL) AF processing, automatic exposure (AE) processing, and flash pre-emission (EF) processing. In addition, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and then the system control unit 50 performs TTL automatic white balance (AWB) processing based on the obtained processing results.

[0039] The image data from the A / D converter 212 is written into the memory 215 through the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written into the memory 215 through the memory control unit 213 without passing through the image processing unit 214. The memory 215 stores the following image data: the image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, and the image data to be displayed on the display unit 108 and the EVF 217. The memory 215 has a storage capacity large enough to store a predetermined number of still images, moving images, and voices within a predetermined time. In addition, the memory 215 serves as an image display memory (video memory).

[0040] The D / A converter 216 converts the image data to be displayed and stored in the memory 215 into an analog signal and supplies the signal to the display unit 108 and the EVF 217. Therefore, the image data to be displayed and written into the memory 215 is displayed on the display unit 108 and the EVF 217 via the D / A converter 216. The display unit 108 and the EVF 217 provide a display in response to the analog signal from the D / A converter 216. The display unit 108 and the EVF 217 are, for example, an LCD or an organic EL display. The digital signal converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216, sequentially transmitted to the display unit 108 and the EVF 217, and displayed thereon, thereby providing a live view display.

[0041] The system control unit 50 is a control unit including at least one processor and / or at least one circuit. In other words, the system control unit 50 can be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements the processing of the flowchart to be described later by running a program recorded in the non-volatile memory 219. In addition, the system control unit 50 performs display control by controlling, for example, the memory 215, the D / A converter 216, the display unit 108, and the EVF 217.

[0042] The camera 100 further includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0043] The system memory 218 is, for example, a RAM. In the system memory 218, constants and variables for the operation of the system control unit 50 and programs read from the non-volatile memory 219 are expanded. The non-volatile memory 219 is an electrically erasable memory. The non-volatile memory 219 is, for example, an EEPROM. In the non-volatile memory 219, constants and programs for the operation of the system control unit 50 are recorded. In this case, the program is a program for implementing the flowchart to be described later. The system timer 220 is a timer unit for measuring the time for various controls and the time of the built-in clock. The communication unit 221 transmits video signals and audio signals to external equipment connected wirelessly or via a wired cable and receives video signals and audio signals from it. The communication unit 221 can also be connected to a wireless local area network (LAN) and the Internet. In addition, the communication unit 221 can also communicate with external equipment via Bluetooth (registered trademark) and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded in the recording medium 227, and receive images and various other information from external equipment. The posture detection unit 222 detects the posture of the camera 100 relative to the direction of gravity. Based on the posture detected by the posture detection unit 222, it can be determined whether the imaging unit 211 captured an image with the camera 100 in a horizontal position or a vertical position. The system control unit 50 can add posture information corresponding to the posture detected by the posture detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image according to the detected posture. The posture detection unit 222 can be, for example, an acceleration sensor or a gyro sensor. The movement (e.g., translation, tilt, lift, stationary or not) of the camera 100 can also be detected by using the posture detection unit 222.

[0044] The eyepiece detection unit 118 can detect an object approaching the eyepiece member 116 (eyepiece viewfinder 117). The eyepiece detection unit 118 can be, for example, an infrared proximity sensor. When an object approaches the eyepiece member 116, the infrared rays from the light-emitting part of the eyepiece detection unit 118 are reflected by the object and received by the light-receiving part of the infrared proximity sensor. The distance from the eyepiece member 116 to the object can be determined based on the amount of infrared radiation received. In this way, the eyepiece detection unit 118 performs proximity detection to detect the proximity distance from the object to the eyepiece member 116. The eyepiece detection unit 118 is an eyepiece detection sensor for detecting the approach and departure of the eye (object) from the eyepiece member 116. If it is detected that the object approaches the eyepiece member 116 within a predetermined distance from a non-proximity state (departure state), the approach of the object is detected. If the object moves away from the eyepiece member 116 by more than a predetermined distance from a proximity state (approach state), the separation of the object is detected. Different thresholds can be used to detect the approach of the object and the separation of the object, for example, using hysteresis. In addition, it is assumed that after the approach of the object is detected, the proximity state continues until the separation of the object is detected. It is assumed that after the separation of the object is detected, the non-proximity state continues until the approach of the object is detected. The system control unit 50 switches the display (display state) / hide (hide state) of the display unit 108 and the EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, at least in the photographic standby state and the automatic switching as a switching setting on the display, during the non-proximity state, the display unit 108 is displayed and the EVF 217 is hidden. During the proximity state, the EVF 217 is displayed and the display unit 108 is hidden. The eyepiece detection unit 118 is not limited to an infrared proximity sensor. The eyepiece detection unit 118 can be other sensors capable of detecting proximity.

[0045] The camera 100 further includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply 225, a recording medium I / F 226, and an operation unit 228.

[0046] The viewfinder external display unit 107 is driven by the viewfinder external display drive circuit 223 and displays various setting values of the camera 100, such as the shutter speed and aperture. The power control unit 224 includes a battery detection circuit, a DC-DC converter, and a switch circuit for switching the blocks to be powered on. The power control unit 224 detects the presence or absence of a battery, the battery type, and the remaining battery capacity, for example. In addition, the power control unit 224 controls the DC-DC converter based on the detection result and an instruction from the system control unit 50, and supplies the required voltage to each part including the recording medium 227 when necessary. The power supply 225 includes: an AC adapter, primary batteries such as alkaline batteries and lithium batteries, and secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries. The recording medium I / F 226 is an interface with the recording medium 227 (for example, a memory card or a hard disk). The recording medium 227 is, for example, a memory card for recording captured images, and includes a semiconductor memory and a magnetic disk. The recording medium 227 can be detached from the camera 100 or can be built into the camera 100.

[0047] The operation unit 228 is an input unit for receiving operations (user operations) from the user. The operation unit 228 is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode change switch 103, a touch panel 109, and other operation units 229. The other operation units 229 include a main electronic dial 104, a sub-electronic dial 105, a moving image button 106, arrow keys 110, a SET button 111, an AE lock button 112, a zoom button 113, a replay button 114, a menu button 115, and a touch bar 119.

[0048] The shutter button 101 includes a first shutter switch 230 and a second shutter switch 231. During the operation of the shutter button 101, that is, during the half-press of the shutter button 101 (photographing preparation instruction), the first shutter switch 230 is turned on to output a first shutter switch signal SW1. The system control unit 50 starts photographing preparation including AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. When the operation of the shutter button 101 is completed, that is, when the shutter button 101 is fully pressed (photographing instruction), the second shutter switch 231 is turned on to output a second shutter switch signal SW2. The system control unit 50 starts a series of photographic operations from reading a signal from the imaging unit 211 to writing an image file including the generated captured image to the recording medium 227 in response to the second shutter switch signal SW2.

[0049] The mode change switch 103 switches the operation mode of the system control unit 50 to any one of a still image shooting mode, a moving image shooting mode, and a replay mode. The still image shooting mode includes modes such as an automatic shooting mode, an automatic scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, it also includes a custom mode and various scene modes used as shooting settings for each shooting scene. The user can directly switch to any shooting mode through the mode change switch 103. Alternatively, after the user switches to the shooting mode list screen through the mode change switch 103, the user can selectively switch to any one of the displayed shooting modes through the operation unit 228. Similarly, the moving image shooting mode can include multiple modes.

[0050] The touch panel 109 is a touch sensor for detecting various touch operations on the display screen (the operation screen of the touch panel 109) of the display unit 108. The touch panel 109 and the display unit 108 can be integrated into a single unit. For example, the touch panel 109 is attached to the upper layer of the display screen of the display unit 108 so that the light transmittance does not interfere with the display of the display unit 108. In addition, the input coordinates on the touch panel 109 and the display coordinates on the display screen of the display unit 108 are correlated with each other, thereby constructing a graphical user interface (GUI) so that the user can directly operate the screen displayed on the display unit 108. The touch panel 109 can be any one of various types including a resistive type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. Depending on the type of the touch panel 109, when the touch panel 109 is touched or when a finger or a pen is brought close to the touch panel 109, a touch is detected. In any case, any type of touch panel can be used.

[0051] For the touch panel 109, the system control unit 50 can detect the following operations or states:

[0052] · The first touch of a finger or a pen on the touch panel 109, that is, the start of a touch (hereinafter referred to as a touch-down)

[0053] · The touch of a finger or a pen on the touch panel 109 (hereinafter referred to as a touch-on)

[0054] · The movement of a finger or a pen in contact with the touch panel 109 (hereinafter referred to as a touch-move)

[0055] · The removal (release) of a finger or a pen from the touch panel 109, that is, the end of a touch (hereinafter referred to as a touch-up)

[0056] · Nothing is in contact with the touch panel 109 (hereinafter referred to as touch-off).

[0057] When a touch is detected, touch-in is also detected. After a touch, touch-in is continuously detected unless touch completion is detected. In addition, when touch movement is detected, touch-in is detected at the same time. Even if touch-in is detected, touch movement is not detected as long as the touch position does not move. After it is detected that all fingers or the pen have completed touching the touch panel 109, touch-off is determined.

[0058] These operations, states, and the position coordinates of the fingers or the pen on the touch panel 109 are notified to the system control unit 50 via the internal bus. The system control unit 50 determines the operation (touch) on the touch panel 109 based on the notified information. During touch movement, the movement direction of the fingers or the pen on the touch panel 109 can also be determined for each vertical component and horizontal component on the touch panel 109 based on the change in the position coordinates. When touch movement of a predetermined distance or more is detected, it is determined that a swipe operation has been performed. A flick is moving a finger quickly a certain distance on the touch panel 109 and releasing the finger from the touch panel 109. In other words, a flick is quickly moving a finger with a brisk touch on the touch panel 109. When touch movement of a predetermined distance or more is detected at a predetermined speed or higher and then touch completion is detected, it is determined that a flick has been performed (it can be determined that a flick has been performed after a swipe operation). In addition, pinch in is bringing the touch positions of multiple points (multi-touch, for example, two-point touch) closer to each other, while pinch out is separating the touch positions from each other. Pinch out and pinch in are collectively referred to as a pinch operation (or simply pinch).

[0059] Figure 4 is a schematic diagram showing a structural example of the lens unit 300. In Figure 4 the lens unit 300 is attached to the camera 100. In Figure 4 the camera 100 shown, the same constituent elements as in Figure 3 are denoted by the same reference numerals as in Figure 3 and their description is optionally omitted.

[0060] The lens unit 300 is a replaceable lens that can be detached from the camera 100. The lens unit 300 is a dual-lens unit that can capture right and left images with parallax. In this embodiment, the lens unit 300 includes two optical systems. Each optical system can capture a wide viewing angle range of approximately 180°. Specifically, each of the two optical systems of the lens unit 300 can capture an object within a field of view (viewing angle) of 180° in the horizontal direction (horizontal angle, azimuth angle, yaw angle) and 180° in the vertical direction (vertical angle, elevation angle, pitch angle). In other words, each optical system can capture the range of the front hemisphere.

[0061] The lens unit 300 includes a right-eye optical system 301R having a plurality of lenses and mirrors, a left-eye optical system 301L having a plurality of lenses and mirrors, and a lens system control circuit 303. The right-eye optical system 301R is an example of the first optical system. The left-eye optical system 301L is an example of the second optical system. The right-eye optical system 301R has a lens 302R disposed near the object. The left-eye optical system 301L has a lens 302L disposed near the object. The lenses 302R and 302L are oriented in the same direction with their optical axes substantially parallel to each other.

[0062] The lens unit 300 is a dual-lens unit (VR180 lens) for obtaining images in the VR180 format (a format of virtual reality (VR) images that can be viewed in binocular vision). In this embodiment, the lens unit 300 has a fisheye lens in each of the right-eye optical system 301R and the left-eye optical system 301L, and the fisheye lens can capture a range of approximately 180°. The lenses in each of the right-eye optical system 301R and the left-eye optical system 301L can capture a range of approximately 160°, which is smaller than the 180° range. The lens unit 300 can form a right image (first image) and a left image (second image) on one or two image pickup devices of the camera when the lens unit 300 is attached to the camera. The right image is formed by the right-eye optical system 301R, and the left image is formed by the left-eye optical system 301L.

[0063] The lens unit 300 is attached to the camera 100 via the lens mounting portion 304 and the camera mounting portion 305 of the camera 100. With this configuration, the system control unit 50 of the camera 100 and the lens system control unit 300 of the lens circuit 303 are electrically connected to each other via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0064] In this embodiment, the right image formed by the right optical system 301R and the left image formed by the left optical system 301L are simultaneously (as a group) formed in the imaging unit 211 of the camera 100. In other words, two optical images formed by the right optical system 301R and the left optical system 301L are formed on one image pickup device. The imaging unit 211 converts the formed object image (optical signal) into an analog electric signal. By using the lens unit 300, two images with parallax can be simultaneously (as a group) obtained from two points (optical systems) on the right optical system 301R and the left optical system 301L. For each of the left-eye image and the right-eye image of the obtained images, VR display is provided, enabling the user to view a three-dimensional VR image within a range of approximately 180°. In other words, the user can obtain a stereoscopic view of the VR180 image.

[0065] In this case, the VR image is an image viewable in the VR display, which will be described later. The VR image includes a panoramic image and an omnidirectional image (entire spherical image) captured by an omnidirectional camera (entire spherical camera), and the video range (effective video range) of the panoramic image is larger than the display range that can be displayed on the display portion at one time. In addition, the VR image includes a moving image and a live image (an image obtained from the camera substantially in real time) as well as a still image. The VR image has a maximum video range (effective video range) corresponding to a field of view of more than 360° in the horizontal direction and more than 360° in the vertical direction. The VR image also includes an image with a viewing angle or video range larger than the viewing angle that can be captured by an ordinary camera, and even if the video range is less than 360° in the horizontal direction and less than 360° in the vertical direction, the video range is still larger than the display range that can be displayed on the display portion at one time. The image captured by the camera 100 having the lens unit 300 is a type of VR image. By, for example, setting the display mode of a display device (a display device capable of displaying a VR image) to "VR view", the VR image can be viewed in the VR display. A VR image with a 360-degree viewing angle is viewed in the VR display; at the same time, the user changes the posture of the display device in the lateral direction (horizontal rotation direction), so that an omnidirectional video can be viewed seamlessly in the horizontal direction.

[0066] VR display (VR view) is a display method (display mode) for displaying a video in a VR image within a field of view corresponding to the pose of a display device, and this display method can change the display range. VR display includes "single-lens VR display (single-lens VR view)", in which an image is displayed after distortion correction by mapping the VR image onto a virtual sphere. VR display includes "dual-lens VR display (dual-lens VR view)", in which a left-eye VR image and a right-eye VR image are displayed in left and right regions after distortion correction by mapping the VR image onto a virtual sphere. The "dual-lens VR display" is provided using a left-eye VR image and a right-eye VR image with parallax to achieve a stereoscopic view of the VR image. In any type of VR display, for example, if a user wears a display device such as an HMD (head-mounted display), the video will be displayed within a field of view corresponding to the orientation of the user's face. For example, assume that in a VR image, at a certain point in time, a video is displayed within a field of view centered at 0° in the horizontal direction (a specific reference point, such as north) and 90° in the vertical direction (90° from the zenith, i.e., the horizon). In this state, if the pose of the display device is flipped (e.g., the display screen changes from south-facing to north-facing), then in the same VR image, the display range is changed to a video within a field of view centered at 180° in the horizontal direction (the opposite reference point, such as south) and 90° in the vertical direction. In other words, when the face of a user wearing an HMD turns from north to south (i.e., the user turns around), the video displayed on the HMD changes from north to south. The VR image captured by the lens unit 300 in this embodiment is an image captured within a range of approximately 180 degrees in the front (180-degree image). There is no video within the range of approximately 180° in the rear. In the VR display of such an image, when the pose of the display device is changed to the side where there is no video, a blank area is displayed.

[0067] This VR display of the VR image gives the user a feeling of staying in the VR image (VR space) in a visual form (immersive feeling). The display method of the VR image is not limited to the method of changing the pose of the display device. For example, the display range can be moved (scrolled) in response to a user operation via a touch panel or a direction button, etc. In addition, in VR display (display mode "VR view"), in addition to changing the display range by changing the pose, the display range can also be changed in response to a touch movement on the touch panel, a drag operation of a mouse, etc., or a press of a direction button. A smartphone attached to a VR goggle (head-mounted adapter) is a type of HMD.

[0068] Figure 5It is a block diagram showing a configuration example of the PC 500. The control unit 501 is, for example, a central processing unit (CPU) that controls the entire PC 500. The read-only memory (ROM) 502 stores programs and parameters in a non-temporary manner. The random access memory (RAM) 503 temporarily stores programs and data supplied from external devices and the like. The recording medium 504 is, for example, a hard disk or a flash memory fixedly installed in the PC 500, or an optical disc, a magnetic card, an optical card, an IC card, or a memory card that can be detached from the PC 500. An image file taken by the camera 100 is read from the recording medium 504. The operation unit 505 receives user operations on the PC 500. The operation members for user operations can be buttons or touch panels provided on the PC 500, or a keyboard or a mouse that can be detached from the PC 500. The display unit 506 displays, for example, data stored in the PC 500 or data supplied from the outside. The display unit 506 can be a part of the PC 500 or a separate display device of the PC 500. The communication unit 507 communicates with external devices such as the camera 100. The system bus 508 connects the components of the PC 500 to enable communication between the components.

[0069] The characteristics of the image taken in the case where the lens unit 300 (dual lens) is attached will be described below. In the case of the lens unit 200 (ordinary single lens), an image that is vertically and horizontally inverted with respect to its actual view (an image rotated by 180°) is formed on the imaging unit 211. Therefore, the formed image is rotated by 180° as a whole to obtain (form) an image that matches its actual view. In the case of the lens unit 300 (dual lens), the right image and the left image are rotated by 180° from their actual views and formed on the imaging unit 211. The layout of the right image and the left image is not particularly limited. In the present embodiment, it is assumed that the right image is formed on the right side of the imaging unit 211 and the left image is formed on the left side. Similar to the case of the lens unit 200 (ordinary single lens), when the formed image (including the image area of the right image and the image area of the left image) is rotated by 180° as a whole, the right image and the left image can match their actual views, but the positions of the right image and the left image are exchanged with each other. In other words, the positional relationship between the left and right images is reversed, so that an image is taken with the right image set on the left side and the left image set on the right side. In the present embodiment, the right image and the left image can be displayed at positions corresponding to the two optical systems (the right-eye optical system 301R and the left-eye optical system 301L).

[0070] Figure 6This is a flowchart showing an example of the operations (photography mode processing) of the camera 100 in the photography mode. These operations are implemented by the system control unit 50 expanding and running a program recorded in the non-volatile memory 219 in the system memory 218. For example, when the camera 100 is started in the photography mode or the mode of the camera 100 is changed to the photography mode, the Figure 6 operations begin.

[0071] In step S601, the system control unit 50 determines whether a camera 100 with a dual lens (e.g., the lens unit 300) is provided. For example, the system control unit 50 determines whether the firmware version of the system control unit 50 is compatible with the dual lens. If it is determined that the version is compatible with the dual lens, the process proceeds to step S602. Otherwise, the process proceeds to step S615. In this embodiment, different from an ordinary single lens, a dual lens needs to acquire and record dual lens information (lens information; information about the two optical systems of the dual lens) for post-processing. Therefore, the process of step S601 is necessary.

[0072] In step S602, the system control unit 50 determines whether the dual lens is attached to the camera 100. If it is determined that the dual lens is attached, the process proceeds to step S603. Otherwise, the process proceeds to step S615. If the dual lens is attached in a state without a dual lens, the process proceeds to step S603. If the attached dual lens is removed and then a single lens is attached, the process proceeds to step S615.

[0073] In step S603, the system control unit 50 obtains the design values of the attached (connected) dual lens from the dual lens. The design values are parameters in the design and are used to exchange the positions of the left and right images and equidistant cylindrical transformation to be described later. For example, obtain Figure 10B the image circle position, image circle diameter, viewing angle, and distortion correction coefficient in. The object image captured by the lens is reflected circularly on the surface of the image pickup device. The image circle refers to the circular object image (circular image).

[0074] In step S604, the system control unit 50 obtains the individual values of the attached (connected) dual lens from the dual lens. The individual values are parameters unique (inherent) to the lens unit, such as errors in the manufacturing process. For example, obtain Figure 10B the image circle displacement, optical axis tilt, and image magnification deviation in. Using the individual values enables image processing with higher accuracy compared to using only the design values.

[0075] In step S605, the system control unit 50 obtains an image from the imaging unit 211.

[0076] In step S606, the system control unit 50 displays the image obtained in step S605 (live view display) on the EVF 217 or the display unit 108.

[0077] In step S607, the system control unit 50 determines whether the user of the camera 100 has provided an instruction to start recording. If it is determined that the instruction to start recording has been provided, the process proceeds to step S608. Otherwise, the process proceeds to step S614. The live view display on the EVF 217 or the display unit 108 continues until the instruction to start recording is provided or the end instruction in step S614, which will be described later.

[0078] The instruction to start recording is a full press of the shutter button 101. The instruction to start recording can be an instruction to take a still image or an instruction to start shooting a moving image. The image to be captured can be any one of a JPEG still image, an MP4 video, a RAW still image, and a RAW video. During the shooting of a moving image, the moving image being recorded can be displayed on the EVF 217 or the display unit 108, just like in the live view display.

[0079] In step S608, the system control unit 50 obtains an image from the imaging unit 211.

[0080] In step S609, the system control unit 50 obtains photographic information related to the shutter speed and aperture during photography and pose information detected by the pose detection unit 222. In the shooting of a RAW image, data (parameters) required for development is also obtained.

[0081] In step S610, the system control unit 50 stores the data (image data) regarding the image obtained in step S608 in a file format in the recording medium 227 (storage medium).

[0082] In step S611, the system control unit 50 stores the information obtained in step S609 (information about the camera 100) in the image file stored in step S610. Therefore, in the image file, the information obtained in step S609 is added as metadata to the image data obtained in step S608.

[0083] In step S612, the system control unit 50 stores the information obtained in steps S603 and S604 (information about the dual lens) in the image file stored in step S610. Therefore, in the image file, the information obtained in steps S603 and S604 is added as metadata to the image data obtained in step S608.

[0084] In step S613, the system control unit 50 determines whether the user of the camera 100 has provided an instruction to end the recording. If it is determined that the instruction to end the recording has been provided, the process proceeds to step S614. Otherwise, the process proceeds to step S608. Steps S608 to S613 are repeated so that moving image frames in the video file or still images in continuous photography (burst shooting) can be continuously recorded.

[0085] In the photography of still images, the instruction to end the recording is, for example, the reset of the full press of the shutter button 101. When the fully pressed shutter button 101 is released within a predetermined time, a still image is taken. If the full press on the shutter button 101 continues for a predetermined time or longer, still images are continuously taken. In the photography of still images, the full press on the shutter button 101 can be used as both the instruction to start recording and the instruction to end the recording. In the photography of moving images, the instruction to end the recording is, for example, the full press on the shutter button 101. For example, when the full press on the shutter button 101 is performed after starting the photography of moving images by the full press on the shutter button 101, the photography of the moving images ends.

[0086] In step S614, the system control unit 50 determines whether the user of the camera 100 has provided an end instruction. If it is determined that the end instruction has been provided, Figure 6 the operation ends. Otherwise, the process proceeds to step S605. The end instruction is, for example, an instruction to turn off the power of the camera 100 or an instruction to switch the mode of the camera 100 from the photography mode to another mode. Specifically, the end instruction is, for example, pressing the power switch 102 or pressing the mode change switch 103.

[0087] If a single lens is attached to the camera 100, the process of step S615 is performed. In step S615, the system control unit 50 takes an image (photographs) using the single lens (single lens imaging). The single lens imaging operation is similar to the conventional imaging (photography) operation in a camera with a single lens, and thus its detailed description is omitted. In the present embodiment, when the image file of the image taken by the single lens is recorded in the recording medium 227, the system control unit 50 obtains information (e.g., design values and individual values) about the attached single lens from the single lens and stores this information in the image file.

[0088] Figure 7It is a flowchart showing an example of operations (display control) for a PC 500 to display an image based on an image file. These operations are achieved by the control unit 501 expanding and running a program (application program) recorded in the ROM 502 in the RAM 503. For example, when a user of the PC 500 operates the operation unit 505 (operation component) to select an image file that has been taken by a camera from the files stored in the recording medium 504, the Figure 7 operation starts. At this time, the recording medium 227 removed from the camera 100 can be used as the recording medium 504. In this embodiment, both the right image and the left image are circular fisheye images (equidistant projection images). The operations of correcting the positions of the right image and the left image to circular fisheye images by using the information about the dual lens as correction information will be described below.

[0089] In step S701, the control unit 501 reads the image file selected by the user of the PC 500 from the recording medium 504. In this case, in addition to the captured image data, the title and metadata added to the image data are also read. For example, the Figure 9 image file in

[0090] In step S702, the control unit 501 determines whether the image file has been taken by a dual lens (by using a dual lens) based on the image file read in step S701. If it is determined that the image file has been taken by a dual lens, the process proceeds to step S703. Otherwise, the process proceeds to step S719. In step S702, for example, it can be determined whether the image file has been taken by a dual lens according to whether the image file contains information about the dual lens (such as design values and individual values). It can be determined whether the image file has been taken by a dual lens according to whether the image file includes a flag indicating the use of a dual lens instead of the detailed information about the dual lens.

[0091] In step S703, the control unit 501 obtains (extracts) information about the dual lens (design values and individual values) from the image file read in step S701.

[0092] In step S704, the control unit 501 obtains the center coordinates of the right image and the left image in the captured image according to the design values obtained in step S703. The center coordinates of the right image and the left image correspond to the optical axis center coordinates of the two optical systems (left eye optical system 301L and right eye optical system 301R) with a dual lens. For example, as Figure 8A and Figure 8B shown, the center coordinates 804 and 808 are obtained. The center coordinate 804 is the horizontal center coordinate of the right image, and the center coordinate 808 is the horizontal center coordinate of the left image.

[0093] In step S705, the control unit 501 obtains the captured image from the image file read in step S701. If the obtained image is displayed as it is, the positions of the left and right images are swapped in the displayed image. Therefore, the positions of the left and right images are swapped thereafter.

[0094] In step S706, the control unit 501 determines whether to provide circular fisheye display. If it is determined to provide circular fisheye display, the process proceeds to step S707. Otherwise (if equidistant cylindrical display is provided), the process proceeds to step S710. In step S706, for example, according to whether the Figure 11A and Figure 11B radio button 1102 is selected to determine whether to provide circular fisheye display. In Figure 11A , the radio button 1102 is selected, while in Figure 11B , the radio button 1102 is not selected. If the radio button 1102 is selected, it is determined to provide circular fisheye display, and then the process proceeds to step S707. If the radio button 1102 is not selected, the process proceeds to step S710. It is determined (for example, before the screens in Figure 11A and Figure 11B are displayed) that circular fisheye display is provided by default, and the process proceeds to step S707, so that the process starts from circular fisheye display. The operations after step S710 will be described later.

[0095] In step S707, based on the center coordinates obtained in step S704 (the center of the optical axes of the left optical system 301L and the right optical system 301R), the control unit 501 swaps the positions of the right and left images in the captured image and generates a processed image (the positions of the left and right images have been swapped). For example, the control unit 501 specifies the area of the right image in the captured image based on the center coordinates of the right image, and specifies the area of the left image in the captured image based on the center coordinates of the left image. Then the control unit 501 swaps the positions of these two specified areas. In this embodiment, the right and left images are horizontally placed next to each other in the captured image. By swapping the positions of the left and right images, the positional relationship between the right and left images is horizontally reversed. The diameters (diameters or radii) of the right and left images can be obtained from the dual-lens information to more accurately specify the areas of the right and left images.

[0096] The method for swapping the positions of the left and right images is not limited to the above method. For example, the Figure 8AThe displacements 805, 806, 809, and 810 in the [description] can set the right and left images while maintaining the obtained displacements during the process of swapping the positions of the left and right images, and other areas can be filled with black. Displacement 805 is the distance from the left end of the captured image to the left end of the right image, and displacement 806 is the distance from the center of the captured image to the right end of the right image. When the positions of the left and right images are swapped, displacement 805 is the distance from the left end of the captured image to the left end of the left image, and displacement 806 is the distance from the center of the captured image to the right end of the left image. Similarly, displacement 809 is the distance from the right end of the captured image to the right end of the left image, and displacement 810 is the distance from the center of the captured image to the left end of the left image. When the positions of the left and right images are swapped, displacement 809 is the distance from the right end of the captured image to the right end of the right image, and displacement 810 is the distance from the center of the captured image to the left end of the right image.

[0097] In step S708, the control unit 501 displays the processed image generated in step S707 on the display unit 506. For example, the processed image 1101 in the [description] is displayed on the display unit 506. Figure 11A in the [description] is displayed.

[0098] In step S709, the control unit 501 determines whether the user of the PC 500 has provided an end instruction. If it is determined that the end instruction has been provided, then Figure 7 the operation ends. Otherwise, the process proceeds to step S706. This can switch the display (display of the captured image) on the display unit 506 between multiple displays including circular fisheye display and equidistant cylindrical display. The end instruction is provided by using the operation unit 505 (operation member). The end instruction is, for example, pressing the Figure 11A and Figure 11B exit button 1106 in the [description].

[0099] If the image file captured by a single lens is read in step S701, the process of step S719 is performed. In step S719, the control unit 501 displays an image based on the image file captured by the single lens on the display unit 506. The process of step S719 is the same as the conventional process of displaying an image captured by a single lens, so its detailed description is omitted.

[0100] Figure 8A and Figure 8B are schematic diagrams showing the swapping of the positions of the left and right images; Figure 8A shows the conventional swapping of the positions of the left and right images without using information about the dual lenses. Figure 8B shows the swapping of the positions of the left and right images by using information about the dual lenses as correction information according to the present embodiment.

[0101] As Figure 8A andFigure 8B As shown, the image 801 before swapping includes a right image 803 set as a circular fisheye image on the left side and a left image 807 set as a circular fisheye image on the right side.

[0102] In Figure 8A , the image 801 is divided into a left half-image and a right half-image at the center coordinates 802 of the image 801, and then the left half-image and the right half-image are swapped with each other. In other words, the left half-image moves to the right side of the right half-image. The image 811 is the image formed after the left and right images are swapped.

[0103] In Figure 8A , the displacement 806 is smaller than the displacement 805. In other words, in the image 801, the right image 803 deviates from the center of the left half of the image 801 towards the center of the image 801. Similarly, the displacement 810 is smaller than the displacement 809. In other words, in the image 801, the right image 807 deviates from the center of the right half of the image 801 towards the center of the image 801. Therefore, in the image 811, the horizontal center coordinate 813 of the left image 807 is shifted by a distance 814 from the center coordinate 804, and the horizontal center coordinate 816 of the right image 803 is shifted by a distance 817 from the center coordinate 808.

[0104] By using the lens information in this embodiment, the horizontal center coordinate of the left image can be aligned with the center coordinate 804, and the horizontal center coordinate of the right image can be aligned with the center coordinate 808 in the image 837, in which the positions of the left and right images have been swapped ( Figure 8B ).

[0105] Figure 9 is a schematic diagram showing an example of the structure of an image file captured by a dual lens. Figure 9The image file includes a title 901, a camera information section 902, a lens information section 903, other information section 904, and an image data section 905. In the title 901, information including the type of the captured image is recorded. In the camera information section 902, information about the camera used for photography is recorded as metadata. For example, photographic information such as the shutter speed and aperture during photography and the pose information of the camera during photography are recorded. In the lens information section 903, information about the dual lens used for photography is recorded as metadata. For example, the design value and individual value of the dual lens are recorded. In the other information section 904, other information is recorded as metadata. For example, in the case of a moving image, information that changes for each frame is recorded. In the case of a RAW image, data required for development is recorded. In the image data section 905, image data is recorded. In the case of a moving image, voice data as well as image data is recorded. In this example, information about the camera, information about the dual lens, and other information are recorded in the same image file. This information may be recorded in other files associated with the image file.

[0106] Figure 10A is a schematic diagram showing an example of the lens information obtained from the dual lens. The lens information includes:

[0107] 1. Lens design value

[0108] 2. Lens individual value

[0109] 3. Lens mark

[0110] 4. Lens focal length

[0111] 5. Lens temperature

[0112] The lens design value is the design value for correcting aberration. During the manufacturing process of the dual lens, each optical system in the two optical systems (left-eye optical system 301L and right-eye optical system 301R) has errors, such as eccentricity or tilt of the lens. If the positions of the left and right images are exchanged or an equidistant cylindrical transformation is performed without considering the errors, the quality of the dual-lens VR display is degraded, making it difficult to obtain a satisfactory stereoscopic view. For example, the lens individual value is the measurement result of the errors detected during the manufacturing process of the dual lens. The lens design value and the lens individual value will be specifically described later according to Figure 10B will specifically describe the lens design value and the lens individual value.

[0113] If the positions of the left and right images are exchanged or an equidistant cylindrical transformation is performed without considering the individual value (manufacturing error) of the lens, the quality of the dual-lens VR display is degraded, making it difficult to obtain a satisfactory stereoscopic view. Referring to Figure 16A and Figure 16B the reason for this problem will be described below. Figure 16AAn ideal optical system is shown. In Figure 16A , the optical axes of the left and right lenses extend parallel to each other and point in the same direction. Therefore, the parallax increases as the object approaches the image sensor, enabling a good stereoscopic view. Figure 16B An example of an optical system with an error deviating from the design value is shown. In Figure 16B , the optical axes of the left and right lenses do not extend parallel to each other and point in different directions. This results in parallax at infinity, interfering with a good stereoscopic view. Additionally, the individual values of the left and right lenses may cause parallax in the vertical direction or cause a change in image magnification. This may interfere with a good stereoscopic view. Therefore, in this embodiment, the captured images are corrected by utilizing the errors of the left and right images actually used for photography, thereby correcting the images to be close to those captured by the ideal optical system. This enables a good stereoscopic view.

[0114] The lens mark is a mark indicating a dual lens and can be used to determine whether a dual lens has been used. The lens focal length is the distance from the "principal point" which is the center of the lens to the image pickup device (image formation position). The lens focal length may or may not be a common parameter for the two optical systems (left-eye optical system 301L and right-eye optical system 301R) of the dual lens. In order to provide a high-quality dual-lens VR display by accurately swapping the positions of the left and right images or performing an equidistant cylindrical transformation, a detailed (high-precision) lens focal length is required. The lens temperature is the temperature of the dual lens and is used to identify the ambient temperature during imaging, etc.

[0115] Figure 10B is a schematic diagram showing the details of the lens design value and the lens individual value. In this embodiment, the lens design value and the lens individual value are used as correction information for swapping the positions of the left and right images and the equidistant cylindrical transformation.

[0116] The lens design value includes:

[0117] 1. Image circle position

[0118] 2. Image circle diameter

[0119] 3. Viewing angle

[0120] 4. Distortion correction coefficient

[0121] The image circle position is the central coordinate of the optical axis of the optical system in the image to be captured. The image circle position is prepared for each of the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R) of the dual lens. In other words, the image circle position is the central coordinate of the image circle (circular fisheye image) formed on the image pickup device. The image circle position is prepared for each of the right image and the left image. For example, the origin of the coordinates is the center of the image pickup device (the center of the image to be captured). The image circle position includes coordinates in the horizontal direction and coordinates in the vertical direction. The image circle position can be used based on various information about the central axis of the optical system in the image to be captured. For example, the image circle position can be the distance from a predetermined position (center or upper left corner) in the image to the optical axis center.

[0122] The image circle diameter is the diameter of the image circle (circular fisheye image) formed on the image pickup device. The viewing angle is the viewing angle of the image circle (circular fisheye image) formed on the image pickup device. The distortion correction coefficient is the ratio of the designed image height of the lens to the ideal image height. The distortion correction coefficient can be set for each image height, and the distortion correction coefficient can be calculated by using interpolation of multiple distortion correction coefficients. An approximate polynomial can be set for the relationship between the image height and the distortion correction coefficient. The image circle diameter, the viewing angle, and the distortion correction coefficient may or may not be common parameters of the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R) of the dual lens.

[0123] When displaying the circular fisheye image, the PC 500 can display a magic window in the circular fisheye image. The magic window is a display item indicating the area to be cut (first) for VR display. For example, the magic window is displayed based on the image circle position, the image circle diameter, and the viewing angle. This can improve the display quality of the magic window. To correctly display the magic window, the PC 500 optionally edits and uses the image circle position, the image circle diameter, and the viewing angle. For example, the PC 500 multiplies the image circle position or the image circle diameter by a coefficient.

[0124] The individual lens values include:

[0125] 5. Image circle displacement

[0126] 6. Optical axis tilt

[0127] 7. Image magnification deviation

[0128] This information is prepared by measuring the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R) of the dual lens.

[0129] Image circle displacement is the deviation of the center coordinates of the image circle (circular fisheye image) formed on the image pickup device from the design value. For example, the image circle displacement includes the deviation in the horizontal direction and the deviation in the vertical direction. In the case where the coordinates of the design value (two-dimensional coordinates, including the coordinates in the horizontal direction and the vertical direction) are taken as the origin, the deviation in the horizontal direction is represented by the coordinate in the horizontal direction, and the deviation in the vertical direction is represented by the coordinate in the vertical direction. Figure 17A An example showing the image circle displacement. Region 1701 represents the right half region or the left half region of the sensor size screen (imaging area). Image circle 1702 is the actual image circle of the left and right lenses. Image circle 1703 is the ideal image circle (design value) of the optical system. Displacement occurs between the position of the actual image circle 1702 and the position of the ideal image circle 1703.

[0130] Optical axis tilt is the deviation from the design value in the optical axis direction near the object. For example, the optical axis tilt includes the deviation in the horizontal direction and the deviation in the vertical direction. The deviation in each direction is represented by an angle. Figure 17B An example showing the deviation caused by the optical tilt in the horizontal direction. Line 1712 represents the actual direction (tilt) of the optical axes of the left and right lenses in the horizontal direction. Line 1713 represents the ideal direction (design value) of the optical axis of the optical system in the horizontal direction. A deviation occurs between the actual direction 1712 of the optical axis in the horizontal direction and the ideal direction 1713 of the optical axis in the horizontal direction. Figure 17C An example showing the deviation caused by the optical tilt in the vertical direction. Line 1722 represents the actual direction of the optical axes of the left and right lenses in the vertical direction. Line 1723 represents the ideal direction (design value) of the optical axis of the optical system in the horizontal direction. A deviation occurs between the actual direction 1722 of the optical axis in the vertical direction and the ideal direction 1723 of the optical axis in the horizontal direction.

[0131] Image magnification deviation is the deviation of the size of the image circle (circular fisheye image) formed on the image pickup device from the design value. The deviation is represented, for example, by the ratio to the design value. Figure 17D An example showing the image magnification deviation. Image circle 1732 is the actual image circle formed on the image pickup device by the left and right lenses. Image circle 1733 is the ideal image circle (design value) formed on the image pickup device by the optical system. The deviation occurs between the actual image circle 1732 of the optical system and the ideal image circle 1733.

[0132] The information included in the lens information is not limited to the foregoing information. For example, the lens information may include the boundary positions of the right image and the left image in the captured image (the edge positions of the circular fisheye image; the positions represented by displacements 805, 806, 809, and 810, etc.). The lens information may include the midpoint coordinates between the right image and the left image in the captured image. In many cases, the midpoint coordinates match the center coordinates of the captured image. The lens information may include information representing the area of the magic window (e.g., the coordinates at the upper left corner of the magic window, the width of the magic window, and the height of the magic window). The lens information may include correction values for improving the accuracy of swapping the positions of the left and right images and equidistant cylindrical transformation (e.g., correction values obtained through dual-lens calibration).

[0133] Figure 10C is a schematic diagram showing an example of the camera information generated in the camera. For example, the camera information is used to provide a high-quality VR display. The camera information includes:

[0134] 1. Camera recording area information

[0135] 2. Camera internal accelerometer information

[0136] 3. Right exposure correction information

[0137] The camera recording area information is information about the effective image area. The displayable effective image area varies depending on the recording mode or the camera's sensor. The PC 500 uses the camera recording area information to provide a more accurate display. The camera internal accelerometer information is the pose information obtained by using the accelerometer sensor (level) in the camera. This information represents the pose of the camera in the roll direction or the pitch direction. The PC 500 uses the camera internal accelerometer information to identify the pose of the camera during photography. The PC 500 performs, for example, electronic vibration isolation or horizontal correction (zenith correction to make the vertical direction of the display approach the vertical direction of the actual space). The right exposure correction information is the exposure setting value for making the exposure of the right image close to the exposure of the left image. The PC 500 uses the right exposure correction information to provide a natural (less embarrassing) dual-lens VR display.

[0138] Figure 11A is a schematic diagram showing an example of the display on the application screen displayed on the display unit 506 by the control unit 501 after swapping the positions of the left and right images (the display in step S708). The screen 1100 is the application screen. Figure 11AThe screen 1100 therein includes the processed image 1101, radio buttons 1102 and 1103, a checkbox 1104, a save button 1105, and an exit button 1106. The processed image 1101 is an image formed after swapping the left and right images, that is, the processed image generated in step S707. The radio button 1102 is the radio button selected when providing a circular fisheye display. The radio button 1103 is the radio button selected when providing an equidistant cylindrical display. When the radio button 1102 is selected, the radio button 1103 is not selected. When the radio button 1102 is not selected, the radio button 1103 is selected. The checkbox 1104 and the save button 1105 are display items used when providing an equidistant cylindrical display. When the radio button 1102 is selected, the checkbox 1104 and the save button 1105 are grayed out to disable the operations. The use of the checkbox 1104 and the save button 1105 will be described below. The exit button 1106 is a button for terminating the application of the screen 1100.

[0139] The equidistant cylindrical display will be described below. If it is determined in Figure 7 step S706 that the circular fisheye display is not selected but the equidistant cylindrical display is selected, the process advances to step S710 to display the image in an equidistant cylindrical display. For example, when the radio button 1103 is selected from among the Figure 11A and Figure 11B radio buttons 1102 and 1103, it is determined that the circular fisheye display is not selected, but the equidistant cylindrical display is selected.

[0140] In step S710, the control unit 501 of the PC 500 determines whether to perform adjustment using the individual values of the dual lenses (adjustment of individual values). If it is determined to adjust the individual values, the process advances to step S716. Otherwise, the process advances to step S711. In step S710, for example, based on whether the Figure 11B checkbox 1104 in

[0141] is checked, it is determined whether to adjust the individual values. If the checkbox 1104 is checked, it is determined to adjust the individual values, and then the process advances to step S716. If the checkbox 1104 is not checked, the process advances to step S711. The operations after step S716 will be described later.In step S711, the control unit 501 generates a mapping for equidistant cylindrical transformation based on the central coordinates obtained in step S704 (the center of the optical axes of the left optical system 301L and the right optical system 301R). The equidistant cylindrical transformation is a transformation in which a circular fisheye image, assumed to be a sphere, has latitudes (horizontal lines) and longitudes (vertical lines) that intersect at right angles, as in the equidistant cylindrical projection used for mapping. Through the equidistant cylindrical transformation, the circular fisheye image is transformed into an equidistant cylindrical image in the shape of a rectangle. The mapping represents the positions in the pre-transformation image that correspond to the pixels after transformation. In this embodiment, a mapping for equidistant cylindrical transformation is generated such that the circular fisheye image can be transformed into an equidistant cylindrical image, and the positions of the right image and the left image can be corrected. In step S711, in the same manner as in step S707, the control unit 501 designates the region of the right image and the region of the left image in the captured image. And the control unit 501 generates a mapping based on the two designated regions.

[0142] In step S712, the control unit 501 performs equidistant cylindrical transformation by using the mapping generated in step S711, thereby generating a processed image. The swapping of the left and right image positions is regarded as part of the equidistant cylindrical transformation. The swapping of the image positions can be separate from the equidistant cylindrical transformation.

[0143] In step S713, the control unit 501 displays the processed image generated in step S712 or step S718 on the display unit 506. Step S718 will be described in another embodiment later. For example, the processed image 1111 is displayed on the display unit 506. Figure 11B in the processed image.

[0144] In step S714, the control unit 501 determines whether to save the processed image (the image after equidistant cylindrical transformation) displayed in step S713. If it is determined to save the processed image, the process proceeds to step S715. Otherwise, the process proceeds to step S709. In step S714, for example, based on whether the save button 1105 has been pressed Figure 11B in the to determine whether to save the processed image. If the save button 1105 has been pressed, it is determined to save the processed image, and the process proceeds to step S715. If the save button 1105 has not been pressed, the process proceeds to step S709.

[0145] In step S715, the control unit 501 stores an image file of the processed image (the image after equidistant cylindrical transformation) displayed in step S713 in the storage medium 504.

[0146] Figure 12 is a schematic diagram showing the equidistant cylindrical transformation according to this embodiment. AsFigure 12 As shown, the image 1201 before the equidistant cylindrical transformation includes a right image 1202 set as a circular fisheye image on the left side and a left image 1205 set as a circular fisheye image on the right side. The image 1208 is the image after the equidistant cylindrical transformation, and includes equidistant cylindrical images 1209 and 1210. In this embodiment, the mapping for the equidistant cylindrical transformation is generated with the corresponding relationships indicated by arrows 1211 and 1212. In the mapping of this embodiment, the pixels of the equidistant cylindrical image 1209 set on the left side are associated with the positions of the left image 1205 set on the right side, while the pixels of the equidistant cylindrical image 1210 set on the right side are associated with the positions of the right image 1202 set on the left side. By using this mapping, the left image 1205 set on the right side is transformed into the equidistant cylindrical image 1209 set on the left side, and the right image 1202 set on the left side is transformed into the equidistant cylindrical image 1210 set on the right side. In other words, while transforming the circular fisheye image into an equidistant cylindrical image, the positions of the right image and the left image are swapped. This makes the positional relationship between the right image and the left image consistent with the positional relationship between the two optical systems (the right eye optical system 301R and the left eye optical system 301L).

[0147] Figure 11B is a schematic diagram showing an example of the display on the application screen displayed on the display unit 506 by the control unit 501 after the equidistant cylindrical transformation (the display in step S713). In Figure 11A it, the processed image 1101, which is the image after the positions of the left and right images are swapped, is displayed on the screen 1100 (application screen). In Figure 11B it, the processed image 1111 (the processed image generated in step S712 or step S718), which is the image after the equidistant cylindrical transformation, is displayed on the screen 1100. In Figure 11B it, the check box 1104 and the save button 1105 are not grayed out, and the check box 1104 and the save button 1105 can be operated. When adjusting the individual value, the check box 1104 is selected. When storing the image file of the processed image 1111, the save button 1105 is pressed. Since the check box 1104 is not selected, the processed image generated in step S712 is displayed as the processed image 1111.

[0148] In the following example, when providing equidistant cylindrical display, the individual value is adjusted. Specifically, the operations starting from step S716 in Figure 7 will be described below. When providing circular fisheye display, the individual value of the dual lens can also be used. Detailed description is omitted.

[0149] In step S716, the control unit 501 adjusts the design value obtained in step S703 based on the individual value obtained in step S703. For example, based on Figure 10B the image circle displacement in, the image circle position (the center coordinates of the right image and the left image in the captured image) is adjusted. If the individual value is different from the design value, the individual value is added to the design value. If the absolute value of the individual value is equivalent to the design value, the design value is replaced with the individual value.

[0150] In step S717, based on the adjusted center coordinates (the center of the optical axes of the left-eye optical system 301L and the right-eye optical system 301R) in step S716, the control unit 501 generates a mapping for equidistant cylindrical transformation. Figure 18 An example is shown in which the image position in the spatial coordinates is adjusted for each pixel based on the adjusted center coordinates (correction for correcting the pixel position). In the left image, the image position is adjusted for each pixel so that the image position 1801 is adjusted to the image position 1802 in the spatial coordinates. In the right image, the image position is adjusted for each pixel so that the image position 1803 is adjusted to the image position 1804 in the spatial coordinates. This mapping is generated by the same method as in step S711. Using the adjusted center coordinates enables a higher-precision equidistant cylindrical transformation.

[0151] In step S718, the control unit 501 performs an equidistant cylindrical transformation by using the mapping generated in step S717, thereby generating a processed image. This processed image is generated by the same method as in step S712. The operations starting from step S713 are performed as described above. As described above, the design value of the lens information is corrected based on the individual value of the dual lenses, thereby converting the circular fisheye image into an equidistant cylindrical image with higher precision.

[0152] According to the information processing apparatus of the present embodiment, if an image is displayed based on an image file captured by a dual lens, the left and right images obtained through the optical system of the dual lens can be appropriately placed after the centers of the left and right images are corrected. This can display a natural dual-lens image.

[0153] Second Embodiment

[0154] A second embodiment of the present disclosure will be described below. This embodiment will describe an example in which the camera 100 and the PC 500 are connected to communicate with each other, the live view image captured by the camera 100 is sent to the PC 500, and the PC 500 displays the live view image on the display unit 506.

[0155] Figure 13is a flowchart showing an operation example of the camera 100. These operations are implemented by the system control unit 50 expanding and running a program recorded in the non-volatile memory 219 in the system memory 218. For example, when the camera 100 is started, Figure 13 the operation starts. Figure 13 The operation is for the function of displaying a live view image taken by the camera on the display unit of the PC (PC live view). When the camera 100 is in the photography standby state, Figure 13 the operation is performed. When an instruction to start recording is input from the PC 500 during PC live view, still image shooting or moving image shooting is performed. At this time, PC live view can be continued. The photographic control using the dual lens is Figure 6 the processing from step S608 to step S613 in, and its description is omitted.

[0156] In step S1301, the system control unit 50 determines whether the camera 100 for a dual lens (for example, the lens unit 300) is provided. If it is determined that this version is compatible with the dual lens, the processing proceeds to step S1302. Otherwise, the processing proceeds to step S1311. The processing of step S1301 is the same as Figure 6 the processing of step S601 in.

[0157] In step S1302, the system control unit 50 determines whether the dual lens is attached to the camera 100. If it is determined that the dual lens is attached, the processing proceeds to step S1303. Otherwise, the processing proceeds to step S1311. The processing of step S1302 is the same as Figure 6 the processing of step S602 in.

[0158] In step S1303, the system control unit 50 obtains the design value of the attached (connected) dual lens from the dual lens. In step S1304, the system control unit 50 obtains the individual value of the attached (connected) dual lens from the dual lens. The processing of step S1303 and step S1304 is the same as the processing of step S603 and step S604.

[0159] In step S1305, the camera 100 is connected to the PC 500, and the system control unit 50 detects the connection between the camera 100 and the PC 500. In step S1306, the system control unit 50 receives a request to start PC live view from the PC 500. In step S1307, the system control unit 50 receives a request for a live view image from the PC 500. The request for a live view image includes information (resolution information) for specifying the resolution of the live view image as will be described later. The system control unit 50 performs the processing of step S1309 to send the live view image with the specified resolution to the PC 500.

[0160] In step S1308, the system control unit 50 converts the information (lens information regarding the dual lenses) obtained in steps S1303 and S1304 according to the coordinate system transformation of the live view image to be transmitted. The captured image (the image recorded in the image file) and the live view image have different resolutions, so the information obtained in steps S1303 and S1304 cannot be directly used for the image processing of the live view image. Therefore, in this embodiment, the lens information is converted into information that conforms to the coordinate system of the live view image.

[0161] In step S1309, the system control unit 50 transmits the live view image and the lens information converted in step S1308 to the PC 500. The system control unit 50 changes the resolution of the live view image based on the resolution information obtained in step S1307 and transmits the live view image to the PC 500. In this embodiment, the system control unit 50 of the camera 100 converts the lens information. The control unit 501 of the PC 500 can also convert the lens information. At this time, the lens information before conversion and the parameters required for converting the lens information are transmitted to the PC 500.

[0162] In step S1310, the system control unit 50 determines whether to end the PC live view. For example, if the camera 100 and the PC 500 are disconnected from each other or the user instructs the camera 100 or the PC 500 to end the PC live view, it is determined to end the PC live view. If it is determined to end the PC live view, Figure 13 the operation ends. Otherwise, the process proceeds to step S1307.

[0163] If a single lens is attached to the camera 100, the process of step S1311 is performed. In step S1311, the system control unit 50 transmits the live view image captured by the single lens to the PC 500. The process of step S1311 is the same as the conventional process for transmitting the live view image captured by the single lens to external equipment, so its detailed description is omitted. In this embodiment, when the live view image captured by the single lens is transmitted to the PC 500, the system control unit 50 does not obtain the information (including design values and individual values) regarding the attached single lens from the single lens or transmit the information to the PC 500.

[0164] Figure 14 is a flowchart showing an operation example of the PC 500. These operations are realized by the control unit 501 expanding and running the program (application program) recorded in the ROM 502 in the RAM 503. For example, when the user instructs the PC 500 to start a specified application, Figure 14 the operation starts. Figure 14The operation is for the function of displaying a live view image captured by a camera on the display unit of a PC (PC live view).

[0165] In step S1401, a camera (e.g., camera 100) is connected to PC 500, and control unit 501 detects the connection between the camera and PC 500.

[0166] In step S1402, control unit 501 determines whether the camera connected in step S1401 is compatible with a dual lens (e.g., lens unit 300). For example, control unit 501 obtains model information about the camera from the connected camera and determines whether the camera is compatible with the dual lens based on the obtained model information. If it is determined that the camera is compatible with the dual lens, the process proceeds to step S1403. Otherwise, the process proceeds to step S1421. A camera compatible with a dual lens is, for example, a camera to which a dual lens can be attached.

[0167] In step S1403, control unit 501 determines whether the firmware of the camera connected in step S1401 is compatible with the dual lens. For example, control unit 501 obtains version information about the camera firmware from the connected camera and determines whether the firmware version of the connected camera is compatible with the dual lens based on the obtained information. If it is determined that the firmware is compatible with the dual lens, the process proceeds to step S1404. Otherwise, the process proceeds to step S1421.

[0168] Even if a camera compatible with a dual lens is connected to PC 500, due to the old firmware version of the connected camera, the connected camera may become incompatible with the dual lens. Therefore, the process of step S1403 is necessary. Since various cameras can be connected to PC 500, cameras incompatible with the dual lens can also be connected to PC 500 regardless of the firmware version. Therefore, the process of step S1402 is required before the process of step S1403.

[0169] In step S1404, control unit 501 determines whether a dual lens has been attached to camera 100 connected in step S1401. If it is determined that a dual lens has been attached, the process proceeds to step S1405. Otherwise, the process proceeds to step S1421.

[0170] In step S1405, control unit 501 sends a request to start PC live view to the camera connected in step S1401.

[0171] In step S1406, the control unit 501 determines whether to provide a circular fisheye display. If it is determined to provide a circular fisheye display, the process proceeds to step S1407. Otherwise (if an equidistant cylindrical display is provided), the process proceeds to step S1414. In step S1406, for example, it is determined whether to provide a circular fisheye display based on whether the radio button 1505 in Figures 15A to 15D is selected. In Figure 15A and Figure 15C , the radio button 1505 is selected, while in Figure 15B and Figure 15D , the radio button 1505 is not selected. If the radio button 1505 is selected, it is determined to provide a circular fisheye display, and then the process proceeds to step S1407. If the radio button 1505 is not selected, the process proceeds to step S1414.

[0172] In step S1407, the control unit 501 sends a request for a live view image to the camera connected in step S1401. In this embodiment, the request for a live view image in step S1407 is a request for a live view image with a normal resolution. The normal resolution is, for example, 4K resolution.

[0173] In step S1408, the control unit 501 receives the live view image captured by the camera and the lens information about the dual lens attached to the camera from the camera connected in step S1401. The resolution of the live view image received in step S1408 is the normal resolution. The lens information received in step S1408 is information converted based on the received live view image (for example, the lens information converted in step S1308 of Figure 13 ).

[0174] In step S1409, the control unit 501 determines whether to swap the positions of the left and right images. If it is determined to swap the positions of the left and right images, the process proceeds to step S1410. Otherwise, the process proceeds to step S1412. In step S1409, for example, it is determined whether to swap the positions of the left and right images based on whether the checkbox 1507 in Figure 15A and Figure 15C is selected. If the checkbox 1507 is selected, it is determined to swap the positions of the left and right images, and then the process proceeds to step S1410. If the checkbox 1507 is not selected, the process proceeds to step S1412.

[0175] In step S1410, based on the lens information obtained in step S1408, the control unit 501 exchanges the positions of the left and right images in the live view image obtained in step S1408, and generates a processed live view image (the positions of the left and right images have been exchanged). The method for exchanging the positions of the left and right images is the same as that in Figure 7 step S707 in []. Based on the center coordinates (the optical axis centers of the left optical system 301L and the right optical system 301R) included in the lens information received together with the live view image, the control unit 501 exchanges the positions of the left and right images in the live view image, and generates a processed live view image.

[0176] In step S1411, the control unit 501 displays the processed live view image generated in step S1410 on the display unit 506.

[0177] In step S1412, the control unit 501 displays the live view image obtained in step S1408 on the display unit 506.

[0178] In step S1413, the control unit 501 determines whether to end the PC live view. For example, if the camera 100 and the PC 500 are disconnected from each other or the user instructs the camera 100 or the PC 500 to end the PC live view, it is determined to end the PC live view. The instruction to end the PC live view is, for example, pressing the Figures 15A to 15D exit button 1508 in []. If it is determined to end the PC live view, then Figure 14 the operation of [] ends. Otherwise, the process proceeds to step S1406.

[0179] As described above, if equidistant cylindrical display is provided, the process proceeds from step S1406 to step S1414. In step S1414, the control unit 501 sends a request for a live view image to the camera connected in step S1401. In this embodiment, the request for a live view image in step S1414 is a request for a live view image with a low resolution (lower than the normal resolution). In the case of equidistant cylindrical display, equidistant cylindrical transformation (transformation from a circular fisheye image to an equidistant cylindrical image) is necessary. The higher the resolution of the image to be subjected to equidistant cylindrical transformation, the longer the time required for equidistant cylindrical transformation. This increases the delay caused by equidistant cylindrical transformation. In this embodiment, a live view image with a low resolution is requested to increase the speed of equidistant cylindrical transformation (shorten the time required for equidistant cylindrical transformation). If the equidistant cylindrical transformation results in a delay within the tolerance, a live view image with a normal resolution may be requested in the equidistant cylindrical display.

[0180] In step S1415, the control unit 501 receives a live view image captured by the camera and lens information regarding the dual lens attached to the camera from the camera connected in step S1401. The resolution of the live view image received in step S1415 is a low resolution. The lens information received in step S1415 is information converted based on the received live view image (e.g., the lens information converted in step S1308 of Figure 13 ).

[0181] In step S1416, the control unit 501 determines whether to swap the positions of the left and right images. If it is determined to swap the positions of the left and right images, the process proceeds to step S1417. Otherwise, the process proceeds to step S1419. In step S1416, for example, based on whether the check boxes 1507 in Figure 15B and Figure 15D have been selected, it is determined whether to swap the positions of the left and right images. If the check box 1507 has been selected, it is determined to swap the positions of the left and right images, and then the process proceeds to step S1417. If the check box 1507 has not been selected, the process proceeds to step S1419.

[0182] In step S1417, based on the lens information obtained in step S1415, the control unit 501 swaps the positions of the left and right images in the live view image obtained in step S1415 and converts each of the left and right images into an equidistant cylindrical image. In other words, the control unit 501 swaps the positions of the left and right images and performs an equidistant cylindrical transformation, thereby generating a processed live view image. The transformation to equidistant cylindrical display including the swapping of the left and right image positions is similar to step S712. The transformation to equidistant cylindrical display including the swapping of the left and right image positions can be similar to steps S716 to S718. In other words, the control unit 501 can generate a mapping based on the lens design value corrected by using the individual values included in the lens information received together with the live view image, and perform the transformation to equidistant cylindrical display including the swapping of the left and right image positions based on this mapping. As in the first embodiment, the swapping of the left and right image positions can be performed as part of the equidistant cylindrical transformation or can be performed as a different process.

[0183] In step S1418, the control unit 501 displays the processed live view image generated in step S1417 on the display unit 506.

[0184] In step S1419, the control unit 501 does not swap the positions of the left and right images in the live view image obtained in step S1415, but converts each of the left and right images into an equidistant cylindrical image. In other words, the control unit 501 generates a processed live view image by equidistant cylindrical transformation without swapping the positions of the left and right images.

[0185] In step S1420, the control unit 501 displays the processed live view image generated in step S1419 on the display unit 506.

[0186] If the camera 100 is not compatible with a dual lens or a single lens is attached to the camera 100, the process of step S1421 is performed. In step S1421, the control unit 501 displays the live view image captured by the single lens on the display unit 506. The process of step S1421 is the same as the conventional process of a PC or the like for displaying the live view image captured by a single lens, and thus its detailed description is omitted.

[0187] In steps S1410, S1417, and S1419, the control unit 501 performs image processing on the live view image acquired from the connected camera. In step S1413 after steps S1410, S1417, and S1419, the control unit 501 determines whether to end the PC live view. If the PC live view continues, the process returns to step S1406 before steps S1410, S1417, and S1419. Therefore, in Figure 14 the operation, the image processing in any one of steps S1410, S1417, and S1419 can be repeated.

[0188] To improve the speed of image processing, the control unit 501 can record information related to the performed image processing in the RAM 503 and use the information from subsequent image processing. For example, the control unit 501 records the correspondence (image processing mapping) between the pixels before image processing and the pixels after image processing. The image processing mapping can be continuously used unless the resolution of the live view image or the lens information changes. When performing image processing in any one of steps S1410, S1417, and S1419, the control unit 501 records the image processing mapping for the image processing. When performing the same image processing again, the control unit 501 uses the recorded image processing mapping for the image processing. This can improve the speed of image processing.

[0189] Figures 15A to 15DIt is a schematic diagram showing an example of the display (PC live view display) on the application screen displayed by the control unit 501 on the display unit 506. The screen 1500 is an application screen (remote live view screen). The screen 1500 includes a live view display area 1501, a guidance display area 1502, a guidance display area 1503, an operation area 1504, and an exit button 1508.

[0190] The live view display area 1501 is an area for displaying a live view image. The live view display area 1501 includes a left display area 1501A and a right display area 1501B. The guidance display area 1502 is an area for displaying a string indicating which one of the two optical systems (left optical system 301L and right optical system 301R) of the dual lens displays an image in the left display area 1501A. The guidance display area 1503 is an area for displaying a string indicating which one of the two optical systems (left optical system 301L and right optical system 301R) of the dual lens displays an image in the right display area 1501B. The operation area 1504 is an area for receiving operations for PC live view. The operation area 1504 displays a radio button 1505, a radio button 1506, and a checkbox 1507. The radio button 1505 is a radio button selected when providing a circular fisheye display. The radio button 1506 is a radio button selected when providing an equidistant cylindrical display. When the radio button 1505 is selected, the radio button 1506 is not selected. When the radio button 1505 is not selected, the radio button 1506 is selected. The checkbox 1507 is a box to be checked when swapping the positions of the left and right images. When the checkbox 1507 is operated, the positions of the right image (right-eye video) and the left image (left-eye video) in the live view image are swapped, and at the same time, the strings displayed in the guidance display areas 1502 and 1503 are also swapped with each other. The exit button 1508 is a button for terminating the PC live view.

[0191] In Figure 15A it, the radio button 1505 for providing a circular fisheye display is selected. The checkbox 1507 for swapping the positions of the left and right images is not checked. Therefore, the live view image obtained from the camera is displayed as it is in the live view display area 1501. Specifically, the right-eye video is displayed as a circular fisheye image in the left display area 1501A, and the left-eye video is displayed as a circular fisheye image in the right display area 1501B.

[0192] In Figure 15BIn [a certain situation], the radio button 1506 for providing an equidistant cylindrical display is selected. The checkbox 1507 for swapping the left and right image positions is not selected. Therefore, the right-eye video and the left-eye video (circular fisheye images) in the live view image obtained from the camera are each transformed into equidistant cylindrical images (the positions of the images are not swapped). Thereafter, the live view display area 1501 displays the live view image obtained after being transformed into equidistant cylindrical images. Specifically, the right-eye video is displayed as an equidistant cylindrical image in the left display area 1501A, and the left-eye video is displayed as an equidistant cylindrical image in the right display area 1501B.

[0193] In Figure 15C In [a certain situation], the radio button 1505 for providing a circular fisheye display is selected, and the checkbox 1507 for swapping the left and right images is selected. Therefore, the positions of the right-eye video and the left-eye video in the live view image obtained from the camera are swapped. Thereafter, the live view display area 1501 displays the live view image obtained after the positions are swapped. Specifically, the left-eye video is displayed as a circular fisheye image in the left display area 1501A, and the right-eye video is displayed as a circular fisheye image in the right display area 1501B.

[0194] In Figure 15D In [a certain situation], the radio button 1506 for providing an equidistant cylindrical display is selected, and the checkbox 1507 for swapping the left and right images is selected. Therefore, the positions of the right-eye video and the left-eye video in the live view image obtained from the camera are swapped, and the right-eye video and the left-eye video (circular fisheye images) are each transformed into equidistant cylindrical images. Thereafter, the live view display area 1501 displays the live view image obtained after the left and right images are swapped and transformed into equidistant cylindrical images. Specifically, the left-eye video is displayed as an equidistant cylindrical image in the left display area 1501A, and the right-eye video is displayed as an equidistant cylindrical image in the right display area 1501B.

[0195] The various controls to be performed by the system control unit 50 can be performed by one piece of hardware, or the processing can be shared among multiple pieces of hardware (e.g., multiple processors or circuits) to control the entire device. Similarly, the various controls to be performed by the control unit 501 can be performed by one piece of hardware, or the processing can be shared among multiple pieces of hardware (e.g., multiple processors or circuits) to control the entire device.

[0196] The present disclosure has been specifically described according to the preferred embodiments. The present disclosure is not limited to these specific embodiments. The present disclosure includes various modes that do not deviate from the scope of the present disclosure. The above-described embodiments merely show one embodiment of the present invention, and these embodiments can be combined as needed.

[0197] The present disclosure is not limited to cameras or PCs. The present disclosure is applicable to any electronic device capable of processing two images having parallax. For example, the present disclosure is applicable to PDAs, mobile phone units, or portable image viewers, printers, digital photo frames, music players, video game consoles, and e-book readers. The present disclosure is also applicable to, for example, video players, display devices (including projectors), tablet computers, smart phones, AI speakers, household appliances, and in-vehicle units.

[0198] The present disclosure can provide a display such that the positions of two images captured by a lens unit having two optical systems are aligned with the two optical systems.

[0199] Other embodiments

[0200] Embodiments of the present invention can also be implemented by a method in which software (program) that executes the functions of the above-described embodiments is provided to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0201] Although the present disclosure has been described with reference to exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An information processing apparatus, comprising: an acquisition unit configured to acquire an image and correction information regarding a first optical system and a second optical system, the image including a first image region and a second image region, the first image region corresponding to a first optical image input via the first optical system, the second image region corresponding to a second optical image input via the second optical system, and the second optical system having a predetermined parallax with respect to the first optical system; a correction unit configured to perform correction processing based on the correction information to correct positions of pixels included in the first image region and positions of pixels included in the second image region in the image; a generation unit configured to generate a processed image by performing transformation processing on the corrected first image region and the corrected second image region; and a swapping unit configured to perform swapping processing on positions of the first image region and the second image region in the image based on optical axis centers of the first optical system and the second optical system in the image, wherein the first optical system is located on the right side of the lens unit, the second optical system is located on the left side of the lens unit, and in the image before swapping, the upper part of the shooting range is located on the upper side, the first image region is located on the left side, and the second image region is located on the right side.

2. The information processing apparatus according to claim 1, wherein the correction information includes information regarding the optical axis center of the first optical system in the image and information regarding the optical axis center of the second optical system in the image.

3. The information processing apparatus according to claim 1 or 2, wherein The swapping unit designates the first image region in the image based on the optical axis center of the first optical system in the image, designates the second image region in the image based on the optical axis center of the second optical system in the image, and swaps positions of the two designated regions.

4. The information processing apparatus according to claim 1 or 2, wherein the image is an image in which the first image region and the second image region are arranged side by side, and the swapping unit horizontally inverts the positional relationship between the first image region and the second image region in the image.

5. The information processing apparatus according to claim 1 or 2, wherein the first optical system and the second optical system are fish-eye lenses, and the first image region and the second image region are circular fish-eye image regions.

6. The information processing apparatus according to claim 1 or 2, wherein The generation unit transforms each of the corrected first image region and the corrected second image region from a circular fish-eye image region into an equidistant cylindrical image region.

7. The information processing apparatus according to claim 1 or 2, wherein the correction information regarding the first optical system and the second optical system includes: intrinsic parameters of the lens unit, and design parameters for the lens unit including the first optical system and the second optical system, and the correction unit performs correction processing based on the design parameters and the intrinsic parameters of the lens unit.

8. The information processing apparatus according to claim 1 or 2, wherein, The acquisition unit acquires an image file in which correction information regarding the first optical system and the second optical system is added as metadata to data regarding the image.

9. The information processing apparatus according to claim 1 or 2, wherein the image is a live view image output from an imaging device capable of being connected to a lens unit including a first optical system and a second optical system, and the correction information regarding the first optical system and the second optical system is lens information obtained by the imaging device from the lens unit.

10. The information processing apparatus according to claim 1 or 2, wherein the image is an image captured by an imaging device capable of being connected to a lens unit including a first optical system and a second optical system, and the correction information regarding the first optical system and the second optical system is lens information obtained by the imaging device from the lens unit and is added as metadata to the data regarding the image by the imaging device.

11. A control method for an information processing apparatus, comprising: obtaining an image and correction information regarding a first optical system and a second optical system, the image including a first image region and a second image region, the first image region corresponding to a first optical image input via the first optical system, the second image region corresponding to a second optical image input via the second optical system, the second optical system having a predetermined parallax with respect to the first optical system; performing a correction process that corrects positions of pixels included in the first image region and positions of pixels included in the second image region in the image based on the correction information; generating a processed image by performing a transformation process on the corrected first image region and the corrected second image region; and performing a swapping process on positions of the first image region and the second image region in the image based on optical axis centers of the first optical system and the second optical system in the image, wherein the first optical system is located on the right side of the lens unit, the second optical system is located on the left side of the lens unit, and in the image before swapping, the upper part of the shooting range is on the upper side, the first image region is on the left side, and the second image region is on the right side.

12. A computer-readable storage medium storing a program that causes a computer to execute the steps of the control method for the information processing apparatus according to claim 11.

13. A computer program product storing a program that causes a computer to execute the steps of the control method for the information processing apparatus according to claim 11.

Images