Information processing device, control method, program, and storage medium

Abstract

To provide an information processing apparatus for controlling an imaging apparatus that captures an image using a lens unit including two optical systems, allowing a user to appropriately control operations of the imaging apparatus when the user controls the imaging apparatus while displaying a live view image.SOLUTION: An image processing apparatus includes: a communication unit 507 configured to communicate with a camera 100 configured to capture a two-eye image which is one image in which left and right images are arranged; and a control unit 501 configured to display the two-eye image on a display unit 506 and set a position of an enlarged range to which enlargement processing is to be applied by the camera 100 in the image displayed on the display unit 506. The control unit 501 converts the set position of the enlarged range based on a display format in which the two-eye image is displayed on the display unit 506, and outputs an instruction to the camera 100.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method, a program, and a storage medium. [Background technology]

[0002] There is known a technology for capturing two images with parallax using two cameras and displaying the captured images in a stereoscopic manner. Patent Document 1 discloses a camera equipped with a lens unit having two optical systems, which can capture two images with parallax at the same time.

[0003] In addition, live view images captured by a digital camera may be transmitted to an external terminal of the digital camera, and the live view images may be displayed on a display unit of the external terminal. Furthermore, the external terminal may transmit recording start commands and image processing control commands to the digital camera to control the operation of the digital camera. Examples of external terminals that can display images include personal computers, smartphones, and tablets. Users can use these external terminals to check live view images and control the digital camera from a location away from the digital camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-141052 Summary of the Invention [Problem to be solved by the invention]

[0005] When an image captured using a lens unit with two optical systems (a single image containing two images with parallax) is displayed in the same way as a conventional image, the positional relationship between the two optical systems and the positional relationship between the two images in the single image may be reversed. Therefore, when displaying a live view image acquired from a digital camera on an external terminal and controlling the digital camera, image processing different from that used for images captured with a conventional single lens is required. However, conventional technologies such as Patent Document 1 have not adequately addressed this issue.

[0006] The present invention is an information processing device that controls an imaging device that captures images by attaching a lens unit having two optical systems, and aims to enable the operation of the imaging device to be appropriately controlled when a user controls the imaging device while displaying a live view image. [Means for solving the problem]

[0007] An information processing device according to the present invention comprises a communication means for communicating with an imaging device that captures one third image including a first image corresponding to a first optical image input via a first optical system and a second image corresponding to a second optical image input via a second optical system having a predetermined parallax with respect to the first optical system, a display control means for displaying the third image on a display means, and a setting means for setting the position of a target area to which predetermined image processing is applied by the imaging device for the displayed third image, wherein the setting means converts the position of the target area set for the third image displayed on the display means in accordance with the display format in which the display control means displays the third image on the display means, and the communication means outputs the converted position of the target area to the imaging device. [Effects of the Invention]

[0008] According to the information processing device and the like according to the present invention, when a user controls an imaging device while displaying a live view image, it is possible to appropriately control the operation of the imaging device. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing the overall configuration of a system. [Figure 2] FIG. [Figure 3] FIG. 2 is a block diagram showing the configuration of a camera. [Figure 4] FIG. 2 is a schematic diagram illustrating a configuration of a lens unit. [Figure 5] FIG. 2 is a block diagram showing the configuration of a PC. [Figure 6] 10 is a flowchart showing the operation of the camera. [Figure 7] FIG. 10 is a schematic diagram showing an example of lens information acquired from a twin lens. [Figure 8] 10 is a flowchart showing an example of the operation of a PC. [Figure 9] FIG. 10 is a schematic diagram of left-right swapping. [Figure 10] FIG. 10 is a schematic diagram showing an equirectangular transformation including a left-right swap transformation. [Figure 11] FIG. 10 is a schematic diagram showing a display example of a PC live view. [Figure 12] FIG. 10 is a schematic diagram showing a display example of a PC live view when an enlarged image is displayed. [Figure 13] 10 is a flowchart showing a process for displaying an enlarged image. [Figure 14] FIG. 10 is a schematic diagram for explaining an enlargement frame movement command. [Figure 15] 10 is a flowchart showing enlargement frame movement processing in PC live view display that is not an enlarged image. [Figure 16] 10 is a flowchart showing enlargement frame movement processing in PC live view display that displays an enlarged image. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] An embodiment of the present invention will be described. FIGS. 1(a) and 1(b) are schematic diagrams showing an example of the overall configuration of a system according to this embodiment. The system according to this embodiment includes a digital camera (camera) 100 and a personal computer (PC) 500. A lens unit 300 is attached (connected) to the camera 100. Although details of the lens unit 300 will be described later, attaching the lens unit 300 enables the camera 100 to simultaneously capture two images (still images or video) with a predetermined parallax. The PC 500 is an information processing device that handles images captured by an imaging device such as the camera 100. FIG. 1(a) shows a configuration in which the camera 100 and the PC 500 are connected to each other wirelessly or via a wired connection so that they can communicate with each other. FIG. 1(b) shows a configuration in which images captured by the camera 100 are input to the PC 500 on a file-based basis via an external storage device. The external storage device may or may not be connected to both the camera 100 and the PC 500. For example, an external storage device may be connected to the camera 100, and files of images captured by the camera 100 may be stored in the external storage device. Thereafter, the external storage device may be removed from the camera 100 and connected to the PC 500, and the PC 500 may retrieve the files stored in the external storage device.

[0012] 2(a) and 2(b) are external views showing an example of the appearance of the camera 100. Fig. 2(a) is a perspective view of the camera 100 seen from the front side, and Fig. 2(b) is a perspective view of the camera 100 seen from the back side.

[0013] The camera 100 has, on its top surface, a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an extra-viewfinder display 107. The shutter button 101 is an operation member used to issue a shooting preparation instruction or a shooting instruction. The power switch 102 is an operation member used to switch the power of the camera 100 on and off. The mode selector switch 103 is an operation member used to switch between various modes. The main electronic dial 104 is a rotary operation member used to change settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation member used to move the selection frame (cursor), advance images, etc. The video button 106 is an operation member used to issue an instruction to start or stop video shooting (recording). The extra-viewfinder display 107 displays various settings such as shutter speed and aperture.

[0014] The camera 100 has, on its rear surface, a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnify button 113, a playback button 114, a menu button 115, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar 119. The display unit 108 displays images and various information. The touch panel 109 is an operation member that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation member that includes keys (four-way keys) that can be pressed up, down, left, and right. Processing can be performed according to the position of the directional keys 110 that is pressed. The SET button 111 is an operation member that is pressed mainly to confirm a selection item. The AE lock button 112 is an operation member that is pressed to fix the exposure state in a shooting standby state. The magnify button 113 is an operation member that switches the magnify mode on and off in the live view display (LV display) of the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is used to enlarge the playback image or increase the magnification in playback mode. The playback button 114 is an operating member for switching between the shooting mode and playback mode. In the shooting mode, pressing the playback button 114 switches to the playback mode, and the most recent image recorded on the recording medium 227 (described later) can be displayed on the display unit 108.

[0015] Menu button 115 is an operation member that is pressed to display a menu screen on display unit 108 that allows various settings to be made. The user can intuitively make various settings using the menu screen displayed on display unit 108, direction keys 110, and SET button 111. Eyepiece unit 116 is a portion through which the user places their eye close to eyepiece finder (peek-in type finder) 117 and looks into it. Through eyepiece unit 116, the user can view an image displayed on an EVF 217 (Electronic View Finder) (described later) inside camera 100. Eyepiece detection unit 118 is a sensor that detects whether the user places their eye close to eyepiece unit 116 (eyepiece finder 117).

[0016] The touch bar 119 is a line-shaped touch operation member (line touch sensor) capable of receiving a touch operation. The touch bar 119 is disposed at a position where it can be touched (touched) by the thumb of the right hand when the grip unit 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. That is, the touch bar 119 can be operated while the user places their eye on the eyepiece finder 117, looks through the eyepiece unit 116, and is in a position (shooting posture) where the user is ready to press the shutter button 101 at any time. The touch bar 119 can receive tap operations (operations in which the user touches the touch bar 119 and then releases the touched position without moving it within a predetermined period of time), slide operations to the left or right (operations in which the user touches the touch bar and then moves the touched position while still touching it), and the like. The touch bar 119 is an operation member different from the touch panel 109, and does not have a display function. The touch bar 119 functions as, for example, a multi-function bar (M-Fn bar) to which various functions can be assigned.

[0017] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like. The grip section 120 is a holding section formed in a shape that allows the user to easily hold the camera 100 with their right hand when holding the camera 100. The shutter button 101 and the main electronic dial 104 are positioned so that they can be operated with the index finger of the right hand when the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand. Similarly, the sub electronic dial 105 and the touch bar 119 are positioned so that they can be operated with the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section provided on the back side of the camera 100, in a position where it is easy to place the thumb of the right hand that is holding the grip section 120 when none of the operation members are being operated. The thumb rest section 121 is made of a rubber member or the like to enhance holding strength (grip feeling). Terminal cover 122 protects connectors such as connection cables that connect camera 100 to external devices (external apparatuses). Lid 123 protects recording medium 227 and the slot by closing the slot for storing recording medium 227 (described later). Communication terminal 124 is a terminal for communicating with a lens unit (such as lens unit 200 or lens unit 300 (described later)) that is detachable from camera 100.

[0018] Fig. 3 is a block diagram showing an example of the configuration of camera 100. Note that the same components as those in Fig. 2 are assigned the same reference numerals as in Fig. 2, and descriptions of those components will be omitted as appropriate. In Fig. 3, lens unit 200 is attached to camera 100.

[0019] First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100. The lens unit 200 is a single lens and is an example of a normal lens. The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, etc.

[0020] The aperture 201 is configured to have an adjustable aperture diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 50 (described later). The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203, and adjusts the focus by changing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication is performed via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100.

[0021] Next, a description will be given of the camera 100. The camera 100 has a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0022] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an imaging element (image sensor) composed of a CCD, CMOS, or other element that converts an optical image into an electrical signal. The imaging unit 211 may also have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data from the A / D converter 212 or data from the memory control unit 213. The image processing unit 214 also performs predetermined arithmetic processing using captured image data, and the system control unit 50 controls exposure and distance measurement based on the obtained arithmetic results. This processing allows for TTL (through-the-lens) AF processing, AE (auto-exposure) processing, EF (pre-flash) processing, etc. Furthermore, the image processing unit 214 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs TTL type AWB (auto white balance) processing based on the obtained calculation results.

[0023] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data to be displayed on the display unit 108 and the EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0024] The D / A converter 216 converts the display image data stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display in accordance with the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD or organic EL display. A digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216 and then sequentially transferred to and displayed on the display unit 108 or the EVF 217, thereby performing live view display.

[0025] The system control unit 50 is a control unit including at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 executes programs recorded in the nonvolatile memory 219 to realize each process in the flowcharts described below. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0026] The camera 100 also includes a system memory 218 , a nonvolatile memory 219 , a system timer 220 , a communication unit 221 , an attitude detection unit 222 , and an eye proximity detection unit 118 .

[0027] The system memory 218 may be, for example, a RAM. Constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are loaded into the system memory 218. The nonvolatile memory 219 is an electrically erasable and recordable memory, and may be, for example, an EEPROM. The nonvolatile memory 219 stores constants, programs, and the like for the operation of the system control unit 50. The programs referred to here are programs for executing flowcharts, which will be described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of a built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 221 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive images and various other information from external devices. The orientation detection unit 222 detects the orientation of the camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 222, it is possible to determine whether an image captured by the imaging unit 211 was captured with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image according to the detected orientation. The orientation detection unit 222 can use, for example, an acceleration sensor or a gyro sensor. The orientation detection unit 222 can also be used to detect movement of the camera 100 (panning, tilting, lifting, whether the camera is stationary, etc.).

[0028] The eyepiece detection unit 118 can detect the approach of an object to the eyepiece unit 116 (eyepiece finder 117). For example, an infrared proximity sensor can be used as the eyepiece detection unit 118. When an object approaches, infrared light emitted from a light-emitting unit of the eyepiece detection unit 118 is reflected by the object and received by a light-receiving unit of the infrared proximity sensor. The distance from the eyepiece unit 116 to the object can be determined based on the amount of received infrared light. In this way, the eyepiece detection unit 118 performs eyepiece detection, which detects the proximity of the object to the eyepiece unit 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (approach) and departure (away from) of the eye (object) from the eyepiece unit 116. When an object is detected approaching within a predetermined distance from the eyepiece unit 116 from a non-eyepiece state (non-approach state), it is detected that the eye has been placed in proximity. On the other hand, when an object whose proximity has been detected moves away from the eye-closed state (approaching state) by a distance greater than a predetermined distance, it is detected that the eye has been moved away. The threshold for detecting eye-closedness and the threshold for detecting eye-away may be different, for example, by providing hysteresis. Furthermore, after detecting eye-closedness, the eye-closed state is maintained until eye-away is detected. After detecting eye-away, the non-eye-closed state is maintained until eye-closedness is detected. The system control unit 50 switches the display unit 108 and the EVF 217 between display (display state) and non-display (non-display state) depending on the state detected by the eye-closedness detection unit 118. Specifically, when at least in a shooting standby state and the display destination switching setting is automatic switching, the display unit 108 is set as the display destination and the display is turned on, and the EVF 217 is hidden, when the eye is not placed near the object. Furthermore, when the eye is placed near the object, the EVF 217 is set as the display destination and the display is turned on, and the display unit 108 is hidden. The eye proximity detector 118 is not limited to an infrared proximity sensor, and other sensors may be used as the eye proximity detector 118 as long as they can detect a state that can be regarded as eye proximity.

[0029] The camera 100 also has an outside-viewfinder display unit 107, an outside-viewfinder display drive circuit 223, a power supply control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and the like.

[0030] The viewfinder display unit 107 is driven by an outside-viewfinder display drive circuit 223 and displays various settings of the camera 100, such as shutter speed and aperture. The power supply control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and other components, and detects whether a battery is installed, the battery type, and the remaining battery charge. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50 to supply the required voltage for the required period to each component, including the recording medium 227. The power supply unit 225 may be a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The recording medium I / F 226 is an interface with a recording medium 227, such as a memory card or a hard disk. The recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be detachable from the camera 100 or may be built into the camera 100.

[0031] The operation unit 228 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode selector switch 103, the touch panel 109, and other operation units 229. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, and the like.

[0032] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway (a shooting preparation instruction) and outputs a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing. The second shutter switch 231 is turned on when the shutter button 101 is pressed fully (a shooting instruction) and outputs a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processing operations, from reading out a signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227.

[0033] The mode selector switch 103 switches the operation mode of the system control unit 50 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for different capture scenes. The user can directly switch to one of the above-mentioned capture modes using the mode selector switch 103. Alternatively, the user can first switch to a list screen of capture modes using the mode selector switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video capture mode may also include multiple modes.

[0034] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating input coordinates on the touch panel 109 with display coordinates on the display surface of the display unit 108, a GUI (graphical user interface) can be configured that makes it appear as if the user is directly operating the screen displayed on the display unit 108. The touch panel 109 can be any of a variety of types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or an optical sensor type. Depending on the type, there are types that detect a touch by contact with the touch panel 109, and types that detect a touch by the approach of a finger or a pen to the touch panel 109, but either type may be used.

[0035] The system control unit 50 can detect the following operations or states on the touch panel 109.

[0036] A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down).

[0037] A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On).

[0038] The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move).

[0039] The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the end of the touch (hereinafter referred to as "touch-up").

[0040] A state in which nothing is touching the touch panel 109 (hereinafter referred to as Touch-Off).

[0041] When a touch down is detected, a touch on is also detected at the same time. After a touch down, a touch on is usually continued to be detected unless a touch up is detected. If a touch move is detected, a touch on is also detected at the same time. Even if a touch on is detected, a touch move is not detected unless the touch position moves. Once it is detected that all fingers or pens that were touching have touched up, a touch off occurs.

[0042] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch-move, the movement direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on changes in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 109 as if flicking it. If a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is then detected, it is determined that a flick has been performed (it can be determined that a flick occurred following a slide operation). Furthermore, when multiple points (for example, two points) are touched together (multi-touch), the touch operation of bringing the touched positions closer together is called pinch in, and when the touch operation of moving the touched positions farther apart is called pinch out. Pinch out and pinch in are collectively called pinch operations (or simply pinch).

[0043] Fig. 4 is a schematic diagram showing an example of the configuration of lens unit 300. Fig. 4 shows a state in which lens unit 300 is attached to camera 100. Note that, of the camera 100 shown in Fig. 4, the same components as those explained in Fig. 3 are given the same reference numerals as in Fig. 3, and explanations of those components will be omitted as appropriate.

[0044] Lens unit 300 is a type of interchangeable lens that can be attached to and detached from camera 100. Lens unit 300 is a twin lens that can capture right and left images with parallax. In this embodiment, lens unit 300 has two optical systems, each of which can capture images over a wide viewing angle range of approximately 180 degrees. Specifically, each of the two optical systems of lens unit 300 can capture images of a subject over a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle). In other words, each of the two optical systems can capture images over the range of the forward hemisphere.

[0045] The lens unit 300 includes a right-eye optical system 301R having a plurality of lenses and a reflecting mirror, a left-eye optical system 301L having a plurality of lenses and a reflecting mirror, and a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. The right-eye optical system 301R includes a lens 302R arranged on the subject side, and the left-eye optical system 301L includes a lens 302L arranged on the subject side. The lenses 302R and 302L face in the same direction, and their optical axes are approximately parallel.

[0046] The lens unit 300 is a twin lens (VR180 lens) for obtaining a VR180 image, which is one of the VR image formats that allows for twin-eye stereoscopic viewing. In this embodiment, the lens unit 300 has fisheye lenses in each of the right-eye optical system 301R and the left-eye optical system 301L that can capture a range of approximately 180 degrees. Note that the range that can be captured by the lenses in each of the right-eye optical system 301R and the left-eye optical system 301L may be approximately 160 degrees, which is narrower than the 180-degree range. The lens unit 300 can form a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L on one or two image sensors of a camera to which the lens unit 300 is attached. An image in which the first image and the second image obtained via the lens unit 300 are arranged side by side is called a twin image.

[0047] The lens unit 300 is attached to the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. In this manner, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0048] In this embodiment, a right image formed via the right-eye optical system 301R and a left image formed via the left-eye optical system 301L are simultaneously (as a set) formed on the imaging unit 211 of the camera 100. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on one imaging element. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. By using the lens unit 300 in this manner, two images with parallax can be simultaneously (as a set) acquired from two locations (optical systems), the right-eye optical system 301R and the left-eye optical system 301L. By dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image with a range of approximately 180 degrees. That is, the user can view a VR180 image in stereo.

[0049] Here, a VR image is an image that can be displayed in VR, as described below. VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera) and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on a display unit. VR images are not limited to still images, but also include videos and live images (images acquired from a camera in almost real time). VR images have an image range (effective image range) of up to 360 degrees horizontally and 360 degrees vertically. VR images also include images with a wider angle of view than the angle of view that can be captured by a normal camera, or an image range that can be displayed at one time on a display unit, even if the field of view is less than 360 degrees horizontally or vertically. An image captured by camera 100 using lens unit 300 described above is a type of VR image. VR images can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display VR images) to "VR view." By displaying a VR image with a 360-degree field of view and changing the orientation of the display device left and right (horizontal rotation direction), the user can view seamless, omnidirectional images left and right.

[0050] VR display (VR view) is a display method (display mode) that displays a VR image with a field of view that corresponds to the orientation of the display device. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by mapping the VR image onto a virtual sphere (distortion correction). VR display also includes "two-eye VR view," which displays a VR image for the left eye and a VR image for the right eye side by side by mapping them onto a virtual sphere. "Two-eye VR display" using a VR image for the left eye and a VR image for the right eye with parallax between them allows for stereoscopic viewing of the VR images. Regardless of the VR display, when a user wears a display device such as an HMD (head-mounted display), the image displayed corresponds to the orientation of the user's face. For example, suppose a VR image is displayed with a field of view centered at 0 degrees left and right (a specific direction, e.g., north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is flipped from this state (for example, the display surface is changed from facing south to facing north), the display range of the same VR image is changed to an image with a field of view centered at 180 degrees left and right (the opposite direction, for example, south) and 90 degrees up and down. In other words, when the user is wearing the HMD and turns their face from north to south (i.e., turns backward), the image displayed on the HMD also changes from a north image to a south image. Note that the VR image captured using the lens unit 300 of this embodiment is an image capturing a range of approximately 180 degrees forward, and no image exists in a range of approximately 180 degrees behind. If such an image is VR displayed and the orientation of the display device is changed to the side where no image exists, a blank area is displayed.

[0051] By displaying VR images in this way, the user can visually experience a sense of immersion as if they were inside the VR image (in the VR space). Note that the method of displaying VR images is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons. Furthermore, during VR display (in the "VR view" display mode), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch-move on the touch panel, dragging with a mouse, pressing directional buttons, etc. Note that a smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0052] FIG. 5 is a block diagram showing an example of the configuration of the PC 500. The control unit 501 is, for example, a central processing unit (CPU) and controls the entire PC 500. The read-only memory (ROM) 502 non-temporarily stores programs and parameters. The random access memory (RAM) 503 temporarily stores programs and data supplied from external devices, etc. The recording medium 504 is a hard disk or flash memory fixed to the PC 500, or an optical disk, magnetic card, optical card, IC card, memory card, etc. that are detachable from the PC 500. Files of images captured by the camera 100 are read from the recording medium 504. The operation unit 505 accepts user operations on the PC 500. Operation members used by the user to perform operations may be buttons or a touch panel provided on the PC 500, or may be a keyboard or mouse detachable from the PC 500. The display unit 506 displays data stored in the PC 500 and data supplied from an external device. The display unit 506 may be a part of the PC 500, or may be a display device separate from the PC 500. The communication unit 507 communicates with external devices such as the camera 100. The system bus 508 connects the components of the PC 500 so that they can communicate with each other.

[0053] Here, the characteristics of a twin-lens image captured by attaching the lens unit 300 (twin lens) will be described. In the case of the lens unit 200 (normal single lens), an image that is inverted vertically and horizontally with respect to the actual appearance (an image rotated 180 degrees) is formed on the imaging unit 211. Therefore, the entire formed image is rotated 180 degrees to acquire (capture) an image that matches the actual appearance. On the other hand, in the case of the lens unit 300 (twin lens), the right and left images are each rotated 180 degrees with respect to the actual appearance and formed on the imaging unit 211. The arrangement of the right and left images is not particularly limited, but in this embodiment, it is assumed that the right image is formed on the right side and the left image is formed on the left side on the imaging unit 211. Then, as in the case of the lens unit 200 (normal single lens), if the entire formed image (an image including the right and left images) is rotated 180 degrees, the right and left images can be aligned with the actual appearance, but the positions of the right and left images will be swapped. In other words, the left-right positional relationship is reversed, and an image is captured in which the right image is positioned on the left side and the left image is positioned on the right side. Therefore, even if the captured image is displayed as is (without taking into account the positional swap), it cannot be viewed stereoscopically. In this embodiment, such an image can be viewed stereoscopically.

[0054] Control in this embodiment will be described below. An example will be described in which the camera 100 and the PC 500 are connected to each other so that they can communicate with each other, a live view image captured by the camera 100 is transmitted to the PC 500, and the PC 500 displays the live view image on the display unit 506.

[0055] FIG. 6 is a flowchart showing an example of the operation of the camera 100. This operation is an operation of transmitting a live view image from the camera 100 in order to display the live view image on the PC 500. This operation is realized by the system control unit 50 expanding a program recorded in the non-volatile memory 219 into the system memory 218 and executing it. For example, when the camera 100 starts up, the operation of FIG. 6 begins. The operation of FIG. 6 is an operation for a function (PC live view) that displays a live view image captured by the camera on the display unit of the PC. The operation of FIG. 6 is executed when the camera 100 is in a shooting standby state. If a recording start instruction is input from the PC 500 during the PC live view operation, still image shooting or video shooting is executed.

[0056] In step S601, the system control unit 50 determines whether the camera 100 is compatible with a twin lens (for example, the lens unit 300). For example, the system control unit 50 determines whether the firmware version of the system control unit 50 is compatible with a twin lens. If it is determined that the camera 100 is compatible with a twin lens, the process proceeds to step S602; otherwise, the process proceeds to step S611.

[0057] In step S602, the system control unit 50 determines whether or not a twin lens is attached to the camera 100. If it is determined that a twin lens is attached, the process proceeds to step S603; otherwise, the process proceeds to step S611. Note that if a twin lens is attached when no twin lens was attached, the process also proceeds to step S603. If a twin lens is attached but the twin lens is removed, the process proceeds to step S611.

[0058] In step S603, the system control unit 50 acquires design values ​​of the attached (connected) twin lens from the twin lens. The design values ​​are design parameters and are used for left-right swapping and equirectangular conversion, which will be described later. For example, the image circle position, image circle diameter, angle of view, and distortion correction coefficient shown in FIG. 7(b) are acquired.

[0059] In step S604, the system control unit 50 acquires individual values ​​of the attached (connected) twin lens from the twin lens. The individual values ​​are parameters specific to the lens unit, such as manufacturing errors. For example, the image circle position deviation, optical axis tilt, and image magnification deviation shown in FIG. 7(b) are acquired. By using the individual values, image processing can be performed with higher accuracy than when using only design values.

[0060] The lens information acquired from the lens unit 300 will be described.

[0061] FIG. 7(a) is a schematic diagram showing an example of lens information acquired from a twin lens. 1. Lens design values 2. Lens individual value 3. Lens Flag 4. Lens focal length 5. Lens temperature Includes:

[0062] Lens design values ​​are design values ​​for performing aberration correction. During the manufacturing process of the twin lenses, errors such as lens decentering and tilt occur in each of the two optical systems (the left-eye optical system 301L and the right-eye optical system 301R). If left-right swapping or equirectangular conversion is performed without taking these errors into consideration, the quality of the twin-eye VR display will deteriorate, making good stereoscopic viewing difficult. Lens individual values ​​are the measurement results of errors detected during the manufacturing process of the twin lenses. Details of the lens design values ​​and lens individual values ​​will be described later using Figure 7(b).

[0063] The lens flag is a flag indicating that the lens is a twin lens. The lens focal length is the distance from the "principal point" (center of the lens) to the image sensor (image formation position). The lens focal length may or may not be a parameter common to the two optical systems of the twin lens (left eye optical system 301L and right eye optical system 301R). A detailed (high-precision) lens focal length is required to perform left-right swapping and equirectangular conversion with high precision to display high-quality twin-lens VR images. The lens temperature is the temperature of the twin lenses and is used to understand the environmental temperature at the time of shooting, etc.

[0064] 7B is a schematic diagram showing details of the lens design values ​​and lens individual values. In this embodiment, the lens design values ​​and lens individual values ​​are used for left-right swapping and equirectangular conversion.

[0065] The lens design value is 1. Image circle position 2. Image circle diameter 3. Angle of View 4. Distortion correction coefficient Includes:

[0066] The image circle position is the coordinate of the optical axis center of the optical system in the captured image, and is prepared for each of the two optical systems of the twin lens (the left-eye optical system 301L and the right-eye optical system 301R). In other words, the image circle position is the center coordinate of the image circle (circular fisheye image) formed on the imaging element, and is prepared for each of the right and left images. The origin of the coordinates is, for example, the center of the imaging element (the center of the captured image). The image circle position includes horizontal and vertical coordinates. Note that various information related to the optical axis center of the optical system in the captured image can be used as the image circle position. For example, the distance from a predetermined position in the image (such as the center or the upper left corner) to the optical axis center can be used.

[0067] The image circle diameter is the diameter of the image circle (circular fisheye image) formed on the image sensor. The angle of view is the angle of view of the image circle (circular fisheye image) formed on the image sensor. The distortion correction coefficient is the ratio of the design image height to the ideal image height of the lens. A distortion correction coefficient may be set for each image height, and for image heights for which no distortion correction coefficient is set, the distortion correction coefficient may be calculated by interpolation using multiple distortion correction coefficients. A polynomial that approximates the relationship between image height and distortion correction coefficient may be set. The image circle diameter, angle of view, and distortion correction coefficient may or may not be parameters common to the two optical systems of the twin lens (the left-eye optical system 301L and the right-eye optical system 301R).

[0068] When displaying a circular fisheye image, the PC 500 may display a magic window on the circular fisheye image. The magic window is a display item that indicates an area that is (initially) cropped for single-lens VR display. For example, the magic window is displayed based on the image circle position, image circle diameter, and angle of view. This improves the display quality of the magic window. To properly display the magic window, the PC 500 edits and uses the image circle position, image circle diameter, and angle of view as appropriate. For example, the PC 500 multiplies the image circle position or image circle diameter by a coefficient.

[0069] Lens individual value is 5. Image circle misalignment 6. Optical axis tilt 7. Image magnification deviation This information is prepared by measuring each of the two optical systems (left eye optical system 301L and right eye optical system 301R) of the twin lenses.

[0070] Image circle position deviation is the deviation of the center coordinates of the image circle (circular fisheye image) formed on the image sensor from the design value. For example, image circle position deviation includes horizontal and vertical deviations. With the design value coordinates (two-dimensional coordinates including horizontal and vertical coordinates) as the origin, the horizontal coordinate indicates the horizontal deviation, and the vertical coordinate indicates the vertical deviation. Optical axis tilt is the deviation of the direction of the optical axis on the subject side from the design value. For example, optical axis tilt includes horizontal and vertical deviations. Deviations in each direction are expressed as angles. Image magnification deviation is the deviation of the size of the image circle (circular fisheye image) formed on the image sensor from the design value. This deviation is expressed, for example, as a ratio to the design value.

[0071] Note that the information included in the lens information is not limited to the information described above. For example, the lens information may include the boundary positions of the right and left images in the captured image. The boundary positions are, for example, the positions of the edges of a circular fisheye image, and are positions indicated by offset amounts 905, 906, 909, 910, etc. in FIG. 9 (described later). The lens information may also include the coordinates of the midpoint between the right and left images in the captured image. In many cases, the coordinates of the midpoint coincide with the center coordinates of the captured image. The lens information may also include information indicating the area of ​​the magic window (e.g., the coordinates of the upper left corner of the magic window, the width of the magic window, and the height of the magic window). The lens information may also include correction data (e.g., correction values ​​obtained by calibrating the twin lenses) for improving the accuracy of left-right swapping, equirectangular transformation, etc.

[0072] In step S605, the system control unit 50 detects that the camera 100 has been connected to the PC 500. In step S606, the system control unit 50 receives a PC live view start request from the PC 500. In step S607, the system control unit 50 receives a live view image request from the PC 500. The live view image request includes information (resolution information) that specifies the resolution of the live view image to be transmitted, as will be described later. The system control unit 50 executes the process of step S609 to transmit a live view image of the specified resolution to the PC 500.

[0073] In step S608, the system control unit 50 converts the information (twin-lens lens information) acquired in steps S603 and S604 to fit the coordinate system of the live view image. Because the resolution and other aspects of the captured image (the image recorded in the image file) and the live view image are different, the information acquired in steps S603 and S604 cannot be used as is for image processing of the live view image. For this reason, in this embodiment, the lens information is converted to information that fits the coordinate system of the live view image.

[0074] In step S609, the system control unit 50 transmits the lens information converted in step S608 and the live view image to the PC 500. The system control unit 50 converts the resolution of the live view image based on the resolution information acquired in step S607 and transmits the image to the PC 500. In this embodiment, the system control unit 50 of the camera 100 converts the lens information, but the control unit 501 of the PC 500 may convert the lens information. In this case, the system control unit 50 transmits the unconverted lens information and parameters required for converting the lens information to the PC 500. In step S610, the system control unit 50 determines whether to end the PC live view. For example, if the connection between the camera 100 and the PC 500 is terminated or if the user instructs the camera 100 or the PC 500 to end the PC live view, the system control unit 50 determines to end the PC live view. If it is determined to end the PC live view, the operation of FIG. 6 is terminated; otherwise, the system control unit 50 proceeds to step S607.

[0075] If a single lens is attached to the camera 100, the process of step S611 is performed. In step S611, the system control unit 50 transmits a live view image captured by the single lens to the PC 500. The process of step S611 is similar to conventional processing for transmitting a live view image captured by a single lens to an external device, and therefore a detailed description thereof will be omitted. In this embodiment, when transmitting a live view image captured by the single lens to the PC 500, the system control unit 50 neither acquires information about the attached single lens (such as design values ​​and individual values) from the single lens nor transmits it to the PC 500.

[0076] Fig. 8 is a flowchart showing an example of the operation of PC 500. This operation is a control for PC 500 to execute PC live view display, which displays a live view image of camera 100. This operation is realized by control unit 501 expanding a program (application program) recorded in ROM 502 into RAM 503 and executing it. For example, when a user instructs PC 500 to start a specific application, the operation of Fig. 8 starts. The operation of Fig. 8 is an operation for a function (PC live view) that displays a live view image captured by the camera on the display unit of the PC.

[0077] In step S801, a camera (for example, camera 100) is connected to the PC 500, and the control unit 501 detects that the camera has been connected to the PC 500.

[0078] In step S802, the control unit 501 determines whether the camera connected in step S801 is compatible with twin lenses (for example, the lens unit 300). For example, the control unit 501 acquires model information of the camera from the connected camera, and determines whether the camera is compatible with twin lenses based on the acquired model information. If it is determined that the camera is compatible with twin lenses, the process proceeds to step S803; otherwise, the process proceeds to step S821. A camera that is compatible with twin lenses is, for example, a camera that can be fitted with twin lenses.

[0079] In step S803, the control unit 501 determines whether the firmware of the camera connected in step S801 is compatible with twin lenses. For example, the control unit 501 acquires firmware version information of the connected camera from the camera, and determines, based on the acquired information, whether the firmware version of the connected camera is compatible with twin lenses. If it is determined that the firmware is compatible with twin lenses, the process proceeds to step S804; otherwise, the process proceeds to step S821.

[0080] Even if a camera that supports twin lenses is connected to PC 500, the connected camera may not be compatible with twin lenses due to reasons such as an old firmware version. This necessitates the processing of step S803. Also, various cameras can be connected to PC 500, and a camera that does not support twin lenses may be connected regardless of the firmware version. This necessitates the processing of step S802 before the processing of step S803.

[0081] In step S804, the control unit 501 determines whether or not a twin lens is attached to the camera connected in step S801. If it is determined that a twin lens is attached, the process proceeds to step S805; otherwise, the process proceeds to step S821.

[0082] In step S805, the control unit 501 transmits a PC live view start request to the camera connected in step S801.

[0083] In step S806, the control unit 501 determines whether or not to perform circular fisheye display. If it is determined that circular fisheye display is to be performed, the process proceeds to step S807; otherwise (if equirectangular display is to be performed), the process proceeds to step S814. In step S806, for example, whether or not to perform circular fisheye display is determined based on whether the radio button 1105 in FIGS. 11(a) to 11(c) is selected or unselected. In FIGS. 11(a) and 11(b), the radio button 1105 is selected, and in FIG. 11(c), the radio button 1105 is unselected. If the radio button 1105 is selected, it is determined that circular fisheye display is to be performed, and the process proceeds to step S807. If the radio button 1105 is unselected, the process proceeds to step S814.

[0084] In step S807, the control unit 501 transmits a live view image request to the camera connected in step S801. In this embodiment, the live view image request in step S807 is a request for a live view image in normal resolution. The normal resolution is, for example, 4K resolution.

[0085] In step S808, the control unit 501 receives, from the camera connected in step S801, a live view image captured by the camera and lens information of the twin lenses attached to the camera. The resolution of the live view image received in step S808 is normal resolution. The lens information received in step S808 is information converted to match the received live view image (for example, the lens information converted in step S608 of FIG. 6).

[0086] In step S830, it is determined whether the PC live view image acquired from camera 100 is a full image or an enlarged image. If it is an enlarged image, in step S831 a display sequence for the enlarged image is executed. Details of step S831 will be described later. If the PC live view image acquired from camera 100 is a full image, the process proceeds to step S809.

[0087] In step S809, the control unit 501 determines whether or not to swap the left and right. If it is determined that the left and right should be swapped, the process proceeds to step S810; if not, the process proceeds to step S812. In step S809, the control unit 501 determines whether or not to swap the left and right based on, for example, whether or not the check box 1107 in FIG. 11 is checked. If the check box 1107 is checked, the control unit 501 determines that the left and right should be swapped, and the process proceeds to step S810. If the check box 1107 is not checked, the process proceeds to step S812.

[0088] In step S810, the control unit 501 generates a processed live view image by swapping the positions of the right and left images in the live view image acquired in step S808 based on the lens information acquired in step S808 (left-right swap). The control unit 501 generates a processed image by swapping the positions of the right and left images in the live view image based on the center coordinates (the centers of the optical axes of the left eye optical system 301L and the right eye optical system 301R) included in the lens information received together with the live view image.

[0089] The left-right swapping process will be described in detail. The control unit 501 acquires center coordinates (the optical axis centers of the left-eye optical system 301L and the right-eye optical system 301R) from lens information acquired from the camera 100 along with the live view image. The control unit 501 swaps the positions of the right and left images in the captured image based on the center coordinates to generate a processed image (left-right swapping). For example, the control unit 501 identifies the area of ​​the right image in the captured image based on the center coordinate of the right image, and identifies the area of ​​the left image in the captured image based on the center coordinate of the left image. The control unit 501 then swaps the positions of the two identified areas. In this embodiment, the right and left images are arranged side by side in the captured image, and the left-right positional relationship between the right and left images is reversed by left-right swapping. In order to identify the areas of the right and left images with higher accuracy, the diameters (diameter or radius) of the right and left images may be acquired from information about the twin lenses.

[0090] 9(a) and (b) are schematic diagrams of left-right swapping. Fig. 9(a) shows conventional left-right swapping that does not use information from the twin lenses. Fig. 9(b) shows left-right swapping of this embodiment that uses information from the twin lenses.

[0091] As shown in Figures 9(a) and (b), in an image 901 before left-right swapping, a right image 903, which is a circular fisheye image, is placed on the left side, and a left image 907, which is also a circular fisheye image, is placed on the right side.

[0092] In Figure 9(a), image 901 is divided into a left half image and a right half image at center coordinate 902 of image 901, and the left half image and the right half image are swapped. In other words, the left half image is moved to the right of the right half image. Image 911 is the image after such left-right swapping.

[0093] 9(a), displacement 906 is smaller than displacement 905. That is, in image 901, right image 903 is shifted from the center of the left half of image 901 toward the center of image 901. Similarly, displacement 910 is smaller than displacement 909. That is, in image 901, left image 907 is shifted from the center of the right half of image 901 toward the center of image 901. Therefore, in image 911, center coordinate 913 of left image 907 in the left-right direction is shifted by distance 914 from center coordinate 904, and center coordinate 916 of right image 903 in the left-right direction is shifted by distance 917 from center coordinate 908, making it difficult to obtain good stereoscopic viewing.

[0094] In this embodiment, by using lens information, in the image 837 (FIG. 9(b)) after left and right swapping, the center coordinates of the left image in the left-right direction can be made to coincide with the center coordinates 904, and the center coordinates of the right image in the left-right direction can be made to coincide with the center coordinates 908. As a result, the image 837 can be viewed stereoscopically in a good condition.

[0095] The method of left-right swapping is not limited to the above method. For example, the shift amounts 905, 906, 909, and 910 in FIG. 9(a) may be obtained from information about the twin lenses. When swapping the positions of the right and left images, the right and left images may be positioned so that the obtained shift amount is maintained, and the remaining area may be filled with black or the like. Shift amount 905 is the distance from the left edge of the captured image to the left edge of the right image, and shift amount 906 is the distance from the center of the captured image to the right edge of the right image. When swapping left and right, shift amount 905 is the distance from the left edge of the captured image to the left edge of the left image, and shift amount 906 is the distance from the center of the captured image to the right edge of the left image. Similarly, shift amount 909 is the distance from the right edge of the captured image to the right edge of the left image, and shift amount 910 is the distance from the center of the captured image to the left edge of the left image. When swapping left and right, the displacement amount 909 is the distance from the right edge of the photographed image to the right edge of the right image, and the displacement amount 910 is the distance from the center of the photographed image to the left edge of the right image.

[0096] In step S811, the control unit 501 displays on the display unit 506 the processed live view image generated in step S810.

[0097] In step S812, the control unit 501 displays the live view image acquired in step S808 on the display unit 506. That is, the live view image output from the camera 100 is displayed on the display unit 506 as is.

[0098] In step S813, the control unit 501 determines whether or not to end the PC live view. For example, if the connection between the camera 100 and the PC 500 is released or if the user instructs the camera 100 or the PC 500 to end the PC live view, the control unit 501 determines to end the PC live view. An instruction to end the PC live view is, for example, pressing the end button 1108 in FIGS. 11(a) to 11(c). If it is determined to end the PC live view, the operation in FIG. 8 ends; otherwise, the process proceeds to step S806.

[0099] As described above, when equirectangular display is to be performed, the process proceeds from step S806 to step S814. In step S814, the control unit 501 transmits a live view image request to the camera connected in step S801. In this embodiment, the live view image request in step S814 is a request for a live view image with a low resolution (a resolution lower than the normal resolution). When performing equirectangular display, equirectangular conversion (conversion from a circular fisheye image to an equirectangular image) is required. The higher the resolution of the image to be subjected to equirectangular conversion, the longer the time required for the equirectangular conversion and the greater the delay due to the equirectangular conversion. In this embodiment, a low-resolution live view image is requested to speed up the equirectangular conversion (shorten the time required for the equirectangular conversion). Note that, if the delay due to the equirectangular conversion is within an acceptable range, a normal-resolution live view image may be requested even when performing equirectangular display.

[0100] In step S815, the control unit 501 receives, from the camera connected in step S801, a live view image captured by the camera and lens information of the twin lenses attached to the camera. The live view image received in step S815 has a low resolution. The lens information received in step S815 is information converted to match the received live view image (for example, the lens information converted in step S608 of FIG. 6).

[0101] In step S816, the control unit 501 determines whether or not to swap the left and right. If it is determined that the left and right should be swapped, the process proceeds to step S817; otherwise, the process proceeds to step S819. In step S816, for example, whether or not to swap the left and right is determined based on whether or not the check box 1107 in FIG. 11(c) is checked. If the check box 1107 is checked, it is determined that the left and right should be swapped, and the process proceeds to step S817. If the check box 1107 is not checked, the process proceeds to step S819.

[0102] In step S817, the control unit 501 swaps the positions of the right and left images in the live view image acquired in step S815 based on the lens information acquired in step S815, and converts each of the right and left images into equirectangular images. Conversion to an equirectangular image (equirectangular conversion) is a conversion process in which, like the equirectangular projection of a map, the circular fisheye image is treated as a sphere and converted so that the latitudes (horizontal lines) and longitudes (vertical lines) intersect at right angles. By equirectangular conversion, the circular circular fisheye image is converted into a rectangular equirectangular image.

[0103] The control unit 501 generates a map including pixels of the circular fisheye image and conversion parameters used to render each pixel in the equirectangular image. The map indicates the position in the image before conversion to which each converted pixel corresponds. In this embodiment, a map for equirectangular conversion is generated so that not only can the circular fisheye image be converted into an equirectangular image, but also the positions of the right and left images can be corrected. Note that in this embodiment, the map is generated so that equirectangular conversion and left-right swapping can be performed simultaneously. The control unit 501 may also generate the map based on lens design values ​​corrected using individual values ​​included in lens information received together with the live view image.

[0104] The control unit 501 performs equirectangular transformation using the generated map to generate a processed image. Note that although left-right swapping is described as part of equirectangular transformation, left-right swapping may be performed separately from equirectangular transformation.

[0105] FIG. 10 is a schematic diagram showing equirectangular transformation including left-right swapping transformation according to this embodiment. As shown in FIG. 10, in an image 1001 before equirectangular transformation, a right image 1002, which is a circular fisheye image, is located on the left, and a left image 1005, which is also a circular fisheye image, is located on the right. An image 1008 is an image after equirectangular transformation and includes equirectangular images 1009 and 1010. In this embodiment, a map of equirectangular transformation is generated so that correspondences are made as indicated by arrows 1011 and 1012. In this map, each pixel of the equirectangular image 1009 located on the left corresponds to a position in the left image 1005 located on the right, and each pixel of the equirectangular image 1010 located on the right corresponds to a position in the right image 1002 located on the left. Using this map, the left image 1005 located on the right is transformed into the equirectangular image 1009 located on the left, and the right image 1002 located on the left is transformed into the equirectangular image 1010 located on the right. In other words, not only are the circular fisheye images converted into equirectangular images, but the positions of the right and left images are also swapped, enabling excellent stereoscopic viewing.

[0106] In step S818, the control unit 501 displays on the display unit 506 the processed live view image generated in step S817.

[0107] In step S819, the control unit 501 converts each of the right and left images into an equirectangular image without swapping the positions of the right and left images in the live view image acquired in step S815. In other words, the control unit 501 performs equirectangular conversion without swapping the left and right images to generate a processed live view image.

[0108] In step S820, the control unit 501 displays the processed live view image generated in step S819 on the display unit 506.

[0109] If a single lens is attached to the camera 100, the process of step S821 is performed. In step S821, the system control unit 50 transmits a live view image captured by the single lens to the display unit 506. The process of step S821 is similar to conventional processing in which a PC or the like displays a live view image captured by a single lens, and therefore a detailed description thereof will be omitted.

[0110] In each of steps S810, S817, and S819, the control unit 501 performs image processing on the live view image acquired from the connected camera. In step S813, which follows steps S810, S817, and S819, the control unit 501 determines whether to end the PC live view. If the PC live view is to be continued, the process returns to step S806, which precedes steps S810, S817, and S819. Therefore, in the operation of FIG. 8, there is a possibility that the image processing in any of steps S810, S817, and S819 is repeatedly executed.

[0111] Therefore, in order to speed up image processing, the control unit 501 may record information related to the executed image processing in the RAM 503 and use it the next time image processing is performed. For example, the control unit 501 records the correspondence between pixels before image processing and pixels after image processing (image processing map). The image processing map can continue to be used as long as there is no change in the resolution or lens information of the live view image. When the control unit 501 performs image processing in any of steps S810, S817, and S819, it records the image processing map for that image processing. Then, when the control unit 501 performs the same image processing again, it uses the recorded image processing map to perform the image processing. This makes it possible to speed up image processing.

[0112] 11(a) to 11(c) are schematic diagrams showing an example of an application screen (PC live view display) displayed by the control unit 501 on the display unit 506. A screen 1100 is an application screen (remote live view screen). The screen 1100 includes a live view display area 1101, a guide display area 1102, a guide display area 1103, an operation area 1104, and an end button 1108.

[0113] The live view display area 1101 is an area where a live view image is displayed. The live view display area 1101 is made up of a left display area 1101A and a right display area 1101B. The guide display area 1102 is an area where a character string indicating which of the two optical systems of the twin lens (left eye optical system 301L or right eye optical system 301R) the image displayed in the left display area 1101A is from is displayed. The guide display area 1103 is an area where a character string indicating which of the two optical systems of the twin lens (left eye optical system 301L or right eye optical system 301R) the image displayed in the right display area 1101B is from is displayed. The operation area 1104 is an area where operations related to PC live view are accepted, and radio buttons 1105 and 1106 and a check box 1107 are displayed in the operation area 1104. Radio button 1105 is a radio button that is selected when performing circular fisheye display, and radio button 1106 is a radio button that is selected when performing equirectangular display. When radio button 1105 is selected, radio button 1106 is unselected, and when radio button 1105 is unselected, radio button 1106 is selected. Check box 1107 is a checkbox that is checked when swapping left and right. When check box 1107 is operated, the positions of the right image (right eye image) and the left image (left eye image) in the live view image are swapped, and the character strings displayed in guide display areas 1102 and 1103 are also swapped. End button 1108 is a button for ending PC live view. Enlarge display button 1109 is a button for issuing an instruction to execute enlargement display processing. Frame 1110 is an item (enlargement frame) that indicates the range to be enlarged (enlargement range) on the live view image in the enlargement display processing. The user can operate the position of the frame 1110 in advance to determine the enlargement range, and then press the enlargement display button 1109 to execute the enlargement display, thereby enlarging and displaying the desired range.

[0114] In Figure 11(a), the radio button 1105 for circular fisheye display is selected. The check box 1107 for left-right swapping is not checked. Therefore, the live view image acquired from the camera is displayed as is in the live view display area 1101. Specifically, the right eye video, which is a circular fisheye image, is displayed in the left display area 1101A, and the left eye video, which is also a circular fisheye image, is displayed in the right display area 1101B.

[0115] In Figure 11(b), the radio button 1105 for circular fisheye display is selected, and the check box 1107 for left-right swap is checked. As a result, the positions of the right-eye video and left-eye video in the live-view image acquired from the camera are swapped. The live-view image after the left-right swap is then displayed in the live-view display area 1101. Specifically, the left-eye video, which is a circular fisheye image, is displayed in the left display area 1101A, and the right-eye video, which is a circular fisheye image, is displayed in the right display area 1101B.

[0116] 11(c), a radio button 1106 for performing equirectangular display is selected, and a check box 1107 for performing left-right swapping is checked. As a result, the positions of the right-eye video and left-eye video in the live-view image acquired from the camera are swapped, and the right-eye video and left-eye video (either of which is a circular fisheye image) are each converted into an equirectangular image. The live-view image after left-right swapping and equirectangular conversion is then displayed in the live-view display area 1101. Specifically, the left-eye video, which is an equirectangular image, is displayed in the left display area 1101A, and the right-eye video, which is an equirectangular image, is displayed in the right display area 1101B.

[0117] Note that if the PC live view image acquired from the camera 100 is an enlarged image, a live view image in which only the enlargement frame portion of the circular fisheye image is cropped is sent, and therefore the image information necessary for equirectangular conversion display cannot be obtained. For this reason, if equirectangular conversion display is in effect, the enlargement display button 1109 is disabled to prohibit enlargement display. Alternatively, when the enlargement display button 1109 is pressed, a forced switch to circular fisheye display occurs, and the enlargement display is switched to.

[0118] <Explanation about Live View magnification display control> When a live view display is being performed on the display unit 506 using PC live view, an application on the PC 500 can transmit an instruction to enlarge the twin-lens image to the camera 100, thereby realizing live view display of the enlarged image. In this case, the camera 100 executes the image enlargement process. The PC 500 receives the live view image to which the enlargement process has been applied and displays it on the display unit 506. The PC 500 notifies the camera 100 of the enlargement process execution and cancellation, and can also set the range of the twin-lens image to which the camera 100 applies the enlargement process (enlargement range). The enlargement range is set, for example, by moving an enlargement frame 810 on the live view image displayed on the display unit 506 in response to a user operation. The control unit 501 of the PC 500 transmits information indicating the position of the enlargement frame 810 to the camera 100, thereby notifying the camera 100 of the enlargement range.

[0119] In the PC live view display, as described above, the left-eye video and the right-eye video input from the left and right optical systems of the lens unit 300 are reversed in the left-right arrangement in the twin-eye image, so the control unit 501 can display the left-eye video and the right-eye video interchangeably. In this case, the coordinates at which the enlargement frame 810 is displayed on the screen do not match the coordinates of the camera 100. For example, suppose that the enlargement frame 810 is set on the screen so that the center of the enlargement frame 810 coincides with the center of the left-eye video displayed on the left side. In this case, the center coordinates of the enlargement frame 810 set on the screen correspond to approximately the center of the right-eye video in the coordinate system of the imaging unit 21 of the camera 100. Therefore, simply transmitting information indicating the position of the enlargement frame on the PC live view to the camera 100 may not notify the camera 100 of the enlargement range intended by the user.

[0120] 12 is a diagram showing an example of an application screen displayed on the display unit 506 by the control unit 501 during enlarged display of the PC live view according to this embodiment. The same part numbers are used for parts common to those in FIG. 11, and a description thereof will be omitted.

[0121] The camera 100 outputs an enlarged image obtained by applying enlargement processing to enlarge a portion of the twin-lens image to the PC 500, and the enlarged image is displayed on the display unit 506 of the PC 500. At this time, if only the enlarged image is displayed on the display unit 506, the user cannot determine which area of ​​the twin-lens image the displayed image corresponds to. Therefore, when the enlarged image is transmitted from the camera 100, the PC 500 acquires information indicating the enlargement range of the twin-lens image (the area targeted for the enlargement processing) from the camera 100 and displays a navigation display indicating the enlargement range of the twin-lens image on the display unit 506. Here, the navigation display includes a navigation display area 1211 indicating the area corresponding to the entire twin-lens image and a frame 1214 indicating the enlargement range of the twin-lens image. The navigation display area 1211 includes an area 1212 indicating the left area and an area 1213 indicating the right area when the image before enlargement is displayed in the PC live view display. By checking the navigation display, the user can determine which area of ​​the twin-lens image the displayed enlarged image corresponds to.

[0122] FIG. 12(a) shows a state in which the enlarged display button 1109 on the application is pressed while the live view image of FIG. 11(a) is being displayed, resulting in a transition to an enlarged display state.

[0123] Guide display area 1202 is an area that displays information indicating an image corresponding to the enlarged image being displayed, similar to guide display section 1102 for the left video and guide display section 1103 for the right video. Guide display section 1102 displays character strings such as "left eye video" and "right eye video" according to the target to be enlarged.

[0124] The reduced display button 1209 is an operation member for ending the enlarged display and transitioning to the full display.

[0125] The left / right switching button 1210 is a button for inputting an instruction to switch the currently enlarged and displayed image between the left eye image and the right eye image to the other image to be enlarged. When the left / right switching button 1210 is pressed, the image to be enlarged is switched between the left and right images, and the enlargement range is controlled so that the range corresponding to the enlargement range before the switch is set.

[0126] The navigation display area 1211 is an area that displays a GUI (navigation display) that indicates the currently enlarged enlargement range of the entire area of ​​the twin-eye image.

[0127] FIG. 12(b) is an enlarged view of the navigation display. The navigation display has areas corresponding to the arrangement of the left eye image and right eye image of the twin-eye image before enlargement. Area 1212 is an area corresponding to the right eye image of the twin-eye image before enlargement. Area 1213 is an area corresponding to the left eye image of the twin-eye image before enlargement. In addition, items are displayed to indicate whether the image displayed in each area corresponds to the left eye image or the right eye image. In this embodiment, the items are character strings such as "left eye image" and "right eye image."

[0128] Furthermore, frame 1214 is an enlargement frame display that indicates the area (enlargement range) of the twin-eye image that is the target of enlargement. The user can recognize whether the enlarged area is the right-eye image or the left-eye image based on the position of frame 1214 on the navigation display. Furthermore, the user can recognize the position of the enlarged area in each image.

[0129] FIG. 13 is a flowchart showing the enlarged image display process executed in step S831.

[0130] In step S1301, the control unit 501 draws an enlarged image acquired from the camera 100 in the live view display area 1101. At this time, the control unit 501 may apply a scaling process to fit the enlarged image to the vertical or horizontal size of the live view display area 1101, and display the enlarged image, or may display the enlarged image at the same size with the image centered.

[0131] In step S1302, the control unit 501 executes navigation display that shows the corresponding area of ​​the enlarged image in the twin-eye image.

[0132] In step S1303, the control unit 501 determines whether left-right swapping is enabled. Left-right swapping is a process of swapping the left eye video and the right eye video in the image before enlargement. Switching between enabling and disabling left-right swapping is controlled depending on whether the check box 1107 is checked. If the control unit 501 determines that left-right swapping is enabled, the process proceeds to step S1304. On the other hand, if the control unit 501 determines that left-right swapping is disabled, the process proceeds to step S1305.

[0133] In step S1304, a navigation display corresponding to the left-right swapped twin-eye image is executed. Figure 12(c) shows the navigation display corresponding to the left-right swapped twin-eye image. At this time, the left and right eyes are aligned correctly, so the guide display is applied as is. However, since the enlargement position is different from the coordinate position of the image sensor, the display of the enlargement position is changed to the coordinate position after the swap.

[0134] In step S1305, the control unit 501 executes navigation display corresponding to a twin-eye image in which the left and right images have not been swapped, as shown in FIG. 12(b). As described above, a twin-eye image captured using twin lenses is arranged in a reversed order of the left-eye image and the right-eye image. A guide display showing the area corresponding to the entire twin-eye image is provided and displayed on the navigation display member 1211. At this time, the enlarged position is displayed as the coordinate position of the image sensor.

[0135] In step S1306, the control unit 501 determines whether the currently displayed enlarged image is a right-eye image. If the enlarged image is a right-eye image, in step S1307, the control unit 501 displays a message indicating that it is a right-eye image in the guide display area 1202. In step S1308, the control unit 501 acquires the position from the center position (coordinates) of the right-eye image to the upper left coordinates of the enlargement frame.

[0136] If it is determined in step S1306 that the enlarged image is not a right-eye image, in step S1309, the control unit 501 displays an indication that it is a left-eye image on the guide display unit 1202. In step S1310, the control unit 501 acquires the position from the center position (coordinates) of the left-eye image to the top left coordinates of the enlargement frame.

[0137] <Controlling the movement of the zoom frame when swapping the left and right eyes> As described above, when an enlarged image is displayed in the PC live view display, the navigation display can inform the user of which area of ​​the twin-lens image the enlarged image corresponds to. The PC 500 of this embodiment further enables the user to specify, on the live view display, the target area (enlargement range) of the enlargement process to be executed by the camera 100. The user can determine the position of the enlargement range by moving the enlargement frame indicating the enlargement range displayed in the PC live view using an operation member (not shown). Simply transmitting information indicating the position of the enlargement frame on the PC live view to the camera 100 poses a problem in that the enlargement range intended by the user cannot be notified to the camera 100.

[0138] As described above, in a twin-eye image transmitted by camera 100, the left-eye video is positioned on the right side of the twin-eye image, and the right-eye video is positioned on the left side of the twin-eye image. In PC live view display, an image may be displayed in which the left-eye video and the right-eye video of the twin-eye image acquired from camera 100 are swapped (left-right swap). In such a case, the user moves an enlargement frame indicating the enlargement range in the displayed image, but the coordinate system of the displayed image differs from the coordinate system of camera 100. Specifically, suppose that the user specifies the enlargement frame in the left area of ​​the coordinate system of the displayed image to specify the left-eye video portion of the image in the PC live view display. Meanwhile, in the coordinate system of the twin-eye image processed by camera 100, the left-eye video exists in the right area of ​​the twin-eye image. Therefore, if the coordinates indicating the position of the enlargement frame in the coordinate system of the PC live view display are directly instructed to camera 100, a position different from the user's intention may be specified.

[0139] In view of these issues, control unit 501 converts position information of the enlargement range set on the PC live view display into position information in the image generated by camera 100 according to the display settings (display mode, display form) of the PC live view, and outputs the converted information to camera 100. That is, when a target area for processing to be performed by camera 100 is set on the live view display, control unit 501 converts the set position of the target area according to the display form of the live view display and sends an instruction to camera 100. This makes it possible to apply processing by camera 100 to the range intended by the user, even if image processing including movement of parts of the image is applied to the captured image (twin-lens image) acquired by camera 100 on the PC live view display.

[0140] As shown in Fig. 11(b), when a live view image is displayed on the PC 500 side with the left and right eyes swapped, the coordinate system of the live view image displayed on the display unit 506 of the PC 500 is different from the coordinate system of the live view image in the camera 100. Fig. 14(a) is a schematic diagram showing the coordinate system of the live view image in the camera 100. As shown in Fig. 14(a), live view display control is performed in a state where the left eye video and the right eye video are not swapped in the coordinate system of the camera 100.

[0141] 14(a) is coordinate system information acquired from the camera 100, and the control unit 501 acquires from the camera a live view coordinate plane corresponding to the size of the camera's image sensor. There is a right eye coordinate area 1422 in the twin lenses located on the left side of the acquired live view coordinate plane, and a left eye coordinate area 1423 in the twin lenses located on the right side of the live view coordinate plane.

[0142] The control unit 501 receives from the camera 100 the PC live view image, along with enlargement frame information indicating the position and size of the enlargement frame 810, and the center positions of the left and right fisheye images of the lens information (center 1420 of the right eye lens coordinate area, center 1421 of the left eye lens coordinate area).

[0143] When the live view display of the PC 500 is enabled to display a live view image, the images are swapped based on the center coordinates of the left eye image and the right eye image, as shown in Figure 11(b), and the live view image is displayed on the display unit 506.

[0144] FIG. 14(b) shows the arrangement of the images displayed on the display unit 506 when the control unit 501 performs live view display with the left and right images swapped. As shown in FIG. 14(b), in the live view coordinate plane of the camera 100, the left-eye image is displayed in the left image display area 1101A centered on the center 1420 of the right-eye lens coordinate area. Also, the right-eye image is displayed in the right image display area 1101B centered on the center 1421 of the left-eye lens coordinate area. That is, with respect to the coordinate system of the camera, the left-right relationship of the images is reversed in the X-axis direction and the live view image is displayed. Further, similarly, the enlargement frame 810 also displays the center positions of the left and right fisheye images (the center 1420 of the right-eye lens coordinate area, the center 1421 of the left-eye lens coordinate area) swapped.

[0145] In a state where the left-right swap is effective like this, as shown by the arrow in FIG. 14(b), it is assumed that a command to move the enlargement frame 810 is given to the PC 500 by the user. If, as in the case where the left-right swap is invalid, the coordinates corresponding to the destination of the enlargement frame display in FIG. 14(b) commanded by the user of the enlargement frame 810 are directly instructed to the camera 100 to move, there will be a problem that the enlargement frame 810 moves to a position unintended by the user.

[0146] When the left-right swap is effective and a command to move the enlargement frame 810 is received from the user, the control unit 501 converts the coordinate position of the destination of the movement of the enlargement frame 810 into the live view coordinate system of the camera 100 shown in FIG. 14(c) and issues a command to move the enlargement frame 810 to the camera 100.

[0147] Note that, in the enlargement area movement control of the live view display commanded by the PC 500 to the camera 100, the form of instructing the camera 100 with the coordinates of the upper left of the enlargement area or the form of instructing the center coordinates of the enlargement area may be used.

[0148] <Example of the movement processing flow of the enlargement frame of the PC live view on the PC> FIG. 15 is an explanatory diagram showing a flowchart in the enlargement frame movement processing in the normal state where the live view of the PC 500 is not enlarged.

[0149] This flowchart begins with a state in which, in live view on the PC 500, an unmagnified twin-lens image is received and the live view image is displayed on the display unit 506.

[0150] In step S1501, the control unit 501 determines whether or not there is an enlargement frame movement command from the user. If there is an enlargement frame movement command, the process proceeds to step S1502; if there is no enlargement frame movement command, the process ends the enlargement frame movement control flow.

[0151] In step S1502, the control unit 501 determines whether the display position of the enlargement frame 810, which has been commanded by the user, is within the display area 1101B of the right-eye image when the left and right images are swapped, as shown in Fig. 14(b). If the position of the enlargement frame 810 is within the display area 1101B of the right-eye image, the process proceeds to step S1504; otherwise, the process proceeds to step S1503.

[0152] In step S1503, the control unit 501 determines whether the display position of the destination enlargement frame 810 that received the user command is within the left-eye image display area 1101A shown in Fig. 14(b). If the destination enlargement frame 810 is within the left-eye image display area 1101A, the process proceeds to step S1507, and if the destination enlargement frame 810 is not within either the left or right circular fisheye image area, the process proceeds to step S1510.

[0153] In step S1504, the control unit 501 determines whether the live view image display is currently being swapped left and right, and if the live view image display is being swapped, the process proceeds to step S1505, and if the live view image display is not being swapped, the process proceeds to step S1561. Note that the control unit 501 determines whether an instruction to swap left and right has been issued based on the selection state of the check box 1107 shown in FIG.

[0154] In step S1505, the control unit 501 calculates the offset coordinates from the center coordinates of the right eye coordinate area 1422 shown in 1420 in FIG. 14(b) to the enlargement frame 810 moved by the user, records them in the RAM 503, and proceeds to step S1506.

[0155] In step S1506, the control unit 501 calculates the destination coordinates by adding the offset coordinates of the enlargement frame 810 moved by the user, which are recorded in RAM 503, to the center coordinates of the left eye coordinate area 1423 shown in 1421 of Figure 14(b), and proceeds to step S1511.

[0156] In step S1507, the control unit 501 determines whether the live view image display is currently being swapped left and right, and if the live view image display is being swapped, proceeds to step S1508, and if the live view image display is not being swapped, proceeds to step S1511.

[0157] In step S1508, the control unit 501 calculates the offset coordinates from the center coordinates of the left eye shown in 1421 in FIG. 14B to the enlargement frame 810 moved by the user, records them in the RAM 503, and proceeds to step S1509.

[0158] In step S1509, the control unit 501 calculates the destination coordinates by adding the offset coordinates of the enlargement frame 810 moved by the user, which are recorded in RAM 503, to the center coordinates of the right eye coordinate area 1422 shown in 1420 in Figure 14(b), and proceeds to step S1511.

[0159] In step S1510, the control unit 501 discards the move command and ends the move operation.

[0160] In step S1511, the control unit 501 commands the camera 100 to move the enlargement frame 810 to the calculated coordinates of the destination of the enlargement frame 810.

[0161] With the above flow, the operation of moving the enlargement frame 810 is completed.

[0162] <Example of the process flow for moving the enlarged position of PC Live View during enlargement> FIG. 16 is a flowchart showing the control when a command to move the enlarged position is received from the user when an enlarged image is being displayed as a live view as shown in FIG.

[0163] This flowchart begins with the PC 500 displaying an enlarged live view image received by the PC 500 on the display unit 506 .

[0164] In step S1601, the control unit 501 determines whether or not there is a command from the user to move the enlarged area. If there is a command to move the area, the process proceeds to step S1602; if there is no command to move the area, the process proceeds to step S1605.

[0165] In step S1602, the control unit 501 determines whether the amount of movement of the enlargement frame exceeds the current image of the eye when the enlargement frame is moved in response to a movement command from the user. If it exceeds the amount, the process proceeds to step S1604; if not, the process proceeds to step S1603.

[0166] In step S1603, the control unit 501 determines whether the enlarged area of ​​the live view has completely protruded outside the currently displayed circular fisheye area of ​​the twin lenses, resulting in a display of only a black area. If it has protruded, the process proceeds to step S1604; if it has not protruded, the process proceeds to step S1607.

[0167] In step S1604, the control unit 501 discards the command to move the enlargement position of the live view display and ends the flow.

[0168] In step S1605, the control unit 501 determines whether or not an instruction to move the enlarged area to the opposite eye image from the current eye image has been input. If an instruction has been input, the process proceeds to step S1606; if not, the flow ends.

[0169] In step S1606, the offset amount of the enlargement frame is added to the center coordinates of the area in the right-eye image or the left-eye image to be enlarged to calculate the coordinates of the destination, and the process proceeds to step S1607.

[0170] In step S1607, the control unit 501 commands the camera 100 to move the enlarged live view display position to the calculated coordinates of the enlarged position.

[0171] With the above flow, the operation of moving the enlargement position during enlargement is completed.

[0172] The various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or the entire device may be controlled by a plurality of pieces of hardware (e.g., a plurality of processors or circuits) sharing the processing. Similarly, the various controls described above as being performed by the control unit 501 may be performed by a single piece of hardware, or the entire device may be controlled by a plurality of pieces of hardware (e.g., a plurality of processors or circuits) sharing the processing.

[0173] Furthermore, although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0174] Furthermore, the present invention is not limited to cameras and PCs, and can be applied to any electronic device that can handle two images with parallax. For example, the present invention can be applied to PDAs, mobile phone terminals, portable image viewers, printers, digital photo frames, music players, game consoles, e-book readers, etc. The present invention can also be applied to video players, display devices (including projectors), tablet terminals, smartphones, AI speakers, home appliances, in-vehicle devices, etc.

[0175] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0176] 100 Digital Camera (Camera) 50 System control section 500 personal computers (PCs) 501 Control section

Claims

1. a communication means for communicating with an imaging device that captures one third image including a first image corresponding to a first optical image input via a first optical system and a second image corresponding to a second optical image input via a second optical system having a predetermined parallax with respect to the first optical system; a display control means for displaying the third image on a display unit; a setting means for setting a position of a target area to which predetermined image processing is to be applied by the imaging device with respect to the displayed third image; and the setting means converts the position of the target area set with respect to the third image displayed on the display unit in accordance with a display mode in which the display control means displays the third image on the display unit; The communication means outputs the converted position of the target area to the imaging device.

1. An information processing device comprising:

2. the display control means displays the third image in one of a plurality of display forms including a first display form in which the positions of the first image and the second image in the third image are swapped, and a second display form in which the positions of the first image and the second image in the third image are not swapped, When the third image is displayed in the first display mode and the position of the target area is set to an area of ​​the third image displayed on the display unit that corresponds to the first image, the setting means converts the third image so that the target area is set to an area of ​​the third image captured by the imaging device that corresponds to the first image.

2. The information processing apparatus according to claim 1, wherein:

3. The setting means converts the coordinates of the target area designated for the image displayed on the display unit in accordance with the display format.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The display control means displays, on the display unit, a display indicating the position of the target area in the third image when an image obtained by applying the predetermined image processing to the third image is displayed on the display unit.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The predetermined image processing is an enlargement process for enlarging a portion of the third image.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. When the display control means displays the third image to which the enlargement processing has been applied, the display control means displays information indicating whether the displayed image is a portion corresponding to the first image or a portion corresponding to the second image.

6. The information processing apparatus according to claim 5,

7. acquiring lens information indicating a first coordinate corresponding to an optical axis center of the first optical system and a second coordinate corresponding to an optical axis center of the second optical system in the third image; The setting means converts the position of the target area based on the position of the target area relative to coordinates corresponding to the first coordinates or the second coordinates in the third image displayed on the display unit.

7. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. further comprising an acquisition unit for acquiring information about the first optical system and the second optical system; The display control means displays, on the display unit, an image obtained by applying first image processing that corrects positions of the first image and the second image in the third image based on the information.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

9. the information includes information about an optical axis center of the first optical system in the third image and information about an optical axis center of the second optical system in the third image; The first image processing is processing of exchanging positions of the first image and the second image in the third image based on optical axis centers of the first optical system and the second optical system in the third image.

9. The information processing apparatus according to claim 8,

10. the information about the first optical system and the second optical system includes design parameters of a lens unit including the first optical system and the second optical system and parameters specific to the lens unit; The first image processing is a processing that is executed based on the design parameters and the inherent parameters.

10. The information processing device according to claim 8, wherein the information processing device is a computer.

11. The third image is an image in which the first image and the second image are arranged side by side, The first image processing is processing for inverting the left-right positional relationship between the first image and the second image in the third image.

11. The information processing device according to claim 8, wherein the information processing device is a computer.

12. the first optical system and the second optical system each include a fisheye lens; The first image and the second image are each a circular fisheye image.

12. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

13. a communication step of communicating with an imaging device that captures one third image including a first image corresponding to a first optical image input via a first optical system and a second image corresponding to a second optical image input via a second optical system having a predetermined parallax with respect to the first optical system; a display control step of displaying the third image on a display unit; a setting step of setting a position of a target area to which predetermined image processing is to be applied by the imaging device with respect to the displayed third image; and the setting step converts a position of the target area set with respect to the third image displayed on the display unit in accordance with a display mode in which the third image is displayed on the display unit in the display control step; The communication step outputs the converted position of the target area to the imaging device.

2. A method for controlling an information processing apparatus comprising:

14. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 12.

15. 13. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

Images