Electronic apparatus, control method, and storage medium

Abstract

To provide a technique for enabling a user to grasp a range to be displayed in reproduction display during live view display.SOLUTION: According to an aspect of the present invention, there is provided an electronic apparatus including an acquisition unit configured to acquire a first live view image corresponding to a first optical image input via a first optical system, and a control unit configured to, when displaying the first live view image, perform control to superimpose and display, on the first live view image, an indicator indicating a portion of an angle of view of the first optical system to be displayed at a time of reproduction display.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a control method, a program, and a storage medium, and more particularly to a control method for displaying an image having a wide image range (wide viewing angle). [Background technology]

[0002] There is a known technology that uses two optical systems to capture two images with parallax (wide-angle images) and display the two images in a stereoscopic manner. A twin-lens camera for capturing two images with parallax has two optical systems facing in the same direction, and can capture two images with parallax at the same time.

[0003] Patent Document 1 discloses a stereoscopic imaging device that uses two imaging units arranged side by side with parallax to capture images of the same subject from left and right viewpoints, thereby obtaining an image for the left eye and an image for the right eye, respectively.

[0004] Patent Document 2 discloses an information processing device that, when displaying an omnidirectional image captured by a surveillance camera and an image cut out from the omnidirectional image side by side, superimposes a frame on the omnidirectional image to indicate which part of the omnidirectional image has been cut out and displayed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2011 / 121840 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-17446 Summary of the Invention [Problem to be solved by the invention]

[0006] Here, it is conceivable that during live view display, the entire image captured by a twin-lens camera or the like is displayed, and during playback display, the two captured images are used for VR display. However, while during live view display, it is possible to know the range that can be captured using the twin-lens camera, it is not possible to know what range will be displayed during playback display.

[0007] An object of the present invention is to provide a technique that makes it possible to grasp the range that will be displayed during playback display during live view display. [Means for solving the problem]

[0008] The electronic device of the present invention is characterized by having an acquisition means for acquiring a first live view image corresponding to a first optical image input via a first optical system, and a control means for controlling, when displaying the first live view image, to superimpose on the first live view image an indicator showing a portion of the angle of view of the first optical system that will be displayed during playback display. [Effects of the Invention]

[0009] According to the present invention, during live view display, it is possible to grasp the range that will be displayed during playback display. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing the overall configuration of a system. [Figure 2] FIG. [Figure 3] FIG. 2 is a block diagram showing the configuration of a camera. [Figure 4] FIG. 2 is a schematic diagram illustrating a configuration of a lens unit. [Figure 5] FIG. 2 is a block diagram showing the configuration of a PC. [Figure 6] 10 is a flowchart showing the operation of the camera. [Figure 7] 3A to 3C are schematic diagrams showing displays in various display modes. [Figure 8]10 is a flowchart showing the operation of the camera. [Figure 9] 10 is a flowchart showing the operation of a PC. [Figure 10] FIG. 10 is a schematic diagram of left-right swapping. [Figure 11] FIG. 2 is a schematic diagram showing lens information and camera information. [Figure 12] FIG. 10 is a schematic diagram illustrating a method for generating a magic window. [Figure 13] FIG. 2 is a schematic diagram of a display screen. [Figure 14] FIG. 2 is a schematic diagram showing a live view display and a playback display. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will be described in detail below with reference to the drawings. FIGS. 1(a) and 1(b) are schematic diagrams showing an example of the overall configuration of a system according to this embodiment. The system according to this embodiment includes a digital camera (camera) 100 and a personal computer (PC) 500. A lens unit 300 is attached (connected) to the camera 100. The lens unit 300 will be described in detail below. By attaching the lens unit 300, the camera 100 can simultaneously capture two images (still images or video) with a predetermined parallax. The PC 500 is an information processing device that handles images captured by an imaging device such as the camera 100. FIG. 1(a) shows a configuration in which the camera 100 and the PC 500 are connected to each other so that they can communicate with each other wirelessly or via a wired connection. FIG. 1(b) shows a configuration in which images captured by the camera 100 and the PC 500 are input to the PC 500 on a file-based basis via an external storage device. The external storage device may or may not be connected to both the camera 100 and the PC 500. For example, an external storage device may be connected to camera 100, and files of images captured by camera 100 may be stored in the external storage device. After that, the external storage device may be removed from camera 100 and connected to PC 500, and the files stored in the external storage device may be imported by PC 500.

[0012] 2(a) and 2(b) are external views showing an example of the appearance of the camera 100. Fig. 2(a) is a perspective view of the camera 100 seen from the front side, and Fig. 2(b) is a perspective view of the camera 100 seen from the back side.

[0013] The camera 100 has, on its top surface, a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an extra-viewfinder display 107. The shutter button 101 is an operation member used to issue a shooting preparation instruction or a shooting instruction. The power switch 102 is an operation member used to switch the power of the camera 100 on and off. The mode selector switch 103 is an operation member used to switch between various modes. The main electronic dial 104 is a rotary operation member used to change settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation member used to move the selection frame (cursor), advance images, etc. The video button 106 is an operation member used to issue an instruction to start or stop video shooting (recording). The extra-viewfinder display 107 displays various settings such as shutter speed and aperture.

[0014] The camera 100 has, on the rear surface thereof, a display unit 108, a touch panel 109, directional keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, and a menu button 115. It also has an eyepiece unit 116, an eyepiece detection unit 118, a touch bar 119, and a display mode switching button 125. The display unit 108 displays images and various information. The touch panel 109 is an operation member that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are keys that can be pressed up, down, left, and right (four-way The direction key 110 is an operation unit composed of a plurality of buttons (arrows, arrows, and arrowheads). Processing can be performed according to the position of the direction key 110 that is pressed. The SET button 111 is an operation member that is pressed mainly to confirm a selection item. The AE lock button 112 is an operation member that is pressed to fix the exposure state in a shooting standby state. The enlargement button 113 is an operation member for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge a playback image or increase the magnification ratio in the playback mode. The playback button 114 is an operation member for switching between the shooting mode and the playback mode. In the shooting mode, pressing the playback button 114 switches to the playback mode, and the most recent image among the images recorded on a recording medium 227 (described later) can be displayed on the display unit 108.

[0015] Menu button 115 is an operation member that is pressed to display a menu screen on display unit 108 that allows various settings to be made. The user can intuitively make various settings using the menu screen displayed on display unit 108, direction keys 110, and SET button 111. Eyepiece unit 116 is a portion through which the user places their eye close to eyepiece finder (peek-in type finder) 117 and looks into it. Through eyepiece unit 116, the user can view an image displayed on an EVF 217 (Electronic View Finder) (described later) inside camera 100. Eyepiece detection unit 118 is a sensor that detects whether the user places their eye close to eyepiece unit 116 (eyepiece finder 117).

[0016] The touch bar 119 is a line-shaped touch operation member (line touch sensor) capable of receiving a touch operation. The touch bar 119 is disposed at a position where it can be touched (touched) by the thumb of the right hand when the grip unit 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. That is, the touch bar 119 can be operated while the user places their eye on the eyepiece finder 117, looks through the eyepiece unit 116, and is in a position (shooting posture) where the user is ready to press the shutter button 101 at any time. The touch bar 119 can receive tap operations (operations in which the user touches the touch bar 119 and then releases the touched position without moving it within a predetermined period of time), slide operations to the left or right (operations in which the user touches the touch bar and then moves the touched position while still touching it), and the like. The touch bar 119 is an operation member different from the touch panel 109, and does not have a display function. The touch bar 119 functions as, for example, a multi-function bar (M-Fn bar) to which various functions can be assigned.

[0017] Display mode switching button 125 is an operating member for switching the display mode. When display mode switching button 125 is pressed, the display mode is switched, and the live view image displayed on display unit 108 and the OSD image (for example, a histogram or a spirit level) superimposed on the live view image, etc. are switched. The user operates display mode switching button 125 to switch the display mode so that the display required for their own shooting is displayed.

[0018] The camera 100 also has a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, etc. The grip section 120 is a holding section formed in a shape that allows the user to easily hold the camera 100 with their right hand when holding the camera 100. When the user holds the camera 100 by gripping the grip section 120 with the little finger, ring finger, and middle finger of their right hand, the shutter button 101 and main electronic dial 104 are located in positions that can be operated with the index finger of the right hand. Similarly, the sub electronic dial 105 and touch bar 119 are located in positions that can be operated with the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section provided on the back side of the camera 100 in a position where it is easy to place the thumb of the right hand that is holding the grip section 120 when none of the operation members are being operated. The thumb rest section 121 is made of a rubber member or the like to enhance holding strength (grip feeling). The terminal cover 122 is used to connect the camera 100 to an external device (external apparatus). The cover 123 protects connectors of connecting cables and the like that are inserted into the camera 100. The cover 123 closes a slot for storing a recording medium 227, which will be described later, thereby protecting the recording medium 227 and the slot. The communication terminal 124 is a terminal for communicating with a lens unit (such as lens unit 200 or lens unit 300, which will be described later) that is detachable from the camera 100.

[0019] Fig. 3 is a block diagram showing an example of the configuration of camera 100. Note that the same components as those in Fig. 2 are assigned the same reference numerals as in Fig. 2, and descriptions of those components will be omitted as appropriate. In Fig. 3, lens unit 200 is attached to camera 100.

[0020] First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that can be attached to and detached from the camera 100. The lens unit 200 is a single lens and is an example of a normal lens. The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, etc.

[0021] The aperture 201 is configured to have an adjustable aperture diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 50 (described later). The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203, and adjusts the focus by changing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication is performed via a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100.

[0022] Next, a description will be given of the camera 100. The camera 100 has a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0023] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an imaging element (image sensor) composed of a CCD, CMOS, or other element that converts an optical image into an electrical signal. The imaging unit 211 may also have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data from the A / D converter 212 or data from the memory control unit 213. The image processing unit 214 also performs predetermined arithmetic processing using captured image data, and the system control unit 50 controls exposure and distance measurement based on the obtained arithmetic results. This processing allows for TTL (through-the-lens) AF processing, AE (auto-exposure) processing, EF (pre-flash) processing, etc. Furthermore, the image processing unit 214 performs predetermined calculation processing using the captured image data, and the system control unit 50 performs TTL type AWB (auto white balance) processing based on the obtained calculation results.

[0024] Image data from the A / D converter 212 is written into the memory 215 via the image processing unit 214 and memory control unit 213. Alternatively, image data from the A / D converter 212 is written into the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, and image data to be displayed on the display unit 108 and EVF 217. The memory 215 stores a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0025] The D / A converter 216 converts the display image data stored in the memory 215 into an analog signal and supplies it to the display unit 108 or the EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 or the EVF 217 performs display in accordance with the analog signal from the D / A converter 216. The display unit 108 or the EVF 217 is, for example, an LCD or organic EL display. A digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216 and then sequentially transferred to and displayed on the display unit 108 or the EVF 217, thereby performing live view display.

[0026] The system control unit 50 is a control unit including at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 executes programs recorded in the nonvolatile memory 219 to realize each process in the flowcharts described below. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0027] The camera 100 also includes a system memory 218 , a nonvolatile memory 219 , a system timer 220 , a communication unit 221 , an attitude detection unit 222 , and an eye proximity detection unit 118 .

[0028] The system memory 218 may be, for example, a RAM. Constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 219, and the like are loaded into the system memory 218. The nonvolatile memory 219 is an electrically erasable and recordable memory, and may be, for example, an EEPROM. The nonvolatile memory 219 stores constants, programs, and the like for the operation of the system control unit 50. The programs referred to here are programs for executing flowcharts, which will be described later. The system timer 220 is a timing unit that measures the time used for various controls and the time of a built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 221 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 221 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive images and various other information from external devices. The orientation detection unit 222 detects the orientation of the camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 222, it is possible to determine whether an image captured by the imaging unit 211 was captured with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image according to the detected orientation. The orientation detection unit 222 can use, for example, an acceleration sensor or a gyro sensor. The orientation detection unit 222 can also be used to detect movement of the camera 100 (panning, tilting, lifting, whether the camera is stationary, etc.).

[0029] The eyepiece detection unit 118 can detect the approach of some object to the eyepiece unit 116 (eyepiece finder 117). For example, an infrared proximity sensor can be used for the eyepiece detection unit 118. When an object approaches, infrared rays projected from the light projecting unit of the eyepiece detection unit 118 are reflected by the object and received by the light receiving unit of the infrared proximity sensor. Depending on the amount of infrared rays received, The distance from the eyepiece unit 116 to an object can be determined. In this way, the eyepiece detection unit 118 performs eyepiece detection, which detects the proximity of an object to the eyepiece unit 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (eyepiece) and departure (eye departure) of an eye (object) from the eyepiece unit 116. When an object approaching within a predetermined distance from the eyepiece unit 116 is detected from a non-eyepiece state (non-approach state), the eyepiece detection unit 118 detects that the eye has been placed near the object. On the other hand, when an object that was detected as approaching from the eyepiece state (approach state) moves away by more than a predetermined distance, the eyepiece detection unit 118 detects that the eye has been removed. The threshold for detecting eyepiece placement and the threshold for detecting eye removal may be different, for example, by providing hysteresis. Furthermore, after detecting eye placement, the eyepiece remains in the eyepiece state until eye removal is detected. After detecting eye removal, the eyepiece remains in the non-eyepiece state until eye placement is detected. The system control unit 50 switches between display (display state) and non-display (non-display state) of the display unit 108 and the EVF 217 depending on the state detected by the eyepiece detection unit 118. Specifically, at least in the shooting standby state, and when the display destination switching setting is automatic switching, the display destination is set to the display unit 108 and the display is turned on while the EVF 217 is not in view while the eyepiece is not in view. Furthermore, when the eyepiece is in view while the eyepiece is in view, the display destination is set to the EVF 217 and the display is turned on while the display unit 108 is in view. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor, and other sensors may be used as the eyepiece detection unit 118 as long as they can detect a state that can be considered as eyepiece proximity.

[0030] The camera 100 also has an outside-viewfinder display unit 107, an outside-viewfinder display drive circuit 223, a power supply control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and the like.

[0031] The viewfinder display unit 107 is driven by an outside-viewfinder display drive circuit 223 and displays various settings of the camera 100, such as shutter speed and aperture. The power supply control unit 224 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and other components, and detects whether a battery is installed, the battery type, and the remaining battery charge. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50 to supply the required voltage for the required period to each component, including the recording medium 227. The power supply unit 225 may be a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter. The recording medium I / F 226 is an interface with a recording medium 227, such as a memory card or a hard disk. The recording medium 227 is a memory card or the like for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The recording medium 227 may be detachable from the camera 100 or may be built into the camera 100.

[0032] The operation unit 228 is an input unit that accepts operations from the user (user operations) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode switching switch 103, the touch panel 109, other operation units 229, etc. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction keys 110, the SET button 111, and the AE lock button 112. It also includes the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, the display mode switching button 125, etc.

[0033] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 is turned on when the shutter button 101 is pressed halfway (a shooting preparation instruction) during operation, and outputs a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing. The second shutter switch 231 is turned on when the shutter button 101 is pressed fully (a shooting instruction) and outputs a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 executes a series of shooting processing from reading out a signal from the imaging unit 211 to generating an image file including a captured image and writing it to the recording medium 227. Get started.

[0034] The mode selector switch 103 switches the operation mode of the system control unit 50 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for different capture scenes. The user can directly switch to one of the above-mentioned capture modes using the mode selector switch 103. Alternatively, the user can first switch to a list screen of capture modes using the mode selector switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video capture mode may also include multiple modes.

[0035] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating input coordinates on the touch panel 109 with display coordinates on the display surface of the display unit 108, a GUI (graphical user interface) can be configured that makes it appear as if the user is directly operating the screen displayed on the display unit 108. The touch panel 109 can be any of a variety of types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or an optical sensor type. Depending on the type, there are types that detect a touch by contact with the touch panel 109, and types that detect a touch by the approach of a finger or a pen to the touch panel 109, but either type may be used.

[0036] The system control unit 50 can detect the following operations or states on the touch panel 109. A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down). A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On). The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the end of the touch (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 109 (hereinafter referred to as Touch-Off).

[0037] When a touch down is detected, a touch on is also detected at the same time. After a touch down, a touch on is usually continued to be detected unless a touch up is detected. If a touch move is detected, a touch on is also detected at the same time. Even if a touch on is detected, a touch move is not detected unless the touch position moves. Once it is detected that all fingers or pens that were touching have touched up, a touch off occurs.

[0038] These operation states and the position coordinates of the finger or pen touching the touch panel 109 are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. Regarding touch-move, the direction of movement of a finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on changes in position coordinates. When a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 109 as if flicking. When a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is detected immediately after that, it is determined that a flick has been performed (it can be determined that a slide operation was followed by a flick). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called a pinch-in, and a touch operation in which the touch positions are moved farther apart is called a pinch-out. Pinch-out and pinch-in are collectively referred to as a pinch operation (or simply pinch).

[0039] Fig. 4 is a schematic diagram showing an example of the configuration of lens unit 300. Fig. 4 shows a state in which lens unit 300 is attached to camera 100. Note that, of the camera 100 shown in Fig. 4, the same components as those explained in Fig. 3 are given the same reference numerals as in Fig. 3, and explanations of those components will be omitted as appropriate.

[0040] Lens unit 300 is a type of interchangeable lens that can be attached to and detached from camera 100. Lens unit 300 is a twin lens that can capture right and left images with parallax. In this embodiment, lens unit 300 has two optical systems, each of which can capture images over a wide viewing angle range of approximately 180 degrees. Specifically, each of the two optical systems of lens unit 300 can capture images of a subject over a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle). In other words, each of the two optical systems can capture images over the range of the forward hemisphere.

[0041] The lens unit 300 includes a right-eye optical system 301R having a plurality of lenses and a reflecting mirror, a left-eye optical system 301L having a plurality of lenses and a reflecting mirror, and a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. The right-eye optical system 301R includes a lens 302R arranged on the subject side, and the left-eye optical system 301L includes a lens 302L arranged on the subject side. The lenses 302R and 302L face in the same direction, and their optical axes are approximately parallel.

[0042] The lens unit 300 is a twin lens (VR180 lens) for obtaining a VR180 image, which is one of the VR (Virtual Reality) image formats that allows for two-eye stereoscopic viewing. In this embodiment, the lens unit 300 has fisheye lenses in each of the right-eye optical system 301R and the left-eye optical system 301L that can capture a range of approximately 180 degrees. Note that the range that can be captured by the lenses in each of the right-eye optical system 301R and the left-eye optical system 301L may be approximately 160 degrees, which is narrower than the 180-degree range. The lens unit 300 can form a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L on one or two image pickup elements of a camera to which the lens unit 300 is attached.

[0043] The lens unit 300 is attached to the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. In this manner, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0044] In this embodiment, a right image formed via the right-eye optical system 301R and a left-eye optical system 301L A right image and a left image formed via the right eye optical system 301R and the left eye optical system 301L are simultaneously (as a set) formed on the imaging unit 211 of the camera 100. That is, two optical images formed by the right eye optical system 301R and the left eye optical system 301L are formed on one imaging element. The imaging unit 211 converts the formed subject image (optical signal) into an analog electrical signal. By using the lens unit 300 in this way, two images with parallax can be simultaneously (as a set) acquired from two locations (optical systems), the right eye optical system 301R and the left eye optical system 301L. By dividing the acquired images into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image with a range of approximately 180 degrees. That is, the user can view a VR180 image in stereo.

[0045] Here, a VR image is an image that can be displayed in VR, as described below. VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera) and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on a display unit. VR images are not limited to still images, but also include videos and live images (images acquired from a camera in almost real time). VR images have an image range (effective image range) of up to 360 degrees horizontally and 360 degrees vertically. VR images also include images with a wider angle of view than the angle of view that can be captured by a normal camera, or an image range that can be displayed at one time on a display unit, even if the field of view is less than 360 degrees horizontally or vertically. An image captured by camera 100 using lens unit 300 described above is a type of VR image. VR images can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display VR images) to "VR view." By displaying a VR image with a 360-degree field of view and changing the orientation of the display device left and right (horizontal rotation direction), the user can view seamless, omnidirectional images left and right.

[0046] VR display (VR view) is a display method (display mode) that displays a VR image with a field of view that corresponds to the orientation of the display device, allowing for a change in display range. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by mapping the VR image onto a virtual sphere (distortion correction). Single-eye VR display is used, for example, when a VR180 video is played and displayed on a PC. Another VR display includes "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side by side by mapping them onto a virtual sphere. Two-eye VR display is used, for example, when a VR180 video is played and displayed on an HMD (head-mounted display). By performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax from each other, it is possible to view these VR images in stereo. Regardless of the VR display, for example, when a user wears a display device such as an HMD, an image with a field of view that corresponds to the orientation of the user's face is displayed. For example, suppose that a VR image is displayed with a field of view centered at 0 degrees left and right (a specific direction, e.g., north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is flipped from this state (e.g., the display surface is changed from facing south to facing north), the display range of the same VR image is changed to an image with a field of view centered at 180 degrees left and right (the opposite direction, e.g., south) and 90 degrees up and down. That is, when a user wearing an HMD turns their face from north to south (i.e., turns backward), the image displayed on the HMD also changes from a north image to a south image. Note that the VR image captured using the lens unit 300 of this embodiment is an image capturing a range of approximately 180 degrees forward (a 180-degree image), and no image exists in a range of approximately 180 degrees behind. If such an image is VR-displayed and the orientation of the display device is changed to a side where no image exists, a blank area is displayed.

[0047] By displaying VR images in this way, the user can visually get the sensation (a sense of immersion) of being inside the VR image (in the VR space). Note that the method of displaying VR images is not limited to changing the orientation of the display device. For example, it is possible to change the orientation of the display device using a touch panel or directional buttons. The display range may be moved (scrolled) in response to user operations via a smartphone, etc. Furthermore, during VR display (display mode "VR view"), in addition to changes in the display range due to posture changes, the display range may also be changed in response to touch-move on a touch panel, dragging with a mouse, pressing a directional button, etc. A smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0048] FIG. 5 is a block diagram showing an example of the configuration of the PC 500. The control unit 501 is, for example, a central processing unit (CPU) and controls the entire PC 500. The read-only memory (ROM) 502 non-temporarily stores programs and parameters. The random access memory (RAM) 503 temporarily stores programs and data supplied from external devices, etc. The recording medium 504 is a hard disk or flash memory fixed to the PC 500, or an optical disk, magnetic card, optical card, IC card, memory card, etc. that are detachable from the PC 500. Files of images captured by the camera 100 are read from the recording medium 504. The operation unit 505 accepts user operations on the PC 500. Operation members used by the user to perform operations may be buttons or a touch panel provided on the PC 500, or may be a keyboard or mouse detachable from the PC 500. The display unit 506 displays data stored in the PC 500 and data supplied from an external device. The display unit 506 may be a part of the PC 500, or may be a display device separate from the PC 500. The communication unit 507 communicates with external devices such as the camera 100. The system bus 508 connects the components of the PC 500 so that they can communicate with each other.

[0049] Here, during live view display, the entire VR image captured by the lens unit 300 (twin lenses) or the like is displayed, and during playback display, only a portion of the VR image is displayed. The playback display is, for example, a VR display, and is performed on a PC, HMD, or the like. FIGS. 14(a) and 14(b) show an example of live view display, and FIG. 14(c) shows an example of playback display. In FIG. 14(a), the entire circular fisheye image (equidistant projection image) that is the VR image is displayed, and in FIG. 14(b), the entire equirectangular image that is the VR image is displayed. The angle of view of the equirectangular image in FIG. 14(b) is the same as the angle of view of the circular fisheye image in FIG. 14(a). In FIG. 14(c), a portion of the VR image shown in FIGS. 14(a) and 14(b) is displayed. For example, the display shown in FIG. 14(c) can be realized by performing equirectangular conversion on the circular fisheye image, then performing perspective projection conversion, and then cropping and displaying a portion of the image after perspective projection conversion. Perspective projection transformation is a correction that brings the appearance of an image closer to how humans actually see it (the real thing), and is a process that reduces image distortion. Equirectangular transformation may be omitted. In two-eye VR display, two images, each a portion of the image after perspective projection transformation, are displayed side by side. For example, when displaying a left image (VR image for the left eye) and a right image (VR image for the right eye) with parallax using an HMD or other device for two-eye VR display, perspective projection transformation is performed on each of the left and right images, and a portion of the left image and a portion of the right image are displayed side by side.

[0050] However, from the live view display such as that shown in FIGS. 14(a) and 14(b), it is possible to know the extent of the angle of view captured, but it is not possible to know the angle of view displayed during playback display. Therefore, in this embodiment, when live view display is performed, it is possible to know the angle of view displayed during playback display. Note that, although an example in which a right image and a left image are captured and displayed as a live view will be described below, it is also possible to display either the right image or the left image as a live view. Furthermore, it is also possible to display a single VR image captured by a single lens as a live view. The VR image displayed as a live view may be a 360° image (omnidirectional image; celestial sphere image) obtained by combining multiple images captured in different directions.

[0051] FIG. 6 is a flowchart showing an example of the operation (shooting mode processing) of the camera 100 in the shooting mode. This operation is performed by the system control unit 50 in accordance with the data stored in the nonvolatile memory 219. This is realized by expanding the program stored in the system memory 218 and executing it. For example, when the camera 100 is started in the shooting mode or when the mode of the camera 100 is switched to the shooting mode, the operation shown in FIG. 6 starts. In the shooting mode process, in addition to the processes described below, still image shooting and video shooting are also performed. In other words, the processes described below are performed while the camera is waiting to shoot or while shooting.

[0052] In step S601, the system control unit 50 obtains information about the display mode that was used last time from the nonvolatile memory 219, and displays an image (such as a live view image or an OSD image) in the last display mode on the display unit 108. The display mode information is not particularly limited as long as it is information that can identify the display mode, but in this embodiment it is assumed to be a number.

[0053] In step S602, the system control unit 50 determines whether or not a display mode switching instruction has been issued by the user of the camera 100. If it is determined that a display mode switching instruction has been issued, the process proceeds to step S603; otherwise, the process proceeds to step S610. The display mode switching instruction is, for example, pressing the display mode switching button 125.

[0054] In step S603, the system control unit 50 determines whether the number N of the current display mode (the display mode before switching in response to the display mode switching instruction in step S602) is 5. If it is determined that the number N is 5, the process proceeds to step S606, and if it is determined that the number N is not 5, the process proceeds to step S604.

[0055] In step S604, the system control unit 50 increments the number N by 1 so that the display modes are switched in order. In this embodiment, it is assumed that there are six display modes numbered 1 to 6.

[0056] In step S605, the system control unit 50 updates the display on the display unit 108 so that an image is displayed on the display unit 108 in the display mode numbered N.

[0057] In step S606, the system control unit 50 determines whether or not a VR lens (a lens that enables VR images to be captured, such as a fisheye lens) such as the lens unit 300 is attached to the camera 100. If it is determined that a VR lens is attached, the process proceeds to step S608; otherwise, the process proceeds to step S607.

[0058] In step S607, the system control unit 50 sets the number N to 1 so as to skip the display mode numbered 6 and transition to the display mode numbered 1.

[0059] In this way, when a VR lens is not attached, the display mode is switched among five display modes numbered 1 to 5, with the display mode number incremented by 1 each time a display mode switching instruction is given. Then, when a display mode switching instruction is given in display mode numbered 5, the display mode is returned to display mode numbered 1.

[0060] FIG. 7(a) shows an example of a display in display mode number 1, FIG. 7(b) shows an example of a display in display mode number 2, and FIG. 7(c) shows an example of a display in display mode number 3. FIG. 7(d) shows an example of a display in display mode number 4, and FIG. 7(e) shows an example of a display in display mode number 5. In FIGS. 7(a) to 7(c), a live view image and display items (OSD image) are displayed, in FIG. 7(d) only a live view image is displayed, and in FIG. 7(e) a menu screen is displayed. The types of display items differ between FIGS. 7(a) to 7(c). The displays in FIGS. 7(a) to 7(e) are similar to conventional displays, and therefore detailed description will be omitted.

[0061] In step S608, the system control unit 50 sets the number N to 6 to transition to the display mode numbered 6. The display mode numbered 6 is a display mode dedicated to the VR lens.

[0062] In step S609, the display on the display unit 108 is updated so that the screen is displayed in the display mode numbered 6.

[0063] In this way, when a VR lens is attached, the display mode is switched among six display modes numbered 1 to 6, with the display mode number incremented by 1 each time a display mode switching instruction is given. Then, when a display mode switching instruction is given in display mode numbered 6, the display mode is returned to display mode numbered 1.

[0064] FIG. 7(f) shows an example of a display in the display mode numbered 6. In FIG. 7(f), a live view image in which a right image and a left image are arranged side by side is displayed. An indicator 701 is displayed superimposed on the right image, and an indicator 702 is displayed superimposed on the left image. In this embodiment, a single image including a right image and a left image is called a live view image. However, each of the right image and the left image may also be called a live view image. The indicator 701 is a display item (frame) indicating a portion of the field of view of the right-eye optical system 301R that is displayed during playback display (during twin-eye VR display). The indicator 702 is a display item (frame) indicating a portion of the field of view of the left-eye optical system 301L that is displayed during playback display (during twin-eye VR display). More specifically, the indicators 701 and 702 indicate portions that are initially displayed in VR display (twin-eye VR display) and that remain displayed unless the user changes their viewing direction (viewpoint). Hereinafter, display items such as the indicators 701 and 702 will be referred to as magic windows. The magic window can be said to be the part of the field of view of the VR image that is displayed when the user looks in a specific direction (for example, the front direction). In this embodiment, the magic window is displayed only in display mode number 6 (a specific display mode).

[0065] By displaying such a magic window, the user can understand the angle of view that will be displayed during playback. Furthermore, when the image is played back on a browser or a head-mounted display (HMD), the user can visually confirm the range that the viewer will first see during shooting, thereby more effectively capturing the user's desired composition and the subject that the user most wants to show. Furthermore, by displaying the right and left images side by side, the user can understand the differences between the right and left images and whether the right and left images were captured without any problems. Furthermore, by displaying the magic window for the right image and the magic window for the left image, the user can understand the differences between the image seen by the right eye and the image seen by the left eye in VR display.

[0066] In FIG. 7(f), indicator 701 indicates the center of the right image, and indicator 702 indicates the center of the left image, but the position of the area indicated by the magic window is not particularly limited. The position of the area indicated by the magic window may be changeable. Only one of indicator 701 and indicator 702 may be displayed. As long as the user can see the area indicated by the magic window, it is not necessarily required that it be displayed at all times during subsequent shooting.

[0067] Furthermore, the method for determining the magic window area is not particularly limited. For example, as shown in FIG. 7(g), a rectangular area showing a part of the image after perspective projection transformation is predetermined as the area (angle of view) that is first displayed in VR display. Then, by applying an inverse transformation of the perspective projection transformation or an inverse transformation of the equirectangular transformation to this rectangular area, the area of ​​the magic window that is superimposed on the live view image before perspective projection transformation is determined.

[0068] In step S610, the system control unit 50 detects whether the VR lens has been removed from the camera 100. If it is determined that the VR lens has been removed, the process proceeds to step S611, and if not, the process proceeds to step S613.

[0069] In step S611, the system control unit 50 determines whether the number N of the current display mode is 6. If it is determined that the number N is 6, the process proceeds to step S612, and if it is determined that the number N is not 6, the process proceeds to step S613.

[0070] In step S612, the system control unit 50 sets the number N to 1 to transition to the display mode with number 1. As described above, the display mode with number 6 is a display mode exclusively for VR lenses. Since it is not appropriate to continue the display mode with number 6 when the VR lens is removed, the system control unit 50 transitions to the display mode with number 1, which is the display mode next to number 6 in the display mode switching order.

[0071] In step S613, the display on the display unit 108 is updated so that the screen is displayed on the display unit 108 in the display mode numbered 1.

[0072] In step S614, the system control unit 50 determines whether an end instruction has been issued by the user of the camera 100. If it is determined that an end instruction has been issued, the operation of FIG. 6 ends; if not, the process proceeds to step S602. An end instruction may be an instruction to turn off the power to the camera 100 or an instruction to switch the mode of the camera 100 from the shooting mode to another mode. In other words, an end instruction may be an instruction to press the power switch 102, the mode selector switch 103, or the like.

[0073] Note that there are no particular limitations on the types and number of display modes that can be switched by a display mode switching instruction. The display modes that can be switched by a display mode switching instruction may be customizable from a menu screen or the like.

[0074] FIG. 8 is a flowchart showing another example of the operation of the camera 100. This operation is realized by the system control unit 50 loading a program recorded in the non-volatile memory 219 into the system memory 218 and executing it. For example, when the camera 100 starts up, the operation of FIG. 8 starts. The operation of FIG. 8 is an operation for a function (PC live view) that displays a live view image captured by the camera on the display unit of a PC. The operation of FIG. 8 is executed when the camera 100 is in a shooting standby state. If a recording start instruction is input from the PC 500 during PC live view operation, still image shooting or video shooting is executed. At this time, PC live view may continue.

[0075] In step S801, the system control unit 50 determines whether the camera 100 is compatible with a twin lens (for example, the lens unit 300). In the operation of FIG. 8, a twin VR lens is assumed as the twin lens. For example, the system control unit 50 determines whether the firmware version of the system control unit 50 is compatible with a twin lens. If it is determined that the camera 100 is compatible with a twin lens, the process proceeds to step S802; otherwise, the process proceeds to step S811.

[0076] In step S802, the system control unit 50 determines whether or not a twin lens is attached to the camera 100. If it is determined that a twin lens is attached, the process proceeds to step S803; otherwise, the process proceeds to step S811. Note that if a twin lens is attached when no twin lens was attached, the process also proceeds to step S803. If the twin lens is removed from a state in which a twin lens was attached and a single lens is attached, the process proceeds to step S811. The operation in FIG. 8 assumes that the single lens is a normal single lens (a single lens that is not a VR lens).

[0077] In step S803, the system control unit 50 acquires design values ​​of the attached (connected) twin lens from the twin lens. The design values ​​are design parameters and are used for left-right swapping and equirectangular conversion, which will be described later. For example, the image circle position, image circle diameter, angle of view, and distortion correction coefficient shown in FIG. 11(b) are acquired.

[0078] In step S804, the system control unit 50 acquires individual values ​​of the attached (connected) twin lens from the twin lens. The individual values ​​are parameters specific to the lens unit, such as manufacturing errors. For example, the image circle position deviation, optical axis tilt, and image magnification deviation shown in FIG. 11(b) are acquired. By using the individual values, image processing can be performed with higher accuracy than when using only design values.

[0079] In step S805, the camera 100 is connected to the PC 500, and the system control unit 50 detects that the camera 100 has been connected to the PC 500. In step S806, the system control unit 50 receives a PC live view start request from the PC 500. In step S807, the system control unit 50 receives a live view image request from the PC 500. The live view image request includes information specifying the resolution of the live view image (resolution information), as will be described later. The system control unit 50 executes the process of step S809 to transmit a live view image of the specified resolution to the PC 500.

[0080] In step S808, the system control unit 50 converts the information acquired in steps S803 and S804 (twin-lens lens information; information about the two optical systems of the twin-lens) so that it matches the coordinate system of the live-view image to be transmitted. Since the information acquired in steps S803 and S804 cannot be used as is for image processing of the live-view image, in this embodiment, the lens information is converted to information that matches the coordinate system of the live-view image. Note that the lens information may include information other than the information acquired in steps S803 and S804. For example, in this embodiment, the lens information includes information about the area (angle of view) of the magic window. The lens information includes information about the magic window to be superimposed on the right image and information about the magic window to be superimposed on the left image.

[0081] In step S809, the system control unit 50 transmits the lens information converted in step S808 and the live view image to the PC 500. The system control unit 50 converts the resolution of the live view image based on the resolution information acquired in step S807 and transmits the converted image to the PC 500. Note that in this embodiment, the system control unit 50 of the camera 100 converts the lens information, but the control unit 501 of the PC 500 may also convert the lens information. In this case, the system control unit 50 transmits the lens information before conversion and parameters necessary for converting the lens information to the PC 500.

[0082] In step S810, the system control unit 50 determines whether to end the PC live view. For example, if the connection between the camera 100 and the PC 500 is released or if the user instructs the camera 100 or the PC 500 to end the PC live view, the system control unit 50 determines to end the PC live view. If it is determined to end the PC live view, the operation in FIG. 8 ends; otherwise, the system control unit 50 proceeds to step S807.

[0083] If a single lens (a normal single lens; a single lens that is not a VR lens) is attached to the camera 100, the process of step S811 is performed. In step S811, the system control unit 50 transmits a live view image captured by the single lens to the PC 500. The process of step S811 is the same as the conventional process of transmitting a live view image captured by a single lens to an external device, so a detailed description will be omitted. In this embodiment, when transmitting a live view image captured by the single lens to the PC 500, the system control unit 50 The information of the single lens (such as design values ​​and individual values) will not be acquired from the single lens or transmitted to PC500.

[0084] Fig. 9 is a flowchart showing an example of the operation of PC 500. This operation is realized by control unit 501 expanding a program (application program) recorded in ROM 502 into RAM 503 and executing it. For example, when a user instructs PC 500 to start a specific application, the operation of Fig. 9 starts. The operation of Fig. 9 is an operation for a function (PC live view) that displays a live view image captured by a camera on the display unit of the PC.

[0085] In step S901, a camera (for example, camera 100) is connected to the PC 500, and the control unit 501 detects that the camera has been connected to the PC 500.

[0086] In step S902, the control unit 501 determines whether the camera connected in step S901 is compatible with a twin lens (for example, the lens unit 300). In the operation of FIG. 9, a twin VR lens is assumed as the twin lens. For example, the control unit 501 acquires model information of the connected camera from the camera, and determines whether the camera is compatible with a twin lens based on the acquired model information. If it is determined that the camera is compatible with a twin lens, the process proceeds to step S903; otherwise, the process proceeds to step S925. A camera compatible with a twin lens is, for example, a camera to which a twin lens can be attached.

[0087] In step S903, the control unit 501 determines whether the firmware of the camera connected in step S901 is compatible with twin lenses. For example, the control unit 501 acquires firmware version information of the connected camera from the camera, and determines, based on the acquired information, whether the firmware version of the connected camera is compatible with twin lenses. If it is determined that the firmware is compatible with twin lenses, the process proceeds to step S904; otherwise, the process proceeds to step S925.

[0088] Even if a camera that supports twin lenses is connected to PC 500, the connected camera may not be compatible with twin lenses due to reasons such as an old firmware version. This necessitates the processing of step S903. Also, various cameras can be connected to PC 500, and a camera that does not support twin lenses may be connected regardless of the firmware version. This necessitates the processing of step S902 before the processing of step S903.

[0089] In step S904, the control unit 501 determines whether or not a twin lens is attached to the camera connected in step S901. If it is determined that a twin lens is attached, the process proceeds to step S905; otherwise, the process proceeds to step S925.

[0090] In step S905, the control unit 501 transmits a PC live view start request to the camera connected in step S901.

[0091] In step S906, the control unit 501 determines whether or not to perform circular fisheye display. If it is determined that circular fisheye display is to be performed, the process proceeds to step S907; otherwise (if equirectangular display is to be performed), the process proceeds to step S916. In step S906, for example, whether or not to perform circular fisheye display is determined based on whether the radio button 1306 in FIGS. 13(a) to 13(d) is selected or unselected. In FIGS. 13(a) and 13(c), the radio button 1306 is selected, and in FIGS. 13(b) and 13(d), the radio button 1306 is unselected. If the radio button 1306 is selected, it is determined that circular fisheye display is to be performed, and the process proceeds to step S907. If the radio button 1306 is unselected, the process proceeds to step S916.

[0092] In step S907, the control unit 501 transmits a live view image request to the camera connected in step S901. In this embodiment, the live view image request in step S907 is a request for a live view image in normal resolution. The normal resolution is, for example, 4K resolution.

[0093] In step S908, the control unit 501 receives, from the camera connected in step S901, a live view image captured by the camera and lens information of the twin lenses attached to the camera. The resolution of the live view image received in step S908 is normal resolution. The lens information received in step S908 is information converted to match the received live view image (for example, the lens information converted in step S808 of FIG. 8).

[0094] In step S909, the control unit 501 determines whether or not to perform left-right swapping, which corrects (swaps) the positions of the right image and the left image. If it is determined that left-right swapping is to be performed, the process proceeds to step S910; otherwise, the process proceeds to step S912. In step S909, the control unit 501 determines whether or not to perform left-right swapping based on, for example, whether or not the check box 1308 in FIGS. 13(a) and 13(c) is checked. If the check box 1308 is checked, it determines that left-right swapping is to be performed, and the process proceeds to step S910. If the check box 1308 is not checked, the process proceeds to step S912.

[0095] Here, the characteristics of an image captured using a twin lens will be described. In the case of a normal single lens, an image that is inverted vertically and horizontally with respect to the actual appearance (an image rotated 180 degrees) is formed on the imaging unit 211. Therefore, the entire formed image is rotated 180 degrees to acquire (capture) an image that matches the actual appearance. On the other hand, in the case of a twin lens, the right and left images are each rotated 180 degrees with respect to the actual appearance and formed on the imaging unit 211. The arrangement of the right and left images is not particularly limited, but in this embodiment, it is assumed that the right image is formed on the right side and the left image is formed on the left side on the imaging unit 211. Then, as in the case of a normal single lens, if the entire formed image (an image including the right and left images) is rotated 180 degrees, the right and left images can be aligned with the actual appearance, but the positions of the right and left images will be swapped. In other words, an image is captured in which the left and right positions are reversed, with the right image positioned on the left side and the left image positioned on the right side. By switching the left and right, the positions of the right and left images can be displayed so as to correspond to the two optical systems (the right eye optical system 301R and the left eye optical system 301L). In the operation of Fig. 6 described above, a live view image in which the left and right images are not switched (a live view image in which the right image is positioned on the left side and the left image is positioned on the right side) is displayed on the display unit 108 of the camera 100.

[0096] In step S910, the control unit 501 swaps the positions of the right and left images in the live-view image acquired in step S908 based on the lens information acquired in step S908 to generate a processed live-view image (left-right swap). The control unit 501 swaps the positions of the right and left images in the live-view image based on center coordinates (the optical axis centers of the left-eye optical system 301L and the right-eye optical system 301R) included in the lens information received together with the live-view image to generate a processed live-view image. For example, the control unit 501 identifies the area of ​​the right image in the live-view image based on the center coordinates of the right image (the optical axis center of the right-eye optical system 301R), and identifies the area of ​​the left image in the live-view image based on the center coordinates of the left image (the optical axis center of the left-eye optical system 301L). The control unit 501 then swaps the positions of the two identified areas. In this embodiment, the right and left images are arranged side by side in the live-view image, and the left-right positional relationship between the right and left images is reversed by the left-right swap. In order to specify the regions of the right and left images with higher accuracy, the diameters (or radii) of the right and left images may be obtained from the information of the two lenses and used.

[0097] Note that the method of left-right swapping is not limited to the above method. For example, the shift amounts 1005, 1006, 1009, and 1010 in FIG. 10(a) may be obtained from information about the twin lenses. When swapping the positions of the right and left images, the right and left images may be positioned so that the obtained shift amount is maintained, and the remaining area may be filled with black or the like. The shift amount 1005 is the distance from the left edge of the live-view image to the left edge of the right image, and the shift amount 1006 is the distance from the center of the live-view image to the right edge of the right image. When swapping left and right, the shift amount 1005 is the distance from the left edge of the live-view image to the left edge of the left image, and the shift amount 1006 is the distance from the center of the live-view image to the right edge of the left image. Similarly, the shift amount 1009 is the distance from the right edge of the live-view image to the right edge of the left image, and the shift amount 1010 is the distance from the center of the live-view image to the left edge of the left image. When swapping left and right, the shift amount 1009 is the distance from the right edge of the live view image to the right edge of the right image, and the shift amount 1010 is the distance from the center of the live view image to the left edge of the right image.

[0098] In step S911, the control unit 501 displays the processed live view image generated in step S910 on the display unit 506.

[0099] In step S912, the control unit 501 displays the live view image acquired in step S908 on the display unit 506.

[0100] In step S913, the control unit 501 generates a magic window (frame) based on the lens information (magic window information) acquired in step S908. In this embodiment, the control unit 501 generates a magic window to be superimposed on the right image based on the information of the magic window to be superimposed on the right image, and generates a magic window to be superimposed on the left image based on the information of the magic window to be superimposed on the left image. Details of the magic window generation method will be described later with reference to FIGS. 12(a) and 12(b).

[0101] In step S914, the control unit 501 displays the magic window generated in step S913 on the display unit 506. Note that when enlarging and displaying a portion of a live view image in response to a user instruction, it is preferable not to generate or display a magic window. For example, enlarging a live view image is often performed to check a portion of the live view image in detail. The magic window may interfere with checking the details of the live view image. Therefore, it is preferable to hide the magic window when enlarging the live view image. The magic window may be displayed when enlarging the live view image, but in that case, it is necessary to enlarge the magic window in accordance with the enlargement of the live view image so that the magic window indicates the angle of view displayed during playback display.

[0102] In step S915, the control unit 501 determines whether to end the PC live view. For example, if the connection between the camera 100 and the PC 500 is released or if the user instructs the camera 100 or the PC 500 to end the PC live view, the control unit 501 determines to end the PC live view. The instruction to end the PC live view is, for example, pressing the end button 1309 in FIGS. 13(a) to 13(d). If it is determined to end the PC live view, the operation in FIG. 9 ends; otherwise, the process proceeds to step S906. In this way, the display (live view display) on the display unit 506 can be switched among a circular fisheye display with left-right swap, a circular fisheye display without left-right swap, an equirectangular display with left-right swap (described later), and an equirectangular display without left-right swap (described later).

[0103] As described above, when equirectangular display is performed, the process proceeds from step S906 to step S916. In step S916, the control unit 501 controls the camera connected in step S901. In this embodiment, the live view image request in step S916 is a request for a live view image with a low resolution (a resolution lower than the normal resolution). When performing equirectangular display, equirectangular conversion (conversion from a circular fisheye image to an equirectangular image) is required, and the higher the resolution of the image to be subjected to equirectangular conversion, the longer the time required for the equirectangular conversion and the greater the delay due to the equirectangular conversion. In this embodiment, a low-resolution live view image is requested to speed up the equirectangular conversion (shorten the time required for the equirectangular conversion). Note that if the delay due to the equirectangular conversion is within an acceptable range, a normal-resolution live view image may be requested even when performing equirectangular display.

[0104] In step S917, the control unit 501 receives, from the camera connected in step S901, a live view image captured by the camera and lens information of the twin lenses attached to the camera. The resolution of the live view image received in step S917 is low. The lens information received in step S917 is information converted to match the received live view image (for example, the lens information converted in step S808 of FIG. 8).

[0105] In step S918, the control unit 501 determines whether or not to swap the left and right. If it is determined that the left and right should be swapped, the process proceeds to step S919; otherwise, the process proceeds to step S921. In step S918, the control unit 501 determines whether or not to swap the left and right based on whether the check box 1308 in FIGS. 13(b) and (d) is checked, for example. If the check box 1308 is checked, the control unit 501 determines that the left and right should be swapped, and the process proceeds to step S919. If the check box 1308 is not checked, the process proceeds to step S921.

[0106] In step S919, the control unit 501 swaps the positions of the right and left images in the live view image acquired in step S917 based on the lens information acquired in step S917, and converts each of the right and left images into equirectangular images. In other words, the control unit 501 performs left-right swapping and equirectangular conversion to generate a processed live view image.

[0107] The control unit 501 generates a map for equirectangular conversion based on the center coordinates (the optical axis centers of the left-eye optical system 301L and the right-eye optical system 301R) included in the lens information received along with the live-view image. For example, the control unit 501 identifies the regions of the right and left images in the live-view image using a method similar to that described in step S910, and generates a map based on the two identified regions. Equirectangular conversion is a conversion process that treats a circular fisheye image as a sphere and converts it so that the latitudes (horizontal lines) and longitudes (vertical lines) intersect at right angles, similar to the equirectangular projection of a map. Equirectangular conversion converts a circular circular fisheye image into a rectangular equirectangular image. The map indicates which position each pixel after conversion corresponds to in the image before conversion. Here, the map for equirectangular conversion is generated not only to convert the circular fisheye image into an equirectangular image but also to correct the positions of the right and left images. The control unit 501 performs equirectangular conversion using the generated map to generate a processed live-view image.

[0108] Although the left-right swapping is described as part of the equirectangular conversion, the left-right swapping may be performed separately from the equirectangular conversion. Furthermore, the control unit 501 may adjust and use the design values ​​included in the lens information based on the individual values ​​included in the lens information received together with the live view image. For example, the image circle position (the central coordinates of the right and left images in the live view image) is adjusted based on the image circle position deviation in FIG. 11(b). If the individual value is a difference from the design value, the individual value is added to the design value. If the individual value is the same absolute value as the design value, the design value is replaced with the individual value. Then, the control unit 501 calculates the position of the image circle based on the adjusted central coordinates (the optical axis centers of the left eye optical system 301L and the right eye optical system 301R). A map for equirectangular transformation is generated using the same method as described above. By using the adjusted center coordinates, equirectangular transformation with higher accuracy is possible.

[0109] In step S920, the control unit 501 displays the processed live view image generated in step S919 on the display unit 506.

[0110] In step S921, the control unit 501 converts each of the right and left images into an equirectangular image without swapping the positions of the right and left images in the live view image acquired in step S917. In other words, the control unit 501 performs equirectangular conversion without swapping the left and right images to generate a processed live view image.

[0111] In step S922, the control unit 501 displays the processed live view image generated in step S921 on the display unit 506.

[0112] In step S923, the control unit 501 generates a magic window (frame) based on the lens information (magic window information) acquired in step S917. In this embodiment, the control unit 501 generates a magic window to be superimposed on the right image based on the information of the magic window to be superimposed on the right image, and generates a magic window to be superimposed on the left image based on the information of the magic window to be superimposed on the left image. Details of the magic window generation method will be described later with reference to FIGS. 12(a) and 12(b).

[0113] In step S924, the control unit 501 displays the magic window generated in step S923 on the display unit 506. As described above, when enlarging and displaying a portion of a live view image in response to an instruction from the user, it is preferable not to generate or display a magic window.

[0114] If camera 100 is not compatible with twin lenses, or if camera 100 is equipped with a single lens, the process of step S925 is performed. In the operation of FIG. 9, it is assumed that the single lens is a normal single lens (a single lens that is not a VR lens). In step S925, control unit 501 displays a live view image captured by the single lens on display unit 506. The process of step S925 is similar to conventional processing in which a PC or the like displays a live view image captured by a single lens, and therefore a detailed description thereof will be omitted.

[0115] In each of steps S910, S919, and S921, the control unit 501 performs image processing on the live view image acquired from the connected camera. In step S915, which follows steps S910, S919, and S921, the control unit 501 determines whether to end the PC live view. If the PC live view is to be continued, the process returns to step S906, which precedes steps S910, S919, and S921. Therefore, in the operation of FIG. 9, there is a possibility that any of the image processing steps S910, S919, and S921 may be repeatedly executed.

[0116] Therefore, in order to speed up image processing, the control unit 501 may record information related to the executed image processing in the RAM 503 and use it the next time image processing is performed. For example, the control unit 501 records the correspondence between pixels before image processing and pixels after image processing (image processing map). The image processing map can continue to be used as long as there is no change in the resolution or lens information of the live view image. When the control unit 501 executes any of the image processing in steps S910, S919, and S921, it records the image processing map for that image processing. Then, when the control unit 501 executes the same image processing again, it executes the image processing using the recorded image processing map. In this way, it is possible to speed up image processing. Cut.

[0117] Note that the operations in Figures 8 and 9 are operations assuming that a twin VR lens or a normal single lens (a single lens that is not a VR lens) is attached to the camera, but may be modified as appropriate to assume other cases. For example, the operations in Figures 8 and 9 may be modified so that a magic window is superimposed on a live view image even when a single VR lens is attached. The operations in Figures 8 and 9 may be modified so that a magic window is superimposed on a live view image even when a 360-degree image (omnidirectional image; celestial sphere image) is displayed as the live view image. The operations in Figures 8 and 9 may be modified so that a magic window is not superimposed on a live view image when a twin lens that is not a VR lens is attached.

[0118] 10(a) and (b) are schematic diagrams of left-right swapping. Fig. 10(a) shows conventional left-right swapping that does not use information from the twin lenses. Fig. 10(b) shows left-right swapping of this embodiment that uses information from the twin lenses.

[0119] 10(a) and 10(b), in an image 1001 before left-right swapping, a right image 1003, which is a circular fisheye image, is positioned on the left side, and a left image 1007, which is also a circular fisheye image, is positioned on the right side. The center coordinate of the right image in the left-right direction is coordinate 1004, and the center coordinate of the left image in the center left-right direction is coordinate 1008.

[0120] In Figure 10(a), image 1001 is divided into a left half image and a right half image at center coordinate 1002 of image 1001, and the left half image and the right half image are swapped. In other words, the left half image is moved to the right of the right half image. Image 1011 is the image after this left-right swap.

[0121] In FIG. 10( a), the displacement 1006 is smaller than the displacement 1005. That is, in image 1001, the right image 1003 is shifted from the center of the left half of image 1001 toward the center of image 1001. Similarly, the displacement 1010 is smaller than the displacement 1009. That is, in image 1001, the left image 1007 is shifted from the center of the right half of image 1001 toward the center of image 1001. Therefore, in image 1011, the center coordinate 1013 of the left image 1007 in the left-right direction is shifted by a distance 1014 from the center coordinate 1004, and the center coordinate 1016 of the right image 1003 in the left-right direction is shifted by a distance 1017 from the center coordinate 1008.

[0122] In this embodiment, by using lens information, in the image 1037 (Figure 10(b)) after left and right swapping, the center coordinates of the left image in the left-right direction can be made to match the center coordinates 1004 and the center coordinates of the left image in the left-right direction can be made to match the center coordinates 1008.

[0123] FIG. 11(a) is a schematic diagram showing an example of lens information acquired from a twin lens and transmitted to a PC. 1. Lens design values 2. Lens individual value 3. Lens Flag 4. Lens focal length 5. Lens temperature 6. Magic Window Information Includes:

[0124] The lens design values ​​are the design values ​​for aberration correction. In the manufacturing process of the twin lens, errors such as lens decentering and tilt occur in each of the two optical systems (left eye optical system 301L and right eye optical system 301R). If left-right swapping or equirectangular conversion is performed without taking these errors into consideration, The quality of the binocular VR display will be reduced, making good stereoscopic viewing difficult. The lens individual values ​​are the measurement results of errors detected during the manufacturing process of the binocular lenses. Details of the lens design values ​​and lens individual values ​​will be described later using Figure 11(b).

[0125] The lens flag indicates that a twin lens is used and can be used to determine whether a twin lens is used. The lens focal length is the distance from the "principal point" (center of the lens) to the image sensor (image formation position). The lens focal length may or may not be a parameter common to the two optical systems of the twin lens (the left-eye optical system 301L and the right-eye optical system 301R). A detailed (high-precision) lens focal length is required to perform high-quality twin-lens VR display by highly accurately performing left-right swapping and equirectangular transformation. The lens temperature is the temperature of the twin lens and is used to determine the environmental temperature at the time of image capture. The magic window information is information about the magic window area and is used to generate the magic window in this embodiment. The magic window information is prepared for each of the two optical systems of the twin lens (the left-eye optical system 301L and the right-eye optical system 301R). The magic window information may be prepared in advance in the camera 100 or generated within the camera 100. The magic window information may be prepared in advance in the PC 500 or generated within the PC 500.

[0126] 11B is a schematic diagram showing details of the lens design values ​​and lens individual values. In this embodiment, the lens design values ​​and lens individual values ​​are used for left-right swapping and equirectangular conversion.

[0127] The lens design value is 1. Image circle position 2. Image circle diameter 3. Angle of View 4. Distortion correction coefficient Includes:

[0128] The image circle position is the coordinate of the optical axis center of the optical system in the captured image, and is prepared for each of the two optical systems of the twin lens (the left-eye optical system 301L and the right-eye optical system 301R). In other words, the image circle position is the center coordinate of the image circle (circular fisheye image) formed on the imaging element, and is prepared for each of the right and left images. The origin of the coordinates is, for example, the center of the imaging element (the center of the captured image). The image circle position includes horizontal and vertical coordinates. Note that various information related to the optical axis center of the optical system in the captured image can be used as the image circle position. For example, the distance from a predetermined position in the image (such as the center or the upper left corner) to the optical axis center can be used.

[0129] The image circle diameter is the diameter of the image circle (circular fisheye image) formed on the image sensor. The angle of view is the angle of view of the image circle (circular fisheye image) formed on the image sensor. The distortion correction coefficient is the ratio of the design image height to the ideal image height of the lens. A distortion correction coefficient may be set for each image height, and for image heights for which no distortion correction coefficient is set, the distortion correction coefficient may be calculated by interpolation using multiple distortion correction coefficients. A polynomial that approximates the relationship between image height and distortion correction coefficient may be set. The image circle diameter, angle of view, and distortion correction coefficient may or may not be parameters common to the two optical systems of the twin lens (the left-eye optical system 301L and the right-eye optical system 301R).

[0130] The magic window may be displayed based on the image circle position, the image circle diameter, and the angle of view. This method also allows the magic window to be displayed with high quality. To properly display the magic window, the PC 500 The circle position, image circle diameter, and field angle are edited as needed. For example, the PC500 multiplies the image circle position and image circle diameter by a coefficient.

[0131] Lens individual value is 5. Image circle misalignment 6. Optical axis tilt 7. Image magnification deviation This information is prepared by measuring each of the two optical systems (left eye optical system 301L and right eye optical system 301R) of the twin lenses.

[0132] Image circle position deviation is the deviation of the center coordinates of the image circle (circular fisheye image) formed on the image sensor from the design value. For example, image circle position deviation includes horizontal and vertical deviations. With the design value coordinates (two-dimensional coordinates including horizontal and vertical coordinates) as the origin, the horizontal coordinate indicates the horizontal deviation, and the vertical coordinate indicates the vertical deviation. Optical axis tilt is the deviation of the direction of the optical axis on the subject side from the design value. For example, optical axis tilt includes horizontal and vertical deviations. Deviations in each direction are expressed as angles. Image magnification deviation is the deviation of the size of the image circle (circular fisheye image) formed on the image sensor from the design value. This deviation is expressed, for example, as a ratio to the design value.

[0133] Note that the information included in the lens information is not limited to the information described above. For example, the lens information may include the boundary positions of the right and left images in the live view image (the positions of the edges of the circular fisheye image; the positions indicated by the offset amounts 1005, 1006, 1009, 1010, etc.). The lens information may also include the coordinates of the midpoint between the right and left images in the live view image. In many cases, the coordinates of the midpoint coincide with the coordinates of the center of the live view image. The lens information may also include correction data (for example, correction values ​​obtained by calibrating the twin lenses) for improving the accuracy of left-right swapping, equirectangular transformation, etc.

[0134] FIG. 11(c) is a schematic diagram showing an example of camera information generated within a camera. For example, the camera information is used to perform high-quality VR display. The camera information is 1. Camera recording area information 2.In-camera accelerometer information 3. Right exposure compensation information Includes:

[0135] Camera recording area information is information about the effective image area. The displayable effective image area varies depending on the camera's sensor and recording mode. PC500 uses the camera recording area information to provide more accurate display. In-camera accelerometer information is attitude information obtained using the camera's in-camera acceleration sensor (level), and indicates the camera's attitude in the roll and pitch directions, etc. PC500 uses the in-camera accelerometer information to determine the camera's attitude at the time of shooting. Based on the determined attitude, PC500 performs electronic image stabilization and horizontal correction (zenith correction, which brings the up-and-down direction of the display closer to the up-and-down direction of the real space). Right exposure compensation information is an exposure setting value that brings the exposure of the right image closer to the exposure of the left image. PC500 uses the right exposure compensation information to provide a natural (reduced) two-eye VR display.

[0136] Figures 12(a) and (b) show an example of a method for generating a magic window. Figure 12(a) shows the process for performing cylindrical fisheye display, i.e., the process of step S913 in Figure 9, and Figure 12(b) shows the process for performing equirectangular display, i.e., the process of step S923 in Figure 9. Below, we will explain how to generate a magic window to be displayed on the left side. A magic window to be displayed on the right side is also generated using a similar method.

[0137] Here, it is assumed that the magic window information includes a position 1202 (e.g., the coordinates of the upper left corner), a width 1203, and a height 1204 of a rectangle 1201 circumscribing the magic window. The control unit 501 of the PC 500 recognizes the rectangle 1201 from the position 1202, the width 1203, and the height 1204.

[0138] The process (step S913) for performing circular fisheye display will be described using FIG. 12(a). The ROM 502 or RAM 503 stores a frame image 1211 (point cloud) in advance. The control unit 501 generates a magic window 1212 by enlarging or reducing the frame image 1211 so that the frame image 1211 is inscribed in the rectangle 1201 (the frame image 1211 becomes the magic window 1212). Note that the magic window information may be a function representing a graphic. Similarly, a function representing a graphic may be used instead of the frame image 1211.

[0139] 12(b), a process (step S923) for performing equirectangular display will be described. Similar to the process of converting the circular fisheye image into an equirectangular image, the control unit 501 applies equirectangular conversion to the frame image 1211 to obtain a frame image 1221. The control unit 501 then generates a magic window 1222 by enlarging or reducing the frame image 1221 so that the frame image 1221 is inscribed in the rectangle 1201. Note that the equirectangular conversion may also be applied to the rectangle 1201, and the frame image 1221 may be inscribed in the rectangle after the equirectangular conversion.

[0140] 13(a) to 13(d) are schematic diagrams showing an example of an application screen (PC live view display) displayed by the control unit 501 on the display unit 506. A screen 1300 is an application screen (remote live view screen). The screen 1300 includes a live view display area 1301, a guide display area 1302, a guide display area 1303, an operation area 1305, and an end button 1309.

[0141] The live view display area 1301 is an area where a live view image is displayed. The live view display area 1301 is made up of a left display area 1301A and a right display area 1301B. The guide display area 1302 is an area where a character string indicating which of the two optical systems of the twin lens (the left eye optical system 301L or the right eye optical system 301R) the image displayed in the left display area 1301A is from is displayed. The guide display area 1303 is an area where a character string indicating which of the two optical systems of the twin lens (the left eye optical system 301L or the right eye optical system 301R) the image displayed in the right display area 1301B is from is displayed. The operation area 1305 is an area where operations related to PC live view are accepted, and radio buttons 1306 and 1307 and a check box 1308 are displayed in the operation area 1305. Radio button 1306 is a radio button that is selected when performing circular fisheye display, and radio button 1307 is a radio button that is selected when performing equirectangular display. When radio button 1306 is selected, radio button 1307 is unselected, and when radio button 1306 is unselected, radio button 1307 is selected. Check box 1308 is a checkbox that is checked when swapping left and right. When check box 1308 is operated, the positions of the right image (right eye image) and left image (left eye image) in the live view image are swapped, and the character strings displayed in guide display areas 1302 and 1303 are also swapped. Exit button 1309 is a button for ending PC live view.

[0142] In FIG. 13(a), the radio button 1306 for circular fisheye display is selected. The check box 1308 for left-right swapping is not checked. Therefore, the live view image acquired from the camera is displayed as is in the live view display area 1301. Specifically, the right-eye video, which is a circular fisheye image, is displayed in the left display area 1301A. The left-eye image, which is a circular fisheye image, is displayed in the right display area 1301B. Furthermore, a magic window 1304A is superimposed on the right-eye image displayed in the left display area 1301A, and a magic window 1304B is superimposed on the left-eye image displayed in the right display area 1301B. The magic window 1304A superimposed on the right-eye image indicates the portion of the angle of view of the right-eye optical system 301R that is displayed during playback display (during twin-eye VR display). The magic window 1304B superimposed on the left-eye image indicates the portion of the angle of view of the left-eye optical system 301L that is displayed during playback display (during twin-eye VR display).

[0143] In FIG. 13(b), a radio button 1307 for performing equirectangular display is selected. A checkbox 1308 for performing left-right swapping is not checked. Therefore, the right-eye video and left-eye video (either of which is a circular fisheye image) in the live-view image acquired from the camera are each converted into an equirectangular image (left-right swapping is not performed). The live-view image after equirectangular conversion is then displayed in the live-view display area 1301. Specifically, the right-eye video, which is an equirectangular image, is displayed in the left-side display area 1301A, and the left-eye video, which is an equirectangular image, is displayed in the right-side display area 1301B. Furthermore, a magic window 1304C is superimposed on the right-eye video displayed in the left-side display area 1301A, and a magic window 1304D is superimposed on the left-eye video displayed in the right-side display area 1301B. The magic window 1304C superimposed on the right-eye image is obtained by converting the magic window 1304A described in Fig. 13(a) into an equirectangular shape, and has a different shape from the magic window 1304A. The magic window 1304C superimposed on the left-eye image is obtained by converting the magic window 1304B described in Fig. 13(a) into an equirectangular shape, and has a different shape from the magic window 1304B.

[0144] In FIG. 13(c), a radio button 1306 for circular fisheye display is selected, and a check box 1308 for left-right swapping is checked. Therefore, the positions of the right-eye video and the left-eye video in the live-view image acquired from the camera are swapped. The live-view image after the left-right swap is then displayed in the live-view display area 1301. Specifically, the left-eye video, which is a circular fisheye image, is displayed in the left-side display area 1301A, and the right-eye video, which is also a circular fisheye image, is displayed in the right-side display area 1301B. Furthermore, the magic windows 1304A and 1304B described in FIG. 13(a) are displayed. However, the positions of the magic windows 1304A and 1304B are also swapped in accordance with the swapping of the positions of the right-eye video and the left-eye video.

[0145] In FIG. 13(d), the radio button 1307 for performing equirectangular display is selected, and the checkbox 1308 for performing left-right swapping is checked. Therefore, the positions of the right-eye video and the left-eye video in the live-view image acquired from the camera are swapped, and the right-eye video and the left-eye video (either of which is a circular fisheye image) are converted into equirectangular images. The live-view image after left-right swapping and equirectangular conversion is then displayed in the live-view display area 1301. Specifically, the left-eye video, which is an equirectangular image, is displayed in the left-side display area 1301A, and the right-eye video, which is an equirectangular image, is displayed in the right-side display area 1301B. Furthermore, the magic windows 1304C and 1304D described in FIG. 13(b) are displayed. However, the positions of the magic windows 1304C and 1304D are also swapped in accordance with the swapping of the positions of the right-eye video and the left-eye video.

[0146] The various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or the entire device may be controlled by a plurality of pieces of hardware (e.g., a plurality of processors or circuits) sharing the processing. Similarly, the various controls described above as being performed by the control unit 501 may be performed by a single piece of hardware, or the entire device may be controlled by a plurality of pieces of hardware (e.g., a plurality of processors or circuits) sharing the processing.

[0147] Furthermore, although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0148] Furthermore, the present invention is not limited to cameras and PCs, and can be applied to any electronic device that can handle VR images. For example, the present invention can be applied to PDAs, mobile phone terminals, portable image viewers, printers, digital photo frames, music players, game consoles, e-book readers, etc. The present invention can also be applied to video players, display devices (including projectors), tablet terminals, smartphones, AI speakers, home appliances, and in-vehicle devices.

[0149] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0150] 100: Digital camera (camera) 50: System control unit 500: Personal computer (PC) 501: Control unit

Claims

1. an acquisition means for acquiring a first live view image corresponding to a first optical image input via a first optical system; a control means for controlling, when the first live view image is displayed, to superimpose on the first live view image an indicator indicating a portion of the angle of view of the first optical system that is to be displayed during playback display; and An electronic device comprising:

2. the acquisition unit acquires the first live view image and a second live view image corresponding to a second optical image input via a second optical system having a predetermined parallax with respect to the first optical system, The control means controls the display of the indicator when the first live view image and the second live view image are displayed side by side.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. The control means controls the indicator, which indicates the portion of the angle of view of the first optical system that is displayed during the playback display, to be superimposed on the first live view image, and the indicator, which indicates the portion of the angle of view of the second optical system that is displayed during the playback display, to be superimposed on the second live view image.

3. The electronic device according to claim 2.

4. The control means controls the indicator to be displayed while waiting for or during shooting.

4. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

5. The control means controls the indicator to be displayed in a specific display mode, and does not control the indicator to be displayed in a display mode different from the specific display mode.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

6. The portion indicated by the indicator is the portion that is displayed first during playback display.

6. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

7. the first optical system is a fisheye lens, The first live view image is a circular fisheye image.

7. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

8. The control means controls the first live view image to be displayed as the circular fisheye image or after being converted into an equirectangular image.

8. The electronic device according to claim 7,

9. When the first live view image is converted into the equirectangular image and then displayed, the acquisition means acquires, as the first live view image, an image with a lower resolution than when the first live view image is displayed as the circular fisheye image.

9. The electronic device according to claim 8.

10. The control means controls the indicator to be also subjected to the conversion process and displayed when the first live view image is converted into the equirectangular image and displayed.

10. The electronic device according to claim 8 or 9.

11. the acquiring means further acquires information about the first optical system; The control means controls the display device to enlarge or reduce a predetermined image based on the information to generate the indicator.

11. The electronic device according to claim 1.

12. When the first live view image is enlarged and displayed, the control means does not control the display of the indicator.

12. The electronic device according to claim 1.

13. The electronic device is an imaging device that captures the first live view image.

13. The electronic device according to claim 1.

14. The electronic device is an information processing device connectable to an imaging device that captures the first live view image.

13. The electronic device according to claim 1.

15. acquiring a first live view image corresponding to a first optical image input via a first optical system; a step of controlling, when displaying the first live view image, to superimpose on the first live view image an indicator indicating a portion of the angle of view of the first optical system that is to be displayed during playback display; 1. A method for controlling an electronic device, comprising:

16. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

Images