Electronic device, control method of electronic device, computer program product, and computer readable storage medium

Abstract

The invention discloses an electronic device, a control method of the electronic device, a computer program product and a computer readable storage medium. The electronic apparatus includes: an acquisition unit configured to acquire an image having a plurality of image areas captured by an image capturing apparatus via a plurality of optical systems; a display control unit for performing control such that a display image based on the acquired image is displayed, and performing control such that an item indicating a position in the display image is displayed; and a receiving unit for receiving a user operation for changing a position of the item, in which the display control unit: performs control such that the item is displayed in a first form in a case where the position of the item corresponds to a position in a predetermined region of the image, and performs control such that the item is displayed in a second form in a case where the position of the item corresponds to a position in the predetermined region of the image. The item is displayed in a first form such that the item is displayed in a second form in a case where the position of the item corresponds to a position outside the predetermined area.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, a control method of the electronic device, a computer program product, and a computer-readable storage medium. Background Art

[0002] There is known a technique for acquiring an image having two image areas with parallax using two optical systems facing the same direction and displaying the two image areas to allow stereoscopic vision thereof. When a circular fisheye lens is used as each optical system, an image area having a wide range indicating 180 degrees (hemisphere, 90 degrees in all directions from the image center) or more vertically and horizontally can be obtained as each image area.

[0003] In addition, a function of connecting a camera (digital camera) to a PC (personal computer) and displaying an image captured by the camera in real time on a display device of the PC (PC live view) has been proposed (JP 2022-183845A). In PC live view, when a user specifies an arbitrary point on a displayed image (live view image), a specific instruction (AF instruction or light metering instruction, etc.) related to the position is transmitted to the camera. When receiving the specific instruction, the camera performs a specific operation (AF or light metering, etc.) based on the position specified by the user.

[0004] However, depending on the lens mounted on the camera, the area where a specific operation (AF or metering, etc.) can be performed appropriately (stably) may be limited to a specific area in the live view image (JP 2019-144401 A). Summary of the invention

[0005] The present disclosure provides a technology for enabling a user to easily designate a position in an area where a specific operation (predetermined processing) can be appropriately performed.

[0006] According to the present disclosure, the electronic device includes: an acquisition unit, which is configured to acquire an image having multiple image areas respectively photographed by a camera device via multiple optical systems; a display control unit, which is configured to control so that a display image based on the acquired image is displayed, and to control so that an item for indicating a position in the display image is displayed; and a receiving unit, which is configured to receive a user operation for changing the position of the item, wherein the display control unit: controls so that when the position of the item corresponds to a position in a predetermined area of ​​the image, the item is displayed in a first form, and controls so that when the position of the item corresponds to a position outside the predetermined area, the item is displayed in a second form.

[0007] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A and Figure 1B This is the appearance diagram of the camera;

[0009] Figure 2 is the block diagram of the camera;

[0010] Figure 3 is a schematic diagram showing the configuration of a lens unit;

[0011] Figure 4 is a cross-sectional view of the lens unit;

[0012] Figure 5A and Figure 5B It is an exploded stereogram of the lens unit;

[0013] Figure 6 is a schematic diagram showing the positional relationship between each optical axis and the imaging circle;

[0014] Figure 7 is a block diagram of the camera system;

[0015] Figure 8 is a schematic diagram showing the configuration of a PC live view system;

[0016] Fig. 9 It is the block diagram of PC;

[0017] Fig.10 is a flow chart showing the operation of the camera;

[0018] Fig.11 is a flowchart showing the operation of the PC; and

[0019] Fig. 12A and Fig. 12B This is a schematic diagram of the application screen. DETAILED DESCRIPTION

[0020] In the following description, embodiments of the present disclosure are described with reference to the accompanying drawings.

[0021] Figure 1A and Figure 1B 1 is an external view showing an external appearance example of a digital camera (camera) 100 according to the present embodiment. Figure 1A is a perspective view of the camera 100 when viewed from the front side, and Figure 1B 1 is a perspective view of the camera 100 when viewed from the back.

[0022] The camera 100 includes thereon a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub-electronic dial 105, an animation button 106, and a viewfinder-outside display unit 107. The shutter button 101 is an operation member for providing a shooting preparation instruction or a shooting instruction. The power switch 102 is an operation member for turning on or off the power of the camera 100. The mode selector switch 103 is an operation member for switching among various modes. The main electronic dial 104 is a rotation operation member for changing setting values ​​such as a shutter speed and an aperture value. The sub-electronic dial 105 is a rotation operation member for moving a selection frame (cursor) and feeding an image. The animation button 106 is an operation member for providing an instruction to start or stop movie shooting (recording). The viewfinder-outside display unit 107 displays various setting values ​​such as a shutter speed and an aperture value.

[0023] The camera 100 includes a display unit 108, a touch panel 109, a direction key 110, a set (SET) button 111, an AE lock button 112, a magnification button 113, a playback button 114, a menu button 115, an eyepiece portion 116, an eyepiece detection unit 118, and a touch bar 119 on the back. The display unit 108 displays images and various types of information. The touch panel 109 is an operation member for detecting a touch operation on the display surface (touch operation surface) of the display unit 108. The direction key 110 is an operation member configured with keys (four-way keys) that can be pressed in the up, down, left, and right directions. Processing corresponding to the pressed position of the direction key 110 can be performed. The set button 111 is an operation member to be pressed mainly when determining a selected item. The AE lock button 112 is an operation member to be pressed when the exposure state is fixed in the shooting standby state. The magnification button 113 is an operation member for turning on or off the magnification mode in the live view display (LV display) of the shooting mode. In the case of the enlargement mode being turned on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. In addition, the enlargement button 113 is used to enlarge the playback image or increase the magnification in the playback mode. The playback button 114 is an operation member for switching between the shooting mode and the playback mode. In the case of the shooting mode, according to the pressing of the playback button 114, the mode is changed to the playback mode, so that the latest image among the images recorded in the recording medium 227 described below can be displayed on the display unit 108.

[0024] The menu button 115 is an operation member to be pressed in order to display a menu screen capable of performing various settings on the display unit 108. The user can intuitively perform various settings by using the menu screen displayed in the display unit 108, the direction key 110, and the setting button 111. The eyepiece portion 116 is a portion that the user approaches with the eye and looks through the eyepiece finder (perspective viewfinder) 117. The user can visually recognize an image displayed in the EVF 217 (electronic viewfinder) located inside the camera 100 described below via the eyepiece portion 116. The eyepiece detection unit 118 is a sensor that detects whether the user approaches the eyepiece portion 116 (eyepiece finder 117) with the eye.

[0025] The touch bar 119 is a linear touch operation member (line touch sensor) that can receive a touch operation. The touch bar 119 is arranged at a position where a touch operation can be performed (touchable) with the thumb of the right hand in a state where the grip portion 120 is held with the right hand (in a state where the grip portion 120 is held with the little finger, ring finger, and middle finger of the right hand) in such a manner that the shutter button 101 can be pressed by the index finger of the right hand. That is, the touch bar 119 can be operated in a state (shooting posture) in which the user approaches the eyepiece finder 117 with the eye, looks through the eyepiece portion 116, and holds up the camera 100 so that the shutter button 101 can be pressed at any time. The touch bar 119 can receive a tap operation (an operation of touching the touch bar within a predetermined period of time and releasing the touch bar without moving the touch position) and a left or right slide operation (an operation of touching the touch bar and then moving the touch position while keeping the touch) on the touch bar 119, etc. The touch bar 119 is an operation member different from the touch panel 109, and does not have a display function. The touch bar 119 functions as, for example, a multi-function bar (M-Fn bar) to which various functions can be assigned.

[0026] Furthermore, the camera 100 further includes a grip 120, a thumb rest 121, a terminal cover 122, a cover 123, a communication terminal 124, and the like. The grip 120 is a holding portion formed in a shape that can be easily held by the right hand of the user when the user holds up the camera 100. The shutter button 101 and the main electronic dial 104 are arranged at positions where the user can operate the shutter button 101 and the main electronic dial 104 with the index finger of the right hand in a state where the user holds the camera 100 while holding the grip 120 with the little finger, the ring finger, and the middle finger of the right hand. Furthermore, in the same state, the sub-electronic dial 105 and the touch bar 119 are arranged at positions where the user can operate the sub-electronic dial 105 and the touch bar 119 with the thumb of the right hand. The thumb rest 121 (thumb standby position) is a grip portion provided at a position where the user can easily place the thumb of the right hand holding the grip 120 on the back side of the camera 100 in a state where no operation member is operated. The thumb rest 121 is provided with a rubber member for enhancing the holding force (grip feeling). The terminal cover 122 protects a connector such as a connection cable for connecting the camera 100 to an external device (external equipment). The cover 123 closes a slot for storing a recording medium 227 described below to protect the recording medium 227 and the slot. The communication terminal 124 is a terminal for communicating with a lens unit (a lens unit 200 or a lens unit 300 described later, etc.) that can be attached to and detached from the camera 100.

[0027] Figure 2 is a block diagram showing a configuration example of the camera 100. Figure 2 In, with Figure 1A and Figure 1B The same components in Figure 1A and Figure 1B The same reference numerals are used to denote the same components, and descriptions of these components are appropriately omitted. Figure 2 In FIG. 1 , the lens unit 200 is mounted to the camera 100 .

[0028] First, the lens unit 200 is described. The lens unit 200 is an interchangeable lens unit (interchangeable lens) that can be attached to and detached from the camera 100. The lens unit 200 is a single lens unit (single lens) and is an example of a normal lens unit. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an auto focus (AF) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0029] The aperture 201 is configured in such a manner that the aperture diameter can be adjusted. The lens 202 is configured with a plurality of lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 adjusts the focus by driving the lens 202. The lens system control circuit 205 controls the aperture drive circuit 203 and the AF drive circuit 204, etc. based on an instruction from the system control unit 50 described below. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203, and adjusts the focus by changing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication is performed via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal that enables the lens unit 200 to communicate with the camera 100 side.

[0030] Next, the camera 100 is described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0031] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 based on the instruction of the system control unit 50. The imaging unit 211 is an imaging element (image sensor) configured with a CCD or CMOS element or the like that converts an optical image into an electrical signal. The imaging unit 211 may include an imaging plane phase difference sensor for outputting defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (pixel interpolation, resizing processing such as reduction, and color conversion processing, etc.) on the data from the A / D converter 212 or the data from the memory control unit 213. In addition, the image processing unit 214 performs predetermined arithmetic processing by using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation result. Through this processing, through-the-lens (TTL) type AF processing, automatic exposure (AE) processing, EF (flash pre-emission) processing, etc. are performed. Furthermore, the image processing unit 214 performs predetermined arithmetic processing by using the captured image data, and the system control unit 50 performs TTL type automatic white balance (AWB) processing based on the obtained calculation result.

[0032] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without the intervention of the image processing unit 214. The memory 215 stores the image data obtained by the camera unit 211 and converted into digital data by the A / D converter 212, and the image data to be displayed on the display unit 108 or the EVF 217. The memory 215 includes a storage capacity sufficient to store a predetermined number of still images and a predetermined length of moving images and voices. In addition, the memory 215 is also used as a memory (video memory) for displaying images.

[0033] The D / A converter 216 converts the image data for display stored in the memory 215 into an analog signal, and supplies the analog signal to the display unit 108 or the EVF 217. Thus, the image data for display written to the memory 215 is displayed on the display unit 108 or the EVF 217 via the D / A converter 216. The display unit 108 and the EVF 217 provide a display in response to the analog signal from the D / A converter 216. The display unit 108 and the EVF 217 are, for example, an LCD or an organic EL display. The digital signal A / D-converted by the A / D converter 212 and accumulated in the memory 215 is converted into an analog signal by the D / A converter 216, and the analog signal is sequentially transmitted to the display unit 108 or the EVF 217 and displayed on the display unit 108 or the EVF 217, thereby performing a live view display.

[0034] The system control unit 50 is a control unit including at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, may be a circuit, or may be a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements the processing of the flowchart described below by executing a program recorded in the nonvolatile memory 219. In addition, the system control unit 50 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, and the like.

[0035] The camera 100 also includes a system memory 218 , a nonvolatile memory 219 , a system timer 220 , a communication unit 221 , a posture detection unit 222 , and an eyepiece detection unit 118 .

[0036] For example, RAM is used as system memory 218. In system memory 218, constants, variables and programs read from nonvolatile memory 219 for operation of system control unit 50 are loaded. Nonvolatile memory 219 is an electrically erasable and recordable memory. For example, EEPROM is used as nonvolatile memory 219. In nonvolatile memory 219, constants and programs for operation of system control unit 50 are recorded. The program used here includes a program for performing the following flowchart. System timer 220 is a timer unit for counting the time for various types of control and the time of the built-in clock. Communication unit 221 transmits video signals and voice signals to external devices connected wirelessly or via wired cables and receives video signals and voice signals from external devices. Communication unit 221 can also be connected to a wireless local area network (LAN) and the Internet. In addition, communication unit 221 can also communicate with external devices via Bluetooth (Bluetooth, registered trademark) and Bluetooth low energy. The communication unit 221 can transmit images (including real-time images) captured by the camera unit 211 and images recorded in the recording medium 227, and can receive images and other various types of information from an external device. The posture detection unit 222 is a posture detection sensor that detects the posture of the camera 100 relative to the gravity direction. Based on the posture detected by the posture detection unit 222, it can be determined whether the image captured by the camera unit 211 is an image captured with the camera 100 held in a horizontal position or in a vertical position. The system control unit 50 can add orientation information corresponding to the posture detected by the posture detection unit 222 to the image file of the image captured by the camera unit 211, and can rotate the image according to the detected posture. For example, an acceleration sensor or a gyro sensor can be used for the posture detection unit 222. It is also possible to detect the movement of the camera 100 (whether it is panning, tilting, lifting, or stationary, etc.) by using the posture detection unit 222.

[0037] The eyepiece detection unit 118 can detect an object approaching the eyepiece portion 116 (eyepiece viewfinder 117). For example, an infrared proximity sensor can be used as the eyepiece detection unit 118. When an object approaches, infrared light emitted from the light emitting portion of the eyepiece detection unit 118 is reflected on the object and received by the light receiving portion of the infrared proximity sensor. The distance from the eyepiece portion 116 to the object can be determined based on the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eye proximity detection to detect the distance between the eyepiece portion 116 and the object approaching the eyepiece portion 116. The eyepiece detection unit 118 is an eyepiece detection sensor for detecting the approach (eye approach) and separation (eye separation) of the eye (object) relative to the eyepiece portion 116. When it is detected that the object approaches to within a predetermined distance relative to the eyepiece portion 116 in a non-eye proximity state (non-proximity state), the eyepiece detection unit 118 detects the eye proximity. On the other hand, in the case where it is detected that the approaching object is separated by a predetermined distance or longer in the eye approach state (approach state), the eyepiece detection unit 118 detects eye separation. The threshold for detecting eye approach and the threshold for detecting eye separation may be different to provide, for example, hysteresis. In addition, after the eye approach is detected, the eye approach state is assumed until the eye separation is detected. After the eye separation is detected, the non-eye approach state is assumed until the eye approach is detected. The system control unit 50 switches between display (display state) and non-display (non-display state) of each of the display unit 108 and the EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, in the case where at least the shooting standby state is established and the switching setting of the display destination is set to automatic switching, during the non-eye approach state, the display destination is set to the display unit 108 and the display is turned on, and the EVF 217 is set to non-display. In addition, during the eye approach state, the display destination is set to the EVF 217 and the display is turned on, and the display unit 108 is set to non-display. Note that the eyepiece detection unit 118 is not limited to the infrared proximity sensor, and other sensors may be used as the eyepiece detection unit 118 as long as they can detect a state that can be regarded as an eye approaching.

[0038] Furthermore, the camera 100 includes the viewfinder out-of-view display unit 107 , an out-of-viewfinder display unit drive circuit 223 , a power control unit 224 , a power supply unit 225 , a recording medium I / F 226 , and an operation unit 228 .

[0039] The viewfinder external display unit 107 is driven by the viewfinder external display unit driving circuit 223, and displays various setting values ​​of the camera 100 such as shutter speed and aperture value. The power control unit 224 is configured with a battery detection circuit, a DC-DC converter, and a switch circuit that switches a block to be powered, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery power, etc. In addition, the power control unit 224 controls the DC-DC converter based on the detection result and the instruction from the system control unit 50, and supplies a required voltage to a part including the recording medium 227 for a necessary period of time. The power supply unit 225 is a primary battery such as an alkaline battery and a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, and a Li battery, or an AC adapter, etc. The recording medium I / F 226 is an interface with a recording medium 227 such as a memory card and a hard disk. The recording medium 227 is a memory card for recording captured images, etc., and is configured with a semiconductor memory and a magnetic disk, etc. The recording medium 227 may be attached to and detached from the camera 100 , or may also be embedded in the camera 100 .

[0040] The operation unit 228 is an input unit (receiving unit) that can receive an operation from a user (user operation) and is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode selector switch 103, the touch panel 109, and the other operation units 229. The other operation units 229 include the main electronic dial 104, the sub-electronic dial 105, the movie button 106, the direction key 110, the set button 111, the AE lock button 112, the magnification button 113, the playback button 114, the menu button 115, and the touch bar 119.

[0041] The shutter button 101 includes a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 turns on in the middle of the operation of the shutter button 101 in response to so-called half-pressing (shooting preparation instruction), and outputs a first shutter switch signal SW1. The system control unit 50 starts shooting preparation processing such as AF processing, AE processing, AWB processing, and EF processing in response to the first shutter switch signal SW1. The second shutter switch 231 turns on when the operation of the shutter button 101 is completed in response to so-called full pressing (shooting instruction), and outputs a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processing involving reading a signal from the imaging unit 211, generating an image file including a captured image, and writing the generated image file to the recording medium 227.

[0042] The mode selector switch 103 switches the operation mode of the system control unit 50 to any one of a still image shooting mode, a movie shooting mode, and a playback mode. Examples of modes of the still image shooting mode include an automatic shooting mode, an automatic scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). Examples of modes also include various scene modes with shooting settings for different shooting scenes, a custom mode, and the like. The user can directly switch the operation mode to any one of the above-mentioned shooting modes with the mode selector switch 103. Alternatively, the user can switch the screen to a list screen of the shooting mode once with the mode selector switch 103, and then selectively switch the operation mode to any one of a plurality of display modes by using the operation unit 228. Similarly, the movie shooting mode can include a plurality of modes.

[0043] The touch panel 109 is a touch sensor for detecting various touch operations on the display surface of the display unit 108 (the operation surface of the touch panel 109). The touch panel 109 and the display unit 108 may be configured integrally. For example, the touch panel 109 is attached to the upper layer of the display surface of the display unit 108 so that the transmittance of light does not hinder the display on the display unit 108. In addition, the input coordinates on the touch panel 109 and the display coordinates on the display surface of the display unit 108 are associated with each other, thereby configuring a graphical user interface (GUI), wherein the user can operate the screen displayed on the display unit 108 using the graphical user interface, just like the user directly operates the screen. The touch panel 109 can use any of various methods including resistive film, capacitance, surface acoustic wave, infrared, electromagnetic induction, image recognition, and optical sensor methods. Depending on these methods, there are methods for detecting touch based on contact with the touch panel 109 and methods for detecting touch based on a finger or pen approaching the touch panel 109, but any method may be adopted.

[0044] For the touch panel 109, the system control unit 50 can detect the following operations or states:

[0045] An operation in which a finger or a pen that is not in contact with the touch panel 109 newly touches the touch panel 109 , that is, the start of touch (hereinafter referred to as touchdown).

[0046] A state where a finger or a pen is in contact with the touch panel 109 (hereinafter referred to as touch hold).

[0047] An operation in which a finger or a pen is moved while being in contact with the touch panel 109 (hereinafter referred to as touch movement).

[0048] An operation of separating the finger or pen in contact with the touch panel 109 from the touch panel 109 (releasing the finger or pen from the touch panel 109 ), that is, ending the touch (hereinafter referred to as touch-off).

[0049] A state where nothing is in contact with the touch panel 109 (hereinafter referred to as "not touched").

[0050] When a touch is detected, touch continuation is detected at the same time. After the touch, unless the touch stop is detected, the touch continuation is usually detected continuously. In addition, when the touch movement is detected, the touch continuation is detected continuously. Even if the touch continuation is detected, the touch movement is not detected as long as the touch position does not move. After the touch stop of all fingers and pens that have been in contact with the touch panel 109 is detected, the untouch is established.

[0051] These operations and states and the position coordinates of the finger or pen in contact with the touch panel 109 are notified to the system control unit 50 through the internal bus. The system control unit 50 determines which operation (touch operation) is performed on the touch panel 109 based on the notified information. Regarding the touch movement, for each vertical component and each horizontal component on the touch panel 109, the moving direction of the finger or pen on the touch panel 109 can be determined based on the change in the position coordinates. When a touch movement of a predetermined distance or longer is detected, it is determined that a sliding operation has been performed. An operation in which a finger quickly moves a specific distance while in contact with the touch panel 109 and separates is called a "flick". In other words, a "flick" is an operation in which a finger quickly slides on the touch panel 109 to flick the touch panel 109. When a touch movement that moves a predetermined distance or longer at a predetermined speed or higher is detected, and then the touch stops without changing, it is determined that a flick has been performed (it can be determined that a flick has been performed after a sliding operation). In addition, a touch operation in which multiple locations (e.g., two points) are touched (multi-touch) and the touch positions are close to each other is called "pinch-in," and a touch operation in which the touch positions are far from each other is called "pinch-out." "Pinch-out" and "pinch-in" are collectively referred to as a pinch operation (or simply "pinch").

[0052] Figure 3 is a schematic diagram showing a configuration example of the lens unit 300 . Figure 3 2 shows a state where the lens unit 300 is mounted on the camera 100. Figure 3 In the camera 100 shown, Figure 2 The same components in Figure 2 Components related to the right eye are represented by R at the end of the reference numerals, components related to the left eye are represented by L at the end of the reference numerals, and components related to both the right eye and the left eye are represented by neither R nor L at the end.

[0053] The lens unit 300 is an interchangeable lens unit that can be attached to and detached from the camera 100. The lens unit 300 is a dual-lens unit capable of capturing right and left images with parallax. The lens unit 300 includes two optical systems, and each of the two optical systems can capture images in a wide viewing angle range of approximately 180 degrees. Specifically, each of the two optical systems of the lens unit 300 can capture an image of a subject corresponding to a field of view (viewing angle) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle). That is, each of the two optical systems can capture images in the front hemisphere range.

[0054] The lens unit 300 includes an optical system 301R including a plurality of lenses and a reflector, an optical system 301L including a plurality of lenses and a reflector, and a lens system control circuit 303. The optical system 301R includes a lens 302R arranged near a subject, and the optical system 301L includes a lens 302L arranged near a subject. That is, the lens 302R and the lens 302L are arranged on the subject side of the lens unit 300. The lens 302R and the lens 302L face the same direction, and their optical axes are substantially parallel to each other.

[0055] The lens unit 300 is a dual lens unit (VR180 lens unit) for obtaining a VR180 image, which is one of the virtual reality (VR) image formats capable of realizing binocular stereoscopic vision. In the lens unit 300, the optical system 301R and the optical system 301L each include a fisheye lens capable of photographing a range of approximately 180 degrees. Note that the range that can be photographed by each lens of the optical system 301R and the optical system 301L can be a range of approximately 120 degrees or 160 degrees narrower than the range of 180 degrees. In other words, the optical systems 301R and 301L can be in a range of approximately 120 degrees to approximately 180 degrees. The lens unit 300 can form a right image formed by the optical system 301R and a left image formed by the optical system 301L on one or two imaging elements of a camera to which the lens unit 300 is attached. In the camera 100 , the right image and the left image are formed on one imaging element (imaging sensor), and one image (binocular image) is generated in which the right image area (area of ​​the right image) and the left image area (area of ​​the left image) are arranged side by side.

[0056] The lens unit 300 is mounted to the camera 100 via a lens mount 304 of the lens unit 300 and a camera mount 305 of the camera 100. In this way, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are thus electrically connected to each other via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0057] exist Figure 3 In the imaging unit 211 of the camera 100, a right image formed by the optical system 301R and a left image formed by the optical system 301L are formed side by side. In other words, the optical system 301R and the optical system 301L form two optical images (object images) in two areas of one imaging element (imaging sensor). The imaging unit 211 converts the formed optical image (optical signal) into an analog electrical signal. By using the lens unit 300 in this way, an image including two image areas with parallax can be acquired from two places (optical systems) of the optical system 301R and the optical system 301L. By dividing the acquired image into a left eye image and a right eye image and performing VR display of the image, the user can view a three-dimensional VR image in a range of approximately 180 degrees. That is, the user can view the image of VR180 in a stereoscopic manner.

[0058] Here, the VR image is an image that can be viewed in the VR display described below. Examples of VR images include omnidirectional images (global images) taken by an omnidirectional camera (global camera) and panoramic images having a wider video range (effective video range) than the display range that can be displayed on the display unit at one time. Examples of VR images also include moving images and real-time images (images acquired from the camera in substantially real time) and still images. The VR image has a maximum video range (effective video range) corresponding to a field of view of 360 degrees in the left and right directions and 360 degrees in the up and down directions. Examples of VR images also include images having a wider viewing angle than that capable of being captured by an ordinary camera or a wider video range than the display range that can be displayed in the display unit at one time, even when the viewing angle or video range is less than 360 degrees in the left and right directions and less than 360 degrees in the up and down directions. The image captured by the camera 100 having the above-mentioned lens unit 300 is a VR image. By setting the display mode of, for example, a display device (a display device capable of displaying a VR image) to "VR view", a VR image can be viewed in a VR display. A specific range of a VR image with a 360-degree viewing angle is displayed so that the user can move the displayed range by changing the posture of the display device in the left and right directions (horizontal rotation direction), thereby viewing seamless omnidirectional video in the left and right directions.

[0059] VR display (VR view) is a display method (display mode) for displaying a video in a field of view according to the posture of a display device in a VR image, and the display method is capable of changing its display range. Examples of VR display include "single-lens VR display (single-lens VR view)", in which one image is displayed after deformation (distortion correction) for mapping a VR image onto a virtual sphere. Examples of VR display include "dual-lens VR display (dual-lens VR view)", in which a left-eye VR image and a right-eye VR image are displayed side by side in the left and right area sides after deformation for mapping the VR image onto a virtual sphere. "Dual-lens VR display" is performed by using a left-eye VR image and a right-eye VR image with parallax, thereby achieving stereoscopic vision of the VR image. In any type of VR display, for example, when a user wears a display device such as a head-mounted display (HMD), a video in a field of view corresponding to the user's facial orientation is displayed. For example, it is assumed that: in a VR image, at a certain point in time, a video is displayed in a field of view range with a center of 0 degrees in the left-right direction (a specific orientation such as the north) and 90 degrees in the up-down direction (90 degrees from the zenith, where the zenith is a horizontal line). In this state, if the posture of the display device is reversed (for example, the display surface is changed from the south direction to the north direction), in the same VR image, the display range is changed to a video in a field of view range with a center of 180 degrees in the left-right direction (an opposite orientation such as the south) and 90 degrees in the up-down direction. That is, when the user wearing the HMD faces from the north to the south (or looks back), the video displayed on the HMD changes from the video of the north to the video of the south. Note that the VR image captured by the lens unit 300 is an image (180-degree image) obtained by capturing a range of approximately 180 degrees in the front, and there is no video in a range of approximately 180 degrees in the back. In the VR display of such an image, when the posture of the display device is changed to a side where no video image exists, a blank area is displayed.

[0060] This VR display of a VR image makes the user visually feel that they are in the VR image (in the VR space) (immersion). Note that the VR image display method is not limited to the method for changing the posture of the display device. For example, the display range can be moved (scrolled) in response to a user operation via a touch panel or a direction button, etc. During VR display (in the "VR view" display mode), in addition to changing the display range by changing the posture, the display range can also be changed in response to a touch movement on the touch panel, a drag operation with a mouse device, etc., or pressing a direction button. In addition, a smartphone mounted to VR goggles (head-mounted adapter) is a type of HMD.

[0061] The configuration of the lens unit 300 is described in more detail. Figure 4is a cross-sectional view showing a configuration example of the lens unit 300, Figure 5A and Figure 5B is an exploded perspective view showing a configuration example of the lens unit 300 . Figure 5A is a perspective view of the lens unit 300 as viewed from the front side, Figure 5B 3 is a perspective view of the lens unit 300 as viewed from the back side.

[0062] The optical system 301R and the optical system 301L are each fixed to the lens top base 310 by screw fastening or the like. The optical axis of the optical system 301R includes, from the subject side, a first optical axis OA1R, a second optical axis OA2R substantially perpendicular to the first optical axis OA1R, and a third optical axis OA3R substantially parallel to the first optical axis OA1R. Similarly, the optical axis of the optical system 301L includes a first optical axis OA1L, a second optical axis OA2L, and a third optical axis OA3L.

[0063] The optical system 301R includes a first lens 311R, a second lens 321R, a third lens 331R, and a fourth lens 341R. The first lens 311R is arranged on the first optical axis OA1R, and the surface 311AR of the first lens 311R on the subject side has a convex shape. The second lens 321R is arranged on the second optical axis OA2R. The third lens 331R and the fourth lens 341R are arranged on the third optical axis OA3R. Similarly, the optical system 301L includes a first lens 311L, a second lens 321L, a third lens 331L, and a fourth lens 341L.

[0064] In addition, the optical system 301R includes a first prism 320R and a second prism 330R. The first prism 320R bends the light beam entering the first lens 311R from the object side from a direction substantially parallel to the first optical axis OA1R to a direction substantially parallel to the second optical axis OA2R, and guides the light beam to the second lens 321R. The second prism 330R bends the light beam entering the second lens 321R from a direction substantially parallel to the second optical axis OA2R to a direction substantially parallel to the third optical axis OA3R, and guides the light beam to the third lens 331R (and the fourth lens 341R). Similarly, the optical system 301L includes a first prism 320L and a second prism 330L.

[0065] Figure 6: is a schematic diagram showing the positional relationship between each optical axis and the imaging circle on the imaging unit 211. An imaging circle ICR corresponding to the effective angle of view of the optical system 301R and an imaging circle ICL corresponding to the effective angle of view of the optical system 301L are formed in parallel on the imaging unit 211 of the camera 100. It is preferable to set the diameter ΦD2 of the imaging circle ICR and the ICL and the distance between the imaging circle ICR and the imaging circle ICL so that the imaging circle ICR and the imaging circle ICL do not overlap each other. For example, the arrangement of the imaging circle ICR and the ICL is set so that in two areas obtained by dividing the light receiving range of the imaging unit 211 into two in the left and right directions, the center of the imaging circle ICR is substantially arranged at the center of the right area and the center of the imaging circle ICL is substantially arranged at the center of the left area. The size and arrangement of the imaging circle ICR and the ICL are determined by, for example, the configuration of the lens unit 300 (and the camera 100).

[0066] exist Figure 6 , the distance L1 is the distance (baseline length) between the first optical axis OA1R and the first optical axis OA1L. In the stereoscopic vision of the image obtained by using the lens unit 300, as the baseline length L1 becomes longer, a higher stereoscopic effect can be obtained. For example, it is assumed that the sensor size (the size of the imaging surface (light receiving surface, light receiving range)) of the imaging unit 211 is 24 mm long × 36 mm wide, and the diameter ΦD2 of the imaging circles ICR and ICL is 17 mm. In addition, the distance L2 between the third optical axis OA3R and the third optical axis OA3L is 18 mm, and the lengths of the second optical axes OA2R and OA2L are 21 mm. Assuming that the second optical axes OA2R and OA2L extend in the horizontal direction, the baseline length L1 is 60 mm, which is substantially equal to the width of an adult's eye (the distance between the right eye and the left eye).

[0067] The diameter ΦD of the lens mount 304 may be longer or shorter than the baseline length L1. When the distance L2 between the third optical axis OA3R and the third optical axis OA3L is shorter than the diameter ΦD of the lens mount 304, the third lenses 331R and 331L and the fourth lenses 341R and 341L may be arranged inside the lens mount 304. Figure 6 In the equation, the relationship L1>ΦD>L2 holds.

[0068] When performing dual-lens VR display with a field of view (angle of view) of about 120 degrees, a sufficient stereoscopic effect can be obtained. However, since there is still discomfort when the field of view is about 120 degrees, in many cases, the angle of view (effective angle of view) of the optical systems 301R and 301L is about 180 degrees. Figure 6, the viewing angles (effective viewing angles) of the optical systems 301R and 301L are greater than 180 degrees, and the diameter ΦD3 of the imaging circle within the 180 degree range is smaller than the diameter ΦD2 of the imaging circles ICR and ICL.

[0069] Figure 7 is a block diagram showing a configuration example of a camera system according to the present embodiment. Figure 7 The camera system in illustrative embodiment includes a camera 100 and a lens unit 300 .

[0070] The lens unit 300 includes optical systems 301R and 301L, drive units 363R and 363L, and a lens information storage unit 350. The optical systems 301R and 301L are as described above. The drive unit 363R drives the optical system 301R, and the drive unit 363L drives the optical system 301L. The lens information storage unit 350 stores lens information related to the lens unit 300. The lens information includes, for example, configuration information of the optical systems 301R and 301L. The lens information may include information (an identifier) ​​indicating whether the lens unit 300 is a dual-lens unit (a lens unit for obtaining a VR image capable of binocular stereoscopic vision).

[0071] As described above, the camera 100 includes the imaging unit 211, the operation unit 228, and the system control unit 50. The system control unit 50 includes the parallax calculation unit 152, the focus detection unit 153, and the drive amount determination unit 154. Note that the parallax calculation unit 152, the focus detection unit 153, and the drive amount determination unit 154 may be included in a device separate from the camera 100. For example, these components may be included in the ( Figure 7 The lens system control circuit 303 (not shown) may be included in the personal computer (PC) 500 described below.

[0072] As described above, the imaging unit 211 is provided with one imaging element, and a right image formed via the optical system 301R and a left image formed via the optical system 301L are formed on an imaging plane of the imaging unit 211. The operation unit 228 includes, for example, a touch panel or a joystick, and is used by the user to designate an AF position (focus detection position) in AF processing.

[0073] The parallax calculation unit 152 calculates the amount of parallax between the right image formed via the optical system 301R and the left image formed via the optical system 301L based on the lens information stored in the lens information storage unit 350. Based on the calculated amount of parallax and the AF position corresponding to the optical system 301R (AF position in the right image), the parallax calculation unit 152 determines the AF position corresponding to the optical system 301L (AF position in the left image). The two AF positions are image formation positions of the same subject. The parallax calculation unit 152 can determine the AF position corresponding to the optical system 301R based on the calculated amount of parallax and the AF position corresponding to the optical system 301L.

[0074] The focus detection unit 153 acquires an AF evaluation value (focus detection evaluation value) for the AF position designated by the user or the AF position determined by the parallax calculation unit 152. For example, when the user designates the AF position corresponding to the optical system 301R, the parallax calculation unit 152 determines the AF position corresponding to the optical system 301L. Then, the focus detection unit 153 acquires two AF evaluation values ​​corresponding to the two AF positions, respectively.

[0075] The drive amount determination unit 154 determines the drive amount of the optical system 301R and the drive amount of the optical system 301L based on the AF evaluation value acquired by the focus detection unit 153, outputs the drive amount of the optical system 301R to the drive unit 363R, and outputs the drive amount of the optical system 301L to the drive unit 363L. The drive units 363R and 363L drive the optical systems 301R and 301L with the drive amounts determined by the drive amount determination unit 154.

[0076] Figure 8 : is a schematic diagram showing an overall configuration example of a PC live view system according to the present embodiment. Figure 8 The PC live view system in the embodiment includes a camera 100 and a PC 500. The lens unit 300 is mounted (connected) to the camera 100. As described above, by mounting the lens unit 300, the camera 100 can capture a single image (still image or animation) including two image areas with a specified parallax. The PC 500 is an information processing device that handles images captured by an imaging device such as the camera 100. Figure 8 A configuration is shown in which the camera 100 and the PC 500 are communicably connected to each other in a wireless or wired manner.

[0077] Fig. 9 is a block diagram showing a configuration example of a PC 500. The PC 500 includes a CPU 501, a work memory 502, a nonvolatile memory 503, an operation unit 504, a display unit 505, and an external I / F 506.

[0078] For example, the CPU 501 controls the various units of the PC 500 by using the working memory 502 as a working memory according to the program stored in the nonvolatile memory 503. The working memory 502 is configured with, for example, a RAM (volatile memory using a semiconductor element, etc.). The nonvolatile memory 503 stores image data, audio data, other data, various programs for operating the CPU 501, etc. The nonvolatile memory 503 is configured with, for example, a hard disk (HD) and a ROM, etc.

[0079] The operation unit 504 is an input device (receiving unit) capable of receiving user operations. For example, the operation unit 504 includes a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, a button, a dial, a joystick, a touch sensor, and a touch pad. For example, the user uses the operation unit 504 to specify the AF position in the AF process.

[0080] The display unit 505 displays various images and screens, etc. under the control of the CPU 501. For example, the display unit 505 displays a live view image obtained by the camera 100, or displays a GUI screen configuring a graphical user interface (GUI). The CPU 501 controls each unit of the PC 500 to generate a display control signal according to a program, generate a video signal to be displayed on the display unit 505, and output the video signal to the display unit 505. Note that the display unit 505 may be configured with an external monitor (a television set, etc.).

[0081] The external I / F 506 is an interface for connecting to an external device (for example, the camera 100 ) through a wired cable or wirelessly and performing input / output (data communication) of a video signal or an audio signal.

[0082] Fig.10 1 is a flowchart showing an example of the operation of the camera 100. These operations are realized by the system control unit 50 loading a program recorded in the nonvolatile memory 219 into the system memory 218 and executing the program. For example, when the camera 100 is started, the system starts Fig.10 operation. Fig.10 The operation is an operation of a function for displaying a live view image captured by a camera on a display unit of a PC (PC live view). Fig.10 The operation is performed when the camera 100 is in a shooting standby state. When an instruction for starting recording is input from the PC 500 during the PC live view operation, still image shooting or movie shooting is performed. At this time, the PC live view may continue.

[0083] In step S1001, the system control unit 50 determines whether the camera 100 is compatible with a dual lens unit (a lens unit for obtaining a VR image capable of realizing binocular stereoscopic vision, such as the lens unit 300). For example, the system control unit 50 determines whether the firmware version of the system control unit 50 is compatible with the dual lens unit. If it is determined that the version is compatible with the dual lens unit, the process proceeds to step S1002, otherwise Fig.10 The operation is completed. Fig.10 In the operation of , as the twin-lens unit, a twin-lens unit with an AF function (a twin-lens unit capable of performing AF processing of each optical system) is assumed.

[0084] In step S1002, the system control unit 50 determines whether a dual lens unit is mounted to the camera 100. For example, the system control unit 50 acquires identification information of the lens unit from the lens unit mounted to the camera 100, and determines whether the lens unit is a dual lens unit based on the acquired identification information. The identification information is part of the above-mentioned lens information. If it is determined that a dual lens unit is mounted, the process proceeds to step S1003, otherwise Fig.10 The operation is completed.

[0085] In step S1003, the system control unit 50 acquires the design value of the dual lens unit from the mounted (connected) dual lens unit. The design value is part of the lens information described above. As described above, one or more than one design value may be acquired.

[0086] In step S1004, the system control unit 50 acquires individual values ​​of the dual lens unit from the mounted (connected) dual lens unit. The individual values ​​are parameters specific to the lens unit, such as errors during manufacturing. The individual values ​​are part of the lens information described above. One or more individual values ​​may be acquired, and each individual value may be a parameter specific to the lens unit.

[0087] In step S1005 , the camera 100 is connected to the PC 500 , and the system control unit 50 detects the connection of the camera 100 to the PC 500 .

[0088] In step S1006 , the system control unit 50 receives a request to start PC live view from the PC 500 .

[0089] In step S1007 , the system control unit 50 receives a request for a live view image from the PC 500 .

[0090] In step S1008, the system control unit 50 converts the information (lens information) obtained in steps S1003 and S1004 into a coordinate system that conforms to the live view image to be transmitted. The information obtained in steps S1003 and S1004 cannot be used for image processing of the live view image as it is. Therefore, the lens information is converted into information that conforms to the coordinate system of the live view image.

[0091] In step S1009, the system control unit 50 transmits the lens information converted in step S1008 and the live view image to the PC 500. In the present embodiment, the system control unit 50 of the camera 100 converts the lens information. However, the CPU 501 of the PC 500 may convert the lens information. In such an embodiment, the lens information before conversion and the parameters required for converting the lens information are transmitted to the PC 500.

[0092] In step S1010, the system control unit 50 determines whether there is an AF instruction from the PC 500. If it is determined that there is an AF instruction, the process proceeds to step S1011, and otherwise the process proceeds to step S1012.

[0093] In step S1011, the system control unit 50 performs AF processing according to the AF instruction. The AF instruction is, for example, Fig.11 The AF instruction transmitted from the PC 500 in step S1124 of . The AF instruction includes the coordinates (coordinate information) of the AF position specified by the user. The system control unit 50 performs AF processing of the optical systems 301R and 301L based on the specified AF position. The parallax calculation unit 152 acquires two AF positions corresponding to the two optical systems 301R and 301L, respectively, and the focus detection unit 153 determines two AF evaluation values ​​corresponding to the two AF positions, respectively. Then, the drive amount determination unit 154 determines the drive amount of the optical system 301R, outputs the drive amount to the drive unit 363R, determines the drive amount of the optical system 301L, and outputs the drive amount to the drive unit 363L.

[0094] In step S1012, the system control unit 50 determines whether to end the PC live view. For example, if the camera 100 and the PC 500 are disconnected from each other, or the user instructs the camera 100 or the PC 500 to end the PC live view, it is determined that the PC live view is to be ended. If it is determined that the PC live view is to be ended, Fig.10 The operation is completed, otherwise the processing proceeds to step S1007.

[0095] Fig.115 is a flowchart showing an example of the operation of the personal computer 500. These operations are realized by the CPU 501 loading the program (application program) recorded in the nonvolatile memory 503 into the working memory 502. For example, when the user instructs the PC 500 to start a specific application, the PC 500 starts Fig.11 operation. Fig.11 The operation is an operation of a function for displaying a live view image captured by a camera on a display unit of a PC (PC live view).

[0096] In step S1101 , a camera (eg, the camera 100 ) is connected to the PC 500 , and the CPU 501 detects the connection of the camera to the PC 500 .

[0097] In step S1102, the CPU 501 determines whether the camera connected in step S1101 is a camera that is compatible with a dual-lens unit (a lens unit for obtaining a VR image that can achieve binocular stereoscopic vision, such as the lens unit 300). For example, the CPU 501 obtains model information related to the camera from the connected camera, and determines whether the camera is compatible with the dual-lens unit based on the obtained model information. If it is determined that the camera is compatible with the dual-lens unit, the process proceeds to step S1103, otherwise Fig.11 The operation is ended. A camera compatible with a dual lens unit is, for example, a camera to which a dual lens unit can be mounted. Fig.11 In the operation of , as the twin-lens unit, a twin-lens unit with an AF function (a twin-lens unit capable of performing AF processing of each optical system) is assumed.

[0098] In step S1103, the CPU 501 determines whether the firmware of the camera connected in step S1101 is compatible with the twin-lens unit. For example, the CPU 501 acquires version information related to the firmware of the camera from the connected camera, and determines whether the version of the firmware of the connected camera is compatible with the twin-lens unit based on the acquired information. If it is determined that the version is compatible with the twin-lens unit, the process proceeds to step S1104, otherwise Fig.11 The operation is completed.

[0099] In step S1104, the CPU 501 determines whether a twin-lens unit is attached to the camera connected in step S1101. For example, the CPU 501 acquires information indicating whether a twin-lens unit is attached to the camera from the connected camera, and determines whether a twin-lens unit is attached to the connected camera based on the acquired information. If it is determined that a twin-lens unit is attached, the process proceeds to step S1105, otherwise Fig.11 The operation is completed.

[0100] In step S1105 , the CPU 501 transmits a request to start PC live view to the camera connected in step S1101 .

[0101] In step S1106 , the CPU 501 transmits a request for a live view image to the camera connected in step S1101 .

[0102] In step S1107, the CPU 501 receives the live view image captured by the camera and the lens information of the dual lens unit attached to the camera from the camera connected in step S1101. The received lens information is information converted to conform to the received live view image (for example, in Fig.10 The received live view image and the received lens information are, for example, the live view image and the lens information transmitted from the camera 100 in step S1009.

[0103] It is assumed that the lens information received in step S1107 includes the following information. The AF-capable area information is area information indicating an AF-capable area. The AF-capable area is a predetermined area where AF processing can be appropriately performed. AF processing outside the AF-capable area (AF processing in which a position outside the AF-capable area is set as an AF position) may not be performed or may be performed. The AF-capable area is considered to be a recommended AF area. The AF-capable area may be considered to be an area where an AF position can be set, or may be considered to be an area including a plurality of recommended AF positions (candidates). The lens information may include at least one of the following items:

[0104] Capable of AF area information;

[0105] The center position of the imaging circle;

[0106] The location of the boundaries of the imaging circle; and

[0107] The diameter of the imaging circle.

[0108] Note that the lens information received in step S1107 may include the following information. The magic window area is an area at a predefined predetermined position (e.g., a center portion) and is an area cut out (first) for VR display. That is, the magic window area is cut out from the captured image, the magic window area is perspectively projected, and the magic window area is displayed on a display device such as a head-mounted display. The lens information may include at least one of the following items:

[0109] Information indicating the magic window;

[0110] Information indicating manufacturing errors of the dual-lens unit; and

[0111] Correction data for improving the accuracy of image processing of live view images (e.g. correction values ​​obtained by calibrating a dual-lens unit).

[0112] In step S1108, the CPU 501 determines whether to perform equirectangular cylindrical conversion display. If it is determined that equirectangular cylindrical conversion display is to be performed, the process proceeds to step S1109, otherwise (when circular fisheye display is to be performed), the process proceeds to step S1116. For example, the CPU 501 Fig. 12A and Fig. 12B The state of radio buttons 1205 and 1206 in the display is used to determine whether to perform equirectangular cylinder conversion display. Fig. 12A and Fig. 12B details.

[0113] In step S1109, the CPU 501 determines whether to perform layout conversion on the multiple image areas (multiple image areas captured via multiple optical systems) in the live view image received (acquired) in step S1107. In the present embodiment, it is assumed that a live view image in which two image areas (a right image area and a left image area) are arranged side by side is received, and as a layout conversion, a conversion of swapping the positions of the two image areas (left-right swap) is performed. If it is determined that the left-right swap is performed, the processing proceeds to step S1110, otherwise the processing proceeds to step S1113. For example, the CPU 501 performs a layout conversion based on Fig. 12A and Fig. 12B Whether to swap left and right is determined by checking whether check box 1207 in is checked.

[0114] In step S1110, the CPU 501 performs left-right swapping and equirectangular cylindrical conversion on the live view image received in step S1107 based on the lens information (e.g., the center position of each imaging circle) received (acquired) in step S1107. As a result, an image obtained by performing left-right swapping and equirectangular cylindrical conversion on the received live view image is obtained as a display image (image to be displayed). For example, left-right swapping is performed so that the right image area is arranged around the center of the left image area before the left-right swapping, and the left image area is arranged around the center of the right image area before the left-right swapping. Then, the right image area and the left image area are each subjected to equirectangular cylindrical conversion. In this embodiment, it is assumed that the right image area and the left image area in the received live view image are each an area of ​​a circular fisheye image. The equirectangular cylindrical conversion is a conversion process in which the circular fisheye image is assumed to be a sphere and converted into a latitude (horizontal line) and meridian (vertical line) crossing each other at right angles as in the equirectangular cylindrical projection of the map. The circular fisheye image is converted into a rectangular equirectangular transformed image through equirectangular cylindrical transformation.

[0115] In step S1111 , the CPU 501 displays the display image generated in step S1110 (live view image after left-right swapping and equirectangular cylindrical conversion) on the display unit 505 .

[0116] In step S1112 , the CPU 501 performs image processing similar to that in step S1110 (left-right interchange and equirectangular cylindrical conversion) on the capable AF area information included in the lens information received in step S1107 .

[0117] In step S1113, as in step S1110, the CPU 501 performs equirectangular cylindrical transformation on the live view image received in step S1107. Left-right swapping is not performed. As a result, an image obtained by performing equirectangular cylindrical transformation on the received live view image is obtained as a display image.

[0118] In step S1114 , the CPU 501 displays the display image generated in step S1113 (live view image after equirectangular cylindrical conversion) on the display unit 505 .

[0119] In step S1115 , the CPU 501 performs image processing (equirectangular cylindrical conversion) similar to that in step S1113 on the capable AF area information included in the lens information received in step S1107 .

[0120] In step S1116, as in step S1109, the CPU 501 determines whether arrangement conversion (left-right interchange) is performed. If it is determined that left-right interchange is performed, the process proceeds to step S1117, otherwise the process proceeds to step S1120.

[0121] In step S1117, as in step S1110, the CPU 501 performs left-right swapping on the live view image received in step S1107. No equirectangular cylindrical conversion is performed. As a result, an image obtained by performing left-right swapping on the received live view image is obtained as a display image.

[0122] In step S1118 , the CPU 501 displays the display image generated in step S1117 on the display unit 505 (the live view image after left-right swapping).

[0123] In step S1119 , the CPU 501 performs image processing (left-right interchange) similar to that in step S1118 on the capable AF area information included in the lens information received in step S1107 .

[0124] In step S1120 , the CPU 501 displays the live view image received in step S1107 on the display unit 505 (as a display image).

[0125] The CPU 501 displays an item indicating a position in the image (a position specifying item, such as Fig. 12A and Fig. 12B A cursor 1210 shown in FIG. 1 is displayed together with the above-mentioned display image on the display unit 505. The operation unit 504 can receive a user operation for changing the position of an item (a position indicated by an item, a display position of an item).

[0126] In step S1121, the CPU 501 determines whether the position of the position designation item is outside the AF area based on the AF area information. In the case where it is determined that the position of the position designation item is outside the AF area, the processing proceeds to step S1122, otherwise, the processing proceeds to step S1123. When neither equirectangular cylindrical conversion nor left-right swapping is performed, the AF area information received in step S1107 is used. When left-right swapping is performed, the AF area information after the left-right swapping (after the layout conversion) is used. When equirectangular cylindrical conversion is performed, the AF area information after the equirectangular cylindrical conversion is used. When left-right swapping and equirectangular cylindrical conversion are performed, the AF area information after the left-right swapping and equirectangular cylindrical conversion is used.

[0127] In step S1122, the CPU 501 changes the form (display format; display form) of the position designation item. Through this processing, when the position of the position designation item is within the AF-capable area, the position designation item is displayed in the first form, and when the position of the position designation item is outside the AF-capable area, the position designation item is displayed in the second form.

[0128] In step S1123, the CPU 501 determines whether a user operation for performing AF processing (AF performing operation) is performed. The AF performing operation is performed using the operation unit 504. If it is determined that the AF performing operation is performed, the process proceeds to step S1124, otherwise the process proceeds to step S1125.

[0129] In step S1124, the CPU 501 transmits an AF instruction including the current coordinates of the position designation item as the coordinates of the AF position (AF position designated by the user) to the camera connected in step S1101. As a result, the camera is controlled to perform AF processing based on the position of the position designation item.

[0130] In the present embodiment, steps S1123 and S1124 are performed only when the position of the position designation item is within the AF-capable area. Therefore, when the position of the position designation item is outside the AF-capable area, AF processing is not performed, and when the position of the position designation item is within the AF-capable area, AF processing is performed. However, steps S1123 and S1124 may be performed regardless of whether the position of the position designation item is within the AF-capable area. That is, regardless of whether the position of the position designation item is within the AF-capable area, AF processing may be performed. For example, the processing may proceed to step S1123 after step S1122.

[0131] In step S1125, the CPU 501 determines whether to terminate the PC live view. If it is determined that the PC live view is to be terminated, Fig.11 The operation is completed, otherwise the processing proceeds to step S1106.

[0132] Fig. 12A and Fig. 12B 1 is a schematic diagram showing a display example of an application screen displayed by the CPU 501 on the display unit 505 during PC live view when the camera 100 is connected to the PC 500. The screen 1200 is an application screen (remote live view screen). The screen 1200 includes a live view display area 1201, a guide display area 1202, a guide display area 1203, an operation area 1204, and an end button 1208.

[0133] The live view display area 1201 is an area for displaying a live view image. The live view display area 1201 includes a left display area 1201A and a right display area 1201B.

[0134] The guide display area 1202 is an area for displaying a character string indicating which of the two optical systems 301L and 301R in the twin-lens unit the image displayed in the left display area 1201A is taken via. The guide display area 1203 is an area for displaying a character string indicating which of the two optical systems 301L and 301R in the twin-lens unit the image displayed in the right display area 1201B is taken via.

[0135] The operation area 1204 is an area for receiving a PC live view operation. Radio buttons 1205 and 1206 and a check box 1207 are displayed in the operation area 1204.

[0136] Radio button 1205 is selected in the case of performing circular fisheye display, and radio button 1206 is selected in the case of performing equirectangular cylindrical conversion display. When radio button 1205 is selected, radio button 1206 is not selected. When radio button 1205 is not selected, radio button 1206 is selected.

[0137] exist Fig. 12A In the example, radio button 1205 is selected and radio button 1206 is not selected. Therefore, circular fisheye display is performed (circular fisheye images are displayed in display areas 1201A and 1202B). Fig. 12B In the display area 1201A and 1202B, the radio button 1205 is not selected, but the radio button 1206 is selected. Therefore, the equirectangular cylindrical conversion display is performed (the equirectangular cylindrical conversion image is displayed in the display areas 1201A and 1202B).

[0138] Checkbox 1207 is a checkbox to be checked when the left and right are swapped. When checkbox 1207 is checked, the display positions of the right image area (area for right-eye video; right-eye video area) and the left image area (area for left-eye video area; left-eye video area) in the live view image can be swapped. Therefore, the character strings displayed in guide display areas 1202 and 1203 are also swapped.

[0139] exist Fig. 12A , check box 1207 is not checked. Therefore, the right and left are not swapped, and the right image (right eye video) captured via the optical system 301R is displayed in the display area 1201A, and the left image (left eye video) captured via the optical system 301L is displayed in the display area 1201B. Fig. 12B , check box 1207 is checked. Therefore, left and right are swapped, the right image is displayed in display area 1201B, and the left image is displayed in display area 1201B.

[0140] The end button 1208 is a button for ending the PC live view.

[0141] Frame 1209 is a frame (item) indicating the AF-capable area. As described above, when the live view image is swapped left and right, the AF-capable area is also swapped left and right, and when the live view image is equirectangularly transformed, the AF-capable area is also equirectangularly transformed. Fig. 12A The box 1209 in FIG. 1 shows an AF-capable area that has neither been subjected to left-right swapping nor to equirectangular cylindrical conversion, and Fig. 12BThe frame 1209 in FIG. 1 shows the AF-capable area after left-right swapping and equirectangular cylindrical conversion. Note that the frame 1209 may be displayed or not. The item indicating the AF-capable area is not limited to the frame 1209, and may be, for example, a mask covering the AF-capable area with a predetermined transparency.

[0142] The cursor 1210 is an item indicating a position in the display image (position designation item). Fig. 12A and Fig. 12B ) show two cursors 1210, but one cursor 1210 is displayed on one application screen.

[0143] The user can move the cursor 1210 by a user operation (e.g., moving a mouse) using the operation unit 504. In addition, the user can instruct to perform AF processing by a user operation (e.g., clicking a mouse button) using the operation unit 504. When the AF processing is instructed, an AF instruction including the current coordinates of the cursor 1210 is transmitted to the camera 100. Then, the camera 100 performs AF processing based on the coordinates of the cursor 1210.

[0144] like Fig. 12A and Fig. 12B As shown, the form of the cursor 1210 changes depending on whether the cursor 1210 is within the frame 1209 (within the AF-capable area). As a result, the user can easily grasp whether the cursor 1210 is within the AF-capable area (even if the frame 1209 is not displayed) and whether AF processing can be appropriately performed, etc. Therefore, the user can easily specify the inside of the AF-capable area. The change in the form of the cursor 1210 may include a change in the type of the cursor 1210, may include a change in the color of the cursor 1210, may include a change in the size of the cursor 1210, and may include a change in other parameters of the cursor 1210.

[0145] Note that the various types of control described above may be processes performed by one hardware (e.g., a processor or circuit) or other devices. Processing may be shared among multiple hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) to perform control of the entire device.

[0146] In addition, the above-mentioned processor is a processor in a broad sense, and includes a general-purpose processor and a special-purpose processor. Examples of general-purpose processors include a central processing unit (CPU), a microprocessing unit (MPU), a digital signal processor (DSP), etc. Examples of special-purpose processors include a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. Examples of PLDs include a field programmable gate array (FPGA), a complex programmable logic device (CPLD), etc.

[0147] The above embodiments (including variant examples) are only examples. Any configuration obtained by appropriately modifying or changing some configurations of the embodiments within the scope of the subject matter of the present disclosure is also included in the present disclosure. The present disclosure also includes other configurations obtained by appropriately combining various features of the embodiments.

[0148] For example, it is described that one image is acquired in which two image areas having parallax are arranged side by side, but the number of image areas, that is, the number of optical systems may be greater than two, and the arrangement of a plurality of image areas is not particularly limited.

[0149] The form of the frame 1209 may also be changed depending on whether the cursor 1210 is in the AF-capable area. Whether to change the form of the cursor 1210 depending on whether the cursor 1210 is in the AF-capable area may be set according to an instruction from the user. Similarly, whether to change the form of the frame 1209 depending on whether the cursor 1210 is in the AF-capable area may be set according to an instruction from the user. Whether to display the frame 1209 may be set according to an instruction from the user. Depending on whether the cursor 1210 is in the AF-capable area, the form of the cursor 1210 may be changed without changing the form of the frame 1209. Depending on whether the cursor 1210 is in the AF-capable area, the form of the frame 1209 may be changed without changing the form of the cursor 1210.

[0150] Among a plurality of operating members such as a mouse, keyboard, and touch panel, the cursor 1210 and the frame 1209 may be displayed in a form corresponding to the operating member used for user operation (eg, moving the cursor 1210). This allows the user to easily grasp which operating member is used.

[0151] An example of considering (using) an area suitable for AF processing (focus adjustment) is described, but an area suitable for white balance adjustment, enlarged display, or photometry (exposure adjustment), etc. may be considered. For example, a plurality of different types of areas such as an AF-capable area (an area suitable for AF processing) and a WB-capable area (an area suitable for white balance adjustment), etc. may be considered. The cursor 1210 and the frame 1209 may be displayed in a form corresponding to the positional relationship between the plurality of areas and the cursor 1210. This allows the user to easily grasp in which area the cursor 1210 is suitable for processing. The user may be able to instruct to perform white balance adjustment, enlarged display, and photometry (exposure adjustment), etc., by a user operation using the operation unit 504.

[0152] At least part of the processing described as being performed by the PC 500 may be performed by the camera 100 or other external devices (eg, a cloud server). At least part of the processing described as being performed by the camera 100 may be performed by the PC 500 or other external devices (eg, a cloud server).

[0153] In addition, the present disclosure is not limited to cameras and PCs, and can be applied to any electronic device that can handle images with multiple image areas corresponding to multiple optical systems. For example, the present disclosure can be applied to PDAs, mobile phone terminals or portable image browsers, printer devices, digital photo frames, music players, electronic game consoles, e-book readers, and cloud servers, etc. In addition, the present disclosure can also be applied to, for example, video players, display devices (including projectors), tablet terminals, smart phones, AI speakers, home appliance devices, and vehicle-mounted devices. The present disclosure is also applicable to multi-view smart phones, etc., having multiple different types of optical systems such as standard lenses, wide-angle lenses, and zoom lenses.

[0154] According to the present disclosure, the user can easily designate an area where a specific operation (predetermined processing) can be appropriately performed.

[0155] Other embodiments

[0156] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0157] Although the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An electronic device, comprising: an acquisition unit configured to acquire an image having a plurality of image areas respectively photographed by an imaging device via a plurality of optical systems; a display control unit configured to control a display image based on the acquired image to be displayed, and to control an item indicating a position in the display image to be displayed; as well as a receiving unit configured to receive a user operation for changing a position of the item, Wherein, the display control unit: controlling so that, in a case where the position of the item corresponds to the position in the predetermined area of ​​the image, the item is displayed in the first form, and Control is performed so that, in a case where the position of the item corresponds to a position outside the predetermined area, the item is displayed in the second form.

2. The electronic device according to claim 1, further comprising: an information acquisition unit configured to acquire area information indicating the predetermined area from the imaging device, The display control unit determines whether the position of the item corresponds to the position in the predetermined area based on the area information.

3. The electronic device according to claim 2, in, The displayed image is an image obtained by performing equirectangular cylindrical transformation on the image, and The display control unit performs the equirectangular cylindrical transformation on the region information, and determines whether the position of the item corresponds to the position in the predetermined region based on the region information after the equirectangular cylindrical transformation.

4. The electronic device according to claim 2 or 3, in, The display image is an image obtained by performing arrangement conversion of the plurality of image areas, and The display control unit performs the arrangement conversion on the region information, and determines whether the position of the item corresponds to the position in the predetermined region based on the region information after the arrangement conversion.

5. The electronic device according to claim 4, in, The image is an image in which two image areas are arranged side by side, and The arrangement conversion is a conversion for interchanging the positions of the two image areas.

6. The electronic device according to any one of claims 1 to 3, further comprising: a control unit configured to perform control so as to perform a predetermined process based on the position of the item, Wherein, the control unit performs control to: In the case where the position of the item corresponds to a position outside the predetermined area, not performing the predetermined processing, and In a case where the position of the item corresponds to the position in the predetermined area, the predetermined process is performed.

7. The electronic device according to claim 6, in, The image is acquired from the camera device, and The predetermined processing is processing for transmitting an instruction of focus adjustment, white balance adjustment, or enlarged display to the image pickup apparatus.

8. The electronic device according to any one of claims 1 to 3, in, The display control unit further controls to display a second item indicating an area corresponding to the predetermined area in the display image.

9. The electronic device according to any one of claims 1 to 3, further comprising: A setting unit is configured to set whether to change the form of the item according to whether the position of the item corresponds to the position in the predetermined area according to an instruction from a user.

10. The electronic device according to any one of claims 1 to 3, in, The display control unit performs control so that the item is displayed in a form corresponding to an operation member used for operation by the user among a plurality of operation members.

11. The electronic device according to claim 10, in, The plurality of operating members include at least one of a mouse, a keyboard, and a touch panel.

12. The electronic device according to any one of claims 1 to 3, in, A plurality of areas of different types are used as the predetermined area, and The display control unit performs control so that the items are displayed in a form corresponding to a positional relationship between the plurality of areas and the items.

13. The electronic device according to any one of claims 1 to 3, in, The acquisition unit acquires a live view image as the image.

14. The electronic device according to any one of claims 1 to 3, in, The predetermined area corresponds to an area suitable for white balance adjustment, enlarged display, exposure adjustment, and focus adjustment in the image pickup apparatus.

15. The electronic device according to any one of claims 1 to 3, in, The image is an image in which two optical images are respectively formed in two areas of one imaging element.

16. A control method for an electronic device, the control method comprising: An acquisition step for acquiring an image having a plurality of image areas respectively photographed by an imaging device via a plurality of optical systems; a display control step for controlling a display image based on the acquired image to be displayed, and controlling an item indicating a position in the display image to be displayed; as well as a receiving step for receiving a user operation for changing the position of the item, Wherein, in the display control step, controlling so that, in a case where the position of the item corresponds to the position in the predetermined area of ​​the image, the item is displayed in the first form, and Control is performed so that, in a case where the position of the item corresponds to a position outside the predetermined area, the item is displayed in the second form.

17. A computer program product comprising a program for causing a computer to execute each step of the control method according to claim 16.

18. A computer-readable storage medium for storing a program for causing a computer to execute each step of the control method according to claim 16.

Images