Electronic devices, methods for controlling electronic devices, programs

Abstract

To provide a technique to create a marker at a position suitable for imaging performed by a user.SOLUTION: An electronic apparatus has: acquisition means that acquires a picked-up image acquired by imaging through one or more optical systems, the picked-up image having an area of an image for each of the optical systems; type determination means that determines a type of a marker to be combined with the picked-up image; determination means that determines the number of the areas of images in the picked-up image; and composition means that, (1) when the number of the areas of images is one, combines the marker at a position according to the type of the marker in the picked-up image, and (2) when the number of the areas of images is two or more, combines the marker, for each of the areas of images, at a position based on information on the optical system used for the imaging and the type of the marker.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an electronic device, a method for controlling an electronic device, and a program.

Background Art

[0002] Two images having parallax can be obtained by one-time shooting, and a stereoscopic image can be displayed using these two images. Patent Document 1 describes that an image for the left eye and an image for the right eye having parallax with each other are arranged and displayed in left and right regions.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In addition, when shooting is performed using a normal single-eye camera, a marker may be displayed at the center position of a single captured image obtained by processing the signal of the imaging sensor. However, when a single captured image obtained using a two-eye lens includes a left-eye region and a right-eye region, if a marker is displayed at the center position of the single captured image, the user cannot grasp the center positions of the respective regions.

[0005] An object of the present invention is to provide a technique for generating a marker at a position suitable for imaging by a user.

Means for Solving the Problems

[0006] One aspect of the present invention is an acquired image obtained by imaging through one or more optical systems, the acquisition means for acquiring an acquired image having a region of an image for each optical system, type determination means for determining the type of a marker to be synthesized in the acquired image A determination means for determining the number of regions of the image in the captured image, 1) When the number of regions in the image is 1, the marker is composited at a position corresponding to the type of marker in the captured image; 2) When the number of regions in the image is 2 or more, the marker is composited for each region of the image at a position based on optical system information relating to the optical system used for imaging and the type of marker. to have death, The optical system information includes the design values ​​of the optical axis center coordinates of each of the multiple optical systems used for imaging, and information on the deviation from the design values. If the number of regions of the image is two or more, the synthesis means determines a reference position for each region of the image based on the design value and the displacement information of each of the multiple optical systems, and synthesizes the markers based on the reference position for each region of the image. This is an electronic device characterized by the following features.

[0007] One aspect of the present invention is, An acquisition step of acquiring an image obtained by imaging through one or more optical systems, wherein the image has a region for each optical system, A type determination step for determining the type of marker to be synthesized on the captured image, A determination step of determining the number of regions of the image in the captured image, 1) If the number of regions in the image is 1, the marker is composited at a position corresponding to the type of marker in the captured image; 2) If the number of regions in the image is 2 or more, the marker is composited for each region of the image at a position based on the optical system information relating to the optical system used for imaging and the type of marker; to have death, The optical system information includes the design values ​​of the optical axis center coordinates of each of the multiple optical systems used for imaging, and information on the deviation from the design values. In the synthesis step, if the number of regions of the image is two or more, the reference position for each region of the image is determined based on the design value and the displacement information for each of the multiple optical systems, and the markers are synthesized based on the reference position for each region of the image. This is a control method for electronic equipment characterized by the following features. [Effects of the Invention]

[0008] According to the present invention, it becomes possible to generate markers at positions suitable for imaging by the user. [Brief explanation of the drawing]

[0009] [Figure 1]It is an external view of the camera. [Figure 2] It is an internal configuration diagram of the camera. [Figure 3] It is a diagram showing the configuration of the lens unit. [Figure 4A] It is a flowchart of the marker control process. [Figure 4B] It is a flowchart of the marker control process. [Figure 5] It is a diagram for explaining lens information. [Figure 6] It is a diagram for explaining lens individual values. [Figure 7] It is a diagram for explaining the center marker. [Figure 8] It is a diagram for explaining the vertical marker. [Figure 9] It is a diagram for explaining the horizontal marker. [Figure 10] It is a diagram for explaining the user marker.

Mode for Carrying Out the Invention

[0010] <Embodiment> Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings. In this embodiment, the case where the electronic device is a digital camera (imaging device) will be described as an example.

[0011] FIGS. 1A and 1B are diagrams showing an example of the external configuration of a digital camera 100 (hereinafter referred to as a camera). FIG. 1A is a perspective view of the camera 100 seen from the front. FIG. 1B is a perspective view of the camera 100 seen from the back.

[0012] Camera 100 has a shutter button 101, a power switch 102, a mode selector switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an external viewfinder display 107 on its top surface. The shutter button 101 is an operation for preparing to shoot or giving a shooting command. The power switch 102 is an operation for switching the power of camera 100 on and off. The mode selector switch 103 is an operation for switching between various modes. The main electronic dial 104 is a rotary operation for changing settings such as shutter speed and aperture. The sub electronic dial 105 is a rotary operation for moving the selection frame (cursor) and advancing images. The video button 106 is an operation for giving a command to start and stop video recording. The external viewfinder display 107 displays various settings such as shutter speed and aperture.

[0013] Furthermore, the camera 100 has a display unit 108, a touch panel 109, a directional key 110, a SET button 111, and an AE lock button 112 on its back. The camera 100 also has a zoom button 113, a playback button 114, a menu button 115, an eyepiece 116, an eyepiece detection unit 118, and a touch bar 119 on its back.

[0014] The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation unit consisting of keys that can be pressed in the up, down, left, and right directions (four directional keys). The camera 100 can perform processing according to the pressed position (position of the directional keys 110).

[0015] The SET button 111 is an operation button that is mainly pressed when confirming a selection item. The AE lock button 112 is an operation button that is pressed when fixing the exposure state in shooting standby mode.

[0016] The zoom button 113 is an operation unit for switching the zoom mode on and off in the live view display (LV display) of the shooting mode. When the zoom mode is on, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 104. The zoom button 113 is also used in playback mode to enlarge the playback image or increase the magnification ratio.

[0017] The playback button 114 is an operation unit for switching between shooting mode and playback mode. By pressing the playback button 114 in shooting mode, the camera switches to playback mode, and the most recent image recorded on the recording medium 227 (described later) can be displayed on the display unit 108.

[0018] The menu button 115 is an operation button that is pressed to display a menu screen on the display unit 108 that allows for various settings. Users can intuitively make various settings using the menu screen displayed on the display unit 108 and the directional keys 110 or SET button 111.

[0019] The eyepiece section 116 is the part for placing the eye into the eyepiece viewfinder (a type of viewfinder) 117. The user can view the image displayed on the internal EVF 217 (Electronic View Finder), which will be described later, through the eyepiece section 116. The eyepiece detection section 118 is a sensor that detects whether or not the user is placing their eye into the eyepiece section 116.

[0020] The touch bar 119 is a line-shaped touch operation area (line touch sensor) capable of accepting touch operations. The touch bar 119 is positioned so that it can be touched by the right thumb when the grip section 120 is held with the right hand (with the little finger, ring finger, and middle finger) so that the shutter button 101 can be pressed with the right index finger. In other words, the touch bar 119 can be operated when the user is looking through the eyepiece viewfinder 117 with their eyepiece in the eyepiece section 116 and is ready to press the shutter button 101 at any time (shooting posture). The touch bar 119 can accept tap operations (touching and releasing without moving within a predetermined period of time), left and right sliding operations (touching and then moving the touch position while keeping the touch on the screen), etc. The touch bar 119 is a different operation area from the touch panel 109 and does not have a display function. The touch bar 119 in this embodiment is a multifunction bar and functions, for example, as an M-Fn bar.

[0021] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, and a communication terminal 124. The grip section 120 is a holding section shaped to be easily gripped by the user with their right hand when holding the camera 100. When the camera 100 is held with the grip section 120 held by the little finger, ring finger, and middle finger of the right hand, the shutter button 101 and the main electronic dial 104 are positioned to be operated by the index finger of the right hand. Similarly, in the same position, the sub electronic dial 105 and the touch bar 119 are positioned to be operated by the thumb of the right hand.

[0022] The thumb rest section 121 (thumb waiting position) is a grip section located on the back of the camera 100, in a position where the thumb of the right hand holding the grip section 120 can be easily rested when no controls are being operated. The thumb rest section 121 is made of rubber material or the like to enhance the holding power (grip feel).

[0023] The terminal cover 122 protects connectors such as connection cables that connect the camera 100 to external devices. The lid 123 protects the recording medium 227 and the slot for storing the recording medium 227, which will be described later, by closing the slot. The communication terminal 124 is a terminal for the camera 100 to communicate with the lens unit 200, which will be described later and can be attached to and detached.

[0024] Figure 2 shows an example of the internal configuration of camera 100. Components identical to those in Figures 1A and 1B are denoted by the same reference numerals, and their descriptions are omitted as appropriate. A lens unit 200 is attached to camera 100.

[0025] First, let's describe the lens unit 200. The lens unit 200 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 200 is a single-lens reflex lens and is an example of a normal lens. The lens unit 200 includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0026] The aperture 201 is configured to have an adjustable aperture diameter. The lens 202 is composed of multiple lenses. The aperture drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the aperture 201. The AF drive circuit 204 drives the lens 202 to focus. The lens system control circuit 205 controls the aperture drive circuit 203, the AF drive circuit 204, etc., based on instructions from the system control unit 50, which will be described later. The lens system control circuit 205 controls the aperture 201 via the aperture drive circuit 203 and focuses by displacing the position of the lens 202 via the AF drive circuit 204. The lens system control circuit 205 can communicate with the camera 100. Specifically, communication takes place via the communication terminal 206 of the lens unit 200 and the communication terminal 124 of the camera 100. The communication terminal 206 is a terminal for the lens unit 200 to communicate with the camera 100.

[0027] Next, the camera 100 will be described. The camera 100 includes a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0028] The shutter 210 is a focal-plane shutter that can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 is an image sensor composed of a CCD or CMOS element that converts an optical image into an electrical signal. The imaging unit 211 may have an image plane phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal.

[0029] The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing, and color conversion) on data from the A / D converter 212 or the memory control unit 213. The image processing unit 214 also performs predetermined calculations using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. This process enables TTL (through-the-lens) AF processing, AE (automatic exposure) processing, and EF (flash pre-flash) processing. Furthermore, the image processing unit 214 performs predetermined calculations using the captured image data and performs TTL AWB (auto white balance) processing based on the obtained calculation results.

[0030] Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, as well as image data for display on the display unit 108 and EVF 217. The memory 215 has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, and audio. The memory 215 also serves as a memory for image display (video memory).

[0031] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 and EVF 217. Therefore, the display image data written to the memory 215 is displayed on the display unit 108 and EVF 217 via the D / A converter 216. The display unit 108 and EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and EVF 217 are displays such as LCDs and OLEDs. The digital signal, which has been A / D converted by the A / D converter 212 and stored in the memory 215, is converted into an analog signal by the D / A converter 216 and sequentially transferred to the display unit 108 and EVF 217 for display, thereby performing live view display.

[0032] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 50 controls the entire camera 100. The system control unit 50 implements each process of the flowchart described later by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, D / A converter 216, display unit 108, EVF 217, etc.

[0033] The camera 100 also includes a system memory 218, a non-volatile memory 219, a system timer 220, a communication unit 221, a posture detection unit 222, and an eyepiece detection unit 118.

[0034] System memory 218 may be RAM, for example. System memory 218 stores constants and variables for the operation of the system control unit 50, as well as programs read from non-volatile memory 219. Non-volatile memory 219 is electrically erasable and recordable memory. For example, EEPROM may be used for non-volatile memory 219. Non-volatile memory 219 stores constants and programs for the operation of the system control unit 50. The program referred to here is a program for executing the flowchart described later.

[0035] The system timer 220 is a timing unit that measures the time used for various controls and the time of the built-in clock. The communication unit 221 transmits and receives video signals and audio signals to and from external devices connected by wireless or wired cables.

[0036] The communication unit 221 can connect to a wireless LAN (Local Area Network) and the internet. Furthermore, the communication unit 221 can communicate with external devices using Bluetooth® and Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 and images recorded on the recording medium 227, and can receive image data and other various information from external devices.

[0037] The attitude detection unit 222 detects the attitude of the camera 100 relative to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 was taken with the camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211, or rotate the image according to the detected attitude. For example, an acceleration sensor or a gyroscope sensor can be used for the attitude detection unit 222. It is also possible to use the attitude detection unit 222 to detect the movement of the camera 100 (pan, tilt, lift, whether it is stationary or not, etc.).

[0038] The eyepiece detection unit 118 can detect the approach of any object to the eyepiece section 116 of the eyepiece finder 117, which incorporates the EVF 217. The eyepiece detection unit 118 can detect, for example, infrared A linear proximity sensor can be used. When an object approaches, infrared light emitted from the light emitter of the eyepiece detection unit 118 is reflected by the object and received by the light receiver of the infrared proximity sensor. The distance from the eyepiece unit 116 to the object can be determined by the amount of infrared light received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity distance of an object to the eyepiece unit 116.

[0039] The eyepiece detection unit 118 is an eyepiece detection sensor that detects when an eye (object) approaches (eyepieces) and moves away (eyes away) from the eyepiece section 116 of the eyepiece finder 117. The eyepiece detection unit 118 detects that an eyepiece has been placed when an object that approaches within a predetermined distance from the eyepiece section 116 is detected from a non-eyepiece state (non-approach state). On the other hand, the eyepiece detection unit 118 detects that an eye has been removed when the object that was detected approaching moves away from the eyepiece state (approach state) by a predetermined distance or more. The threshold for detecting eyepieces and the threshold for detecting eye separation may be different, for example, by providing hysteresis. After detecting eyepieces, the eyepiece is considered to be in an eyepiece state until eye separation is detected. After detecting eye separation, the eyepiece is considered to be in a non-eyepiece state until eyepieces are detected again.

[0040] The system control unit 50 switches the display (display state) / hidden (hidden state) of the display unit 108 and EVF 217 according to the state detected by the eyepiece detection unit 118. Specifically, when the camera is in at least the shooting standby state and the display destination switching setting is set to automatic switching, the display destination is set to the display unit 108 and the EVF 217 is hidden when the user is not using an eyepiece. When the user is using an eyepiece, the display destination is set to the EVF 217 and the display unit 108 is hidden. Note that the eyepiece detection unit 118 is not limited to an infrared proximity sensor; other sensors that can detect a state that can be considered as eyepiece use may be used.

[0041] The camera 100 also includes an external viewfinder display unit 107, an external viewfinder display drive circuit 223, a power control unit 224, a power supply unit 225, a recording medium interface 226, an operation unit 228, and the like.

[0042] The external viewfinder display unit 107 displays various settings of the camera 100, such as shutter speed and aperture, via the external viewfinder display drive circuit 223.

[0043] The power control unit 224 includes a battery detection circuit, a DC-DC converter, and a switch circuit for switching which blocks are energized. The power control unit 224 detects whether a battery is installed, the type of battery, and the remaining battery level. Based on the detection results and instructions from the system control unit 50, the power control unit 224 controls the DC-DC converter and supplies the necessary voltage to each part, including the recording medium 227, for the required period of time.

[0044] The power supply unit 225 is a primary battery (such as an alkaline battery or a lithium battery), a secondary battery (such as a NiCd battery, NiMH battery, or Li battery), or an AC adapter.

[0045] The recording medium I / F 226 is an interface to a recording medium 227, such as a memory card or hard disk. The recording medium 227 is a memory card or the like for recording captured images, and is composed of semiconductor memory or magnetic disks. The recording medium 227 may be detachable from the camera 100, or it may be built into the camera 100.

[0046] The operation unit 228 is an input unit that receives user input (user operation) and is used to input various instructions to the system control unit 50. The operation unit 228 includes a shutter button 101, a power switch 102, a mode selector switch 103, a touch panel 109, and other operation units 229.

[0047] Other control units 229 include a main electronic dial 104, a sub electronic dial 105, a video button 106, a directional key 110, a SET button 111, an AE lock button 112, a zoom button 113, a play button 114, a menu button 115, a touch bar 119, and the like.

[0048] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 turns on during the operation of the shutter button 101, so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. Upon generation of the first shutter switch signal SW1, the system control unit 50 starts shooting preparation processing (AF processing, AE processing, AWB processing, or EF processing, etc.). The second shutter switch 231 turns on when the operation of the shutter button 101 is completed, so-called full-press (shooting instruction), and generates a second shutter switch signal SW2. Upon generation of the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processes (from reading the signal from the imaging unit 211 to generating an image file containing the captured image and writing it to the recording medium 227).

[0049] The mode switch 103 switches the operating mode of the system control unit 50 to one of the following: still image shooting mode, video shooting mode, or playback mode. Modes included in still image shooting mode include auto shooting mode, auto scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). Modes included in still image shooting mode also include various scene modes and custom modes, which are shooting settings for different shooting scenes. The user can switch directly to any of the above shooting modes using the mode switch 103. Alternatively, the user can switch to the shooting mode list screen using the mode switch 103, and then selectively switch to one of the displayed modes using the operation unit 228. Similarly, the video shooting mode may also include multiple modes.

[0050] The touch panel 109 is a touch sensor that detects various touch operations on the display surface of the display unit 108 (the operating surface of the touch panel 109). The touch panel 109 and the display unit 108 can be configured as an integrated unit. For example, the touch panel 109 is mounted on the upper layer of the display surface of the display unit 108 so that its light transmittance does not interfere with the display of the display unit 108. By associating the input coordinates on the touch panel 109 with the display coordinates on the display surface of the display unit 108, a GUI (Graphical User Interface) can be configured that makes it appear as if the user can directly operate the screen displayed on the display unit 108.

[0051] The touch panel 109 can use any of the following methods: resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. Depending on the method, a touch may be detected when there is contact with the touch panel 109, or when a finger or pen approaches the touch panel 109. Either method is acceptable.

[0052] The system control unit 50 can detect the following operations or states on the touch panel 109. - A finger or pen that was not previously touching the touch panel 109 now touches the touch panel 109, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which the touch panel 109 is being touched with a finger or pen (hereinafter referred to as Touch-On). • The touch panel 109 is being moved while a finger or pen is touching it (hereinafter referred to as Touch-Move). • The finger or pen that was touching the touch panel 109 was lifted off the touch panel 109 (Lili The touch has been completed, that is, the touch has ended (hereinafter referred to as Touch-Up). • The state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0053] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. Touch-ons are also detected simultaneously if a touch-move is detected. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. After all fingers or pens that were touching have been detected as having touched up, a touch-off occurs.

[0054] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 109, are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 determines what kind of operation (touch operation) was performed on the touch panel 109. For touch moves, the direction of movement of the finger or pen moving on the touch panel 109 can also be determined for each vertical and horizontal component on the touch panel 109 based on the change in position coordinates. If it is detected that a touch move has occurred beyond a predetermined distance, it is determined that a slide operation has been performed.

[0055] A flick is an operation in which you touch the touch panel 109 with your finger, quickly move it a certain distance, and then release it. In other words, a flick is an operation in which you quickly trace the touch panel 109 with your finger as if flicking it. A flick is determined to have occurred when a touch move of a certain distance or more at a certain speed or more is detected, and a touch up is detected immediately afterward (it can be determined that a flick occurred following a slide operation). Furthermore, a touch operation in which you touch multiple locations (for example, two points) together (multitouch) and bring the touch positions closer together is called a pinch-in, and a touch operation in which you move the touch positions further apart is called a pinch-out. Pinch-out and pinch-in are collectively referred to as a pinch operation (or simply a pinch).

[0056] Figure 3 is a schematic diagram showing an example of the configuration of the lens unit 300. Figure 3 shows the lens unit 300 attached to the camera 100. Note that components of the camera 100 shown in Figure 3 that are the same as those described in Figure 2 are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0057] The lens unit 300 is a type of interchangeable lens that can be attached to and removed from the camera 100. The lens unit 300 is a twin-lens system that enables parallax in the left and right images. The lens unit 300 has two optical systems, each with a wide field of view of approximately 180 degrees, and can capture an area of ​​the front hemisphere. Specifically, the two optical systems of the lens unit 300 can each capture subjects within a field of view (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation / depression angle, pitch angle).

[0058] The lens unit 300 includes a right-eye optical system 301R having multiple lenses and a reflective mirror, a left-eye optical system 301L having multiple lenses and a reflective mirror, and a lens system control circuit 303. The right-eye optical system 301R is an example of a first optical system, and the left-eye optical system 301L is an example of a second optical system. The right-eye optical system 301R has a lens 302R positioned on the subject side, and the left-eye optical system 301L has a lens 302L positioned on the subject side. Lenses 302R and 302L face the same direction, and their optical axes are approximately parallel.

[0059] The lens unit 300 of this embodiment is a VR180 lens for obtaining images in the so-called VR180 format, which is a VR (Virtual Reality) image format that enables binocular stereoscopic viewing. In the VR180 lens, the right eye optical system 301R and the left eye optical system 301L each have a fisheye lens capable of capturing a range of approximately 180 degrees. The VR180 lens only needs to be capable of acquiring images that enable binocular VR display as VR180, with the right eye optical system 301R and the left eye optical system 301L each capable of capturing a wide field of view of about 160 degrees, which is narrower than the 180-degree range. The VR180 lens can form a right image (first image) formed via the right eye optical system 301R and a left image (second image) formed via the left eye optical system 301L, which has parallax with the right image, onto one or two image sensors of the attached camera.

[0060] Furthermore, the lens unit 300 is attached to the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. When the lens unit 300 is attached to the camera 100, the system control unit 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 are electrically connected via the communication terminals 124 and 306.

[0061] In this embodiment, the right image formed via the right-eye optical system 301R and the left image formed via the left-eye optical system 301L, which has parallax with respect to the right image, are imaged side by side on the imaging unit 211 of the camera 100. That is, two optical images formed by the right-eye optical system 301R and the left-eye optical system 301L are formed on a single image sensor. The imaging unit 211 converts the imaged subject (optical signal) into an analog electrical signal. In this way, by using the lens unit 300, two images with parallax can be acquired simultaneously (as a set) from two locations (optical systems) - the right-eye optical system 301R and the left-eye optical system 301L. Furthermore, by separating the acquired images into a left-eye image and a right-eye image and displaying them in VR, the user can view a three-dimensional VR image with a range of approximately 180 degrees, so-called VR180.

[0062] Here, a VR image is an image that can be displayed in VR, as described later. VR images include omnidirectional images (spherical images) taken with an omnidirectional camera (spherical camera), and panoramic images with a wider field of view (effective field of view) than the display area that can be shown on the display unit at once. Furthermore, VR images are not limited to still images, but also include videos and live images (images acquired from a camera in near real-time). A VR image has a field of view (effective field of view) of up to 360 degrees horizontally and vertically. In addition, VR images also include images that have a wider field of view than that that can be captured by a normal camera, or a wider field of view than the display area that can be shown on the display unit at once, even if the field of view is less than 360 degrees horizontally or vertically.

[0063] The image captured by the camera 100 using the lens unit 300 described above is a type of VR image. VR images can be displayed in VR, for example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR View". By displaying a VR image with a 360-degree field of view in VR, the user can change the orientation of the display device left or right (horizontal rotation direction) to view a seamless, all-around image in the left-right direction.

[0064] Here, VR display (VR view) refers to a display method (display mode) in which the display range of a VR image is changed, displaying an image within the field of view range corresponding to the orientation of the display device. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by performing a transformation (a transformation with distortion correction) that maps the VR image to a virtual sphere. VR display also includes "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side-by-side in the left and right regions by performing transformations that map each to a virtual sphere. By using "two-eye VR display" with a VR image for the left eye and a VR image for the right eye that have parallax with each other, the user can experience stereoscopic vision.

[0065] Regardless of the VR display method, for example, when a user wears a display device such as an HMD (Head-Mounted Display), the image displayed will have a field of view that corresponds to the orientation of the user's face. For example, suppose a VR image is displaying a field of view centered at 0 degrees horizontally (a specific direction, e.g., north) and 90 degrees vertically (90 degrees from the zenith, i.e., horizontal) at a given point in time. If the orientation of the display device is reversed (for example, changing the display surface from facing south to facing north), the display range of the same VR image will change to a field of view centered at 180 degrees horizontally (the opposite direction, e.g., south) and 90 degrees vertically. In other words, if a user wearing an HMD turns their face from north to south (i.e., turns their back), the image displayed on the HMD will also change from a north image to a south image.

[0066] Note that the VR image captured using the lens unit 300 of this embodiment is a VR180 image capturing a range of approximately 180 degrees in front, and there is no image of a range of approximately 180 degrees behind. When such a VR180 image is displayed in VR, if the orientation of the display device is changed to the side where there is no image, a blank area will be displayed.

[0067] By displaying VR images in VR in this way, users can visually experience the sensation of being inside the VR image (in a VR space). Note that the method of displaying VR images is not limited to changing the orientation of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel or directional buttons. Furthermore, during VR display (in the "VR View" display mode), in addition to changes in the display range due to changes in orientation, the display range may also be changed in response to touch movements on the touch panel, drag operations with a mouse, or pressing of directional buttons. Note that a smartphone mounted on VR goggles (head-mounted adapter) is a type of HMD (Head-Mounted Display).

[0068] The image captured by the camera 100 via the lens unit 300 is divided into two regions because it is captured via the right-eye optical system and the left-eye optical system, respectively.

[0069] (Regarding marker control processing) Referring to the flowcharts in Figures 4A and 4B, the marker control process that controls the display of markers (indicators used by photographers as a guide when deciding on the composition of a shot) will be explained. The flowcharts in Figures 4A and 4B are realized by the system control unit 50 loading the program recorded in the non-volatile memory 219 into the system memory 218 and executing it. Furthermore, the process in the flowchart of Figure 4A starts when the digital camera 100 is set to still image shooting mode or video shooting mode.

[0070] Furthermore, users can pre-configure settings such as whether or not to display markers (a function used as a guide when photographers decide on the composition of their shots) and the type of marker (the display format of the markers when they are displayed).

[0071] In step S401, the system control unit 50 acquires the captured image generated by the imaging process of the imaging unit 211.

[0072] In step S402, the system control unit 50 acquires information about the type of lens unit attached to the camera 100 (hereinafter referred to as the "attached lens unit"). Then, the system control unit 50 refers to the information about the type of the attached lens unit and determines whether or not the attached lens unit is a twin-lens system. Specifically, the system control unit 50 acquires information about the type of the attached lens unit from the attached lens unit by communicating with the attached lens unit via communication terminals 124, 206, and 306. If it is determined that the attached lens unit is a twin-lens system, the system proceeds to step S403. On the other hand, if it is determined that the attached lens unit is not a twin-lens system, the system proceeds to step S433 in Figure 4B. If it is determined that the attached lens unit is not a twin-lens system, for example, if it is determined that the lens unit is not attached, or if the attached lens unit is a normal lens unit such as a single-lens system. This is the case when it is determined to be a twin lens. The system control unit 50 may also determine whether the mounted lens unit is a twin lens based on the lens flag (described later) in the lens information.

[0073] In step S402, the system control unit 50 only needs to determine whether the number of image regions corresponding to the optical system (image regions included in the captured image) is 2 or not. For example, the system control unit 50 can determine whether the number of image regions is 2 or not by analyzing the captured image. In this case, if the number of image regions is 2, the system proceeds to step S403, and if the number of image regions is 1, the system proceeds to step S433 in Figure 4B.

[0074] In step S403, the system control unit 50 determines whether the setting for displaying markers is set to "display markers" (i.e., the setting to display markers is enabled). If it is determined that the setting is "display markers," the process proceeds to step S410. If it is determined that the setting is "display markers," the process proceeds to step S404.

[0075] In step S404, the system control unit 50 generates a live view display image (LV display image to be displayed on the display unit 108) from the captured image acquired in step S401. The live view display image may also include icons indicating the shooting settings.

[0076] In step S405, the system control unit 50 displays the live view image on the display unit 108.

[0077] In step S406, the system control unit 50 determines whether or not it has received an instruction to terminate the LV display (an instruction from the user). If it is determined that an instruction to terminate has been received, the process in this flowchart ends. If it is determined that an instruction to terminate has not been received, the process proceeds to step S401.

[0078] In step S410, the system control unit 50 calculates the coordinates of the center positions (center coordinates) of the region of the right image (first image) formed via the right-eye optical system 301R formed on the image sensor, and the region of the left image (second image) formed via the left-eye optical system 301L. When a bilens system is attached to the camera 100, errors occur in the coordinates of the center positions of the right image region and the left image region from the design values ​​due to the reasons described below. Therefore, in step S410, the system control unit 50 corrects the center positions of each region obtained by dividing the image formed on the image sensor (captured image) into left and right halves, according to the lens information (information on the image circle position and image circle position shift described below). As a result, the system control unit 50 calculates the coordinates of the center position of the right image and the center position of the left image in the captured image as reference position coordinates when determining the position for compositing the markers. Note that the coordinates calculated in step S410 may not be the coordinates of the center position, but may be, for example, the coordinates of the right edge, left edge, upper edge, and lower edge of each image region. Furthermore, the coordinates calculated in step S410 may be, for example, the coordinates of the four vertices if the region of each image is rectangular, or the coordinates of the two foci if the region of each image is elliptical. In other words, the coordinates calculated in step S410 may be the coordinates of any reference position, as long as they serve as the reference coordinates when compositing the markers.

[0079] Figure 5A is a schematic diagram showing an example of lens information (information about the optical system mounted on the camera 100; optical system information). The system control unit 50 of the camera 100 acquires lens information from the lens unit 300 via the communication terminal 124 and the communication terminal 306 of the lens unit 300. The lens information includes the lens design value, the individual lens value, the lens flag, and the lens focal length. This includes distance and lens temperature, etc.

[0080] Lens design values ​​are design values ​​for aberration correction. During the manufacturing process of a twin-lens system, errors such as lens eccentricity and tilt occur in each of the two optical systems (left-eye optical system 301L and right-eye optical system 301R). Lens individual values ​​are measurement results of errors (errors from set values) detected during the manufacturing process of the twin-lens system. Details of lens design values ​​and lens individual values ​​will be described later using Figure 5B. The lens flag is a flag indicating that it is a twin-lens system. The lens flag can be used to identify whether the lens unit 300 attached to the camera 100 is a twin-lens system or not. Lens focal length is the distance from the "principal point," which is the center of the lens, to the image sensor (imaging position). Lens focal length may or may not be a common parameter for the two optical systems of the twin-lens system (left-eye optical system 301L and right-eye optical system 301R). Lens temperature is the temperature of the twin-lens system and is used to understand the ambient temperature at the time of shooting.

[0081] Figure 5B is a schematic diagram showing details of lens design values ​​and individual lens values. Lens design values ​​include image circle position, image circle diameter, angle of view, and distortion correction coefficient.

[0082] The image circle position is the coordinate of the optical axis center of the optical system in the captured image (design value of the optical axis center coordinate), and is provided for each of the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R). In other words, the image circle position is the coordinate of the center of the image circle (circular fisheye image) formed on the image sensor (coordinate of the center position), and is provided for both the right image and the left image. The origin of the coordinates is, for example, the center of the image sensor (center of the captured image). The image circle position includes horizontal and vertical coordinates. Various information regarding the optical axis center of the optical system in the captured image can be used as the image circle position. For example, the distance from a predetermined position in the captured image (such as the center or upper left corner) to the optical axis center can be used as the image circle position.

[0083] The image circle diameter is the diameter of the image circle (circular fisheye image) formed on the image sensor. The field of view is the field of view of the image circle (circular fisheye image) formed on the image sensor. The distortion correction coefficient is the ratio of the design image height to the ideal image height of the lens. A distortion correction coefficient may be set for each image height, and for image heights for which a distortion correction coefficient has not been set, the distortion correction coefficient may be calculated by interpolation using multiple distortion correction coefficients. A polynomial approximating the relationship between image height and distortion correction coefficient may also be set. The image circle diameter, field of view, and distortion correction coefficient may or may not be common parameters for the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R).

[0084] Lens individual values ​​include image circle misalignment, optical axis tilt, and image magnification misalignment. This information is prepared by measuring each of the two optical systems of the twin-lens system (left eye optical system 301L and right eye optical system 301R).

[0085] Image circle misalignment is the deviation of the center coordinates of the image circle (circular fisheye image) formed on the image sensor from the design value. For example, image circle misalignment includes both horizontal and vertical misalignment. With the design value coordinates (a two-dimensional coordinate system including horizontal and vertical coordinates) as the origin, the horizontal misalignment is shown by the horizontal coordinates, and the vertical misalignment is shown by the vertical coordinates.

[0086] Figure 6A shows an example of image circle misalignment. Region 601 represents the right or left half of the sensor size screen (imaging plane). Image circle 602 is the actual image circle of the right or left lens. Image circle 603 is the image circle of the ideal optical system (design value). This is an image circle. There is a discrepancy between the actual position of image circle 602 and the ideal image circle 603.

[0087] Optical axis tilt is the deviation of the optical axis orientation on the subject side from the design value. For example, optical axis tilt includes both horizontal and vertical deviations. The deviation in each direction is expressed as an angle. Figure 6B shows an example of horizontal deviation due to optical tilt. Line 612 shows the actual horizontal optical axis orientation (tilt) of the right or left lens. Line 613 shows the horizontal optical axis orientation of the ideal optical system (design value). A deviation occurs between the actual horizontal optical axis orientation 612 and the ideal horizontal optical axis orientation 613.

[0088] Figure 6C also shows an example of vertical displacement due to optical tilt. Line 622 indicates the actual horizontal optical axis orientation of the right or left lens. Line 623 indicates the horizontal optical axis orientation of the ideal optical system (design value). A displacement occurs between the actual horizontal optical axis orientation 622 and the ideal horizontal optical axis orientation 623.

[0089] Image magnification shift is the deviation of the size of the image circle (circular fisheye image) formed on the image sensor from the design value. This deviation is expressed, for example, as a ratio to the design value. Figure 6D shows an example of image magnification shift. Image circle 632 is the image circle formed on the image sensor through the actual right or left lens. Image circle 633 is the image circle formed on the image sensor through the ideal optical system (design value). A deviation occurs between the actual image circle 632 and the image circle 633 of the ideal optical system.

[0090] The information included in the lens information is not limited to the information described above. For example, the lens information may include the boundary positions of the right and left images in the captured image (the position of the edge of the circular fisheye image). The lens information may also include the midpoint coordinates between the right and left images in the captured image. In many cases, the midpoint coordinates coincide with the center coordinates of the captured image.

[0091] As indicated by the lens design values ​​and individual lens values ​​mentioned above, the image circle position of the lens unit 300 mounted on the digital camera 100 is subject to error. Therefore, the center of the area obtained by simply dividing the screen left and right does not coincide with the center of the left and right images.

[0092] In step S411, the system control unit 50 determines whether the marker type (type of marker) set by the user is a center marker. If it is determined that the marker type is a center marker, the process proceeds to step S412. If it is determined that the marker type is not a center marker, the process proceeds to step S413.

[0093] In step S412, the system control unit 50 synthesizes center markers at the center positions of the right image region and the left image region calculated in step S410 to generate a live view display image (composite image).

[0094] In step S413, the system control unit 50 determines whether the marker type set by the user is a vertical marker. If it is determined that the marker type is a vertical marker, the process proceeds to step S414. If it is determined that the marker type is not a vertical marker, the process proceeds to step S415.

[0095] In step S414, the system control unit 50 combines two vertical markers (a vertical line passing through the center position of the right image region calculated in step S410 and a vertical line passing through the center position of the left image region) into the captured image. As a result, the system control unit 50 generates a live view display image (composite image).

[0096] In step S415, the system control unit 50 determines whether the marker type set by the user is a horizontal marker. If it is determined that the marker type is a horizontal marker, the process proceeds to step S416. If it is determined that the marker type is not a horizontal marker, the process proceeds to step S421.

[0097] In step S416, the system control unit 50 composites horizontal markers (a horizontal line passing through the center position of the right image region calculated in step S410 and a horizontal line passing through the center position of the left image region) onto the captured image. This allows the system control unit 50 to generate an image for live view display.

[0098] The system control unit 50 may determine whether the region of the right image (first image) formed via the right eye optical system 301R and the region of the left image (second image) formed via the left eye optical system 301L are displayed side by side or side by side vertically. The system control unit 50 may then display the horizontal marker only when the regions of the two images are displayed side by side. The system control unit 50 may also display the horizontal marker when the regions of the two images are displayed side by side vertically. The arrangement of the regions of the right image and the left image is determined by the orientation of the camera at the time of shooting, the structure of the lens unit, or user settings.

[0099] In step S421, the system control unit 50 determines that the marker type set by the user is a user marker, and determines whether the display position of the user marker has already been set. If it is determined that the display position of the user marker has already been set, the process proceeds to step S422. If it is determined that the display position of the user marker has not already been set, the process proceeds to step S423.

[0100] In step S422, the system control unit 50 synthesizes user markers at the positions of the user markers set in the right image region and at the display positions of the user markers set in the left image region. This allows the system control unit 50 to generate an image for live view display.

[0101] In step S423, the system control unit 50 adjusts the display position of the user marker in either the right image region or the left image region (the region of the image selected by the user) according to the user's operation. This allows the system control unit 50 to set (determine) the display position of the user marker.

[0102] In step S424, the system control unit 50 sets (determines) the display position of the user marker for the right image region and the left image region that was not used for user adjustment in step S423. The system control unit 50 sets the position determined in step S423 to a corrected position as the display position of the user marker, according to the lens design value and lens individual value information described above. For example, the system control unit 50 sets the display positions of the user markers in the right image region and the left image region so that the relative position of the user marker display position with respect to the center position of the right image region is the same as the relative position of the user marker display position with respect to the center position of the left image region.

[0103] In step S425, the system control unit 50 superimposes (conjugates) user markers onto the display positions set in steps S423 and S424 of the captured image. This allows the system control unit 50 to generate an image for live view display.

[0104] In step S433 of Figure 4B, the system control unit 50, similar to step S403, The system determines whether the setting is "Marker display enabled" (i.e., whether the setting is configured to display markers). If it is determined that the setting is "Marker display enabled," the system proceeds to step S440, where the system control unit 50 obtains the coordinates of the center position of the screen (the entire captured image) of the display unit 108. If it is determined that the setting is "Marker display disabled," the system proceeds to step S404.

[0105] In steps S441 to S446, the system control unit 50 performs the same processing as in steps S411 to S416 based on the coordinates of the center position acquired in step S440. Note that in steps S411 to S416, the system control unit 50 composited a marker into the captured image for each region of the image, but in steps S441 to S446, it composites only one marker into the captured image (see Figures 7A, 8A, and 9A).

[0106] In step S451, the system control unit 50 determines that the marker type set by the user is a user marker, and determines whether the display position of the user marker has already been set. If it is determined that the display position of the user marker has already been set, the process proceeds to step S452. If it is determined that the display position of the user marker has not already been set, the process proceeds to step S453.

[0107] In step S452, the system control unit 50 composites the user marker at the designated display position of the user marker. This allows the system control unit 50 to generate an image for live view display.

[0108] In step S453, the system control unit 50 adjusts the display position of the user marker on the entire screen (captured image) of the display unit 108 according to the user's operation and sets (determines) the display position of the user marker.

[0109] As described above, the system control unit 50 can operate as a type determination unit that determines the type of marker set by the user, and as a determination unit that determines the number of image regions in the captured image. The system control unit 50 can also operate as a generation unit that generates a live view display image by combining the markers with the captured image.

[0110] (Regarding the display of markers) The following explanation of marker display uses examples of live view displays shown in Figures 7A to 10B. Note that the marker display forms in each figure are examples only and are not limited to those shown. For example, the shape of the center marker and user marker may be star-shaped or circular. Horizontal and vertical markers may be represented by dashed or dotted lines.

[0111] Figure 7A shows the live view display 701 when the lens unit attached to the camera 100 is a single-lens reflex lens and the marker type is a center marker (YES in step S441). In the example in Figure 7A, the center marker 702 is displayed at the center of the screen (captured image).

[0112] Figure 7B shows the live view display 703 when the lens unit mounted on the camera 100 is a twin-lens system and the marker type is a center marker (YES in step S411). Region 704 is the region of the right image obtained by imaging via the right eye optical system 301R in the lens unit 300. Region 705 is the region of the left image obtained by imaging via the left eye optical system 301L in the lens unit 300. In the example in Figure 7B, the center marker 706 is displayed at the center of region 704, and the center marker 707 is displayed at the center of region 705.

[0113] Figure 8A shows the live view display 801 when the lens unit attached to the camera 100 is a single-lens reflex lens and the marker type is a vertical marker (YES in step S443). In the example in Figure 8A, a vertical marker 802 is displayed that passes through the center of the screen (captured image) and extends in the vertical direction.

[0114] Figure 8B shows the live view display 803 when the lens unit attached to the camera 100 is a twin-lens system and the marker type is a vertical marker (YES in step S413). The vertical marker 806 passes through the center position in the right image region 804 and extends vertically. The vertical marker 807 passes through the center position in the left image region 805 and extends vertically.

[0115] Figure 9A shows the live view display 901 when the lens unit attached to the camera 100 is a single-lens reflex lens and the marker type is a horizontal marker (YES in step S445). In the example in Figure 9A, a horizontal marker 902 is displayed that passes through the center of the screen (captured image) and extends in the left-right direction.

[0116] Figure 9B shows the live view display 903 when the lens unit attached to the camera 100 is a twin-lens system and the marker type is a horizontal marker (YES in step S415). The horizontal marker 906 passes through the center position in the right image region 904 and extends in the left-right direction. The horizontal marker 907 passes through the center position in the left image region 905 and extends in the left-right direction.

[0117] Figure 10A shows the live view display 1001 when the lens unit attached to the camera 100 is a single-lens reflex lens and the marker type is a user marker (NO in step S445). In the example in Figure 10A, the user marker 1002 is displayed at the display position set by the user.

[0118] Figure 10B shows an example of the live view display 1003 when the lens unit attached to the camera 100 is a twin-lens system and the marker type is a user marker (when NO is selected in step S415). User marker 1006 is a user marker displayed in the area 1004 of the right image. User marker 1007 is a user marker displayed in the area 1005 of the left image.

[0119] In this embodiment, examples of displaying markers on live view images using a single-lens and a dual-lens system have been described. However, markers may also be displayed on images captured using a compound lens with three or more lenses instead of a dual-lens system. In other words, the captured image may include the regions of two or more images (optical images). When displaying markers on images captured using a compound lens with three or more lenses (a lens unit with three or more optical systems), in steps S411 to S425, markers are combined for each image region corresponding to the optical system. For example, if the marker type is a horizontal marker, a horizontal marker passing through the center position of each image region is displayed. If the marker type is a user marker, a user marker is displayed at the user-set display position in one image region, and in the regions of other images (other than one image), a user marker is displayed at a position corrected from the user-set display position for each image region.

[0120] Furthermore, although this embodiment describes four types of markers (center marker, vertical marker, horizontal marker, and user marker), the marker types may also include other markers such as grid markers.

[0121] Furthermore, although this embodiment describes the case where the imaging device has one image sensor, markers may also be displayed on images captured using an imaging device that uses two or more image sensors.

[0122] According to this embodiment, the system control unit 50 determines the type of lens unit attached to the camera 100 and controls the position of the marker according to the determination result. Furthermore, if the lens unit attached to the camera 100 is a twin-lens system, the system control unit 50 controls (corrects) the position of the marker for each image region according to information about the lens (optical system) (such as lens design values ​​or individual lens values). This makes it possible to display the marker in a position suitable for imaging by the user.

[0123] Furthermore, in the above, the statement "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" can be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, the statement "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" can be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, the expression "greater than or equal to A" can be replaced with "A or greater than (higher; longer; more)" or rephrased as "greater than (higher; longer; more)." On the other hand, the expression "less than or equal to A" can be replaced with "A or less than (lower; shorter; fewer)" or rephrased as "less than (lower; shorter; fewer)." Furthermore, "larger than A (higher; longer; more)" can be rephrased as "greater than or equal to A," and "smaller than A (lower; shorter; fewer)" can be rephrased as "less than or equal to A."

[0124] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0125] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0126] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0127] The above-disclosed embodiments include the following configurations, methods, and programs. [Configuration 1] An acquisition means for acquiring an image obtained by imaging through one or more optical systems, the image having a region for each optical system, A type determination means for determining the type of marker to be synthesized on the captured image, A determination means for determining the number of regions of the image in the captured image, 1) When the number of regions in the image is 1, the marker is composited at a position corresponding to the type of marker in the captured image; 2) When the number of regions in the image is 2 or more, the marker is composited for each region of the image at a position based on optical system information relating to the optical system used for imaging and the type of marker. An electronic device characterized by having the following features. [Configuration 2] The optical system information includes the design values ​​of the optical axis center coordinates of each of the multiple optical systems used for imaging, and information on the deviation from the design values. The electronic device according to configuration 1, characterized by the features described above. [Configuration 3] If the number of regions of the image is two or more, the synthesis means determines a reference position for each region of the image based on the design value and the displacement information of each of the multiple optical systems, and synthesizes the markers based on the reference position for each region of the image. The electronic device according to configuration 2, characterized by the features described above. [Structure 4] If the type of the marker indicates a first marker, the synthesis means 1) synthesizes the first marker at the central position in the captured image if the number of regions in the image is one, and 2) if the number of regions in the image is two or more, it determines the central position for each region of the image based on the optical system information and synthesizes the first marker at the central position for each region of the image. An electronic device according to any one of configurations 1 to 3, characterized by the features described herein. [Composition 5] If the type of the marker indicates a second marker, the synthesis means 1) synthesizes the second marker extending in the left-right direction through the central position in the captured image if the number of regions in the image is one, and 2) if the number of regions in the image is two or more, it determines the central position for each region of the image based on the optical system information, and synthesizes the second marker extending in the left-right direction through the central position of each region of the image. An electronic device according to any one of configurations 1 to 4, characterized by the features described herein. [Composition 6] If the type of the marker indicates a third marker, the synthesis means 1) synthesizes the third marker extending vertically through the central position in the captured image if the number of regions in the image is one, and 2) if the number of regions in the image is two or more, it determines the central position for each region of the image based on the optical system information and synthesizes the third marker extending vertically through the central position of each region of the image. An electronic device according to any one of configurations 1 to 5, characterized by the features described herein. [Composition 7] If the type of the marker indicates a fourth marker, the combining means 1) if the number of regions of the image is one, combine the fourth marker at a user-defined position in the captured image; and 2) if the number of regions of the image is two or more, combine the fourth marker with the first image region at a user-defined position, and for each of the image regions other than the first image, combine the fourth marker at a position corrected for each image region based on the optical system information from the user-defined position. An electronic device according to any one of configurations 1 to 6, characterized by the features described herein. [Structure 8] The determination means determines the number of image regions based on information about the type of lens unit including the one or more optical systems. An electronic device according to any one of configurations 1 to 7, characterized by the features described herein. [Composition 9] The aforementioned marker is a marker used by the user as a guide when deciding on the composition of the photograph. be, An electronic device according to any one of configurations 1 to 8, characterized by the above. [method] An acquisition step of acquiring an image obtained by imaging through one or more optical systems, wherein the image has a region for each optical system, A type determination step for determining the type of marker to be synthesized on the captured image, A determination step of determining the number of regions of the image in the captured image, 1) If the number of regions in the image is 1, the marker is composited at a position corresponding to the type of marker in the captured image; 2) If the number of regions in the image is 2 or more, the marker is composited for each region of the image at a position based on the optical system information relating to the optical system used for imaging and the type of marker; A method for controlling electronic equipment, characterized by having the following features. [program] A program for causing a computer to function as one of the electronic devices described in any one of sections 1 through 9. [Explanation of symbols]

[0128] 100: Camera, 211: Imaging unit, 50: System control unit

Claims

1. An acquisition means for acquiring an image obtained by imaging through one or more optical systems, the image having a region for each optical system, A type determination means for determining the type of marker to be synthesized on the captured image, A determination means for determining the number of regions of the image in the captured image, 1) When the number of regions in the image is one, the marker is composited at a position corresponding to the type of marker in the captured image; 2) When the number of regions in the image is two or more, the marker is composited for each region of the image at a position based on optical system information relating to the optical system used for imaging and the type of marker; It has, The optical system information includes the design values ​​of the optical axis center coordinates of each of the multiple optical systems used for imaging, and information on the deviation from the design values. If the number of regions of the image is two or more, the synthesis means determines a reference position for each region of the image based on the design value and the displacement information of each of the multiple optical systems, and synthesizes the markers based on the reference position for each region of the image. An electronic device characterized by the following features.

2. If the type of the marker indicates a first marker, the synthesis means 1) synthesizes the first marker at the central position in the captured image if the number of regions in the image is one, and 2) if the number of regions in the image is two or more, it determines the central position for each region of the image based on the optical system information and synthesizes the first marker at the central position for each region of the image. The electronic device according to feature 1.

3. If the type of the marker indicates a second marker, the synthesis means 1) synthesizes the second marker extending in the left-right direction through the central position in the captured image if the number of regions in the image is one, and 2) if the number of regions in the image is two or more, it determines the central position for each region of the image based on the optical system information, and synthesizes the second marker extending in the left-right direction through the central position of each region of the image. The electronic device according to feature 1.

4. If the type of the marker indicates a third marker, the synthesis means 1) synthesizes the third marker extending vertically through the central position in the captured image if the number of regions in the image is one, and 2) if the number of regions in the image is two or more, it determines the central position for each region of the image based on the optical system information and synthesizes the third marker extending vertically through the central position of each region of the image. The electronic device according to feature 1.

5. If the type of the marker indicates a fourth marker, the synthesis means: 1) If the number of regions in the image is one, synthesizes the fourth marker at a user-defined position in the captured image; 2) If the number of regions in the image is two or more, synthesizes the fourth marker to the first image region at a user-defined position, and for each of the image regions other than the first image, synthesizes the fourth marker to a position corrected for each image region based on the optical system information from the user-defined position. The electronic device according to feature 1.

6. The determination means determines the number of image regions based on information about the type of lens unit including the one or more optical systems. The electronic device according to feature 1.

7. The aforementioned marker is used by the user as a guide when deciding on the composition of the photograph. The electronic device according to feature 1.

8. An acquisition step of acquiring an image obtained by imaging through one or more optical systems, the image having a region for each optical system, A type determination step for determining the type of marker to be synthesized on the captured image, A determination step of determining the number of regions of the image in the captured image, 1) If the number of regions in the image is one, the marker is composited at a position corresponding to the type of marker in the captured image; 2) If the number of regions in the image is two or more, the marker is composited for each region of the image at a position based on the optical system information relating to the optical system used for imaging and the type of marker; It has, The optical system information includes the design values ​​of the optical axis center coordinates of each of the multiple optical systems used for imaging, and information on the deviation from the design values. In the synthesis step, if the number of regions of the image is two or more, the reference position for each region of the image is determined based on the design value and the displacement information for each of the multiple optical systems, and the markers are synthesized based on the reference position for each region of the image. A method for controlling electronic equipment characterized by the following features.

9. A program for causing a computer to function as one of the electronic devices described in any one of claims 1 to 7.

Images