Imaging device and its control method

Abstract

To seamlessly switch a field angle from wide angle to telephoto, without interrupting imaging, when performing the imaging in parallel in a plurality of imaging modes.SOLUTION: An imaging apparatus having a plurality of lenses having field angles different from each other, includes: reception means for receiving a switching operation for switching the lens used for imaging, when imaging an image in parallel in a first imaging mode and in a second imaging mode; and control means for performing control so as to switch the lens used for imaging according to the reception of the switching operation of the lens used for imaging. The control means performs the control so as to switch the lens used in the first imaging mode and in the second imaging mode, so that a different lens is used in the first imaging mode and in the second imaging mode.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an imaging device and a control method for an imaging device.

Background Art

[0002] Conventionally, a shooting method of simultaneously recording a subject using both an imaging unit for still image shooting and an imaging unit for moving image shooting provided in an imaging device has been well known. In recent years, imaging devices such as smartphones have mounted a plurality of imaging units with different focal lengths and have realized recording a subject simultaneously with each imaging unit. Also, when switching the angle of view, a smartphone uses the imaging unit with the focal length to be switched. Patent Document 1 discloses a technique of automatically moving the zoom position of an imaging device for moving image shooting in conjunction with the zoom operation of an imaging device for still image shooting.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, each imaging device can be switched to an arbitrary angle of view by performing optical zoom by lens driving. On the other hand, a smartphone switches the angle of view using a plurality of imaging units (lenses) with different focal lengths. For example, when imaging in parallel in two imaging modes of still images and moving images with a smartphone, in order to switch the lens for moving images to the lens in use for still images, the use of the imaging unit for still images needs to be stopped once, and then the angle of view for moving image shooting is switched. Therefore, it is difficult to perform seamless angle of view adjustment from wide angle to telephoto without interrupting imaging.

[0005] Therefore, the present invention aims to seamlessly switch the field of view from wide-angle to telephoto without interrupting imaging when imaging is performed in parallel using multiple imaging modes. [Means for solving the problem]

[0006] To achieve the above objective, the imaging apparatus according to the present invention is An imaging device having multiple lenses with different angles of view, A receiving means for receiving a switching operation to switch the lens used for imaging when images are being captured in parallel using the first imaging mode and the second imaging mode, It includes a control means that controls the switching of the lens used for imaging in response to receiving a switching operation for the lens used for imaging, The control means is Control to switch the lenses used in the first imaging mode and the second imaging mode so that different lenses are used in the first imaging mode and the second imaging mode. death, When the first lens is used in the first imaging mode and the second lens is used in the second imaging mode, if a switching operation is received to switch the lens used in the first imaging mode to the second lens, the system controls the system to switch the lens used in the second imaging mode to a third lens that is different from both the first and second lenses, and to switch the lens used in the first imaging mode back to the second lens. do, It is characterized by the following: [Effects of the Invention]

[0007] According to the present invention, when capturing images in parallel using multiple imaging modes, the field of view can be seamlessly switched from wide-angle to telephoto without interrupting the imaging process. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram of the smartphone's exterior. [Figure 2] This is a block diagram showing the configuration of a smartphone. [Figure 3] This diagram explains the operation of switching the field of view. [Figure 4] This is a flowchart illustrating the angle of view switching process according to the first embodiment. [Figure 5]This is a flowchart illustrating the angle of view switching process according to the second embodiment. [Figure 6] This diagram illustrates the LV display switching process according to the third embodiment. [Figure 7] This is a flowchart illustrating the LV display switching process according to the third embodiment. [Figure 8] This diagram illustrates the zoom-out process in still image LV display. [Figure 9A] This is a flowchart illustrating the zoom-in and zoom-out process. [Figure 9B] This flowchart illustrates the details of the zoom-in process. [Figure 9C] This flowchart illustrates the details of the zoom-out process. [Figure 10] This figure illustrates the video LV superimposition display process in the third embodiment. [Figure 11] This flowchart illustrates the process of overlaying video LV (Live View) images. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Figures 1(a) and 1(b) are external views of a smartphone 100, which is an example of an imaging device (electronic device) to which the present invention can be applied. Figure 1(a) is a front view of the smartphone 100, and Figure 1(b) is a rear view of the smartphone 100.

[0010] The display 105 is a display unit provided on the front surface of the smartphone 100 that displays images and various types of information. The touch panel 106a is a touch operation member configured integrally with the display 105, and the smartphone 100 can detect a touch operation on the display surface (operation surface) of the display 105 using the touch panel 106a. The smartphone 100 can perform a live view display (LV display) of an image captured by the out-camera 114 or the in-camera 115 on the display 105. The LV display is a display of an image that represents the subject in substantially real time, and the image displayed in the LV display is called an LV image. The out-camera 114 includes a telephoto lens 114a, a standard lens 114b, a wide-angle lens 114c, etc., which will be described later.

[0011] The power button 106b is an operation button for switching the lighting / extinguishing of the display 105 or switching the ON / OFF of the power supply of the smartphone 100. For example, when the user presses the power button 106b for a time shorter than a predetermined time (e.g., 3 seconds) (short presses the power button 106b), the lighting / extinguishing of the display 105 is switched. When the user continues to press the power button 106b for a predetermined time (long presses the power button 106b), the ON / OFF of the power supply of the smartphone 100 is switched.

[0012] The volume plus button 106c and the volume minus button 106d are volume buttons for adjusting the volume of the audio signal output from the audio output terminal 112a or the volume of the audio output from the speaker 112b. When the volume plus button 106c is pressed, the volume increases, and when the volume minus button 106d is pressed, the volume decreases. In the shooting standby state when using the camera, the volume plus button 106c and the volume minus button 106d function as a shutter button for instructing shooting.

[0013] The user can arbitrarily set so that a specific function is executed when the power button 106b and the volume minus button 106d are pressed simultaneously, or when the volume minus button 106d is quickly pressed several times.

[0014] The home button 106e is an operation button for displaying the home screen on the display 105. When the user has launched and used various applications on the smartphone 100, by pressing the home button 106e, various applications can be temporarily closed and the home screen can be displayed on the display 105. In this embodiment, it is assumed that the home button 106e is a physically pressable button, but it may also be a touch-operable touch button (display item) displayed on the display 105.

[0015] The audio output terminal 112a is a headphone jack and is a terminal for outputting an audio signal to headphones, an external speaker, etc. The speaker 112b is a built-in speaker for outputting sound. For example, when headphones, an external speaker, etc. are connected to the audio output terminal 112a, an audio signal is output from the audio output terminal 112a. When headphones, an external speaker, etc. are not connected to the audio output terminal 112a, sound is output from the speaker 112b.

[0016] FIG. 2 is a block diagram showing a configuration example of the smartphone 100. A CPU 101, a memory 102, a non-volatile memory 103, an out-camera image processing unit 104, a display 105, an operation unit 106, a storage medium I / F 107, an external I / F 109, and a communication I / F 110 are connected to an internal bus 150. Also, an audio output unit 112, an attitude detection unit 113, an out-camera 114, an in-camera 115, and an in-camera image processing unit 116 are connected to the internal bus 150. Each unit connected to the internal bus 150 can exchange data with each other via the internal bus 150.

[0017] The CPU 101 is a control unit that controls the entire smartphone 100 and consists of at least one processor or circuit. The memory 102 consists of, for example, RAM (volatile memory using semiconductor elements). The CPU 101 controls each part of the smartphone 100 by using the memory 102 as work memory, according to a program stored in, for example, the non-volatile memory 103. The non-volatile memory 103 stores image data, audio data, other data, and various programs for the operation of the CPU 101. The non-volatile memory 103 is composed of, for example, flash memory or ROM.

[0018] The rear camera 114 is a camera located on the back of the smartphone 100 (the side opposite to the side on which the display 105 is located). In the example shown in Figure 2, the rear camera 114 includes an imaging unit 114s, an imaging unit 114t, an imaging unit 114u, a telephoto lens 114a, a standard lens 114b, and a wide-angle lens 114c. In this embodiment, imaging is performed using the imaging unit 114s when using the telephoto lens 114a, the imaging unit 114t when using the standard lens 114b, and the imaging unit 114u when using the wide-angle lens 114c. However, the configuration is not limited to this, and for example, the rear camera 114 may have an imaging unit for still images and an imaging unit for video, and each imaging unit may be configured to switch between a telephoto lens, a standard lens, and a wide-angle lens for shooting.

[0019] The imaging units 114s, 114t, and 114u are image sensors such as CCD image sensors and CMOS image sensors, and perform imaging using a telephoto lens 114a, a standard lens 114b, and a wide-angle lens 114c, respectively. The rear camera 114 can perform three types of imaging simultaneously (in parallel or time-division) using the telephoto lens 114a, the standard lens 114b, and the wide-angle lens 114c. Of course, the rear camera 114 can also perform only two of the three types of imaging simultaneously, or only one type of imaging.

[0020] The focal length of the telephoto lens 114a is longer than that of the standard lens 114b, so images taken with the telephoto lens 114a will be more telephoto than images taken with the standard lens 114b. The focal length of the wide-angle lens 114c is shorter than that of the standard lens 114b, so images taken with the wide-angle lens 114c will be wider-angle than images taken with the standard lens 114b. In other words, the focal length decreases in the order of telephoto lens 114a, standard lens 114b, and wide-angle lens 114c, and the angle of view (imaging range) widens accordingly. In this embodiment, the zoom magnification (focal length) of the telephoto lens 114a is assumed to be a predetermined fixed value, but it may be changeable by the user.

[0021] In this embodiment, the smartphone 100 can capture images in multiple imaging modes, and by switching between the imaging units 114s, 114t, and 114u in each imaging mode, the lens can be switched and the field of view can be changed for each imaging mode. The multiple imaging modes include, for example, a still image mode for capturing still images and a video mode for capturing videos. When capturing images in parallel in still image mode and video mode, the smartphone 100 uses different imaging units, i.e., different lenses, for each imaging mode.

[0022] Smartphone 100 switches the lens used (telephoto lens 114a, standard lens 114b, wide-angle lens 114c) by switching between image units 114s, 114t, and 114u in each imaging mode. In the following description, "switching lenses" means switching the lens by switching the image unit.

[0023] The front camera 115 is a camera located on the front of the smartphone 100 (the same side as the display 105).

[0024] Images captured by the rear camera 114 and images captured by the front camera 115 can both be displayed as Live View (LV) on the display 105. The user can use the touch panel 106a to select which camera's images to display as Live View on the display 105. For example, the user (photographer) can select the rear camera 114 when photographing the scene in front of them and the front camera 115 when taking a selfie.

[0025] Furthermore, when displaying images captured by the rear camera 114 on the display 105 via Live View (LV), the user can operate the touch panel 106a to select which lens was used to capture the image and display it on the display 105 via Live View. If the user selects the telephoto lens 114a, a more magnified image can be displayed on the display 105 via Live View than when the standard lens 114b is selected. If the user selects the standard lens 114b, a wider-angle image can be displayed on the display 105 via Live View than when the telephoto lens 114a is selected, and a more magnified image can be displayed on the display 105 via Live View than when the wide-angle lens 114c is selected. If the user selects the wide-angle lens 114c, a wider-angle image can be displayed on the display 105 via Live View than when the telephoto lens 114a or the standard lens 114b is selected.

[0026] The rear camera image processing unit 104 performs various image processing and subject recognition processing using images captured by the rear camera 114, based on the control of the CPU 101. The rear camera image processing unit 104 includes a telephoto image processing unit 104a, a standard image processing unit 104b, and a wide-angle image processing unit 104c. The telephoto image processing unit 104a performs various image processing using images captured by the telephoto lens 114a. The standard image processing unit 104b performs various image processing using images captured by the standard lens 114b. The wide-angle image processing unit 104c performs various image processing using images captured by the wide-angle lens 114c. These three image processing units can operate simultaneously (in parallel or in time-division).

[0027] In this embodiment, three image processing units are used, each corresponding to one of the three lenses. However, the number of lenses corresponding to one image processing unit is not particularly limited. One image processing unit may be used for two lenses, or one image processing unit may be used for three lenses. In other words, the rear camera image processing unit 104 does not contain multiple image processing units. This is also acceptable. In that case, the rear camera image processing unit 104 may simultaneously (in parallel or time-division) execute multiple processes using multiple images.

[0028] The in-camera image processing unit 116 performs various image processing using the image captured by the in-camera 115, based on the control of the CPU 101.

[0029] Each image processing unit can perform various image processing operations using images stored in the non-volatile memory 103 or storage medium 108, images acquired via the external I / F 109, or images acquired via the communication I / F 110. Image processing operations performed by each image processing unit include A / D conversion, D / A conversion, image data encoding, compression, decoding, resizing, noise reduction, and color conversion. Each image processing unit may be composed of a dedicated circuit block for specific image processing. Furthermore, depending on the type of image processing, the CPU 101 may perform image processing according to a program without using each image processing unit.

[0030] The display 105 displays images and GUI (Graphical User Interface) screens based on the control of the CPU 101. The CPU 101 generates display control signals according to the program and controls various parts of the smartphone 100 to generate image signals for display on the display 105 and output them to the display 105. The display 105 displays images based on the output image signals. The smartphone 100 itself only has an interface for outputting image signals for display on the display 105, and the display 105 may be an external monitor (such as a television).

[0031] The operation unit 106 is an input device (reception unit) for receiving user operations. The operation unit 106 includes text information input devices such as keyboards, pointing devices such as mice and touch panels, buttons, dials, joysticks, touch sensors, and touchpads. The operation unit 106 also includes the aforementioned operating components such as the touch panel 106a, power button 106b, volume up button 106c, volume down button 106d, and home button 106e.

[0032] The storage medium interface 107 allows for the insertion and removal of storage media 108, such as memory cards, CDs, and DVDs. Based on the control of the CPU 101, it reads data from the inserted storage media 108 and writes data to the storage media 108. The storage media 108 may also be the built-in storage within the smartphone 100. The external interface 109 is an interface for connecting to external devices via wired or wireless cables and for inputting and outputting image and audio signals. The communication interface 110 is an interface for communicating with external devices and the internet 111 to send and receive various data such as files and commands.

[0033] The audio output unit 112 outputs audio from video data and music data, as well as operation sounds, ringtones, and various notification sounds. The audio output unit 112 includes the audio output terminal 112a and speaker 112b described above. The audio output unit 112 may also output audio via wireless communication or other means.

[0034] The attitude detection unit 113 detects the attitude (tilt) of the smartphone 100 relative to the direction of gravity, and the attitude (tilt) of the smartphone 100 relative to the yaw, roll, and pitch axes. Based on the attitude detected by the attitude detection unit 113, it is possible to determine whether the smartphone 100 is held horizontally, vertically, pointed upwards, pointed downwards, or in an oblique position. The attitude detection unit 113 can use any of several sensors, such as an accelerometer, gyroscope, geomagnetic sensor, compass sensor, or altitude sensor. One can be used. Alternatively, a combination of two or more sensors can be used as the attitude detection unit 113.

[0035] As described above, the operation unit 106 includes a touch panel 106a. The touch panel 106a is an input device that is superimposed on the display 105 to form a planar structure and outputs coordinate information corresponding to the position of contact. The CPU 101 can detect the following operations or states on the touch panel 106a. - A finger or pen that was not previously touching the touch panel 106a now touches the touch panel 106a, i.e., the start of a touch (hereinafter referred to as Touch-Down). • The state in which a finger or pen is touching the touch panel 106a (hereinafter referred to as Touch-On). • The finger or pen is moving while touching the touch panel 106a (hereinafter referred to as Touch-Move). - The finger or pen that was touching the touch panel 106a has been removed (released), i.e., the touch has ended (hereinafter referred to as Touch-Up). - When nothing is being touched on the touch panel 106a (hereinafter referred to as Touch-Off)

[0036] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. If a touch-move is detected, a touch-on is also detected simultaneously. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. A touch-off is detected when all fingers or pens that were touching the screen have been detected as having touched up.

[0037] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 106a, are notified to the CPU 101 via the internal bus 150. Based on the notified information, the CPU 101 determines what kind of operation (touch operation) was performed on the touch panel 106a. For touch moves, the direction of movement of the finger or pen moving on the touch panel 106a can also be determined for each vertical and horizontal component on the touch panel 106a based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation was performed. An operation in which a finger is touched on the touch panel 106a and then quickly moved a certain distance and released is called a flick. In other words, a flick operation is an operation in which the finger is quickly swiped across the touch panel 106a. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected, and a touch-up is then detected, it can be determined that a flick was performed (it can be determined that a flick followed a slide operation). Furthermore, touching multiple points (for example, two points) simultaneously (multitouch) to bring them closer together is called pinch-in, and touching them further apart is called pinch-out. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch). The touch panel 106a may use any of the various types of touch panels, such as resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, and optical sensor types. There are methods that detect a touch when there is contact with the touch panel, and methods that detect a touch when a finger or pen approaches the touch panel, and either method is acceptable.

[0038] The following embodiments relate to controlling the switching of the field of view when a smartphone 100 takes still images and takes video images in parallel.

[0039] (First embodiment) In the first embodiment, the rear camera 114 is equipped with three lenses: a telephoto lens 114a, a standard lens 114b, and a wide-angle lens 114c, each with a different field of view. The rear camera 114 can capture images in parallel in multiple imaging modes using the telephoto lens 114a, the standard lens 114b, and the wide-angle lens 114c. For example, the rear camera 114 can capture images in video mode using the standard lens 114b and in still image mode using the telephoto lens 114a.

[0040] The rear camera 114 captures images by switching the lens used for video mode and still image mode based on user input. The video mode and still image mode share three lenses (image capture units), and when capturing images in parallel in each mode, they use different lenses for each mode.

[0041] Referring to Figures 3 and 4, the process of switching the field of view of captured images in video mode and still image mode will be explained. Figure 3 is a diagram illustrating the user's operation to switch the field of view. Figure 4 is a flowchart illustrating the field of view switching process by the rear camera 114 and the rear camera image processing unit 104. The field of view switching process in Figure 4 is realized when the CPU 101 of the smartphone 100 loads the program recorded in the non-volatile memory 103 into memory 102 and executes it.

[0042] The field of view switching process is initiated, for example, when a camera application is launched on the smartphone 100. The field of view switching process is not limited to the camera application; it may also be initiated when a means for capturing still images and videos in parallel is launched.

[0043] In S401, the CPU 101 starts imaging with the rear camera 114 using three lenses: a telephoto lens 114a, a standard lens 114b, and a wide-angle lens 114c. The images captured using each lens are then processed by the telephoto image processing unit 104a, the standard image processing unit 104b, and the wide-angle image processing unit 104c of the rear camera image processing unit 104.

[0044] In S402, the CPU 101 displays captured images in video mode and still image mode as live view (LV). For example, the CPU 101 displays images captured using the standard lens 114b in video mode and images captured using the telephoto lens 114a in still image mode as live view (LV) on the display 105.

[0045] Figure 3(a) shows an example of a screen displaying an image captured using the standard lens 114b in video mode and an image captured using the telephoto lens 114a in still image mode. The display 105 is divided into a main image area 301 that displays the image captured in video mode using the standard lens 114b, and a sub-image area 305 that displays the image captured in still image mode using the telephoto lens 114a. In Figure 3(a), the main image area 301 is the area displayed on almost the entire display 105, and the sub-image area 305 is the area superimposed on the upper right of the main image area 301.

[0046] The main image area 301 displays the index 302, indicator 303, and video recording button 304 superimposed on the LV image in video mode. The index 302 indicates the lenses that can be switched in video mode. The indicator 303 points to the lens currently in use in video mode, as indicated by the index 302. The example in Figure 3(a) shows that the standard lens 114b is being used in video mode. The indicator 303 is also an operating element (item) for switching the lens being used. The user can switch the lens being used by touching the indicator 303 and moving it on the index 302 using touch move. The item consisting of the index 302 and indicator 303 is an example of the first item. Video recording button 3 04 is an operating component (item) for instructing the start and stop of video recording.

[0047] In the sub-image area 305, the still image capture button 306 and the lens switching member 307 are displayed superimposed on the LV image in still image mode. The still image capture button 306 is an operating member (item) for capturing still images. The still image capture button 306 can be moved between the captured image in still image mode and the captured image in video mode by touch movement. In other words, the still image capture button 306 can be moved from the sub-image area 305 to the main image area 301.

[0048] The user can take a still image of the captured image displayed in LV on the main image area 301 by moving the still image capture button 306 to the main image area 301 and touching it. If the center of the still image capture button 306 is on the captured image in still image mode in the sub-image area 305, the CPU 101 takes a still image with the lens used in still image mode. If the center of the still image capture button 306 is on the captured image in video mode in the main image area 301, the CPU 101 takes a still image with the lens used in video mode. The still image capture button 306 is an example of the fourth item.

[0049] The lens switching member 307 indicates the lens used for the still image captured in the sub-image area 305. As described in S407, the lens switching member 307 is an operating member (item) for changing the lens used in still image mode.

[0050] In S403, the CPU 101 determines whether or not an operation to switch the video lens used in video mode has occurred. Based on the operation information detected by the touch panel 106a, the CPU 101 determines whether or not the position of the indicator 303 has changed. If the indicator 303 at the time the touch move is completed has selected a lens different from the lens currently in use, the CPU 101 determines that an operation to switch the video lens has occurred. If the indicator 303 at the time the touch move is completed has selected the same lens as the lens currently in use, the CPU 101 determines that no operation to switch the video lens has occurred. If an operation to switch the video lens has occurred, the process proceeds to S404; if no operation to switch the video lens has occurred, the process proceeds to S406.

[0051] In S404, CPU101 switches the video lens from the currently used lens to the lens selected by the user in S403. When CPU101 switches the video lens, it displays the image captured by the switched video lens in the main image area 301 as a live view (LV).

[0052] If the video lens is switched, for example, from the currently used standard lens 114b to the telephoto lens 114a used in still image mode, the CPU 101 pauses the captured image displayed in LV in the sub-image area 305 in still image mode. After pausing the LV display in still image mode, the CPU 101 switches the video lens to the telephoto lens 114a.

[0053] In S405, the CPU 101 performs the process of switching the still image lens. If the video lens is to be switched to the lens currently being used as a still image lens, the CPU 101 switches the still image lens to a lens that is not being used in either video or still image mode. For example, the CPU 101 switches the still image lens to a wide-angle lens 114c, which is different from both the standard lens 114b used in video mode and the telephoto lens 114a used in still image mode. The CPU 101 displays the image captured in still image mode using the switched lens in LV in the sub-image area 305.

[0054] In S404, if the video lens is switched to a lens not being used as a still image lens, in S405, CPU101 does not change the still image lens and sub-image area 3 The LV display for 05 will not be changed.

[0055] In the explanation above, the video lens was switched in S404, and then the still image lens was switched in S405. However, when switching from a video lens to a lens being used as a still image lens, the captured image displayed in LV in sub-image area 305 in still image mode must be paused. Therefore, when switching to a lens being used in still image mode, it may be better to perform the still image lens switching in S405 first, and then the video lens switching process in S404. In this case, the lenses for still image mode and video mode can be switched without pausing the captured image displayed in LV in sub-image area 305 in still image mode.

[0056] In S406, the CPU 101 determines whether or not an operation to switch the still image lens was performed. Based on the operation information detected by the touch panel 106a, the CPU 101 determines whether or not the lens switching member 307 was touched. If the lens switching member 307 is touched, the CPU 101 determines that an operation to switch the still image lens was performed. If the lens switching member 307 is not touched, the CPU 101 determines that no operation to switch the still image lens was performed. If an operation to switch the still image lens was performed, the process proceeds to S407; if no operation to switch the still image lens was performed, the process proceeds to S410.

[0057] In S407, CPU 101 displays a list of lenses that can be switched to as still image lenses. CPU 101 controls the display of the candidates so that it is also possible to switch back to the currently used lens. Figure 3(b) shows an example screen when an operation to switch the still image lens is received while recording a video displayed in the main image area 301.

[0058] When the lens switching member 307 is touched on the screen shown in Figure 3(a), the CPU 101 superimposes a pop-up window 308, including the index 309 and indicator 310, onto the sub-image area 305, as shown in Figure 3(b). The index 309 indicates the lens that can be switched in still image mode. The indicator 310 points to the lens currently in use in still image mode on the index 309. The example in Figure 3(b) shows that the standard lens 114b is being used in video mode and the telephoto lens 114a is being used in still image mode. The indicator 310 is an operating member (item) for the user to switch the lens being used by touching and moving on the index 309. The item consisting of the index 309 and indicator 310 is an example of a second item.

[0059] The sub-image area 305 is displayed larger than when the pop-up window 308 is not displayed, as shown in Figure 3(b), while the pop-up window 308 is displayed for switching lenses. This makes it easier for the user to view the still image LV when switching lenses.

[0060] While recording video displayed in the main image area 301, the CPU 101 controls the system so that the lens being used in video mode cannot be selected in still image mode, in order to prevent interruption of video recording. In the example in Figure 3(b), the standard lens 114b being used in video mode is not displayed on the indicator 309. In other words, the CPU 101 controls the system so that it does not accept switching to the lens being used in video mode while recording video, effectively making it impossible to switch lenses used in video mode. In contrast, when not recording video, the CPU 101 allows the lens being used in video mode to be selected, and the lens being used in video mode is also displayed on the indicator 309.

[0061] In S408, CPU101 selects the still image lens from the one selected by the user in S407. Switch to the lens. When the still image lens is switched, the CPU 101 displays the image captured by the new still image lens in LV in the sub-image area 305.

[0062] If the still image lens is switched, for example, from the currently used telephoto lens 114a to the standard lens 114b used in video mode, the CPU 101 pauses the captured image displayed in the main image area 301 in video mode. After pausing the LV display in video mode, the CPU 101 switches the still image lens back to the standard lens 114b.

[0063] In S409, the CPU 101 performs the process of switching the video lens. If the still image lens is to be switched to the lens currently being used as the video lens, the CPU 101 switches the video lens to a lens that is not being used in either video mode or still image mode. For example, the CPU 101 switches the video lens to a wide-angle lens 114c that is different from both the telephoto lens 114a used in still image mode and the standard lens 114b used in video mode. The CPU 101 displays the video image captured by the switched lens in LV in the main image area 301.

[0064] In S408, if the still image lens is switched to a lens not being used as a video lens, in S409, the CPU 101 does not change the video lens, nor does it change the LV display in the main image area 301. Also, if video is being recorded, the still image lens will not be switched to a lens being used as a video lens, so the CPU 101 does not change the video lens or the LV display in the main image area 301.

[0065] Furthermore, in the explanation above, the lens for still image mode was switched in S408, and then the lens for video mode was switched in S409. However, when switching a still image lens to a lens currently in video mode, it is necessary to pause the video mode LV display. Therefore, when switching a still image lens to a lens currently in video mode, it may be better to execute the process of switching the video lens in S409 first, and then the process of switching the still image lens in S407. In this case, the lens can be switched without pausing the video mode LV display.

[0066] In S410, the CPU 101 determines whether or not a video recording operation has occurred. A video recording operation is an operation to start or stop video recording. Based on the operation information detected by the touch panel 106a, the CPU 101 determines whether or not the video recording button 304 has been touched. If the video recording button 304 has been touched, the CPU 101 determines that a video recording operation has occurred. If the video recording button 304 has not been touched, the CPU 101 determines that no video recording operation has occurred. If a video recording operation has occurred, the process proceeds to S411; if no video recording operation has occurred, the process proceeds to S412.

[0067] In S411, the CPU 101 starts or stops video recording. If video recording is not currently in progress, the CPU 101 starts recording the video captured by the video lens and displayed in the main image area 301. If video recording is currently in progress, the CPU 101 stops the video recording.

[0068] When the standard lens 114b is used as the lens for video recording, when video recording starts, the CPU 101 processes and encodes the video signal output from the standard lens 114b using the standard image processing unit 104b. The CPU 101 stores the image content processed by the standard image processing unit 104b in the memory 102. The CPU 101 generates a video file from the image content stored in the memory 102 and records it to the storage medium 108 via the non-volatile memory 103 or the storage medium I / F 107.

[0069] When video recording is stopped, the CPU 101 stops the encoding process by the standard image processing unit 104b and stops the process of generating a video file from the video content stored in memory 102 and recording it to the non-volatile memory 103 or storage medium 108.

[0070] In S412, the CPU 101 determines whether a still image capture operation was performed. Based on the operation information detected by the touch panel 106a, the CPU 101 determines whether the still image capture button 306 was touched. If the still image capture button 306 was touched, the CPU 101 determines that a still image capture operation was performed. If the still image capture button 306 was not touched, the CPU 101 determines that no still image capture operation was performed. If a still image capture operation was performed, the process proceeds to S413; if no still image capture operation was performed, the process proceeds to S414.

[0071] In S413, the CPU 101 captures a still image and records the generated still image file to the non-volatile memory 103 or storage medium 108. When the still image capture button 306 is on the sub-image area 305, the CPU 101 generates a still image file from the image captured by the still image lens, and when the still image capture button 306 is on the main image area 301, the CPU 101 generates a still image file from the image captured by the video lens.

[0072] For example, if a telephoto lens 114a is used as a lens for still images and the still image capture button 306 is located on the sub-image area 305, the telephoto image processing unit 104a processes and encodes the video signal output from the telephoto lens 114a to generate a still image file. Similarly, if a standard lens 114b is used as a lens for video and the still image capture button 306 is located on the main image area 301, the standard image processing unit 104b processes and encodes the video signal output from the standard lens 114b to generate a still image file.

[0073] The CPU 101 stores the image content processed by each part of the rear camera image processing unit 104 in the memory 102. The CPU 101 generates a still image file from the image content stored in the memory 102 and records it to the storage medium 108 via the non-volatile memory 103 or the storage medium interface.

[0074] In S414, the CPU 101 determines whether or not video recording is in progress. If video recording is in progress, the process returns to S403; otherwise, the process proceeds to S415. The CPU 101 can determine that video recording is in progress if any part of the out-camera image processing unit 104 is performing video encoding processing, and that video recording is not in progress if encoding processing is not being performed.

[0075] In S415, the CPU 101 determines whether the shooting standby state has ended. The shooting standby state is a state in which the LV image in video mode and the LV image in still image mode are displayed on the display, and the system is waiting to start recording video or taking a still image. If the shooting standby state has ended, the angle of view switching process shown in Figure 4 ends; if the shooting standby state has not ended, the process returns to S403. The CPU 101 can determine that the shooting standby state has ended, for example, when the user closes the camera application.

[0076] Note that the indicator 302 and indicator 303 of the main image area 301 described in S403, and the indicator 309 and indicator 310 of the sub-image area 305 described in S407, may be implemented using a common indicator. In the example shown in Figure 3(c), the indicator 312, the indicator 313 for video, and the indicator 311 for still images are displayed superimposed on the LV image in video mode in the main image area 301. The item consisting of the indicator 312, indicator 313, and indicator 311 is an example of a third item. The user can switch between the lenses used in video mode and still image mode using the third item. Cut.

[0077] The indicator 312 shows the lenses that can be switched in video mode on the upper side (towards the center of the main image area 301) and the lenses that can be switched in still image mode on the lower side (towards the periphery of the main image area 301). In the example in Figure 3(c), since video is being recorded using the standard lens 114b, the scale for selecting the standard lens 114b is not displayed on the lower side of the indicator 312. When video is not being recorded, the scale for selecting the standard lens 114b is displayed, and the user can select the standard lens 114b.

[0078] Indicator 313 is displayed above index 312 and, like indicator 303 in Figure 3(a), indicates the lens being used in video mode. The example in Figure 3(c) shows that the standard lens 114b is being used in video mode. Indicator 313 is also an operating component (item) for switching the lens used in video mode. The user can switch the lens used in video mode by touching indicator 313 in one place (single touch) and moving it along index 312 with a touch move.

[0079] When indicator 313 is touched and another area is touched at the same time (multi-touch with two or more locations), indicator 311, which indicates the lens being used in still image mode, is displayed below index 312, as shown in Figure 3(c). The example in Figure 3(c) shows that the telephoto lens 114a is being used in still image mode. Indicator 311 is also an operating element (item) for switching the lens used in still image mode. The user can switch the lens used in still image mode by moving the touch-move gesture on index 312 while multi-touching indicator 311.

[0080] The still image lens scales of indicator 311 and index 312 may be displayed while multi-touch is detected on indicator 313 or index 312, or they may be displayed at all times.

[0081] In the first embodiment described above, when switching from a video lens to a lens used in still image mode, the CPU 101 switches the still image lens to a lens not used in either video or still image mode. Conversely, when switching from a still image lens to a lens used in video mode, the CPU 101 switches the video lens to a lens not used in either video or still image mode. By controlling the switching of the field of view in this way, the user can seamlessly switch the field of view from wide-angle to telephoto without interrupting image capture. Furthermore, when capturing images in parallel in video and still image modes, the effort required to switch the field of view for each imaging mode can be reduced.

[0082] (Second embodiment) In the first embodiment, except when recording video in video mode, the user can switch between a video lens and a lens used in still image mode, or vice versa. In contrast, in the second embodiment, a lens used in a preferred imaging mode (hereinafter referred to as the preferred imaging mode) is controlled not to be used in other imaging modes (hereinafter referred to as non-preferred imaging modes). The preferred imaging mode can be set by the user.

[0083] In the second embodiment, detailed explanations of aspects similar to those in the first embodiment will be omitted, and the differences and processes from the first embodiment will be explained in detail. Figure 5 is a flowchart illustrating the field of view switching process by the rear camera 114 and the rear camera image processing unit 104. The field of view switching process in Figure 5 is realized by the CPU 101 of the smartphone 100 loading the program recorded in the non-volatile memory 103 into the memory 102 and executing it.

[0084] The processing in S501 and S502 is the same as in S401 and S402, so the explanation is omitted. In S503, the CPU 101 sets the imaging mode selected by the user as the priority imaging mode. The priority imaging mode is not limited to what the user sets; for example, it may be set to video mode as the initial setting. Images captured in priority imaging mode are displayed as LV in the main image area 301, and images captured in non-priority imaging mode are displayed as LV in the sub-image area 305.

[0085] The operation to set the priority imaging mode is, for example, to touch and hold the image of the target still image mode in the sub-image area 305, then touch and move the finger to the main image area 301 and release it. For example, if the image of the still image mode displayed in the sub-image area 305 is touched and moved to the main image area 301, the CPU 101 sets the still image mode to the priority imaging mode and displays the image of the still image mode in LV on the main image area 301.

[0086] Furthermore, the operation to set the priority imaging mode may also be the operation to change the orientation of the smartphone 100's display 105. The orientation of the display 105 can be determined based on the orientation of the smartphone 100 detected by the orientation detection unit 113. For example, the CPU 101 sets the video mode as the priority imaging mode when the orientation of the display 105 becomes vertical, and sets the still image mode as the priority imaging mode when it becomes horizontal.

[0087] When the priority imaging mode is changed, the CPU 101 displays the image captured in the priority imaging mode in the main image area 301 as an LV (Live View) and the image captured in the non-priority imaging mode in the sub-image area 305. Furthermore, when the priority imaging mode is changed, the CPU 101 may also use the lens that was used in the priority imaging mode before the change in the priority imaging mode after the change. In this case, the CPU 101 switches the lens used in the non-priority imaging mode after the change to a lens that is not used in the priority imaging mode after the change.

[0088] In the following process, the priority imaging mode will be described as video mode, and the non-priority imaging mode as still image mode. The process from S504 to S507 is the same as the process from S403 to S406 in Figure 4.

[0089] In S504, the CPU 101 determines whether or not there has been an operation to switch the lens used in the video mode set as the priority imaging mode (priority lens). If there has been an operation to switch the priority lens, in S505, the CPU 101 switches the priority (video) lens to the lens selected by the user in S504. If a lens that was being used as a non-priority (still image) lens is switched as a priority (video) lens in S504, in S506, the CPU 101 executes the switching process for the non-priority (still image) lens. If the lens selected as the priority (video) lens in S504 is not a lens that was being used as a non-priority (still image) lens, the process in S506 is not executed. In other words, the non-priority (still image) lens is maintained without being switched. Note that for the processes in S505 and S506, S506 may be executed first, followed by S505. In S507, CPU101 determines whether or not there was an operation to switch to a non-priority (still image) lens.

[0090] In S508, the CPU 101 displays candidate lenses that can be switched to as non-priority (still image) lenses. In the first embodiment, when recording video in video mode, the CPU 101 does not display the standard lens 114b being used in video mode as a candidate for still image lenses. In contrast, in the second embodiment, the CPU 101 does not display the standard lens 114b being used in priority video mode as a candidate for non-priority still image lenses, regardless of whether video recording is in progress or not. Also, when the priority imaging mode is still image mode, the priority Lenses currently in use in still image mode will not be displayed as candidates for non-priority video lenses. However, in any imaging mode, the system is controlled to allow switching to the currently used lens.

[0091] The processing in S509 is the same as the processing in S408 in Figure 4. In S509, the CPU 101 switches the non-priority still image lens to the lens selected by the user in S508. Since the lens being used in the video mode, which is the priority imaging mode, is not displayed as a candidate for the non-priority (still image) lens in S508, the process of switching the priority (video) lens after the processing in S509 is not executed as in the first embodiment.

[0092] The processing in S510 to S515 is the same as the processing in S410 to S415 in Figure 4. In S510, the CPU 101 determines whether or not a video recording operation has occurred. In S511, the CPU 101 starts or stops video recording. In S512 and S513, the CPU 101 determines whether or not a still image capture operation has occurred, and if a still image capture operation has occurred, it captures a still image.

[0093] In S514, the CPU 101 determines whether or not video recording is in progress. If video recording is in progress, the process returns to S503; otherwise, the process proceeds to S515. In S515, the CPU 101 determines whether or not the shooting standby state has ended. If the shooting standby state has ended, the angle of view switching process shown in Figure 5 is terminated; otherwise, the process returns to S503.

[0094] In the second embodiment described above, switching a non-priority lens to a lens being used in priority imaging mode is restricted. On the other hand, switching a priority lens to a lens being used in non-priority imaging mode is not restricted. When switching a priority lens to a lens being used in non-priority imaging mode, the CPU 101 switches the non-priority lens to a lens that is not being used in either priority or non-priority imaging mode. By switching the field of view via an unused lens in this way, the user can seamlessly switch the field of view from wide-angle to telephoto without interrupting the imaging process being performed in parallel. Furthermore, when imaging in parallel in video mode and still image mode, the effort required for the user to switch the field of view of the imaging mode they want to prioritize can be reduced.

[0095] Although the example described shows the image captured in priority imaging mode being displayed as LV in the main image area 301, the CPU 101 may not change the area where LV is displayed even if the priority imaging mode is changed, and may instead set the imaging mode of the sub-image area 305 as the priority imaging mode. In this case, the CPU 101 controls the lens used in the non-priority imaging mode of the main image area 301 so as not to be changed to the lens used in the priority imaging mode of the sub-image area 305. Furthermore, changing the priority imaging mode can be achieved, for example, by displaying an item indicating that it is in priority imaging mode in either the main image area 301 or the sub-image area 305, and the user moving the item between the respective areas using touch-move.

[0096] (Third embodiment) In the third embodiment, the user can switch the LV display for video to the LV display for still image shooting while continuing to record video and take still images. The rear camera 114 has three lenses with different angles of view: a telephoto lens 114a, a standard lens 114b, and a wide-angle lens 114c. The rear camera 114 can capture images in parallel using multiple imaging modes with the three lenses. In the third embodiment, the display 105 is not divided into areas as in the first and second embodiments, but rather the LV display for still image mode and the LV display for video mode are switched according to the user's operation.

[0097] Figures 6(a) to 6(e) illustrate the switching process for LV display in the third embodiment. The LV display screen 601 in Figure 6(a) is a screen on the display 105 where an image captured in video mode using the standard lens 114b is displayed in LV.

[0098] Icon 602 is an icon indicating the display status, showing that the LV display screen 601 is displaying a video image in LV mode. The display status indicates whether the display 105 is displaying a video image in LV mode or a still image in LV mode. Icon 603 is an icon indicating the video recording status, showing that the device is waiting for the video recording start operation. The LV display switching button 604 is an operating component (item) for switching between LV display in video mode and LV display in still image mode. The LV display switching button 604 is an example of the fifth item.

[0099] The icon 605 in Figure 6(b) is an icon indicating the video recording status, showing that the LV image in video mode displayed on the display 105 is being recorded. Video recording is started and stopped using an item (not shown) similar to the video recording button 304 described in Figure 3(a).

[0100] The LV display screen 611 in Figure 6(c) shows the image captured in still image mode using the telephoto lens 114a, displayed in LV on the display 105. Icon 612 is an icon indicating the display status, and indicates that the LV display screen 611 is displaying an image captured in still image mode in LV.

[0101] Image 621 in Figure 6(d) is a still image generated by a still image capture operation from the image of the LV display screen 611 in Figure 6(c), which was captured in still image mode using the telephoto lens 114a. Image 631 in Figure 6(e) is a still image with the same field of view as the video, generated by extracting one frame from a video captured in video mode using the standard lens 114b.

[0102] Figure 7 is a flowchart illustrating the LV display switching process according to the third embodiment. The LV display switching process in Figure 7 is realized when the CPU 101 of the smartphone 100 loads the program recorded in the non-volatile memory 103 into memory 102 and executes it.

[0103] In S701, the CPU 101 accepts user input. If the accepted user input is an operation to switch the LV display, processing proceeds to S702. The operation to switch the LV display is expected to be, for example, an operation to touch the LV display switching button 604 displayed on the display 105, but is not limited to this. The operation to switch the LV display may also be an operation to double-tap or touch-move on the LV image displayed on the display 105, or it may be an operation to press a physical button. The user can display the captured image in LV mode, including the subject or the desired location, in LV mode by, for example, double-tapping the subject on the LV image or drawing a circle around the area (location) to be captured with touch-move.

[0104] In S702, the CPU 101 determines the display status of the display 105. The display status is whether the image captured in video mode is displayed in LV (Live View) on the display 105, or whether the image captured in still image mode is displayed in LV (Live View). If the image captured in video mode is displayed in LV, the process proceeds to S703; if the image captured in still image mode is displayed in LV, the process proceeds to S707.

[0105] In S703, the CPU 101 determines whether the lens used for LV display in video mode is the telephoto lens 114a, the standard lens 114b, or the wide-angle lens 114c. If the lens used in video mode is the wide-angle lens 114c or the telephoto lens 114a, the process proceeds to S704. If the lens used in video mode is the standard lens 114b, the process proceeds to S7 Proceed to 05.

[0106] In the S704, the CPU 101 starts capturing images in still image mode using the standard lens 114b. The CPU 101 can then display the captured images taken in still image mode using the standard lens 114b on the display 105 (LV display) or capture them as still images.

[0107] In the S705, the CPU 101 starts capturing images in still image mode using the telephoto lens 114a. The CPU 101 can then display the captured images taken in still image mode using the telephoto lens 114a on the display 105 as live view (LV) images, or capture them as still images.

[0108] Note that S703 to S705 show examples of determining the still image lens based on whether the lens used in video mode is the wide-angle lens 114c, the telephoto lens 114a, or the standard lens 114b, but are not limited to these examples. The still image lens can be determined to be different from the lens used in video mode. For example, the lens used in still mode may be determined to be a more telephoto lens than the lens used in video mode.

[0109] In S706, CPU 101 changes the display on display 105 from LV display in video mode to LV display in still image mode. If video is being recorded in video mode, CPU 101 continues recording video while displaying the captured image in LV display in still image mode. CPU 101 changes the icon indicating the display status of display 105 to icon 612 indicating that it is in LV display mode in still image mode, and returns to S701.

[0110] In S707, the CPU 101 determines whether the lens used for LV display in still image mode is a telephoto lens 114a, a standard lens 114b, or a wide-angle lens 114c. If the lens used in still image mode is a wide-angle lens 114c, the process proceeds to S708. If the lens used in still image mode is a telephoto lens 114a or a standard lens 114b, the process proceeds to S709.

[0111] In the S708, the CPU 101 starts capturing images in video mode using the standard lens 114b. The CPU 101 can then display the captured images captured in video mode using the standard lens 114b on the display 105 as live video (LV) images, or record them as video.

[0112] In the S709, the CPU 101 starts capturing images in video mode using the wide-angle lens 114c. The CPU 101 can then display the captured images captured in video mode using the wide-angle lens 114c on the display 105 as live video (LV) images, or record them as video.

[0113] Note that S707-S709 show examples of determining the video lens based on whether the lens used in still image mode is the wide-angle lens 114c, the telephoto lens 114a, or the standard lens 114b, but are not limited to these examples. The video lens only needs to be different from the lens used in still image mode. For example, the lens used in video mode may be determined to be wider-angle than the lens used in still image mode.

[0114] In S710, CPU101 changes the display on display 105 from still image mode LV display to video mode LV display. CPU101 changes the icon indicating the display status of display 105 to icon 602 indicating video mode LV display, and returns to S701.

[0115] If the user operation received in S701 is a still image capture operation, the process proceeds to S711. In S711, the CPU 101 determines the display state of the display 105. If the display 105 is showing a video capture image in LV, the process proceeds to S712. If the display 105 is showing a still image capture image in LV, the process proceeds to S713.

[0116] In S712, the CPU 101 captures a still image with the lens being used in video mode. The CPU 101 extracts one frame from the video being captured with the lens being used in video mode, generates a still image with the same field of view as the video, as shown in image 631 in Figure 6(e), and returns to S701.

[0117] In S713, the CPU 101 captures a still image using the lens currently in use in still image mode. Since the lens used in still image mode is different from the lens used in video mode, the CPU 101 generates a still image with a different field of view than the video, as shown in image 621 in Figure 6(d), and returns to S701.

[0118] If the user operation received in S701 is a video recording start / stop operation, the process proceeds to S714. In S714, CPU101 determines the video recording status. If it is in standby mode, the process proceeds to S715; if it is recording, the process proceeds to S721.

[0119] In S715, the CPU 101 determines the display status of the display 105. If the display 105 is showing a video image in LV mode, the process proceeds to S716. If the display 105 is showing a still image in LV mode, the process proceeds to S717.

[0120] In S716, CPU101 starts recording video with the lens currently in use in video mode. CPU101 changes the icon indicating the video recording status to icon 605, which indicates that recording is in progress, and returns to S701.

[0121] In S717, the CPU 101 determines whether the lens used for LV display in still image mode is the telephoto lens 114a, the standard lens 114b, or the wide-angle lens 114c. If the lens used in still image mode is the wide-angle lens 114c, the process proceeds to S718. If the lens used in still image mode is the telephoto lens 114a or the standard lens 114b, the process proceeds to S719.

[0122] In S718, the CPU 101 starts imaging in video mode using the standard lens 114b and proceeds to S720. In S719, the CPU 101 starts imaging in video mode using the wide-angle lens 114c and proceeds to S720.

[0123] Note that S717-S719 show examples of determining the video lens depending on whether the lens used in still image mode is the wide-angle lens 114c, the telephoto lens 114a, or the standard lens 114b, but are not limited to these examples. The video lens only needs to be different from the lens used in still image mode. For example, the lens used in video mode may be determined to be wider-angle than the lens used in still image mode.

[0124] In the S720, the CPU 101 changes the display on the display 105 from the LV display in still image mode to the LV display in video mode and starts video recording. The CPU 101 changes the icon indicating the video recording status to icon 605, which indicates that video is being recorded, and returns to the S701.

[0125] If video recording was in progress at S714, at S721, CPU101 stops video recording, changes the icon indicating the video recording status to icon 603 indicating that it is waiting for the operation to start video recording, and returns to S701.

[0126] If the user operation received by S701 is an operation to terminate the LV (Live View) display, the LV display switching process shown in Figure 7 will be terminated.

[0127] Figure 8 illustrates the zoom-out process in still image LV display. When recording video using the wide-angle lens 114c, if the display on the display 105 is changed to the LV display in still image mode and zoomed out using a pinch-in gesture, the widest end of the standard lens 114b will be the most zoomed-out state. This is because the wide-angle lens 114c is used in video mode, and the lens used in still image mode cannot be switched to the wide-angle lens 114c. In the LV display screen 801 of Figure 8, when the user zooms out using a pinch-in gesture and reaches the widest end of the standard lens 114b, the CPU 101 displays a notification 802 on the display 105 stating, "You cannot zoom out any further."

[0128] Conversely, if, while recording video using the telephoto lens 114a, the display on the display 105 is changed to the LV display for still image mode and zoomed in using a pinch-out gesture, the telephoto end of the standard lens 114b will be the most zoomed-in state. This is because the telephoto lens 114a is being used in video mode, and the lens used in still image mode cannot be switched to the telephoto lens 114a. In this case, when the user zooms in using a pinch-out gesture and reaches the telephoto end of the standard lens 114b, the CPU 101 displays a notification on the display 105 stating, "You cannot zoom in any further."

[0129] Referring to Figures 9A to 9C, the zoom-in and zoom-out processing in still image LV display according to the third embodiment will be described. Figure 9A is a flowchart illustrating the zoom-in and zoom-out processing. Figure 9B is a flowchart illustrating the details of the zoom-in processing. Figure 9C is a flowchart illustrating the details of the zoom-out processing. The zoom-in and zoom-out processing in Figures 9A to 9C is realized by the CPU 101 of the smartphone 100 loading the program recorded in the non-volatile memory 103 into the memory 102 and executing it.

[0130] In S900, CPU 101 accepts user input. If the accepted user input is a pinch-out operation, CPU 101 proceeds to S910 and performs a zoom-in (enlargement) process by changing the zoom magnification of the image captured in still image mode. If the accepted user input is a pinch-in operation, CPU 101 proceeds to S930 and performs a zoom-out (reduction) process by changing the zoom magnification of the image captured in still image mode. If the accepted user input is an operation to end LV, the zoom-in / zoom-out process shown in Figure 9A is terminated.

[0131] Refer to Figure 9B to explain the details of the zoom-in process in S910. In S911, the CPU 101 determines whether the lens used for LV display in still image mode is the wide-angle lens 114c, the standard lens 114b, or the telephoto lens 114a. If the lens used in still image mode is the wide-angle lens 114c, the process proceeds to S912. If the lens used in still image mode is the standard lens 114b, the process proceeds to S920, and if the lens used in still image mode is the telephoto lens 114a, the process proceeds to S924.

[0132] In the S912, the CPU 101 obtains the zoom position (zoom magnification) based on the user's pinch-out operation. The zoom position due to the pinch-out operation is the same as the wide-angle lens 114c. If the zoom position is wider than the telephoto end, the process proceeds to S913. If the zoom position obtained by pinching out is telephoto beyond the telephoto end of wide-angle lens 114c and wider than the widephoto end of telephoto lens 114a, the process proceeds to S914. If the zoom position obtained by pinching out is telephoto beyond the widephoto end of telephoto lens 114a, the process proceeds to S916.

[0133] In S913, the CPU 101 performs zoom-in processing using the wide-angle lens 114c and returns to S900 in Figure 9A. The CPU 101 can perform zoom-in processing, for example, by changing the zoom position to the position acquired in S912 by digitally zooming the image captured by the wide-angle lens 114c.

[0134] In S914, the CPU 101 determines whether or not video recording is currently underway with the standard lens 114b. If video recording is underway with the standard lens 114b, the process proceeds to S913. If video recording is not underway with the standard lens 114b, the process proceeds to S915.

[0135] In S915, the CPU 101 changes the lens used for LV display in still image mode to the standard lens 114b, which has a narrower field of view. The CPU 101 performs zoom-in processing using the standard lens 114b and returns to S900 in Figure 9A. The CPU 101 can perform zoom-in processing, for example, by changing the zoom position to the position obtained in S912 through digital zoom of the image captured by the standard lens 114b.

[0136] In S916, the CPU 101 determines whether or not video recording is in progress with the telephoto lens 114a. If video recording is in progress with the telephoto lens 114a, the process proceeds to S915. If video recording is not in progress with the telephoto lens 114a, the process proceeds to S917. In S917, the CPU 101 changes the lens used for LV display in still image mode to the telephoto lens 114a, which has a narrower field of view. The CPU 101 performs a zoom-in process using the telephoto lens 114a and returns to S900 in Figure 9A. The CPU 101 can perform a zoom-in process, for example, by changing the zoom position to the position obtained in S912 through digital zoom of the image captured by the telephoto lens 114a.

[0137] In S920, the CPU 101 obtains the zoom position based on the user's pinch-out operation. If the zoom position obtained by the pinch-out operation is wider than the telephoto end of the standard lens 114b, the process proceeds to S921. If the zoom position obtained by the pinch-out operation is telephoto beyond the telephoto end of the standard lens 114b, the process proceeds to S922.

[0138] In S921, the CPU 101 performs zoom-in processing using the standard lens 114b and returns to S900 in Figure 9A. The CPU 101 can perform zoom-in processing by, for example, changing the zoom position to the position acquired in S920 by digitally zooming the image captured by the standard lens 114b.

[0139] In S922, the CPU 101 determines whether or not video recording is in progress with the telephoto lens 114a. If video recording is in progress with the telephoto lens 114a, the process proceeds to S921. If video recording is not in progress with the telephoto lens 114a, the process proceeds to S923.

[0140] In S923, the CPU 101 changes the lens used for LV display in still image mode to the telephoto lens 114a, which has a narrower field of view. The CPU 101 performs zoom-in processing using the telephoto lens 114a and returns to S900 in Figure 9A. The CPU 101 can perform zoom-in processing, for example, by changing the zoom position to the position acquired in S920 through digital zoom of the image captured by the telephoto lens 114a.

[0141] In S924, the CPU 101 performs zoom-in processing using the telephoto lens 114a and returns to S900 in Figure 9A. The CPU 101 can perform zoom-in processing by obtaining the zoom position based on the user's pinch-out operation and changing the zoom position to the obtained zoom position by digitally zooming the image captured by the telephoto lens 114a.

[0142] Refer to Figure 9C to explain the details of the zoom-out process in S930. In S931, the CPU 101 determines whether the lens used for LV display in still image mode is the wide-angle lens 114c, the standard lens 114b, or the telephoto lens 114a. If the lens used in still image mode is the telephoto lens 114a, the process proceeds to S932. If the lens used in still image mode is the standard lens 114b, the process proceeds to S930, and if the lens used in still image mode is the wide-angle lens 114c, the process proceeds to S935.

[0143] In S932, the CPU 101 obtains the zoom position (zoom magnification) based on the user's pinch-in operation. If the zoom position obtained by the pinch-in operation is on the telephoto side of the wide-angle end of the telephoto lens 114a, processing proceeds to S933. If the zoom position obtained by the pinch-in operation is on the wide-angle side of the telephoto lens 114a and on the telephoto side of the wide-angle end of the standard lens 114b, processing proceeds to S934. If the zoom position obtained by the pinch-in operation is on the wide-angle side of the standard lens 114b, processing proceeds to S937.

[0144] In S933, the CPU 101 performs a zoom-out process using the telephoto lens 114a and returns to S900 in Figure 9A. The CPU 101 can perform the zoom-out process by changing the zoom position to the position obtained in S932, for example, by digitally zooming the image captured by the telephoto lens 114a.

[0145] In S934, the CPU 101 determines whether or not video recording is currently underway with the standard lens 114b. If video recording is underway with the standard lens 114b, the process proceeds to S936. If video recording is not underway with the standard lens 114b, the process proceeds to S935.

[0146] In S935, the CPU 101 changes the lens used for LV display in still image mode to the standard lens 114b, which has a wider field of view. The CPU 101 performs zoom-out processing using the standard lens 114b and returns to S900 in Figure 9A. The CPU 101 can perform zoom-out processing, for example, by changing the zoom position to the position obtained in S932 by digitally zooming the image captured with the standard lens 114b.

[0147] In S936, the CPU 101 changes the lens used for LV display in still image mode to the wide-angle lens 114c, which has a wider field of view. The CPU 101 performs zoom-out processing using the wide-angle lens 114c and returns to S900 in Figure 9A. The CPU 101 can perform zoom-out processing by changing the zoom position to the position obtained in S932, for example, by digital zooming the image captured with the wide-angle lens 114c.

[0148] In S937, CPU101 determines whether or not video recording is in progress with the wide-angle lens 114c. If video recording is in progress with the wide-angle lens 114c, the lens used in still image mode is not switched to the wide-angle lens 114c, and processing proceeds to S938. If video recording is not in progress with the wide-angle lens 114c, processing proceeds to S936.

[0149] In S938, the CPU 101 displays a notification 802 on the display 105 to inform the user that the zoom-out will not proceed any further, as explained in Figure 8, and then returns to S900 in Figure 9A.

[0150] In the S940, the CPU101 determines the zoom position based on the user's pinch-in gesture. This is advantageous. If the zoom position obtained by pinch-in operation is on the telephoto side of the wide end of the standard lens 114b, the process proceeds to S941. If the zoom position obtained by pinch-in operation is on the wide end of the standard lens 114b, the process proceeds to S942.

[0151] In S941, the CPU 101 performs zoom-out processing using the standard lens 114b and returns to S900 in Figure 9A. The CPU 101 can perform zoom-out processing by changing the zoom position to the position acquired in S940, for example, by digitally zooming the image captured by the standard lens 114b.

[0152] In S942, the CPU 101 determines whether or not video recording is in progress with the wide-angle lens 114c. If video recording is in progress with the wide-angle lens 114c, the lens used in still image mode is not switched to the wide-angle lens 114c, and processing proceeds to S944. If video recording is not in progress with the wide-angle lens 114c, processing proceeds to S943.

[0153] In S943, the CPU 101 changes the lens used for LV display in still image mode to the wide-angle lens 114c, which has a wider field of view. The CPU 101 performs zoom-out processing using the wide-angle lens 114c and returns to S900 in Figure 9A. The CPU 101 can perform zoom-out processing, for example, by changing the zoom position to the position acquired in S940 by digitally zooming the image captured by the wide-angle lens 114c.

[0154] In S944, the CPU 101 displays a notification 802 on the display 105 to inform the user that the zoom-out will not proceed any further, as explained in Figure 8, and then returns to S900 in Figure 9A.

[0155] In S945, the CPU 101 obtains the zoom position based on the user's pinch-in operation. If the zoom position obtained by the pinch-in operation is on the telephoto side of the wide-angle lens 114c's wide end, the process proceeds to S946. If the zoom position obtained by the pinch-in operation is on the wide side of the wide-angle lens 114c's wide end, the process proceeds to S947.

[0156] In S946, the CPU 101 performs zoom-out processing using the wide-angle lens 114c and returns to S900 in Figure 9A. The CPU 101 can perform zoom-out processing by, for example, changing the zoom position to the position acquired in S945 by digitally zooming the image captured by the wide-angle lens 114c.

[0157] In S947, the CPU 101 displays a notification 802 on the display 105 to inform the user that the zoom-out will not proceed any further, as explained in Figure 8, and then returns to S900 in Figure 9A.

[0158] Referring to Figure 10, the process of overlaying an image captured in video mode onto an image captured in still image mode displayed in LV mode (hereinafter referred to as the video LV overlay process) will be explained. If the display on display 105 is switched to the LV display in still image mode during video recording, the range recorded in the video will change if the camera orientation changes for still image shooting.

[0159] Therefore, in the third embodiment, when a still image is being displayed in LV mode, the CPU 101 superimposes a video LV image onto the still image LV image when the user performs a predetermined operation. The predetermined operation is, for example, pan operation, tilt operation, pinch-out operation, pinch-in operation, flick operation, etc.

[0160] The still image LV display screen 1001 in Figure 10(a) shows the telephoto lens 1 on the display 105. This is a screen displaying an image captured in still image mode using lens 14a in Live View mode. Icon 605 indicates that video is being recorded in video mode using a lens other than the telephoto lens 114a.

[0161] The still image LV display screen 1002 in Figure 10(b) is a screen where the imaging range has been changed and the position of the subject has moved as a result of the user panning the rear camera 114, compared to the still image LV display screen 1001 in Figure 10(a).

[0162] The sub-screen 1003, which is superimposed on the still image LV display screen 1002, is a screen displaying the captured image captured in video mode using the standard lens 114b and being recorded as video, in LV format. By displaying the captured image in video mode on the sub-screen 1003 in LV format, the user can check the video shooting status (recording status).

[0163] Figure 11 is a flowchart illustrating the video LV superimposed display process according to the third embodiment. The video LV superimposed display process in Figure 11 is realized by the CPU 101 of the smartphone 100 loading the program recorded in the non-volatile memory 103 into memory 102 and executing it.

[0164] In S1101, the CPU 101 accepts user input. If the accepted user input is a pan or tilt operation of the rear camera 114, the process proceeds to S1102. If the accepted user input is a pinch-in or pinch-out operation, the process proceeds to S1105. If the accepted user input is a flick operation, the process proceeds to S1106. If the accepted operation is an operation to end the LV display in still image mode, the process proceeds to S1111.

[0165] In S1102, the CPU 101 superimposes a sub-screen 1003, which displays the captured image in video mode as LV, onto the still image LV display screen 1002, as explained in Figure 10(b). In S1103, the CPU 101 determines whether a predetermined time has elapsed since the sub-screen 1003 was superimposed in S1102. If the predetermined time has elapsed, the process proceeds to S1104. If the predetermined time has not elapsed, the CPU 101 returns to S1103 and enters a waiting state until the predetermined time has elapsed. Once the predetermined time has elapsed, in S1104, the CPU 101 erases the sub-screen 1003, which displays the captured image in video mode as LV.

[0166] In S1105, if the CPU 101 receives a pinch-out operation in S1101, it zooms in on the LV image in still image mode; if the CPU 101 receives a pinch-in operation in S1101, it zooms out on the LV image in still image mode and proceeds to S1102. Since zooming in or out on the LV image in still image mode may change the imaging range in video mode, the CPU 101 displays a sub-screen 1003 that displays the video image in LV mode, similar to the case of panning operations.

[0167] In S1106, the CPU 101 determines the direction of the flick operation received in S1101. If the flick direction is from the edge of the touch panel 106a toward the center, the CPU 101 controls the display of the sub-screen 1003 which displays the captured image in video mode as a live view. On the other hand, if the flick direction is from the center of the touch panel 106a toward the edge, the CPU 101 controls the display of the sub-screen 1003 which displays the captured image in video mode as a live view.

[0168] If the flick direction is from the edge of the touch panel 106a towards the center, the process proceeds to S1107. If the flick direction is from the center of the touch panel 106a towards the edge, the process proceeds to S1109.

[0169] In S1107, the CPU 101 determines whether a sub-screen 1003, which displays the captured image in video mode as LV, is superimposed on the still image LV display screen 1002. If the sub-screen 1003 is displayed, the process returns to S1101. If the sub-screen 1003 is not displayed, the process proceeds to S1108. In S1108, the CPU 101 superimposes the sub-screen 1003, which displays the captured image in video mode as LV, onto the still image LV display screen 1002, and returns to S1101.

[0170] In S1109, the CPU 101 determines whether a sub-screen 1003, which displays the captured image in video mode as LV, is superimposed on the still image LV display screen 1002. If the sub-screen 1003 is displayed, the process proceeds to S1110. If the sub-screen 1003 is not displayed, the process returns to S1101. In S1110, the CPU 101 erases the sub-screen 1003 that is superimposed on the still image LV display screen 1002 and returns to S1101.

[0171] If the operation received in S1101 is an operation to end the LV display in still image mode, the process proceeds to S1111. An operation to end the LV display in still image mode is, for example, an operation to switch to the LV display in video mode, which is the operation of touching the LV display switching button 604 in Figure 10(a). In S1111, the CPU 101 ends the LV display in still image mode, and the video LV superimposed display process shown in Figure 11 ends.

[0172] According to the third embodiment described above, even during video recording, the user can display the captured image in still image mode on approximately the entire screen of the display 105 using LV (Live View). This allows the user to easily confirm the subject they want to photograph and capture the desired still image.

[0173] Furthermore, by performing a predetermined operation, the user can overlay a sub-screen for displaying the currently recorded video in LV (Live View) mode onto the LV display screen in still image mode. Even if the imaging range in video mode changes due to panning or other operations for still image shooting, the user can check the video shooting status by looking at the LV display in video mode on the sub-screen.

[0174] The rear camera 114 is equipped with multiple lenses with different angles of view, and an example has been described in which different lenses are used for video mode and still image mode, but it is not limited to this. The rear camera 114 may be equipped with a single lens. In this case, the image captured in still image mode is generated by extracting one frame from the image captured in video mode (moving image). When a pinch-out or pinch-in operation is received in still image mode, the CPU 101 can perform zoom-in or zoom-out processing by digital zooming the extracted image.

[0175] (Other embodiments) Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the embodiments described above may be combined as appropriate. The present invention has been described using the application to a smartphone as an example, but this is not limited to this example, and it can be applied to any electronic device having a shooting means that can shoot in parallel in multiple imaging modes using multiple lenses with different angles of view. In other words, the present invention can be applied to digital cameras, tablet terminals, and the like.

[0176] The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be performed by a circuit that implements one or more functions (for example, an ASIC). [Explanation of symbols]

[0177] 100: Imaging device (smartphone), 101: CPU, 106: Control panel, 106a: Touch panel, 114: Rear camera, 114a: Telephoto lens, 114b: Standard lens, 114c: Wide-angle lens

Claims

1. An imaging device having multiple lenses with different angles of view, A receiving means for receiving a switching operation to switch the lens used for imaging when images are being captured in parallel using the first imaging mode and the second imaging mode, It includes a control means that controls the switching of the lens used for imaging in response to receiving a switching operation for the lens used for imaging, The control means is The system controls the switching of lenses used in the first imaging mode and the second imaging mode so that different lenses are used in the first imaging mode and the second imaging mode. When the first lens is used in the first imaging mode and the second lens is used in the second imaging mode, if a switching operation is received to switch the lens used in the first imaging mode to the second lens, the system controls the system to switch the lens used in the second imaging mode to a third lens that is different from both the first and second lenses, and to switch the lens used in the first imaging mode back to the second lens. An imaging device characterized by the following features.

2. When the first lens is used in the first imaging mode and the second lens is used in the second imaging mode, and a switching operation is received to switch the lens used in the first imaging mode to a third lens which is different from both the first and second lenses, the control means controls the system to switch the lens used in the first imaging mode to the third lens and to maintain the second lens used in the second imaging mode. The imaging apparatus according to feature 1.

3. The control means controls the receiving means so as not to accept a switching operation for the lens used in the second imaging mode, when it receives a switching operation for the lens used in the first imaging mode. The imaging apparatus according to feature 1.

4. When receiving an operation to switch the lens used in the first imaging mode, the receiving means is controlled so that it can also accept a switch to the lens used in the first imaging mode. ru, The imaging device according to feature 3.

5. The control means controls the lens used in the first imaging mode to be switchable from the currently used lens to another lens when no video is being recorded in the first imaging mode, and to prevent the lens used in the first imaging mode from being switchable when video is being recorded in the first imaging mode. The imaging apparatus according to feature 1.

6. The receiving means displays, on the display means of the imaging device, an item for receiving a switching operation for the lens used for imaging, along with the image captured in the first imaging mode, and accepts the switching operation in response to an operation on the item. The imaging apparatus according to feature 1.

7. The receiving means includes a first item superimposed on the image captured in the first imaging mode for switching the lens used in the first imaging mode, and a second item superimposed on the image captured in the second imaging mode for switching the lens used in the second imaging mode. The imaging apparatus according to feature 1.

8. The receiving means includes a third item which is superimposed on the image captured in the first imaging mode and is used to switch between the lenses used in the first imaging mode and the second imaging mode, respectively. The control means is If the operation on the third item is performed by a single touch at one location, the lens used in the first imaging mode will be switched to the selected lens. If the operation on the third item is a multi-touch operation using two or more locations, the lens used in the second imaging mode is switched to the selected lens. The imaging apparatus according to feature 1.

9. The receiving means, when recording video in the first imaging mode, does not include the lens used in the first imaging mode as a candidate lens that can be switched to in the second imaging mode. The imaging apparatus according to feature 1.

10. The aforementioned receiving means is The operator accepts an operation to set the first imaging mode as a priority imaging mode that takes precedence over the second imaging mode. The lens used in the first imaging mode, which is set as the priority imaging mode, is not included in the list of lenses that can be switched in the second imaging mode. The imaging apparatus according to feature 1.

11. The receiving means includes a first item superimposed on the image captured in the first imaging mode for switching the lens used in the first imaging mode, and a second item superimposed on the image captured in the second imaging mode for switching the lens used in the second imaging mode. The imaging apparatus according to feature 10.

12. The receiving means includes a third item that is superimposed on the image captured in the first imaging mode and is used to switch between the lenses used in the first imaging mode and the second imaging mode, respectively. fruit, The control means is If the operation on the third item is performed by a single touch at one location, the lens used in the first imaging mode will be switched to the selected lens. If the operation on the third item is a multi-touch operation using two or more locations, the lens used in the second imaging mode is switched to the selected lens. The imaging apparatus according to feature 10.

13. The operation to set the priority imaging mode involves touching and moving the image captured in the target imaging mode to the area where the image captured in the priority imaging mode is displayed. The imaging apparatus according to feature 10.

14. The operation to set the priority imaging mode is an operation to change the orientation of the display of the imaging device, and the imaging mode that is set in advance according to the orientation of the display is set as the priority imaging mode. The imaging apparatus according to feature 10.

15. The receiving means can be moved between the image captured in the first imaging mode and the image captured in the second imaging mode, and includes a fourth item for taking a still image. The control means is If the fourth item is present in the image captured in the first imaging mode, a still image is taken using the lens used in the first imaging mode. If the fourth item is present in the image captured in the second imaging mode, control the system to capture a still image using the lens used in the second imaging mode. The imaging apparatus according to feature 1.

16. A control method for an imaging device having multiple lenses with different angles of view, A receiving step that accepts a switching operation to switch the lens used for imaging when images are being captured in parallel using the first imaging mode and the second imaging mode, The system includes a control step that controls the system to switch the lens used for imaging in response to a request to switch the lens used for imaging, In the control step described above, The system controls the switching of lenses used in the first imaging mode and the second imaging mode so that different lenses are used in the first imaging mode and the second imaging mode. When the first lens is used in the first imaging mode and the second lens is used in the second imaging mode, if a switching operation is received to switch the lens used in the first imaging mode to the second lens, the system controls the system to switch the lens used in the second imaging mode to a third lens that is different from both the first and second lenses, and to switch the lens used in the first imaging mode back to the second lens. A control method for an imaging device, characterized by the following:

17. A program for causing a computer to function as one of the means of an imaging apparatus according to any one of claims 1 to 15.

18. A computer-readable storage medium storing a program for causing a computer to function as one of the means of an imaging apparatus according to any one of claims 1 to 15.

Images