Electronic apparatus, control method thereof, and program

Abstract

To prevent the captured image to be superimposed from overlapping with an object.SOLUTION: Disclosed is an electronic apparatus having a plurality of lenses. The electronic device includes: first acquisition means for acquiring a first captured image captured via a first lens among the plurality of lenses; second acquisition means for acquiring a second captured image captured via a second lens different from the first lens; first generation means for generating a composite image in which the second captured image is superimposed on the first captured image; detection means for detecting an object from the first captured image; third acquisition means for acquiring a third captured image captured at a wider view angle than the first captured image when a region of the object detected by the detection means overlaps with a region on which the second captured image is superimposed in the composite image generated by the first generation means; and second generation means for generating a composite image in which the second captured image is superimposed on the third captured image.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an electronic device, a control method thereof, a program, and a recording medium.

Background Art

[0002] There is a shooting method (Picture-in-Picture, hereinafter also referred to as PIP) that superimposes one of a plurality of captured images on another captured image for synthesis and output. Shooting with PIP is used not only for still image shooting but also in various scenes such as video or live video distribution, video display in a web conference, and TV video.

[0003] In recent years, the development of smartphones with compound eyes has advanced, and many of them are equipped with cameras having lenses with different angle-of-view, such as wide-angle, standard, and telephoto. Among them, as a camera function of a smartphone, there is also one having a function of displaying and recording captured images with compound eyes in PIP. When performing shooting and display with PIP using an electronic device such as a smartphone, there may be a case where the subject on the captured image and the display of the captured image to be superimposed (hereinafter also referred to as a PIP window) overlap, resulting in a decrease in visibility.

[0004] Patent Document 1 discloses an image display device that detects a subject and moves the position of the PIP window if the subject is facing the direction of the PIP window.

[0005] Patent Document 2 discloses an imaging device that automatically changes the magnification of a lens as the subject that the photographer presses an operation button to capture is cut off or moves, and controls so that the subject is within the angle of view.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

[0007] However, in Patent Document 1, the subject may still overlap with the PIP window even after it has been moved. For example, when the lens magnification is increased to make the subject appear larger, there is a high possibility that the subject will overlap with the PIP window even after it has been moved, making it difficult to reduce the poor visibility of the subject.

[0008] Patent Document 2 describes how the field of view can be changed based on the subject's movement to keep the subject within the field of view, but it does not take into account in-camera displays such as PIP windows. Therefore, it is difficult to avoid overlap with in-camera displays. Furthermore, Patent Document 2 requires the photographer to be conscious of camera operation. For example, a photographer may be simultaneously hosting a broadcast program and operating the camera or smartphone used for broadcasting. The photographer must constantly check the appearance of the subject and the PIP window, and if the subject and the PIP window overlap, they must interrupt video recording or broadcasting to change the camera's field of view or position. As a result, the photographer cannot concentrate on filming and broadcasting (or hosting a broadcast program, etc.).

[0009] The present invention aims to prevent the subject from overlapping with the superimposed captured image. [Means for solving the problem]

[0010] The present invention relates to an electronic device having a plurality of lenses, comprising: a first acquisition means for acquiring a first image captured through a first lens among the plurality of lenses; a second acquisition means for acquiring a second image captured through a second lens different from the first lens; a first generation means for generating a composite image by superimposing the second image onto the first image; a detection means for detecting a subject from the first image; a third acquisition means for acquiring a third image captured at a wider angle than the first image when the area of ​​the subject detected by the detection means in the composite image generated by the first generation means overlaps with the area on which the second image is superimposed; and a second generation means for generating a composite image by superimposing the second image onto the third image. [Effects of the Invention]

[0011] According to the present invention, it is possible to prevent the subject and the superimposed captured image from overlapping. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram of the smartphone's exterior. [Figure 2] This is a block diagram showing the configuration of a smartphone. [Figure 3] This is a diagram to explain the problem that we are trying to solve. [Figure 4] This is a flowchart showing the control process of this embodiment. [Figure 5] This flowchart shows the subject detection and display change process of this embodiment. [Figure 6] This diagram shows the video recording screen and the image display settings screen. [Figure 7] This diagram shows the relationship between the subject and the display area of ​​the PIP window. [Modes for carrying out the invention]

[0013] Preferred embodiments of the present invention will be described below with reference to the drawings. In this embodiment, a smartphone will be used as an example of an electronic device. Figure 1 is an external view of smartphone 100. Figure 1(a) is a front view of smartphone 100, and Figure 1(b) is a rear view of smartphone 100. The smartphone 100 has a display 105, a rear camera 114, and a front camera 115. The display 105 is a display unit located on the front of the smartphone that displays images and various information. The smartphone 100 can display live view images (LV images) taken with the rear camera 114 or the front camera 115 on the display 105. The rear camera 114 includes a telephoto camera 114a, a standard camera 114b, and an ultra-wide-angle camera 114c. The front camera 115 includes a standard front camera 115a and an ultra-wide-angle front camera 115b.

[0014] The smartphone 100 has an operating unit 106. The operating unit 106 includes a touch panel 106a, a power button 106b, a volume up button 106c, a volume down button 106d, a home button 106e, and the like. The touch panel 106a is a touch operation member and can detect a touch operation on the display surface (operation surface) of the display 105. The power button 106b is an operation member and can switch the on / off state of the display 105. If the pressing of the power button 106b is continued (long-pressed) for a certain period of time, for example, 3 seconds, the power of the smartphone 100 can be switched on / off. The volume plus button 106c and the volume minus button 106d are volume buttons for controlling the volume output from the audio output unit 112 described later. Pressing the volume plus button 106c increases the volume, and pressing the volume minus button 106d decreases the volume. Also, in the shooting standby state when using the camera, pressing the volume plus button 106c or the volume minus button 106d also functions as a shutter button for instructing shooting. The user can arbitrarily set so that specific functions are executed when the power button 106b and the volume minus button 106d are pressed simultaneously or when the volume minus button 106d is quickly pressed several times.

[0015] The home button 106e is an operation button for causing the display 105 to display the home screen which is the startup screen of the smartphone 100. When various applications are started and used in the smartphone 100, pressing the home button 106e can temporarily close the various applications that are running and display the home screen. Although the home button 106e is assumed to be a physically pressable button, it may also be a touchable button having the same function as that displayed on the display 105 instead of a physical button.

[0016] Also, the smartphone 100 has an audio output terminal 112a and a speaker 112b. The audio output terminal 112a is an earphone jack and is a terminal for outputting audio to earphones, external speakers, etc. The speaker 112b is a built-in speaker inside the body that emits sound. When the audio is output from the smartphone 100, for example, when an earphone cord is not attached to the terminal for outputting audio to the audio output terminal 112a, the sound is output from the speaker 112b.

[0017] Figure 2 is a block diagram showing an example of the configuration of the smartphone 100. In Figure 2, the same components as those in Figure 1 are labeled with the same reference numerals. The smartphone 100 has a CPU 101, a memory 102, a non-volatile memory 103, an out-camera image processing unit 104, a display 105, an operation unit 106, a recording medium I / F 107, an external I / F 109, and a communication I / F 110 connected to an internal bus 150. Also, the smartphone 100 has an audio output unit 112, an attitude detection unit 113, an out-camera 114, an in-camera 115, and an in-camera image processing unit 116 connected to the internal bus 150. Each unit connected to the internal bus 150 can exchange data with each other via the internal bus 150.

[0018] The CPU 101 is a control unit that controls the entire smartphone 100 and consists of at least one processor or circuit. The memory 102 consists of, for example, a RAM (volatile memory using semiconductor elements, etc.). The CPU 101 controls each unit of the smartphone 100 using the memory 102 as a work memory according to a program stored in the non-volatile memory 103, for example. The non-volatile memory 103 stores image data, audio data, other data, various programs for the operation of the CPU 101, etc. The non-volatile memory 103 is composed of, for example, a flash memory, a ROM, etc.

[0019] The rear camera image processing unit 104 performs various image processing and subject recognition processing on images captured by the rear camera 114, based on the control of the CPU 101. There is a telephoto camera image processing unit 104a, a standard camera image processing unit 104b, and an ultra-wide-angle camera image processing unit 104c, respectively. Each image processing unit performs processing on images captured by its respective camera. In this embodiment, each of the three rear cameras is configured to have its own image processing unit, but it is not necessary for all of them to be separate; any two cameras may share one image processing unit, or all three cameras may share an image processing unit. Similarly, the front camera image processing unit 116 performs various image processing and subject recognition processing on images captured by the front camera 115. Each image processing unit can perform various image processing operations on images stored in the non-volatile memory 103 or recording medium 108, video signals acquired via the external I / F 109, images acquired via the communication I / F 110, etc. The image processing performed by each image processing unit includes A / D conversion, D / A conversion, image data encoding, compression, decoding, scaling (resizing), noise reduction, color conversion, etc. Each image processing unit may be composed of a dedicated circuit block for performing a specific image processing operation. The out-camera image processing unit 104 may be integrated into a single processing block and configured to handle images from each camera in parallel or time-division processing. Furthermore, depending on the type of image processing, the CPU 101 can perform image processing according to a program without using each image processing unit.

[0020] The display 105 displays images and GUI (Graphical User Interface) screens based on the control of the CPU 101. The CPU 101 generates display control signals according to the program and controls various parts of the smartphone 100 to generate video signals for display on the display 105 and output them to the display 105. The display 105 displays images based on the output video signals. The smartphone 100 itself only has an interface for outputting video signals for display on the display 105, and the display 105 may be an external monitor (such as a television).

[0021] The operation unit 106 is an input device for receiving user input, including a keyboard and other character information input device, a mouse and a touch panel, buttons, dials, joysticks, touch sensors, and a touchpad. The touch panel is a flat input device superimposed on the display 105, and outputs coordinate information corresponding to the position of contact. The operation unit 106 includes the touch panel 106a, power button 106b, volume up button 106c, volume down button 106d, and home button 106e as described above.

[0022] The recording medium interface 107 can accommodate recording media 108 such as memory cards, CDs, and DVDs, and, based on the control of the CPU 101, reads data from the inserted recording media 108 and writes data to the recording media 108. The recording media 108 may also be the built-in storage of the smartphone 100. The external interface 109 is an interface for connecting to external devices via wired or wireless cables and for inputting and outputting video and audio signals. The communication interface 110 is an interface for communicating with external devices and the internet 111, etc., to send and receive various data such as files and commands.

[0023] The audio output unit 112 outputs audio from video and music data, operation sounds, ringtones, various notification sounds, etc. The audio output unit 112 includes an audio output terminal 112a for connecting earphones, etc., and a speaker 112b, but audio output may also be performed via wireless communication, etc. The attitude detection unit 113 detects the attitude of the smartphone 100 relative to the direction of gravity, as well as the tilt of the attitude relative to the yaw, roll, and pitch axes. Based on the attitude detected by the attitude detection unit 113, it is possible to determine whether the smartphone 100 is held horizontally, vertically, facing upwards, facing downwards, or in an oblique position. The attitude detection unit 113 can use at least one of the following: an acceleration sensor, a gyroscope, a geomagnetic sensor, a compass sensor, an altitude sensor, etc., and it is also possible to use a combination of multiple sensors.

[0024] The rear camera 114 is a camera located on the side of the smartphone 100's casing opposite to the display 105. The telephoto camera 114a has a longer focal length than the standard camera 114b, allowing it to capture images at a greater telephoto range than the standard camera 114b. The ultra-wide-angle camera 114c has a shorter focal length than the standard camera 114b, allowing it to capture images at a wider angle than the standard camera 114b. In other words, the focal lengths decrease in the order of telephoto camera 114a, standard camera 114b, and ultra-wide-angle camera 114c, and the field of view widens accordingly. In this embodiment, the telephoto camera 114a is assumed to have a lens with a mechanism that optically magnifies to a predetermined magnification, but it may also have a mechanism that allows the user to vary the magnification. The front camera 115 is a camera located on the same side of the smartphone 100's casing as the display 105. The ultra-wide-angle front camera 115b has a shorter focal length than the standard front camera 115a, allowing for wider-angle shots than the standard front camera 115a.

[0025] The telephoto camera 114a, standard camera 114b, and ultra-wide-angle camera 114c can all take pictures simultaneously. However, it is not necessary for all three cameras to take pictures at the same time; any two of the three cameras may be operated, or one camera may take pictures independently. Live View images taken by the rear camera 114 and the front camera 115 can all be displayed on the display 105. The touch panel 106a can be used to select which camera's image to display on the display 105. Specifically, selecting the telephoto camera 114a will display a more magnified image on the display 105 than the standard camera 114b. Selecting the standard camera 114b will display an image that is wider-angle than the telephoto camera 114a and more magnified than the ultra-wide-angle camera 114c. Selecting the ultra-wide-angle camera 114c will display an image that is wider-angle than both the telephoto camera 114a and the standard camera 114b. Alternatively, by using either the rear camera 114 or the front camera 115, users can choose to photograph the scene in front of them or take a selfie of themselves.

[0026] The operation unit 106 includes a touch panel 106a. The CPU 101 can detect the following operations or states on the touch panel 106a. - A finger or pen that was not previously touching the touch panel 106a now touches the touch panel 106a, i.e., the start of a touch (hereinafter referred to as Touch-Down). - The touch panel 106a is in a state where a finger or pen is touching it (hereinafter referred to as Touch-On). - The finger or pen is moving while touching the touch panel 106a (hereinafter referred to as Touch-Move). - The finger or pen that was touching the touch panel 106a is lifted off the touch panel 106a, i.e., the touch is finished (hereinafter referred to as Touch-Up). - The state in which nothing is being touched on the touch panel 106a (hereinafter referred to as Touch-Off).

[0027] When a touchdown is detected, a touch-on is also detected simultaneously. After a touchdown, touch-ons are usually detected continuously unless a touch-up is detected. If a touch-move is detected, a touch-on is also detected simultaneously. Even if a touch-on is detected, a touch-move will not be detected if the touch position has not moved. A touch-off is detected when all fingers or pens that were touching the screen have been detected as having touched up. These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 106a, are notified to the CPU 101 via the internal bus. Based on the notified information, the CPU 101 determines what kind of operation (touch operation) was performed on the touch panel 106a. For touch moves, the direction of movement of the finger or pen moving on the touch panel 106a can also be determined for each vertical and horizontal component on the touch panel 106a based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation was performed. An operation in which a finger is touched on the touch panel 106a and then quickly moved a certain distance and released is called a flick. In other words, a flick is an operation in which the finger is quickly swiped across the touch panel 106a. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected, and a touch-up is then detected, it can be determined that a flick was performed (it can be determined that a flick followed a slide operation). Furthermore, touching multiple locations (for example, two points) simultaneously to bring them closer together is called a pinch-in, and touching them further apart is called a pinch-out. Pinch-out and pinch-in are collectively referred to as a pinch operation (or simply a pinch). The touch panel 106a may use any of the various types of touch panels, such as resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, or optical sensor. There are methods that detect a touch when there is contact with the touch panel, and methods that detect a touch when a finger or pen approaches the touch panel, and either method is acceptable.

[0028] This embodiment describes the display process of the subject and the PIP window, as well as the field of view change process, when recording and streaming video on a smartphone 100.

[0029] Figure 3 is a diagram illustrating the problem that this embodiment aims to solve. Figure 3(a) shows an example of video recording or video streaming in which the photographer 301 uses the telephoto camera 114a of the smartphone 100 to photograph the subject 302 while simultaneously using the standard front camera 115a to photograph the photographer 301's facial expression. The photographer 301 is expected to perform picture-in-picture (PIP) shooting, superimposing the LV image acquired by the standard front camera 115a onto the LV image acquired by the telephoto camera 114a. Furthermore, for PIP shooting, any combination of the rear camera 114 and front camera 115 is acceptable, not limited to the combination of the telephoto camera 114a and the standard front camera 115a.

[0030] Figure 3(b) shows the image displayed on the smartphone 100's display 105 as a result of the shooting described in Figure 3(a). The LV image 304 acquired by the standard front camera 115a is superimposed on the LV image 303 acquired by the telephoto camera 114a and displayed on the display 105 in a PIP window. In Figure 3(b), the subject 302 in the LV image 303 and the LV image 304 overlap in some parts, and the entire subject 302 is not visible. Therefore, the photographer 301 cannot confirm the entire image of the subject 302. For this reason, the photographer 301 needs to operate the smartphone 100 itself or the control unit 106 to prevent overlap.

[0031] Figure 3(c) shows the operation to move the superimposed LV image 304 in order to avoid the overlap between the subject 302 and the LV image 304 that occurred in Figure 3(b). The photographer 301 operates the touch panel 106a of the smartphone 100 with his finger 305 to move the LV image 304 to another position 306. However, if the area of ​​the subject 302 is large, the subject 302 and the LV image 304 may overlap even if the LV image 304 is moved to another position 306. In addition, the photographer 301 has to interrupt his shooting and broadcasting and operate the control unit 106 of the smartphone 100 to avoid the overlap between the subject 302 and the LV image 304, which prevents him from concentrating on shooting and broadcasting.

[0032] The following describes an embodiment that automatically suppresses the overlap between the subject and the PIP window. Figure 4 is a flowchart of the control process in this embodiment. The flowchart in Figure 4 is realized by the CPU 101 executing a program stored in the non-volatile memory 103. The flowchart in Figure 4 is also started by switching the smartphone 100 to the camera function.

[0033] In S401, the CPU 101 starts driving the rear camera 114, the front camera 115, the rear camera image processing unit 104, and the front camera image processing unit 116, and transitions to the camera's shooting standby state. In S402, the CPU 101 displays the LV image acquired through the first lens on the display 105. In this embodiment, the LV image acquired through the first lens is the LV image taken by the telephoto camera 114a. However, the LV images to be displayed may also be those acquired by the standard camera 114b, the ultra-wide-angle camera 114c, the standard front camera 115a, or the ultra-wide-angle front camera 115b. Furthermore, the photographer may be able to select the LV images from the rear camera 114 and the front camera 115 that they wish to display on the display 105 by operating the operation unit 106 from the candidates displayed on the display 105.

[0034] In S403, the CPU 101 stores information about the camera being used as the main image on the display 105 in the memory 102. Here, the main image is the image currently displayed as an LV image across the entire display 105, and is the image that is superimposed as the first captured image. In this embodiment, the memory 102 stores information that the telephoto camera 114a (first lens) is being used.

[0035] In S404, the CPU 101 determines whether or not it has received an input from the control unit 106 to start video recording. If it has received an input, it proceeds to S405; otherwise, it proceeds to S410. In S405, CPU 101 executes the process to start video recording and proceeds to S500. The process to start video recording includes at least one of the following: creating a new video file, recording the start time, and starting recording to the recording medium 108.

[0036] In the S500, the CPU 101 performs subject detection and display change processing. This process will be described later using the flowchart in Figure 5.

[0037] In S406, CPU101 determines whether or not video recording is in progress. If video recording is in progress, the process proceeds to S407; otherwise, it determines that video streaming is in progress and proceeds to S412.

[0038] In S407, the CPU 101 writes the LV images and composite images acquired by the rear camera 114 and front camera 115 as video frames to the recording medium 108 via the recording medium I / F 107. Here, the photographer can choose whether to record the composite image or record the LV images separately. That is, when displaying a composite image on the display 105 by superimposing the second captured image onto the first captured image, as will be described later, the photographer can choose whether to record the composite image as a single video file or to record the first and second captured images as separate video files. The CPU 101 records the LV images and composite images in a recording format according to the photographer's selection. Note that when the CPU 101 records the first and second captured images as separate video files, it performs the following recording process. That is, even when displaying a composite image on the display 105 by superimposing the second captured image onto the third captured image, as will be described later, the CPU 101 continues to record the first and second captured images as separate video files.

[0039] In S408, CPU101 determines whether or not it has received input indicating the end of video recording or video streaming. If it has received such input, it proceeds to S409; otherwise, it proceeds to S413.

[0040] In S409, the CPU 101 stops video recording or video distribution processing. The CPU 101 stops video distribution processing by ceasing transmission to the Internet 111 via the communication I / F 110.

[0041] In S410, the CPU 101 determines whether or not it has received an input from the operation unit 106 to start video streaming. If it has received an input, it proceeds to S411; otherwise, it returns to S402. In S404 and S410, if neither an input for starting video recording nor video streaming has been received, the CPU 101 may return to S402 and continue in the shooting standby state, or it may determine that it is taking a still image and execute the process of taking a still image.

[0042] In S411, the CPU 101 executes the process of starting video distribution. This process includes at least one of the following: connecting to the Internet 111 via the communication I / F 110, determining the video file destination server, and authentication with the video file destination server. In addition to starting video distribution, the CPU 101 can also write the composite image as a video frame to the recording medium 108 via the recording medium I / F 107. At this time, the CPU 101 records the composite image as a single video file.

[0043] In S412, the CPU 101 transmits the LV images and composite images acquired by the rear camera 114 and the front camera 115 as video frames to the internet 111 via the communication I / F 110.

[0044] In S413, the CPU 101 determines whether the camera used for acquiring the main image is the third lens and not the first lens. If it is the third lens, the process proceeds to S505 in Figure 5, which will be described later; otherwise, it returns to S500, which is immediately after executing S405 and S411. In S414, the CPU 101 determines whether an operation to terminate the camera function has been performed from the operation unit 106. If an operation to terminate the function has been performed, the operation of the rear camera 114, front camera 115, rear camera image processing unit 104, and front camera image processing unit 116 is stopped, and the camera function is terminated. Otherwise, the process returns to S402.

[0045] Figure 5 is a flowchart showing the subject detection and display change process in the S500 described above. In S501, the CPU 101 acquires the LV image captured through the first lens as the first image capture image. In this embodiment, the LV image captured through the first lens is the LV image captured by the telephoto camera 114a. In S502, the CPU 101 acquires the LV image captured through the second lens as the second image capture image. In this embodiment, the LV image captured through the second lens is the LV image captured by the standard in-camera 115a. The first lens and the second lens may be any combination of the rear camera 114 and the front camera 115.

[0046] In S503, the CPU 101 superimposes the second captured image onto the first captured image with a region size So and displays it on the display 105 in picture-in-picture (PIP) format. In this embodiment, the CPU 101 generates a composite image by combining the second captured image with the first captured image by controlling the telephoto camera image processing unit 104a and the standard in-camera image processing unit 116a.

[0047] In S504, the CPU 101 calculates the size, position coordinates (xo,yo), width Wo, and height Ho of the region So on which the second captured image is superimposed, and stores them in memory 102. Note that processes S503 and S504 may be performed at either the start of video recording or the start of video streaming.

[0048] In S505, the CPU 101 controls the out-camera image processing unit 104 or the in-camera image processing unit 116 to perform subject detection processing to determine if a subject is visible in the first captured image. In this embodiment, the CPU 101 controls the telephoto camera image processing unit 104a to perform subject detection processing based on the LV image acquired by the telephoto camera 114a. In S506, the CPU 101 determines whether or not a subject has been detected based on the results of the subject detection process. If a subject is detected, the process proceeds to S507; otherwise, the process in S500 is terminated.

[0049] In S507, the CPU 101 controls the out-camera image processing unit 104 or the in-camera image processing unit 116 to calculate the size, position coordinates (x,y), width W, and height H of the detection area S of the subject in the first captured image, and stores them in the memory 102. In this embodiment, the CPU 101 controls the telephoto camera image processing unit 104a to calculate the detection area S of the subject in the LV image acquired by the telephoto camera 114a. Furthermore, as will be described later, if the main image acquisition lens is changed to a third lens (in this embodiment, the standard camera 114b or the ultra-wide-angle camera 114c), the detection area S' of the subject changes to position coordinates (x',y'), width W', and height H'. That is, in this embodiment, when proceeding from S413 to S505, S506, and S507 in Figure 4, the CPU 101 controls the standard camera image processing unit 104b or the ultra-wide-angle camera image processing unit 104c to calculate the detection area S' of the subject.

[0050] In S508, the CPU 101 determines whether the region So on which the second captured image is superimposed overlaps with the subject detection region S. If they overlap, the process proceeds to S509; otherwise, the process in S500 is terminated. Furthermore, the process does not necessarily have to proceed immediately to S508 when region So and the detection region S of the subject overlap; it may also involve a process such as comparing with a threshold as shown below.

[0051] The first step is to compare the overlapping area with a threshold. Specifically, the CPU 101 determines whether the overlapping area So on which the second captured image is superimposed and the subject detection area S is greater than or equal to a predetermined first threshold. If it is greater than or equal to the predetermined first threshold, the process proceeds to S509; otherwise, the process in S500 is terminated. Secondly, the process involves comparing the overlapping time with a threshold. Specifically, the CPU 101 determines whether the overlapping time between the region So on which the second captured image is superimposed and the subject detection region S is greater than or equal to a predetermined second threshold. If it is greater than or equal to the predetermined second threshold, the process proceeds to S509; otherwise, the process in S500 is terminated.

[0052] In S509, the CPU 101 determines whether it is possible to switch to a camera with a wide-angle lens. Specifically, the CPU 101 determines whether it is possible to switch to a camera with a wider-angle lens than the camera currently acquiring the first image, from among multiple cameras located on the same plane as the camera currently acquiring the first image. For example, if the first image is currently being acquired using the telephoto camera 114a, it is possible to switch to the standard camera 114b or the ultra-wide-angle camera 114c as the camera with a wide-angle lens, so it is determined that it is possible to switch to a camera with a wide-angle lens. On the other hand, if the first image is currently being acquired using the ultra-wide-angle camera 114c, it is determined that it is not possible to switch to a camera with a wide-angle lens because there is no camera with a wider-angle lens than the ultra-wide-angle camera 114c. Note that although this explanation describes the case where the first image is taken by the rear camera 114, the same determination can be made even if the first image is taken by the front camera 115. If it's possible to switch to the camera with the wide-angle lens, proceed to S510; otherwise, proceed to S513.

[0053] In the S510, the CPU 101 determines that the camera information used as the main image on the display 105 is switchable, changes it to the information of the wide-angle lens camera, and stores it in memory 102. In this embodiment, the information stored in memory 102 is changed to the information that the standard camera 114b (third lens) is being used.

[0054] In S511, the CPU 101 acquires the LV image captured through the third lens as the third image capture image. In this embodiment, the LV image captured through the third lens is the LV image captured by the standard camera 114b.

[0055] In S512, the CPU 101 superimposes the second captured image onto the third captured image with a region size So and displays it on the display 105 in picture-in-picture (PIP) format. In this embodiment, the CPU 101 generates a composite image by combining the second captured image with the third captured image by controlling the standard camera image processing unit 104b and the standard in-camera image processing unit 116a. In this embodiment, the case described in the series of processes S509 to S512 is when the LV image captured through the third lens is the LV image captured by the standard camera 114b. However, if it is possible to switch to the ultra-wide-angle camera 114c, the LV image may be acquired by switching to the ultra-wide-angle camera 114c without going through the standard camera 114b.

[0056] In S513, the CPU 101 determines whether it is possible to change the cropping area of ​​the first captured image, that is, to change the field of view to a wider angle by electronic zoom. If it is possible, proceed to S514; otherwise, proceed to S516. In S514, the CPU 101 modifies the cropping area of ​​the first captured image, i.e., performs electronic zoom processing. In this embodiment, the CPU 101 controls the telephoto camera image processing unit 104a to perform the process of modifying the cropping area of ​​the LV image acquired by the telephoto camera 114a.

[0057] In S515, the CPU 101 superimposes the second captured image onto the first captured image after the cropping area has been changed, with the size of area So, and displays it on the display 105 in picture-in-picture (PIP) format. In S516, the CPU 101 changes the position in which the second captured image is superimposed on the first captured image. At this time, the CPU 101 changes the position of the second captured image to one of the four corners of the first captured image. Note that the CPU 101 is not limited to changing the position of the second captured image, but may also erase (not display) the second captured image from the display 105 and display at least one of the first or third captured image on the display 105.

[0058] Figure 6 shows the video recording and image display settings screen on a smartphone in this embodiment. Figure 6(a) shows an example of the screen when the photographer selects the camera function of the smartphone 100 using the operation unit 106 and the device enters a shooting standby state. By touching down icons 601 to 608 displayed on the display 105 using the touch panel 106a, users can activate various settings and functions. Touching down icon 601 allows users to switch between the rear camera 114 and the front camera 115, which are used to acquire LV images and display them on the display 105.

[0059] Icons 602-604 represent shooting functions. Touching down the "Photo" text icon 602 activates the still image shooting function, touching down the "Video" text icon 603 activates the video recording function, and touching down the "LIVE" text icon 604 activates the video streaming function to the internet 111. The magnification icon 605 represents the current magnification, and touching it down switches the magnification of the currently used rear camera 114 or front camera 115. It may be configured so that increasing the magnification above a certain threshold switches to the telephoto camera 114a, and decreasing it below a certain threshold switches to the ultra-wide-angle camera 114c. Similarly, for the front camera, it may be configured so that decreasing the magnification below a certain threshold switches to the ultra-wide-angle front camera 115b. In addition, even without touching down the magnification icon 605, pinching in on the touch panel 106a switches the magnification to the wide-angle side, and pinching out switches it to the telephoto side.

[0060] Icon 606 represents the camera settings function; touching it down will take you to a screen where you can configure settings such as the video recording method and video streaming destination. Touching down icon 607 allows you to set the screen display format. Details will be described later in Figure 6(b). Touching down icon 608 will take a still image and save it if you are using the still image shooting function, start or stop video recording if you are using the video shooting function, or start or stop video streaming if you are using the video streaming function. These operations may be performed not only on the touch panel 106a, but also on at least one of the operation units 106.

[0061] Figure 6(b) shows the screen display format settings screen that transitions when icon 607 is touched down. Options 609 represent the display format for each LV image, and the photographer can select the option 609 corresponding to the format they want to display during shooting by touching down. For example, option 609a represents displaying using the telephoto camera 114a and the standard front camera 115a for PIP shooting. When option 609a is selected, LV images are acquired with the telephoto camera 114a and the standard front camera 115a, and the LV image acquired with the standard front camera 115a is superimposed and displayed in PIP window format. Note that option 609 may be set in conjunction with the screen display and the video recording / video streaming images, or it may be possible to set them separately, such as setting the screen display to PIP and the video recording / video streaming images to be only the LV images from the telephoto camera 114a.

[0062] In option 609, in addition to the combinations shown in the diagram, other combinations are also possible, such as using two rear cameras 114, or superimposing the LV image acquired by the rear camera 114 onto the LV image acquired by the front camera 115.

[0063] Figure 7 shows the relationship between the subject and the display area of ​​the PIP window in this embodiment. Figure 7(a) shows an example of displaying the LV image 701 acquired by the telephoto camera 114a on the display 105. To superimpose the LV image acquired by the standard in-camera 115a onto the LV image 701 with the size of the region So (702 in Figure 7) shown by the thick line, the coordinates (xo, yo), width Wo, and height Ho are calculated. On the other hand, the detection region S of the subject shown by the dotted line (703 in Figure 7) is where the subject is detected by the telephoto camera image processing unit 104a in steps S505 to S507, and the size of the subject is calculated.

[0064] In Figure 7(a), the position coordinates (x,y) of the subject are calculated as the top left corner as an example, but the position coordinates can be any position as long as they are within the subject's detection area S (703 in Figure 7). Also, the width W and height H are calculated to calculate the size of the subject's detection area S (703 in Figure 7) as a rectangle, but the rectangle of the area is not limited to this; any method that can calculate the size of the subject is acceptable. Here, when the LV image acquired by the standard in-camera 115a is superimposed on the LV image 701 in region So (702 in Figure 7), i.e., displayed in a PIP window format, a part of region So and a part of the subject detection region S overlap in overlapping region 704.

[0065] The LV image 705 in Figure 7(b) was acquired when the CPU 101 determined in S508 that there was overlap and switched from the telephoto camera 114a to the standard camera 114b or the ultra-wide-angle camera 114c. Note that the LV image 705 may also be an LV image acquired with the telephoto camera 114a that has been electronically zoomed. Because the field of view changes due to the switching of the rear camera 114 or the electronic zooming process, the subject detection area S (706 in Figure 7) becomes smaller than the subject detection area S in Figure 7(a) (703 in Figure 7). To acquire the LV image 705, the position coordinates (x', y'), width W', and height H' of the subject are recalculated. The CPU 101 executes the processes S508 to S516 until the overlapping region 704 in Figure 7(a) is eliminated by switching the rear camera 114 or by electronic zoom processing. If the overlapping region 704 reappears after the process has been completed to eliminate it once, the processes S508 to S516 may be executed again.

[0066] As described above, according to this embodiment, when the subject and the superimposed captured image (PIP window) overlap as shown in Figure 7(a), the CPU 101 changes the field of view by switching to the wide-angle out-camera or in-camera. By displaying the subject and the PIP window so that they do not overlap, the entire subject is visible, improving the visibility of the subject. In addition, the photographer does not need to interrupt the progress of the shooting or broadcasting they wish to do to operate the field of view with the control unit 106, so they can concentrate on the progress of the shooting or broadcasting they wish to do.

[0067] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or recording medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0068] Furthermore, the various controls described above, which are performed by the CPU 101, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0069] Although this invention has been described using a smartphone as an example, it is not limited to this example and can be applied to any electronic device having multiple lenses. In other words, this invention can be applied to personal computers, PDAs, digital cameras, portable image viewers, digital photo frames, music players, game consoles, e-book readers, and the like.

[0070] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate. [Explanation of symbols]

[0071] 100: Smartphone (electronic device) 101: CPU 105: Display 114: Rear camera 114a: Telephoto camera 114b: Standard camera 114c: Ultra-wide-angle camera 115: Front camera 115a: Standard front camera 115b: Ultra-wide-angle front camera

Claims

1. An electronic device having a plurality of lenses, a first acquisition means for acquiring a first captured image captured through a first lens among the plurality of lenses; a second acquisition means for acquiring a second captured image captured through a second lens different from the first lens; a third acquisition means for acquiring a third captured image captured through a third lens among the plurality of lenses, the third lens having a wider angle than the first lens; a control means for controlling the second captured image to be superimposed on the first captured image and displayed on a display means; a detection unit that detects a subject from the first captured image, The control means when an area of ​​the subject detected from the first captured image by the detection means overlaps with an area of ​​the first captured image on which the second captured image is superimposed, control is performed so that the second captured image is superimposed on the third captured image and displayed on the display means; When the area of ​​the subject detected from the first captured image by the detection means does not overlap with the area of ​​the first captured image on which the second captured image is superimposed, the second captured image is controlled to be superimposed on the first captured image and displayed on the display means.

1. An electronic device comprising:

2. the first lens and the third lens are provided on a first surface of the electronic device; The electronic device according to claim 1 , wherein the second lens is provided on a second surface different from the first surface.

3. A recording means for recording an image on a recording medium, The recording means is 3. The electronic device according to claim 1, further comprising: a display unit configured to display a captured image of a first captured image on the display means; a display unit configured to display a captured image of a second captured image on the display means;

4. an output means for outputting the image to an external device; The output means includes:

3. The electronic device according to claim 1, further comprising: an image obtained by superimposing the second captured image on the first captured image or an image obtained by superimposing the second captured image on the third captured image, the image being output via a network.

5. A recording means for recording an image on a recording medium, The recording means is 5. The electronic device according to claim 4, further comprising: an image in which the second captured image is superimposed on the first captured image; or an image in which the second captured image is superimposed on the third captured image, the image being recorded.

6. The control means The electronic device described in any one of claims 1 to 5, characterized in that when an overlapping area between the subject area and the area on which the second captured image is superimposed is equal to or greater than a predetermined first threshold, the second captured image is controlled to be superimposed on the third captured image and displayed on the display means.

7. The control means The electronic device described in any one of claims 1 to 5, characterized in that when the overlap time between the subject area and the area on which the second captured image is superimposed is equal to or greater than a predetermined second threshold, the second captured image is controlled to be superimposed on the third captured image and displayed on the display means.

8. The control means 8. The electronic device according to claim 1, further comprising: when the third captured image cannot be acquired, the electronic device is controlled so that the area in which the second captured image is superimposed is moved and displayed on the display means.

9. The control means 8. The electronic device according to claim 1, further comprising control for displaying only the first captured image or the third captured image on the display means when the third captured image cannot be acquired.

10. A method for controlling an electronic device having a plurality of lenses, comprising the steps of: a first acquisition step of acquiring a first captured image captured through a first lens among the plurality of lenses; a second acquisition step of acquiring a second captured image captured through a second lens different from the first lens; a third acquisition step of acquiring a third captured image captured through a third lens among the plurality of lenses, the third lens having a wider angle than the first lens; a control step of controlling the second captured image to be superimposed on the first captured image and displayed on a display means; a detection step of detecting a subject from the first captured image, In the control step, when an area of ​​the subject detected from the first captured image by the detection step overlaps with an area of ​​the first captured image on which the second captured image is superimposed, control is performed so that the second captured image is superimposed on the third captured image and displayed on the display means; When the area of ​​the subject detected from the first captured image by the detection step does not overlap with the area of ​​the first captured image on which the second captured image is superimposed, control is performed so that the second captured image is superimposed on the first captured image and displayed on the display means.

23. A method for controlling an electronic device comprising:

11. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 9.

12. 10. A computer-readable recording medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 9.

Images