Electronic device and control method thereof

Abstract

To shoot a moving image applied with depth composition while suppressing reduction in a frame rate.SOLUTION: An electronic apparatus for shooting a moving image at a set frame rate comprises: an image pickup device which outputs an image signal corresponding to electric charges accumulated by performing photoelectric conversion on light incident via an image formation optical system; acquisition means which acquires a driving speed of a focus lens included in the image formation optical system; control means which sets a driving amount of the focus lens at the driving speed, an electric charge accumulation time of the image pickup device, the frame rate and the number of a plurality of focal positions so as to acquire an image signal from the image pickup device at each of the plurality of focal positions in each frame period at the frame rate; and combining means which converts depth of field by combining the image signal obtained from the image pickup device at each of the plurality of focal positions in each frame period.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an electronic device and a control method thereof, and more particularly to a technique for capturing and combining multiple images with different focus positions in an electronic device equipped with an image sensor. [Background technology]

[0002] A technique known as depth stacking has been known for some time. This technique involves capturing multiple images while changing the focus position along the optical axis (focus bracketing), extracting the in-focus area from each image, and then compositing an image with an extended depth of field. In this type of depth stacking technique, for example in macro photography, images captured while moving the focus position to cover the depth of the subject are combined, thereby expanding the depth of field while blurring the background and bringing the entire subject into focus.

[0003] In such focus stacking, Patent Document 1 describes a configuration in which, in order to confirm the effect of focus stacking before capturing an image for recording, fewer images are captured than the number of images to be composited during recording, thereby shortening the composite processing time and making it easier to obtain a confirmation image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-231058 Summary of the Invention [Problem to be solved by the invention]

[0005] Here, when the subject of macro photography is a moving subject such as an insect, there may be a desire to record it as a moving image.

[0006] However, the conventional technology disclosed in Patent Document 1 is basically premised on capturing still images, and does not include any specific description of video generation. Therefore, if the technique in Patent Document 1 is directly used to generate video, the resulting video will have a low frame rate and will not look as good as a video.

[0007] The present invention has been made in consideration of the above problems, and has an object to shoot moving images with focus stacking while suppressing a decrease in frame rate. [Means for solving the problem]

[0008] In order to achieve the above object, an electronic device of the present invention is an electronic device that shoots moving images at a set frame rate, and includes an image sensor that outputs an image signal corresponding to accumulated charges obtained by photoelectrically converting incident light via an imaging optical system, an acquisition means that acquires a drive speed of a focus lens included in the imaging optical system, a control means that sets a drive amount of the focus lens at the drive speed, a charge accumulation time of the image sensor, the frame rate, and the number of the plurality of focus positions so as to acquire an image signal from the image sensor at each of the plurality of focus positions during each frame period at the frame rate, and a synthesis means that synthesizes the image signals obtained from the image sensor at each of the plurality of focus positions for each frame period to convert a depth of field. The control means controls the accumulation of charge in the image sensor and the driving of the focus lens to be performed at different timings, and when the frame rate is F, the number of the plurality of focus positions is N, the drive amount of the focus lens at the drive speed is P, and the charge accumulation time is T, the control means (T+P)×N≦1 / F Controlled to satisfy do. [Effects of the Invention]

[0009] According to the present invention, it is possible to shoot a moving image with focus stacking while suppressing a decrease in frame rate. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the basic configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 2]6 is a flowchart of a process for capturing a depth stacking video according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of macro photography in the first embodiment. [Figure 4] 5A to 5C are diagrams showing an example of timing for capturing and generating a depth stacking video according to the first embodiment. [Figure 5] 10A and 10B are diagrams showing an example of timing for capturing and generating a depth stacking video according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] First Embodiment 1 is a block diagram showing the basic configuration of an image capturing device 100 according to the first embodiment. The image capturing device 100 may be, for example, a camera such as a digital camera or a digital video camera, or may be an electronic device with a camera function such as a camera-equipped mobile phone or a camera-equipped computer.

[0013] The optical system 101 is an imaging optical system equipped with a group of lenses, a shutter, an aperture, etc. The group of lenses includes a correction lens that corrects camera shake and the like, a focus lens, etc. The optical system 101 forms an image of subject light on the image sensor 102 in accordance with a control signal from the CPU 103. The image sensor 102 is an imaging device such as a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor, and converts light incident via the optical system 101 into an image signal by photoelectric conversion and outputs the image signal.

[0014] The CPU 103 executes a program stored in advance in the memory to control each component of the imaging device 100 in accordance with input signals and the like.

[0015] The primary storage device 104 is a volatile storage device such as a RAM (Random Access Memory), which stores temporary data and is used as a work memory for the CPU 103. Information stored in the primary storage device 104 is used by the image processing unit 105 and recorded on a recording medium 106. The secondary storage device 107 is a non-volatile storage device such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The secondary storage device 107 stores programs (firmware) for controlling the imaging device 100 and various setting information, and is used by the CPU 103.

[0016] The recording medium 106 records image data obtained by capturing images and stored in the primary storage device 104. The recording medium 106 may be a recording medium that is removable from the imaging device 100, such as a semiconductor memory card. The data recorded on the recording medium 106 can be read by an external device such as a PC (personal computer) by attaching the recording medium 106 to the external device. In other words, the imaging device 100 has an attachment / detachment mechanism and a read / write function for the recording medium 106.

[0017] The display unit 108 displays a viewfinder image during shooting, a shot image, a GUI (Graphical User Interface) image for interactive operation, etc. The operation unit 109 is a group of input devices that accept user operations and transmit input information to the CPU 103, and includes, for example, buttons, levers, a touch panel, etc. Input devices that use voice, line of sight, etc. can also be used for operation.

[0018] The imaging device 100 has a plurality of image processing patterns that the image processing unit 105 applies to images captured, and these patterns can be set as shooting modes by the operation unit 109. The image processing unit 105 performs various types of processing, such as image processing known as development processing, as well as color tone adjustment according to the shooting mode. Note that at least some of the functions of the image processing unit 105 may be realized by the CPU 103 through software processing.

[0019] 2 is a flowchart of the process of capturing a depth stacking moving image in this embodiment. Before describing the process of capturing a depth stacking moving image with reference to this flowchart, the characteristics of the subject and shooting conditions to be captured as the depth stacking moving image will first be described with reference to FIG.

[0020] 3(a) shows a screen image of an insect, which is a subject 301, moving on a plant 302, and FIG. 3(b) shows a situation in which the subject 301 is being photographed using the imaging device 100. Typically, in macro photography, photographing is performed to blur the background and emphasize the subject 301 by adjusting the photographing conditions, such as the aperture and shutter speed, so that the depth of field is shallow, for example, as indicated by arrow 303. If the subject 301 moves along a trajectory indicated by arrow 306, it will fall outside the depth of field 303, resulting in a blurred image of the subject 301.

[0021] In such a case, it is possible to track subject 301 by, for example, moving the focus position to a position where the depth of field is as indicated by arrow 304. However, if the subject is moving as indicated by trajectory 306 to trajectory 307, it is necessary to frequently move the focus position in accordance with the movement of the subject, which will change the way the surroundings are captured (how the background is blurred). It is also possible to widen the depth of field to the depth indicated by arrow 305 by narrowing the aperture, for example, but in that case the background will not be blurred as much, which may result in an image that is not satisfactory for macro photography.

[0022] In such cases, by applying depth stacking technology and performing depth stacking using, for example, an image captured at depth of field 303 and an image captured at depth of field 304, it is possible to virtually capture subject 301 at depth of field 305 while generating an image with the same background blur as when depth of field is 303 or 304.

[0023] In the depth stacking video of this embodiment, images are captured alternately at two depths of field 303 and 304, and one video frame is generated from the two images obtained by repeating this process to generate the depth stacking video. Note that while Fig. 3 shows an example in which two images are combined, the number of images to be combined is not limited to two, and it is also possible to combine three or more images.

[0024] To generate such a depth stacking moving image, first, in S201, CPU 103 sets the shooting conditions necessary for capturing images to be used in generating the depth stacking moving image. This setting method will be described using the example shown in FIG.

[0025] 4 shows an example of the focus drive time 401, charge accumulation time 402 of the image sensor 102, time 403 for reading out an image signal corresponding to the accumulated charge, development processing time 404 for the read-out image signal, and focus stacking processing time 405 required to drive the optical system 101 to move from the depth of field 303 to the depth of field 304 shown in FIG. 3. When generating a depth stacking video, this series of processes must be contained within each frame period. Under these conditions, the following formula (1) must be satisfied, where T (sec) is the charge accumulation time, P (sec) is the focus drive time, F (fps) is the frame rate, and N is the number of images used for focus stacking.

[0026] (T+P)×N≦1 / F …(1) 4, N is 2 (frames), but can also be a value of 3 or more if the conditions are met. Note that increasing the number of frames used for focus stacking may increase the time required for the synthesis process, so the number of frames may be limited so that the synthesis process time 405 fits within each frame period. Alternatively, the read image signals can be written directly to the recording medium 106, etc., and development and synthesis processes can be performed as post-processing.

[0027] To give a specific example of the condition determination shown in equation (1), if you want to shoot a video at a frame rate of 30 fps, and the charge accumulation time T is 1 / 250 sec, and the number of combined images is 2, the focus drive time P is P≦(1 / F) / NT=(1 / 30) / 2-(1 / 250) ≒12.6 msec …(2) Here, the focus drive time P increases or decreases depending on, for example, the lens performance and the drive amount of the focus lens. For example, when an image is acquired in a positional relationship where the depth of field is adjacent, such as the depth of field 303 and 304 shown in FIG. 3, if the drive amount of the focus lens required to move the shooting distance of the depth of field 303 is D (mm) and the drive speed of the focus lens in the lens used is V (mm / sec), the focus drive time P is P=D / V …(3) The depth of field may be calculated from the focal length, aperture value, shooting distance, etc. of the lens, or the drive amount may be determined in advance according to the state of the lens.

[0028] In equation (3), for example, if the drive speed V is 100 mm / sec, the focus drive time P = 10 msec is required to move the drive amount D by 1 mm. In such a case, since the focus drive time P satisfies equation (2), it is possible to set the maximum value (maximum amount) of the drive amount D to, for example, 1 mm. In contrast, for example, if the drive speed V is 75 mm / sec, the focus drive time P ≈ 13 msec is required to move the drive amount D by 1 mm, and the focus drive time P does not satisfy equation (2). In such a case, it is possible to satisfy equation (2) by limiting the maximum value (maximum amount) of the drive amount D to, for example, about 0.7 mm.

[0029] In this way, in S201, the shooting conditions required for shooting a depth stacking video, such as the frame rate, aperture value, charge accumulation time, focus lens drive amount, and number of images to be synthesized, are set prior to shooting, and the process proceeds to S202.

[0030] In S202, the CPU 103 controls the optical system 101 and the image sensor 102 based on the set shooting conditions to accumulate charges and read out image signals for the first image. Then, in S203, the CPU 103 controls the image processing unit 105 to perform development processing on the read-out image signals.

[0031] Furthermore, in S204, the CPU 103 drives the focus lens of the optical system 101 based on the drive amount set in S201 to change the focal position. Note that these processes can also be performed partially in parallel as shown in FIG.

[0032] Once the focal position has been changed, in steps S205 to S207, the same processes as in steps S202 to S204 are performed to capture a second image. However, in step S207, the focus lens is driven in the direction opposite to the direction in which it was driven in step S204 by the drive amount set in step S201. In this way, once an image with a depth of field of 303 is obtained in steps S202 and S203, and an image with a depth of field of 304 is obtained in steps S205 and S206, CPU 103 performs focus stacking in step S208. Note that the focus stacking process can be performed using a known method, and therefore a description thereof will be omitted here.

[0033] A single frame image of a moving image is generated through a series of processes from S202 to S208. The CPU 103 can generate a moving image with focus stacking by repeating the processes from S202 until it is determined in S209 that recording has ended.

[0034] As described above, according to the first embodiment, it is possible to capture a depth stacking video while suppressing a decrease in frame rate.

[0035] <Second embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment, an example was described in which the charge accumulation time and the focus lens drive time are set at different timings so as not to overlap. However, it is also possible to perform imaging in which the focus lens is driven while charge is being accumulated. How to determine the drive amount in such a case will be explained using FIG. 5. Note that the imaging device in the second embodiment can be the one explained with reference to FIG. 1 in the first embodiment, and therefore explanation thereof will be omitted here.

[0036] 5 differs from FIG. 4 in that accumulation time 502 and focus drive time 501 overlap. In such a case, to capture a depth stacking video, the following equation (4) must be satisfied, where T (sec) is the charge accumulation time, P (sec) is the focus drive time, F (fps) is the frame rate, and N is the number of images used for depth stacking.

[0037] MAX(T, P)×N≦1 / F …(4) (MAX(X, Y) represents the larger of X and Y) Since MAX(T, P) is smaller than T+P in equation (1), the condition in equation (4) is more lenient. For example, in the first embodiment, when the drive speed V is 75 mm / sec, the focus drive time P≈13 msec is required to move the drive amount D by 1 mm, so equation (2) could not be satisfied. In contrast, in the example shown in FIG. 5, P×N=0.013×2≦1 / F=(1 / 30)=0.033 …(5) This satisfies equation (4), and therefore the drive amount D can be set to 1 mm.

[0038] Furthermore, if the focus speed V is 100 mm / sec, then P = 10 msec, P×N=0.010×3≦1 / F=(1 / 30)=0.033 …(6) It is also possible to set the number of composite images N to 3, as shown above.

[0039] However, because charge is accumulated while the focus lens is being driven, the image quality tends to be slightly inferior to that of the first embodiment. For this reason, for example, the first embodiment may be set to "image quality priority" and the second embodiment may be set to "frame rate priority," allowing the user to select between them.

[0040] As described above, according to the second embodiment, by driving the focus lens in parallel with charge accumulation, it becomes possible to capture a focus stacking moving image more flexibly.

[0041] <Third embodiment> Next, a third embodiment of the present invention will be described. In the first and second embodiments described above, an example of determining the drive amount of the focus lens has been described, but the present invention is not necessarily limited to this, and other elements can also be determined by calculation.

[0042] In the first embodiment, an example was shown in which the drive amount was limited due to insufficient drive speed of the focus lens, but instead of limiting the drive amount, the settable range of the accumulation time may be limited. For example, when the drive speed V is 75 mm / sec and the drive amount D is 1 mm, the focus drive time P is approximately 13 msec, so equation (2) cannot be satisfied. In such a case, the accumulation time T is set to a time shorter than 1 / 400. This allows P≦(1 / F) / NT=(1 / 30) / 2-(1 / 400) ≒14.2 msec …(7) As a result, the condition can be satisfied. Similarly, the condition can also be satisfied by limiting the frame rate F.

[0043] As described above, according to the third embodiment, by adjusting the charge accumulation time and / or frame rate based on the recording frame rate and lens information, it is possible to capture a depth stacking video while suppressing a decrease in frame rate.

[0044] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0045] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0046] 100...imaging device, 101...optical system, 102...imaging element, 103...CPU, 104...primary storage device, 105...image processing unit, 106...recording medium, 107...secondary storage device, 108...display unit, 109...operation unit

Claims

1. An electronic device that captures video at a set frame rate, an imaging element that photoelectrically converts incident light via an imaging optical system and outputs an image signal corresponding to accumulated charges; an acquisition unit for acquiring a drive speed of a focus lens included in the imaging optical system; a control means for setting a drive amount of the focus lens at the drive speed, a charge accumulation time of the image sensor, the frame rate, and the number of the plurality of focus positions so that an image signal is acquired from the image sensor at each of the plurality of focus positions during each frame period at the frame rate; a combining unit that combines image signals obtained from the image pickup device at each of the plurality of focal positions for each frame period to convert a depth of field, the control means controls the accumulation of charge in the image sensor and the driving of the focus lens so as to be performed at different times; When the frame rate is F, the number of the plurality of focal positions is N, the drive amount of the focus lens at the drive speed is P, and the charge accumulation time is T, the control unit (T+P)×N≦1 / F Control to satisfy An electronic device characterized by:

2. An electronic device that shoots video at a set frame rate, an imaging element that photoelectrically converts incident light via an imaging optical system and outputs an image signal corresponding to accumulated charges; an acquisition unit for acquiring a drive speed of a focus lens included in the imaging optical system; a control means for setting a drive amount of the focus lens at the drive speed, a charge accumulation time of the image sensor, the frame rate, and the number of the plurality of focus positions so that an image signal is acquired from the image sensor at each of the plurality of focus positions during each frame period at the frame rate; a combining unit that combines image signals obtained from the image pickup device at each of the plurality of focal positions for each frame period to convert a depth of field, the control means performs accumulation of charges in the image sensor and driving of the focus lens in parallel; When the frame rate is F, the number of the plurality of focal positions is N, the drive amount of the focus lens at the drive speed is P, and the charge accumulation time is T, the control unit MAX(T,P)×N≦1 / F (However, MAX(T, P) is the larger of T and P.) Control to satisfy An electronic device characterized by:

3. 3. The electronic device according to claim 1, wherein the control means limits the driving amount so that an image signal can be acquired from the image sensor at each of the plurality of focal positions during each frame period.

4. 3. The electronic device according to claim 1, wherein the control means limits the number of the plurality of focus positions so that the time required for the composition process by the composition means falls within each frame period.

5. 3. The electronic device according to claim 1, wherein the control means limits the charge accumulation time so that an image signal can be acquired from the image sensor at each of the plurality of focal positions during each frame period.

6. 3. The electronic device according to claim 1, wherein the control means limits the frame rate so that an image signal can be acquired from the image sensor at each of the plurality of focal positions during each frame period.

7. 7. The electronic device according to claim 1, wherein the driving amount is an amount by which the focus lens is driven to a focal position where the depths of field do not overlap, or a maximum amount by which the focus lens can be driven at the driving speed in the time allocated to driving the focus lens in each frame period.

8. 8. The electronic device according to claim 1, further comprising a recording unit for recording the image signal synthesized by the synthesizing unit.

9. A control method for an electronic device that shoots video at a set frame rate, comprising: an imaging step of outputting an image signal corresponding to electric charges accumulated by photoelectrically converting incident light via an imaging optical system using an imaging element; an acquisition step of acquiring a drive speed of a focus lens included in the imaging optical system by an acquisition means; a setting step in which, prior to the imaging step, a control unit sets a drive amount of the focus lens at the drive speed, a charge accumulation time of the image sensor, the frame rate, and the number of the plurality of focus positions so that an image signal is acquired from the image sensor at each of the plurality of focus positions during each frame period at the frame rate; a combining step in which the combining means combines image signals obtained from the image pickup device at each of the plurality of focus positions for each frame period to convert the depth of field. The accumulation of charge in the image pickup element in the imaging step and the driving of the focus lens are controlled to be performed at different timings, When the frame rate is F, the number of the plurality of focal positions is N, the driving amount of the focus lens at the driving speed is P, and the charge accumulation time is T, in the setting step, (T+P)×N≦1 / F Set to satisfy A method for controlling an electronic device.

10. A control method for an electronic device that shoots video at a set frame rate, comprising: an imaging step of outputting an image signal corresponding to electric charges accumulated by photoelectrically converting incident light via an imaging optical system using an imaging element; an acquisition step of acquiring a drive speed of a focus lens included in the imaging optical system by an acquisition means; a setting step in which, prior to the imaging step, a control unit sets a drive amount of the focus lens at the drive speed, a charge accumulation time of the image sensor, the frame rate, and the number of the plurality of focus positions so that an image signal is acquired from the image sensor at each of the plurality of focus positions during each frame period at the frame rate; a combining step in which a combining means combines image signals obtained from the image pickup device at each of the plurality of focus positions for each frame period to convert a depth of field, In the imaging step, charge accumulation in the imaging element and driving of the focus lens are performed in parallel, When the frame rate is F, the number of the plurality of focal positions is N, the driving amount of the focus lens at the driving speed is P, and the charge accumulation time is T, in the setting step, MAX(T,P)×N≦1 / F (However, MAX(T, P) is the larger of T and P.) Set to satisfy A method for controlling an electronic device.

11. A program for causing a computer to execute each step of the control method according to claim 9 or 10.

12. A computer-readable storage medium storing the program according to claim 11.

Images