Image capture device and control method thereof

Abstract

To pick up, display, and record images at frame rates suitable for the imaging, display, and recording, and obtain images suitable for display and recording.SOLUTION: An imaging apparatus can set an imaging frame rate for driving an image pick-up device, a display frame rate for display means, and a recording frame rate for recording means to values different from each other, based on a set electric charge storage time and the imaging frame rate, and controls the electric charge storage time in frames of the imaging frame rate to drive the image pick-up device. Based on the imaging frame rate, the display frame rate, and the electric charge storage time, the imaging apparatus creates, from an image output from the image pick-up device at the imaging frame rate, a display image corresponding to exposure set in the display frame rate and outputs the display image, and creates and outputs a recording image corresponding to exposure set in the recording frame rate.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] Conventionally, in imaging devices, it is common to drive and control the frame rate of a display device (hereinafter referred to as a "display frame rate") and the frame rate at which data is recorded in a memory or the like (hereinafter referred to as a "recording frame rate") in synchronization with a set frame rate at the time of shooting (hereinafter referred to as an "imaging frame rate"). For example, when the imaging frame rate is 30p, the display frame rate and the recording frame rate are also driven and controlled at 30p.

[0003] Meanwhile, in recent years, display devices using organic light-emitting diodes (OLED displays) have been widely used. OLED displays have the characteristic of having a faster response speed for switching between frames than liquid crystal display devices (LCDs) that have been commonly used in the past. With LCDs, which have a slow response speed for switching between frames, when shooting at a low imaging frame rate (e.g., 30p) while moving the angle of view by panning, etc., blurring of the captured image is recognized as an afterimage when displayed, and it is perceived as if interpolation is being performed between frames. This reduces the deterioration of visibility. However, with OLED displays, the response speed for switching between frames is fast, so when the image is displayed at the same display frame rate as the imaging frame rate, it is recognized as blurring as it is, resulting in a problem of worse visibility compared to LCDs.

[0004] In order to improve the visibility of an OLED display, it is conceivable to speed up the display update cycle by setting a high display frame rate capable of driving the OLED display, and to speed up the image capture frame rate in accordance with the display frame rate. However, when the image capture frame rate becomes faster than the recording frame rate capable of being recorded, it is not possible to record all captured frame images, as shown in FIG.

[0005] Patent Document 1 describes the following regarding the imaging frame rate, display frame rate, and recording frame rate. That is, when the imaging frame rate is faster than the display frame rate and the recording frame rate, the captured images of each frame are temporarily written to memory. Then, for display, the frame rate is converted by thinning out and reading the images from the memory at the display frame rate and displaying them. Also, for recording, the read frame rate of the captured images from the memory is controlled to the recording frame rate, so that the captured images are recorded without thinning out.

[0006] Patent Document 2 also discloses that when shooting at a high frame rate and performing long exposure live view at a low frame rate, the number of shot images necessary to achieve appropriate brightness are sequentially synthesized and displayed according to the frame rate of the display device. It also discloses that when recording moving images, similarly to live view, the number of shot images necessary to achieve appropriate brightness are sequentially synthesized and the synthesized images are sequentially recorded. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 4424174 [Patent Document 2] JP 2020-10140 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, the method described in Patent Document 1 has a problem in that if the subject is dark and sufficient charge accumulation time (exposure time) cannot be ensured in each frame, the obtained image will be dark.

[0009] Furthermore, the method described in Patent Document 2 is premised on the fact that the display frame rate and the recording frame rate are the same. Therefore, when a display device capable of updating the display at a high display frame rate such as an OLED display is used, the recording may not keep up, or if the display frame rate is matched to the recording frame rate, improvement in visibility due to blurring cannot be expected.

[0010] The present invention has been made in consideration of the above problems, and has an object to capture, display and record images at frame rates suitable for each, and to obtain images suitable for both display and recording. [Means for solving the problem]

[0011] In order to achieve the above object, an imaging device according to the present invention comprises: 1. Charge Storage Time and exposure and a control means for controlling an imaging frame rate for driving the imaging element, a display frame rate for the display means, and a recording frame rate for the recording means; 1. Based on the charge accumulation time and the imaging frame rate, Second a driving means for controlling a charge accumulation time to drive the imaging element; When the display frame rate is equal to or lower than the imaging frame rate, the imaging frame rate, the display frame rate, 1. Charge storage time The second charge accumulation time a first generating means for generating a display image corresponding to the exposure set by the setting means at the display frame rate from an image output from the imaging element at the imaging frame rate based on the above-mentioned, and outputting the display image; When the recording frame rate is equal to or lower than the imaging frame rate, The imaging frame rate, the recording frame rate, and 1. Charge storage time The second charge accumulation time and a second generating means for generating a recording image corresponding to the exposure set by the setting means at the recording frame rate from the image output from the imaging element at the imaging frame rate based on the above. do It is characterized by: Effect of the Invention

[0012] According to the present invention, images can be captured, displayed, and recorded at frame rates suited to each, and images suited to each of display and recording can be obtained. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a conceptual diagram for explaining accumulation control in the first embodiment. [Diagram 3] FIG. 4 is a conceptual diagram for explaining synthesis control in the first embodiment. [Figure 4] 5 is a flowchart showing synthesis control in the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing a schematic configuration of an imaging apparatus according to a second embodiment. [Figure 6] FIG. 11 is a conceptual diagram for explaining accumulation control in the second embodiment. [Figure 7] FIG. 11 is a conceptual diagram for explaining synthesis control in the second embodiment. [Figure 8] 10 is a flowchart showing synthesis control in the second embodiment. [Figure 9] FIG. 13 is a conceptual diagram for explaining accumulation control in a modified example of the second embodiment. [Figure 10] FIG. 13 is a conceptual diagram for explaining gain control and synthesis control in a modified example of the second embodiment. [Figure 11] 10 is a flowchart showing gain control and synthesis control in a modified example of the second embodiment. [Figure 12] FIG. 13 is a conceptual diagram showing timings of imaging, display, and recording for explaining a conventional example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0015] <First embodiment> A first embodiment of the present invention will now be described. Fig. 1 is a block diagram showing an example of a schematic configuration of an imaging device 100 according to a first embodiment. As shown in Fig. 1, the imaging device 100 shows the configuration of an electronic device equipped with a camera, such as a so-called digital still camera, a digital video camera, or a mobile terminal with a camera.

[0016] The amount of incident light entering from the outside through lens 101 is controlled by light attenuation section 102 and enters imaging section 103. In Fig. 1, lens 101 is shown as a single lens, but is usually made up of multiple lenses. Light attenuation section 102 is made up of aperture blades, an ND (Neutral Density) filter, etc.

[0017] The imaging unit 103 is composed of an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD, a driver for driving the imaging element, a timing generation circuit, a CDS / AGC circuit, an A / D converter, and the like. The driver and timing generation circuit supply driving pulses and the like for driving the imaging element to the imaging element, and read out the image captured by the imaging element and adjust the exposure time. The imaging element captures an image of a subject by photoelectrically converting incident light whose amount is controlled by the light reduction unit 102, and the CDS / AGC circuit samples and amplifies image information based on the electric charge (image signal) stored in each pixel of the imaging element. Note that correlated double sampling (CDS) is performed for sampling, and automatic gain control (AGC) is performed for amplification. The A / D converter converts the image signal (analog signal) output from the CDS / AGC circuit into a digital signal and outputs it.

[0018] The image processing unit 104 performs various signal processing such as AWB (Auto White Balance) and gamma control on the image signal (digital signal) output from the A / D converter, and develops it by adjusting the color, brightness, etc., and outputs the generated image.

[0019] The microcomputer 105 is a circuit also called a microcontroller, and performs overall control of the operations of the imaging device 100, including the above-mentioned controls.

[0020] The imaging device 100 of this embodiment also has a frame rate switching control unit 106, which determines switching of the imaging frame rate, the display frame rate, and the recording frame rate. If switching is necessary, the microcomputer 105 is notified of the switched frame rate. The microcomputer 105 controls the accumulation control unit 107, the first synthesis control unit 108, the second synthesis control unit 109, and the third synthesis control unit 112 in response to an instruction from the frame rate switching control unit 106.

[0021] The accumulation control unit 107 calculates the charge accumulation time for each frame of the imaging unit 103 according to the imaging frame rate, and notifies the imaging unit 103 of the calculated charge accumulation time.

[0022] The display unit 110 displays the display image output from the first synthesis control unit 108 at the display frame rate of the display unit 110, and the external display unit 210 displays the display image output from the third synthesis control unit 112 at the display frame rate of the external display unit 210. In addition, the recording unit 111 records the display image output from the second synthesis control unit 109 at the recording frame rate of the recording unit 111.

[0023] Next, the control of the charge accumulation time in the accumulation control unit 107 and the specific operations in the first synthesis control unit 108, the second synthesis control unit 109, and the third synthesis control unit 112 will be described with reference to FIG. 2 and FIG.

[0024] For example, when the image capture frame rate is 30p and the charge accumulation time is 1 / 80 seconds, if the image capture frame rate is changed to 60p, a charge accumulation time of 1 / 80 seconds can be ensured for each frame. Therefore, the accumulation control unit 107 sets the charge accumulation time for each frame to 1 / 80 seconds (FIG. 2(a)).

[0025] Also, when the image capture frame rate is 30p and the charge accumulation time is 1 / 40 seconds, if the image capture frame rate is changed to 60p, it is not possible to take a charge accumulation time of 1 / 40 seconds for each frame. In such a case, the accumulation control unit 107 controls the charge accumulation time of each frame to 1 / 80 seconds, which is half of the charge accumulation time of 1 / 40 seconds, so that the charge accumulation time for two frames is 1 / 40 seconds (FIG. 2(b)). In this way, when one frame period (cycle) of the image capture frame rate after the change is shorter than the charge accumulation time, the charge accumulation time is divided and allocated to multiple frames. Note that, in this allocation, the charge accumulation time is allocated to multiple frames in the number corresponding to one frame period of the image capture frame rate before the change.

[0026] The first synthesis control unit 108 synthesizes a required number of images as necessary based on the imaging frame rate notified by the microcomputer 105, the display frame rate of the display unit 110, and the charge accumulation time set by photometry or a user instruction to generate a display image. Then, the first synthesis control unit 108 outputs the obtained display image to the display unit 110 at the display frame rate.

[0027] For example, when the imaging frame rate is 30p, the charge accumulation time is 1 / 80 seconds (corresponding to the time before the frame rate in FIG. 2(a) is changed), and the display frame rate of the display unit 110 is 30p, the first synthesis control unit 108 displays the image output in each frame from the image processing unit 104 as is on the display unit 110. Note that the same display is performed when the charge accumulation time is 1 / 40 seconds (corresponding to the time before the frame rate in FIG. 2(b) is changed).

[0028] Also, when the imaging frame rate is 60p, the charge accumulation time is 1 / 80 seconds (corresponding to the frame rate after change in FIG. 2(a)), and the display frame rate of the display unit 110 is 60p, the first synthesis control unit 108 displays the image output from the image processing unit 104 for each frame as is on the display unit 110 (FIG. 3(a)). On the other hand, when the imaging frame rate is 60p, the charge accumulation time is 1 / 40 seconds (corresponding to the frame rate after change in FIG. 2(b)), and the display frame rate of the display unit 110 is 60p, the first synthesis control unit 108 sequentially synthesizes images two frames at a time while shifting, and displays the display image obtained by synthesizing on the display unit 110 (FIG. 3(b)). This makes the total charge accumulation time of each display image 1 / 40 seconds.

[0029] The second synthesis control unit 109 synthesizes a required number of images as necessary based on the imaging frame rate notified by the microcomputer 105, the recording frame rate of the recording unit 111, and the charge accumulation time set by photometry or a user instruction to generate a recording image. Then, the obtained display image is output to the recording unit 111 at the recording frame rate.

[0030] For example, when the imaging frame rate is 30p, the charge accumulation time is 1 / 80 seconds (corresponding to the time before the frame rate is changed in FIG. 2(a)), and the recording frame rate of the recording unit 111 is 30p, the second synthesis control unit 109 outputs the images output in each frame from the image processing unit 104 as is to the recording unit 111. Note that the same output is performed when the charge accumulation time is 1 / 40 seconds (corresponding to the time before the frame rate is changed in FIG. 2(b)).

[0031] Furthermore, when the imaging frame rate is 60p, the charge accumulation time is 1 / 80 seconds (corresponding to the frame rate after change in FIG. 2(a)), and the recording frame rate of the recording unit 111 is 30p, the images output from the image processing unit 104 for each frame are thinned out and recorded by the recording unit 111 (FIG. 3(a)). On the other hand, when the imaging frame rate is 60p, the charge accumulation time is 1 / 40 seconds (corresponding to the frame rate after change in FIG. 2(b)), and the recording frame rate of the recording unit 111 is 30p, the second synthesis control unit 109 synthesizes images two frames at a time, and outputs the recorded image obtained by synthesis to the recording unit 111 (FIG. 3(b)). This makes the total charge accumulation time of each recorded image 1 / 40 seconds.

[0032] The third synthesis control unit 112 synthesizes a required number of images as necessary based on the imaging frame rate notified from the microcomputer 105, the display frame rate of the external display unit 210, and the charge accumulation time set by photometry or a user instruction. Then, the obtained display images are output to the external display unit 210 at the display frame rate. Note that the processing in the third synthesis control unit 112 is similar to the processing in the first synthesis control unit 108, and therefore a detailed description thereof will be omitted.

[0033] Next, the control of the frame rate, charge accumulation time, and image synthesis in the imaging device 100 of this embodiment will be described with reference to the flowchart in Fig. 4. Here, the control when the external display unit 210 is connected to the imaging device 100 will be described, and it is assumed that the default imaging frame rate, display frame rate, and recording frame rate are the same when the external display unit 210 is not connected. It is also assumed that the recording frame rate cannot be changed from the default frame rate.

[0034] When the external display unit 210 is connected to the imaging device 100, in S101, it is determined whether the responsiveness of the external display unit 210 in switching between frames is fast. If it is determined that the responsiveness of the external display unit 210 in switching between frames is not fast, the process proceeds to S104, and images are displayed without changing either the imaging frame rate or the display frame rate. In this case, images captured at a default imaging frame rate are output to the external display unit 210 via the third synthesis control unit 112. Note that, if the display frame rate of the external display unit 210 is slower than the default imaging frame rate, the third synthesis control unit 112 may thin out the images before outputting them.

[0035] In the next step S105, the recording unit 111 records images at a default recording frame rate.

[0036] On the other hand, if it is determined in S101 that the response of switching between frames on the external display unit 210 is fast, then in S102, the imaging device 100 determines whether or not the imaging element of the imaging unit 103 can be driven at the same imaging frame rate as the high-speed display frame rate of the external display unit 210. If it is determined that the imaging element cannot be driven at the same frame rate as the display frame rate of the external display unit 210, the imaging frame rate is not changed. Then, in S104, images are displayed at the default display frame rate, and in S105, images are recorded at the default recording frame rate. In this case, the image obtained at the default imaging frame rate is also output to the third synthesis control unit 112.

[0037] If it is determined in S102 that the imaging device 100 can drive the imaging element of the imaging section 103 at the same imaging frame rate as the high-speed display frame rate of the external display section 210, then in S103 the imaging frame rate is changed to match the high-speed display frame rate of the external display section 210. Then, in S106, the charge accumulation time set in the imaging device 100 is compared with one frame period (cycle) of the imaging frame rate after the change. If the one frame period is equal to or greater than the charge accumulation time, the process proceeds to S107, and if the one frame period is less than the charge accumulation time, the process proceeds to S109.

[0038] In S107, the display frame rate of the display unit 110 is set to the changed image capture frame rate. This allows the display unit 110 to quickly switch the images to be displayed at the same cycle as the high-speed image capture frame rate. In addition, in S108, the frames of the captured images are thinned out to achieve the default recording frame rate, and recorded in the recording unit 111 (control in FIG. 3(a)). Note that, if the display frame rate of the display unit 110 cannot be increased to the image capture frame rate, the display frame rate of the display unit 110 is switched to a higher rate within the possible range. Then, the frames of the captured images are thinned out and displayed to achieve the changed display frame rate.

[0039] On the other hand, in S109, as described with reference to FIG. 2(b), the charge accumulation time is divided into a plurality of frames and allocated to the plurality of frames. Then, in S110, the display frame rate of the display unit 110 is set to the changed imaging frame rate. Also, the first synthesis control unit 108 and the third synthesis control unit 112 are controlled to synthesize images of a plurality of frames while sequentially shifting as shown in FIG. 3(b), and output the images to the display unit 110 and the external display unit 210 at the display frame rate. This allows the display unit 110 and the external display unit 210 to display images of the required charge accumulation time at a high display frame rate. Note that, when the display frame rate of the display unit 110 cannot be increased to the imaging frame rate, the display frame rate of the display unit 110 is switched to be higher within the possible range. Then, the images of a plurality of frames are added while shifting and output at the display frame rate.

[0040] Next, in S111, the second synthesis control unit 109 synthesizes images of the number of frames according to the charge accumulation time, as shown in Fig. 3(b), and outputs the images at the default recording frame rate to the recording unit 111. This makes it possible to record images at the recording frame rate with the same exposure as before the imaging frame rate was changed.

[0041] 4, the process proceeds to S102 on the condition that it is determined in S101 that the responsiveness of switching between frames on the external display unit 210 is fast, but the present invention is not limited to this. For example, the process may proceed to S102 on the condition that the display frame rate of the set external display unit 210 or the display unit 110 is fast.

[0042] As described above, according to the first embodiment, imaging, display, and recording can be driven at frame rates suitable for each device, and images with equivalent exposure can be displayed and recorded before and after changing the imaging frame rate.

[0043] <Second embodiment> Next, a second embodiment of the present invention will be described. Fig. 5 is a block diagram showing an example of a schematic configuration of an image pickup device 200 according to an embodiment of the present invention. In Fig. 5, the same components as those in the image pickup device 100 shown in Fig. 1 are given the same reference numerals, and description thereof will be omitted. The difference from the configuration shown in Fig. 1 is that a first gain control unit 208 is provided instead of the first synthesis control unit 108, and a second gain control unit 209 is provided instead of the third synthesis control unit 112.

[0044] In the first embodiment described above, when one frame period (cycle) of the imaging frame rate is shorter than the charge accumulation time, the charge accumulation time is distributed so that charge accumulation is performed at equal timing in each frame. FIG. 6(a) shows the charge accumulation control in the first embodiment. Hereinafter, the charge accumulation time allocated to multiple frames is called "split accumulation time" to be distinguished from the charge accumulation time set by photometry or user instruction. Then, when generating a display image, images for multiple frames are sequentially shifted while being synthesized, so that the charge accumulation time of each display image is made equal before and after changing the imaging frame rate, and control is performed so that there is no difference in exposure of the display image.

[0045] However, in the above control, the timing at which the image of each frame starts to be read (end of the divided accumulation time) is the end position of each frame. In this case, since the divided accumulation time At is shorter than the length of one frame, a period of time (non-accumulation time) t during which no charge accumulation is performed occurs between the start of the frame and the start of the divided accumulation time. There is a concern that the non-accumulation time t occurring between frames may cause unnatural effects such as misalignment or blurring when photographing a fast-moving subject.

[0046] Therefore, in the second embodiment, the accumulation control unit 107 performs control when allocating charge accumulation time to a plurality of frames so as to eliminate non-accumulation time t between combined frames.

[0047] FIG. 6(b) is a timing chart for explaining accumulation control in the case where one frame period of the changed imaging frame rate is shorter than the charge accumulation time in the second embodiment. As shown in FIG. 6(b), the divided accumulation time of the first frame is started at a timing that is shifted backward so that the end (read-out start) timing of the divided accumulation time of the first frame coincides with the start timing of the divided accumulation time of the second frame. In other words, charge accumulation is started after a time 2t, which is obtained by subtracting the charge accumulation time 2At from the time for two frames, has elapsed from the start of the first frame. In this way, it is possible to prevent the non-accumulation time t between the frames in the charge accumulation method of the first embodiment from being inserted between the first and second frames while performing charge accumulation in the same manner as normal charge accumulation.

[0048] Next, the timing and method of generating a display image and a recording image from the image of each frame obtained by storing and reading out the charge at the timing shown in Fig. 6(b) will be described with reference to Fig. 7(a). Fig. 7(a) shows, as an example, a case where the changed display frame rate and imaging frame rate are 60p, the recording frame rate is 30p, and the charge storage time is longer than one frame period of the display frame rate and shorter than two frame periods.

[0049] As shown in Fig. 7(a), the recording image is composed of two frames at a time and recorded at the recording frame rate. Here, since the recording frame rate is 30p, every two frames are composed by the second composition control unit 109 and recorded by the recording unit 111. In the second embodiment, since charge accumulation and readout are performed as shown in Fig. 6(b), it is possible to reduce discomfort such as misalignment and blur when photographing a fast-moving subject.

[0050] On the other hand, if images are synthesized while shifting for each frame as in the first embodiment, display images will be generated in which the non-accumulation time 2t is sandwiched between the frames in which the images are synthesized, and in which the non-accumulation time 2t is not sandwiched between the frames in which the images are synthesized. This will result in a noticeable sense of incongruity, such as misalignment and blurring, when shooting a fast-moving subject.

[0051] Therefore, in the second embodiment, instead of synthesizing images of a plurality of frames, gain control is performed on the images obtained in each frame, so that the exposure of the image displayed before and after the change of the image capture frame rate does not change. In the example shown in FIG. 7(a), since the display frame rate is 60p, the first gain control unit 208 and the second gain control unit 209 perform gain-up control on the image read out at 60p, and output it to the display unit 110 and the external display unit 210. Note that the first gain control unit 208 and the second gain control unit 209 are switched to drive at the same frame rate as the display frame rate by the instruction of the frame rate switching control unit 106. This makes it possible to display a display image with the same exposure before and after the change of the image capture frame rate for each frame. Note that the gain value applied to the image of each frame may be a value obtained by dividing the charge accumulation time 2At by the divided accumulation time of each frame.

[0052] Also, in FIG. 7(a), the case where the recording frame rate is 30p is shown, but when the recording frame rate is lower (for example, 15p) and three or more frames of images need to be synthesized to maintain the exposure, the control is performed as follows. That is, when the non-accumulation time t is not generated between frames of the imaging frame rate and the readout is assigned to a plurality of frames so as to be performed at the end of the frames of the recording frame rate, for example, it becomes as shown in FIG. 7(b). In this case, too, since the length and timing of the divided accumulation time differ for each frame, the image to be displayed is controlled so that the exposure does not change by performing gain-up control according to the divided accumulation time of each frame on the image read out in each frame. Also, by bringing the charge accumulation time to the rear of the recording frame rate, if a frame to which the divided accumulation time is not assigned occurs, additional charge accumulation is performed to obtain a display image.

[0053] Next, control of the frame rate, charge accumulation time, and image synthesis in the second embodiment will be described with reference to the flowchart in Fig. 8. In the process shown in Fig. 8, the processes in S101 to S108 and S111 are similar to the processes shown in Fig. 4 described in the first embodiment, so the same step numbers are used and descriptions are omitted.

[0054] If it is determined in S106 that one frame period (cycle) of the changed imaging frame rate is shorter than the charge accumulation time, the process proceeds to S209. In S209, the accumulation control unit 107 starts reading out the charges accumulated at the end of each frame of the recording frame rate, and determines the start timing of the charge accumulation time so that no non-accumulation time occurs between frames of the imaging frame rate (FIG. 6(b)).

[0055] Next, in S210, it is determined whether or not there are frames of the imaging frame rate to which the divided accumulation time has not been allocated due to the slow recording frame rate, and if so, charge accumulation for display is performed in S211. This allows the image obtained in each frame to be displayed continuously without interruption even when the recording frame rate is at a lower frame rate.

[0056] Next, in S212, the first gain control unit 208 and the second gain control unit 209 apply gains to the images of each frame according to the divided accumulation time in each frame, and output the images to the display unit 110 and the external display unit 210 (FIG. 7(a)). This allows the display unit 110 and the external display unit 210 to display images with equivalent exposure before and after the change in the imaging frame rate at a high-speed display frame rate.

[0057] Then, in S111, a plurality of images captured for recording in each frame of the recording frame rate are composited and output at the recording frame rate to the recording unit 111. This makes it possible to record images with equivalent exposure before and after the change in the imaging frame rate at the recording frame rate.

[0058] As described above, according to the second embodiment, in addition to the effects of the first embodiment, it is possible to reduce the sense of incongruity in an image when a subject is moving at high speed.

[0059] <Modification> Next, a modification of the second embodiment will be described. In the second embodiment, as shown in FIG. 6(b), the charge accumulation time is controlled to be aligned to the end of each frame of the recording frame rate so that no non-accumulation time t occurs between frames of the imaging frame rate during the charge accumulation time spanning multiple frames.

[0060] In contrast, in this modification, as shown in FIG. 9, the charge accumulation time is divided and allocated to a plurality of frames of the imaging frame rate so that the center of the charge accumulation time coincides with the center of each frame of the recording frame rate.

[0061] Next, the timing and method for generating a display image and a recording image from an image obtained by storing and reading out charges at the timing shown in FIG. 9 will be described with reference to FIG. 10(a).

[0062] As shown in Fig. 10(a), the recording image is composed of two frames at a time and recorded at the recording frame rate. Here, since the imaging frame rate is 60p and the recording frame rate is 30p, every two frames are composed by the second composition control unit 109 and recorded by the recording unit 111. Since charge accumulation and readout are performed as shown in Fig. 9, it is possible to reduce discomfort such as misalignment and blur when photographing a fast-moving subject.

[0063] On the other hand, if the display image is synthesized while shifting it for each frame as in the first embodiment, the timing of the non-accumulation time t differs for each frame, which can result in noticeable misalignment, blurring, and other unnatural effects when photographing a fast-moving subject.

[0064] Therefore, in the modified example, instead of synthesizing images of a plurality of frames, gain control is performed on the images obtained in each frame in the same manner as in the second embodiment, so that the exposure of the image displayed before and after the change of the image capture frame rate does not change. In the example shown in FIG. 10(a), since the display frame rate is 60p, the image read out at 60p is gain-up by the first gain control unit 208 and the second gain control unit 209, and is displayed every frame by the display unit 110 and the external display unit 210. The gain value applied to the image of each frame may be a value obtained by dividing the charge accumulation time 2At by the divided accumulation time in each frame, and in the example shown in FIG. 10(a), the gain value is 2At / At=2.

[0065] Also, in FIG. 10(a), the case where the recording frame rate is 30p is shown, but when the recording frame rate is lower (for example, 15p) and it is necessary to synthesize images of three or more frames to maintain the exposure, the control is performed as follows. That is, when the charge accumulation time is divided and allocated to a plurality of frames of the imaging frame rate so that no non-accumulation time t occurs between frames of the imaging frame rate, it becomes, for example, as shown in FIG. 10(b). In this case, too, the length and timing of the divided accumulation time differ depending on the frame, so that the image to be displayed is controlled so that no change in exposure occurs by performing gain-up control according to the divided accumulation time of each frame on the image read out in each frame. Also, by allocating the charge accumulation time so that the center of the charge accumulation time coincides with the center of the recording frame rate, if a frame to which the divided accumulation time is not allocated occurs, additional charge accumulation is performed to obtain a display image.

[0066] Fig. 11 is a flowchart showing control related to the frame rate, charge accumulation, and image synthesis in the modified example. In the process shown in Fig. 11, instead of S209 in Fig. 8, in S309, the divided accumulation time is allocated to multiple frames of the imaging frame rate so that the center of the charge accumulation time is aligned with the center of each frame period of the recording frame rate, and charge accumulation is started. Since the other processes are the same as those shown in Fig. 8, the same step numbers are used and the description is omitted.

[0067] As described above, according to the modified example, it is possible to obtain the same effects as those of the second embodiment.

[0068] In the above-described embodiment, the frame rate control and image synthesis are performed by independent circuits as shown in Fig. 1 and Fig. 5. However, these configurations or a part of them may be configured within the microcomputer 105.

[0069] Furthermore, the display frame rate of the display unit 110 can be switched between high and low speeds, the default frame rate is low, and the above control may be performed when the display frame rate is switched to high speed.

[0070] <Other embodiments> The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0071] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0072] 101: lens, 102: dimming section, 103: imaging section, 104: image processing section, 105: microcomputer, 106: frame rate switching control section, 107: accumulation control section, 108: first synthesis control section, 109: second synthesis control section, 110: display section, 111: recording section, 112: third synthesis control section, 208: first gain control section, 209: second gain control section, 210: external display section

Claims

1. A setting means for setting a first charge accumulation time and an exposure; a control means for controlling an imaging frame rate for driving the imaging element, a display frame rate for the display means, and a recording frame rate for the recording means; a driving means for controlling a second charge accumulation time in each frame of the imaging frame rate based on the first charge accumulation time and the imaging frame rate to drive the imaging element; a first generating means for generating, when the display frame rate is equal to or lower than the imaging frame rate, a display image corresponding to the exposure set by the setting means at the display frame rate from an image output from the imaging element at the imaging frame rate, based on the imaging frame rate, the display frame rate, the first charge accumulation time, and the second charge accumulation time, and outputting the display image; a second generating means for generating, when the recording frame rate is equal to or lower than the imaging frame rate, a recording image corresponding to the exposure set by the setting means at the recording frame rate from an image output from the imaging element at the imaging frame rate, based on the imaging frame rate, the recording frame rate, the first charge accumulation time, and the second charge accumulation time, and outputting the generated recording image; An imaging device comprising:

2. The imaging device according to claim 1, characterized in that, when the imaging frame rate is changed, the first generation means and the second generation means generate a display image and a recording image so that the display image and recording of each frame period have equivalent exposure before and after the change in the imaging frame rate.

3. The imaging device according to claim 1 or 2, characterized in that, when the display frame rate is changed, the control means changes the imaging frame rate to the same frame rate as the display frame rate if the imaging element can be driven at the changed display frame rate.

4. 4. The imaging device according to claim 1, wherein when the first charge accumulation time is longer than a period of the imaging frame rate, the driving means divides the first charge accumulation time and allocates it to a plurality of frames to drive the imaging element.

5. The second generating means is when a period of the image capturing frame rate is equal to or longer than the first charge accumulation time and the recording frame rate is equal to the image capturing frame rate, an image output in each frame is directly used as the recording image; when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the recording frame rate is slower than the imaging frame rate, an image output for each frame is thinned out in accordance with a ratio between the recording frame rate and the imaging frame rate to generate the recording image; When the period of the imaging frame rate is shorter than the first charge accumulation time and the recording frame rate is slower than the imaging frame rate, images of a plurality of frames are synthesized so that the sum of the second charge accumulation times assigned to each frame becomes the first charge accumulation time before division, thereby generating the recording image.

5. The imaging device according to claim 4.

6. 6. The imaging device according to claim 4, wherein when the first charge accumulation time is longer than a period of the imaging frame rate, the driving means allocates the second charge accumulation time to the same timing of the plurality of frames.

7. The first generating means is when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is equal to the imaging frame rate, an image output in each frame is directly used as the display image; when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is slower than the imaging frame rate, an image output in each frame is thinned out in accordance with a ratio between the display frame rate and the imaging frame rate to generate the display image; When the period of the imaging frame rate is shorter than the first charge accumulation time and the display frame rate is equal to or slower than the imaging frame rate, images of a plurality of frames are sequentially synthesized so that the sum of the second charge accumulation times assigned to each frame becomes the first charge accumulation time before division, thereby generating the display image.

7. The imaging device according to claim 6.

8. The imaging device described in claim 4 or 5, characterized in that when the period of the imaging frame rate is shorter than the first charge accumulation time and the period of the recording frame rate is longer than the first charge accumulation time, the driving means divides and allocates the first charge accumulation time so that the end timing of the first charge accumulation time coincides with the end timing of each frame of the recording frame rate and so that there is no time between the multiple frames when charge accumulation is not performed.

9. The imaging device described in claim 4 or 5, characterized in that when the first charge accumulation time is longer than a period of the imaging frame rate and shorter than a period of the recording frame rate, the driving means divides and allocates the first charge accumulation time so that a center of the first charge accumulation time coincides with a center of the recording frame rate and so that no time is generated between the multiple frames during which charge accumulation is not performed.

10. The first generating means is when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is equal to the imaging frame rate, an image output in each frame is directly used as the display image; when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is slower than the imaging frame rate, an image output in each frame is thinned out in accordance with a ratio between the display frame rate and the imaging frame rate to generate the display image; When a period of the imaging frame rate is shorter than the first charge accumulation time, a gain is applied to an image obtained from each frame in accordance with a ratio between the second charge accumulation time assigned to each frame and the first charge accumulation time before division, thereby generating the display image.

10. The imaging device according to claim 8, wherein the first and second lenses are arranged in a first direction.

11. 11. The imaging device according to claim 8, wherein when allocating the first charge accumulation time to the plurality of frames, if a frame to which the second charge accumulation time is not allocated occurs, the driving means drives the imaging element to perform charge accumulation for a predetermined time in that frame.

12. a setting step in which a setting means sets a first charge accumulation time and an exposure; a control step in which a control means controls an imaging frame rate for driving the imaging element, a display frame rate of the display means, and a recording frame rate of the recording means; a driving step of driving the image sensor by controlling a second charge accumulation time in each frame of the image capture frame rate based on the first charge accumulation time and the image capture frame rate; a first generation step in which, when the display frame rate is equal to or lower than the imaging frame rate, a first generation means generates and outputs a display image corresponding to the exposure set in the setting step at the display frame rate from an image output from the imaging element at the imaging frame rate based on the imaging frame rate, the display frame rate, the first charge accumulation time, and the second charge accumulation time; a second generation step in which, when the recording frame rate is equal to or lower than the imaging frame rate, a second generation step generates and outputs a recording image corresponding to the exposure set in the setting step at the recording frame rate from an image output from the imaging element at the imaging frame rate based on the imaging frame rate, the recording frame rate, the first charge accumulation time, and the second charge accumulation time; 13. A method for controlling an imaging apparatus comprising:

13. A program for causing a computer to function as each of the means of an imaging device described in any one of claims 1 to 11.

14. A computer-readable storage medium storing the program according to claim 13.

Images