Imaging device, control method thereof, and program

Abstract

To obtain a high-quality image according to a consecutive shooting mode and the state of blur.SOLUTION: Plural types of consecutive shooting modes (ultra-high-speed, high-speed, low-speed) with different consecutive shooting speeds can be selected. A motion vector in an input image is separated into a background vector expressing the motion of a background and a subject vector expressing the motion of a subject. A blur signal about blur of an imaging apparatus 100 is acquired from the background vector and a detected gyro signal. An exposure condition in imaging is controlled on the basis of the type of consecutive shooting mode, the subject vector (motion of the subject) and the blur signal (blur information). Further, the composition number is decided when image composition is performed.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an imaging apparatus, a control method thereof, and a program. [Background technology]

[0002] Imaging devices such as digital cameras and smartphones have been put to practical use, using factors other than photometry to control exposure conditions (exposure parameters) (exposure time, ISO sensitivity, aperture value, etc.) and capture multiple images to obtain high-quality images. For example, conventional imaging devices detect the amount of subject movement or blur and change the exposure time accordingly, select images with minimal blur from multiple captured images, or combine multiple images to reduce blur. Such technologies are disclosed in Patent Documents 1, 2, and 3.

[0003] Patent Document 1 discloses a device that corrects camera shake by combining multiple underexposed images taken in succession at a fast shutter speed that is less susceptible to the effects of camera shake. In this device, even if the number of combined images reaches the upper limit, if the image is still underexposed, the sensitivity is controlled to be increased based on the upper limit of the sensitivity.

[0004] Patent Document 2 discloses a device that, in order to reduce the generation of images with unintended exposure during continuous shooting, does not perform automatic exposure control during continuous shooting in a continuous shooting mode that performs continuous shooting at high speed if the exposure compensation value is not within a predetermined range.

[0005] Patent Document 3 discloses an apparatus that employs a second program diagram in which the shutter speed is faster in a part of the luminance region of the field when camera shake is detected, and sets the shutter speed to the faster side. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-152803 [Patent Document 2] Japanese Patent Application Publication No. 2019-71536 [Patent Document 3] Japanese Patent Application Laid-Open No. 2006-337458 Summary of the Invention [Problem to be solved by the invention]

[0007] There are many different conditions and situations when shooting, such as the type and characteristics of the continuous shooting mode (signal readout time, time between frames, predicted vibration, etc.), the state of the subject (still or moving, etc.), power supply status, the amount and frequency characteristics of camera shake, etc. It is difficult to set the exposure time and number of combined images to suit all of these many conditions.

[0008] However, in Patent Document 1, the number of combined images and the sensitivity are set using the same method regardless of the continuous shooting mode setting, making it difficult to set the number of combined images and the sensitivity that are suitable for the characteristics of the continuous shooting mode.

[0009] In Patent Document 2, in a continuous shooting mode in which continuous shooting is performed at high speed, automatic exposure control may not be performed during the continuous shooting operation, making it difficult to always obtain appropriate exposure.

[0010] In Patent Document 3, when camera shake is detected, the second program diagram with a higher shutter speed is uniformly adopted regardless of the characteristics of the continuous shooting mode, making it difficult to set a shutter speed (exposure time) that is suitable for the characteristics of the continuous shooting mode.

[0011] Therefore, there is room for improvement in terms of obtaining high-quality images that comprehensively take into account the characteristics of the continuous shooting mode and the state of blur.

[0012] An object of the present invention is to obtain high-quality images in accordance with the continuous shooting mode and the state of blur. [Means for solving the problem]

[0013] In order to achieve the above object, the present invention Imaging device as one aspect ofis an imaging device comprising: a setting means for setting a plurality of types of continuous shooting modes with different continuous shooting speeds; a first acquisition means for acquiring the movement of a subject from an input image; a second acquisition means for acquiring blur information relating to the blur of the imaging device; and a control means for controlling exposure conditions during shooting based on the type of continuous shooting mode, the movement of the subject, and the blur information, and for determining the number of images to be combined when combining images. When the continuous shooting mode is set, the control means performs at least one of increasing the exposure time, decreasing the number of combined images, or lowering the ISO sensitivity in a second continuous shooting mode that is slower than the first continuous shooting mode, compared to the first continuous shooting mode. It is characterized by: Another aspect of the present invention is an imaging device comprising: a setting means for setting a plurality of types of continuous shooting modes with different continuous shooting speeds; a first acquisition means for acquiring the movement of a subject from an input image; a second acquisition means for acquiring blur information relating to blurring of the imaging device; and a control means for controlling exposure conditions during shooting based on the type of continuous shooting mode, the movement of the subject, and the blur information, and for determining the number of images to be combined when combining images, wherein the control means shortens the exposure time when the movement of the subject is a second movement that is faster than the first movement, compared to when the movement of the subject is a first movement. [Effects of the Invention]

[0014] According to the present invention, high-quality images can be obtained in accordance with the continuous shooting mode and the state of blur. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of an imaging device. [Figure 2] 10 is a flowchart illustrating an exposure control process. [Figure 3] 10 is a flowchart showing a process for separating camera shake. [Figure 4] 10 is a table showing the relationship between frequency and addition coefficients of hand-shake vector signal values. [Figure 5] 10A and 10B are diagrams showing the results of frequency analysis of hand shake signals and motion vectors generated during a predetermined period before the start of shooting. [Figure 6] 10 is a flowchart showing the process of calculating the exposure coefficient and the number of combined images and selecting the single shooting mode or the continuous shooting mode. [Figure 7] FIG. 10 is a diagram showing an example of a program diagram. [Figure 8] FIG. 1 is a block diagram of an imaging device. [Figure 9] 10 is a flowchart illustrating an exposure control process. [Figure 10] FIG. 10 is a diagram showing an example of a program diagram. [Figure 11] FIG. 10 is a diagram showing an example of a program diagram. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] (First embodiment) 1 is a block diagram of an imaging device according to a first embodiment of the present invention. The imaging device 100 is configured as, for example, a digital camera. The imaging device 100 is configured such that an interchangeable lens 31 is detachably attached to an imaging device body 1.

[0018] In the imaging device body 1, a lens mount 2 is arranged to mount an interchangeable lens 31. Light passing through the photographic optical system in the interchangeable lens 31 forms a subject image on the imaging element 3, which is then photoelectrically converted. The imaging unit 4 generates an image signal by performing various image processing on the electrical signal photoelectrically converted by the imaging element 3. The A / D conversion unit 5 converts the analog image signal generated by the imaging unit 4 into a digital image signal. A memory (VRAM) 6, such as a buffer memory, receives the output of the A / D conversion unit 5 and temporarily stores this image data. A D / A conversion unit 7 reads the image data stored in the VRAM 6 and converts it into an analog image signal in a format suitable for playback output. An image display device (hereinafter referred to as LCD) 8, such as a liquid crystal display (LCD), displays this image signal. A storage memory 10, such as a semiconductor memory, stores image data.

[0019] The compression / decompression unit 9 has a compression unit that performs compression processing and encoding processing of image data to read image data temporarily stored in the VRAM 6 and convert it into a format suitable for storage in the storage memory 10. The compression / decompression unit 9 also has an expansion unit that performs decoding processing and expansion processing to convert the image data stored in the storage memory 10 into a format optimum for playback display, etc. The AE processing unit 11 receives the output from the A / D conversion unit 5 and performs automatic exposure (AE) processing. The AF processing unit 12 receives the output from the A / D conversion unit 5 and generates an AF evaluation value for performing automatic focus adjustment (AF) processing, and also detects the defocus amount.

[0020] The shake detection sensor 14 detects movement of the imaging device 100, such as camera shake. The shake detection sensor 14 is composed of inertial sensors such as a gyro sensor and an accelerometer, and multiple sensors are used to detect shake on multiple axes. The shake detection unit 13 processes the gyro signal output (detection signal) from the shake detection sensor 14. The CPU 15 is a microcomputer with built-in memory for calculations that controls the entire imaging device 100. The timing generator (hereinafter referred to as TG) 16 generates a predetermined timing signal. The sensor driver 17 drives the imaging element 3. The operation switch 18 is composed of a group of various switches (SW). The EEPROM 19 is an electrically rewritable read-only memory that pre-stores programs for various controls and data used to perform various operations.

[0021] Battery 20 is the power source for the entire imaging device 100. Communication driver 21 communicates with interchangeable lens 31. LED 22 is a display element that displays warnings and the like. Sensor movement motor 25 is a drive source for moving imaging element 3 horizontally, vertically, and rotationally. Sensor movement control unit 24 controls the operation of sensor movement motor 25. Motion vector detection unit 27 receives output from A / D conversion unit 5 and performs processing to detect the motion vector of the subject. Main subject detection unit 26 receives outputs from motion vector detection unit 27, A / D conversion unit 5, and CPU 15 and performs main subject detection processing. Main subject detection unit 26 identifies the main subject and detects its position and size within the screen.

[0022] The image transformation and cropping unit 28 performs image processing such as image rotation, enlargement / reduction, and trimming (cropping). The image synthesis unit 29, which serves as a synthesis means, synthesizes the images cropped by the image transformation and cropping unit 28. The image transformation and cropping unit 28 and the image synthesis unit 29 can process multiple images obtained by continuous shooting to obtain a new image. The speaker 23 is a sound source for announcing focus and warning of out-of-focus.

[0023] Meanwhile, in the interchangeable lens 31, the image stabilization lens 32 is an optical system for moving the subject image on the image plane of the image sensor 3 in order to correct image blur. The focus lens 33 is an optical system for adjusting the imaging position of the subject image along the optical axis to achieve focus. The iris 34 controls the amount of light passing through the photographic optical system consisting of the image stabilization lens 32, focus lens 33, etc. The communication driver 35 communicates with the image capture device body 1. The motor control unit 36 ​​drives and controls the iris drive motor that drives the iris 34, the focus drive motor that drives the focus lens 33, and the image stabilization lens drive motor that drives the image stabilization lens 32 (none of which are shown). The EEPROM 37 is an electrically rewritable read-only memory that pre-stores data used to perform various operations. The interchangeable lens 31 also includes a zoom ring that changes the focal length and a manual focus ring that adjusts the focus (not shown).

[0024] The storage medium, storage memory 10, may be a fixed semiconductor memory such as a flash memory, or a card-shaped or stick-shaped semiconductor memory such as a card-type flash memory that is detachably attached to the device. Alternatively, storage memory 10 may be a hard disk or the like.

[0025] The operation switches 18 also include a main power switch for starting up the imaging device body 1 and supplying power, a release switch for starting a video shooting operation (recording operation), a playback switch for starting a playback operation, etc. The release switch is a two-stage switch with a first stroke (hereinafter SW1) and a second stroke (hereinafter SW2). When SW1 is turned on, an instruction signal is generated to start shooting preparation operations such as AE processing and AF processing that are performed prior to the shooting operation. When SW2 is turned on, an instruction signal is generated to start the actual exposure operation. The operation switches 18 also include a shooting mode setting dial, an exposure compensation amount change dial, an exposure time change dial, an aperture value change dial, a continuous shooting mode setting switch, etc.

[0026] The continuous shooting mode setting switch mentioned above allows selection of single or continuous shooting mode, and when continuous shooting mode is selected, multiple continuous shooting modes with different sensor readout rates (different continuous shooting speeds) can be selected. Therefore, the type of continuous shooting mode is set based on user operation. The types of continuous shooting modes that can be set include ultra-high-speed continuous shooting mode, high-speed continuous shooting mode, and low-speed continuous shooting mode. These continuous shooting modes differ in the number of channels and readout drive frequency when reading signals from the sensor, resulting in different characteristics such as the time and interval (blanking period) for reading from the sensor. This results in different continuous shooting mode characteristics such as the number of shots per unit time, the effectiveness of suppressing subject movement distortion, and power consumption.

[0027] The operation of the imaging device 100 configured as described above will be described. First, a light beam from a subject that has passed through the interchangeable lens 31 and whose light intensity has been adjusted is focused on the light receiving surface of the image sensor 3. The focused subject image is converted into an electrical signal through photoelectric conversion processing by the image sensor 3 and output to the imaging unit 4. The imaging unit 4 performs various signal processing on the input signal, generating a predetermined image signal. This image signal is output to the A / D conversion unit 5 and converted into a digital signal (image data), after which it is temporarily stored in the VRAM 6 and also output to the AE processing unit 11, the AF processing unit 12, the motion vector detection unit 27, the main subject detection unit 26, and the image transformation and cropping unit 28.

[0028] The image data stored in VRAM 6 is output to D / A conversion unit 7, where it is converted into an analog image signal in a format suitable for display, and then displayed as an image on LCD 8. The image data stored in VRAM 6 is also output to compression / decompression unit 9. The output image data is compressed by a compression unit in compression / decompression unit 9, then converted into image data in a format suitable for storage, and stored in storage memory 10.

[0029] The AE processing unit 11 receives the input digital image signal and calculates an AE evaluation value corresponding to the brightness of the subject. This AE evaluation value is output to the CPU 15. The CPU 15 then calculates the exposure time for the image sensor 3 and the aperture value of the diaphragm 34 based on this AE evaluation value, and transmits this information to the interchangeable lens 31 via the communication driver 21. In response to this, the interchangeable lens 31 performs diaphragm drive processing and the like, adjusting the aperture value of the diaphragm 34 so that it is appropriate.

[0030] The AF processing unit 12 detects the amount of defocus by correcting the image signal acquired by the image sensor 3 having imaging pixels for focus adjustment and performing correlation calculations on the corrected image signal. The CPU 15 determines the amount and direction of drive of the focus lens 33 and transmits this to the interchangeable lens 31 via the communication driver 21. In response, the interchangeable lens 31 performs drive processing for the focus lens 33, enabling AF control to achieve a focused state.

[0031] Main subject detection unit 26 detects subjects that appear to be people, animals, or other possible subjects from the image data input from D / A conversion unit 7. Main subject detection unit 26 receives from CPU 15 the setting status of operation switch 18, the results of AE processing unit 11, and color temperature information of the subject determined for AWB processing, as well as subject distance information of the suspected subject received from AF processing unit 12. Furthermore, main subject detection unit 26 receives movement information of the suspected subject from motion vector detection unit 27, and receives a detection-processed signal (integrated gyro signal) from blur detection unit 13. Main subject detection unit 26 then uses this received information to identify the main subject and detect its position and size.

[0032] The motion vector detection unit 27 receives the input digital image signal (standard image) and performs correlation calculations with the digital image signal (reference image) of the previous frame according to the divided areas in response to instructions from the CPU 15. In this way, the motion vector detection unit 27 obtains the motion vector of the subject within the divided areas.

[0033] That is, motion vector detection unit 27 performs a difference calculation between the standard image and the reference image while shifting the reference image by a predetermined number of pixels in the horizontal and vertical directions, and determines the pixel shift amount that results in the highest correlation (smallest difference amount) as the amount of motion of the subject in that region. Motion vector detection unit 27 then determines the horizontal and vertical pixel shift directions at that time as the motion direction. This allows the motion vector of the subject within the region between frames to be determined. CPU 15 detects the movement of the background and the movement of the subject from the motion vector for each region and the main subject information received from main subject detection unit 26.

[0034] Image transformation and cropping unit 28 receives information about the up / down / left / right movement of the main subject in the image calculated by CPU 15 in response to outputs from main subject detection unit 26 and motion vector detection unit 27, as well as information about the rotation of imaging device 100. Image transformation and cropping unit 28 then performs image processing such as transformation such as image rotation and cropping a portion of the image in order to correct the movement or rotation according to the received information. The resulting image is recorded in a predetermined area of ​​VRAM 6 or output to image synthesis unit 29.

[0035] For example, the position of the subject detected as the main subject or the background on the screen may change due to camera shake by the photographer. If such blurring occurs between multiple images obtained by continuous shooting, unnatural lines may appear in the combined image and the resolution may decrease.

[0036] Therefore, a motion vector detection unit 27 detects the positional deviation between the multiple images, and an image transformation and cropping unit 28 geometrically transforms the images to correct the positional deviation. Then, an image synthesis unit 29 adds together the multiple images that have been aligned by the geometric transformation, thereby performing blur correction (positional deviation correction) by image synthesis.

[0037] Next, the exposure control operation in the imaging device 100 configured as described above, i.e., the operation of setting the exposure coefficient (exposure condition) and the number of images to be combined, will be described using the flowchart shown in Fig. 2. Here, the exposure coefficient includes exposure parameters such as exposure time (Tv), aperture value (Av), and ISO sensitivity (Sv). The number of images to be combined is the number of images to be combined when combining multiple images captured in continuous shooting mode.

[0038] 2 is a flowchart showing the exposure control process. This process is realized by CPU 15 expanding a program stored in EEPROM 19 into a RAM (not shown) provided in CPU 15 and executing it. This process starts when a single-shot or continuous-shot shooting mode is set. In this process, CPU 15 serves as a first acquisition means, a second acquisition means, and a control means.

[0039] In step S201, the CPU 15 performs processes such as initializing variables used in this process and moving drive members to their initial positions, and also checks initial settings such as the interchangeable lens and shooting mode. The CPU 15 determines whether an interchangeable lens is attached, and if so, acquires information about the image stabilization lens, focus lens, and aperture of the interchangeable lens. If no interchangeable lens is attached, the CPU 15 does not acquire information about the interchangeable lens. The CPU 15 also checks the operation status of the shooting mode setting dial on the operation switch 18, and confirms the set shooting mode (for example, shutter speed priority, aperture priority, sports mode, landscape mode, etc.).

[0040] In step S202, CPU 15 checks the status of the continuous shooting mode setting switch on operation switch 18, which serves as setting means, and checks the setting of single shooting / continuous shooting mode. That is, CPU 15 checks whether the mode is set to single shooting mode, ultra-high speed continuous shooting mode, high speed continuous shooting mode, low speed continuous shooting mode, or continuous / single shooting automatic setting mode. Ultra-high speed continuous shooting mode is a special continuous shooting mode that speeds up sensor readout, operates more channels in parallel, and has a higher drive frequency than normal readout.

[0041] In this ultra-high speed continuous shooting mode, it is possible to suppress motion distortion of the subject without operating the mechanical shutter. By not requiring mechanical shutter operation, it is possible to take more pictures per unit time than with a mechanical shutter. However, since ultra-high speed continuous shooting mode is generally set to a higher speed by limiting the number of read bits and the read range of the OB (optical black) area, it is sometimes not set for normal shooting.

[0042] In step S203, CPU 15 receives motion vector information for each divided region over a predetermined period from motion vector detection unit 27. CPU 15 also receives signals (integrated gyro signals) for each axis over a predetermined period from shake detection unit 13. The predetermined period here refers to a predetermined period before the start of shooting, and is assumed to be a period from several seconds before the time of information reception (for example, when SW1 is turned on) to the time of information reception.

[0043] In step S204, CPU 15 executes a camera shake separation process. CPU 15 separates the motion vectors in the input image into a background vector representing the movement of the background and an object vector representing the movement of the object. Therefore, CPU 15 can obtain the object vector by separating the background vector from the motion vectors in the input image. This camera shake separation process will be described in detail with reference to FIGS. 3 and 4, but will be briefly outlined here.

[0044] Since motion vector information is a combination of information caused by camera shake by the photographer and information caused by the movement of the subject within the frame, it is necessary to separate these information. Furthermore, gyro signals have signal fluctuations such as offset and drift. Filtering makes relatively high-frequency signals more reliable than motion vectors, but relatively low-frequency signals less reliable. Therefore, CPU 15 uses both the motion vector and the gyro signal to generate a signal with high reliability across the low- to high-frequency range. CPU 15 compares the separated camera shake vector with the processed signal (integrated gyro signal) obtained from the shake detection unit 13 to obtain a camera shake signal (camera shake information) with the fluctuation component removed from the gyro signal. This allows for the generation of a highly reliable camera shake signal with minimal fluctuation that is separated from subject movement. Note that in this embodiment, the camera shake caused by the movement of the imaging device 100 is estimated to be primarily due to camera shake caused by the user during handheld shooting, and the camera shake information is obtained as blur information. However, the present invention is not limited to this configuration. For example, even when the camera is fixed or the user is standing still, the imaging device 100 itself may move due to external factors, and the blur information described above is not limited to information caused by camera shake.

[0045] In step S205, CPU 15 predicts camera shake during exposure (in the case of continuous shooting, from the start of the first shooting to the end of the last shooting; the same applies below). CPU 15 predicts the frequency and amount of camera shake occurring for each frequency during exposure by analyzing the frequency of the camera shake signal that occurred during the predetermined period before the start of shooting, which was determined in step S204. The camera shake prediction method will be described later with reference to FIG. 5.

[0046] In step S206, CPU 15 predicts the movement of the main subject during exposure. First, main subject detection unit 26 receives the motion vector of the detected main subject region from motion vector detection unit 27. CPU 15 performs frequency analysis of this motion vector information for a predetermined period before the start of shooting, thereby predicting the occurrence frequency and amount of main subject movement that occurs during exposure for each frequency. The method of predicting main subject movement will be described later with reference to FIG. 6.

[0047] In step S207, CPU 15 determines the amount of camera shake after correction during exposure from the camera shake predicted in step S205. This is determined from the response characteristics of vibration compensation lens 32 and the vibration compensation sensor drive system, the vibration suppression performance determined from the frequency characteristics of the filter processing system used to determine the amount of shake compensation, and the predicted camera shake. The amount of camera shake after correction for each frequency is determined from the predicted camera shake for each frequency and the vibration suppression performance for each frequency, so the amount of camera shake after correction is determined by adding these amounts for each axis and then calculating the square root of the sum of squares. CPU 15 then determines the stroke of vibration compensation lens 32 and the vibration compensation sensor drive system required for that correction.

[0048] In step S208, CPU 15 executes AF processing and AE processing. CPU 15 then performs scene determination based on subject distance information and brightness information obtained thereby, as well as main subject information received from main subject detection unit 26. In scene determination, CPU 15 determines whether the scene corresponds to the following first, second, or third scene.

[0049] First, the first scene corresponds to a case where the main subject is a person, a pet such as a dog or cat, a vehicle, etc., and the value of the motion vector is equal to or greater than a certain value, or the AF information indicates that the subject is moving in the distance direction. The first scene also corresponds to a case where the shooting mode is set to Sports mode or Kids & Pets mode. In this case, the subject's movement is small before shooting, and there is a high possibility that a large movement will occur due to some trigger. Macro shooting (determined from the focal length and AF information) also corresponds to the first scene. It is desirable to set ultra-high speed continuous shooting or the like in the first scene. In this embodiment, the ultra-high speed continuous shooting mode is set in principle in the first scene.

[0050] The second scene includes a case where the main subject is a person, a pet such as a dog or cat, a vehicle, or the like, but the motion vector value is less than a certain value and the distance is far away, or a case where the main subject is a still life or other object and the motion vector value is less than a certain value. The second scene also includes a case where the shooting mode is set to landscape mode. In this case, even if there is movement of the subject before shooting, it is highly likely that it is not the main subject. In the second scene, ultra-high-speed continuous shooting is not necessary, and low-speed continuous shooting may be preferable from the viewpoint of energy saving, etc. In this embodiment, the high-speed continuous shooting mode is set as a general rule for the second scene.

[0051] The third scene corresponds to a night scene where the brightness determined from the AE information is darker than a predetermined value and the subject is motionless, or a scene where some areas are extremely bright and the overall image is darker than a predetermined value based on the color temperature information and AE information, and it can be determined that the scene is a sunrise or sunset. Because the exposure period is long in the third scene, slow continuous shooting is preferable, as it allows for long shooting intervals and centering of the correction system. This also enables blur correction during the long exposure period. In this embodiment, the slow continuous shooting mode is set as a general rule in the third scene.

[0052] In step S209, the CPU 15 calculates the limit exposure time TvLMT, which is the longest exposure time, using equation 1 or 2 based on the magnitude of the subject movement. The limit exposure time TvLMT is the limit value of the exposure time that can be set on the long side. If the subject movement is equal to or greater than a predetermined value, the exposure time Tv is set to a time that does not cause subject blur. Therefore, the CPU 15 calculates the exposure time TvLMT using equation 1. The exposure time TvLMT is set to, for example, the maximum value that satisfies equation 1.

[0053] Exposure time TvLMT ≦ length of one side of one pixel ÷ subject movement speed (μm / sec) ... (Equation 1) Here, the subject motion speed is the value at the frequency at which the amount of main subject motion occurring during exposure predicted in step S206 is maximum for each frequency. Alternatively, the subject motion speed may be calculated by multiplying the subject motion speed for each of a plurality of representative frequencies by the occurrence frequency.

[0054] On the other hand, if the subject movement is less than a predetermined value, the CPU 15 estimates the amount of camera shake after correction and controls the exposure time Tv, ISO sensitivity, and aperture value Av. Therefore, the CPU 15 calculates the limit exposure time TvLMT using Equation 2. The exposure time TvLMT is set to, for example, the maximum value that satisfies Equation 2.

[0055] Exposure time TvLMT ≦ length of one side of one pixel ÷ amount of camera shake after correction (μm / sec) ... (Equation 2) Here, if the actual stroke of the correction system is insufficient for the stroke of the shake correction system calculated in step S207, CPU 15 may determine the limit exposure time TvLMT as follows: That is, CPU 15 compares the exposure time at which the amount of camera shake predicted in step S205 becomes a predetermined ratio (for example, 80%) of the actual stroke of the correction system with the result of calculation using equation 2 above, and determines the shorter time as the limit exposure time TvLMT.

[0056] The corrected amount of camera shake used here was found from the corrected amount of camera shake found in step S207. The value found in step S207 represents the sum of the amplitudes of the camera shake waveforms at each frequency. Therefore, multiplying this sum by the exposure time can be thought of as the corrected amount of camera shake during exposure under the most unfavorable conditions (however, because camera shake is a periodic wave, the maximum value is the value found in step S207). Strictly speaking, it is necessary to determine the exposure start and end times and make calculations from the waveform obtained by adding and combining the camera shake signals of each frequency, but the above simple method is used to determine the limit exposure time TvLMT.

[0057] In step S210, the CPU 15 calculates the exposure coefficient and the number of combined images, and also selects continuous / single shooting and the type of continuous shooting mode in the case of continuous shooting, the details of which will be described later with reference to FIGS.

[0058] In step S211, the CPU 15 executes exposure processing corresponding to the set shooting mode (including continuous shooting processing or single shooting processing). For example, when the CPU 15 confirms that the release switch SW1 is turned on, it drives the focus lens 33 to the in-focus position based on the AF processing result, and controls the exposure time, aperture value, and the like based on the results of the calculations and settings in step S210. At this time, the CPU 15 also executes image stabilization processing. This processing may be performed using known image stabilization processing, etc. Then, when SW2 is turned on, the CPU 15 executes actual exposure processing. In the case of continuous shooting, exposure is performed for the number of frames to be combined.

[0059] In step S212, in the case of continuous shooting, the CPU 15 performs blur correction by synthesis processing and ends the processing shown in Fig. 2. Note that in the case of single shooting, the CPU 15 ends the processing shown in Fig. 2 without performing synthesis processing.

[0060] In the synthesis process, the CPU 15 detects misalignment between multiple images input from the A / D converter 5 using the motion vector detector 27, and then performs geometric transformation on the images to correct the misalignment using the image transformation and cropping unit 28. This geometric transformation corrects misalignment in the translational and rotational directions of the images by performing an affine transformation or projective transformation on the images. Geometric transformation requires matching each pixel of the pre- and post-transformation images. When translating or rotating an image in subpixel units, pixels do not correspond one-to-one, so matching is typically achieved using pixels interpolated from multiple surrounding pixels. However, pixel interpolation reduces image resolution and causes image quality degradation. To prevent image quality degradation, it is desirable to avoid translational or rotational movements in subpixel units as much as possible. Limiting translational movements to integer pixel units reduces alignment accuracy, but suppresses image quality degradation due to pixel interpolation. The multiple images aligned by geometric transformation are then added together by the image synthesis unit 29. This allows for image blur correction through image synthesis.

[0061] Fig. 3 is a flowchart showing the hand-shake separation process executed in step S204 of Fig. 2. Here, separation of hand-shake vectors by clustering and generation of a reliable hand-shake signal without fluctuations will be described.

[0062] In step S301, CPU 15 determines whether there is a main subject, that is, whether a main subject has been detected by main subject detection unit 26. If there is a main subject, CPU 15 proceeds to step S302, and if there is no main subject, CPU 15 proceeds to step S311.

[0063] In step S302, CPU 15 performs a first separation process using the main subject information obtained from main subject detection unit 26. In this first separation process, CPU 15 identifies a motion vector detected at a point outside the subject area as a background vector, and identifies a motion vector detected at a point inside the subject area as a subject vector. After step S302, CPU 15 proceeds to step S303.

[0064] In step S311, CPU 15 determines the distance to the farthest object in the image and the distance to the closest object. In step S312, CPU 15 determines whether the difference between these two distances is equal to or greater than a predetermined value. If the difference between the two distances is less than the predetermined value, CPU 15 designates all of the distances as background vectors in step S315 and proceeds to step S321. On the other hand, if the difference between the two distances is equal to or greater than the predetermined value, CPU 15 determines a threshold value in step S313. This threshold value may be, for example, the average value of the distance to the farthest object and the distance to the closest object. In step S314, CPU 15 performs a first separation process in which motion vectors detected in areas farther than the threshold are identified as background vectors and motion vectors detected in areas closer than the threshold are identified as object vectors. Then, CPU 15 proceeds to step S303.

[0065] In step S303 and thereafter, in order to improve the accuracy of separation, the CPU 15 performs "separation based on the amount of motion vectors" using the well-known k-means method as the second separation process.

[0066] First, in step S303, CPU 15 sets the number k of clusters to be classified and an initial value Vk of the center of gravity of each cluster. In this case, the number of clusters is set to k=2 because the image is separated into a background cluster and an object cluster. If there are multiple objects, the number of clusters may be changed depending on the number of objects. Also, a cluster may be provided to separate motion vectors that do not belong to either the background or the object.

[0067] The initial value of the center of gravity of each cluster is determined using the results of the first separation process. Specifically, CPU 15 generates histograms for the background vectors obtained in the first separation process, using the amount of movement in the X direction and the amount of movement in the Y direction, and calculates the most frequent values ​​V1x and V1y of each histogram. Since this most frequent value is the representative value of the background vectors, CPU 15 sets this as the initial value V1=(V1x, V1y) of the center of gravity of the background cluster. In a similar manner, CPU 15 calculates representative values ​​V2x and V2y of the object vectors obtained in the first separation process, and sets this as the initial value V2=(V2x, V2y) of the center of gravity of the object cluster.

[0068] In step S304, the CPU 15 calculates the distance between each motion vector data and the center of gravity of each cluster, and in step S305, the CPU 15 reassigns each motion vector data to a cluster by regarding it as belonging to the cluster with the closest distance.

[0069] In the above process, if the cluster assignments for all vector data have not changed, or if the amount of change is below a predetermined threshold, the CPU 15 determines that the process has converged. Then, the CPU 15 ends the second separation process and proceeds from step S306 to step S321. The background vector obtained in this way becomes the hand-shake vector.

[0070] On the other hand, if it cannot be determined that the process has converged, the CPU 15 proceeds to step S307, where it recalculates the center of gravity from the newly assigned cluster, and then returns to step S304. Therefore, the CPU 15 calculates the distance between each motion vector data and the center of gravity of each cluster, and repeats the process of reassigning the cluster.

[0071] In step S321, the CPU 15 obtains a gyro signal value for each frequency by performing a Fourier transform on the gyro signal after filtering obtained in step S203. In step S322, the CPU 15 obtains a hand shake vector signal value for each frequency by performing a Fourier transform on the signal obtained in the processes up to step S306.

[0072] The bandwidth of the hand-shake vector signal value obtained by conversion to the frequency domain is approximately 0 to 15 Hz. This is because the sampling rate of the digital image signal input to the motion vector detection unit 27 is approximately 30 fps. In contrast, the bandwidth of the gyro signal value is approximately 0 to 100 Hz. The sampling period of the gyro signal for the three axes (pitch, yaw, and roll) is approximately 600 Hz, so the sampling period for one axis is the value shown above. Therefore, for the sake of convenience, the CPU 15 sets any portion of the hand-shake vector signal value above 15 Hz to 0.

[0073] In step S323, the CPU 15 determines an addition coefficient to be used when performing weighted addition on the gyro signal value and the hand-shake vector signal value for each frequency. The CPU 15 determines the addition coefficient according to the table shown in FIG. 4 so that the value is larger in each frequency band with high reliability. FIG. 4 is a table showing the relationship between frequency and the addition coefficient for the hand-shake vector signal value. The addition coefficient for the hand-shake vector signal value is 1.0 at a frequency of 0 Hz (DC) and 0 at frequencies above which the reliability of the gyro signal value is sufficiently high, with values ​​linearly interpolated for frequencies in between. The addition coefficient for the gyro signal value is 1 minus the addition coefficient for the hand-shake vector signal value.

[0074] In step S324, CPU 15 adds the gyro signal value and the hand-shake vector signal value for each frequency using the addition coefficient calculated in step S323. In step S325, CPU 15 performs an inverse Fourier transform to generate a hand-shake signal, and then ends the process shown in Fig. 3. In this way, CPU 15 can obtain the hand-shake signal based on the background vector separated from the motion vector in the input image and the gyro signal from shake detection sensor 14.

[0075] Next, the prediction of camera shake during exposure, which is executed in step S205, and the prediction of the movement of the main subject during exposure, which is executed in step S206, will be described with reference to FIG.

[0076] 5(a) and 5(c) are diagrams showing the results of frequency analysis (e.g., Fourier transform results) of hand shake signals generated during a predetermined period before the start of shooting. FIG. 5(b) is a diagram showing the results of frequency analysis of motion vectors generated during a predetermined period before the start of shooting. These frequency analyses are performed in steps S321 and S322.

[0077] Compared to the frequency distribution of camera shake, the frequency distribution of motion vectors generally exists on the low-frequency side. This is shown in Figures 5(a) and (b). Also, in Figure 5(c), in addition to the normal camera shake, there is a large signal value on the lower frequency side. This is caused by moving the imaging device to reframe or check the settings. Since this does not occur during exposure, its effect must be eliminated.

[0078] Therefore, the CPU 15 obtains the maximum value Bmax of the signal values obtained from the frequency analysis result and the average value Bave of the signal values of each frequency obtained from the frequency analysis result. Then, as the predicted value of the hand shake during exposure, if Bmax < Bave × Cs holds, the CPU 15 selects the frequency having a hand shake signal value of Bmax × Rs or more. Cs and Rs are constants. On the other hand, if Bmax ≥ Bave × Cs holds, the CPU 15 selects the frequency having a hand shake signal value that is Bstd (corresponding to the maximum value near Bave) × Rs or more and less than Bstd. <0********><0********><0********>This Bstd is the average value of the values that are less than or equal to the threshold value (Bth), which is the average value of the values greater than or equal to Bave, and greater than or equal to Bave. The constant Cs may be set to a value of about 3 to 4 in consideration of the properties of the normal distribution function and the like. The constant Rs may be set to a value of about 0.5. The predicted value of the movement of the main subject during exposure can also be obtained in the same manner. <0********><0********>[[ID=​​​​​​​​To obtain Bstd, the CPU 15 first calculates the average value Bth of values greater than or equal to Bave and uses this as the threshold value. Then, the CPU 15 calculates the average of the signal values that are less than or equal to this threshold value and greater than or equal to Bave, and designates this as Bstd. Thereafter, the CPU 15 selects the frequency having a camera shake signal value that is greater than or equal to Bstd × Rs and less than Bstd. Here, the frequency corresponding to the vicinity of the signal value Bt1 in Fig. 5(c) will be selected.

[0083] The maximum value of the movement signal value of the main subject corresponds to the signal value Bu at the frequency Fu in Fig. 5(b). In this case, since the maximum value Bmax is Bu and the average value is approximately half of Bu, Bmax < Bave × Cs holds. Therefore, the CPU 15 selects the frequency having a camera shake signal value that is greater than or equal to Bmax × Rs. Here, the frequency in the vicinity of the frequency Fu in Fig. 5(b) will be selected.

[0084] In this way, it is predicted that the main subject will move at the selected frequency. This prediction result takes into account both camera shake and subject blur. In the explanation in Fig. 5, for the sake of simplicity of calculation, the frequency values were limited, but all the converted frequencies may be targeted.

[0085] Next, the calculation of the exposure coefficient and the number of composite images and the selection of the single-shot / continuous-shot mode executed in step S210 will be described with reference to Figs. 6 and 7.

[0086] Fig. 6 is a flowchart showing the process of calculating the exposure coefficient and the number of composite images and selecting the single-shot / continuous-shot mode, which is executed in step S210. Fig. 7 is a diagram showing an example of a program diagram used in the process shown in Fig. 6.

[0087] In step S601, the CPU 15 acquires the results of the AE process and the AF process executed in step S208. In step S602, the CPU 15 sets three exposure coefficients according to the program diagram shown in Fig. 7. This program diagram reflects the limit exposure time TvLMT that has already been set in step S209.

[0088] In Fig. 7, the solid line is a program diagram used when the subject movement is large (larger than the first predetermined value), the dashed line is a program diagram used when the subject movement is intermediate (less than the first predetermined value and equal to or greater than the second predetermined value), and the dashed line is a program diagram used when the subject does not move (less than the second predetermined value). Here, when the subject movement is large, it means that the subject is determined to be a running person, a vehicle, or a subject that is moving quickly. When the subject movement is intermediate, it means that the subject is determined to be a slowly moving person, a pet, or the like. When the subject movement is small, it means that the subject is determined to be a stationary person or a still object.

[0089] In the program diagram of FIG. 7, the exposure time Tv is shorter when the subject movement is a first movement (e.g., when it is greater than the first predetermined value) than when it is a second movement that is greater than the first movement (e.g., when it is less than the second predetermined value).

[0090] If it is determined in step S602 that there is movement of the subject, the CPU 15 proceeds from step S602 to step S603. On the other hand, if it is determined that there is no movement of the subject, the CPU 15 evaluates the influence of shift blur.

[0091] That is, the CPU 15 detects the shooting distance (subject distance) from the AF processing result, and estimates the degree of shift blur and the shift blur correction capability from that distance and the focal length of the shooting lens.The CPU 15 then changes the limit exposure time TvLMT to a shorter exposure time based on the estimation result.That is, the CPU 15 estimates the amount of blur after correction during exposure, and recalculates the limit exposure time TvLMT based on the estimated amount of blur.

[0092] The shift blur compensation performance at each shooting distance is estimated from the stroke and frequency characteristics of the sensor drive system for image stabilization required for image stabilization calculated in step S207. Shift blur is compensated for by the sensor drive system. Therefore, shift blur compensation performance is determined by how much of the actual sensor drive system's stroke margin remains relative to the required stroke, as well as the low-frequency (e.g., 0.5 Hz or less) response characteristics of the sensor drive system and the frequency characteristics of the filter processing system.

[0093] There is no significant delay in the low-frequency response characteristics, and most shift vibration is low-frequency, so if there is sufficient stroke margin in the sensor drive system, it is expected that a considerable amount of compensation will be made. Therefore, the insufficient stroke in the sensor drive system will account for the majority of uncompensated shift vibration. Specifically, the uncompensated shift vibration is calculated using Equation 3, where Rsb is a constant of approximately 0.1.

[0094] Uncorrected shift wobble = stroke shortage + Rsb x stroke margin...(Equation 3) Next, CPU 15 recalculates the limit exposure time TvLMT using the amount of camera shake corrected during exposure and uncorrected shift shake. If the exposure time Tv has been changed, CPU 15 determines the aperture value Av according to the amount of deviation in the optical axis direction (amount of blur in the optical axis direction). That is, CPU 15 detects how much the image capture device 100 has moved in the optical axis direction from the AF processing result executed in step S208, and determines the aperture value Av so that the farthest and nearest objects fall within the depth of field. At this time, extreme values ​​(for example, values ​​outside the range of the standard deviation and average) may be excluded, taking measurement error into consideration.

[0095] If the aperture value Av has changed from the initially set value, the CPU 15 adjusts the ISO sensitivity accordingly. For example, if the aperture value Av is narrowed by one stop, the CPU 15 adjusts the ISO sensitivity by increasing it by one stop. If the exposure time Tv is changed to take into account the effects of shift blur, the CPU 15 adjusts the ISO sensitivity to reflect not only the change in aperture value Av but also the change in exposure time Tv. However, in all of the above cases, an initial upper limit is set for the ISO sensitivity, and if this limit is exceeded, the ISO sensitivity is clipped to the upper limit.

[0096] Thereafter, in step S603, CPU 15 determines whether or not an appropriate exposure amount can be obtained with the set exposure coefficient. If CPU 15 determines that an appropriate exposure amount can be obtained, it proceeds to step S621, and if it determines that an appropriate exposure amount cannot be obtained, it proceeds to step S604.

[0097] In step S621, if the photographer has not set the continuous shooting mode, the CPU 15 determines the exposure coefficient and also sets the single shooting mode. Therefore, in this case, it is determined that image composition will not be performed. On the other hand, if the photographer has set the continuous shooting mode, the CPU 15 sets the continuous shooting mode (type) accordingly. Therefore, in this case, it is determined that image composition will be performed. After step S621, the CPU 15 ends the processing shown in FIG. 6.

[0098] In step S604, the CPU 15 sets the continuous shooting mode. If the continuous shooting mode setting (ultra-high speed, high speed, low speed) has been intentionally made by the photographer, the CPU 15 proceeds to step S605 with that continuous shooting mode (type) set.

[0099] On the other hand, if the continuous shooting mode has not been intentionally set, CPU 15 sets the continuous shooting mode according to the scene determination in step S208. This allows a continuous shooting mode appropriate for the movement of the subject to be set. Note that the continuous shooting mode may also be set taking into account the remaining battery charge or the estimated margin of correction stroke. For example, CPU 15 may set the energy-saving low-speed continuous shooting mode even when the remaining battery charge is less than a predetermined level, and if the estimated margin of correction stroke is low, set the low-speed continuous shooting mode, which has a long interval between continuous shots and enables centering of the correction system. Note that the estimated margin of correction stroke can be calculated from the amount of camera shake predicted during exposure in step S205 and the positions (amount of deviation from the center) of the image stabilization lens and the image stabilization sensor drive system at that time.

[0100] After the continuous shooting mode is set, in step S605 and subsequent steps, at least one of the following is performed in the second continuous shooting mode: lengthening the exposure time, reducing the number of combined shots, or lowering the ISO sensitivity, compared to the first continuous shooting mode. Here, the second continuous shooting mode is a slower continuous shooting mode than the first continuous shooting mode. For example, if the first continuous shooting mode is the ultra-high-speed continuous shooting mode, the second continuous shooting mode corresponds to the high-speed continuous shooting mode or the low-speed continuous shooting mode. If the first continuous shooting mode is the high-speed continuous shooting mode, the second continuous shooting mode corresponds to the low-speed continuous shooting mode. In other words, when a relatively low-speed continuous shooting mode is set, a relatively small number of combined shots, a relatively long exposure time, and a relatively low ISO sensitivity are set. When a relatively high-speed continuous shooting mode is set, a relatively large number of combined shots, a relatively short exposure time, and a relatively high ISO sensitivity are set. In this respect, the CPU 15 serves as a control means in the present invention.

[0101] First, in step S605, the CPU 15 sets the number of composite images and resets the exposure time Tv accordingly. That is, the CPU 15 first adjusts the number of composite images to compensate for the shortfall in the exposure coefficient set in step S602. The number of composite images is set so that it is a positive integer equal to 2 raised to the power Nc (Nc is a positive real number) and so that the amount of exposure is appropriate or over. The upper limit of Nc differs depending on the continuous shooting mode.

[0102] There is a limit to the number of images that can be combined. This is because the more images that can be combined, the less exposure each image has, which reduces the signal-to-noise ratio during image combination, and the more images that can be combined, the less accurate the vector alignment becomes. Furthermore, a reduced exposure reduces the accuracy of detecting the amount of image misalignment when performing alignment, and an increased number of alignments not only accumulates minute errors, but also increases the probability of an error in detecting the amount of alignment. Therefore, an upper limit for the number of images that can be combined is set taking the above into consideration. The first, second, and third upper limits for the number of images that can be combined are considered.

[0103] The low-speed, high-speed, and ultra-high-speed continuous shooting modes correspond to first, second, and third upper limits for the number of composite images, respectively. The magnitude relationship of Nc is such that the first upper limit for the number of composite images < the second upper limit for the number of composite images < the third upper limit for the number of composite images. The first upper limit for the number of composite images Nc is a value that is approximately 3 to 4 steps smaller than the third upper limit for the number of composite images Nc. The second upper limit for the number of composite images Nc is a value that is approximately 2 to 3 steps smaller than the third upper limit for the number of composite images Nc. If the exposure amount is higher than appropriate due to the set number of composite images, the CPU 15 adjusts the exposure amount by shortening the exposure time Tv.

[0104] In step S606, CPU 15 determines whether an appropriate exposure amount can be obtained with the number of combined images and exposure coefficient set in step S605. If CPU 15 determines that an appropriate exposure amount can be obtained, it confirms the exposure coefficient and continuous shooting mode settings and ends the processing shown in Fig. 6. However, if it determines that an appropriate exposure amount cannot be obtained, CPU 15 proceeds to step S607.

[0105] In step S607, an upper sensitivity limit higher than the default ISO sensitivity limit is set. The upper sensitivity limit in this case is determined based on the type of continuous shooting mode and the predicted movement of the main subject during exposure. CPU 15 then adjusts (increases) the ISO sensitivity within the upper sensitivity limit. The upper sensitivity limit is lower in a slow continuous shooting mode than in a high-speed continuous shooting mode.

[0106] In step S608, the CPU 15 determines whether an appropriate exposure amount can be obtained as a result of adjusting the ISO sensitivity in step S607. If the CPU 15 determines that an appropriate exposure amount can be obtained, it confirms the settings of the exposure coefficient and the continuous shooting mode and ends the processing shown in Fig. 6. However, if it determines that an appropriate exposure amount cannot be obtained, the CPU 15 proceeds to step S609.

[0107] In step S609, CPU 15 determines whether the movement of the main subject is large, i.e., whether the predicted value of the movement of the main subject during exposure is equal to or greater than a predetermined value. If the predicted value of the movement of the main subject during exposure is equal to or greater than the predetermined value, CPU 15 proceeds to step S610; otherwise, CPU 15 proceeds to step S611.

[0108] In step S610, the CPU 15 readjusts the ISO sensitivity and the number of combined images to obtain the appropriate exposure. For example, the CPU 15 raises the ISO sensitivity by up to half a stop to obtain the appropriate exposure. If the exposure is still insufficient, the CPU 15 increases the number of combined images by up to 1.4 times (with decimal points rounded down to the nearest integer) to obtain the appropriate exposure. If the exposure is still insufficient, the CPU 15 repeats these operations until the appropriate exposure is obtained.

[0109] In step S611, the exposure time Tv, ISO sensitivity, and number of combined images are readjusted to obtain the appropriate exposure amount. For example, the CPU 15 lengthens the exposure time Tv by up to one-third of a stop to obtain the appropriate exposure amount. If the exposure amount is still insufficient, the CPU 15 increases the ISO sensitivity by up to one-third of a stop to obtain the appropriate exposure amount. If the exposure amount is still insufficient, the CPU 15 increases the number of combined images by up to 1.3 times (however, decimal points are rounded down to an integer value) to obtain the appropriate exposure amount. If the exposure amount is still insufficient, the CPU 15 repeats these operations until the appropriate exposure amount is obtained.

[0110] Note that there are hardware configuration restrictions on the exposure time Tv, ISO sensitivity, and number of combined images. Therefore, in steps S610 and S611, the CPU 15 clips the exposure amount at that value if it is insufficient even when the values ​​of these restrictions are reached. Once the appropriate exposure amount is obtained in this way, the CPU 15 confirms the settings of the exposure coefficient and continuous shooting mode, and ends the processing shown in FIG.

[0111] According to this embodiment, the exposure conditions are controlled based on the type of continuous shooting mode (ultra-high speed, high speed, low speed), the subject vector (subject movement), and the camera shake signal (camera shake information), and the number of images to be combined is also determined. This allows high-quality images to be obtained according to the continuous shooting mode and the state of shake. For example, it is possible to suppress a decrease in image resolution due to shake, and an increase in noise and a decrease in color saturation in high-sensitivity shooting.

[0112] Furthermore, when continuous shooting mode is set, at least one of the following is performed for slower continuous shooting modes: lengthening the exposure time, reducing the number of combined images, or lowering the ISO sensitivity. This allows high-quality images to be obtained by setting exposure conditions and the number of combined images appropriate for the continuous shooting speed.

[0113] Furthermore, if the subject moves more rapidly, the exposure time Tv is set to a shorter value. Furthermore, the limit exposure time TvLMT is set based on the magnitude of the subject's movement. This allows for the acquisition of an image in which the loss of resolution due to blurring is suppressed.

[0114] Furthermore, a hand shake signal relating to the shaking of the image capturing device 100 is acquired from the background vector separated from the motion vector and the detected gyro signal, so that the shaking of the image capturing device 100 can be accurately determined.

[0115] In the present embodiment, an example has been described in which each image to be combined is underexposed, and the combined images are properly exposed when all the combined images are added together. However, the sensitivity may be increased to properly expose each image, and all the combined images may be averaged, or both addition and averaging may be used.

[0116] Although the example in which image stabilization is not performed before exposure begins has been shown, image stabilization may be performed at all times. In this case, the vector information will be image stabilization information, so in order to determine the actual movement, it is necessary to subtract the amount of image stabilization from the detected vector.

[0117] In step S605, the continuous shooting mode may be changed to a slower mode, and the exposure time Tv may be changed to a longer time.

[0118] The present invention may be applied to the imaging device body 1 or to an imaging device with an integrated lens. Also, an imaging device body without an interchangeable lens, such as the imaging device body 1, may be called an imaging device.

[0119] (Second embodiment) 8 is a block diagram of an imaging device according to a second embodiment of the present invention. In this embodiment, the imaging device 101 is configured as an information communication terminal, for example, a smartphone. In FIG. 8, the same elements as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0120] In the imaging device 101, the touch panel 108 is an image display input device. The touch panel 108 has a function of displaying images using a liquid crystal display (LCD) or the like and a function of inputting images using the touch panel. The AF processing unit 1012 receives output from the A / D conversion unit 5 and generates an AF evaluation value for performing autofocus (AF) processing. The LED 1022 is a display element that functions as an illumination and flashlight during AF, and also notifies users of incoming calls, etc.

[0121] A focus lens movement motor 1025 drives the focus lens 1033 in the optical axis direction. A focus lens movement control unit 1024 controls the focus lens movement motor 1025. In addition to the selection of single shooting mode or continuous shooting mode by the user operating the touch panel 108, if the continuous shooting mode is selected, multiple types of continuous shooting modes with different continuous shooting speeds can be selected, as in the first embodiment.

[0122] The CPU 15 controls the movement of the focus lens 1033 via the focus lens movement control unit 1024 and the focus lens movement motor 1025, while instructing the AF processing unit 1012 to generate an AF evaluation value. The CPU 15 then records this AF evaluation value in association with the position of the focus lens 1033, and calculates the position of the focus lens 1033 at which the AF evaluation value reaches its peak. The CPU 15 then calculates a correction value for the calculated position, taking into account various characteristics of the optical system, to determine the in-focus position (the position to which the focus lens 1033 is moved). The CPU 15 then controls the focus lens 1033 to this in-focus position, thereby enabling AF control to obtain an in-focus state.

[0123] 9 is a flowchart showing the exposure control process. This process is realized by CPU 15 expanding a program stored in EEPROM 19 into a RAM (not shown) provided in CPU 15 and executing it. This process starts when a single-shot or continuous-shot shooting mode is set. This process corresponds to the process shown in FIG. 2 in the first embodiment.

[0124] In step S901, the CPU 15 performs processes such as initializing variables used in this process and moving drive members to their initial positions, and also checks the set shooting mode (for example, shutter speed priority, aperture priority, sports mode, landscape mode, etc.).

[0125] In steps S902 and S903, the CPU 15 executes the same processes as steps S203 and S204 in FIG. 2. Therefore, the filtered gyro signal obtained in step S902 serves as the camera shake signal. However, since a smartphone is light and lacks a support member such as a grip, it is prone to large low-frequency camera shake. Therefore, large low-frequency camera shake is eliminated by combining images captured with short exposure times. Small high-frequency shakes contained in each image are eliminated by moving the shake detection sensor 14 in the horizontal and vertical rotation directions. Therefore, the shake detection unit 13 performs high-pass filtering to extract only the high-frequency shake components.

[0126] In step S904, similar to step S205, CPU 15 performs frequency analysis of the camera shake signal that occurred during the predetermined period before the start of shooting, which was determined in step S903, to predict the frequency and amount of camera shake that will occur for each frequency during exposure. In step S905, similar to step S206, CPU 15 predicts the movement of the main subject during exposure.

[0127] In step S906, the CPU 15 sets the continuous shooting mode (and its type). It is assumed that the type of continuous shooting mode is not intentionally set in a smartphone. Therefore, the CPU 15 sets the continuous shooting mode based on the scene determination result obtained by the scene determination function installed in the smartphone, the remaining battery level, and the estimated margin of correction stroke. This process is the same as the process in step S604 when the continuous shooting mode is not intentionally set.

[0128] For example, the CPU 15 sets the low-speed continuous shooting mode when the remaining battery power is less than a predetermined level, and also sets the low-speed continuous shooting mode when there is little margin for correction stroke. In other cases, the continuous shooting mode is set based on the scene determination result. The margin for correction stroke can be calculated from the amount of camera shake predicted during exposure in step S904 and the position (deviation from the center) of the shake correction sensor drive system at that time. For example, if the deviation is 30% or more of the total correction stroke, it is determined that there is little margin for correction stroke.

[0129] The continuous shooting mode is set based on the scene determination result, based on the speed of the main subject's movement obtained from motion vector information and AF information, the shooting distance obtained from AF information, brightness obtained from AE information, and the like. This is roughly the same as in the first embodiment. Note that since the image capture device 101 is a smartphone, a server may perform processing such as estimating the shooting point from seasonal time and location information, and a preferred continuous shooting mode may be set based on the results.

[0130] In step S907, the CPU 15 sets three exposure coefficients in accordance with the program diagram shown in FIG. 10 or FIG.

[0131] 10 and 11 are diagrams showing examples of program diagrams used in setting the exposure coefficient. The program diagram in FIG. 10 is applied when the low-speed continuous shooting mode is set. The program diagram in FIG. 11 is applied when the high-speed continuous shooting mode is set. In addition, a program diagram to be applied when the ultra-high-speed continuous shooting mode is set may also be provided. Alternatively, the program diagram in FIG. 11 may be applied when the ultra-high-speed continuous shooting mode is set.

[0132] In Figures 10 and 11, solid lines indicate cases where the subject movement is large (greater than the first predetermined value), dashed lines indicate cases where the subject movement is intermediate (less than the first predetermined value and equal to or greater than the second predetermined value), and dashed lines indicate cases where the subject movement is no (less than the second predetermined value). According to these program diagrams, when a relatively slow continuous shooting mode is selected, a relatively small number of combined images, a relatively long exposure time, and a relatively low ISO sensitivity are set. Also, when a relatively high-speed continuous shooting mode is selected, a relatively large number of combined images, a relatively short exposure time, and a relatively high ISO sensitivity are set. However, these diagrams are merely examples, and other diagrams may be used.

[0133] In step S907, if the CPU 15 does not obtain an appropriate exposure amount by setting the exposure coefficient, it repeats the process of resetting the exposure time, the number of combined images, or the ISO sensitivity until an appropriate exposure amount is obtained.

[0134] For example, the CPU 15 lengthens the exposure time to obtain proper exposure depending on the brightness, and when a predetermined exposure time is reached, increases the ISO sensitivity stepwise (for example, by two steps). Next, the CPU 15 increases the number of combined images (for example, up to four), and if proper exposure is still not obtained, increases the ISO sensitivity again, for example, by 1.5 steps.

[0135] If underexposure still occurs, the CPU 15 determines the exposure coefficient without further control if there is no subject movement (less than the second predetermined value). If the subject movement is intermediate (less than the first predetermined value but equal to or greater than the second predetermined value), the CPU 15 lengthens the exposure time (for example, to 1 / 2 second) and lowers the ISO sensitivity by about one step. The CPU 15 then again raises the ISO sensitivity by about one step to adjust to the correct exposure. If underexposure still occurs at this point, the CPU 15 determines the exposure coefficient without further control.

[0136] If the subject motion is large (greater than the first predetermined value), the CPU 15 increases the number of combined images (for example, up to 16), then lengthens the exposure time (for example, to 1 / 2 second) and lowers the ISO sensitivity by about one step. The CPU 15 then raises the ISO sensitivity again by about one step to adjust the exposure to the correct level. If underexposure still occurs at this point, the CPU 15 determines the exposure coefficient without further control.

[0137] In steps S908 and S909, the CPU 15 executes the same processes as in steps S211 and S212.

[0138] According to this embodiment, the program chart to be used is switched depending on the type of continuous shooting mode. By applying a program chart according to the type of continuous shooting mode, at least one of the following is performed in the case of a slower continuous shooting mode: lengthening the exposure time, reducing the number of combined images, or lowering the ISO sensitivity. Therefore, the same effect as in the first embodiment can be achieved in terms of obtaining high-quality images according to the continuous shooting mode and the state of blur.

[0139] Furthermore, even if the user does not intentionally set or is unable to set the type of continuous shooting mode, the appropriate exposure coefficient and number of combined images are set, providing high usability.

[0140] The first embodiment may be applied to an information communication terminal such as that exemplified in the second embodiment.

[0141] In the first embodiment, a program diagram according to the type of continuous shooting mode as shown in FIGS. 10 and 11 may also be applied.

[0142] In the first and second embodiments, the processing of gyro signals has been described with some exceptions for only one axis. However, in reality, camera shake and subject movement occur on multiple axes, so the same processing can be applied to other axes.

[0143] In the first and second embodiments, the brightness of the shooting environment is measured using a signal from a sensor for obtaining a photographed image, but a dedicated photometric sensor provided separately from this may also be used.

[0144] In the first and second embodiments, the number of types of continuous shooting modes that can be set may be four or more.

[0145] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0146] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a non-transitory storage medium, and having one or more processors in the computer of the system or device read and execute the program. The above program and the storage medium storing the program constitute the present invention. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions. [Explanation of symbols]

[0147] 1. Imaging device body 13 Shake detection unit 15 CPU 18 Operation switch 26 Main subject detection unit 29 Image synthesis unit

Claims

1. An imaging device, a setting means for setting a plurality of types of continuous shooting modes with different continuous shooting speeds; a first acquisition means for acquiring a motion of a subject from an input image; a second acquisition means for acquiring blur information relating to blur of the imaging device; a control means for controlling exposure conditions when taking pictures based on the type of continuous shooting mode, the movement of the subject, and the blur information, and for determining the number of images to be combined when combining images; When the continuous shooting mode is set, the control means performs at least one of increasing the exposure time, decreasing the number of combined images, or decreasing the ISO sensitivity in a second continuous shooting mode that is slower than the first continuous shooting mode, compared to the first continuous shooting mode.

2. An imaging device, a setting means for setting a plurality of types of continuous shooting modes with different continuous shooting speeds; a first acquisition means for acquiring a motion of a subject from an input image; a second acquisition means for acquiring blur information relating to blur of the imaging device; a control means for controlling exposure conditions when taking pictures based on the type of continuous shooting mode, the movement of the subject, and the blur information, and for determining the number of images to be combined when combining images; The imaging device, wherein the control means shortens the exposure time when the subject's movement is a second movement that is faster than the first movement, compared to when the subject's movement is a first movement.

3. 3. The imaging apparatus according to claim 2, wherein the control means determines a limit value of the length of the exposure time that can be set based on the speed of the movement of the subject.

4. 3. The imaging device according to claim 2, wherein the control means estimates the amount of blur after correction during exposure based on the subject distance, and determines the limit value of the length of the exposure time that can be set based on the estimated amount of blur.

5. The imaging device according to any one of claims 1 to 4, characterized in that the first acquisition means acquires the movement of the subject by separating a vector representing the movement of the background and a vector representing the movement of the subject from the motion vectors in the input image.

6. 6. The imaging device according to claim 5, wherein the second acquisition means acquires the blur information based on a vector representing the movement of the background separated from a motion vector in the input image and a detection signal of a sensor that detects blur of the imaging device.

7. 7. The imaging device according to claim 1, wherein the first acquisition means and the second acquisition means acquire the subject movement information and the blur information, respectively, for a predetermined period before the start of imaging.

8. 8. The imaging apparatus according to claim 1, wherein the control means determines to perform image composition when the continuous shooting mode is set.

9. 9. The imaging apparatus according to claim 1, wherein the control means determines not to perform image composition when a single-shot mode is set.

10. 10. The imaging apparatus according to claim 1, wherein the setting unit sets the type of the continuous shooting mode based on a user operation.

11. The imaging device according to any one of claims 1 to 10, characterized in that, when the type of continuous shooting mode is not set by user operation, the setting means determines a scene based on at least the movement of the subject, and sets the type of continuous shooting mode according to the determined scene.

12. 12. The image pickup apparatus according to claim 11, wherein the setting means sets the type of the continuous shooting mode by further taking into consideration a remaining battery charge or a stroke of a sensor drive system for image stabilization.

13. 13. The imaging apparatus according to claim 1, further comprising a combining unit that combines the number of images captured in the continuous shooting mode.

14. 2. The imaging device according to claim 1, wherein, when the continuous shooting mode is set, the control means switches the program chart to be used between the first continuous shooting mode and the second continuous shooting mode.

15. A control method for an imaging device, comprising: a setting step for setting a plurality of types of continuous shooting modes with different continuous shooting speeds; a first acquisition step of acquiring a motion of a subject from an input image; a second acquisition step of acquiring blur information relating to blurring of the imaging device; a control step of controlling exposure conditions when taking pictures based on the type of continuous shooting mode, the movement of the subject, and the blur information, and determining the number of images to be combined when combining images; a control method for an imaging device, characterized in that, when the continuous shooting mode is set in the control step, at least one of the following is performed in the case of a second continuous shooting mode that is slower than the first continuous shooting mode: increasing the exposure time, reducing the number of combined images, or lowering the ISO sensitivity, compared to the case of the first continuous shooting mode.

16. A control method for an imaging device, comprising: a setting step for setting a plurality of types of continuous shooting modes with different continuous shooting speeds; a first acquisition step of acquiring a motion of a subject from an input image; a second acquisition step of acquiring blur information relating to blurring of the imaging device; a control step of controlling exposure conditions when taking pictures based on the type of continuous shooting mode, the movement of the subject, and the blur information, and determining the number of images to be combined when combining images; A control method for an imaging device, characterized in that, in the control step, the exposure time is shortened when the movement of the subject is a second movement that is faster than the first movement, compared to when the movement of the subject is a first movement.

17. 17. A program causing a computer to execute the method for controlling an imaging device according to claim 15.

Images