Control device, imaging device, control method, and program

Abstract

The present invention provides a control device that can suppress the degradation of resolution of low-contrast, high-frequency subjects when generating composite images. [Solution] The control device (15) has an acquisition means (15a) for acquiring the degree of degradation of the resolution of a first region in at least one of the multiple images when generating a composite image using multiple images, and a determination means (15b) for determining the number of images to be composited to generate the composite image according to the degree of degradation.

Description

Technical Field

[0001] The present invention relates to a control device, an imaging device, a control method, and a program.

Background Art

[0002] Patent Document 1 discloses a method of controlling exposure conditions during shooting based on the type of continuous shooting mode, the subject vector (movement of the subject), and the blur signal (blur information), and determining the number of synthesized images when performing image synthesis. Patent Document 2 discloses a method of performing multiple exposures based on the limit exposure time at which no blur occurs determined according to the appropriate exposure time and the focal length of the photographing lens, and adding and outputting a plurality of images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document,2

Summary of the Invention

Problems to be Solved by the Invention

[0004] When generating a composite image, if the number of composite images is determined without considering the alignment error during the composite process, the resolution of a low-contrast and high-frequency subject, which is referred to as the texture of the composite image, may deteriorate due to the influence of the alignment error. On the other hand, the methods disclosed in Patent Document 1 and Patent Document 2 do not consider the deterioration of the texture caused by the alignment error.

[0005] Therefore, an object of the present invention is to provide a control device capable of suppressing the deterioration of the resolution of a low-contrast and high-frequency subject when generating a composite image.

Means for Solving the Problems

[0006] One aspect of the present invention is a control device that, when generating a composite image using a plurality of images, includes an acquisition means for acquiring the degree of degradation of the resolution of a first region in at least one of the plurality of images, and a determination means for determining the number of images to be composited in order to generate the composite image, according to the degree of degradation.

[0007] Other objects and features of the present invention are described in the following embodiments. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a control device that can suppress the degradation of resolution of low-contrast, high-frequency subjects when generating a composite image. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of the imaging system in this embodiment. [Figure 2] This is a flowchart showing the process in this embodiment. [Figure 3] This is an example of a program diagram in this embodiment. [Figure 4A] This flowchart shows the process for estimating the degree of texture degradation and changing the number of composite images in this embodiment. [Figure 4B] This flowchart shows the process for estimating the degree of texture degradation and changing the number of composite images in this embodiment. [Figure 5] This figure shows an example of the relationship between the texture evaluation value and the number of composite images in this embodiment. [Figure 6] This figure shows an example of a P-diagram when the ISO sensitivity is fixed in this embodiment. [Figure 7] This figure shows an example of the relationship between the texture evaluation value and the number of composite images in the medium ISO sensitivity region in this embodiment. [Figure 8] This figure shows an example of frequency components included in the subject in this embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0011] First, the imaging system 100 in this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram of the imaging system 100. The imaging system 100 is composed of an imaging device (camera body) 1 and a lens device (interchangeable lens) 31 that can be attached to or removed from the imaging device 1. However, this embodiment is not limited to this and can also be applied to imaging devices in which the camera body and lens device are integrally configured.

[0012] 1 is an imaging device, 2 is a lens mount on which the lens device 31 is attached, and 3 is an image sensor that photoelectrically converts the subject image (optical image) formed by the imaging optical system of the lens device 31. 4 is an imaging circuit that generates a predetermined image signal by applying various image processing to the electrical signal (image data) output from the image sensor 4. 5 is an A / D conversion circuit that converts the analog image signal generated by the imaging circuit 6 into a digital image signal. 6 is a VRAM (memory), such as a buffer memory, that temporarily stores the digital image signal output from the A / D conversion circuit 5. 7 is a D / A conversion circuit that reads the image signal stored in the VRAM 6, converts it into an analog signal, and converts it into an image signal in a format suitable for playback output. 8 is an LCD (image display device such as a liquid crystal display device) that displays the image signal output from the D / A conversion circuit 7. 10 is a storage memory, such as a semiconductor memory, that stores image data.

[0013] 9 is a compression / expansion circuit that includes a compression circuit and an expansion circuit. The compression circuit performs compression processing and encoding processing on the image data in order to store the image signal temporarily stored in VRAM 6 in a form suitable for storage in the storage memory 10. The expansion circuit performs decoding processing, expansion processing, etc. to put the image data stored in the storage memory 10 into a form optimal for reproduction display. 11 is an AE processing circuit that performs automatic exposure (AE) processing based on the output signal from the A / D conversion circuit 5. 12 is an AF processing circuit that detects the defocus amount for performing automatic focus adjustment (AF) processing based on the output signal from the A / D conversion circuit 5.

[0014] 14 is a shake detection sensor that detects the movement of the imaging device 1 such as camera shake. The shake detection sensor 14 has an inertial sensor such as a gyro sensor or an accelerometer, and can detect multi-axis shake by using a plurality of sensors. 13 is a shake detection circuit that processes the signal of the shake detection sensor 14. The shake detection sensor 14 and the shake detection circuit 13 constitute shake detection means.

[0015] 15 is a CPU (control device) that incorporates a memory for arithmetic operations for controlling the imaging device 1. The CPU 15 has an acquisition means 15a and a determination means 15b. The acquisition means 15a acquires the degree of deterioration of the resolution of the first region in at least one of the plurality of images when generating a composite image using the plurality of images. The determination means 15b determines the number of composite images of the plurality of images used for generating the composite image according to the degree of deterioration.

[0016] 16 is a timing generator that generates a predetermined timing signal. 17 is a sensor driver. 18 is an operation switch that includes various switch groups. 19 is an EEPROM, which is an electrically rewritable read-only memory that stores in advance programs for performing various controls, etc., and data used for performing various operations. 21 is a communication driver for communicating with the lens device 31.

[0017] 25 is a sensor movement motor for moving the image sensor 3 in the horizontal and vertical rotational directions. 24 is a sensor movement control circuit for controlling the movement of the sensor movement motor 25. 26 is a main subject detection circuit that identifies the main subject and detects its position and size within the image. 27 is a motion vector detection circuit that uses the output signal from the A / D conversion circuit 5 to detect the motion vector of the subject. 28 is an image deformation and cropping circuit that performs image processing such as rotation, scaling, and cropping (cutting out) of the image. 29 is an image synthesis circuit that combines the images cropped by the image deformation and cropping circuit 28. The image deformation and cropping circuit 28 and the image synthesis circuit 29 constitute an image synthesis means that processes multiple images acquired by continuous shooting to acquire a new image.

[0018] 32 is an image stabilization lens, 33 is a focus lens, and 34 is an aperture (diaphragm). The image stabilization lens 32, focus lens 33, and aperture 34 constitute the imaging optical system. The aperture 34 functions as a light intensity adjustment means that controls the amount of light beam transmitted through the imaging optical system. 35 is a communication driver for communicating with the imaging device 1. 36 is a control circuit. The control circuit 36 ​​controls an aperture drive motor (not shown) that drives the aperture 34. The control circuit 36 ​​also controls a focus drive motor (not shown) that drives the focus lens 33. The control circuit 36 ​​also controls an image stabilization lens drive motor (not shown) that drives the image stabilization lens 32. 37 is an EEPROM, which is an electrically rewritable read-only memory that stores data used to perform various operations. The lens device 31 further includes a zoom mechanism for changing the focal length of the lens device 31, a zoom ring for operating the zoom mechanism, and a manual focus ring for adjusting the focus.

[0019] The memory used as a storage medium for image data, etc., is a fixed-type semiconductor memory such as flash memory, or a semiconductor memory such as a card-type flash memory that is detachably attached to the imaging device 1 in a card or stick shape. However, this embodiment is not limited to these, and various forms of memory, such as magnetic storage media like hard disks or floppy disks, can be applied.

[0020] The operation switches 18 include a main power switch for starting the imaging device 1 and supplying power, and a release switch for starting video recording (recording) operations, etc. The operation switches 18 also include a playback switch for starting playback operations, a shooting mode setting dial, an exposure compensation amount change dial, an exposure time change dial, an aperture value change dial, and a continuous shooting mode setting switch, etc.

[0021] In this embodiment, a blur correction means is configured to optically correct blur using a blur correction lens 32, a control circuit 36, an image sensor 3, a sensor movement motor 25, and a sensor movement control circuit 24.

[0022] The image data digitized by the A / D conversion circuit 5 is output to the VRAM 6, as well as to the AE processing circuit 11, AF processing circuit 12, motion vector detection circuit 27, main subject detection circuit 26, and image deformation and cropping circuit 28.

[0023] The AE processing circuit 11 is a photometering means that receives an input digital image signal, calculates an AE evaluation value according to the brightness of the subject, and outputs it to the CPU 15. Based on the AE evaluation value, the CPU 15 calculates the exposure time of the image sensor 3 and the aperture value (aperture amount) of the aperture 34, and transmits them to the lens device 31 via the communication driver 21. The lens device 31 performs aperture drive processing and other operations to control the aperture 34 so that the aperture amount of the aperture 34 is appropriate.

[0024] The AF processing circuit 12 performs image correction on the image signal acquired by the image sensor 3, which has imaging pixels for focus adjustment, and correlation calculations on the corrected image signal to detect the amount of defocus. The CPU 15 acquires the drive amount and drive direction of the focus lens 33 and transmits it to the lens device 31 via the communication driver 21. The lens device 31 can perform AF control to obtain a focused state by performing drive processing on the focus lens 33.

[0025] The main subject detection circuit 26 detects subjects that appear to be people, animals, or other subjects from the image data output from the A / D conversion circuit 5. The main subject detection circuit 26 then identifies the main subject from among the detected subjects and detects the position and size of the main subject. The main subject is identified based on the settings of the operation switch 18 received from the CPU 15, the results of the AE processing circuit 11, or the color temperature information of the subject obtained for AWB processing. The main subject is also identified using subject distance information of what is thought to be the subject received from the AF processing circuit 12, motion information of what is thought to be the subject received from the motion vector detection circuit 27, or camera shake information received from the blur detection circuit 13.

[0026] The motion vector detection circuit 27, based on the input digital image signal (reference image), calculates the motion vector for each divided region by performing a correlation calculation with the digital image signal (reference image) from the previous frame, according to the regions divided according to instructions from the CPU 15. The motion vector detection circuit 27 also determines the reliability of the detection based on the contrast of the reference image and the reference image. Specifically, the motion vector detection circuit 27 performs a difference calculation between the reference image and the reference image while shifting the reference image by a predetermined number of pixels in the horizontal and vertical directions. The pixel shift amount that yields the highest correlation (smallest difference amount) is taken as the amount of motion of the subject in that region. The motion vector detection circuit 27 takes the direction of the horizontal and vertical pixel shift as the direction of motion. This allows the motion vector for each region to be determined.

[0027] A motion detection means for detecting background movement and subject movement is provided by a motion vector detection circuit 27 that detects motion vectors for each region and a main subject detection circuit 26 that detects main subject information.

[0028] The image deformation and cropping circuit 28 performs image processing based on the output signals of the main subject detection circuit 26 or the motion vector detection circuit 27. The image processing by the image deformation and cropping circuit 28 includes deformation such as image rotation or cropping of a portion of the image in order to correct changes in information about the up, down, left, and right movement of the main subject on the image and the rotation of the imaging device 1 calculated by the CPU 15. The resulting image is recorded in a predetermined area of ​​the VRAM 6 or output to the image synthesis circuit 29.

[0029] For example, the position of the subject or background detected as the main subject may change on the screen due to the user's camera shake. If such blur occurs between multiple images obtained by continuous shooting, unnatural lines may appear in the combined image, or the resolution may decrease. Therefore, the motion vector detection circuit 27 detects the positional shift between multiple images, and the image deformation and cropping circuit 28 geometrically deforms the images to correct the positional shift. Then, the multiple images that have been aligned by geometric deformation are added together in the image synthesis circuit 29 to perform blur correction (positional shift correction) by image synthesis.

[0030] Next, with reference to Figure 2, the exposure control operation in the imaging device 1, namely the setting operation of the exposure coefficient (exposure time, aperture value, sensitivity) and the number of composite images, will be described. Figure 2 is a flowchart of the operation (processing) of the imaging device 1 in this embodiment. Each step in Figure 2 is mainly performed by the CPU 15.

[0031] First, in step S201, the CPU 15 initializes the variables used in the processing and moves the drive member to its initial position, and then checks the initial settings of the lens device 31 and the shooting mode. The CPU 15 also determines whether the lens device 31 is attached or not. If the lens device 31 is attached, it acquires information about the image stabilization lens 32, the focus lens 33, and the aperture 34. On the other hand, if the lens device 31 is not attached, the CPU 15 does not perform the process of acquiring information about the lens device 31 (lens information). The CPU 15 also checks the operation status of the shooting mode setting dial on the operation switch 18 and confirms the set shooting mode (for example, shutter speed priority, aperture priority, manual exposure mode, etc.) and the sensitivity (ISO sensitivity) at the time of shooting.

[0032] Next, in step S202, the CPU 15 calculates the limit exposure time by predicting the amount of blur during the exposure time from the acquired amount of blur. The CPU 15 acquires motion vector information for each region of the screen divided from the motion vector detection circuit 27 for a predetermined period, and then separates the subject movement to obtain camera shake information. The CPU 15 also receives camera shake information obtained by integrating the gyro signal information for each axis from the blur detection circuit 13.

[0033] The hand shake calculated from motion vector information consists of angular shake and shift shake, but the hand shake calculated from gyro signal information is solely due to angular shake. Therefore, CPU 15 separates and calculates the angular shake and shift shake.

[0034] The CPU 15 determines the subject distance during shooting from the AF processing based on the information of the main subject received from the main subject detection circuit 26, and calculates the amount of blur on the screen due to angular blur and shift blur. The CPU 15 also determines the corrected amount of camera shake for each axis direction from the vibration suppression performance of the blur correction means, which consists of the blur correction lens 32, control circuit 36, image sensor 3, sensor movement motor 25, and sensor movement control circuit 24. The CPU 15 then calculates the predicted amount of blur on the screen by calculating the square root of the sum of squares.

[0035] Based on the predicted amount of blur on the screen and the acceptable amount of blur, the limit exposure time can be determined as follows:

[0036] Limit exposure time (sec) = Allowable blur amount (μm) ÷ Absolute value of the slope where the absolute value of the predicted blur amount on the screen is the maximum value (μm / sec) The CPU 15 then fine-tunes the exposure time based on the above calculation results. The set exposure time may not be able to be set to the value of the above calculation result due to the limitations of the resolution of the image sensor 3 and the mechanical shutter. In that case, the CPU 15 sets the longest exposure time below the limit exposure time as the limit exposure time.

[0037] Next, in step S203, the CPU 15 performs AE processing from the main subject information received from the main subject detection circuit 26 and obtains an AE evaluation value. From the AE evaluation value (Ev), the CPU 15 determines the exposure coefficient (exposure time, aperture value, sensitivity) that will result in an appropriate captured image when no composite image is generated, using a program diagram according to the confirmed shooting mode.

[0038] Figure 3 shows a program diagram that reflects the fixed sensitivity (ISO 100 and 1600) and the limit exposure time for automatically setting the sensitivity between ISO 100 and 12800 when program mode is set. However, this diagram is just one example to illustrate this embodiment, and other diagrams can also be applied to this embodiment.

[0039] Here, the operation of this embodiment will be explained using the case where the sensitivity is automatically set between ISO 100 and 12800 as an example. According to Figure 3, in environments where the AE evaluation value is brighter than Ev16, the exposure time is changed by one stop, and up to Ev9, the exposure time and aperture value are changed by 0.5 stops in increments of one stop, and the ISO sensitivity is set to the lowest value of 100, thereby obtaining the appropriate amount of exposure.

[0040] From Ev9 to Ev1, where the exposure time is the limit, the exposure time is set to the limit, and the aperture value and ISO sensitivity are changed. If the maximum aperture is F2.8, the aperture value is changed by one stop while keeping the ISO sensitivity at 100 until Ev7. After that, the ISO sensitivity is increased by one stop to obtain the appropriate amount of exposure. From Ev1, where the ISO sensitivity is at its upper limit of ISO12800, the exposure time is increased to a longer duration.

[0041] In this case, since the exposure time will be longer than the limit, there is a possibility of image degradation due to camera shake. Therefore, multiple images are taken at exposure times shorter than the limit, and a composite image is generated from these images to create a good image with almost no degradation due to camera shake.

[0042] The number of images needed to create the composite image (the number of images to be composited) can be determined by "the exposure time required for an image to be properly captured without generating a composite image ÷ the limit exposure time". The exposure time for each individual image is the limit exposure time, and the ISO sensitivity for each individual image is the value shown in the P-diagram multiplied by the number of images to be composited. However, since the number of images to be composited is a natural number greater than or equal to 2, if the result of the above calculation is not a natural number, the exposure time for each individual image cannot be set to the limit exposure time. In this case, it is necessary to adjust the ISO sensitivity and exposure time.

[0043] Therefore, to account for the margin of vibration suppression, if the fractional part of the number of composite images is less than or equal to a predetermined value (for example, 0.3), the number of composite images is rounded down, and the exposure time for each individual image is made longer to the extent that image degradation due to camera shake does not occur, and the ISO sensitivity is adjusted accordingly. For example, in the case of Ev-0.2, the exposure time from the P-line diagram is 0.072 seconds, and the limit exposure time is 0.03125 seconds, so the number of composite images is calculated to be 2.3. Therefore, the number of composite images is set to 2, the exposure time for each individual image is 0.036 seconds, and the ISO sensitivity for each individual image is set to 25600. In this case, the ISO sensitivity in terms of SN is effectively equivalent to 18101.

[0044] Furthermore, in the case of Ev-2.2, the exposure time is calculated to be 0.29 seconds from the P-diagram, resulting in 9.19 composite images. Therefore, assuming 9 composite images, the exposure time for each image is 0.032 seconds, and the ISO sensitivity for each image is set to 115200. In this case, the ISO sensitivity in terms of signal-to-noise ratio is effectively equivalent to 38400.

[0045] Conversely, if the fractional part of the number of composite images exceeds a predetermined value (e.g., 0.3), the number of composite images is rounded up, the exposure time for each individual image is shortened, and the ISO sensitivity is adjusted accordingly. For example, in the case of Ev-0.8, the exposure time from the P-diagram is 0.11 seconds, and the number of composite images is calculated to be 3.48. Therefore, the number of composite images is set to 4, the exposure time for each individual image to 0.027 seconds, and the ISO sensitivity for each individual image to 51200. In this case, the ISO sensitivity in terms of SN is effectively equivalent to 25600.

[0046] Furthermore, in the case of Ev-3.3, the exposure time from the P-diagram is calculated to be 0.62 seconds, resulting in 19.7 composite images. Therefore, assuming 20 composite images, the exposure time for each image is 0.03 seconds, and the ISO sensitivity for each image is 256000. In this case, the ISO sensitivity in terms of signal-to-noise ratio is effectively equivalent to 57243.

[0047] This prevents image degradation due to camera shake without changing the exposure time and aperture value, which are related to photographic expression. The same applies when the ISO sensitivity is fixed at 100 or 1600. In bright environments where the AE evaluation value exceeds the upper limit of the aperture value, the exposure time is changed by one stop, and thereafter the exposure time and aperture value are changed by 0.5 stops in increments of one stop. When the exposure time reaches the limit exposure time, the aperture value is changed by 0.5 stops, and after the aperture value reaches its widest value, the exposure time is increased to a longer duration.

[0048] As mentioned above, this diagram is just one example. For example, when the sensitivity is automatically set between ISO 100 and 12800, the synthesis process may be started before the maximum ISO sensitivity is reached, the ISO sensitivity may be increased again after the number of synthesized images reaches a predetermined number, and preparations may be made to further increase the number of synthesized images after the maximum ISO sensitivity is reached. If it is determined that the generation of a synthesized image is necessary, the process proceeds from step S204 to S205. On the other hand, if it is determined that the generation of a synthesized image is not necessary, the process proceeds from step S204 to S221.

[0049] In step S205, the CPU 15 (acquisition means 15a) estimates (acquires) the degree of degradation of the texture (first region), which is the resolution of a relatively low-contrast, high-frequency subject, based on the sensitivity and number of composite images taken during shooting. If the CPU 15 determines that the texture has degraded as a result, it performs a process to change the number of composite images in step S207. Details of steps S205 and S207 will be described later.

[0050] Then, in step S208, the CPU 15 determines whether or not a release instruction (release request) has been issued. If a release instruction has been issued, the process proceeds to step S209, and the CPU 15 performs the exposure process.

[0051] In step S209, the CPU 15 checks the status of the release switch. Once the CPU 15 confirms that SW1 is ON, it drives the focus lens 33 to the focus position based on the AF processing result. The CPU 15 also controls the exposure time and aperture value based on the result of step S210. Simultaneously, the CPU 15 performs image stabilization processing. After that, the CPU 15 executes the actual exposure process.

[0052] Next, in step S210, the CPU 15 performs synthesis processing. The motion vector detection circuit 27 detects the positional misalignment between multiple images input from the A / D conversion circuit 5, and the image deformation and extraction circuit 28 geometrically deforms the images to correct the positional misalignment. Geometric deformation corrects the positional misalignment in the translational and rotational directions of the images by performing an affine transformation or projection transformation on the images. Geometric deformation requires that each pixel of the image before and after deformation be associated. When translating or rotating an image on a sub-pixel basis, each pixel cannot be associated one-to-one, so it is common to use pixels interpolated with multiple surrounding pixels to perform the association.

[0053] However, pixel interpolation compromises image resolution and degrades image quality. To prevent image quality degradation, it is desirable to minimize translational and rotational movements at the sub-pixel level. Regarding translational movements, limiting them to integer pixel units reduces alignment accuracy, but avoids image quality degradation due to pixel interpolation. Then, the multiple images aligned by geometric deformation are added together in the image synthesis circuit 29. This enables blur correction through image synthesis.

[0054] In step S221, the CPU 15 determines whether a release instruction (release request) has been issued. If a release instruction has been issued, in step S222, the CPU 15 performs the exposure process.

[0055] Here, with reference to Figures 4A and 4B, the texture degradation estimation and the processing of changing the number of composite images performed in steps S205 and S207 will be explained. Figures 4A and 4B are flowcharts showing the texture degradation estimation and processing of changing the number of composite images. Each step in Figures 4A and 4B is mainly performed by the CPU 15.

[0056] First, in step S401, the CPU 15 determines whether the set ISO sensitivity is lower than a predetermined value. Here, the set ISO sensitivity is either a value set by the user or a value set automatically. If the set ISO sensitivity is lower than the predetermined value, the CPU 15 determines that no texture degradation will occur due to the number of composite images, terminates this process, and proceeds to step S208 in Figure 2. In the case of the medium ISO sensitivity range and the high ISO sensitivity range, the CPU 15 evaluates the degree of texture degradation due to the number of composite images and, based on the result, determines whether or not to change the number of composite images in steps S402 and beyond.

[0057] In step S402, the CPU 15 reads characteristic data (table) that shows the relationship between texture degradation and the number of layers used for synthesis. The CPU 15 then determines the degree of texture degradation based on this characteristic data.

[0058] Figures 5(A) and (B) show an example of the relationship between texture evaluation value and the number of composite images. The table above is created from the relationship shown in the graphs in Figures 5(A) and (B). In Figures 5(A) and (B), the horizontal axis represents the number of composite images, including images without composites, and the vertical axis represents the texture evaluation value, which evaluates the texture value, which is the resolution of relatively low-contrast, high-frequency subjects. The higher this value, the better the low-contrast, high-frequency subjects are reproduced in the image. Furthermore, if the value is above a certain threshold, the texture degradation is minimal and the image is judged to be good.

[0059] The dashed lines in Figure 5(A) represent the texture evaluation values ​​for each number of composite images at low ISO sensitivities (e.g., less than 400), specifically ISO 100 (exposure times of 1 / 30 second and 1 / 8 second; all other ISO sensitivities use an exposure time of 1 / 8 second) and ISO 200. Figure 5(B) shows the average values. Compared to the average values ​​for the entire ISO sensitivity range shown by the dashed line, the texture evaluation values ​​themselves are higher, and the degree of degradation due to the number of composite images is also smaller. Note that the range of low ISO sensitivities is determined for each image sensor or imaging device, as it depends on the characteristics of the image sensor.

[0060] Texture evaluation values ​​generally decrease as the number of images combined increases. This is due to calculation errors in the alignment of each image during the combination process and the fact that alignment is performed at the pixel level. However, in the low ISO sensitivity range, the alignment position is calculated relatively accurately because the images have low noise, and the texture evaluation values ​​of each combined image are high due to the low noise. Therefore, as mentioned above, it is determined that no texture degradation occurs.

[0061] The solid lines in Figure 5(A) show the texture evaluation values ​​for each number of composite images at medium ISO sensitivities (e.g., 400 or higher, less than 5000) of ISO 400, 800, and 3200. This range is determined for each sensor or imaging device, similar to the low ISO range. Figure 5(B) shows the average values. Compared to the average value for the entire ISO sensitivity range, the texture evaluation value itself is lower, and the degree of degradation due to the number of composite images is also greater. Because the images contain a certain amount of noise, calculating the alignment position is more difficult than in the low ISO sensitivity range, and the texture evaluation value of each composite image is also lower than the average value for the entire ISO sensitivity range. For this reason, the degradation due to the number of composite images has a greater impact on image quality.

[0062] The dashed lines in Figure 5(A) show the texture evaluation values ​​for each number of composite images at high ISO sensitivities (e.g., above 5000), namely ISO 6400 and 25600. This range is determined for each image sensor or imaging device, similar to the low ISO range. Figure 5(B) shows the average values. Although the texture evaluation values ​​themselves are lower than the average value for the entire ISO sensitivity range, the degree of degradation due to the number of composite images is small. This is because, in the high ISO sensitivity range, the texture is inherently low due to noise reduction processing. Also, when using high ISO sensitivities, the illumination is often very low and long exposures are required, so reducing the number of composite images may result in excessively long exposure times for each individual image. Therefore, no changes to the number of composite images are made except for the processing in step S404, which will be described later.

[0063] Here, we will explain how degradation is determined based on texture evaluation values. This determination is made using the degree to which the texture evaluation value decreases due to an increase in the number of composite images, and the acceptable value of the texture evaluation value. The specific method is explained using Figures 5(A) and (B) as examples.

[0064] In the medium ISO sensitivity range shown in Figures 5(A) and (B), the acceptable value for texture evaluation is defined as a value that is 20% lower than the texture evaluation value without image synthesis, based on the results of the sensory evaluation. At ISO 400, the evaluation value clearly decreases when the number of synthesized images exceeds 8, and at ISO 800, it decreases with the number of synthesized images, and at 16 images or more, although the difference in values ​​is small, it is recognized that the absolute value has also decreased, even though it exceeds the acceptable value. At ISO 3200, the absolute value of the evaluation value becomes low at 4 synthesized images, and the decrease becomes more gradual thereafter, but it approaches the acceptable value at 16 images and falls below the acceptable value at 32 images. The average value also approaches the acceptable value at 16 images and falls below the acceptable value at 32 images.

[0065] Using this data as a reference, we will determine the number of lossless composite images for each ISO sensitivity. Therefore, for ISO 400, the number of lossless composite images will be 8, for ISO 800, it will be 16, and for ISO 3200, it will be 16.

[0066] Similarly, in the high ISO sensitivity range, for both ISO 6400 and ISO 25600, the absolute value of the evaluation value becomes low with 4 composite images, and the decrease becomes gradual thereafter, approaching the acceptable value at 16 images and falling below the acceptable value at 32 images. Therefore, for both ISO 6400 and ISO 25600, the number of non-degrading composite images is set to 16.

[0067] The same determination is made for ISO sensitivities not shown in Figures 5(A) and (B). However, for ISO sensitivities for which no characteristic data is available, the characteristic data is obtained by linear interpolation from the data of the preceding and succeeding ISO sensitivities, and the determination is made in the same manner. For example, in the above-mentioned medium ISO sensitivity range, the number of non-degraded composite images is 16 for ISO sensitivities from 800 to 3200.

[0068] In step S403, the CPU 15 determines whether the number of composite sheets set in step S203 is less than or equal to the number of composite sheets without degradation. If the number of composite sheets is less than or equal to the number of composite sheets without degradation, this process ends and the process proceeds to step S208. On the other hand, if the number of composite sheets exceeds the number of composite sheets without degradation, the process proceeds to step S404.

[0069] In step S404, the CPU 15 modifies the exposure time and the number of composite images as follows. Here, the CPU attempts to improve the texture degradation by reducing the number of composite images by one, which is a contributing factor to the degradation. However, this alone may increase the noise in the composite image, so the exposure time is slightly increased to a level where the image quality degradation due to camera shake is negligible.

[0070] The exposure time set in step S203 may not be set to the value calculated above due to the resolution limitations of the image sensor 3 and the mechanical shutter. In such cases, the exposure time may be set to a shorter duration than the desired exposure time, such as the longest exposure time below the limit exposure time. In such cases, the exposure time should be set to a duration that is longer than the desired exposure time in increments of the resolution. The ISO sensitivity for a single shot should also be adjusted accordingly.

[0071] The specific operation will be explained using the P-diagrams shown in Figure 6, which are fixed at ISO sensitivities of 1600 and 6400, as an example. The P-diagram for ISO sensitivity 6400, which corresponds to the high ISO sensitivity range, is shown by the dashed line in Figure 6.

[0072] When the exposure time is 0.6 seconds and the limit exposure time is 1 / 32 of a second, the number of composite images is calculated to be 19.2. Therefore, in step S203, the number of composite images is set to 19 and the exposure time of each image is set to 0.03158 seconds. However, this time (0.03158 seconds) cannot be set due to resolution constraints, so it is set to the longest exposure time (0.03125 seconds) that is less than or equal to the limit exposure time. Therefore, the exposure time of one image is changed to the time (0.03516 seconds) that is closest to the set time among the available times that are greater than the set time (0.03158 seconds), and the ISO sensitivity of the single image is changed accordingly from 122881 to 109228. As a result, the effective ISO sensitivity after compositing changes from 28190 to 25745, and the signal-to-noise ratio is also improved.

[0073] The P-diagram for ISO 1600, which corresponds to the medium ISO sensitivity range, is shown by the dashed line in Figure 6.

[0074] If the exposure time is 0.7 seconds and the limit exposure time is 1 / 32 second, the number of composite images is calculated to be 22.4. Therefore, in step S203, the number of composite images is set to 23, and the exposure time for each image is set to 0.03044 seconds. However, this time (0.03044 seconds) cannot be set due to resolution constraints, so it is set to the longest exposure time (0.02734 seconds) that is less than or equal to the limit exposure time. Therefore, the exposure time for one image is changed to the closest time (0.03125 seconds) among the available time ranges that exceed the set time (0.02734 seconds), and the ISO sensitivity of the single image is changed accordingly from 40970 to 35849. As a result, the effective ISO sensitivity after compositing changes from 8543 to 7643, and the signal-to-noise ratio is also improved.

[0075] In step S405, the CPU 15 determines whether the ISO sensitivity is within the high ISO sensitivity range. If the ISO sensitivity is within the high ISO sensitivity range, this process ends and the process proceeds to step S208 in Figure 2.

[0076] Next, in step S406, similar to step S404, the CPU 15 determines whether the currently set number of composite sheets is less than or equal to the non-degradation composite sheet. If the number of composite sheets is less than or equal to the non-degradation composite sheet, this process ends and the process proceeds to step S208. On the other hand, if the number of composite sheets exceeds the non-degradation composite sheet, the process proceeds to step S407.

[0077] In step S407, the CPU 15 checks the set shooting mode and confirms the operation status of the setting dial. If the set shooting mode is a mode in which it is possible to set the exposure time to a short duration (such as program mode or aperture priority mode, where the user does not directly specify the exposure time), the process proceeds from step S408 to step S409. On the other hand, if the set shooting mode is a mode in which this process is not possible (such as shutter speed priority mode or manual exposure mode, where the user directly sets the exposure time), the process proceeds from step S408 to step S410.

[0078] In step S409, the CPU 15 sets the new number of composite images to 25% more than the number of non-degradable composite images set in step S402, without changing the exposure time for each shot. This is because an increase of about 25% compared to the number set in step S402 is considered to result in an acceptable degree of texture degradation. The CPU 15 adjusts the ISO sensitivity accordingly. Since the number of composite images has decreased from the value set in step S203, the total exposure time, calculated as number of composite images × exposure time per image, becomes shorter, so the ISO sensitivity is increased to compensate. If the number of composite images before the change is Np and the number of composite images after the change is Nn, the ISO sensitivity becomes (Np ÷ Nn) times.

[0079] In step S412, the CPU 15 determines whether the texture evaluation value has decreased (i.e., whether there is no texture degradation) due to the change in the number of textures and ISO sensitivity, based on the characteristic data showing the relationship between texture degradation and the number of composite textures read in step S402. If it is determined that the texture evaluation value has not decreased, this process ends and the process proceeds to step S208 in Figure 2. On the other hand, if it is determined that the texture evaluation value has decreased, the process proceeds to step S421.

[0080] Here, we will explain the determination method in step S412 using Figure 7 as an example. Figure 7 shows an example of the relationship between the texture evaluation value and the number of composite images in the medium ISO sensitivity range. In Figure 7, the solid line represents the texture evaluation value for each number of composite images at ISO sensitivity 1600, the dashed line at ISO sensitivity 1920, and the dotted line at ISO sensitivity 2560.

[0081] For example, if the number of images to be composited and the ISO sensitivity are set to 32 and 1600 in step S203, the number of images to be composited and the ISO sensitivity are changed to 20 and 2560 in the process of S409. Reading the respective texture evaluation values ​​from the graph, we can see that they are 510 and 525, respectively, indicating an improvement in the texture evaluation value. In this case, it is determined that the texture evaluation value has not decreased, and this process is terminated, proceeding to step S208 in Figure 2.

[0082] On the other hand, if the number of images to be composited and the ISO sensitivity are set to 24 and 1600 in step S203, the number of images to be composited and the ISO sensitivity are changed to 20 and 1920 in the processing of S409. Reading the respective texture evaluation values ​​from the graph, we can see that they are 550 and 546, respectively, indicating a decrease in the texture evaluation value. In this case, it is determined that the texture evaluation value has decreased, and the process proceeds to step S421.

[0083] In step S410, the CPU 15 determines whether it is possible to increase the exposure time for a single image. If it is possible to increase the exposure time, the process proceeds to step S411. On the other hand, if it is not possible to increase the exposure time, the process proceeds to step S412.

[0084] In step S202, the CPU 15 determines the limit exposure time using the maximum slope of the predicted blur amount on the screen. However, such a predicted maximum blur amount rarely occurs. Therefore, here the limit exposure time is determined using a realistic predicted blur amount, and if that time allows for a reduction of one or more composite images, it is determined that it is possible to increase the exposure time of a single image.

[0085] The realistic predicted amount of deviation is determined as follows. The predicted amount of deviation on the screen is expressed as a function of time F(t) over a given time interval. Differentiating this function with respect to time gives the time function G(t) (=dF(t) / dt) of the slope (μm / sec). After converting this function to a function of absolute value, its average value Slpm is calculated as follows.

[0086] Slpm=∫ t t 2 1(abs(G(t))dt÷(t1-to) (∫ t t 2 1(abs(G(t))dt represents the definite integral of the absolute value of G(t) from t1 to t2). Using the value Slpm and the average value (Slpx) of the absolute value (μm / sec) of the slope with the maximum absolute value (μm / sec) of the predicted amount of blur on the screen, the limit exposure time is calculated as follows.

[0087] Limit exposure time (sec) = Allowable blur (μm) ÷ Slpx (μm / sec) However, if that value exceeds 1.5 times the time obtained in step S202, the limit exposure time will be 1.5 times that value.

[0088] The CPU 15 uses this limit exposure time to reset the exposure time and the number of images to be combined in step S411. Then, in step S412, the CPU 15 determines whether the reset number of images to be combined is less than or equal to the number of images to be combined without degradation. If the number of images to be combined is less than or equal to the number of images to be combined without degradation, this process ends and the process proceeds to step S208. On the other hand, if the number of images to be combined exceeds the number of images to be combined without degradation, the process proceeds to step S421.

[0089] In step S421, the CPU 15 temporarily records the exposure coefficient (exposure time per image, number of images combined, aperture value, ISO sensitivity) in a storage unit such as internal memory. Subsequently, in step S422, the CPU 15 determines whether the set ISO sensitivity is in an ISO sensitivity range where the influence of the number of images combined is small.

[0090] For example, at an ISO sensitivity of 400, the number of lossless composite images is 8. According to the graph in Figure 5(A), the region between 8 and 16 composite images shows a significant decrease in texture evaluation value due to the number of composite images, indicating that it is affected by an increase in the number of composite images. Therefore, it is determined that this is not an ISO sensitivity region where the effect is small, and the process proceeds from step S422 to step S431.

[0091] On the other hand, for ISO sensitivities of 800 and 3200, the number of lossless composite images is 16. In the range of 16 to 32 composite images, the decrease in texture evaluation value due to the number of composite images is relatively small, and the effect of increasing the number of composite images is less pronounced. Therefore, it is determined that this is an ISO sensitivity range with little effect, and the process proceeds from step S422 to step S423.

[0092] In step S423, the CPU 15 sets the number of composite images to be (+1) the number of non-degradable composite images. Then, in step S424, the CPU 15 determines whether the scene has high alignment accuracy for each image when generating the composite image. If the CPU 15 determines that the scene has high alignment accuracy, it sets the number of composite images to a predetermined constant multiple (for example, 1.1 times) and rounds it to the nearest integer. Here, a scene with high alignment accuracy is, for example, one where "both the contrast of the main subject and the background are high" or "the reliability of motion vector detection in each region obtained by the vector detection circuit 27 is high," and these can be determined by comparing the calculated index with a predetermined value. In other words, the CPU 15 may change the number of composite images according to at least one of the contrast of the subject, the contrast of the background, and the reliability of motion vector detection. The reason for increasing the number of composite images in this way is that if the increase is of this magnitude compared to the number of non-degradable composite images set in step S402, it can be determined that the impact on image quality is small.

[0093] Next, in step S425, the CPU 15 compares the number of composite sheets increased in steps S423-S424 (Nadd) with the number of composite sheets set in step S203 (Norg). If Nadd ≤ Norg, the number of composite sheets is set to Nadd, this process ends, and the process proceeds to step S208 in Figure 2. On the other hand, if Nadd > Norg, the process proceeds to step S431.

[0094] In step S431, the CPU 15 determines whether it has previously changed the exposure time and the number of images to be combined (processing in S411). If the exposure time and the number of images to be combined have already been changed, the process proceeds to step S434. On the other hand, if the exposure time and the number of images to be combined have not been changed, the process proceeds to step S432.

[0095] In step S432, the CPU 15 determines whether it is possible to increase the exposure time for a single image (make it longer) in the same manner as in step S410. If it is possible to increase the exposure time, the CPU 15 changes the limit exposure time (sec) in the same manner as in step S412 and resets the exposure time for a single image and the number of images to be combined. Then, in step S433, the CPU 15 determines whether the reset number of images to be combined is less than or equal to the number of images to be combined without degradation. If the number of images to be combined exceeds the number of images to be combined without degradation, the process proceeds to step S434. On the other hand, if the number of images to be combined is less than or equal to the number of images to be combined without degradation, the process proceeds to step S439, and then to step S208 in Figure 2. However, regarding the number of images to be combined without degradation here, if the processing in steps S423 to S424 has been performed, the resulting increased number of images to be combined without degradation is used.

[0096] The processing from step S434 onward cannot achieve the same number of images for non-degraded synthesis (including after processing in steps S423-S424), but it is a process to ensure a balanced image quality, taking into account factors such as the signal-to-noise ratio (SNR).

[0097] First, in step S434, the CPU 15 restores the exposure coefficients (single exposure time, number of composite images, aperture value, ISO sensitivity) recorded in step S421. Next, in step S435, the CPU 15 decreases the set number of composite images by 1. Then, in step S436, the CPU 15 checks the set shooting mode and, based on the operation status of the setting dial, determines whether the set shooting mode is a mode that allows for a short exposure time (whether it is possible to change the ISO sensitivity (to a shorter exposure time)). If the set shooting mode is a mode that allows for a short exposure time, the process proceeds to step S437. On the other hand, if the set shooting mode is a mode that does not allow for a short exposure time, this process ends and the process proceeds to step S208 in Figure 2.

[0098] In step S437, the CPU 15 determines whether it is possible to increase the ISO sensitivity beyond what was done in step S409, taking into account the tolerance for noise and the spatial frequency of the subject in the area considered to be in focus. If the maximum ISO sensitivity in the case of automatic ISO sensitivity setting is set by the user at least one stop higher than the factory setting, it is determined that it is possible to increase the ISO sensitivity. Alternatively, if the maximum frequency of the maximum spatial frequency of the subject in the area of ​​the image where AF processing has been performed is below a predetermined frequency, it is determined that it is possible to increase the ISO sensitivity.

[0099] The ISO sensitivity is increased by half the number of stops higher than the factory setting at this point. For example, if the factory setting is 12800, the user setting is 51200, and the current setting is 3200, the difference between the factory setting and the user setting is 2 stops, so the current setting of 3200 is shifted by 1 stop, changing the ISO sensitivity to 64000. Next, if the maximum frequency of the spatial frequency maxima of the subject in the image area where AF processing has been performed is below a predetermined frequency, the ISO sensitivity is increased according to that frequency. The frequency components contained in the subject are determined by FFT, etc., and if the frequency of the maximum value is below a predetermined frequency (for example, one-quarter of the Nyquist frequency), the ISO sensitivity is increased according to the relationship between that frequency and a predetermined threshold frequency.

[0100] The specific setting method will be explained using Figures 8(A) and 8(B) as examples. Figures 8(A) and 8(B) show examples of frequency components contained in a subject. In the example shown in Figure 8(A), if the maximum frequency (ft1a) of the subject's spatial frequency maxima exceeds a predetermined threshold frequency (fth), the ISO sensitivity will not be increased further.

[0101] In the example shown by the solid line in Figure 8(B), the maximum frequency of the spatial frequency maxima of the subject (ft1b) exceeds fth, but since its magnitude is below the threshold (Bth) for the magnitude that is thought to affect image quality, it is determined that there are no maximal frequency components that exceed fth. Therefore, corresponding to the maximal frequency component (ft0b) that is less than or equal to fth, ft0b is less than or equal to the frequency that is one-third of the threshold (fth) (fth / 3), so the ISO sensitivity is increased by one stop.

[0102] In the example shown by the dashed line in Figure 8(B), the maximum frequency of the spatial frequency maxima of the subject (ft1c) exceeds fth, but its magnitude is less than Bth, so it is determined that there are no frequency components of the maxima that exceed fth. Therefore, the ISO sensitivity is increased according to the maximum frequency component of the maxima that is less than or equal to fth (ft0c). Since ft0c is less than or equal to the frequency that is half the threshold (fth) (fth / 2) (greater than fth / 3), the ISO sensitivity is increased by 2 / 3 stop.

[0103] Although not shown as an example, if the maximum frequency component of a maximum value below fth is above the frequency that is half the threshold (fth) (fth / 2) but below the threshold (fth) (fth / 3), the ISO sensitivity should be increased by 1 / 3 stop.

[0104] In step S437, the CPU 15 changes the number of composite images and the total exposure time according to the ISO sensitivity set in step S436. If the ISO sensitivity before setting in step S436 is set to ISObe and the ISO sensitivity after setting is set to ISOaf, the number of composite images and the total exposure time are calculated as follows.

[0105] Number of composite images = Number of composite images before change × ISObe ÷ ISOaf Total exposure time = Exposure time per image × Number of images combined before modification × ISObe ÷ ISOaf Note that the exposure time per image and the number of images to be combined before modification are the values ​​read in step 434. Then proceed to step S208 in Figure 2.

[0106] On the other hand, in step S439, since the exposure time for one image is increased in S432, there is a possibility that the blur of the images used for synthesis will increase. Therefore, the control of the blur correction means is changed to correct large low-frequency blurs that affect image quality more effectively. In other words, the blur control means of the CPU 15 changes the control of the blur correction means, for example, in accordance with the change in the number of images to be synthesized.

[0107] In the control before the long exposure time, if the stroke of the correction means is less than half, the blur correction means is moved by 100% of the difference between the target control position and the current position. Similarly, if the stroke is less than two-thirds, the blur correction means is moved by 90% of the difference; if the stroke is less than three-quarters, it is moved by 75% of the difference; and if it exceeds that, it is moved by 60% of the difference.

[0108] On the other hand, in the control after the long exposure time, if the stroke of the correction means is less than half, the shake correction means is moved by 90% of the difference between the target control position and the current position. Similarly, if the stroke is less than two-thirds, the shake correction means is moved by 80% of the difference; if the stroke is less than three-quarters, it is moved by 60% of the difference; and if it exceeds that, it is moved by 50% of the difference. Then, the process proceeds to step S208 in Figure 2.

[0109] In this embodiment, the degree of texture degradation is estimated using an index (texture evaluation value) representing the resolution of a relatively low-contrast, high-frequency subject and the number of composite images, and the processing related to setting the number of composite images is changed according to the ISO sensitivity. This makes it possible to generate a composite image with suppressed texture degradation.

[0110] As described above, when generating a composite image using multiple images, the acquisition means 15a acquires the degree of resolution degradation of the first region in at least one of the multiple images. The determination means 15b determines the number of images to be used to generate the composite image according to the degree of degradation.

[0111] Preferably, the determination means changes the number of composite images from a first number to a second number depending on the degree of degradation. Also preferably, at least one of the multiple images includes a first region and a second region, where the first region is a subject region with lower contrast and higher frequency than the second region. Also preferably, the acquisition means 15a acquires the degree of degradation using information regarding the sensitivity and the number of composite images at the time of shooting.

[0112] Preferably, the acquisition means 15a determines the number of composite images based on the appropriate exposure time when no composite image is generated from the measurement value of the photometric means, and the limit exposure time at which blur is determined not to occur. More preferably, the acquisition means 15a acquires the limit exposure time using at least one of the output signal of the blur detection means, the focal length, and the subject distance. Even more preferably, the determination means 15b determines the number of composite images using the limit exposure time. Also preferably, depending on the degree of degradation, the determination means 15b sets the exposure time for each image used for composite imaging to be longer than the limit exposure time, and determines the number of composite images.

[0113] Preferably, the determination means 15b determines the sensitivity during shooting according to the degree of degradation. Also preferably, the determination means 15b determines the sensitivity during shooting according to the tolerance for noise in at least one of the multiple images, according to the upper limit of the automatically set sensitivity set by the user and at least one of the spatial frequencies of the subject in the region determined to be in focus. Also preferably, the criteria for changing the number of composite images differ according to the sensitivity during shooting.

[0114] Although this embodiment describes an interchangeable-lens imaging device as an example, it is not limited to this. This embodiment is broadly applicable to electronic devices equipped with camera functions, such as digital video cameras or smartphones.

[0115] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0116] Each embodiment of the disclosure includes the following configuration and method. (Composition 1) When generating a composite image using multiple images, an acquisition means for acquiring the degree of resolution degradation of a first region in at least one of the multiple images, A control device characterized by having a determination means for determining the number of composite images to be used to generate the composite image, according to the degree of degradation. (Configuration 2) The control device according to configuration 1, characterized in that the determination means changes the number of composite sheets from a first number to a second number according to the degree of deterioration. (Composition 3) At least one of the aforementioned plurality of images includes the first region and the second region, The control device according to configuration 1 or 2, characterized in that the first region is a subject region with lower contrast and higher frequency than the second region. (Composition 4) The control device according to any one of configurations 1 to 3, characterized in that the acquisition means acquires the degree of degradation using information regarding the sensitivity at the time of shooting and the number of composite images. (Composition 5) The control device according to any one of configurations 1 to 4, characterized in that the acquisition means determines the number of composite images according to an appropriate exposure time when no composite image is generated from the measurement value of the photometric means and a limit exposure time at which blurring is determined not to occur. (Composition 6) The control device according to configuration 5, characterized in that the acquisition means acquires the limit exposure time using at least one of the output signal of the blur detection means, the focal length, and the subject distance. (Composition 7) The control device according to configuration 5 or 6, characterized in that the determination means determines the number of composite images using the limit exposure time. (Composition 8) The control device according to any one of configurations 5 to 7, characterized in that the determination means sets the exposure time for capturing each image used for synthesis to be longer than the limit exposure time, according to the degree of degradation, and determines the number of images to be synthesized. (Composition 9) The control device according to any one of configurations 1 to 8, characterized in that the determination means determines the sensitivity during shooting according to the degree of degradation. (Composition 10) The control device according to any one of configurations 1 to 8, wherein the determination means determines the sensitivity during shooting according to at least one of the tolerance for noise in at least one of the plurality of images, which is set by the user, and the spatial frequency of the subject in the region where it is determined to be in focus. (Composition 11) The control device according to any one of configurations 1 to 10, characterized in that the determination means determines the number of composite images according to at least one of the contrast of the subject, the contrast of the background, and the reliability of motion vector detection. (Composition 12) Image stabilization means to correct blur, A control device according to any one of configurations 1 to 11, further comprising control means for changing the control of the blur correction means in accordance with the change in the number of composite images. (Composition 13) A control device according to any one of configurations 1 to 12, characterized in that the criteria for changing the number of composite images differ depending on the sensitivity during shooting. (Composition 14) An imaging device characterized by having a control device according to any one of configurations 1 to 13 and an image sensor. (Method 1) When generating a composite image using multiple images, the steps include obtaining the degree of resolution degradation of a first region in at least one of the multiple images, A control method characterized by comprising the step of determining the number of composite images used to generate the composite image according to the degree of degradation. (Composition 15) A program characterized by causing a computer to execute the control method described in Method 1.

[0117] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of symbols]

[0118] 15. CPU (Control Unit) 15a Means of acquisition 15b Determining means

Claims

1. When generating a composite image using multiple images, an acquisition means for acquiring the degree of resolution degradation of a first region in at least one of the multiple images, A control device characterized by having a determination means for determining the number of composite images to be used to generate the composite image, according to the degree of degradation.

2. The control device according to claim 1, characterized in that the determination means changes the number of composite sheets from a first number to a second number according to the degree of deterioration.

3. At least one of the aforementioned plurality of images includes the first region and the second region, The control device according to claim 1, characterized in that the first region is a subject region with lower contrast and higher frequency than the second region.

4. The control device according to claim 1, characterized in that the acquisition means acquires the degree of degradation using information regarding the sensitivity at the time of shooting and the number of composite images.

5. The control device according to claim 1, characterized in that the acquisition means determines the number of composite images according to the appropriate exposure time when the composite image is not generated from the measurement value of the photometric means and the limit exposure time at which blurring is determined not to occur.

6. The control device according to claim 5, characterized in that the acquisition means acquires the limit exposure time using at least one of the output signal of the blur detection means, the focal length, and the subject distance.

7. The control device according to claim 5, characterized in that the determination means determines the number of composite images using the limit exposure time.

8. The control device according to claim 5, characterized in that the determination means determines the exposure time for taking each image used for synthesis to be longer than the limit exposure time, according to the degree of degradation, and determines the number of images to be synthesized.

9. The control device according to claim 1, characterized in that the determination means determines the sensitivity during shooting according to the degree of degradation.

10. The control device according to claim 1, wherein the determination means determines the sensitivity during shooting according to at least one of the tolerance for noise in at least one of the plurality of images, which is set by the user, and the spatial frequency of the subject in the region where it is determined to be in focus.

11. The control device according to claim 1, characterized in that the determination means determines the number of composite images according to at least one of the contrast of the subject, the contrast of the background, and the reliability of motion vector detection.

12. Image stabilization means to correct blur, The control device according to claim 1, further comprising control means for changing the control of the blur correction means in accordance with the change in the number of composite images.

13. The control device according to claim 1, characterized in that the criteria for changing the number of composite images differ depending on the sensitivity during shooting.

14. An imaging device comprising a control device according to any one of claims 1 to 13 and an image sensor.

15. When generating a composite image using multiple images, the steps include obtaining the degree of resolution degradation of a first region in at least one of the multiple images, A control method characterized by comprising the step of determining the number of composite images used to generate the composite image according to the degree of degradation.

16. A program characterized by causing a computer to execute the control method described in claim 15.

Images