Imaging apparatus and method for controlling the same

Abstract

To provide an imaging apparatus capable of acquiring a higher quality image by reducing the influence of an image magnification fluctuation.SOLUTION: An imaging system includes an imaging apparatus 1 and a lens device 31 and performs focus adjustment by the control of the focus adjustment operation of an imaging optical system included in the lens device 31. The imaging apparatus 1 acquires AF (autofocus) information or gyro information and calculates the shake amount in the optical axis direction of the imaging optical system (S202). The imaging apparatus 1 detects an image magnification change when performing the focus adjustment (S205) and calculates the degree of recovery of image resolution, to perform setting for post-photographing processing (S206). The imaging apparatus 1 determines whether to fix or adjust the focus on the basis of an image magnification change amount and the degree of recovery of the image resolution (S207).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for correcting image blur in an imaging device. [Background technology]

[0002] When performing macro photography or the like with an imaging device, one method of correcting image blur caused by shaking in the optical axis direction is to perform focus adjustment and then image synthesis. Patent Document 1 discloses a technology that can reduce image degradation caused by environmental changes during photography. A control means of the imaging device adjusts the imaging plane position from the focus plane position by a correction amount based on imaging environment information, and the imaging element outputs a captured image at the adjusted imaging plane position. An image processing means performs image restoration processing on the captured image using an image restoration filter corresponding to the imaging plane position.

[0003] Patent Document 2 discloses a technique for acquiring multiple pieces of image data while changing the focal position of a focus lens. The imaging device includes a lens control unit that moves the lens to control the focal position, and a distance variation detection unit detects distance variations between the subject and the imaging unit in the direction of the lens optical axis. The lens focal position is corrected to coincide with the target focal plane according to the amount of camera shake detected by the distance variation detection unit. The shooting instruction unit changes shooting instructions according to the detection result of the distance variation detection unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-052104 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-005224 [Patent Document 3] Patent No. 3449860 Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional techniques, when image blur occurs due to shaking in the optical axis direction of the imaging optical system, if focus adjustment is performed to correct the image blur, the resolution of the captured image may decrease due to the influence of fluctuations in image magnification. Patent Documents 1 and 2 do not disclose any technique for determining whether to perform focus adjustment to correct image blur in a situation where it is necessary to correct image blur caused by shaking in the optical axis direction of the imaging optical system (hereinafter referred to as "image blur correction").

[0006] In the technology disclosed in Patent Document 1, an image restoration process is performed on a captured image using an image restoration filter that corresponds to the photographing plane position after focus correction, so there is a possibility that sufficient resolution cannot be obtained for images that are difficult to restore. Also, in the technology disclosed in Patent Document 2, the lens focal position is corrected to coincide with the target focal plane according to the detected amount of camera shake, so there is a possibility that the resolution will decrease due to the influence of changes in image magnification.

[0007] An object of the present invention is to provide an imaging apparatus that can reduce the influence of image magnification fluctuations and obtain higher quality images. [Means for solving the problem]

[0008] An imaging device according to an embodiment of the present invention includes an imaging means, a detection means for detecting the amount of blur in the optical axis direction of the imaging optical system, a control means for controlling focus adjustment by a focus adjustment operation of the imaging optical system, and a calculation means for calculating, from the detected amount of blur, the amount of change in image magnification of an image acquired by the imaging means, and the control means controls the focus adjustment in accordance with the calculated amount of change in image magnification. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an imaging device that can reduce the influence of image magnification fluctuations and obtain higher quality images. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a configuration according to a first embodiment; [Figure 2] 10 is a flowchart illustrating processing in an embodiment. [Figure 3] 3 is a flowchart illustrating the process of S206 in FIG. 2. [Figure 4] 4 is a flowchart illustrating processing following FIG. 3. [Figure 5] 3 is a flowchart illustrating the process of S210 in FIG. 2. [Figure 6] FIG. 2 is a schematic diagram partially illustrating a pixel configuration of an imaging element. [Figure 7] FIG. 10 is a block diagram showing a configuration example according to a second embodiment. [Figure 8] 10A and 10B are explanatory diagrams of a method for calculating the amount of shake in the optical axis direction using gyro information. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will be described in detail with reference to the accompanying drawings. When an imaging device performs focus adjustment to correct image blur, the image may become multiplexed due to the influence of image magnification fluctuation, which may result in a decrease in resolution. Furthermore, when moving the lens in the optical axis direction of the imaging optical system, measures must be taken to address the following issues in addition to image magnification fluctuation. - If the optical axis is misaligned or there is backlash during lens operation, the resolution of the captured image will decrease. Exposure cannot be made while the lens is moving to adjust the focus, which increases the time it takes to complete the entire exposure. -Power is required to drive the lens. Below, we will explain a technology for reducing the effects of image magnification fluctuations and the like and further improving the image quality of captured images by determining whether or not to perform focus adjustment when blurring occurs in the optical axis direction of the imaging optical system.

[0012] [First embodiment] 1 is a block diagram showing an example of the configuration of an imaging system according to this embodiment. This shows an example of a lens-interchangeable system in which a lens device 31 can be attached to the main body of an imaging device 1. The imaging device 1 has a lens mount 2 for attaching the lens device 31. Light that passes through the imaging optical system of the lens device 31 forms an image of a subject on an imaging element 3.

[0013] The image sensor 3 performs photoelectric conversion and outputs an image signal, etc. For example, the image sensor 3 is a pupil-splitting type image sensor capable of image-plane phase-difference AF (autofocus), and has pixel groups with different exit pupil positions. The image sensor 3 is equipped with multiple microlenses and multiple photoelectric conversion units corresponding to each microlens. The amount of defocus can be detected by reading out phase-difference signals from the multiple photoelectric conversion units. In addition, an image signal (such as a still image signal for recording) can be obtained by adding and reading out signals from adjacent photoelectric conversion units.

[0014] The imaging circuit 4 receives the electrical signal photoelectrically converted by the imaging element 3 and performs various processes to generate an analog image signal. The A / D conversion circuit 5 converts the analog image signal generated by the imaging circuit 4 into a digital image signal. The memory 6 is a buffer memory that temporarily stores the digital image signal output from the A / D conversion circuit 5, and will hereinafter also be referred to as "VRAM." The D / A conversion circuit 7 reads the digital image signal from the VRAM 6 and converts it into an analog signal, as well as converting it into an image signal in a form suitable for playback output. The image display device 8 is a liquid crystal display device (hereinafter also referred to as "LCD") that receives the image signal from the D / A conversion circuit 7 and displays an image.

[0015] The compression / decompression circuit 9 is connected to the VRAM 6 and the image storage memory 10. The image storage memory 10 is a fixed semiconductor memory such as a flash memory, or a card- or stick-shaped card-type semiconductor memory that is detachable from the imaging device. Alternatively, it can be applied in various forms, such as a magnetic storage medium such as a hard disk or flexible disk.

[0016] The compression / decompression circuit 9 has a compression circuit and an decompression circuit. The compression circuit acquires image data temporarily stored in the VRAM 6 and performs compression processing and encoding processing on the image data to make it suitable for storage in the image storage memory 10. The decompression circuit also performs decoding processing and decompression processing on the image data stored in the image storage memory 10 to make it suitable for playback display, etc.

[0017] The AE processing circuit 11 receives the output signal of the A / D conversion circuit 5 and performs automatic exposure (AE) processing. The AF processing circuit 12 receives the output signal of the A / D conversion circuit 5 and generates an AF evaluation value for performing automatic focusing (AF) processing, and detects the amount of defocus. The A / D conversion circuit 5, AE processing circuit 11, and AF processing circuit 12 are connected to a CPU (Central Processing Unit) 15. The CPU 15 has a built-in memory for calculations to control the imaging device.

[0018] Shake detection sensor 14 detects movement of imaging device 1 due to camera shake or the like. For example, shake detection sensor 14 is composed of an inertial sensor such as a gyro sensor or an accelerometer. Using multiple inertial sensors makes it possible to detect shake on multiple axes. Shake detection circuit 13 acquires and processes the output signal of shake detection sensor 14 and outputs the detection results to CPU 15 and main subject detection circuit 26. In this embodiment, AF processing circuit 12 and CPU 15 constitute means for detecting the amount of shake in the optical axis direction of the imaging optical system.

[0019] A timing generator (hereinafter referred to as "TG") 16 is connected to the CPU 15, generates predetermined timing signals, and outputs them to the imaging circuit 4 and a sensor driver 17. The sensor driver 17 is a drive circuit for the imaging element 3, and drives the imaging element 3 in accordance with signals from the TG 16 based on control commands from the CPU 15.

[0020] The operation switch (SW) group 18 has operation elements such as various switches and dials. Operation signals from the switches etc. are output to the CPU 15, and the CPU 15 executes processing according to the operation instructions. For example, there are a main power switch for starting up the imaging device 1 and supplying power, a release switch for starting the shooting operation (recording operation), a playback switch for starting the playback operation, etc.

[0021] The EEPROM 19 in the main body of the imaging device is an electrically rewritable read-only memory and is connected to the CPU 15. Programs for various controls and data used for various operations are stored in advance in the EEPROM 19. A battery 20 supplies the power required for the circuitry such as the CPU 15.

[0022] The communication driver 21 communicates with the lens device 31. The display element 22 is, for example, a light-emitting diode (LED) or other display device that displays warnings, etc. The speaker 23 outputs audio to notify the user that the target subject is in focus or to warn the user that the target subject is out of focus, etc. The communication driver 21, the display element 22, and the speaker 23 are connected to the CPU 15.

[0023] The sensor movement control circuit 24 controls the movement of the image sensor 3 within a predetermined plane in accordance with a control command from the CPU 15. The predetermined plane is a plane perpendicular to the optical axis of the imaging optical system, and the sensor movement control circuit 24 controls the drive of the motor 25. The motor 25 constitutes a drive mechanism for the image sensor 3, and can move the image sensor 3 horizontally, vertically, or in a rotational direction.

[0024] Main subject detection circuit 26 identifies the main subject within the shooting screen and detects its position and size. Main subject detection circuit 26 receives the output signal of A / D conversion circuit 5 and the detection signal of motion vector detection circuit 27, performs detection processing, and outputs the main subject detection result to CPU 15. Motion vector detection circuit 27 receives the output signal of A / D conversion circuit 5, performs processing to detect the motion vector of the subject, and outputs the detection result to main subject detection circuit 26.

[0025] The image processing circuit 28 performs processes such as image deformation, image extraction, image alignment, and synthesis in accordance with control commands from the CPU 15. The image processing circuit 28 acquires the output signal of the A / D conversion circuit 5 and generates a new image from multiple captured images by rotating, enlarging, or reducing the image, cropping (image extraction), and synthesizing multiple extracted images. This image processing corrects fluctuations in image magnification. In other words, the image processing circuit 28 serves as a means for correcting fluctuations in image magnification.

[0026] The image resolution recovery circuit 29 performs the following processing to recover the image resolution in accordance with a control command from the CPU 15. -Sharpness processing such as edge emphasis. A process that changes the combination of pixels with different exit pupils added together to be used for image-plane phase-detection AF. -Image restoration processing using image restoration filters (such as Wiener filters). Super-resolution processing (image processing that obtains a relatively high-resolution output image from a relatively low-resolution input image). Each process will be described later.

[0027] Next, the configuration of the lens device 31 will be described. The lens device 31 includes a motion compensation lens 32, a focus lens 33, and an aperture 34. The motion compensation lens 32 is, for example, a shift lens for image blur correction. The focus lens 33 is a focus adjustment lens that can move in the direction of the optical axis of the imaging optical system. The aperture 34 adjusts the amount of light that passes through the imaging optical system.

[0028] A communication driver 35 in the lens device 31 communicates with the main body of the imaging device 1. A control circuit 36 ​​controls the components of the lens device 31, such as the image stabilization lens 32, the focus lens 33, and the aperture 34. The control circuit 36 ​​controls the driving of various motors (not shown) corresponding to the optical members. These motors are the drive motor for the image stabilization lens 32, the drive motor for the focus lens 33, and the drive motor for the aperture 34. The control circuit 36 ​​controls the motors to move the focus lens 33 and perform focus adjustment (focusing). An EEPROM 37 in the lens device 31 is an electrically rewritable read-only memory that pre-stores data and the like used for performing various operations.

[0029] Next, the operation of the imaging device 1 will be described. The user operates the power switch and release switch to start up the imaging device 1 with the lens device 31 attached. The release switch has a two-stage switch configuration, and the user can issue an operation instruction corresponding to a first stroke (hereinafter referred to as SW1) and an operation instruction corresponding to a second stroke (hereinafter referred to as SW2). When SW1 is turned on, an instruction signal is generated to start AE processing and AF processing, which are performed prior to the shooting operation. Furthermore, when SW2 is turned on, an instruction signal is generated to start the actual exposure operation.

[0030] Light from the subject passes through the imaging optical system of the lens device 31 and reaches the image sensor 3 where it is photoelectrically converted. The imaging circuit 4 processes the output of the image sensor 3, and the digital image signal from the A / D conversion circuit 5 is sent to the VRAM 6. The digital image signal is also output to the AE processing circuit 11, the AF processing circuit 12, the motion vector detection circuit 27, the main subject detection circuit 26, the image processing circuit 28, and the image resolution restoration circuit 29.

[0031] The AE processing circuit 11 calculates an AE evaluation value corresponding to the brightness of the subject based on the digital image signal input from the A / D conversion circuit 5, and outputs the calculated value to the CPU 15. Based on the acquired AE evaluation value, the CPU 15 calculates the exposure time of the image sensor 3 and the aperture value of the diaphragm 34, and transmits these values ​​to the lens device 31 via the communication driver 21. In the lens device 31, a control circuit 36 ​​controls the diaphragm drive, and adjusts the diaphragm 34 to an appropriate aperture value.

[0032] The AF processing circuit 12 performs image correction on the image signal acquired by the image sensor 3 having imaging pixels for focus adjustment. The AF processing circuit 12 detects the defocus amount (or distance information) by performing correlation calculations on the corrected image signal and outputs the information to the CPU 15. The CPU 15 calculates the drive amount and drive direction of the focus lens 33 and transmits the calculation results to the lens device 31 via the communication driver 21. In the lens device 31, a control circuit 36 ​​controls the drive of the focus lens 33, and a focused state is obtained by AF control.

[0033] Main subject detection circuit 26 detects the main subject based on the digital image signal input from A / D conversion circuit 5. A photographed person, animal, or other subject is detected. For example, main subject detection circuit 26 determines the main subject using the following information: -Information on the setting status of the operation switch. Information indicating the processing results of the AE processing circuit 11. -Color temperature information of the subject calculated for AWB (auto white balance) processing. Subject distance information acquired by the AF processing circuit 12. Subject movement information obtained by the motion vector detection circuit 27. Hand-shake information obtained by the shake detection sensor 14 and the shake detection circuit 13. The main subject detection circuit 26 detects the position and size of the determined main subject within the image.

[0034] The motion vector detection circuit 27 detects a motion vector based on the digital image signal input from the A / D conversion circuit 5. A standard image corresponding to the digital image signal of the current frame is compared with a reference image corresponding to the digital image signal of the previous frame. The motion vector detection circuit 27 performs a correlation calculation between the standard image and the reference image for multiple image regions divided in accordance with instructions from the CPU 15, and calculates a motion vector for each divided image region. Specifically, the reference image is shifted horizontally or vertically by a predetermined pixel shift amount, and a difference calculation is performed between the standard image and the reference image. The pixel shift amount that yields the highest correlation (smallest difference amount) in the difference calculation result is determined as the amount of motion of the subject in the target region, and the pixel shift direction at that time is determined as the movement direction. This allows the motion vector for each image region to be determined.

[0035] Image processing circuit 28 acquires the outputs of main subject detection circuit 26 and motion vector detection circuit 27 and performs predetermined image processing. The predetermined image processing includes, for example, deformation such as image rotation to correct changes in the up / down / left / right movement of the image of the main subject and information about the rotation of imaging device 1, and cropping to cut out a portion of the image. The image data processed by image processing circuit 28 is stored in a predetermined area of ​​VRAM 6, or is further subjected to image synthesis processing.

[0036] For example, the positions of the detected main subject and background areas on the screen may change due to camera shake by the photographer, etc. If blur occurs between multiple captured images, unnatural lines may appear in the composite image, and image resolution may decrease. Therefore, the motion vector detection circuit 27 detects positional deviations between the multiple images, and the image processing circuit 28 performs geometric transformation processing on the images to correct the positional deviations between the images. The multiple images that have been aligned by geometric transformation are added by the image processing circuit 28, thereby performing blur correction (positional deviation correction) through image synthesis.

[0037] The image resolution restoration circuit 29 performs image resolution restoration processing on the digital image signal input from the A / D conversion circuit 5 in accordance with instructions from the CPU 15. The image resolution restoration processing is one or more of the following processes: · Sharpness processing: Main subject area information is used. -Processing to change the combination of adding different pixels of the exit pupil used for image plane phase difference AF: Main subject area information and the defocus amount of each area are used. Image restoration processing using a Wiener filter, etc.: Information such as main subject area information, the defocus amount of each area, the focal length and aperture value of the lens device 31, the point spread function of each image height, aberration, distortion, etc. is used. Hereinafter, the point spread function will be abbreviated as PSF. Super-resolution processing: Main subject area information and the defocus amount of each area are used.

[0038] The image data resulting from processing by the image resolution restoration circuit 29 is stored in a predetermined area of ​​the VRAM 6. In this processing, the image resolution restoration circuit 29 can obtain necessary information from the lens device 31, AF processing circuit 12, etc. via the CPU 15. For example, the image resolution restoration circuit 29 obtains information about the main subject area from the main subject detection circuit 26, and obtains the defocus amount of each area from the AF processing circuit 12. The image resolution restoration circuit 29 also obtains information about the focal length and aperture value of the imaging optical system, as well as information about the PSF, aberration, and distortion for each image height from the lens device 31.

[0039] Next, blur correction in the optical axis direction in the imaging device 1 will be described with reference to the flowchart shown in FIG. 2. The following processing is realized by the CPU 15 executing a program. In S201, the CPU 15 initializes variables used in the processing, moves a driving member to its initial position, and so on, and then acquires information about the lens device 31 and information about the brightness of the shooting environment. If it is determined that the lens device 31 is attached to the main body of the imaging device 1, processing is executed to acquire information about the focal length of the imaging optical system, the blur correction lens 32, the focus lens 33, and the aperture 34. If the lens device 31 is not attached to the main body of the imaging device 1, the processing to acquire information about the lens device 31 is not performed, and the lens device 31 enters a standby state until the main body of the imaging device 1 is attached. Next, the processing proceeds to S202.

[0040] In S202, the CPU 15 performs a process of acquiring AF information used to control the focus adjustment operation over at least several frames, and calculates the amount of blur in the optical axis direction from the AF information. The CPU 15 acquires a defocus amount based on the AF information from the AF processing circuit 12. The defocus amount includes information corresponding to the distance to the subject. The CPU 15 chronologically calculates the amount of blur in the optical axis direction, which corresponds to changes in the distance from the imaging device to the subject, based on information about the focus lens 33 acquired from the lens device 31. Next, the process proceeds to S203, where the CPU 15 performs a calculation to determine whether the subject is within the depth of field based on information about the aperture 34 acquired from the lens device 31 and the subject distance information acquired in S202. In S204, the CPU 15 determines whether the subject is within the depth of field. If the subject is not within the depth of field (Yes in S204), it is determined that focus adjustment is necessary, and the process proceeds to S205. If the subject is within the depth of field (No in S204), it is determined that focus adjustment is not necessary, and the process proceeds to S208.

[0041] In S205, the CPU 15 performs a process to detect a change in image magnification when focus adjustment is performed. This change in image magnification occurs when the distance to the subject changes due to blurring of the image capture device 1 along the optical axis, or when the effective focal length of the lens device 31 changes due to focus adjustment. This change in focal length occurs during macro photography, etc. When the distance to the subject changes due to blurring along the optical axis and the focus lens 33 moves in response to this change to adjust the focus, the size of the image on the sensor surface of the image sensor 3 changes. When blurring along the optical axis moves the image capture device 1 closer to the subject, the image on the sensor surface becomes larger. However, depending on the configuration of the imaging optical system of the lens device 31, focus adjustment may further enlarge the image, amplifying the change in image magnification, or conversely, may reduce the image, reducing the change in image magnification. Similarly, when the image capture device 1 moves away from the subject, the change in image magnification may either be amplified or reduced.

[0042] Therefore, the CPU 15 calculates the amount of change in image magnification when focus adjustment is performed based on the change in object distance calculated up to this point. Specifically, the difference between the maximum and minimum values ​​of the image size when focus adjustment is performed is calculated. If the magnitude of the difference is equal to or less than a predetermined threshold, the CPU 15 determines that the subject is a candidate for focus locking (i.e., that focus locking is possible). If the magnitude of the difference is greater than the predetermined threshold, the CPU 15 determines that focus locking is not possible. In this case, a margin may be provided for the change in object distance by multiplying the calculated value up to this point by a predetermined constant. Note that even if it is determined that focus locking is possible, if the amount of change in image magnification decreases due to focus adjustment, the CPU 15 determines that focus adjustment should be actively performed (i.e., the subject is determined not to be a candidate for focus locking). In this way, the CPU 15 calculates the amount of change in image magnification due to focus adjustment and performs a determination process regarding focus locking. After S205, the process proceeds to S206.

[0043] In S206, the CPU 15 calculates the degree of image resolution restoration. Based on information acquired from the lens device 31, the AF processing circuit 12, the main subject detection circuit 26, etc., the CPU 15 calculates the degree of image resolution restoration using image processing to compensate for the loss of resolution caused by fixing the focus. The CPU 15 sets the image resolution restoration process to be performed after shooting. Image resolution restoration methods include sharpness processing such as edge enhancement. Another method involves changing the combination of pixels with different exit pupils used in image-plane phase-difference AF. Other methods include image restoration using a Wiener filter or super-resolution processing. The CPU 15 evaluates the degree of image resolution restoration achieved by each process. This process will be described later. Note that, depending on the structure of the image sensor, changing the combination of pixels with different exit pupils may not be possible. If such an image sensor is used, a different image resolution restoration process is selected. The CPU 15 determines that fixation of focus is possible if the degree of image resolution restoration achieved by one process or a combination of multiple processes is equal to or greater than a predetermined value (threshold) and sufficient image resolution is obtained.

[0044] In this way, the CPU 15 calculates the degree of resolution recovery through image processing using the image resolution recovery circuit 29 and sets the post-photography processing. Next, in S207, the CPU 15 determines whether to fix the focus or perform focus adjustment. The CPU 15 executes a process for determining whether to perform focus adjustment based on the amount of change in image magnification and the degree of image resolution recovery. If a candidate for focus fixation is determined in S205 and S206, the CPU 15 determines to fix the focus and proceeds to the process of S208. On the other hand, if a candidate for focus fixation is not determined in S205 or S206, the CPU 15 proceeds to the process of S213.

[0045] In S208, the CPU 15 determines whether image composition is necessary to improve the image quality of the captured image. The CPU 15 acquires the magnitude of blur of the image capture device 1, excluding the optical axis direction, from the blur detection circuit 13 and compares the amount of blur with a threshold value. If the amount of blur of the image capture device 1, excluding the optical axis direction, is greater than the threshold value, the capture is divided into multiple captures, resolution degradation due to blur in each image is suppressed, and the multiple images are aligned and composed. The CPU 15 also determines that image composition is necessary if focus adjustment is determined to be necessary in S204. This is because, without focus adjustment, the image will contain resolution degradation due to blur, which must be restored to increase the resolution. The processing content differs for images with different degrees of focus (in-focus). Therefore, multiple images with different degrees of focus are acquired, and appropriate image resolution restoration processing is performed for each image. If it is determined that image composition is necessary in S208, the processing proceeds to S209. If it is determined that image composition is not necessary, the processing proceeds to S212.

[0046] The processes of S201 to S208 are basically performed after SW1 is turned on, and continue to be performed while SW1 is kept on. However, taking into consideration the so-called one-press operation (an operation in which SW1 and SW2 are turned on almost simultaneously), the processes of S201 to S208 are performed by transitioning to a low power consumption mode, such as by lowering the readout rate of the image sensor, before SW1 is turned on. At this time, the processes may be performed when the detection signal of the shake detection circuit 13 falls below a predetermined value and the CPU 15 determines that framing has stabilized.

[0047] In S209, continuous shooting is performed in synchronization with the operation of SW2. Data on captured images is stored sequentially in VRAM 6. If it is determined in S204 that focus adjustment is unnecessary, the exposure for each image is set based on the processing results of the AE processing circuit 11. A lower limit (longest exposure time) for the shutter speed is set. For example, the lower limit may be set to 1 / focal length / 2. Multiple images are continuously shot according to the exposure setting. Furthermore, if it is determined in S204 that focus adjustment is necessary, the adjustment is performed based on the settings based on the processing results of the AE processing circuit 11. The values ​​of each element that determines exposure are adjusted according to the position of the focus lens 33, a component of the focus adjustment means. The CPU 15 determines the positions where the speed of blur is maximum and minimum based on the period of blur in the optical axis direction detected using the AF information up to that point. Then, at the position where the speed of blur is maximum, the CPU 15 sets the exposure time to half the standard aperture value without changing the standard aperture value, and adjusts the sensitivity accordingly. Furthermore, when the speed of the camera shake is at its minimum, the CPU 15 does not change the aperture value set as the reference, but doubles the exposure time and adjusts the sensitivity in accordance with that setting. Next, the process proceeds to step S210.

[0048] In S210, the CPU 15 restores the resolution through image processing using the image resolution restoration circuit 29. The resolution restoration process is performed on the image input from the A / D conversion circuit 5 or a predetermined area of ​​the VRAM 6, and the processed image data is stored in a predetermined area of ​​the VRAM 6. Note that this process is not performed if it is determined in S204 that focus adjustment is not required. Details of the image resolution restoration process will be described later.

[0049] Next, in S211, the continuously captured images are transformed, aligned, and combined. The motion vector detection circuit 27 detects misalignment between multiple images input from the A / D conversion circuit 5 or a predetermined area of ​​the VRAM 6, and the image processing circuit 28 performs geometric transformation of the images to correct the misalignment. The geometric transformation includes correcting the size of the images caused by changes in image magnification and correcting misalignment in the translational and rotational directions of the images.

[0050] The geometric transformation in S211 requires that pixels be associated with each other between the pre-transformation image and the post-transformation image. When translating or rotating an image on a sub-pixel basis, pixels do not correspond one-to-one, so the correspondence is typically achieved using pixels interpolated from multiple surrounding pixels. However, pixel interpolation can reduce image resolution and cause image quality degradation. To prevent image quality degradation, it is desirable to avoid translation or rotation on a sub-pixel basis as much as possible. Regarding translation, limiting movement to integer pixel units reduces alignment accuracy, but can avoid image quality degradation due to pixel interpolation. The data of the multiple images aligned by geometric transformation is added in the image processing circuit 28, and blur-corrected image data is generated by image synthesis.

[0051] In S212, a single image is captured in synchronization with the operation of SW2 according to the exposure setting based on the processing result of the AE processing circuit 11. In addition, in S213, continuous shooting begins while the focus is adjusted in synchronization with the operation of SW2, and multiple image data are stored in a predetermined area of ​​the VRAM 6. Continuous shooting while adjusting the focus is performed as follows. (1) For all pixels, the values ​​of adjacent pixels with different exit pupil positions are added together and read out (additional readout). At this time, for the area targeted for AF (hereinafter referred to as the "target area"), such as the image area corresponding to the main subject, the values ​​of pixels with different exit pupil positions are read out separately, and AF control is performed using this pixel information. (2) After additive reading for image recording, etc., the values ​​of pixels at different exit pupil positions are read out separately for the relevant region only, and AF control is performed using this pixel information. In this reading, the operation of reading out the values ​​of pixels at different exit pupil positions separately for the relevant region only may be performed multiple times. (3) The values ​​of pixels at different exit pupil positions are read out separately for all pixels, and this pixel information is used for AF control. The separately read-out pixels are then added together to generate an image for recording or display. An appropriate method can be selected from the methods (1) to (3) according to the performance of the imaging device, etc.

[0052] After S213, the process proceeds to S211. The image data stored in VRAM 6 in S213 is read out in S211, and a composite image is generated. In this case, even if a process for recovering image resolution is set in S206, it is ignored.

[0053] After S211 or S212, the series of processes ends. Note that image stabilization processing is executed during shooting in S209, S212, and S213. Since a known method can be applied to this processing, a description thereof will be omitted. Furthermore, image processing may be performed directly on the image data acquired by continuous shooting in S209 and S213 without first storing the data in VRAM 6.

[0054] Next, the process shown in S206 in Fig. 2 (calculating the degree of image resolution recovery and setting post-photography processing) will be described with reference to Figs. 3 and 4. In this embodiment, the process determines which of a plurality of image resolution recovery methods is applied to recover the resolution to an appropriate degree, and whether the result satisfies the desired resolution. The following process is realized by the CPU 15, which is the main controller, executing a program loaded into memory.

[0055] First, in S301, a determination process is performed to determine whether the maximum defocus amount detected by the AF processing circuit 12 satisfies a predetermined condition. The predetermined condition is whether the defocus amount is equal to or less than α1 times (for example, α1=2) the depth of focus (Fδ: F is the effective F-number, and δ is the allowable circle of confusion diameter) under the shooting conditions. If the predetermined condition is satisfied, the process proceeds to S302, and if the predetermined condition is not satisfied, the process proceeds to S311 in FIG. 4.

[0056] In S302, the contrast of the main subject region determined by main subject detection circuit 26 (hereinafter referred to as "contrast A") and the contrast between the background region and the main subject region (hereinafter referred to as "contrast B") are calculated. For contrast A, the maximum and minimum luminance signal values ​​within the main subject region are determined, and the difference between these values ​​is normalized over the full range of luminance signal values ​​to obtain an index representing the contrast. For contrast B, the maximum and minimum luminance signal values ​​at the boundary between the main subject region and the background region are determined, and the difference between these values ​​is normalized over the full range of luminance signal values ​​to obtain an index representing the contrast. This boundary region is defined by setting a detection region of a predetermined size outside and inside the portion that forms the outermost outline of the main subject region. For example, the predetermined size is set to ±2% of the screen size or ±100 pixels. Note that instead of using the difference between the maximum and minimum luminance signal values, the maximum differential value of the luminance signal may be used as an index representing the contrast. Next, the process proceeds to S303.

[0057] In S303, a determination process is performed to determine whether or not both contrast A and contrast B are equal to or greater than a first threshold value. The first threshold value is referred to as "predetermined value 1." The determination condition is "contrast A ≧ predetermined value 1" and "contrast B ≧ predetermined value 1." If the determination condition is met, the process proceeds to S306. If contrast A or contrast B is less than "predetermined value 1," the process proceeds to S304.

[0058] In S304, the determination process is performed using the following determination condition expressions. The second threshold is denoted as "predetermined value 2" and is assumed to satisfy "predetermined value 2 > predetermined value 1". The third threshold is denoted as "predetermined value 3" and is assumed to satisfy "predetermined value 3 < predetermined value 1". (I) "Contrast A ≧ predetermined value 2" and "Contrast B ≧ predetermined value 3" or (II) "Contrast A ≧ predetermined value 3" and "Contrast B ≧ predetermined value 2" If the determination condition (I) or (II) is met in S304, the process proceeds to S306. If the determination condition (I) or (II) is not met, the process proceeds to S305.

[0059] In S305, the focus is fixed, and the settings at the time of shooting are recorded in memory so that sharpness processing will be performed after processing to change the combination of pixels with different exit pupils used for image-plane phase-difference AF. Meanwhile, in S306, the focus is fixed, and the settings at the time of shooting are recorded in memory so that sharpness processing will be performed. After S305 and S306, processing ends.

[0060] 4, a determination process is performed to determine whether the maximum defocus amount detected by the AF processing circuit 12 satisfies a predetermined condition. The predetermined condition is whether the defocus amount is equal to or less than α2 times (for example, α2=3) the depth of focus (Fδ) under the shooting conditions. If the predetermined condition is satisfied, the process proceeds to S312, and if the predetermined condition is not satisfied, the process proceeds to S313.

[0061] In S312, the memory is recorded with a change in the combination of pixels with different exit pupils added together for use in image-plane phase-difference AF. Next, in S313, the degree of resolution restoration in the image restoration process is calculated. Then, in S314, it is determined whether the resolution obtained as a result of the image restoration process is equal to or greater than a predetermined threshold. The fourth threshold corresponding to the desired resolution is denoted as "predetermined value 4." For example, the determination condition is that the results of an FFT (fast Fourier transform) of the main subject area show that components above a predetermined frequency are equal to or greater than a percentage corresponding to predetermined value 4. The numerical value for components above the predetermined frequency (predetermined value 4) is variably set based on the FFT result of the image that is considered to be the most in-focus among the images actually acquired up to that point. Image restoration processing can be performed on an image acquired before shooting when the image capture device 1 is closest to the subject or when the image capture device 1 is farthest from the subject, and then FFT processing can be performed on the processing result. However, since using an image resolution restoration circuit requires a large computational load, the following method may be used instead.

[0062] Image capture device 1 acquires the highest spatial frequency of the main subject area from the image determined to be most in focus, and performs processing to calculate the area requiring restoration from information from main subject detection circuit 26. Image capture device 1 acquires PSF information, aberration and distortion information for the area to be restored from lens device 31, and estimates image restoration performance. Depending on the value of each piece of information, the degree of image restoration and the resulting resolution are measured in advance, and the results are stored in EEPROM 19.

[0063] Image restoration becomes more difficult the higher the spatial frequency of the main subject area (the more detailed the area), the wider the range requiring restoration, and the greater the aberration or distortion in the area to be restored. When the spatial frequency of the main subject area is high, high-frequency components are lost not only due to loss of focus, but also due to noise in the image itself and driving errors in the focus lens 33. This loss cannot be restored by image restoration processing, so image resolution may not be fully restored. When the range requiring restoration is wide, the parameter values ​​for image restoration cannot be considered constant, and resolution may not be fully restored in areas requiring parameter values ​​different from the adopted parameter values. Furthermore, when the area to be restored has a large aberration or distortion, the aberration or distortion varies depending on the position on the screen, increasing the number of factors that cause resolution degradation not included in the parameters for image restoration, and the resolution may not be fully restored. Therefore, based on the above acquired information and the data stored in EEPROM 19, CPU 15 performs a determination process to determine whether image resolution will be fully restored if image restoration processing is performed. If the determination condition of S314 is met, the process proceeds to S315, and if the determination condition of S314 is not met, the process proceeds to S316.

[0064] In S315, the focus is fixed and the following processing is performed. If it is recorded in memory in step S312 that processing for changing the combination of adding pixels with different exit pupils used for image plane phase detection AF is to be performed: After processing for changing the combination, it is recorded in memory that image restoration processing and sharpness processing will be performed when shooting. If the process of changing the combination of pixels added together with different exit pupils used for image plane phase detection AF is not recorded in memory in step S312: The process of performing sharpness processing after image restoration processing when shooting is recorded in memory.

[0065] In S316, a process is executed to determine the degree of resolution restoration in the super-resolution process. Next, in S317, a determination process is executed to determine whether the resolution obtained as a result of the super-resolution process is equal to or greater than a "predetermined value 4" corresponding to the desired resolution. For example, the determination condition is that, as a result of the FFT of the main subject area, components above a predetermined frequency are equal to or greater than a percentage corresponding to the predetermined value 4. The predetermined value 4 is variably set, as in the case of image restoration. It would be possible to actually perform image restoration using super-resolution processing and then perform FFT processing on the results, but this would be a process with a large computational load, so the following method may be used instead.

[0066] As with image restoration, the imaging device 1 calculates the highest spatial frequency of the main subject area and the area requiring restoration, and estimates the super-resolution performance based on the aberration and distortion information for the area being processed. The extent of image restoration and the resulting resolution are measured in advance based on the values ​​of each piece of information, and the results are stored in EEPROM 19. The higher the spatial frequency of the main subject area, the wider the area requiring restoration, and the greater the aberration or distortion in the area being restored, the more difficult the super-resolution processing. Therefore, based on the acquired information and the data stored in EEPROM 19, CPU 15 performs a determination process to determine whether the image resolution will be restored to a sufficient level if super-resolution processing is performed. If the determination condition of S317 is met, the process proceeds to S318; if the determination condition of S317 is not met, the process proceeds to S319.

[0067] In S318, the focus is fixed and the following processing is performed. If it is recorded in memory in step S312 that processing will be performed to change the combination of adding pixels with different exit pupils used for image plane phase detection AF: After processing to change the combination, it is recorded in memory that super-resolution processing and sharpness processing will be performed when shooting. If the memory does not record in step S312 the process of changing the combination of pixels added together that have different exit pupils used for image plane phase detection AF: The memory records that sharpness processing will be performed after super-resolution processing when shooting.

[0068] In S319, it is recorded in the memory that focus adjustment will be performed during shooting. After S315, S318, and S319, the process ends.

[0069] The process shown in S210 in Fig. 2 (restoring resolution through image processing) will be described with reference to Fig. 5. The following process is realized by the CPU 15, which is the main controller, executing a program loaded into memory.

[0070] First, in S401, it is determined whether or not it is recorded in memory that processing other than sharpness processing will be performed in the processing shown in S206 of Fig. 2. If it is recorded in memory that only sharpness processing will be performed, the processing proceeds to S408. If it is recorded in memory that processing other than sharpness processing will be performed, the processing proceeds to S402.

[0071] In S402, it is determined whether or not the process of changing the combination of addition of pixels with different exit pupils is recorded in memory. If the process of changing the combination of addition of pixels with different exit pupils is recorded in memory, the process proceeds to S403. If the process of changing the combination of addition of pixels with different exit pupils is not recorded in memory, the process proceeds to S404.

[0072] In S403, a process of changing the combination of addition of pixels with different exit pupils is performed. Next, in S404, it is determined whether or not the execution of image restoration processing is recorded in memory. If the execution of image restoration processing is recorded in memory, the process proceeds to S405, and if the execution of image restoration processing is not recorded in memory, the process proceeds to S406.

[0073] In S405, image restoration processing (setting of PSF, implementation of Wiener filter processing) is performed. Next, the process proceeds to S408. In S406, it is determined whether or not the execution of super-resolution processing is recorded in memory. If the execution of super-resolution processing is recorded in memory, the process proceeds to S407, and if the execution of super-resolution processing is not recorded in memory, the process proceeds to S408.

[0074] Super-resolution processing is performed in S407, and sharpness processing is performed in S408. Then, in S409, it is determined whether or not all processing related to the target image stored in VRAM 6 has been completed. If it is determined that processing has not been completed, the process returns to S401 and continues. If it is determined that processing has been completed, the series of processes ends.

[0075] The image processing for restoring resolution performed at each step will be described below. Patent Document 3 discloses details of image sharpness processing (S408 in FIG. 5), so a detailed description will be omitted. In this embodiment, processing is performed on multiple images with different focus states, so parameters such as the unsharp mask and the amount of mixed edge components for each pixel are changed depending on the focus state. If it is determined that the image is more out of focus, stronger sharpness processing is applied to the image.

[0076] Next, with reference to FIG. 6, the process of changing the combination of pixels with different exit pupils used for image-plane phase-difference AF (FIG. 5: S403) will be described. FIG. 6 is a schematic diagram partially illustrating the pixel configuration of the image sensor 3. To perform image-plane phase-difference AF, the image sensor 3 has a configuration in which two independent pixels sharing one microlens are arranged two-dimensionally. The image sensor 3 has first and second photoelectric conversion units corresponding to one microlens. The An pixel corresponds to the first photoelectric conversion unit, and the Bn pixel corresponds to the second photoelectric conversion unit (n is a variable representing a natural number). In the focused state, light from a specific portion of the subject passes through different exit pupil positions and forms an image on pixels that share the microlens (for example, an image is formed on pixel A1 and pixel B1, ..., pixel A8 and pixel B8). Signals of the A image sequence are acquired from pixels A1, A2, ..., and A8, and signals of the B image sequence are acquired from pixels B1, B2, ..., and B8.

[0077] In Figure 6, the in-focus state of the subject is represented by a solid line, and the out-of-focus state is represented by a dotted line. Pixels An and Bn, which share a microlens, form a pair, and adding the signals from these pixels produces an in-focus image signal of the subject. However, in an out-of-focus state, light from the subject does not form an image on the pixels that share the microlens. For example, light that forms an image on pixels A7 and B7 in an in-focus state forms an image on pixels A6 and B8 in an out-of-focus state. Therefore, adding the signals from pixels that share a microlens does not produce an in-focus image. Therefore, image column B can be shifted by the amount of shift between the calculated image column A (A1·A2···A8) and image column B (B1·B2···B8) and then its signal can be added to image column A. This results in a more in-focus image than when adding the signal from image column A without shifting image column B. In the example of Figure 6, it is sufficient to shift two pixels and add the signals of pixel A6 and pixel B8 (similarly, A5 and B7, A4 and B6, etc.). However, if the amount of defocus becomes large, shifting the image column B can make the centers of the light beams coincide, but since the images incident on each pixel will be blurred in the first place, it will not be possible to obtain an in-focus image. Therefore, only when it is determined in S301 of Figure 3 or S311 of Figure 4 that the amount of defocus is equal to or less than a predetermined value (threshold value), processing is performed to change the combination of pixels to be added that are different in the exit pupil.

[0078] The image restoration process shown in S405 of FIG. 5 is performed by a Wiener filter using a PSF. A known method may be used for the image restoration process using a Wiener filter using a PSF, and a detailed description thereof will be omitted. In this embodiment, the PSF is selected taking into consideration that the resolution restoration process shown in S405 is performed by changing the combination of pixels added together that have different exit pupils, prior to the image restoration process. In other words, it is assumed that the focus state changes due to the change in the combination of added pixels, resulting in a more focused state. Then, the PSF is set based on the amount of focus deviation that cannot be adjusted by adding pixels together.

[0079] The defocus amount obtained from different pixels in the exit pupil on the imaging plane is calculated from the amount of shift of image sequence B relative to image sequence A when the difference in brightness values ​​between image sequence A and image sequence B in Figure 6 is minimized. The amount of shift is obtained in sub-pixel units through interpolation calculations, rather than in pixel units. In contrast, changing the combination of pixels to be added is performed in pixel units, and this difference remains as an amount that cannot be fully adjusted. Therefore, the PSF is set based on this amount that cannot be fully adjusted.

[0080] On the other hand, if resolution restoration by changing the combination of pixels added together with different exit pupils is not performed, the PSF is set based on the defocus amount calculated by the AF processing circuit 12. Furthermore, since the PSF represents the amount of image blur in an out-of-focus state, it differs depending on the position of the main subject on the screen. Therefore, the PSF is set using the image height corresponding to the range in which the main subject is present and the defocus amount set above.

[0081] If the range of the main subject within the screen is wide and distributed from the center to the periphery, performing image restoration processing using a single PSF may result in some areas not being sufficiently restored. When area division is performed, processing of the boundaries becomes complicated, and unnaturalness may remain at the boundaries after image restoration. Therefore, image restoration processing is performed to restore resolution only when the main subject is distributed within a range that can be considered equal. Furthermore, if the area to be restored is located in the periphery of the screen, sufficient resolution may not be obtained even after image restoration processing due to the influence of aberrations and distortions depending on the focal length and aperture value of the lens device 31. In such cases, image restoration processing is not performed.

[0082] The super-resolution processing shown in S407 of Figure 5 is performed using a method called example-based super-resolution, which uses a separate set of image data for learning in advance. This super-resolution method involves machine learning of a conversion lookup table consisting of pairs of feature blocks of low-resolution images and feature blocks of high-resolution images in advance. The super-resolution processing of this embodiment regards a reduction in resolution due to out-of-focus caused by blurring in the optical axis direction as a reduction in the number of pixels, and restores resolution by generating multiple pixels from a pixel regarded as a single tentative pixel by the super-resolution processing.

[0083] For example, assuming that four pixel regions on the top, bottom, left, and right sides can be considered the same pixel due to out-of-focus, resolution can be restored by generating four-pixel information from these combined pixels. A low-resolution image, in which multiple pixels are considered as one pixel, is divided into multiple blocks, and a process is performed to search for the low-resolution block in a lookup table with the closest feature values. A high-resolution image can be generated using the high-resolution block corresponding to the searched low-resolution block. If the main subject is large within the image, it becomes difficult to find the block with the closest feature values ​​in the lookup table. Therefore, resolution restoration using super-resolution processing is performed only when the area in which the main subject is distributed is within a predetermined size (threshold range). Furthermore, if the target area for image restoration is affected by aberrations or distortion, sufficient image resolution may not be achieved. In such cases, super-resolution processing is not performed. Unlike image restoration, the same super-resolution processing is performed regardless of whether resolution restoration processing by changing the combination of pixels added together with different exit pupils has been performed in advance.

[0084] In this embodiment, it is preferable to use an image sensor capable of high-speed signal readout. An image sensor capable of faster signal readout can detect blurring in the optical axis direction with a shorter delay time, allowing for more accurate reflection of parameter values ​​used in image processing such as image restoration. Furthermore, in this embodiment, an example of image-surface phase-difference AF has been described. The AF method of the image capture device is not limited, and an AF method using an external AF sensor dedicated to AF, a TOF (Time Of Flight) sensor, or the like can be applied.

[0085] [Second embodiment] Next, a second embodiment will be described. In this embodiment, an example of application to a portable electronic device such as a smartphone is shown. The difference from the first embodiment is that the amount of blur in the optical axis direction of the imaging optical system is detected using a blur detection sensor 14. Below, a description of matters similar to the first embodiment will be omitted, and the differences will be mainly described.

[0086] FIG. 7 is a block diagram showing an example of the configuration of an imaging device 101 of this embodiment. In this embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. The image display input device 108 includes a display means such as a liquid crystal display (LCD) and an input means having a touch panel function (hereinafter referred to as a "touch panel LCD"). The liquid crystal display (LCD) displays images according to the output signal of the D / A conversion circuit 7. The touch panel LCD is electrically connected to the CPU 15, and signals input by operating the touch panel are processed by the CPU 15. The user can make desired settings and selections by operating the touch panel LCD.

[0087] The AF processing circuit 1012 performs automatic focus adjustment processing by acquiring the output from the A / D conversion circuit 5. The display element 1022 is an LED or the like, and performs illumination during AF, a flashlight function, an incoming call notification, etc. according to control commands from the CPU 15.

[0088] The focus lens 1033 constitutes an imaging optical system and is movable in the direction of its optical axis. The motor 1025 is a driving means for moving the focus lens 1033 in the direction of the optical axis. The focus lens movement control circuit 1024 controls the movement of the focus lens 1033 by driving the motor 1025.

[0089] The CPU 15 transmits a control command to the focus lens movement control circuit 1024, and performs processing to instruct the AF processing circuit 1012 to generate an AF evaluation value while controlling the movement of the focus lens 1033. The CPU 15 and the focus lens movement control circuit 1024 perform AF control by moving the focus lens 1033 to a position where the AF evaluation value reaches its peak value.

[0090] Next, the processing in the imaging device 101 will be described with reference to FIG. 2. In this embodiment, the processing in S202 is different. In the first embodiment, AF information is acquired and the amount of blur in the optical axis direction is calculated, but in this embodiment, gyro information is acquired and the amount of blur in the optical axis direction is calculated. The gyro information is information based on the output of an inertial sensor. Note that, instead of the gyro information, information based on the output of a distance measurement sensor may be acquired and the amount of blur in the optical axis direction may be calculated.

[0091] Referring to FIG. 8, a method for calculating the amount of blur in the optical axis direction using gyro information will be described. In FIG. 8, the optical axis direction of the imaging optical system is defined as the z-axis direction, and the two axes perpendicular to the z-axis are defined as the x-axis and y-axis, respectively. FIG. 8(A) is a schematic diagram viewed from the x-axis direction, and FIG. 8(B) is a diagram showing the settings of each axis with respect to the imaging device 101. The gyro sensor can acquire the angular velocity of rotation around the x-axis, y-axis, and z-axis. When calculating the amount of blur in the z-axis direction, the angular velocity of rotation around the x-axis and motion vector information in the z-axis direction are used. The imaging device 101 can be considered to be rotating around a certain point. In FIG. 8(A), the radius of rotation of the rotational motion is denoted as L, and the angle of rotation around the x-axis is denoted as θ. The radius of rotation L can be calculated using the formula "L = (motion vector information in the z-axis direction) / (angular velocity around the x-axis)". The angular velocity is the time derivative of the rotation angle θ. Since it is not possible to obtain motion vector information in the z-axis direction during exposure, the CPU 15 calculates the amount of blur in the z-axis direction from the radius of rotation L and the angular velocity around the x-axis. The calculation formula is "amount of blur in the z-axis direction = L × (angular velocity around the x-axis)".

[0092] In this embodiment, it is possible to obtain information on the detected amount of blur in the optical axis direction of the imaging optical system from a sensor and determine whether or not to perform focus adjustment to correct image blur.

[0093] As described above, in the above embodiment, the amount of blur in the optical axis direction of the imaging optical system is detected based on the output of the AF processing circuit 12 or the output of the blur detection circuit 13, which processes the signal from the blur detection sensor 14. The output of an inertial sensor such as a gyro sensor or the output of an AF sensor that generates a signal used to control focus adjustment can be acquired. The CPU 15 acquires information about the aperture 31 and subject distance information from the lens device 31 and executes processing to calculate the amount of focus change, the amount of focus adjustment associated with the focus change, and the amount of image magnification change. Furthermore, the CPU 15 determines whether to perform focus adjustment based on the amount of image magnification change after focus adjustment and the degree of image resolution restoration through image processing. The CPU 15 and the image resolution restoration circuit 29 calculate the degree of resolution restoration through image processing based on the maximum defocus amount detected by the AF processing circuit 12, information about the main subject area calculated by the main subject detection circuit 26, and information about the area to be subjected to image restoration processing. According to the above embodiment, if it is determined that focus adjustment should not be performed, it is determined not to perform operations that would result in significant changes in image magnification, such as lens drive for focus adjustment. Furthermore, it is possible to minimize the frequency of processes that have a large effect on the image, such as image restoration processing and super-resolution processing, and to provide a more natural, high-resolution image.

[0094] Embodiments of the present disclosure include the following configurations and methods. [Configuration 1] An imaging means; a detection means for detecting the amount of blur in the optical axis direction of the imaging optical system; a control means for controlling focus adjustment by a focus adjustment operation of the imaging optical system; a calculation unit that calculates an amount of change in image magnification of the image acquired by the imaging unit from the detected amount of blur, The control means controls the focus adjustment in accordance with the calculated change amount of the image magnification. An imaging device characterized by: [Configuration 2] a determination means for determining whether to perform the focus adjustment or fix the focus, using the amount of change in the image magnification; The control means controls the focus adjustment in accordance with the result of the determination by the determination means. 2. The imaging device according to claim 1, [Configuration 3] The determining means determines that the focus adjustment should be performed when the magnitude of the change in the image magnification is greater than a threshold value, and the control means controls the focus adjustment. 3. The imaging device according to configuration 2. [Configuration 4] When it is determined that the focus is to be fixed or when it is determined that the focus adjustment is not necessary, the determining means determines whether or not to combine the plurality of images acquired by the imaging means. 4. The imaging device according to configuration 2 or 3. [Configuration 5] a resolution recovery means for performing a process to recover the image resolution of the image; the calculation means calculates the degree of recovery of the image resolution, The determining means determines whether or not to perform the focus adjustment based on the calculated degree of recovery and the amount of change in the image magnification. 5. The imaging device according to any one of configurations 2 to 4. [Configuration 6] image processing means for processing the image; When the determination means determines that the focus is fixed and that the captured images are to be combined, the resolution recovery means performs a process to recover the image resolution, and the image processing means performs a process to combine the images whose image resolutions have been restored. 6. The imaging device according to configuration 5. [Configuration 7] The image processing means performs processing to correct the variation in image magnification by image deformation, image extraction, and image synthesis. 7. The imaging device according to configuration 6, [Configuration 8] a main subject detection means for detecting a main subject area within an image; The calculation means calculates the degree of restoration based on the defocus amount related to the imaging optical system and the imaging means, information on the main subject area detected by the main subject detection means, and information on the area to be subjected to image restoration processing. 6. The imaging device according to configuration 5. [Configuration 9] the main subject detection means calculates a first contrast related to the main subject region and a second contrast related to a background region in the image and the main subject region; The resolution recovery means changes the content of the process for recovering the image resolution depending on whether the first and second contrasts satisfy a determination condition. 9. The imaging device according to configuration 8. [Configuration 10] The resolution restoration means performs one or more processes among sharpness processing, image restoration processing, and super-resolution processing according to the calculated degree of restoration. 10. The imaging device according to any one of configurations 5 to 9. [Configuration 11] the imaging means has an imaging element capable of detecting an imaging surface phase difference, The resolution recovery means performs a process of changing the combination of additions of different pixels of the exit pupil used for focus adjustment. 11. The imaging device according to any one of configurations 5 to 10. [Configuration 12] The calculation means calculates the amount of change in the image magnification corresponding to the amount of blur detected by the detection means and the focal length of the imaging optical system. 12. The imaging device according to any one of configurations 1 to 11. [Configuration 13] The detecting means detects the amount of shake from the output of an inertial sensor or a distance measuring sensor, or from a signal used to control a focus adjustment operation. 13. The imaging device according to any one of configurations 1 to 12. [method] A control method executed in an imaging device, comprising: a detection step of detecting an amount of blur in the optical axis direction of the imaging optical system; a calculation step of calculating a change amount of image magnification of an image acquired by an imaging means from the detected amount of blur; a control step of controlling focus adjustment by a focus adjustment operation of the imaging optical system in accordance with the calculated change amount of the image magnification. A control method comprising: [Explanation of symbols]

[0095] 1,101 imaging devices 15 CPU 28 Image processing circuit 29 Image resolution recovery circuit

Claims

1. An imaging means; a detection means for detecting the amount of blur in the optical axis direction of the imaging optical system; a control means for controlling focus adjustment by a focus adjustment operation of the imaging optical system; a calculation unit that calculates an amount of change in image magnification of the image acquired by the imaging unit from the detected amount of blur, The control means controls the focus adjustment in accordance with the calculated change amount of the image magnification. An imaging device characterized by:

2. a determination means for determining whether to perform the focus adjustment or fix the focus, using the amount of change in the image magnification; The control means controls the focus adjustment in accordance with the result of the determination by the determination means.

2. The imaging device according to claim 1.

3. The determining means determines that the focus adjustment should be performed when the magnitude of the change in the image magnification is greater than a threshold value, and the control means controls the focus adjustment.

3. The imaging device according to claim 2.

4. When it is determined that the focus is to be fixed or when it is determined that the focus adjustment is not necessary, the determining means determines whether or not to combine the plurality of images acquired by the imaging means.

3. The imaging device according to claim 2.

5. a resolution recovery means for performing a process to recover the image resolution of the image; the calculation means calculates the degree of recovery of the image resolution, The determining means determines whether or not to perform the focus adjustment based on the calculated degree of recovery and the amount of change in the image magnification.

3. The imaging device according to claim 2.

6. image processing means for processing the image; When the determination means determines that the focus is fixed and that the captured images are to be combined, the resolution recovery means performs a process to recover the image resolution, and the image processing means performs a process to combine the images whose image resolutions have been restored.

6. The imaging device according to claim 5.

7. The image processing means performs processing to correct the variation in image magnification by image deformation, image extraction, and image synthesis.

7. The imaging device according to claim 6.

8. a main subject detection means for detecting a main subject area within an image; The calculation means calculates the degree of restoration based on the defocus amount related to the imaging optical system and the imaging means, information on the main subject area detected by the main subject detection means, and information on the area to be subjected to image restoration processing.

6. The imaging device according to claim 5.

9. the main subject detection means calculates a first contrast related to the main subject region and a second contrast related to a background region in the image and the main subject region; The resolution recovery means changes the content of the process for recovering the image resolution depending on whether the first and second contrasts satisfy a determination condition.

9. The imaging device according to claim 8.

10. The resolution restoration means performs one or more processes among sharpness processing, image restoration processing, and super-resolution processing according to the calculated degree of restoration.

6. The imaging device according to claim 5.

11. the imaging means has an imaging element capable of detecting an imaging surface phase difference, The resolution recovery means performs a process of changing the combination of additions of different pixels of the exit pupil used for focus adjustment.

6. The imaging device according to claim 5.

12. The calculation means calculates the amount of change in the image magnification corresponding to the amount of blur detected by the detection means and the focal length of the imaging optical system.

2. The imaging device according to claim 1.

13. The detecting means detects the amount of shake from the output of an inertial sensor or a distance measuring sensor, or from a signal used to control a focus adjustment operation.

13. The imaging device according to claim 1, wherein the imaging device is a lens.

14. A control method executed in an imaging device, comprising: a detection step of detecting an amount of blur in the optical axis direction of the imaging optical system; a calculation step of calculating a change amount of image magnification of an image acquired by an imaging means from the detected amount of blur; a control step of controlling focus adjustment by a focus adjustment operation of the imaging optical system in accordance with the calculated change amount of the image magnification. A control method comprising:

Images