Image processing device, image processing method, imaging device, program, and recording medium

Abstract

To provide an image processing device that enables a depth-of-field to be automatically set to be a value intended by a user.SOLUTION: An image processing device has determination means performing scene determination and compositing means compositing a plurality of images with different focus positions in the optical axis direction. The compositing means automatically performs the compositing on the basis of the results of the scene determination. The composited image generated when the compositing means performs the compositing has a deeper depth of field than the plurality of images.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, and more particularly to an imaging device that captures images with different focus positions. [Background technology]

[0002] A technique known as depth stacking is known in which multiple images with different focus positions in the optical axis direction (focus bracketing imaging) are captured, the in-focus areas of each image are extracted, and an image with an extended depth of field is synthesized.

[0003] In order to generate the depth synthesis image that the user intends, it is necessary to set an appropriate depth of field.

[0004] Patent Document 1 discloses a method in which a user specifies an arbitrary range on a screen, and a composite image is generated based on distance information of the specified range. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2016-39613 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the conventional technology disclosed in the above-mentioned Patent Document 1, the image processing device cannot determine the range of focus stacking without user operation.

[0007] Therefore, an object of the present invention is to provide an image processing device that can set focus stacking without user operation. [Means for solving the problem]

[0008] In order to achieve the above object, the present invention provides: The imaging means for acquiring images and the images captured in live viewA determination means for performing scene determination; Image a synthesis means for performing synthesis, the determining means determines importance in the scene determination based on the position of the subject, the proportion of the subject to the angle of view, and the number of subjects, and if the importance is greater than a predetermined first threshold, acquires a plurality of images with different focus positions in the optical axis direction; The synthesis means For the plurality of images The synthesis is performed to obtain an image having a deeper depth of field than the plurality of images. A composite image is generated, and if the importance is equal to or less than a first threshold value, the imaging means captures one image having a predetermined depth of field, and the composition means does not perform the composition. The present invention provides an imaging device characterized by the above features. [Effects of the Invention]

[0009] According to the present invention, an imaging device can set focus stacking without any user operation. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating the hardware configuration of a digital camera 100 as an image processing device according to an embodiment of the present invention. [Figure 2] 10A and 10B are diagrams for explaining a display image on a display unit when a focus stacking image is live-viewed. [Figure 3] 4 is a flowchart illustrating focus stacking according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating an example of detection of a main subject in the first embodiment of the present invention. [Figure 5] FIG. 3 is a diagram illustrating an example of background region detection in the first embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating focus stacking processing according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0012] (First embodiment) FIG. 1 is a block diagram illustrating the hardware configuration of a digital camera 100 serving as an image processing device according to this embodiment. In FIG. 1, the digital camera 100 includes an imaging system including a shutter 101, a barrier 102, a focus lens 103, and an imaging unit 22. The shutter 101 is a shutter with an aperture function. The barrier 102 covers the imaging system of the digital camera 100 to prevent the imaging system from becoming dirty or damaged. The focus lens 103 is a lens included in a lens group (not shown) disposed between the shutter 101 and the barrier 102. Note that the lens group also includes other lenses, such as a zoom lens. The imaging unit 22 includes an imaging element, such as a CCD or CMOS element, that converts an optical image into an electrical signal, and an A / D conversion function. Output data (captured images) from the imaging unit 22 is written to memory 32 via an image processing unit 24 and a memory control unit 15, or directly via the memory control unit 15. During focus bracketing imaging, which will be described later, a set number of captured images are all written to memory 32.

[0013] The digital camera 100 further includes an AF evaluation value detector 23, a flash 90, an image processor 24, a focus stacking unit 25, a motion detector 26, a status detector 27, a memory 32, a D / A converter 13, a display 28, a nonvolatile memory 56, a system controller 50, a system memory 52, and a system timer 53. The AF evaluation value detector 23 is located within the image capture unit 22 and calculates an AF evaluation value from contrast information and other data obtained from a digital image signal. The AF evaluation value is output from the image capture unit 22 to the system controller 50. The flash 90 can compensate for illuminance when capturing images in low-light or backlit scenes by emitting light during capture. The image processor 24 performs predetermined pixel interpolation, resizing (e.g., reduction), and color conversion on image data output from the image capture unit 22 or image data from the memory controller 15. The image processor 24 also performs predetermined arithmetic processing using the captured image data, and the system controller 50 controls exposure and distance measurement based on the obtained arithmetic results. This allows TTL (through-the-lens) AE (auto exposure) processing and EF (auto flash) processing to be performed. The image processing unit 24 also performs AF (auto focus) processing, sometimes using the output of the AF evaluation value detection unit 23 provided in the imaging unit 22. The image processing unit 24 also performs predetermined calculations using the captured image data, and performs TTL AWB (auto white balance) processing based on the calculation results.

[0014] The depth stacking unit 25 generates an image with an extended depth of field by outputting in-focus pixels in each image using multiple captured images obtained by focus bracketing imaging in the imaging unit 22. Details will be described later.

[0015] The motion detection unit 26 uses two pieces of image data to perform template matching between a region of interest and its surroundings, and calculates a motion vector for each region into which the image is divided, or for each pixel. If the calculated motion vector is equal to or greater than a threshold, it detects that there is motion in the subject and notifies the system control unit 50. The state detection unit 27 detects the state of the digital camera 100, such as the angular velocity measured by the gyro sensor, the attachment / detachment status of the tripod, and the settings made by the user via the operation unit 70. The system control unit 50 is notified of the detection results.

[0016] The memory 32 stores image data acquired and A / D converted by the imaging unit 22, as well as image data to be displayed on the display unit 28. The memory 32 has sufficient storage capacity to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 32 also serves as a memory for displaying images (video memory). The D / A converter 13 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28. The display image data written to the memory 32 is then displayed on the display unit 28 via the D / A converter 13. The display unit 28 displays the image on a display device such as an LCD in accordance with the analog signal from the D / A converter 13. The digital signal once A / D converted by the imaging unit 22 and stored in the memory 32 is converted to analog by the D / A converter 13 and sequentially transferred to and displayed on the display unit 28, thereby functioning as an electronic viewfinder and enabling through-image display (hereinafter referred to as live view). The non-volatile memory 56 is an electrically erasable and recordable memory, such as a flash memory. The non-volatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are programs for executing various flowcharts described later in the first, second, and third embodiments.

[0017] The system control unit 50 controls the entire digital camera 100. Specifically, by executing a program recorded in the nonvolatile memory 56 described above, focus bracketing imaging is realized based on subject information, subject distance, and image contrast information. That is, during this imaging, the system control unit 50 controls the drive of the focus lens 103 and the shutter 101, thereby sequentially capturing multiple images with different focus positions. The amount of change in focus position (focus step) between adjacent captured images obtained by this imaging process is set from a value calculated by the system control unit 50.

[0018] The system memory 52 is composed of RAM etc. and stores constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 56, etc. The system control unit also controls the display by controlling the memory 32, D / A converter 13, display unit 28 etc. The system timer 53 is a timing unit that measures the time used for various controls and the time of the built-in clock.

[0019] The digital camera 100 also includes operating means for inputting various operational instructions to the system control unit 50, including a mode selector switch 60, a shutter button 61, a first shutter switch 64, a second shutter switch 62, an operation unit 70, and a power switch 72. The mode selector switch 60 switches the operating mode of the system control unit 50 between still image recording mode, video recording mode, playback mode, etc. Modes included in the still image recording mode include auto capture mode, auto scene determination mode, manual mode, various scene modes that provide capture settings for specific capture scenes, program AE mode, and custom mode. The mode selector switch 60 can directly switch to one of these modes included in the still image capture mode. Alternatively, after switching to the still image capture mode with the mode selector switch 60, other operating members can be used to switch to one of these modes included in the still image capture mode. Similarly, the video capture mode may include multiple modes. The shutter button 61 is an operating unit for issuing image capture instructions. The first shutter switch 64 is turned on when the shutter button 61 is pressed halfway, i.e., halfway down, and generates a first shutter switch signal SW1. The first shutter switch signal SW1 initiates operations such as AF (autofocus), AE (autoexposure), AWB (auto white balance), and EF (auto flash) processing. In other words, parameters for image capture are acquired under the control of the system control unit 50. The user can select center-point AF or face AF as the AF processing to be initiated upon receiving signal SW1. Center-point AF refers to AF performed on a single point at the center of the image capture screen, while face AF refers to AF performed on a face detected within the image capture screen by the face detection function. The second shutter switch 62 is turned ON upon completion of shutter button 61 operation, i.e., full press (image capture command), and generates a second shutter switch signal SW2. The system control unit 50 initiates a series of image capture operations, from reading a signal from the image capture unit 22 to writing image data to the recording medium 200, based on the second shutter switch signal SW2.Each operating member of the operating unit 70 is assigned a function appropriate for each situation by selecting and operating various function icons displayed on the display unit 28, and functions as various function buttons. Examples of function buttons include an end button, a back button, an image forward button, a jump button, a filter button, and an attribute change button. For example, when the menu button is pressed, a menu screen on the display unit 28 that allows various settings to be made is displayed. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four-way buttons (up, down, left, and right), and the SET button. The power switch 72 switches the power of the digital camera 100 on and off.

[0020] The digital camera 100 further includes a power supply control unit 80, a power supply unit 40, and a recording medium I / F 18. The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and other components, and detects whether a battery is installed, the type of battery, and the remaining battery charge. The power supply control unit 80 controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each component, including the recording medium 200. The power supply unit 40 may include primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or Li batteries, or an AC adapter. The recording medium I / F 18 is an interface with the recording medium 200, such as a memory card or hard disk. The recording medium 200 is a recording medium, such as a memory card, for recording captured images, and may include a semiconductor memory or a magnetic disk.

[0021] Next, an overview of focus bracketing imaging will be described with reference to Fig. 1. Focus bracketing imaging refers to imaging in which the focus lens 103 is moved to a desired focus position, and then exposure and readout of the imaging unit 22 is performed.

[0022] Next, the basic operation of focus stacking processing in the digital camera 100 will be described with reference to the drawings. FIG. 2 is a diagram illustrating an image displayed on the display unit 28 when a focus stacking image is live-viewed in this embodiment. Position 201 is the close-side focus position of focus bracketing imaging indicated by an AF frame, position 202 is the infinity-side focus position, and position 203 is the subject (an insect). Displayed on the display unit 28 are an AF frame indicating the close-side focus position 201 and subject 203. The user operates the operation unit 70 to specify a reference focus position on the image displayed on the display unit 28. Alternatively, the reference focus position may be automatically specified for the subject detected by the AF evaluation value detection unit 23. In FIG. 2, it is assumed that the closest point on the subject 203 is specified as the reference focus position. As a result, the system control unit 50 recognizes the specified reference focus position as the close-side focus position 201 of focus bracketing imaging and displays an AF frame at position 201. Once the closest side focus position 201 is determined, the focus position in the Z axis direction (depth direction) where focus bracketing image capture ends, i.e., the infinity side focus position 202, is determined based on the focus interval according to the focus step setting and the number of images captured. In Fig. 2, it is assumed that the entire subject 203 falls within the focus range indicated by the closest side focus position 201 to the infinity side focus position 202. Also, it is assumed that 10 images are captured to generate one depth stacked image, and the depth stacking unit 25 performs depth stacking processing using the 10 captured images.

[0023] Next, an example of the focus stacking process in this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart for explaining focus stacking in this embodiment. In step S301, the system control unit 50 performs scene determination on an image captured by the imaging unit 22 in live view. A specific method of determination will be described later. In step S302, the system control unit 50 uses the scene determination result to determine whether to enable the focus stacking function, and sets one focus position (step S306) or multiple focus positions (step S303).

[0024] In step S303, the system control unit 50 determines the focus position and number of times to capture images for focus stacking. The system control unit 50 may also set a focus step according to the exposure control value and change the number of focus positions. This is a process aimed at shortening processing time and reducing the amount of blur by reducing the number of images captured in response to the increase in shooting time and amount of camera shake that accompanies an increase in exposure time. The system control unit 50 may also determine whether to change the number of focus positions according to the tripod attachment / detachment status detected by the status detection unit 27.

[0025] In step S304, the system control unit 50 controls the drive of the focus lens 103 and the shutter 101, causing the imaging unit 22 to sequentially capture images at the multiple different focus positions set in step S303. In step S305, the depth stacking unit 25 generates an image with an extended depth of field using the multiple captured images captured in step S304. In step S307, the system control unit 50 controls the drive of the focus lens 103 and the shutter 101, causing the imaging unit 22 to capture an image at the focus position set in step S306.

[0026] Here, the focus stacking process in step S305 will be described in detail.

[0027] The focus stacking process in this embodiment may use a known technique, and is performed as follows, for example.

[0028] First, the image processing unit 24 performs alignment on the image to be subjected to depth stacking. The image processing unit 24 sets multiple blocks in a reference image, which is one of multiple images to be subjected to depth stacking. The image processing unit 24 preferably sets each block to have the same size. Next, the image processing unit 24 sets a search range in the image to be aligned, at the same position as each block in the reference image, but with a wider range than the block in the reference image. Finally, the image processing unit 24 calculates a corresponding point in each search range of the target image that minimizes the sum of absolute differences (SAD) in brightness between the block in the reference image and the corresponding point described above. The image processing unit 241 calculates a positional deviation as a vector from the center of the block in the reference image and the corresponding point described above. In addition to SAD, the image processing unit 241 may also use the sum of squared differences (SSD) or normalized cross correlation (NCC) to calculate the corresponding point described above.

[0029] Next, the image processing unit 24 calculates transformation coefficients from the amount of positional deviation between the reference image and the target image. The image processing unit 24 uses, for example, projective transformation coefficients as the transformation coefficients. However, the transformation coefficients are not limited to projective transformation coefficients, and affine transformation coefficients or simplified transformation coefficients that only include horizontal and vertical shifts may also be used.

[0030] For example, the image processing unit 24 can perform the transformation using the formula shown in (Formula 1).

[0031]

number

[0032] In (Equation 1), (x', y') indicate the coordinates after transformation, and (x, y) indicate the coordinates before transformation. Matrix A indicates the transformation coefficients.

[0033] After the alignment, the depth stacking unit 25 calculates a contrast value for the image after the alignment.

[0034] As an example of a method for calculating the contrast value, for example, first, the luminance Y is calculated from the color signals Sr, Sg, and Sb of each pixel using the following (Equation 2). Y=0.299Sr+0.587Sg+0.114Sb (Formula 2)

[0035] Next, a contrast value I is calculated for a matrix L of luminance Y of 3×3 pixels using a Sobel filter as shown in the following (Equation 3) to (Equation 5).

[0036]

number

[0037]

number

[0038]

number

[0039] Furthermore, the above-mentioned method of calculating the contrast value is merely an example, and it is also possible to use an edge detection filter such as a Laplacian filter or a band-pass filter that passes through a predetermined band.

[0040] Next, the depth stacking unit 25 generates a synthesis map by comparing the contrast values ​​of pixels at the same position in each image and calculating a synthesis ratio according to the magnitude of the contrast value.

[0041] An example of a specific calculation method is shown below.

[0042] The depth stacking unit 25 generates a composite map Am(x,y) using the contrast value Cm(x,y). Here, m is the mth image among multiple images with different focus positions, x is the horizontal coordinate of the image, and y is the vertical coordinate. To generate the composite map, the depth stacking unit 25 compares the contrast values ​​of pixels at the same position in each image and calculates a composite ratio according to the magnitude of the contrast value. Specifically, among the images at the same position, a composite ratio of 100% is assigned to the pixel with the largest contrast value, and a composite ratio of 0% is assigned to other pixels at the same position. In other words, the following (Equation 6) holds.

[0043]

number

[0044] However, it is necessary to adjust the blending ratio appropriately so that the boundaries do not look unnatural. As a result, the blending ratio of the blending map for one image is not a binary value between 0% and 100%, but rather changes continuously.

[0045] The depth stacking unit 257 generates an omni-focus image O(x, y) by combining the captured images according to the calculated synthesis map. If the captured original image is Im(x, y), the image is generated by the following (Equation 7).

[0046]

number

[0047] The above is a description of an example of the focus stacking process in step S305.

[0048] Next, the scene determination described in step S301 will be described. In this embodiment, as an example, a case will be described in which the face of the person being photographed is detected as the main subject from a live view image, and scene determination is performed based on the position, size, number, etc. of the face of the person being photographed. FIG. 4 is a diagram for explaining an example of main subject detection in this embodiment. First, the AF evaluation value detection unit 23 detects the face of the person being photographed. Next, the system control unit 50 calculates an importance 401 based on the position of the face of the person being photographed detected by the AF evaluation value detection unit 23, an importance 402 based on the proportion of the face of the person being photographed occupying the angle of view, and an importance 403 based on the number of faces of the person being photographed. Specifically, the system control unit 50 may set the importance 401 to a value that is negatively correlated with the distance between the center position 404 of the face of the person being photographed and the center 405 of the angle of view. The system control unit 50 may set the importance 402 to a value that is positively correlated with the proportion of the face 407 of the person being photographed occupying the angle of view 406. The system control unit 50 may set the importance 403 to a value that is positively or negatively correlated with the number of faces of the person being photographed.

[0049] If the importance of the main subject is higher than a predetermined threshold, the system control unit 50 determines to perform depth stacking. When the system control unit 50 performs depth stacking, it determines to place the focus position on the main subject (the face of the person being photographed), and can generate a composite image that focuses on the main subject and has a blurred background. On the other hand, when the system control unit 50 does not perform depth stacking, it captures only one image with a deep depth of field that includes the background.

[0050] Furthermore, in the above description, it is assumed that the main subject is the face of the person being photographed, but this is not limiting.

[0051] In step S301, the system control unit 50 calculates the area other than the main subject detected by the AF evaluation value detection unit 23 as a background area 501. FIG. 5 is a diagram illustrating an example of background area detection in this embodiment. The system control unit 50 acquires an image focused on the background area and calculates the importance of the background area. If the importance of the background is lower than a predetermined threshold, the system control unit 50 determines to perform focus stacking. For example, in FIG. 5, the system control unit 50 increases the importance of the background in image 501 if the AF evaluation value detection unit 23 detects subject 502 within the background area. For images 501 and 503, the system control unit 50 may determine whether the image is outdoors based on the calculation results of the exposure control value and ranging control value obtained by the image processing unit 24, and increase the importance of the background if the image is outdoors. When capturing outdoor images, particularly when landmarks are present in the background, it is preferable to capture a single image with a deep depth of field so that the landmarks can be composited into the image. Increasing the importance of the background when capturing outdoors is a process that assumes that landmarks are present in the background of outdoor image capture.

[0052] Alternatively, system control unit 50 may perform both main subject detection and background detection, and determine whether to perform focus stacking based on the importance of both.

[0053] According to the first embodiment, the image processing device detects the main subject, background, and the like from a live view image, and can automatically determine whether to perform focus stacking based on the detection result.

[0054] (Second embodiment) A second embodiment of the present invention will be described below with reference to the drawings. Unlike the first embodiment, the second embodiment also determines whether to perform focus stacking before and after scene determination. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0055] 6 is a flowchart for explaining the focus stacking process in the second embodiment. The second embodiment is characterized in that a determination as to whether to perform focus stacking is made in steps S601 and S602 before and after the scene determination in step S301.

[0056] In step S601, the system control unit 50 determines whether to perform focus stacking based on the live view image before scene determination or the settings of the digital camera 100. If the system control unit 50 determines in step S601 that focus stacking will not be performed, the process proceeds to step S306, and controls the system to capture only one image.

[0057] As an example of the determination to perform focus stacking in step S601, if the motion detection unit 26 detects a moving object, the system control unit 50 determines not to perform focus stacking and transitions to S306. This is processing to prevent deterioration in the quality of the composite image due to the synthesis of moving areas.

[0058] Next, the determination of whether to perform focus stacking in step S602 will be described. After the system control unit 50 enables focus stacking in step S302 and determines the focus position in step S303, it determines whether to perform focus stacking in step S602. That is, unlike step S601, at the time of step S602, the settings for focus stacking (for example, the number of images to be captured, focus step, etc.) have already been made.

[0059] In step S602, the system control unit 50 determines whether to perform focus stacking based on the settings for focus stacking determined in step S303. For example, if the determined number of images to be captured is greater than a predetermined threshold, the system control unit 50 decides not to perform focus stacking. This is a process to prevent an increase in processing time due to an increase in the number of images to be captured. Note that in this case, in the image capture in step S307, the aperture may be adjusted so that a single image is captured having a depth of field equivalent to the depth of field of the composite image when focus stacking is performed.

[0060] Furthermore, in step S602, the system control unit 50 predicts the depth of field of the depth stacking image from the determined focus position and the depth of field of one image, and if the depth of field of the depth stacking image is shallower than a predetermined threshold, it decides not to perform depth stacking. This is because there is little benefit to performing depth stacking if the depth of field of the composite image is shallow. Here, the system control unit 50 may also simply predict the depth of field of the depth stacking image from the focus positions on both ends of the determined focus position.

[0061] In this embodiment, only one of the processing in step S601 and the processing in step S602 may be executed.

[0062] According to the second embodiment, if it is determined that there is no need to perform focus stacking before or after scene determination based on the state of the camera, moving object detection, or determined focus position, it can be decided not to perform focus stacking.

[0063] (Other embodiments) The above embodiment has been described based on implementation in a digital camera, but is not limited to a digital camera. For example, the present invention may be implemented in a mobile device with a built-in image sensor, or in a network camera capable of capturing images.

[0064] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and run the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0065] 100 digital cameras 101 Shutter 102 Barrier 103 Focus Lens 13 D / A converter 15 Memory control unit 18 Recording Media I / F 22 Imaging unit 23 AF evaluation value detection section 24 Image processing section 25 Depth synthesis section 26 Motion detection unit 27 Status detection unit 28 Display section 32 memory 40 Power supply section 50 System control section 52 system memory 53 System Timer 56 Non-volatile memory 60 Mode switch 61 Shutter button 62 Second shutter switch 64 First shutter switch 70 Operation section 72 Power switch 80 Power supply control unit 90 Strobe 200 Recording Media

Claims

1. An imaging means for acquiring an image; A determination means for performing scene determination on an image captured by live view; a synthesis means for synthesizing images, the determining means determines the importance of the scene based on the position of the subject, the proportion of the subject to the angle of view, and the number of subjects; If the importance is greater than a predetermined first threshold, Acquire multiple images with different focus positions in the optical axis direction, the combining means performs the combining on the plurality of images to generate a combined image having a deeper depth of field than the plurality of images; If the importance is equal to or less than a predetermined first threshold, the imaging means captures one image having a predetermined depth of field, The synthesis means does not perform the synthesis.

1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the determining means performs the scene determination for the plurality of images.

3. 2. The image processing device according to claim 1, wherein the importance is negatively correlated with the distance between the position of the subject and the center of the angle of view.

4. 2. The image processing device according to claim 1, wherein the proportion of the subject in the angle of view and the importance are positively correlated.

5. When the determination means determines that the image captured using the live view was captured outdoors, the determination means increases the importance of a background region in the image; If the importance of the background region is lower than a predetermined second threshold, the synthesis is performed; 5. The image processing device according to claim 1, wherein when the importance of the background region is equal to or greater than the second threshold, the combining means does not perform the combining and acquires a single image having a predetermined depth of field.

6. The determination means determines whether there is a moving object, 6. The image processing apparatus according to claim 1, wherein the combining means does not perform the combining when the moving object is present.

7. 7. The image processing apparatus according to claim 1, wherein the combining means extracts in-focus areas of the plurality of images to generate the combined image.

8. 8. The image processing device according to claim 1, wherein if the depth of field of the composite image generated by the combining means when performing the combining is shallower than a predetermined second threshold, the combining means does not perform the combining.

9. An imaging step for acquiring an image; a determination step of performing scene determination on an image captured by live view; a synthesis step of synthesizing the images, In the determining step, the importance in determining the scene is determined based on the position of the subject, the proportion of the subject to the angle of view, and the number of subjects; If the importance is greater than a predetermined first threshold, In the imaging step, a plurality of images having different focus positions in the optical axis direction are acquired, In the combining step, the combining is performed on the plurality of images to generate a combined image having a deeper depth of field than the plurality of images; If the importance is equal to or less than a predetermined first threshold, In the imaging step, an image having a predetermined depth of field is captured; In the synthesis step, the synthesis is not performed. An image processing method comprising:

10. A program for causing a computer to operate an image processing device, an imaging step of acquiring an image; a determination step of performing scene determination on an image captured by live view; a synthesis step of synthesizing the images; In the determining step, the importance in determining the scene is determined based on the position of the subject, the proportion of the subject to the angle of view, and the number of subjects; If the importance is greater than a predetermined first threshold, In the imaging step, a plurality of images are acquired at different focus positions in the optical axis direction, In the combining step, the combining is performed on the plurality of images to generate a combined image having a deeper depth of field than the plurality of images; If the importance is equal to or less than a predetermined first threshold, In the imaging step, an image having a predetermined depth of field is captured; In the synthesis step, the synthesis is not performed. A program characterized by:

11. A computer-readable storage medium storing the program according to claim 10.

Images