Image processing device, imaging device, image processing method, program, and storage medium

Abstract

To provide an image processing device capable of performing setting relating to depth synthesis without a user operation.SOLUTION: An image processing device (100) includes acquisition means (50) for acquiring lens information, synthesis means (25) for synthesizing a plurality of images different in focal position from each other photographed by using a lens corresponding to the lens information and generating a synthetic image having a depth of field deeper than those of the plurality of images, and determination means (50) for determining whether or not photographing is macro- photographing. The synthesis means synthesizes the plurality of images on the basis of information relating to a photographing magnification acquired from the lens information. The synthesis means synthesizes the plurality of images, when it is determined that the photographing is the macro-photographing by the determination means.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, a program, and a storage medium. [Background technology]

[0002] There is a known technology called depth stacking, which captures multiple images with different focus positions in the optical axis direction (focus bracketing imaging), extracts the in-focus areas of each image, and synthesizes an image with an extended depth of field. In order to generate a depth stacked image as intended by a user, it is necessary to set an appropriate depth of field. Patent Document 1 discloses a method in which a user specifies an arbitrary range on a screen and a synthetic image is generated based on distance information of the range specified by the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-39613 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the method disclosed in Patent Document 1, settings related to focus stacking, such as the range of focus stacking, cannot be made without user operation.

[0005] Therefore, an object of the present invention is to provide an image processing device that allows settings related to focus stacking to be made without user operation. [Means for solving the problem]

[0006] An image processing device as one aspect of the present invention has an acquisition means for acquiring lens information, a synthesis means for synthesizing a plurality of images having different focus positions taken using a lens corresponding to the lens information to generate a synthetic image having a deeper depth of field than the plurality of images, and a determination means for determining whether or not macro photography is being performed, wherein the synthesis means synthesizes the plurality of images based on information relating to the shooting magnification acquired from the lens information, and the synthesis means synthesizes the plurality of images if the determination means determines that macro photography is being performed.

[0007] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device that allows settings related to focus stacking to be made without user operation. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram of an imaging device in each embodiment. [Diagram 2] FIG. 2 is an explanatory diagram of a depth stacking image in the first embodiment. [Diagram 3] 4 is a flowchart showing a process of focus stacking in the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram of an example of detection of a main subject in the first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram of an example of detection of a background region in the first embodiment. [Figure 6] 13 is a flowchart showing a process of focus stacking in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0011] (First embodiment) First, an imaging device (image processing device) 100 according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram of the imaging device 100. Note that in this embodiment, the imaging device 100 is a digital camera, but is not limited to this, and may be other devices that are equipped with an imaging element and can acquire images, such as a mobile device or a network camera.

[0012] The imaging device 100 includes an imaging system including a shutter 101, a barrier 102, a focus lens 103, and an imaging unit 22. In the imaging device 100, the camera body and the lens device may be integrated together, or the lens device may be detachably attached to the camera body.

[0013] The shutter 101 is a shutter with an aperture function. The barrier 102 covers the imaging system of the imaging device 100 to prevent the imaging system from being dirty or damaged. The focus lens 103 is a lens included in a lens group (not shown) arranged between the shutter 101 and the barrier 102. The lens group described above also includes other lenses such as a zoom lens. The imaging unit 22 includes an imaging element configured with a CCD or CMOS element that converts an optical image into an electrical signal, and an A / D conversion processing function. Output data (captured image) from the imaging unit 22 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. During focus bracketing imaging described later, all of the captured images up to the set number are written into the memory 32.

[0014] The imaging device 100 includes an AF evaluation value detection section (detection means) 23, a strobe 90, an image processing section 24, a depth stacking section (synthesizing means) 25, a motion detection section (moving object determination means) 26, a state detection section 27, a memory 32, a D / A converter 13, and a display section 28. The imaging device 100 also includes a non-volatile memory 56, a system control section 50, a system memory 52, and a system timer 53. The AF evaluation value detection section 23 is provided inside the imaging section 22, and calculates an AF evaluation value from contrast information and the like obtained from a digital image signal, and outputs the obtained AF evaluation value from the imaging section 22 to the system control section 50. The strobe 90 can compensate for illuminance when imaging in a low-illuminance scene or a backlit scene by emitting light during imaging.

[0015] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the image data output from the imaging unit 22 or image data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic results. This allows TTL (through-the-lens) type AE (automatic exposure) processing and EF (automatic flash light control) processing to be performed. The image processing unit 24 also performs AF (autofocus) processing. At this time, the output of the AF evaluation value detection unit 23 provided in the imaging unit 22 may also be used. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the obtained arithmetic results.

[0016] The depth stacking unit 25 generates an image with an extended depth of field by outputting pixels in focus in each image using a plurality of captured images obtained by focus bracketing imaging in the imaging unit 22. Details of this will be described later.

[0017] The motion detection unit 26 uses two pieces of image data to perform template matching processing between a region of interest and its surroundings, and calculates a motion vector for each region into which the image is divided, or for each pixel. If the calculated motion vector is equal to or greater than a threshold, the motion detection unit 26 determines that there is motion in the subject (there is a moving object), and notifies the system control unit 50. The status detection unit 27 detects the status of the imaging device 100, such as the angular velocity detected by the gyro sensor, the attachment / detachment state of the tripod, and the settings made by the user via the operation unit 70. The status detection unit 27 notifies the system control unit 50 of the detection result.

[0018] The memory 32 stores image data acquired and A / D converted by the imaging unit 22, and image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 32 also serves as a memory for displaying images (video memory). The D / A converter 13 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28. As a result, the image data for display written in the memory 32 is displayed on the display unit 28 via the D / A converter 13. The display unit 28 performs display according to the analog signal from the D / A converter 13 on a display such as an LCD. The digital signal once A / D converted by the imaging unit 22 and stored in the memory 32 is converted into an analog signal by the D / A converter 13, and is sequentially transferred to and displayed on the display unit 28, thereby functioning as an electronic viewfinder and enabling through image display (live view).

[0019] The non-volatile memory 56 is an electrically erasable and recordable memory, such as a flash memory. Constants, programs, and the like for operating the system control unit 50 are stored in the non-volatile memory 56. The programs referred to here include, for example, programs for executing various flowcharts described later.

[0020] The system control unit 50 controls the entire imaging device 100. Specifically, the system control unit 50 executes a program recorded in the non-volatile memory 56 described above to realize focus bracketing imaging based on subject information, subject distance, and image contrast information. That is, the system control unit 50 controls the driving of the focus lens 103 and the shutter 101 during imaging, thereby sequentially capturing a plurality of images with different focus positions. Note that the amount of change in focus position (focus step) between adjacent captured images obtained by imaging processing is set from a value calculated by the system control unit 50.

[0021] The system memory 52 is composed of RAM etc., and develops constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 56, etc. The system control unit 50 also performs display control by controlling the memory 32, D / A converter 13, display unit 28 etc. The system timer 53 is a timing unit that measures the time used for various controls or the time of a built-in clock.

[0022] The imaging device 100 also includes an operating unit for inputting various operation instructions to the system control unit 50, which is composed of a mode changeover switch 60, a shutter button 61, a first shutter switch 64, a second shutter switch 62, an operating unit 70, and a power switch 72. The mode changeover switch 60 switches the operation mode of the system control unit 50 to one of a still image recording mode, a moving image recording mode, a playback mode, etc. Modes included in the still image recording mode include an auto imaging mode, an auto scene discrimination mode, a manual mode, various scene modes in which imaging settings are made for each imaging scene, a program AE mode, a custom mode, etc. The mode changeover switch 60 directly switches to one of these modes included in the still image imaging mode. Alternatively, after switching to the still image imaging mode once with the mode changeover switch 60, it is also possible to switch to one of these modes included in the still image imaging mode using other operating members. Similarly, the moving image imaging mode may also include a plurality of modes.

[0023] The shutter button 61 is an operation unit for issuing an image capture instruction. The first shutter switch 64 is turned on when the shutter button 61 is pressed halfway during operation, i.e., pressed halfway, and generates a first shutter switch signal SW1. The first shutter switch signal SW1 starts operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (automatic flash light control) processing. That is, parameters for image capture are acquired under the control of the system control unit 50. Note that the user can select a central one-point AF processing or a face AF processing as the AF processing to be started upon receiving the signal SW1. Here, the central one-point AF processing refers to a processing to perform AF on a central point in the image capture screen, and the face AF processing refers to a processing to perform AF on a face in the image capture screen detected by a face detection function. The second shutter switch 62 is turned on when the shutter button 61 is pressed fully (image capture instruction), and generates a second shutter switch signal SW2. The system control unit 50 starts a series of imaging processing operations from reading out a signal from the imaging unit 22 to writing image data to the recording medium 200 in response to the second shutter switch signal SW2.

[0024] Each operation member of the operation unit 70 is assigned a function for each scene by selecting and operating various function icons displayed on the display unit 28, and acts as various function buttons. Examples of the function buttons include an end button, a back button, an image forward button, a jump button, a filter button, and an attribute change button. For example, when the menu button is pressed, a menu screen in which various settings can be made is displayed on the display unit 28. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four directional buttons (up, down, left, and right) and the SET button. The power switch 72 switches the power of the imaging device 100 on and off.

[0025] The imaging device 100 includes a power supply control unit 80, a power supply unit 40, and a recording medium I / F 18. The power supply control unit 80 is configured with a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and the like, and detects whether a battery is attached, the type of battery, and the remaining battery level. The power supply control unit 80 also controls the DC-DC converter based on the detection result and an instruction from the system control unit 50, and supplies the required voltage to each unit including the recording medium 200 for the required period. The power supply unit 40 is configured with a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, and the like. The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is configured with a semiconductor memory, a magnetic disk, and the like.

[0026] It should be noted that focus bracketing imaging refers to imaging in which the focus lens 103 is moved to a desired focus position, and then exposure and readout of the imaging unit 22 are performed.

[0027] Next, a basic operation of the depth stacking process in the imaging device 100 will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram of a depth stacking image in this embodiment, and shows an example of a display image displayed on the display unit 28 when the depth stacking image is live-viewed.

[0028] Position 201 is the close side focus position of focus bracketing imaging indicated by an AF frame, position 202 is the infinity side focus position, and position 203 is the subject (insect). The AF frame indicating the close side focus position 201 and the subject 203 are displayed on the display unit 28. The user operates the operation unit 70 to specify a reference focus position on the image displayed on the display unit 28. Alternatively, the reference focus position may be specified automatically for the subject detected by the AF evaluation value detection unit 23. In FIG. 2, the closest location of the subject 203 is specified as the reference focus position. As a result, the system control unit 50 recognizes the specified reference focus position as the close side focus position 201 of focus bracketing imaging, and displays an AF frame at the close side focus position 201.

[0029] Once the close side focus position 201 is determined, the focus position where the focus bracketing imaging ends, i.e., the infinity side focus position 202, is determined in the Z axis direction (depth direction) based on the focus interval according to the focus step setting and the number of times imaging is performed. In Fig. 2, it is assumed that the entire subject 203 falls within the focus range indicated by the close side focus position 201 to the infinity side focus position 202. Also, it is assumed that the number of times imaging is performed to generate one depth stacked image is 10, and the depth stacking unit 25 performs depth stacking processing using the 10 captured images.

[0030] Next, the focus stacking process in this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the focus stacking process in this embodiment.

[0031] First, in step S301, the system control unit 50 performs scene determination on an image captured in live view by the imaging unit 22. Note that a specific method of scene determination will be described later.

[0032] Next, in step S302, the system control unit (acquisition means) 50 acquires information (lens information) about the lens (optical system) including the focus lens 103, and performs macro shooting determination based on the lens information. Macro shooting is a shooting method for taking a large image of a small subject, and the subject is photographed by approaching it. Therefore, the shooting magnification tends to be large, the shooting distance tends to be short, and the depth of field tends to be shallow. In the case of macro shooting, the depth of field becomes shallow, and the area in which the subject to be photographed is in focus becomes small. For this reason, in order to generate an image in which a wider range of the subject is in focus by depth stacking, it is determined in advance whether or not it is macro shooting. The lens information is acquired by the system control unit 50 communicating with the lens via a lens communication unit (not shown). The lens information includes the ID of the lens or information about the shooting magnification (the numerical value of the shooting magnification itself, or specific information corresponding to the shooting magnification), etc. In the macro shooting determination, for example, if the ID of the lens is the ID of a specific close-up lens (specific lens information), or if the shooting magnification calculated from the lens information is equal to or greater than a certain threshold, it is determined that it is macro shooting. The shooting magnification may be calculated by the system control unit 50 from the lens information, or the shooting magnification may be directly acquired as the lens information.

[0033] Next, in step S303, the system control unit 50 determines whether or not to enable the focus stacking function using the scene determination result in step S301 and the macro shooting determination result in step S302. If it is determined that the focus stacking function is to be enabled, the process proceeds to step S304. On the other hand, if it is determined that the focus stacking function is not to be enabled, the process proceeds to step S307. In step S304, the system control unit 50 sets multiple focus positions. In step S307, the system control unit 50 sets one focus position.

[0034] Although the details will be described later, in the scene determination process (step S301), the system control unit 50 mainly obtains the importance of the main subject and the background by image analysis, and determines whether or not to perform (enable) depth stacking based on the obtained importance. This makes it possible to generate an image in which only the main subject is in focus, and also to generate an image in which background landmarks are in focus. However, it is difficult to determine by image analysis alone that the shooting is difficult to focus on, such as macro shooting, because the depth of field is shallow. Therefore, in the macro shooting determination process (step S302), the system control unit 50 determines that the shooting is difficult to focus on by analyzing the lens information, and performs depth stacking based on the determination result. This makes it possible to realize appropriate depth control in a wider range of shooting scenes.

[0035] In step S304, the system control unit 50 determines the focus position and the number of times of imaging for depth stacking. The system control unit 50 may also set a focus step according to the exposure control value and change the number of focus positions. This is a process aimed at shortening the processing time and reducing the amount of blur by reducing the number of shots taken in response to the increase in shooting time and the amount of camera shake that accompanies an increase in exposure time. The system control unit 50 may also determine whether or not to change the number of focus positions according to the attachment / detachment state of the tripod detected by the state detection unit 27.

[0036] Next, in step S305, the system control unit 50 performs drive control of the focus lens 103 and the shutter 101, so that the imaging unit 22 sequentially captures images at the multiple different focus positions set in step S304. Next, in step S306, the depth stacking unit (synthesizing means) 25 generates an image (synthetic image) with an expanded depth of field using the multiple images (captured images) captured in step S305. That is, the depth stacking unit 25 synthesizes multiple images with different focus positions to generate a synthetic image with a deeper depth of field than the multiple images. At this time, the depth stacking unit 25 extracts the respective focus areas of the multiple images to generate a synthetic image.

[0037] In step S308, the system control unit 50 controls the driving of the focus lens 103 and the shutter 101, so that the imaging unit 22 captures an image at the focus position set in step S307.

[0038] Next, the focus stacking process in step S306 will be described in detail. Note that the focus stacking process in this embodiment may use a known technique, and is performed, for example, as follows.

[0039] First, the image processing unit 24 performs position alignment on the image to be subjected to depth stacking. The image processing unit 24 sets a plurality of blocks in a reference image, which is one of a plurality of images to be subjected to depth stacking. The image processing unit 24 preferably sets the size of each block to be the same. Next, the image processing unit 24 sets a search range in the image to be subjected to position alignment, which is a range wider than the block of the reference image, at the same position as each block of the reference image. Finally, the image processing unit 24 calculates a corresponding point in each search range of the target image, which has a minimum sum of absolute difference (SAD) between the block of the reference image and the corresponding point described above. The image processing unit 241 calculates a positional deviation as a vector from the center of the block of the reference image and the corresponding point described above. In addition to SAD, the image processing unit 241 may use Sum of Squared Difference (SSD) or Normalized Cross Correlation (NCC) in calculating the corresponding point described above.

[0040] Next, the image processing unit 24 calculates a conversion coefficient from the amount of positional deviation between the reference image and the target image. The image processing unit 24 uses, for example, a projection transformation coefficient as the conversion coefficient. However, the conversion coefficient is not limited to the projection transformation coefficient, and an affine transformation coefficient or a simplified conversion coefficient including only a horizontal and vertical shift may be used.

[0041] For example, the image processing unit 24 can perform the transformation using the following (Equation 1).

[0042]

number

[0043] In (Equation 1), (x', y') indicate the coordinates after transformation, and (x, y) indicate the coordinates before transformation. Matrix A indicates the transformation coefficients.

[0044] After the alignment, the depth stacking unit 25 calculates a contrast value for the image after the alignment. As a method of calculating the contrast value, for example, first, the luminance Y is calculated by the following (Equation 2) using the color signals Sr, Sg, and Sb of each pixel.

[0045] Y=0.299Sr+0.587Sg+0.114Sb (Formula 2) Next, a contrast value I is calculated for a matrix L of luminance Y of 3×3 pixels using a Sobel filter as shown in the following (Equation 3) to (Equation 5).

[0046]

number

[0047] Furthermore, the above-mentioned method of calculating the contrast value is merely an example, and for example, it is also possible to use an edge detection filter such as a Laplacian filter or a bandpass filter that passes through a predetermined band.

[0048] Next, the depth stacking unit 25 generates a synthesis map. As a method of generating the synthesis map, the depth stacking unit 25 compares the contrast values ​​of pixels at the same position in each image and calculates a synthesis ratio according to the magnitude of the contrast value.

[0049] An example of a specific calculation method is shown below. The depth stacking unit 25 generates a synthesis map Am(x,y) using the contrast value Cm(x,y). Here, m is the mth image among the multiple images with different focus positions, x is the horizontal coordinate of the image, and y is the vertical coordinate. As a method of generating the synthesis map, the depth stacking unit 25 compares the contrast values ​​of pixels at the same position in each image and calculates a synthesis ratio according to the magnitude of the contrast value. Specifically, among the images at the same position, a synthesis ratio of 100% is given to the pixel with the largest contrast value, and a synthesis ratio of 0% is given to other pixels at the same position. In other words, the following (Equation 6) is established.

[0050]

number

[0051] However, it is necessary to adjust the blending ratio appropriately so that the boundaries do not look unnatural. As a result, the blending ratio of the blending map for one image is not a binary ratio of 0% and 100%, but rather changes continuously.

[0052] The depth stacking unit 257 generates an all-focus image O(x, y) by synthesizing the captured images according to the calculated synthesis map. If the captured original image is Im(x, y), an image is generated as expressed by the following (Equation 7).

[0053]

number

[0054] The above is a description of an example of the focus stacking process in step S306.

[0055] Next, the scene determination in step S301 will be described with reference to FIG. 4. In this embodiment, as an example, a case will be described in which a face is detected as a main subject from a live view image, and scene determination is performed based on the position, size, number, etc. of the face. FIG. 4 is an explanatory diagram of an example of detection of a main subject in this embodiment. First, the AF evaluation value detection unit 23 detects a face (main subject). Next, the system control unit 50 calculates an importance 401 based on the position of the face detected by the AF evaluation value detection unit 23, an importance 402 based on the proportion of the face occupying the angle of view, and an importance 403 based on the number of faces. Specifically, the system control unit 50 may set a value negatively correlated to the distance between the face center position 404 and the center 405 of the angle of view in the importance 401. The system control unit 50 may set a value positively correlated to the proportion of the face 407 occupying the angle of view 406 in the importance 402. The system control unit 50 may set a value positively correlated or negatively correlated to the number of faces in the importance 403.

[0056] If the importance of the main subject is higher than a predetermined threshold, the system control unit 50 determines to perform depth stacking. If the system control unit 50 determines to perform depth stacking, it determines to place the focal position on the main subject (face), and can generate a composite image that focuses on the main subject and has a blurred background. On the other hand, if the system control unit 50 determines not to perform depth stacking, it captures only one image with a deep depth of field that includes the background.

[0057] If the face of the main subject is not detected, the system control unit 50 refers to the result of the macro shooting determination in step S302, and if it is determined that macro shooting is performed, it determines that depth stacking is to be performed. As described above, in the case of macro shooting, the depth of field tends to be shallow, and the area in which the subject to be photographed is in focus tends to be small. Even in this case, by performing depth stacking, it is possible to generate a composite image in which a wider range of the subject is in focus. Note that in the above description, it has been described on the assumption that the main subject is a face, but this is not limited to this.

[0058] In step S301, the system control unit 50 calculates, as a background region, the region other than the main subject detected by the AF evaluation value detection unit 23. Figures 5(a) and (b) are explanatory diagrams of an example of background region detection in this embodiment, where Figure 5(a) shows an outdoor image 501 and Figure 5(b) shows an indoor image 503.

[0059] The system control unit 50 acquires an image in which the focus is on the background region, and calculates the importance of the background region. If the importance of the background is determined to be lower than a predetermined threshold, the system control unit 50 determines to perform (enable) depth stacking. For example, in the image 501 of FIG. 5(a), the system control unit 50 increases the importance of the background when the AF evaluation value detection unit 23 detects a subject 502 in the background region. For each of the image 501 of FIG. 5(a) and the image 503 of FIG. 5(b), the system control unit 50 determines whether or not the image is outdoors according to the calculation results of the exposure control value and the distance measurement control value obtained by the image processing unit 24. In the case of an outdoor image, the importance of the background may be increased. In the case of an outdoor image, particularly when a landmark is present in the background, it is preferable to capture one image with a deep depth of field in order to also composite the landmark. In the case of outdoors, increasing the importance of the background is a process assuming that a landmark is present in the background of the outdoor image capture.

[0060] The system control unit 50 may also perform both detection of the main subject and detection of the background, and determine whether or not to perform focus stacking based on the importance of both. In this embodiment, when the face of the main subject is not detected, the system control unit 50 determines whether or not to perform focus stacking based on the macro shooting determination result, but this is not limited to this. For example, even when a face is detected, the system control unit 50 may determine whether or not to perform focus stacking based on the macro shooting determination result. Also, the system control unit 50 may perform only macro shooting determination without performing scene determination, and determine whether or not to perform focus stacking based on the macro shooting determination result.

[0061] According to this embodiment, it is possible to automatically determine whether or not to perform focus stacking based on the shooting magnification obtained from the lens information.

[0062] Second embodiment Next, a second embodiment of the present invention will be described with reference to Fig. 6. Unlike the first embodiment, this embodiment determines whether or not to perform focus stacking both before and after scene determination. Note that in this embodiment, descriptions common to the first embodiment will be omitted. Fig. 6 is a flowchart showing the process of focus stacking in this embodiment. Fig. 6 differs from Fig. 3 in that steps S601 and S602 are inserted before and after the scene determination in step S301, respectively, to determine whether or not to perform focus stacking.

[0063] First, in step S601, the system control unit 50 determines whether or not to perform depth stacking based on the live view image before scene determination or the settings of the imaging device 100. If it is determined that depth stacking is not to be performed, the process proceeds to step S307, where the system control unit 50 controls the imaging unit 22 to capture only one image. In step S601, for example, if the motion detection unit 26 detects a moving object, the system control unit 50 determines that depth stacking is not to be performed. This is to prevent deterioration of the quality of the composite image caused by the composition of a moving area. Also, in step S601, for example, the system control unit 50 may determine whether or not the imaging device 100 is in a self-portrait mode in which the user himself or herself is photographed, depending on the state (camera state) of the imaging device 100 detected by the state detection unit 27. An example of the camera state is a state in which the monitor faces the lens direction. In the case of a vari-angle monitor, the monitor is opened and rotated 180 degrees. In the case of a tilt monitor, the monitor is moved in any direction and faces the lens direction. In addition, even in an imaging device where the lens direction matches the monitor, such as the front camera of a smartphone, it is possible to automatically determine whether to perform depth stacking. This is because in selfie mode, the main subject and the background are far apart in the optical axis direction and are not suitable for depth stacking.

[0064] Next, the determination of whether to perform focus stacking in step S602 will be described. After enabling the focus stacking function in step S303 and determining the focus position in step S304, the system control unit 50 determines whether to perform focus stacking in step S602. That is, unlike step S601, the settings for focus stacking (e.g., the number of imaging times, focus steps, etc.) have already been completed at the time of step S602.

[0065] In step S602, the system control unit 50 determines whether to perform depth stacking based on the settings for depth stacking determined in step S304. For example, if the determined number of captured images is greater than a predetermined threshold, the system control unit 50 determines not to perform depth stacking. This is to prevent an increase in processing time due to an increase in the number of captured images. In this case, in the image capturing in step S308, the diaphragm (aperture diaphragm) may be adjusted so as to capture one image having a depth of field equivalent to the depth of field of the composite image when depth stacking is performed.

[0066] In step S602, the system control unit 50 predicts the depth of field of the depth stacking image from the determined focus position and the depth of field of one image, and determines not to perform depth stacking if the depth of field of the depth stacking image is shallower than a predetermined threshold. This is because there is little benefit to performing depth stacking if the depth of field of the composite image is shallow. Here, the system control unit 50 may simply predict the depth of field of the depth stacking image from the focus positions at both ends of the determined focus position. Note that in this embodiment, only one of the processing of step S601 and the processing of step S602 may be executed.

[0067] According to this embodiment, if it is determined that there is no need to perform depth stacking before or after scene determination and macro shooting determination based on the state of the camera, moving object detection, or a determined focus position, it is possible to avoid performing depth stacking.

[0068] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0069] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium that enable settings related to focus stacking to be configured without user operation.

[0070] The disclosure of each embodiment includes the following configurations and methods.

[0071] (Configuration 1) An acquisition means for acquiring lens information; a synthesis means for synthesizing a plurality of images having different focus positions to generate a synthetic image having a deeper depth of field than the plurality of images, The image processing device according to claim 1, wherein the combining means combines the plurality of images based on information relating to a shooting magnification obtained from the lens information. (Configuration 2) The imaging device further includes a detection unit for detecting a main subject, 2. The image processing device according to configuration 1, wherein, when the detection means has not detected the main subject, the synthesis means synthesizes the plurality of images based on information related to the photographing magnification. (Configuration 3) The apparatus further includes a moving object determination means for determining whether or not a moving object is present, 3. The image processing device according to configuration 1 or 2, wherein when the moving object determination means determines that there is a moving object, the synthesis means does not synthesize the multiple images. (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the synthesis means extracts in-focus areas of the plurality of images to generate the synthetic image. (Configuration 5) 5. The image processing device according to any one of claims 1 to 4, wherein the lens information includes information relating to the shooting magnification. (Configuration 6) 6. The image processing device according to configuration 5, wherein the information regarding the imaging magnification is the imaging magnification or specific lens information. (Configuration 7) 5. The image processing device according to any one of claims 1 to 4, wherein the acquisition means calculates the shooting magnification by using the lens information. (Configuration 8) 8. The image processing device according to claim 7, wherein, when the photographing magnification is equal to or greater than a certain threshold, the combining means combines the plurality of images. (Configuration 9) An image processing device according to any one of configurations 1 to 8; an imaging section for acquiring the plurality of images. (Method 1) An acquisition step of acquiring lens information; a synthesis step of synthesizing a plurality of images having different focus positions to generate a synthetic image having a deeper depth of field than the plurality of images, an image processing method comprising: combining the plurality of images in the combining step based on information relating to a shooting magnification obtained from the lens information; (Configuration 10) A program for causing a computer to execute the image processing method according to Method 1. (Configuration 11) A computer-readable storage medium storing the program according to configuration 10.

[0072] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0073] 100 Imaging device (image processing device) 23 AF evaluation value detection unit (detection means) 25 Depth synthesis section (synthesis means) 50 System control unit (acquisition means)

Claims

1. An acquisition means for acquiring lens information; a synthesis means for synthesizing a plurality of images having different focus positions that are taken using a lens corresponding to the lens information, to generate a synthetic image having a deeper depth of field than the plurality of images; A determination means for determining whether or not the photograph is macro photography, The synthesizing means synthesizes the plurality of images based on information related to the photographing magnification obtained from the lens information, The image processing device according to claim 1, wherein the combining means combines the plurality of images when the determining means determines that the image is macro photography.

2. 2. The image processing apparatus according to claim 1, wherein the determining means determines whether or not the image is macro photography based on the lens information.

3. The image processing apparatus according to claim 2 , wherein the acquisition means calculates a shooting magnification using the lens information.

4. 4. The image processing apparatus according to claim 3, wherein the determination means determines that the photographing is macro photography when the photographing magnification is equal to or greater than a certain threshold value.

5. 2. The image processing apparatus according to claim 1, wherein the synthesis means extracts in-focus areas of the plurality of images to generate the synthetic image.

6. a scene determination means for determining a scene at the time of shooting from an image different from the plurality of images; a detection unit for determining whether a predetermined subject is detected from an image different from the plurality of images, the combining means combines the plurality of images when the predetermined subject is detected and the scene satisfies a predetermined condition; 2. The image processing apparatus according to claim 1, wherein when the predetermined subject is not detected and the determining means determines that the photographing is macro photography, the image processing apparatus performs the plurality of photographing operations.

7. 7. The image processing apparatus according to claim 6, wherein the predetermined subject is a face.

8. 8. The image processing apparatus according to claim 7, wherein the scene determination means determines the importance based on the position of the face, the proportion of the face occupying an angle of view, and the number of the faces.

9. 9. The image processing apparatus according to claim 8, wherein the predetermined condition is that the importance is higher than a predetermined threshold value.

10. An image processing device according to any one of claims 1 to 9; an imaging section for acquiring the plurality of images.

11. An acquisition step of acquiring lens information; a synthesis step of synthesizing a plurality of images having different focus positions captured using a lens corresponding to the lens information to generate a synthetic image having a deeper depth of field than the plurality of images; A determination step of determining whether or not the photograph is macro photography, In the combining step, the plurality of images are combined based on information regarding a shooting magnification obtained from the lens information; The image processing method according to claim 1, wherein the combining step combines the plurality of images when the determining step determines that the image is macro photography.

12. A program for causing a computer to execute the image processing method according to claim 11.

13. A computer-readable storage medium storing the program according to claim 12.

Images