Image processing device, imaging apparatus, image processing method, computer program, and recording medium

Abstract

To provide a method for correcting the angle of view of an image obtained by focus bracketing in accordance with user settings, etc.SOLUTION: An image processing device has synthesizing means for synthesizing a plurality of images with different focus positions, correction means for correcting the angle of view for the plurality of images, and display means for displaying the corrected images when the correction means performs the correction. When the synthesizing means performs the synthesis, the correction means performs the correction. When the synthesizing means does not perform the synthesis, the correction means does not perform the correction.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an image processing device, and more particularly to an image processing device that displays a plurality of images with different focus positions. [Background technology]

[0002] Patent Document 1 discloses a so-called focus bracketing technique in which a plurality of images are captured while changing the focus position of a subject, and a plurality of images with different focus positions are generated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-10970 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the focus bracketing technique has a problem in that even if multiple images are captured without changing settings other than the focus position, the angle of view between the multiple captured images changes.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an image processing device that corrects the angle of view of images obtained by focus bracketing imaging. [Means for solving the problem]

[0006] The present invention provides an image processing device having a compositing means for compositing a plurality of images having different focus positions, a correction means for correcting the angle of view of the plurality of images, and a display means for displaying the image after the correction has been performed when the correction means has performed the correction, in which the correction means performs the correction when the compositing means performs the compositing, and the correction means does not perform the correction when the compositing means does not perform the compositing. Effect of the Invention

[0007] According to the image processing device of the present invention, the angle of view of images obtained by image capture using focus bracketing can be corrected in accordance with user settings and the like. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a structure of a digital camera according to an embodiment of the present invention. [Diagram 2] 4A and 4B are diagrams for explaining how a subject image is formed on an image forming surface in the embodiment of the present invention. [Diagram 3] 5 is a diagram for explaining a function showing the relationship between the focus position and the image magnification in the embodiment of the present invention. FIG. [Figure 4] 1A to 1C are diagrams illustrating imaging using focus stacking according to an embodiment of the present invention. [Diagram 5] 1 is a flowchart illustrating generation of a composite image according to an embodiment of the present invention. [Figure 6] 4 is a flowchart illustrating imaging according to an embodiment of the present invention. [Figure 7] 1 is a flowchart illustrating alignment in focus stacking according to an embodiment of the present invention. [Figure 8] 1 is a flowchart for explaining image synthesis of focus stacking according to an embodiment of the present invention. [Figure 9] 5 is a flowchart illustrating correction of the angle of view in an embodiment of the present invention. [Figure 10] 5A to 5C are diagrams illustrating an image cut-out process according to an embodiment of the present invention. [Figure 11] FIG. 2 is a diagram for explaining an image display in the embodiment of the present invention. [Figure 12] FIG. 13 is another diagram for explaining the display of an image in the embodiment of the present invention. [Figure 13]1 is a diagram for explaining an example of a display on a display unit 108 of a digital camera 100 according to this embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of image switching according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0010] <Digital Camera Overview> 1 is an example of a block diagram showing the structure of a digital camera for capturing an image according to this embodiment. Digital camera 100 can capture still images, record information on the focal position, calculate contrast values, and synthesize images. Furthermore, digital camera 100 can perform enlargement or reduction processing on captured and saved images or images input from the outside.

[0011] The control unit 101 is, for example, a signal processor such as a CPU or MPU, and controls each part of the digital camera 100 while reading a program previously stored in a ROM 105, which will be described later. For example, as will be described later, the control unit 101 issues commands to an imaging unit 104, which will be described later, to start and end imaging. Alternatively, the control unit 101 issues commands for image processing to an image processing unit 107, which will be described later, based on a program stored in the ROM 105. Commands from a user are input to the digital camera 100 by an operation unit 110, which will be described later, and reach each part of the digital camera 100 through the control unit 101.

[0012] The driving unit 102 is constituted by a motor and the like, and mechanically operates the optical system 103 described below under the command of the control unit 101. For example, based on the command of the control unit 101, the driving unit 102 moves the position of a focus lens included in the optical system 103 to adjust the focal length of the optical system 103.

[0013] The optical system 103 is composed of a zoom lens, a focus lens, an aperture, etc. The aperture is a mechanism for adjusting the amount of light that passes through. The focal position can be changed by changing the position of the lens.

[0014] The imaging unit 104 is a photoelectric conversion element that performs photoelectric conversion to convert an incident optical signal into an electrical signal. For example, a CCD sensor or a CMOS sensor can be applied to the imaging unit 104. The imaging unit 104 has a video imaging mode and can capture a plurality of images that are consecutive in time as each frame of a video.

[0015] ROM 105 is a read-only non-volatile memory serving as a recording medium, and stores parameters and the like necessary for the operation of each block in addition to the operation programs of each block of digital camera 100. RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output during the operation of each block of digital camera 100.

[0016] The image processing unit 107 performs various image processing such as white balance adjustment, color interpolation, filtering, etc. on the image output from the imaging unit 104 or on image signal data recorded in the internal memory 109 described later. In addition, the image processing unit 107 performs compression processing on the image signal data captured by the imaging unit 104 in accordance with a standard such as JPEG.

[0017] Image processing unit 107 is configured as an application specific integrated circuit (ASIC) that is a collection of circuits that perform specific processing. Alternatively, control unit 101 may perform part or all of the functions of image processing unit 107 by processing according to a program read from ROM 105 by control unit 101. When control unit 101 performs all of the functions of image processing unit 107, there is no need to have image processing unit 107 as hardware.

[0018] The display unit 108 is a liquid crystal display or an organic EL display for displaying images temporarily stored in the RAM 106, images stored in the internal memory 109 described below, or a setting screen of the digital camera 100.

[0019] The built-in memory 109 is a place for recording images captured by the imaging unit 104, images processed by the image processing unit 107, and information on the focal position when capturing an image, etc. A memory card or the like may be used instead of the built-in memory.

[0020] The operation unit 110 is, for example, a button, switch, key, or mode dial attached to the digital camera 100, or a touch panel that also serves as the display unit 108. Commands from a user reach the control unit 101 via the operation unit 110.

[0021] <Explanation of subject imaging> Below, subject imaging will be briefly explained.

[0022] FIG. 2 is a diagram for explaining how a subject image is formed on an imaging plane in this embodiment.

[0023] 2(a) shows a state in which a subject 201 is imaged as an image 204 on a surface 203a by an optical lens 202. In other words, if the surface 203a and the image sensor surface of the imaging unit 104 coincide with each other, the subject 201 is imaged as a "point" on the surface 203a and recorded as a focused image.

[0024] FIG. 2(b) shows a case where the image forming plane and the image sensor plane do not coincide. When the image sensor plane 203b is located at a position different from the plane 203a shown in FIG. 2(a), the subject 201 imaged by the optical lens 202 is reflected on the image sensor plane 203b as a circle of confusion 205. In the situation shown in FIG. 2(b), when the circle of confusion 205 is smaller than the allowable circle of confusion of the image sensor, the circle of confusion 205 can be regarded as equivalent to a "point" when in focus, and an image equivalent to a focused image is obtained. On the other hand, when the circle of confusion 205 is larger than the allowable circle of confusion, a blurred image is obtained on the image sensor plane 203b.

[0025] FIG. 2(c) is a side view of the above state. Here, when an image of a subject is formed at a focal point 210 and an imaging sensor surface exists at the position of the surface 211a, a circle of confusion diameter 212a is obtained. The circle of confusion diameter 212a shown in FIG. 2(c) is smaller than the allowable circle of confusion diameter 213 of the imaging sensor. Therefore, an image 217 recorded by the imaging sensor is a focused image without blur. On the other hand, when the imaging sensor surface exists at the position of the surface 214a, the circle of confusion diameter 215a at this time is larger than the allowable circle of confusion diameter 213. Therefore, an image 218a on the imaging sensor surface 214a is a blurred image. The area indicated by the diagonal lines where the circle of confusion diameter 212a is smaller than the allowable circle of confusion diameter 213 is the focal depth 216a, and the focal depth 216a converted and replaced on the subject side is the depth of field.

[0026] FIG. 2(d) is a diagram showing a state where the aperture is narrowed compared to FIG. 2(c). In the narrowed aperture state, the diameter of the incident light varies depending on the depth, and thus the diameter of the circle of confusion 212b for the surface 211b and the diameter of the circle of confusion 215b for the surface 214b change. Compared to the diameter of the circle of confusion 215a in FIG. 2(c), the diameter of the circle of confusion 215b in FIG. 2(d) is smaller. Therefore, the image 218b shown in FIG. 2(d) has a smaller amount of blur than the image 218a. Also, the focal depth 216b shown in FIG. 2(d) is deeper than the focal depth 216a.

[0027] <Focus Breathing> Focus breathing is a phenomenon that occurs when taking pictures with a digital camera, where the angle of view changes even if you move only the focus ring without moving the zoom lens at all. Focus breathing is caused by the magnification change effect that occurs when the focus group moves. When focusing on a close subject, the actual focal length of the lens can change significantly, and the reduction in focus breathing becomes more noticeable.

[0028] When capturing multiple images of a subject, focus breathing can cause the angle of view to change if the focus position is changed, even if other settings remain unchanged. In focus bracketing, simply changing the focus position can result in different angles of view between the captured images.

[0029] FIG. 3 is a diagram for explaining a function showing the relationship between the focal position and the image magnification in this embodiment. In the function shown in FIG. 3, the image magnification monotonically decreases with respect to the focal position. The relationship between the focal position and the image magnification as shown in FIG. 3 differs depending on the type of lens, and is obtained by measurement in advance as information specific to the lens. In addition, the relationship between the focal position and the angle of view can also be obtained from the relationship between the focal position and the image magnification as shown in FIG. 3. Note that the image magnification here refers to the ratio between the size of the subject image formed on the image sensor and the actual size of the subject.

[0030] <Explanation of Focus Stacking> One application of focus bracketing is depth stacking.

[0031] FIG. 4 is a diagram for explaining imaging of depth stacking in this embodiment. Here, subjects 41 to 43 are assumed to be in focus. The subjects 41 to 43 are assumed to be at different distances (subject distances) from each other, and are located in the order of subjects 41, 42, and 43 from the side closer to the digital camera 100 (in the direction from the close distance side to the long distance side). In order to obtain a depth stacking image in which all of the subjects 41 to 43 are in focus, it is necessary to cover the focal range 400 (bracket range) in which focus bracketing imaging is performed with multiple focal depths. The depths of field 411 to 416 indicate the focal depths in each imaging, and are arranged to cover the focal range 300. That is, by performing imaging (six imaging) at focus positions with the depths of field 411 to 416, the subjects 41 to 43 within the focal range 400 are in focus in any of the images. Furthermore, by combining the regions within the focal depth in each of the multiple images captured in this manner, an image that is in focus over the entire focal range 400 (overall bracket range) can be obtained.

[0032] Depth stacking technology is also useful for obtaining high-resolution images. In focus bracketing, if the depth of field is shallow when capturing each image, an image area with extremely high resolution can be obtained in the in-focus area. If each in-focus area is composited, the composite image can maintain a high resolution.

[0033] Next, a flow of generating a focus stacking image in this embodiment will be described.

[0034] 5 is a flowchart for explaining generation of a composite image in this embodiment. In step S501, the imaging unit 104 captures a plurality of images having different focus positions in the optical axis direction. In step S502, the control unit 101 performs alignment on the plurality of images captured by the imaging unit 104 in step S201. In step S503, the image processing unit 107 performs compositing on the images after alignment to generate a composite image with a deeper depth of field.

[0035] Each step shown in FIG. 5 will now be described in detail.

[0036] FIG. 6 is a flowchart for explaining the imaging in step S501 in this embodiment.

[0037] In step S601, the control unit 101 performs settings for imaging. For example, the user specifies a position to be focused on using a touch panel or the like, and the control unit 101 specifies other focus positions in sequence at a predetermined focus interval for a predetermined number of images, with the position specified by the user as the closest focus position. Alternatively, the control unit 101 may specify the position specified by the user as the closest focus position to infinity.

[0038] Alternatively, the control unit 101 may specify the initial focus position through the optical system 103 by using automatic focusing.

[0039] Alternatively, the user may specify two locations using a touch panel or the like, and the control unit 101 may determine the locations specified by the user as the focus positions on the closest to infinity side and closest to the nearest side.

[0040] In step S602, the image capturing unit 104 captures an image at the focus position that has the earliest image capturing order among the focus positions set in step S301 that have not yet been captured.

[0041] In step S603, the control unit 101 determines whether or not imaging has been performed at all focus positions set in step S301. If imaging has been performed at all focus positions, the process in the flowchart shown in Fig. 6 is terminated, and if there is a focus position at which imaging has not yet been performed, the process returns to step S602.

[0042] Furthermore, in the case of a multi-lens camera having a plurality of image capturing units 104, images may be captured simultaneously at the plurality of focus positions set in step S601.

[0043] FIG. 7 is a flowchart for explaining alignment in focus stacking in this embodiment.

[0044] In step S701, the control unit 101 acquires a reference image for alignment from among the images captured by the imaging unit 104 in step S501. The reference image for alignment is, for example, the image captured first. Alternatively, since capturing images while changing the focus position causes a slight change in the angle of view between captured images, the image with the narrowest angle of view may be used.

[0045] In step S702, the control unit 101 acquires a target image for alignment processing. The target image is an image other than the reference image acquired in step S701, and is not yet subjected to alignment processing. If the control unit 101 uses the image captured earliest as the reference image, it is sufficient to acquire the target images sequentially in the order in which they were captured.

[0046] In step S703, the control unit 101 calculates the amount of positional deviation between the reference image and the target image. An example of the calculation method is described below. First, the control unit 101 sets a plurality of blocks in the reference image. It is preferable that the control unit 101 sets each block to have the same size. Next, the control unit 101 sets a search range in the target image at the same position as each block in the reference image, which is wider than the block in the reference image. Finally, the control unit 101 calculates a corresponding point in each search range of the target image that minimizes the sum of absolute difference (Sum of Absolute Difference, hereinafter referred to as SAD) of brightness with the block in the reference image. The control unit 101 calculates the positional deviation referred to in step S703 as a vector from the center of the block in the reference image and the corresponding point described above. In calculating the corresponding point described above, the control unit 101 may use the sum of squared difference (SSD, hereinafter referred to as SSD) or normalized cross correlation (NCC, hereinafter referred to as NCC) in addition to SAD.

[0047] In step S704, the control unit 101 calculates a conversion coefficient from the amount of positional deviation between the reference image and the target image. The control unit 101 uses, for example, a projection transformation coefficient as the conversion coefficient. However, the conversion coefficient is not limited to the projection transformation coefficient, and an affine transformation coefficient or a simplified conversion coefficient including only a horizontal and vertical shift may be used.

[0048] In step S705, the image processing unit 107 performs conversion on the target image using the conversion coefficients calculated in step S704.

[0049] For example, the control unit 101 can perform the transformation using the equation shown in the following (Equation 1).

[0050]

number

[0051] In step S706, the control unit 101 determines whether or not the alignment has been performed for all images other than the reference image. If the alignment has been performed for all images other than the reference image, the process in the flowchart shown in Fig. 7 is terminated, and if there are images that have not yet been processed, the process returns to step S702.

[0052] In addition, when aligning multiple images captured by the above-mentioned multi-lens camera, the amount of parallax caused by differences in the position of the optical system 103 can be found by calculating the amount of shift in step S703, so alignment can be performed using a similar process.

[0053] FIG. 8 is a flowchart for explaining image composition in focus stacking according to this embodiment.

[0054] In step S801, the image processing unit 107 calculates a contrast value for each image (including the reference image) after alignment. As an example of a method for calculating the contrast value, the image processing unit 107 first calculates a luminance Y from the color signals Sr, Sg, and Sb of each pixel using the following (Equation 2). Y=0.299Sr+0.587Sg+0.114Sb (Formula 2)

[0055] Next, a contrast value I is calculated for a matrix L of luminance Y of 3×3 pixels using a Sobel filter as shown in the following (Equation 3) to (Equation 5).

[0056]

number

[0057]

number

[0058]

number

[0059] Furthermore, the above-mentioned method of calculating the contrast value is merely an example, and for example, it is also possible to use an edge detection filter such as a Laplacian filter or a bandpass filter that passes through a predetermined band.

[0060] In step S802, the image processing unit 107 generates a composite map. As a method of generating the composite map, the image processing unit 107 compares the contrast values ​​of pixels at the same position in each image, and calculates a composite ratio according to the magnitude of the contrast value. Specifically, among the images at the same position, a composite ratio of 100% is assigned to the pixel with the largest contrast value, and a composite ratio of 0% is assigned to other pixels at the same position. In other words, the following (Equation 6) is established.

[0061]

number

[0062] However, in step S802, it is necessary to adjust the synthesis ratio appropriately so that the boundary does not become unnatural. As a result, the synthesis ratio of the synthesis map for one image is not binarized between 0% and 100%, but changes continuously.

[0063] In step S803, the image processing unit 107 performs a synthesis process on the images after the alignment using the synthesis map generated in step S802, and generates a synthetic image.

[0064] <Angle of view of composite image for depth stacking> In the above-mentioned depth stacking, the composite image obtained in step S803 may have a narrower angle of view than the image obtained by the imaging in step S602 due to the above-mentioned focus breathing. The multiple images captured by the imaging unit 104 in step S602 may have a different angle of view because the focus position has changed. In other words, the multiple images captured by the imaging unit 104 in step S602 do not have the same angle of view. When the image processing unit 107 generates a composite image using the multiple images captured by the imaging unit 104 in step S602, it must determine the angle of view of the composite image according to the angle of view of the image with the narrowest angle of view. Therefore, the angle of view of the composite image is the same as the angle of view of the image with the narrowest angle of view among the multiple images captured by the imaging unit 104 in step S602. In other words, the angles of view of the images other than the image with the narrowest angle of view are different from the angle of view of the composite image.

[0065] As described above, the difference in the angle of view between the captured image and the composite image may cause the user to feel uncomfortable. For example, when the user captures images using focus bracketing, if the image is captured in a direction that narrows the angle of view, the angle of view that the user first sees on the display unit 108 will differ from the angle of view of the image finally obtained by composition.

[0066] Therefore, it may be possible to correct the angle of view of multiple images obtained by focus bracketing to match the angle of view of the image with the narrowest angle of view, but if depth stacking processing is not performed, it is not necessary to correct the angle of view.

[0067] Therefore, in this embodiment, the control unit 101 determines whether to correct the angle of view of images obtained by imaging with focus bracketing, depending on whether focus stacking processing is to be performed.

[0068] FIG. 9 is a flowchart for explaining the correction of the angle of view in this embodiment.

[0069] In step S901, the imaging unit 104 performs focus bracketing imaging. The process in step S901 may be considered to be the same as the flow shown in Fig. 6. The imaging unit 104 captures a plurality of images at a plurality of different determined focus positions.

[0070] In step S902, the control unit 101 determines whether the mode is focus stacking. Determining whether the mode is focus stacking here means determining whether focus stacking is active as one function of the digital camera 100. For example, if the digital camera 100 has a normal continuous shooting mode in addition to the focus stacking mode, the control unit 101 determines that the focus stacking mode has been selected instead of the normal continuous shooting mode.

[0071] If the focus stacking mode is selected, the flow proceeds to step S903, where the image processing unit 107 corrects the angle of view.

[0072] As a method of correcting the angle of view in step S903, for example, the image processing unit 107 uses the narrowest angle of view among the angles of view of a plurality of images as a reference and performs a cutout process on the other images.

[0073] FIG. 10 is a diagram for explaining the image cropping process in this embodiment. Assume that image 1001 and image 1002 are two images obtained by focus bracketing imaging. The angle of view of image 1002 is the narrowest angle of view among the images obtained by focus bracketing imaging. The image processing unit 107 uses the angle of view of image 1002 as a reference. Then, the image processing unit 107 crops out a portion 1003 of image 1001 that corresponds to the angle of view of image 1002. The image processing unit 107 performs the same process on all images other than image 1002 obtained by focus bracketing imaging. As a result, all images obtained by focus bracketing imaging have the same angle of view.

[0074] Furthermore, the image processing unit 107 may correct the angle of view using the relationship between the focal position and the angle of view, without comparing the angles of view of the images actually captured. As described above, there is an inherent relationship between the focal position and the image magnification as shown in FIG. 3. Since a certain relationship is recognized between the image magnification and the angle of view, the relationship between the angle of view and the focal position is also obtained as information inherent to the lens. The image processing unit 107 obtains the relationship between the angle of view and the focal position, which is information inherent to the lens, and can obtain the relative relationship of the angles of view between the multiple images obtained by the focus bracketing imaging using the focal position of the focus bracketing imaging. In other words, the image processing unit 107 can correct the angle of view of the image to be corrected if there is information on the focal position of the reference image and the focal position of the image to be corrected.

[0075] Fig. 11 is a diagram for explaining the display of images in this embodiment. Fig. 11(a) shows the display of the display unit 108 when generating a depth stacking image. Fig. 11(a) shows a state in which an image (captured image) used to generate a depth stacking image and a composite image are displayed simultaneously. When the image processing unit 107 performs correction to align the angles of view of the images displayed on the display unit 108, the angles of view of the images displayed on the display unit 108 are unified.

[0076] Fig. 11(b) shows the display of the display unit 108 when the focus stacking image is not displayed and only the captured image is displayed. Even in the case of Fig. 11(b), the image processing unit 107 can perform correction to align the angles of view of the images displayed on the display unit 108.

[0077] Fig. 12 is another diagram for explaining the display of images in this embodiment. Fig. 11(a) shows a captured image 1 being displayed on the display unit 108. Fig. 11(b) shows a captured image 2 being displayed on the display unit 108. Fig. 11(c) shows a captured image 15 being displayed on the display unit 108. Fig. 11(d) shows a composite image being displayed on the display unit 108.

[0078] The user operates a key or a touch panel provided on the operation unit 110 to select captured image 1 on a screen as shown in Fig. 11, and the display unit 108 transitions to a display as shown in Fig. 12(a). Similarly, the user operates the operation unit 110 to select captured image 2 on a screen as shown in Fig. 11, and the display unit 108 transitions to a display as shown in Fig. 12(b). Also, the user operates the operation unit 110 to select captured image 2 on a screen as shown in Fig. 11(a), and the display unit 108 transitions to a display as shown in Fig. 12(d).

[0079] Also, the display unit 108 may transition from a state in which one image is displayed to a state in which another image is displayed, as in FIG. 12. For example, the display unit 108 is assumed to be displaying captured image 1, as in FIG. 12(a). The user operates a key or the like provided on the operation unit 110 to switch the image so that the display unit 108 displays captured image 2, as in FIG. 12(b). The user can further operate the imaging unit 110 to sequentially display the captured images on the display unit 108. When the display unit 108 displays all the captured images, the user can further operate the operation unit 110, causing the display unit 108 to display a composite image.

[0080] It should be noted that the number of captured images shown in FIG. 11 and FIG. 12 is merely an example.

[0081] If the image processing unit 107 does not correct the angle of view of the captured images, the angle of view between the captured images appears to change while the captured images are being displayed sequentially on the display unit 108. If the image processing unit 107 corrects the angle of view of the captured images, the angle of view between the captured images appears to be the same while the captured images are being displayed sequentially on the display unit 108. From the user's perspective, the change between the images due to the focus position appears more noticeable.

[0082] Furthermore, the user may select an image to be displayed on the display unit 108 using another method. For example, assume that a plurality of captured images captured by focus bracketing are recorded in the built-in memory 109 of the digital camera 100, and that a touch panel also used as the display unit 108 of the digital camera 100 functions as the operation unit 110. FIG. 13 is a diagram for explaining an example of a display on the display unit 108 of the digital camera 100 in this embodiment. FIG. 13 shows a state in which the display unit 108 displays an arbitrary one image among the plurality of captured images captured by focus bracketing. When the user touches an area 1311 of the touch panel also used as the display unit 108, the display unit 108 displays the image in which the area 1311 is most in focus among the plurality of captured images captured by focus bracketing. In this way, the digital camera 100 can quickly display an image that focuses on an area specified by the user.

[0083] The above image display switching method can be realized by recording in advance information on the focus position of the image together with the image in the built-in memory 109. Alternatively, it can be realized by the following method.

[0084] FIG. 14 is a diagram for explaining an example of image switching in this embodiment. An image 1401 shown in FIG. 14 is an arbitrary image among a plurality of captured images captured by focus bracketing, similar to the image 1301 shown in FIG. 13. The image 1401 is divided into a plurality of blocks for processing, which will be described later. The division of the blocks as shown in FIG. 14 is drawn for convenience of explanation, and the actual division is much finer than that shown in FIG. 14. When the user touches a location corresponding to the area 1311, the control unit 101 specifies a shaded block 1411 corresponding to the area 1311 touched by the user. Next, the image processing unit 107 calculates a contrast value from a plurality of blocks corresponding to the block 1411 of all captured images captured by focus bracketing. The contrast value may be calculated by the method described in (Equation 2) to (Equation 5) described above. The control unit 101 assumes that the block with the highest contrast value among the plurality of blocks corresponding to the block 1411 is in focus. Next, the display unit 108 displays the captured image having the in-focus block.

[0085] Using the above-described method, the display unit 108 can quickly display an image in which the area touched by the user is most in focus.

[0086] Furthermore, the above-described touch operation by the user on the touch panel can be realized in a digital camera 100 that is not provided with a touch panel by using buttons, keys, or the like provided on the operation unit 110 to achieve an equivalent operation. For example, as shown in Fig. 14, an implementation method is also conceivable in which the display unit 108 displays blocks on an image, the user sequentially selects blocks on which the user wants to focus using buttons, keys, or the like, and the display unit 108 displays an image corresponding to the blocks on which the user wants to focus by pressing the confirm button.

[0087] According to this embodiment, for a plurality of captured images having different focus positions, it is possible to determine whether or not to correct the angle of view when the captured images are displayed, depending on whether or not synthesis is performed.

[0088] (Other embodiments) In the above embodiment, the explanation was based on a personal digital camera, but as long as it is equipped with a synthesis function, the present invention can also be applied to mobile devices, smartphones, or network cameras connected to a server.

[0089] Also, the imaging in the above embodiment can be realized by another implementation method. For example, the focus position can be changed while capturing a video in a video format such as MP4, and each frame of the resulting video can be used equivalently to an image captured by focus bracketing. In such a method, multiple images with different focus positions can be obtained with a lower processing load.

[0090] Furthermore, in accordance with the above embodiment, the process of displaying the image after the angle of view has been corrected can be changed to a process of superimposing a frame line corresponding to the angle of view after the correction, as in part 1003 of image 1001, to obtain a similar effect.

[0091] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and run the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0092] 100 Digital Camera 101 Control section 102 Drive unit 103 Optical system 104 Imaging unit 105 ROM 106 RAM 107 Image Processing Unit 108 Display section 109 Internal Memory 110 Operation section

Claims

1. Synthesizing means for performing synthesis on a plurality of images with different focus positions in the optical axis direction, correction means for correcting the difference in the angle of view caused by the difference in the focus positions of the plurality of images, display means for displaying the image after the correction when the correction means performs the correction, and having, the correction means is when the synthesizing means performs the synthesis, the correction is performed, when the synthesizing means does not perform the synthesis, the correction is not performed, and an image processing apparatus characterized by this.

2. The correction means corrects the angle of view by cutting out the image at a predetermined angle of view, and the image processing apparatus according to claim 1.

3. The correction means corrects so that the angles of view of the plurality of images become the same, and the image processing apparatus according to claim 1.

4. The display means simultaneously displays at least some of the plurality of images, and the image processing apparatus according to claim 1.

5. The display means sequentially displays at least some of the plurality of images, and the image processing apparatus according to claim 1.

6. The display means sequentially displays at least some of the plurality of images according to a user operation, and the image processing apparatus according to claim 5.

7. The display means simultaneously displays at least some of the plurality of images, and then displays one of the plurality of images, and the image processing apparatus according to claim 4.

8. The display means is characterized in that, after simultaneously displaying at least some of the plurality of images, it displays one of the plurality of images according to a user operation, in the image processing apparatus according to claim 7.

9. The display means is characterized in that, after displaying a first image among the plurality of images, it displays a second image different from the first image among the plurality of images, in the image processing apparatus according to claim 1.

10. The display means is characterized in that, after displaying the first image, it displays the second image according to a user operation, in the image processing apparatus according to claim 9.

11. After the user designates a first area of the first image being displayed by the display means, the display means displays the second image, and a second area corresponding to the first area of the second image is in focus, in the image processing apparatus according to claim 10.

12. The display means includes a touch panel, and the user designates the first area using the touch panel, in the image processing apparatus according to claim 11.

13. The plurality of images are frames of a video, in the image processing apparatus according to claim 1.

14. The video is an MP4 format video, in the image processing apparatus according to claim 13.

15. The synthesizing means generates a synthesized image by the synthesis, and a depth of field of the synthesized image is deeper than that of any of the plurality of images, in the image processing apparatus according to any one of claims 1 to 14.

16. The synthesizing means performs the synthesis using the in-focus areas of each of the plurality of images, in the image processing apparatus according to claim 15.

17. The image processing apparatus according to claim 16, wherein the combining means identifies a focused area of each of the plurality of images using contrast values of the plurality of images.

18. An imaging means for imaging a plurality of images having different focus positions in the optical axis direction, A combining means for combining the plurality of images, A correcting means for correcting a difference in angle of view caused by a difference in focus position of the plurality of images, A display means for displaying an image after the correction when the correcting means performs the correction, and having: The correcting means, When the combining means performs the combining, performs the correction, An imaging apparatus characterized in that when the combining means does not perform the combining, the correction is not performed.

19. A combining step of combining a plurality of images having different focus positions in the optical axis direction, A correcting step of correcting a difference in angle of view caused by a difference in focus position of the plurality of images, A display step of displaying an image after the correction when the correction is performed in the correcting step, and having: When the combining is performed in the combining step, the correction is performed in the correcting step, An image processing method characterized in that when the combining is not performed in the combining step, the correction is not performed in the correcting step.

20. A program for operating an image processing apparatus on a computer, A combining step of combining a plurality of images having different focus positions in the optical axis direction, A correcting step of correcting a difference in angle of view caused by a difference in focus position of the plurality of images, When the correction is performed in the correcting step, causing the display step of displaying an image after the correction to be performed. When performing the synthesis in the synthesis step, perform the correction in the correction step, A computer program characterized in that when the synthesis is not performed in the synthesis step, the correction is not performed in the correction step.

21. A computer-readable storage medium storing the program according to claim 20.

Images