Image processing device, image processing method, and program

Abstract

To solve the problem in which noise in still areas where the number of synthesized images is reduced is worsened and image quality is sometimes impaired when reproducing subject blurring that differs from area to area by controlling the number of composite images of a plurality of images with a short exposure time.SOLUTION: An image processing device has image acquisition means for acquiring an image, area determination means for determining a plurality of subject areas in the image, and synthesizing means for synthesizing a plurality of images obtained from imaging means for capturing images under predetermined capturing conditions. The synthesizing means has different subject blur correction characteristics that correspond to the characteristics of each of the plurality of subject areas and performs a synthesis process by referring to the different subject blur correction characteristics for each of the plurality of subject areas.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program, and particularly to an image processing technique for reproducing a subject blur suitable for each region while suppressing local noise degradation in shooting where a region where motion blur is to be maintained and a region where suppression is desired are mixed.

Background Art

[0002] Conventionally, in a shooting scene including a local moving subject such as a sports scene or an everyday snapshot, by mixing a subject region that expresses suppressing motion blur and a region that leaves blur according to the motion, the sense of presence of the scene and There is an image expression that emphasizes the sense of movement of the moving subject. In order to perform such an expression, in the prior art described in Patent Document 1, information regarding a moving object is evaluated from a plurality of images continuous in time series to determine the motion blur intensity, and the number of composite images is changed for each region. There has been proposed a method of generating an image with different motion blurs.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art described in Patent Document 1, multiple images taken with a predetermined exposure time are referenced, and control is performed to reduce the number of images combined in areas where motion blur is to be reduced, and to increase the number of images combined, such as through additive averaging, in areas where motion blur is to be added. Consequently, there was a problem that the subject areas where blur was to be suppressed had relatively more noise compared to the subject areas where blur was to be added, and the quality of the final image was impaired. Therefore, the present invention aims to provide an image processing device that reproduces subject blur appropriate for each area while suppressing localized noise deterioration in shooting where areas where motion blur is to be maintained and areas where it is to be suppressed are mixed. [Means for solving the problem]

[0005] In view of the above issues, the image processing apparatus of the present invention comprises an image acquisition means for acquiring an image, a region determination means for determining a plurality of subject regions within the image, and a synthesis means for synthesizing a plurality of images obtained from an imaging means that performs imaging under predetermined shooting conditions, wherein the synthesis means has different subject blur correction characteristics corresponding to the characteristics of the plurality of subject regions, and performs synthesis processing for each of the plurality of subject regions by referring to the different subject blur correction characteristics. [Effects of the Invention]

[0006] In shooting situations where there are areas where motion blur should be maintained and areas where it should be suppressed, it is possible to reproduce appropriate subject blur for each area while suppressing localized noise degradation. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram illustrating the configuration of the digital camera in Example 1. [Figure 2] A diagram illustrating the configuration of the imaging control parameter generation unit in Example 1. [Figure 3] A flowchart illustrating the operation of the shooting control parameter generation unit in Example 1. [Figure 4]A diagram illustrating the images taken in succession in Example 1. [Figure 5] A diagram illustrating the effect of superimposing scene recognition results onto the display image in Example 1. [Figure 6] A flowchart illustrating the conventional method for calculating motion vectors. [Figure 7] A diagram illustrating the conventional method for calculating motion vectors. [Figure 8] A figure showing the image at time t in Example 1 with a motion vector superimposed. [Figure 9] A diagram illustrating the correction gain applied to the motion vector in Example 1. [Figure 10] A diagram illustrating the image captured in Example 1. [Modes for carrying out the invention]

[0008] (Example 1) A preferred embodiment of the present invention will be described in detail below with reference to the drawings. The embodiment described below is an imaging device, and an example of applying the present invention to a digital camera as an example of an imaging device will be described.

[0009] Figure 1 is a block diagram showing the functional configuration of a digital camera according to an embodiment of the present invention. The control unit 101 is, for example, a CPU, which reads the operation programs of each block in the digital camera 100 from the ROM 102, expands them in the RAM 103, and executes them to control the operation of each block in the digital camera 100. The ROM 102 is a rewritable non-volatile memory that stores the operation programs of each block in the digital camera 100, as well as parameters necessary for the operation of each block. The RAM 103 is a rewritable volatile memory that is used as a temporary storage area for data output during the operation of each block in the digital camera 100.

[0010] The optical system 104 forms an image of the subject on the imaging unit 105. The optical system 104 includes, for example, a fixed lens, a variable magnification lens for changing the focal length, and a focus lens for adjusting the focus. The optical system 104 also includes an aperture, which adjusts the aperture diameter of the optical system to adjust the amount of light during shooting. The imaging unit 105 is an image sensor such as a CCD or CMOS sensor, which performs photoelectric conversion of the optical image formed on the image sensor by the optical system 104 and outputs the resulting analog image signal to the A / D conversion unit 106. The A / D conversion unit 106 applies A / D conversion processing to the input analog image signal and outputs the resulting digital image data to the RAM 103 for storage, thereby acquiring image data (image acquisition).

[0011] The image processing unit 107 applies various image processing techniques, such as white balance adjustment, color interpolation, and gamma processing, to the image data stored in the RAM 103 and outputs the resulting image data back to the RAM 103. The image processing unit 107 also includes an image synthesis processing unit 200, which will be described later. This unit recognizes the scene in the image data stored in the RAM 103 and generates shooting parameters for the digital camera 100 based on motion analysis results using the image data and estimation results of the subject's movement direction. The shooting control parameters generated by the image processing unit 107 are output to the control unit 101, which then controls the operation of each block in the digital camera 100.

[0012] The recording medium 108 is a removable memory card or the like, and images processed by the image processing unit 107 stored in the RAM 103, or images converted A / D by the A / D conversion unit 106, are recorded as recorded images. The display unit 109 is a display device such as an LCD, and can function as an electronic viewfinder by displaying the subject image captured by the imaging unit 105. It also displays various information in the digital camera 100, such as playing back images recorded on the recording medium 108. Furthermore, it can display icons based on the scene recognition results of the image data from the image processing unit 107 superimposed on the image.

[0013] The operation input unit 110 includes user input interfaces such as a release switch, setting buttons, a mode setting dial, etc. When detecting an operation input made by the user, it outputs a control signal corresponding to the operation input to the control unit 101. Further, in an aspect where the display unit 109 is provided with a touch panel sensor, the operation input unit 110 also functions as an interface for detecting a touch operation made on the display unit 109.

[0014] As described above, the configuration and basic operations of the digital camera 100 have been explained.

[0015] Next, the operation of the image processing unit 107, which is a feature of Embodiment 1 of the present invention, will be described in detail. In Embodiment 1 of the present invention, the shooting of a subject performing a golf swing will be taken as an example for explanation. Here, a process for obtaining an image in which different blurs are controlled for each region so that there is no motion blur in the face region but motion blur appears in the swinging region, and local noise does not deteriorate, will be described.

[0016] First, a configuration example of the image composition processing unit 200 included in the image processing unit 107 will be described with reference to FIG. 2. FIG. 2 is a diagram showing a configuration example of the image composition processing unit 200.

[0017] The image composition processing unit 2 is composed of a subject region determination unit 201, a feature extraction unit 202, a shooting parameter generation unit 203, a region-specific composition characteristic determination unit 204, a registration processing unit 205, and a composition processing unit 206. The image composition processing unit 200 inputs the image data 207 and 208 captured by the imaging unit 105 and recorded in the RAM 103, and outputs an image composition result 209.

[0018] Furthermore, image data 207 is an image captured during live view operation, which allows the user to determine the shooting timing while recognizing the subject on the display unit during preparation for shooting, while image data 208 is an image captured during actual shooting after the user has given a shooting command. The imaging control of the imaging unit when capturing image data 207 and 208 may be the same. Alternatively, the imaging control may be changed when capturing 208 based on shooting parameters including aperture, shutter speed, and ISO sensitivity, which are determined by the shooting parameter generation unit 203 during live view operation.

[0019] Next, the processing of the image synthesis processing unit 200 will be explained using the flowchart in Figure 3.

[0020] In S301, the user turns on the digital camera 100 and begins preparatory shooting, such as composing the shot. The control unit 101 continuously captures images while maintaining a predetermined frame rate during preparatory shooting. The captured images are displayed on the display unit 109, and the user adjusts the composition while viewing the displayed images. In this embodiment, the frame rate is 120 frames per second. In other words, the imaging unit 105 captures one image every 1 / 120th of a second. The shutter speed at this time is set to be as short as possible. An example of continuously captured images is shown in Figure 4. The image at time t is denoted as 401, and the image at time t+1 is denoted as 402. Figure 4 also shows an attempt to photograph a person swinging a golf club. The face and center of the person subject 403 have little movement, and although the area 404 at the tip of the swinging golf club has a lot of movement, the image does not have much blur because it is captured with a short exposure time.

[0021] In S302, the subject area determination processing unit 201, under the control of the control unit 101, refers to the image data 207 captured (acquired) in S301 and determines a representative subject area, which will be used as a reference for determining the final motion blur and noise reproduction characteristics of the output image. For example, in the golf scene of this embodiment, as shown in Figure 5, the area of ​​the face of the person playing golf is set as the first representative area 503, and the area near the tip of the swinging golf club is set as the second subject area 504.

[0022] One method for determining the subject area is for the user to touch a predetermined area while viewing the captured image displayed on the display unit. Specifically, when the user touches a part of the image on the display unit during the preparation for shooting, rectangular frames 503 and 504 of a predetermined size, including the touched area, are superimposed on the image, as shown in Figure 5. Then, known image recognition processing, such as machine learning, is applied to the image within that area to determine the attributes of the set subject area. Alternatively, multiple subject areas may be determined by setting subjects such as people or subjects with characteristics of specific sports such as golf as detection targets, for example, as disclosed in Patent Document 2.

[0023] In either case, the attributes of the region information, such as a person region or a golf club, will be stored in RAM103 as information for multiple representative subject regions.

[0024] Next, in S303, the feature extraction unit 202, under the control of the control unit 101, refers to the image data 207 and calculates image feature quantities to determine the composite characteristics for each region from the image features within the subject region selected in S303. In this embodiment, the objective is to reproduce the sense of dynamism in the moving subject region while reproducing the noise in the stationary subject region at a good level in the final output image by controlling the composite characteristics based on the amount of motion and noise in the subject region. Therefore, as shown in Figure 2, the feature extraction unit 202 has a motion vector calculation unit 2022 and a noise amount calculation unit 2021 inside, and calculates motion vector information and noise amount as image feature quantities for the selected region, respectively.

[0025] First, the motion vector calculation process in the motion vector calculation unit 2022 will be explained in detail with reference to Figures 6, 7, and 8.

[0026] A motion vector is a vector representation of the horizontal and vertical movement of the subject area between 207 images captured in succession. Figure 6 is a flowchart showing the motion vector and motion vector reliability calculation process by the motion vector calculation unit 2022. Figure 7 is a diagram showing the method for calculating the motion vector using the block matching method. In this embodiment, the block matching method is used as an example to explain the motion vector calculation method, but the motion vector calculation method is not limited to this example, and for example, the gradient method may also be used.

[0027] In step S601 of Figure 6, the motion vector calculation unit 2022 receives two temporally adjacent captured images as input. In this embodiment, the motion vector calculation unit 2022 sets the captured image at time t in Figure 4 as the reference frame and the captured image at time t+1 as the reference frame.

[0028] In step S602 of Figure 6, the motion vector calculation unit 204 places an N×N pixel reference block 702 in the reference frame 701, as shown in Figure 7.

[0029] In this embodiment, by setting the area where the reference block 702 is placed to the representative subject area set by the subject area determination processing unit 202 in S302, it is possible to efficiently analyze only the motion information necessary to generate the shooting parameters. In particular, the correlation calculation in S604, which will be described later, is a computationally intensive operation, so by performing the calculation only in the necessary area, it is possible to generate the shooting parameters at a faster speed.

[0030] In step S603 in Figure 6, the motion vector calculation unit 2022 sets the search range 705 for the reference frame 703 as follows, by setting the area around the center coordinates of the reference block 702 and the same coordinates 704 of the reference frame 701 to (N+n) pixels. The setting of the search range 705 is also limited to the area around the representative subject area set by the subject area determination processing unit 202, just like in step S602.

[0031] In S604 of Figure 6, the motion vector calculation unit 2022 performs a correlation calculation between the reference block 702 of the reference frame 701 and the reference block 706 of N×N pixels at different coordinates within the search range 705 of the reference frame 703, and calculates a correlation value. The correlation value is calculated based on the sum of absolute values ​​of inter-frame differences for the pixels of the reference block 702 and the reference block 706. In other words, the coordinate with the smallest value of the sum of absolute values ​​of inter-frame differences is the coordinate with the highest correlation value. Note that the method for calculating the correlation value is not limited to calculating the sum of absolute values ​​of inter-frame differences; for example, the correlation value may be calculated based on the sum of squares of inter-frame differences or the normal cross-correlation value. In the example in Figure 7, it is assumed that the reference block 706 has the highest correlation.

[0032] In S605 of Figure 6, the motion vector calculation unit 2022 calculates a motion vector based on the reference block coordinates showing the highest correlation value obtained in S604, and the correlation value of that motion vector is taken as the motion vector reliability. In the example in Figure 7, within the search range 705 of the reference frame 703, the motion vector is determined based on the same coordinate 704 corresponding to the center coordinate of the reference block 702 of the reference frame 701 and the center coordinate of the reference block 706. In other words, the distance and direction between the same coordinate 704 and the center coordinate of the reference block 706 are determined as the motion vector. Furthermore, the correlation value, which is the result of the correlation calculation with the reference block 706 during the calculation of the motion vector, is determined as the motion vector reliability. Note that the motion vector reliability increases as the correlation value between the reference block and the reference block increases.

[0033] In S606 of Figure 6, the motion vector calculation unit 2022 determines whether or not it has calculated motion vectors in the target locations where the reference block 702 should be placed in the reference frame 701, that is, in this embodiment, the representative subject area set by the subject area determination processing unit 202. If the motion vector calculation unit 204 determines that motion vectors have been calculated for all target locations, it terminates the motion vector calculation process. On the other hand, if it determines that motion vectors have not been calculated for the target locations, it returns to S602 and repeats the subsequent processing. In this embodiment, the reference block 702 calculates motion vectors and motion vector confidence for each pixel in all pixels included in the face area and the golf club area, and based on the motion vector confidence, it averages the motion vectors with high confidence to obtain a motion vector that represents each area.

[0034] Figure 8(a) shows the motion vectors between captured images calculated based on the above processing. In Figure 8(a), the arrows indicate motion vectors, the length of the arrows indicates the magnitude of the motion vector, and the direction of the arrows indicates the direction of the motion vector. 801 is the motion vector of the face region of the human subject 403, and 802 is the motion vector of the golf club region. It can be seen that the amount of movement between frames is greater than that of the face region because the golf club is being swung at high speed.

[0035] Next, we will explain the noise level calculation process in the noise level calculation unit 2021.

[0036] In the noise amount calculation unit 2021, in the image 207, the pixel values ​​within the representative subject area set by the subject area determination unit 202 in S302 are referenced, and the variance of the pixel values ​​within the area is calculated as the noise amount. Specifically, the noise amounts N(A1) and N(A2) corresponding to the first representative subject area 503 and the second representative subject area 504 are calculated using equations (1) and (2), respectively. N(A1)=1 / n1 Σ(xi1-x_ave1)^2···(1) N(A2)=1 / n2 Σ(xi2-x_ave2)^2···(2)

[0037] Here, n1 and n2 are the number of pixels in each representative subject area, xi1 and xi2 are the pixel values ​​in each representative subject area, and x1_ave and x2_ave are the average values ​​of the pixels in each representative subject area.

[0038] The motion vector information and noise level for each representative subject area calculated in S303 are used in subsequent processing to determine the number of images to be composited.

[0039] Next, in S304, based on the image features within the representative subject area, the region-specific synthesis characteristic determination unit 204, under the control of the control unit 101, performs a process to estimate a reference region for determining the characteristics of the synthesis process during actual shooting. First, from the attributes of the representative subject area determined in S302 and the motion vector amount and direction of the representative subject area calculated in S303, it estimates subject areas in the vicinity of the representative subject area where a motion amount comparable to that of the representative subject area occurs during actual shooting. This process may apply recognition processing such as known machine learning. Alternatively, based on the attributes of the representative subject area, the corresponding points of the motion vector may be extended so that the movement of the representative subject area occurs continuously in the direction of the motion vector for a predetermined period, thereby estimating subject areas where a predetermined amount of motion occurs. Figure 8(b) shows an example of the estimated regions. The first reference region 803 corresponds to the first representative subject area 503 and is a subject area where a motion amount comparable to that of the face of a human subject is estimated to occur. The second reference region 804 corresponds to the second representative subject region 503 and is the subject region where the amount of motion generated when a golf club is swung at high speed is estimated to occur. These estimated regions are later referenced as reference regions when determining the characteristics of the different synthesis processes for each region. In addition, for each reference region, the region name of the individual reference region and the information of the pixel position of the representative point within the reference region are associated and stored in the RAM 103. As an example of a representative point within a reference region, the motion vector of each pixel position within the reference region is referenced, and the region with the largest motion or the region with the smallest amount of motion within the reference region is set as the representative point. Figure 8(c) shows an example of a reference region and a representative point. In the scene of Figure 8(b), the representative point of the first reference region 803 is A1, and its coordinates are A1(x1,y1). Also, in the scene of Figure 8(b), the representative point of the second reference region 804 is A2, and its coordinates are A2(x2,y2).

[0040] Next, in S305, based on the image features within the representative subject area analyzed in S303, the shooting parameter generation unit 203, under the control of the control unit 101, determines the shooting parameters to be applied to this shooting (determination of shooting conditions). Specifically, the shutter speed, aperture value, ISO sensitivity are set, as well as the total exposure time to obtain the final output image and the number of images to be composited. Here, the shutter speed is set to a high speed to reduce blur in the face area, which is the first representative subject area. The aperture value is set to an aperture value that brings both the first representative subject area and the area of ​​the swinging golf club, which is the second representative subject area, within the depth of field. As for the ISO sensitivity, under the conditions of the shutter speed and aperture value mentioned above, it is set so that the average brightness level within the screen, including the first and second representative subject areas, reaches a predetermined level. Furthermore, the total exposure time required to obtain the final output image is determined by referring to the motion vector information obtained in S303 and the amount of noise, based on an exposure time that expresses the motion blur of the second representative subject area to the desired amount of motion blur, while suppressing the amount of noise in the first representative subject area, which is to be reproduced with less blur. In addition, the value obtained by dividing the total exposure time by the shutter speed becomes the number of images taken to be used for synthesis.

[0041] Next, in S306, based on the shooting parameters determined in S305, the imaging unit 105 performs the main shooting to generate the final output image, and captures the image 208 to be used for the main shooting.

[0042] For example, ten consecutive images are taken with a shutter speed of 1 / 500 second, an aperture value of F8, and an ISO sensitivity of ISO 12800. Furthermore, the imaging unit 105 is driven and controlled to minimize the time gap between consecutive image captures.

[0043] Figure 9 shows the order in which each of the 208 images captured in sequence was taken. In the subsequent alignment process, the first image 1501 is used as the alignment reference, but based on the amount of movement of the representative subject area, any image other than the first one may be set as the alignment reference.

[0044] Next, in S307, the attribute information of the reference region determined in S304 is referenced, and the composite characteristics are calculated for each pixel position under the control of the control unit 101.

[0045] First, the 208 images captured in S306 are sequentially input, and for each pixel position in the image, it is determined, based on the inter-pixel distance, which of the multiple reference regions the pixel corresponds to. Here, as shown in Figure 8(c), the pixel position of the representative point of the first reference region is A1(X1,Y1), the pixel position of the representative point of the second reference region is A2(X2,Y2), and the pixel position of the pixel of interest P in the input image is P(Xp,Yp). At this time, the inter-pixel distance D1 between pixel P and the first reference region, and the inter-pixel distance D2 between pixel P and the second reference region are calculated using equations (3) and (4). D1 = |Xp - X1| + |Yp - Y1| ... (3) D2 = |Xp - X1| + |Yp - Y1| ... (4)

[0046] Next, referencing the inter-pixel distances D1 and D2 and a predetermined threshold THmin, a composite index M for controlling the composite characteristics of the pixel of interest is calculated using equations (5) to (8). M=0 (D1 <D2 かつ D1<THmin)···(5) M=1023 (D2 <D1 かつ D2<THmin)···(6) M = D1 / (D1 + D2) (D1 <D2 かつ D1≧THmin)···(7) M = 1023 - D2 / (D1 + D2) (D2 <D1 かつ D2≧THmin)···(8)

[0047] For example, if the pixel of interest is located in the first reference region 803, the composite index M=0 is set according to equation (5). On the other hand, if the pixel of interest P is located in the second reference region 804, the composite index M=1023 is set according to equation (6). Furthermore, as shown in Figure 8(c), if the pixel of interest P is located in an intermediate pixel region not included in either the first reference region 803 or the second reference region 804, the composite index M is set to a value between 1 and 1023 according to equation (7) or equation (8).

[0048] Based on the pixel-by-pixel synthesis index M calculated as described above, the subsequent synthesis process will be controlled such that pixels closer to the representative point of the first reference region 803 will have a greater application of the synthesis process based on the first reference region 803, and pixels closer to the representative point of the second reference region 804 will have a greater application of the synthesis process based on the second reference region 804.

[0049] Here, we will explain the purpose of changing the characteristics of the synthesis process for each domain.

[0050] When shooting in dimly lit environments, setting a fast shutter speed to keep the main subject area within the depth of field while preventing blurring of the face may result in a high ISO sensitivity. For example, this could occur with a shutter speed of 1 / 500 second, an aperture of F8, and an ISO sensitivity of ISO 12800. To generate an image that reproduces motion blur in the moving areas and reproduces the still areas with as little noise as possible using multiple images taken under such conditions, multiple images must be combined even in the still areas; otherwise, the noise level in the still areas will be relatively worse.

[0051] For example, if the desired motion blur is achieved at 1 / 50 second, then 10 images taken at 1 / 500 second will be averaged, and in the motion region, the noise level of the combined (averaged) image will be attenuated to 1 / √10. Therefore, if the same degree of averaging is not performed in the subject region where motion blur is to be suppressed to reduce the noise level, a noise step will occur at the boundary between the motion region and the still region, degrading the image quality. Furthermore, in the still region, it is necessary to correct for local positional shifts of the image due to camera shake and minute subject blur before the image is combined, but in the region where motion is to be reproduced, only camera shake should be corrected, and local subject blur should not be corrected before the image is combined. Therefore, in this invention, the image is configured to perform the image combination process with different correction characteristics for each region within the screen so that blur reproduction and noise suppression reach the desired degree.

[0052] In S308, for each pixel position, prior to synthesis, the alignment processing unit 205, under the control of the control unit 101, determines the amount of positional shift correction based on the synthesis characteristics determined in S307, and performs alignment (position correction) processing.

[0053] In S307, pixels for which the composite characteristic index M=0 to 511 is set are pixels in the vicinity of the first reference area 803. Therefore, the alignment processing unit 205 detects local image blur of the subject in addition to the camera shake component and performs alignment to reduce blur of the face, which is the first representative subject area. On the other hand, in S307, pixels for which the composite characteristic index M=512 to 1023 is set are pixels in the vicinity of the second reference area 804. Therefore, the alignment processing unit 205 performs alignment correction based only on the camera shake component, that is, on the amount of positional shift of the entire field of view of the image, in order to reproduce motion blur in the golf swing area, which is the second representative subject area. As a specific example of the alignment process, the known technology described in Patent Document 3 can be applied, but is not limited to this.

[0054] In S309, for each pixel position, the synthesis processing unit 206 performs synthesis processing under the control of the control unit 101, based on the synthesis characteristics determined in S307. For pixels for which the synthesis characteristic index M=0 to 511 was set in S307, the synthesis characteristics that reduce blur in the first representative subject area (face) and reduce noise are prioritized, and the image to which alignment was applied in S308 is referenced, and a predetermined number of averaging steps are performed. Also, based on the noise amount calculated in S303, the noise amount in the first representative subject area may be greater than the noise amount in the second representative subject area in the image before synthesis. In this case, an averaging step equivalent to that used to reproduce motion blur would result in a noise difference, so a smoothing process may be applied to the synthesized image for each pixel, referencing images of the vicinity of the pixel of interest.

[0055] In S307, pixels with a composite characteristic index M=512~1023 are prioritized for reproducing motion blur in the second representative subject area, which is the golf swing area. Therefore, in S308, images with a different alignment applied than the first representative subject area are referenced, and a predetermined number of images are averaged.

[0056] The above processing is applied to all pixel positions by referring to the continuously captured main image 208, the output results are recorded on the recording medium 108, and the series of processes is completed.

[0057] Figure 10 shows an example of the output image. Motion blur is not present in the face of the human subject. On the other hand, motion blur is present in the club area, resulting in an image with a sense of dynamism.

[0058] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0059] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0060] 100 Digital Cameras 101 Control Unit 102 ROM 103 RAM 104 Optical system 105 Imaging Unit 106 A / D Conversion Unit 107 Image Processing Unit 108 Recording media 109 Display section 110 Operation Input Section

Claims

1. image acquisition means for acquiring an image; a region determining means for determining a plurality of subject regions within the image; A synthesis unit that synthesizes a plurality of images obtained by an imaging unit that captures images under a predetermined photographing condition, the synthesis means has different object blur correction characteristics corresponding to the features of the plurality of object regions, respectively, and performs synthesis processing by referring to the different object blur correction characteristics for each of the plurality of object regions.

2. The image processing apparatus according to claim 1 , wherein the characteristics of the plurality of subject regions are determined based on at least one of an amount of motion and an amount of noise in each of the plurality of subject regions.

3. a photographing condition determining means for determining the predetermined photographing condition, 3. The image processing device according to claim 1, wherein the shooting condition determination means determines, for each of the plurality of subject regions in the image, an exposure time and a number of composite images corresponding to the different subject blur correction characteristics, based on attributes of the plurality of subject regions in the image and amounts of movement within the plurality of subject regions.

4. 4. The image processing apparatus according to claim 3, wherein the attribute of the subject area is determined by image recognition processing.

5. 5. The image processing apparatus according to claim 1, wherein the synthesizing means comprises a position correcting means for correcting local positional deviations between the plurality of images.

6. 6. The image processing apparatus according to claim 5, wherein the synthesizing means synthesizes the plurality of images after the local positional deviation has been corrected by the position correcting means.

7. 7. The image processing device according to claim 6, wherein the position correction means determines a correction amount for the local positional deviation based on at least one of image features of a first subject region and a subject region adjacent to the first subject region.

8. 8. The image processing device according to claim 1, wherein the combining means determines the number of images to be combined in the first subject region based on a difference in noise level between the first subject region and a subject region adjacent to the first subject region.

9. 9. The image processing apparatus according to claim 7, wherein the first object region is a region in which it is desired to suppress object blurring among the plurality of object regions.

10. 9. The image processing apparatus according to claim 7, wherein the first object region is a region in which object blur is desired to be reproduced, among the plurality of object regions.

11. The image processing apparatus according to claim 1 , wherein the plurality of subject regions are determined based on a feature amount of the image.

12. 12. The image processing apparatus according to claim 1, wherein the predetermined photographing conditions include at least one of a shutter speed, an aperture value, an ISO sensitivity, an exposure time, and a number of combined images.

13. a display means for displaying the image, 13. The image processing apparatus according to claim 1, wherein the plurality of subject regions are determined by a user on the display means.

14. an imaging means for capturing an object image formed through an optical system and outputting the image; An imaging device comprising the image processing device according to any one of claims 1 to 13.

15. an image acquisition step of acquiring an image; a region determining step of determining a plurality of subject regions within the image; A synthesis step of synthesizing a plurality of images obtained by an imaging means that captures images under predetermined imaging conditions, the combining step has different object blur correction characteristics corresponding to the features of the plurality of object regions, and performs a combining process by referring to the different object blur correction characteristics for each of the plurality of object regions.

16. A program for causing a computer to execute each step of the control method according to claim 15.

17. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 15.

Images