Image processing apparatus, imaging apparatus, control method of image processing apparatus, and program

Abstract

To provide an image processing device, an imaging apparatus, a control method and a program of the image processing device which can obtain a high quality image by reducing an alignment mistake of plural images.SOLUTION: An image processing device comprises: image composition means 103 which performs sequential alignment composition for performing composition after performing alignment between adjacent images and performs referential alignment composition for performing composition after performing alignment of remaining images to a reference image; and image composition control means 104 which controls the image composition means 103 to generate a composite image from plural images including the same subject imaged continuously in time-series by imaging means 102 by combining the sequential alignment composition and the referential alignment composition.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, a control method for an image processing apparatus, and a program, and more particularly to an image processing apparatus, an imaging apparatus, a control method for an image processing apparatus, and a program for performing image alignment.

Background Art

[0002] Conventionally, there has been known a technique called image synthesis anti-shake in which an image (handshake correction image) equivalent to a long-exposure image without handshake is obtained by aligning and then synthesizing a plurality of images continuously shot in a short time. As one of the methods of this image synthesis anti-shake, there is a method of aligning and synthesizing the compositional deviation between the previous image and the current image (hereinafter referred to as permutation alignment synthesis) (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when performing image synthesis anti-shake by permutation alignment synthesis as in Patent Document 1, once a misalignment occurs, the subsequent image group will be aligned with the image in which the misalignment occurred, and the influence of image degradation due to the misalignment will increase. Note that the occurrence of image degradation due to misalignment is not limited to image synthesis anti-shake, and the same applies to so-called HDR image synthesis in which a plurality of images are synthesized to generate an image with a high dynamic range. That is, the same problem occurs in the technique of aligning a plurality of images, and the problem becomes more prominent as the number of images to be aligned increases.

[0005] The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus, an imaging apparatus, a control method of the image processing apparatus, and a program that can reduce misalignment of a plurality of images and obtain high-quality images.

Means for Solving the Problems

[0006] The image processing apparatus according to claim 1 of the present invention The amount of composition deviation between each of a plurality of images continuously captured in a time series including the same subject by an imaging means sets the number of images in one group according to of and performs permutation alignment for aligning between adjacent images in the group, and the first reference alignment for aligning the remaining images with a reference image in the plurality of images Alignment means, controls the alignment means, in the second group among the plurality of images the permutation alignment other than and with respect to the reference image the reference alignment in the first group Control means for aligning, and image synthesis means for performing permutation alignment synthesis for synthesizing a plurality of images permutation-aligned by the alignment means and reference alignment synthesis for synthesizing a plurality of images reference-aligned by the alignment means, and is provided with resulting image and in the second group the reference alignment resulting image to by the reference alignment The control means controls the image synthesis means to generate a composite image by combining the permutation alignment synthesis the first group and the second group are determined according to the magnitude of the composition deviation between adjacent images in the one group and the reference alignment synthesis result and result which is characterized in that.

Effects of the Invention

[0007] According to the present invention, misalignment of a plurality of images can be reduced, and high-quality images can be obtained.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0010] (Example 1) FIG. 1 is a block diagram showing the hardware configuration of camera 101 including the image processing apparatus according to the present invention.

[0011] In FIG. 1, camera 101 includes imaging means 102, image synthesizing means 103, image synthesis control means 104, shooting condition output means 105, recording means 106, shake detection means 107, anti-shake system 108, RTC 109, operation means 110, and focus detection means 111. Further, a photographing lens 112 is detachably attached to camera 101. In this embodiment, image synthesizing means 103 and image synthesis control means 104 constitute the image processing apparatus according to the present invention. Further, camera 101 of this embodiment has a control unit (not shown) that realizes the functions of camera 101 by controlling each part of camera 101. The control unit is, for example, a CPU, and reads and executes a program stored in a ROM (not shown) into a RAM (not shown), and realizes the functions of camera 101 by controlling each part of camera 101. The ROM is, for example, a rewritable non-volatile memory, and stores programs, setting values, GUI data, etc. that can be executed by the control unit. The RAM is used to read the program executed by the control unit and store the values necessary during the execution of the program. Image synthesizing means 103 and image synthesis control means 104 in FIG. 1 are functions realized when the control unit executes a program.

[0012] Operation means 110 includes an anti-shake switch, a release button, a touch sensor, etc. for turning on the image synthesis anti-shake mode. Here, the image synthesis anti-shake mode is a mode in which a plurality of images taken in a short time are aligned and then synthesized to generate a shake correction image equivalent to a long-exposure without camera shake.

[0013] Imaging means 102 has an imaging sensor such as a CCD or a CMOS. When the release button is fully pressed with the anti-shake switch on, imaging means 102 continuously generates a preset number of images (12 images in this embodiment) in time series, and sequentially outputs the images of the above fixed number to image synthesizing means 103.

[0014] When the image synthesis means 103 acquires the preset number of images from the imaging means 102, it generates a camera shake correction image by aligning and then synthesizing the acquired images (image alignment and synthesis). Note that the image synthesis means 103 may temporarily store the images acquired from the imaging means 102 in a memory (not shown) such as a removable memory card or a built-in memory, and then align and synthesize the images read from the memory. Hereinafter, expressions such as "the images acquired from the imaging means 102" or "the images acquired by the imaging means 102" shall include not only the images output from the imaging means 102 but also the images read from the memory as described above.

[0015] The image synthesis means 103 can execute two methods as the image alignment and synthesis method. The first method is a permutation alignment and synthesis in which, in a plurality of consecutive images acquired by the imaging means 102, the composition shift between the previous image and the next image (between two consecutive images) is aligned and the two images are synthesized. The second method is a reference alignment and synthesis in which one of the plurality of consecutive images is used as a reference image, and the composition shift of each of the remaining images with respect to the reference image is aligned and synthesized with the reference image.

[0016] When the release button is half-pressed with the anti-shake switch on, the shooting condition output means 105 outputs the shooting conditions to the image synthesis control means 104. In this embodiment, the shooting condition output means 105 outputs the shooting focal length detected by the focus detection means 111 to the image synthesis control means 104 as the shooting condition. Note that since the detection of the shooting focal length itself is a known technique, its details are omitted.

[0017] When the image synthesis control means 104 acquires the shooting conditions (here, the shooting focal length) of the camera 101 from the shooting condition output means 105, it causes the image synthesis means 103 to execute at least one of the permutation alignment and synthesis and the reference alignment and synthesis based on the acquired shooting conditions.

[0018] The recording means 106 records the hand-shake corrected image generated by the image synthesizing means 103 in the memory, and temporarily records the image after position adjustment and the synthesized image, which will be described later.

[0019] The vibration detection means 107 has an inertial sensor such as an angular velocity sensor or an acceleration sensor, and detects vibrations such as hand-shake occurring in the camera 101.

[0020] The anti-vibration system 108 is an anti-vibration system that exhibits optical anti-vibration performance by cooperatively controlling the imaging means 102 and the photographing lens 112.

[0021] The RTC (Real-Time Clock) 109 is an IC having a timekeeping function and performs timekeeping.

[0022] Note that the image composition control means 104 and the image composition means 103 constituting the image processing apparatus of the present invention may be provided in an external device different from the camera 101. In such a configuration, the external device may receive an image acquired by the imaging means 102 via a recording medium such as a memory card or communication with the camera 101.

[0023] FIG. 2 is a diagram showing an example in which image alignment synthesis according to the present embodiment is applied to twelve images 21a to 21j continuously photographed in time series by the imaging means 102.

[0024] As shown in FIG. 2, when the release button is fully pressed with the anti-vibration switch on, the imaging means 102 successively captures images 21a to 21l that are the targets of image alignment and synthesis in time series and outputs them to the image synthesis means 103. The arrangement of the subject 22 in each of the images 21a to 21l is different in the vertical direction of the paper surface. This indicates that the composition of each image is different due to camera shake, that is, there is a composition deviation. In practice, there may also be a composition deviation in the horizontal direction of the paper surface due to camera shake, but in this embodiment, for the sake of easy explanation, the case where the composition deviation occurs only in the vertical direction of the paper surface as shown in FIG. 2 will be described. Also, here, for the sake of simplicity of explanation, it is assumed that the subject 22 is stationary and the shooting conditions (exposure time, aperture value, focal length, etc.) of each image are constant.

[0025] The image synthesis control means 104 instructs the image synthesis means 103 to perform permutation alignment and synthesis on the images for the specified number of sheets (here, 4 sheets) in the order in which the images are acquired from the imaging means 102.

[0026] In response to this instruction, the image synthesis means 103 performs permutation alignment and synthesis on the images 21a to 21d acquired from the imaging means 102.

[0027] Specifically, the image synthesis means 103 first detects the deviation (composition deviation) between the position of the subject 22 in the image 21a and the position of the subject 22 in the image 21b, and adjusts the position of the image 21b so as to align the composition deviation. The image 21b after this position adjustment is hereinafter referred to as image 21b'. Similarly, the deviation (composition deviation) between the position of the subject 22 in the image 21b' and the position of the subject 22 in the image 21c is detected, and the position of the image 21c is adjusted so as to align the composition deviation. The image 21c after this position adjustment is hereinafter referred to as image 21c'. Also, the deviation (composition deviation) between the position of the subject 22 in the image 21c' and the position of the subject 22 in the image 21d is detected, and the position of the image 21d is adjusted so as to align the composition deviation. The image 21d after this position adjustment is hereinafter referred to as image 21d'.

[0028] Images 21a to 21l are set to have a short time interval (shooting timing) between two images taken most recently (hereinafter referred to as adjacent images). As a result, the composition deviation between adjacent images becomes small, and the time for calculating the detection of the composition deviation by the image synthesizing means 103 can be shortened. A known method may be used for the method of detecting the composition deviation between adjacent images in the above processing. For example, a method using a motion vector obtained by comparing the positions of feature points between images, a method using the detection results of inertial sensors such as an angular velocity sensor and an acceleration sensor, a method using both the motion vector and the detection results of the inertial sensor, and the like can be mentioned.

[0029] Next, the image synthesizing means 103 synthesizes the aligned images 21a, 21b’, 21c’, 21d’ after performing brightness adjustment and trimming adjustment on the regions that do not overlap during synthesis to generate a single synthesized image 23a.

[0030] In this way, the images for the number of images specified by the image synthesis control means 104 (here, four images) are arranged and synthesized in order as one group, and the number of images belonging to one group is limited. The reason for this is that the larger the number of images belonging to one group, the greater the degree to which the quality of the obtained synthesized image is degraded when an alignment error occurs in the group because the error affects the last image of the group.

[0031] When the number of synthesized images generated by the image synthesizing means 103 has not reached the specified number (here, three images), the image synthesis control means 104 repeatedly instructs the image synthesizing means 103 to perform permutation alignment synthesis on four newly acquired images.

[0032] As a result, in the image synthesizing means 103, after images 21e to 21h are arranged and synthesized in order to generate a single synthesized image 23e, images 21i to 21l are arranged and synthesized in order to generate a single synthesized image 23i.

[0033] When the number of synthesized images generated by the image synthesis means 103 reaches a specified number, the image synthesis control means 104 instructs the image synthesis means 103 to perform a reference alignment synthesis process of aligning and synthesizing the remaining synthesized images based on the first synthesized image generated.

[0034] In response to this instruction, the image synthesis means 103 performs reference alignment synthesis on the synthesized images 23a, 23e, and 23l.

[0035] Specifically, the image synthesis means 103 first detects the deviation (composition deviation) of the position of the subject 22 in the synthesized image 23a and the positions of the subject 22 in the synthesized images 23e and 23i respectively, and adjusts the positions of the synthesized images 23e and 23i so as to align the composition deviation. The synthesized images 23e and 23i after this position adjustment are hereinafter referred to as synthesized images 23e' and 23i'.

[0036] Thereafter, the image synthesis means 103 synthesizes the aligned synthesized images 23a, 23e', and 23i after performing brightness adjustment and trimming adjustment on the non-overlapping regions during synthesis to generate a single camera shake correction image.

[0037] Note that the amount of blur between adjacent images of the synthesized images 23a, 23e, and 23i is assumed to be equal to or greater than the amount of blur between adjacent images of the images 21a to 21l. This is because the time interval between the first images (21a, 21e, 21i) of the permutation alignment synthesis of each synthesized image is longer than the time interval between adjacent images. Therefore, the synthesized images 23a, 23e, and 23i perform reference alignment synthesis instead of permutation alignment synthesis to improve the alignment synthesis accuracy.

[0038] In FIG. 2, four images each of the images 21a to 21d, the images 21e to 21h, and the images 21i to 21l are in permutation positionAlthough they were combined and synthesized, the specified number of permutation alignment syntheses is changed according to the shooting focal length, which is a shooting condition. This is because when the shooting focal length of the camera 101 is long, the amount of compositional deviation between each image becomes large and alignment errors are likely to occur. Therefore, a smaller number of permutation alignment syntheses can improve the alignment synthesis accuracy. Conversely, when the shooting focal length of the camera 101 is short, even if an alignment error occurs, the original compositional deviation is small and not noticeable. Therefore, a larger number of permutation alignment syntheses can speed up the alignment synthesis.

[0039] Figure 3 is a flowchart of the alignment synthesis process according to this embodiment. This process starts when the anti-shake switch is turned on.

[0040] First, in step S301, when the release button is half-pressed, the shooting condition output means 105 detects the shooting focal length by the focus detection means 111 and outputs it to the image synthesis control means 104.

[0041] In step S302, the image synthesis control means 104 sets the number of images (specified number) belonging to one group in the image synthesis means 103 according to the shooting focal length output from the shooting condition output means 105 in step S301. Specifically, when the shooting focal length is within a predetermined range, 4 is set as the specified number. When the shooting focal length is greater than the maximum value of the predetermined range, 3 is set as the specified number. When the shooting focal length is less than the minimum value of the predetermined range, 6 is set as the specified number. In this embodiment, the specified number set here is one of 3, 4, and 6. However, the specified number may be set in more detail according to the shooting focal length. That is, it is sufficient to satisfy the relationship that the specified number set when the shooting focal length is the first value is larger than the specified number set when the shooting focal length is the second value, which is a value larger than the first value.

[0042] In step S303, when there is a full press operation of the release button, the imaging means 102 starts shooting. As a result, images that are continuous in time series are shot by the imaging means 102 and output to the image synthesizing means 103. Note that in step S303, regardless of the specified number of images set in step S302, the imaging means 102 shoots images for the set number of images (12 images in this embodiment). Therefore, for example, when the specified number of images is set to 4 in step S302, the image synthesis control means 104 sets the number of synthesized images (the specified number) generated to generate the camera shake correction image to 3.

[0043] In step S304, the image synthesis control means 104 instructs the image synthesizing means 103 to perform permutation alignment synthesis of the images for the specified number of images. The image synthesizing means 103 performs permutation alignment synthesis in response to this instruction, and the permutation alignment synthesis of the images 21a to 21d in FIG. 2 is performed, and the synthesized image 23a is created.

[0044] In step S305, the image synthesis control means 104 repeatedly gives the instruction of step S304 to the image synthesizing means 103 until the generation of the synthesized images for the specified number of images is completed, and then proceeds to step S306. As a result, the synthesized images 23a, 23e, and 23i in FIG. 2 are generated.

[0045] In step S306, the image synthesis control means 104 instructs the image synthesizing means 103 to perform reference alignment synthesis, and ends this process. As a result, the reference alignment synthesis of the synthesized images 23a, 23e, and 23i is performed, and the camera shake correction image is generated.

[0046] By changing the number of images (specified number) to be subjected to permutation alignment synthesis by the image synthesizing means 103 according to the shooting focal length of the camera 101 in this way, the deterioration of the camera shake correction image due to misalignment is reduced.

[0047] Note that the number of images (specified number) to be synthesized by permutation alignment was set based only on the shooting focal length in this embodiment, but it may be set in consideration of other shooting conditions, for example, the following shooting conditions (a) to (e). Thereby, misalignment in image synthesis shake correction can be further reduced, and a high-quality hand shake correction image can be obtained.

[0048] (a) Blur amount When the blur amount detected by the shake detection means 107 is large, reduce the number of synthesized images by permutation alignment. composition In this case, the shooting condition output means 105 outputs the blur amount detected by the shake detection means 107 as a shooting condition. As the blur amount at this time, the blur amount detected before the shooting start process in step S303 may be used.

[0049] (b) Optical image stabilization performance When there is an image stabilization system 108 that can exhibit optical image stabilization performance through the cooperation of the camera 101 and the shooting lens 112, the optical image stabilization performance changes depending on the combination of the camera 101 and the shooting lens 112. Therefore, in the case of a combination with low anti-shake performance, the compositional deviation due to hand shake becomes large, so the number of synthesized images by permutation alignment is reduced. In this case, the shooting condition output means 105 outputs the optical image stabilization performance based on the combination of the camera 101 and the shooting lens 112 as a shooting condition. For example, when performing optical image stabilization with either the camera 101 or the shooting lens 112, the number of synthesized images by permutation alignment is reduced more than when performing optical image stabilization with both the camera 101 and the shooting lens 112.

[0050] (c) Frequency characteristics of blur Since the anti-shake system 108 mounted on the camera 101 generally has low anti-shake accuracy for low-frequency blur, when the vibration detection means 107 detects low-frequency blur, the number of composite images for alignment in series is reduced. In this case, the shooting condition output means 105 outputs the blur detected by the vibration detection means 107 as a shooting condition. For example, when the main component of the vibration detected by the vibration detection means 107 is the first frequency component, the number of composite images for alignment in series is reduced more than when the main component of the vibration detected by the vibration detection means 107 is the second frequency component which is higher in frequency than the first frequency component.

[0051] (d) Elapsed time since anti-shake start Since the anti-shake accuracy of the anti-shake system 108 mounted on the camera 101 is low at the initial stage of anti-shake start, the number of composite images for alignment in series is reduced for a predetermined time (for example, 1 second) from the anti-shake start. In this case, the shooting condition output means 105 outputs the elapsed time since the anti-shake start by the RTC 109 mounted on the camera 101 as a shooting condition. For example, when shooting starts when the elapsed time since the anti-shake start is the first time, the number of composite images for alignment in series is reduced more than when shooting starts when the elapsed time since the anti-shake start is the second time which is longer than the first time. Here, when the anti-shake switch is turned on, or when the release button is half-pressed, the anti-shake may be started.

[0052] (e) Elapsed time since shooting start While the photographer is holding the camera 101 and shooting, the blur increases as time elapses since the shooting start. This is because the photographer cannot confirm the subject through the viewfinder during shooting. Therefore, during shooting, the number of composite images for alignment in series is reduced after a predetermined time (for example, 2 seconds) has elapsed since the exposure start. In this case, the shooting condition output means 105 outputs the elapsed time since the exposure start by the RTC 109 mounted on the camera 101 as a shooting condition. For example, when the elapsed time since the exposure start is the first time, the number of composite images for alignment in series is reduced more than when the elapsed time since the exposure start is the second time which is longer than the first time.

[0053] As described above, in this embodiment, the image synthesis control means 104 causes the image synthesis means 103 to execute by combining the permutation alignment synthesis and the reference alignment synthesis according to the shooting conditions. Thereby, it is possible to reduce the influence of misalignment in image synthesis anti-shake and obtain a high-quality hand-shake correction image.

[0054] (Embodiment 2) The advantage of the permutation alignment synthesis is that the composition deviation between the two images to be aligned is small. This is because the two images to be aligned are adjacent images, and the time interval between them is extremely short. Therefore, the range for searching the composition deviation can be reduced, and the alignment synthesis can be performed at high speed. On the other hand, in the reference alignment synthesis, for an image with a large time interval from the reference image, the composition deviation becomes large. Therefore, it is necessary to set a wide range for searching the composition deviation, and it takes time until the alignment synthesis. However, in the reference alignment synthesis, since all the remaining images are aligned and synthesized with the reference image, only the image with an alignment error becomes the cause of the deterioration of the synthesized image. For this reason, compared with the case of the permutation alignment synthesis, in the case of the reference alignment synthesis, the degree of deterioration of the synthesized image when an alignment error occurs in the group is small.

[0055] Therefore, in this embodiment, when the composition deviation between the two images to be aligned is small, the configuration is such that the reference alignment synthesis is performed without performing the permutation alignment synthesis.

[0056] Note that, among the hardware configurations of this embodiment, the same configurations as those in Embodiment 1 are denoted by the same reference numerals, and redundant descriptions are omitted.

[0057] In this embodiment, it is assumed that when the release button is fully pressed with the anti-shake switch on, 10 images to be the target of image alignment synthesis are continuously output in time series from the imaging means 102.

[0058] Also, in this embodiment, the shooting condition output means 105 outputs, as shooting conditions, not only the shooting focal length detected by the focus detection means 111 as in the first embodiment, but also the amount of camera 101 shake detected by the shake detection means 107 to the image composition control means 104.

[0059] FIG. 4 is a diagram showing an example in which the image position alignment synthesis according to this embodiment is applied to 10 images 41a to 41j continuously output in time series by the imaging means 102.

[0060] In FIG. 4, a case is illustrated in which in the shooting condition output means 105, a small amount of shake is detected in the first half of shooting (the first period in which images 41a to 41f are shot), and a large amount of shake is detected in the second half of shooting (the second period in which images 41g to 41j are shot). For the sake of simplicity of explanation, here it is assumed that the shooting focal length output from the shooting condition output means 105 is within a predetermined range that is neither large nor small.

[0061] In this case, the image composition control means 104 instructs the image composition means 103 to perform reference position alignment synthesis of the images 41a to 41f shot in the first period based on the amount of shake output from the shooting condition output means 105.

[0062] In response to this instruction, the image composition means 103 aligns the composition deviations of each of the remaining images 41b to 41f with respect to the image 41a and then synthesizes them to generate a synthesized image 42a.

[0063] Also, the image composition control means 104 instructs the image composition means 103 to perform permutation position alignment synthesis of the images 41g to 41j shot in the second period based on the amount of shake output from the shooting condition output means 105.

[0064] In response to this instruction, the image synthesis means 103 adjusts the position of the image 41h so as to align the composition deviation of the image 41h with respect to the image 41g. The image 41h after this position adjustment is hereinafter referred to as the image 41h'. Similarly, the position of the image 41i is adjusted so as to align the composition deviation of the image 41i with respect to the image 41h'. The image 41i after this position adjustment is hereinafter referred to as the image 41i'. Also, the position of the image 41j is adjusted so as to align the composition deviation of the image 41j with respect to the image 41i'. The image 41j after this position adjustment is hereinafter referred to as the image 41j'.

[0065] Next, the image synthesis means 103 generates a single synthesized image 42g by synthesizing the aligned images 21g, 21h', 21i', 21j' after performing brightness adjustment and trimming adjustment on the non-overlapping areas during synthesis.

[0066] Thus, in this embodiment, the image synthesis control means 104 instructs the image synthesis means 103 to perform reference alignment synthesis on a group of images captured in a first period in which the amount of blur is less than a predetermined value. On the other hand, the image synthesis control means 104 instructs the image synthesis means 103 to perform permutation alignment synthesis on a group of images captured in a second period in which the amount of blur is greater than a predetermined value.

[0067] After that, when the generation of the synthesized images 42a, 42g by the image synthesis means is completed, the image synthesis control means 104 instructs the image synthesis means 103 to perform reference alignment synthesis on the synthesized images 42a, 42g.

[0068] In response to this instruction, the image synthesis means 103 performs reference alignment synthesis on the synthesized images 42a, 42g to generate a camera shake correction image.

[0069] With such a configuration, it is possible to reduce the degradation of the synthesized image due to misalignment in each group and shorten the time required for alignment synthesis.

[0070] FIG. 5 is a flowchart of the alignment synthesis process according to this embodiment. This process starts when the anti-shake switch is turned on.

[0071] For the steps that perform the same processing as in FIG. 3, the same reference numerals are assigned.

[0072] First, the processing of steps S301 to S303 is performed.

[0073] In step S501, the shooting condition output means 105 detects the amount of camera shake of the current camera 101 using the shake detection means 107, and outputs these detection results to the image synthesis control means 104 as shooting conditions.

[0074] In step S502, the image synthesis control means 104 calculates the product of the amount of camera shake and the shooting focal length output from the shooting condition output means 105 in step S501. When the value is smaller than a predetermined value, it proceeds to step S503; otherwise, it proceeds to step S504. Here, the reason for calculating the product of the shooting focal length and the amount of camera shake will be explained. Even if the amount of camera shake is small, when the shooting focal length is long, the compositional deviation on the imaging surface of the imaging means 102 becomes large. Conversely, even if the shooting focal length is short, when the amount of camera shake is large, the compositional deviation on the imaging surface of the imaging means 102 becomes large. Therefore, by obtaining the product of the shooting focal length and the amount of camera shake, the magnitude of the compositional deviation on the imaging surface is determined. As another method, the predetermined value may be changed according to the shooting focal length. For example, in the case of the first shooting focal length, the first predetermined value is set, and in the case of the second shooting focal length, the second predetermined value is set, and by comparing the set predetermined value with the amount of camera shake, the same effect as the method of calculating the product of the shooting focal length and the amount of camera shake can be obtained.

[0075] In step S503, the image synthesis control means 104 instructs the image synthesis means 103 to perform reference alignment synthesis, and proceeds to step S505. Thereby, the image synthesis means 103 can perform a synthesis process with less influence of alignment errors. Further, since the image synthesis control means 104 knows that there is little composition deviation between the two images to be aligned, it instructs the image synthesis means 103 to reduce the search range of the composition deviation. Thereby, the image synthesis means 103 can shorten the alignment synthesis time.

[0076] In step S504, the image synthesis control means 104 instructs the image synthesis means 103 to perform permutation alignment synthesis, and proceeds to step S505. Thereby, the image synthesis means 103 can perform alignment synthesis at high speed.

[0077] In step S505, the image synthesis control means 104 repeats the process from step S501 until the imaging means 102 finishes shooting the set number of images (in the example of FIG. 4, 10 images 41a to 41j) and the alignment synthesis of all those images is completed. Thereby, the composite images 42a and 42g in FIG. 4 are generated.

[0078] In step S306, the image synthesis control means 104 instructs the image synthesis means 103 to perform reference alignment synthesis, and ends this process. Thereby, the reference alignment synthesis of the composite images 42a and 42g is performed, and a camera shake correction image is generated.

[0079] According to this process, during shooting, while the product of the shooting focal length and the amount of blur is small, the process transitions from step S502 to step S503, and reference alignment synthesis is performed. On the other hand, during shooting, while the product of the shooting focal length and the amount of blur is large, the process transitions from step S502 to step S504, and permutation alignment synthesis is performed. By combining the reference alignment synthesis and the permutation alignment synthesis in this way, the number of image sheets for the permutation alignment synthesis can be reduced. As a result, compared to the case where permutation alignment synthesis is performed for all of the images 41a to 41j to generate a camera shake correction image, the degradation of the camera shake correction image quality due to alignment errors is reduced.

[0080] Note that after the amount of blur increases and the synthesis is changed from reference alignment synthesis to permutation alignment synthesis, if the amount of blur then becomes small, it may be set to return to reference alignment synthesis again.

[0081] Also, the remaining number of images may be considered when changing from reference alignment synthesis to permutation alignment synthesis. For example, in FIG. 4, when the amount of blur for the images 41a to 41i is small and the amount of blur for the image 41j is large, changing to permutation alignment composition for the image 41j would substantially result in reference alignment synthesis of the images 41a and 41j. Therefore, when the remaining number of images is less than a predetermined number (for example, less than 2 sheets), the synthesis method may not be changed.

[0082] FIG. 6 is a diagram showing an example in which the image alignment synthesis according to a modification of the present embodiment is applied to ten images 61a to 61j continuously output in time series during long-exposure shooting by the imaging means 102.

[0083] As shown in FIG. 6, in the case of long-exposure shooting, during the initial stage of shooting (the period during which the images 61a to 61d are shot), permutation alignment synthesis is performed, and a synthesized image 62a is generated. On the other hand, during the later stage of shooting (the period during which the images 61e to 61j are shot), reference alignment synthesis is performed, and a synthesized image 62e is generated.

[0084] Thus, the reason for performing reference alignment synthesis in the later stage of shooting in long-exposure shooting is that the amount of blur tends to be larger in the later stage of shooting than in the initial stage of shooting, and it is easier to make mistakes when aligning the composition misalignment. Therefore, for the images 61e to 61j captured in the later stage of shooting, a reference alignment synthesis is performed in which the search range for the composition misalignment is widened and the remaining images 61f to 61j are aligned with the image 61e as the reference image, and a composite image 62e is generated.

[0085] After that, when the generation of the composite images 62a and 62e by the image composite means is completed, the image composite control means 104 instructs the image composite means 103 to perform reference alignment synthesis on the composite images 62a and 62e.

[0086] In response to this instruction, the image composite means 103 performs reference alignment synthesis on the composite images 62a and 62e to generate a shake correction image.

[0087] In an example like FIG. 6, although the alignment synthesis time becomes long, accurate alignment synthesis can be performed. Therefore, it may be set according to the shooting mode or the like whether to use reference alignment synthesis or permutation alignment synthesis as the alignment synthesis method to be executed when the amount of blur is large. For example, in the high-quality mode, reference alignment synthesis may be performed when the amount of blur is large, and in other modes, permutation alignment synthesis may be performed when the amount of blur is large.

[0088] (Example 3) In Examples 1 and 2, in order to detect the composition misalignment, it is premised that a plurality of images to be aligned are captured under substantially the same conditions and the motion vectors between the images can be accurately obtained. However, if the plurality of images to be aligned include images with different shooting conditions, such as lighting that is not used when other images are captured, there is a risk that the correct motion vectors between the images cannot be obtained.

[0089] Therefore, in this embodiment, even when images with different shooting conditions are included, a stable alignment synthesis is enabled. Hereinafter, the configuration of this embodiment will be specifically described.

[0090] Note that, among the hardware configurations of this embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and redundant descriptions are omitted.

[0091] FIG. 7 is a diagram showing an example in which the image alignment synthesis according to this embodiment is applied when the 12 images 21a to 21l shown in FIG. 2 are generated by front-curtain synchronous shooting. That is, here, the image 21b shown in FIG. 2 is taken when the strobe flashes, and the other images 21a, 21c to 21l are taken when the strobe does not flash.

[0092] In known front-curtain synchronous shooting, the strobe is fired at the beginning of the exposure, but in this embodiment, the image 21a is taken in a state where the strobe is not fired immediately before the acquisition of the image 21b generated by strobe shooting. As a result, it becomes possible to obtain the motion vector trajectory 71 and correct the composition deviation of the image 21b. Hereinafter, a specific description will be given.

[0093] In this embodiment, when the image 21b is taken at the time of strobe firing at the initial stage of shooting, the images 21a and 21c are taken in a state where the strobe is not fired (the same shooting conditions) immediately before and immediately after that. Note that the image 22b is taken with an exposure time shorter than that of the other images 21a, 21c to 21l taken during the front-curtain synchronous shooting. Therefore, the composition deviation of the image 21b is shifted by approximately an intermediate amount of the composition deviations of the images 21a and 21c. Therefore, when aligning the image 21b with the image 21a in the permutation alignment synthesis, first, the composition deviation of the image 21c with respect to the image 21a is detected, and the position of the image 21c is adjusted so as to align the composition deviation, and the aligned image 21c' is generated. Next, the motion vector trajectory 71 between the image 21a and the image 21c' is calculated, and the image 21b is aligned using the calculated motion vector trajectory 71. That is, even without detecting the composition deviation of the image 21b with respect to the image 21a, an image 21b' with the composition deviation of the image 21b corrected can be obtained.

[0094] Furthermore, the image 21a captured before the stroboscopic emission image 21b may be an image before the release button is fully pressed, that is, a so-called live view image, and may not be used for synthesis. That is, when performing shooting with stroboscopic emission after fully pressing the release button and synthesizing the image at the time of stroboscopic emission and the images captured without stroboscopic emission thereafter, the live view image 21a before fully pressing the release button may be used to align the image 21b. In this way, by using the live view image for aligning the image at the time of stroboscopic emission, the time from when the release button is fully pressed until stroboscopic emission can be shortened, and the possibility of missing a shutter chance can be reduced.

[0095] FIG. 8 is a diagram showing an example in which the image alignment synthesis according to the present embodiment is applied when 12 images 21a to 21l shown in FIG. 2 are generated by rear curtain sync shooting. That is, here, the image 21k shown in FIG. 2 is captured at the time of stroboscopic emission, and the other images 21a to 21j, 21l are captured without stroboscopic emission.

[0096] In known rear curtain sync shooting, the strobe is emitted in the latter stage of exposure, but in this embodiment, the image 21l is captured in a state without stroboscopic emission immediately after the acquisition of the image 21k generated by stroboscopic shooting. Thereby, it becomes possible to obtain the motion vector locus 81 and correct the compositional deviation of the image 21k. This will be specifically described below.

[0097] In this embodiment, when taking an image 21k during the strobe light emission in the post - shooting stage, immediately before and immediately after that, images 21j and 21l are taken in a state where the strobe is not emitting light (under the same shooting conditions). Note that the image 22k is taken with an exposure time shorter than that of the other images 21a - 21j, 21l taken during the rear - curtain sync shooting. Therefore, the compositional deviation of the image 21k will deviate by approximately an intermediate amount between the compositional deviations of the images 21j and 21l. Thus, when aligning the image 21k with the image 21j in the permutation alignment synthesis, first, the compositional deviation of the image 21l with respect to the image 21j is detected, and the position of the image 21l is adjusted so as to align the compositional deviation, and a post - alignment image 21l' is generated. Next, the motion vector trajectory 81 of the images 21j and 21l' is calculated, and the image 21k is aligned using the calculated motion vector trajectory 81. That is, even without detecting the compositional deviation of the image 21k with respect to the image 21j, an image 21k' with the compositional deviation of the image 21k corrected can be obtained.

[0098] Note that the image 21l taken after the strobe - lit image 21k is used only for correcting the compositional deviation of the image 21k generated by performing the strobe shooting and may not be used for synthesis. This is because after the strobe is fired, the subject often moves, and if the subject after the movement is included in the image 21l, the quality of the shake - corrected image may deteriorate.

[0099] In this way, when there is an image with shooting conditions different from those of the other images among the images 21a - 21l in FIG. 2, the image synthesis means 103 corrects the compositional deviation of the image with different shooting conditions using the compositional deviations detected in the images taken immediately before and after it.

[0100] FIG. 9 is a diagram showing an example in which the image alignment synthesis according to Embodiment 3 is applied when the last image 21h of the second permutation alignment group shown in FIG. 2 is an image with different shooting conditions. Here, the permutation alignment group refers to a group that generates a composite image by permutation alignment.

[0101] There is a change in brightness between the image 21h taken at the timing when the fireworks went off and the image g taken immediately before that. In such a case, it is not possible to accurately obtain the compositional deviation between the images 21g and 21h. Therefore, in this embodiment, as in FIGS. 7 and 8, first, the compositional deviation of the image 21h is corrected using the images 21g and 21i taken immediately before and after the image 21h. Specifically, the compositional deviation of the image 21i with respect to the image 21g is detected, the position of the image 21i is adjusted so as to align the compositional deviation, and the image 21i' after alignment is generated. Then, the motion vector trajectory 91 of the images 21g and 21i' is calculated, and the image 21h is aligned using the calculated motion vector trajectory 91. That is, even without detecting the compositional deviation of the image 21h with respect to the image 21g, an image 21h' with the corrected compositional deviation of the image 21h can be obtained.

[0102] As shown in FIG. 9, from the image 21e to the image 21i taken immediately after the image 21h generated by performing fireworks photography are made into a permutation alignment group. However, the image 21i is used only for calculating the motion vector trajectory 91 for correcting the compositional deviation of the image 21h and is not used for generating the composite image 23e. That is, in FIGS. 2, 7, and 8, the images 21e to 21h are used as the permutation alignment group, but in FIG. 9, the images 21e to 21i are used as the permutation alignment group. However, the composite image 23e is generated using the images 21e, 21f' to 21h' in all of FIGS. 2, 7 to 9. The reason for this is that when the number of images for permutation alignment synthesis is made uniform for each group, the alignment accuracy can be made uniform. Therefore, for example, when the image taken by fireworks photography is the image 21g instead of the last image 21h of the group, the images 21e to 21h are used as the permutation alignment group, and the composite image 23e is generated using the images 21e, 21f' to 21h'.

[0103] Note that in FIG. 9, the example shows a case where an image with different shooting conditions has a different subject brightness from other images in the permutation alignment group to which it belongs, but it is not limited to this. For example, it may be the presence or absence of strobe light emission during shooting.

[0104] Figure 10 is a flowchart of the alignment synthesis process according to this embodiment. This process starts when the anti-vibration switch is turned on and is applicable to any of the patterns shown in FIGS. 7 to 9.

[0105] In addition, for steps that perform the same processing as in FIG. 3, the same reference numerals are assigned.

[0106] First, the processing of steps S301 to S303 is performed.

[0107] In step S1001, the image synthesis control means 104 instructs the image synthesis means 103 to detect the compositional deviation of the image output from the imaging means 102. In response to this instruction, the image synthesis means 103 detects the compositional deviation between the output image and the image obtained immediately before it.

[0108] In step S1002, the image synthesis control means 104 instructs the image synthesis means 103 to detect the brightness of the image output from the imaging means 102. In response to this instruction, the image synthesis means 103 detects the brightness of the output image.

[0109] In step S1003, the image synthesis control means 104 instructs the image synthesis means 103 to repeat the processing of steps S1001 and S1002 until it is performed on the specified number of images set in step S302. As a result, the detection of the compositional deviation and brightness of each image in the current permutation alignment group is completed.

[0110] In step S1004, based on the brightness detected for each image in the current permutation alignment group, the image synthesis control means 104 determines whether there is an image in the current permutation alignment group whose brightness is different from that of other images. If there is an image with a different brightness, the process proceeds to step S1005; otherwise, it proceeds to step S1007.

[0111] In step S1005, the image composition control means 104 determines whether the images determined to have different brightnesses in step S1004 are the last images in the current permutation alignment group. If they are not the last images, the process directly proceeds to step S1006. If they are the last images, the process returns to step S1001 to detect the composition deviation and brightness of the first image in the next permutation alignment group. Thereby, when the last image in the current permutation alignment group is an image with different brightnesses, a vector trajectory for correcting the composition deviation of the last image can be obtained by using the image taken immediately after the last image. Although not shown in the flowchart of FIG. 10, if there is no next permutation alignment group, the image composition control means 104 causes the imaging means 102 to take one additional image including the subject 22 and then proceeds to step S1006.

[0112] In step S1006, the image composition control means 104 calculates the composition deviation of the images with different brightnesses. For example, when the images with different brightnesses are image 21h in FIG. 9, the composition deviation between images 21g and 21i taken immediately before and after image 21h is detected, and the position of image 21i is adjusted to align the composition deviation. Next, the motion vector trajectory 91 of image 21i' after the position adjustment with respect to image 21g is calculated. Then, the composition deviation of image 21h is set as approximately the intermediate amount of the composition deviations of the adjacent images 21g and 21i, and the composition deviation of image 21h is calculated based on the motion vector trajectory 91.

[0113] In step S1007, the image composition control means 104 instructs the image composition means 103 to align and then synthesize each image in the current permutation alignment group based on the composition deviations detected and calculated in steps S1001 and S1006. By this process, for example, based on images 21a to 21d in FIG. 9, images 21b' to 21d' are generated, and images 21a, 21b' to 21d' are synthesized to obtain a synthesized image 23a. Also, based on images 21e to 21i in FIG. 9, images 21f' to 21i' are generated, and images 21e, 21f' to 21h' are synthesized to obtain a synthesized image 23e.

[0114] In step S1008, until the generation of the composite images for the specified number of sheets is completed, the image composition control means 104 repeats the processing from step S1001 and then proceeds to step S1009. As a result, the composite images 23a, 23e, 23i in FIG. 9 are generated.

[0115] In step S1009, the image composition control means 104 instructs the image composition means 103 to perform reference alignment composition and ends this processing. In response to this instruction, the image composition means 103 performs reference alignment composition of the composite images 23a, 23e, 23i, and a shake correction image is generated.

[0116] In this way, when images with different brightness levels are generated during shooting with the anti-shake switch turned on, the image composition means 103 detects the composition deviation (trajectory) between the images taken immediately before and after the images with different brightness levels by using the composition deviation (trajectory) between the images taken immediately before and after the images with different brightness levels. And when the last image in the current permutation alignment group is an image with a different brightness level, the image in the next permutation alignment group is also used to detect the composition deviation of the images with different brightness levels.

[0117] As a result, even when there are images with different brightness levels during shooting with the anti-shake switch turned on, highly accurate alignment composition becomes possible.

[0118] As described above, the method for detecting composition deviation when there are images with different brightness levels within the permutation alignment group has been described. Next, a method for detecting composition deviation when there are images with different brightness levels within a group (hereinafter referred to as a reference alignment group) that generates a composite image by reference alignment, such as the images 41a to 41f in FIG. 4 and the images 61e to 61j in FIG. 6, will be described.

[0119] As described above, in the second embodiment, the images taken during the period when the amount of blur of the camera 101 detected by the vibration detection means 107 is small (FIG. 4) or during the latter stage of shooting in long-exposure shooting (FIG. 6) are included in the reference alignment group.

[0120] FIG. 11 is a diagram showing an example in which alignment synthesis according to a modification of this embodiment is applied when there are images with different shooting conditions in the reference alignment group shown in FIG. 6. That is, the reference alignment group consists of six images 1101e to 1101j (corresponding to images 61e to 61j taken in the latter stage of long-exposure shooting in FIG. 6), and the group includes an image 1101i with different brightness.

[0121] The image synthesizing means 103 calculates a motion vector locus 1103 showing the difference in composition deviation between the image 1101h and 1101j taken immediately before and after the image 1101i with different brightness with respect to the reference image 1101e. The composition deviation of the image 1101i with different brightness is taken as the intermediate amount of the composition deviations of the images 1101h and 1101j, and the composition deviation of the image 1101i with different brightness is calculated using the motion vector locus 1103. Then, after aligning each image based on its composition deviation, they are synthesized to obtain a synthesized image 1102e.

[0122] FIG. 12 is a flowchart of the alignment synthesis process according to a modification of Embodiment 3. This process starts from the time when continuous shooting of the imaging means 102 is started by a full press operation of the release button with the anti-shake switch on.

[0123] First, in step S1201, the image synthesis control means 104 instructs the image synthesis means 103 to detect the brightness of the image output from the imaging means 102. In response to this instruction, the image synthesis means 103 detects the brightness of the output image.

[0124] In step S1202, based on the brightness detected in step S1201, the image synthesis control means 104 determines whether the image output from the imaging means 102 has a different brightness from the image output from the imaging means 102 immediately before it. As a result of this determination, if it is an image with different brightness, step S1203 is skipped and the process proceeds to step S1204; otherwise (NO in step S1202), the process proceeds to step S1203.

[0125] In step S1203, the image composition control means 104 instructs the image composition means 103 to detect the composition deviation of the output image. In response to this instruction, the image composition means 103 detects the composition deviation between the output image and the reference image.

[0126] In step S1204, the process from step S1201 is repeated until the shooting of the set number of images (6 images in the example of FIG. 11) to be shot in the latter stage of shooting is completed. That is, for the images 1101f to 1101j output from the imaging means 102 in sequence, the composition deviation with respect to the reference image 1101e is obtained one by one. However, for the image 1101i with a brightness change, the composition deviation with respect to the reference image 1101e is not detected.

[0127] In step S1205, the image composition control means 104 determines whether the image with different brightness is the last image of the reference alignment group. If it is the last image, the process proceeds to step S1206; otherwise (NO in step S1205), the process proceeds to step S1207.

[0128] In step S1206, the image composition control means 104 causes the imaging means 102 to additionally shoot one image including the subject 22, and after causing the image composition means 103 to detect the composition deviation between the additionally shot image and the reference image, the process proceeds to step S1207. Thereby, even if the image with different brightness is the last image of the reference alignment group, the composition deviation with respect to the reference image of the image shot immediately after it, which is used in step S1206 described later, can be obtained.

[0129] In step S1207, the image composition control means 104 calculates the composition deviation of the image with different brightness from the composition deviations with respect to the reference images of the images shot immediately before and after it. In the example of FIG. 11, thereby, the composition deviation of the image 1101i is calculated based on the motion vector locus 1103 indicating the difference between the composition deviations of the images 1101h and 1101j.

[0130] In step S1208, the image composition control means 104 aligns and synthesizes each image of the reference alignment group based on the composition deviation detected in step S1203 and the composition deviation calculated in step S1206, and ends this process.

[0131] In this way, when images with different brightness occur during the shooting of the reference alignment group, the composition deviation of the image with the changed brightness is detected by using the difference in the composition deviation (motion vector trajectory) with respect to the reference image of the images shot immediately before and after it. This enables highly accurate alignment and synthesis even when images with different brightness occur during the shooting of the reference alignment group.

[0132] In addition, in FIG. 12, the case where images with different brightness occur in the latter stage of the long-exposure shooting in FIG. 6 is illustrated, but the same processing can also be applied to the case where images with different brightness occur during the shooting in the period when the shake amount of the camera 101 in FIG. 4 is small.

[0133] As described above, in this embodiment, by adopting the following configuration, stable image alignment and synthesis have become possible.

[0134] 1. When there are images with different shooting conditions such as flash emission or changes in the brightness of the subject, the composition deviation is calculated by using the change in the composition deviation (alignment trajectory) of the images shot immediately before and after the image, and image alignment is performed.

[0135] 2. When acquiring images with different shooting conditions, images with the same shooting conditions are acquired immediately before and after the acquired image. Specifically, when performing front-curtain sync shooting, a non-flash image is acquired immediately before acquiring a flash image, and when performing rear-curtain sync shooting, a non-flash image is acquired immediately after acquiring a flash image.

[0136] 3. When an image with different shooting conditions is the last image of a group, the composition deviation of the last image is calculated using the first image of the next group, and image alignment is performed. However, the first image of the next group is not used in the image composition of the group including the image with different shooting conditions. Also, when there is no next group, the imaging means 102 is additionally caused to shoot one more image including the subject 22, and the composition deviation of the last image is calculated using that image.

[0137] 4. When an image with different shooting conditions is included in the reference alignment group, the difference in composition deviation between the images immediately before and after the image with different shooting conditions with respect to the reference image is calculated, and the composition deviation of the image with different shooting conditions is calculated using the calculated difference.

[0138] Furthermore, by not using the images not used in the image synthesis of the group including the image with different shooting conditions for calculating the composition deviation of the image with different shooting conditions, it becomes easier to obtain the image intended by the photographer. Specifically, in front-curtain sync shooting, the non-flash images before the flash emission image, and in rear-curtain sync shooting, the non-flash images after the flash emission image may not be used for image synthesis.

[0139] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having a computer of the system or device read and execute the program.

[0140] The computer has one or more processors or circuits, and may include a network of separate multiple computers or separate multiple processors or circuits in order to read and execute computer-executable instructions.

[0141] The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). Further, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0142] Note that although all of the above-described embodiments have described the case of synthesizing a plurality of images for the purpose of image synthesis anti-shake, the present invention is applicable not only to anti-shake but also to other techniques for aligning and synthesizing a plurality of images such as HDR image synthesis. Further, the alignment technique of the present invention can be applied to techniques that do not synthesize a plurality of images.

Explanation of Signs

[0143] 101 Camera 102 Imaging means 103 Image synthesis means 104 Image synthesis control means 105 Shooting condition output means 106 Recording means 107 Vibration detection means 108 Anti-shake system 109 RTC 110 Operation means 111 Focus detection means 112 Shooting lens

Claims

1. setting the number of images in one group according to the amount of composition deviation between each of a plurality of images continuously captured in a time series including the same subject by an imaging means, in the first group among the plurality of images, performing permutation alignment for alignment between adjacent images, and in the second group among the plurality of images, performing reference alignment for aligning the remaining images other than a reference image with the reference image; an alignment means; a control means for controlling the alignment means and aligning the image resulting from the permutation alignment in the first group and the image resulting from the reference alignment in the second group by the reference alignment; an image synthesis means for performing permutation alignment synthesis for synthesizing a plurality of images permutation-aligned by the alignment means and reference alignment synthesis for synthesizing a plurality of images reference-aligned by the alignment means; comprising the first group and the second group are determined according to the magnitude of the composition deviation between adjacent images in the one group; the control means controls the image synthesis means to combine the result of the permutation alignment synthesis and the result of the reference alignment synthesis to generate a synthesized image, an image processing apparatus characterized by this.

2. further comprising imaging condition output means for outputting first imaging conditions having a correlation with the amount of composition deviation; based on the first imaging conditions output from the imaging condition output means, the control means controls the combination of the permutation alignment synthesis and the reference alignment synthesis in the image synthesis means, the image processing apparatus according to claim 1, characterized by this.

3. when the plurality of images include a specific image with different second imaging conditions, the alignment means calculates an alignment trajectory of images captured immediately before and after the specific image, and performs alignment of the specific image using the calculated alignment trajectory, the image processing apparatus according to claim 2, characterized by this.

4. The image processing apparatus according to claim 3, wherein the images taken immediately before and after are images in which the second imaging conditions are satisfied.

5. The second imaging condition is the presence or absence of flash emission, The imaging means, When generating the plurality of images by front curtain sync shooting, immediately before the flash emission image which is the specific image, a non-flash emission image which is an image in which the second imaging conditions are satisfied is taken, and when generating the plurality of images by rear curtain sync shooting, immediately after the flash emission image which is the specific image, a non-flash emission image which is an image in which the second imaging conditions are satisfied is taken. The image processing apparatus according to claim 4, characterized by this.

6. The control means does not use the non-flash emission image taken immediately after the flash emission image in the rear curtain sync shooting for generating the composite image. The image processing apparatus according to claim 5, characterized by this.

7. The control means, causes the alignment means to align the images of the second group by the reference alignment according to the first imaging condition, In the second group where the specific image is the last image and there is no next group, the imaging means is caused to additionally take an image including the same subject after shooting the plurality of images, and use it as the image taken immediately after. The image processing apparatus according to any one of claims 3 to 6, characterized by this.

8. The control means, In the group where the specific image is the last image and there is a next group, the first image of the next group is used as the image taken immediately after. The image processing apparatus according to claim 7, characterized by this.

9. The control means, The image processing apparatus according to claim 7 or 8, wherein the immediately-taken image is not used for image composition of a group in which the specific image is the last image.

10. The control means calculates the alignment locus based on a composition deviation between the immediately-taken image and the image taken immediately after that with respect to the immediately-taken image when the specific image is included in the first group, the image processing apparatus according to any one of claims 3 to 6.

11. The control means calculates the alignment locus based on a difference in composition deviation between each of the immediately-taken image and the image taken immediately after that with respect to the reference image when the specific image is included in the second group, the image processing apparatus according to any one of claims 3 to 6.

12. The first shooting condition is a shooting focal length, the image processing apparatus according to any one of claims 2 to 11.

13. The number of images in one group is smaller when the shooting focal length is a first distance than when the shooting focal length is a second distance shorter than the first distance, the image processing apparatus according to claim 12.

14. The first shooting condition is at least any one of a blur amount, an optical image stabilization performance, a frequency characteristic of blur, an elapsed time from the start of image stabilization, and an elapsed time from the start of shooting, the image processing apparatus according to any one of claims 2 to 13.

15. The first shooting condition is a blur amount, and the number of images belonging to one group is smaller when the blur amount is a first amount than when the blur amount is a second amount smaller than the first amount, the image processing apparatus according to any one of claims 2 to 14.

16. In the case of long-exposure shooting, the control means performs the permutation alignment synthesis at the initial stage of shooting and the reference alignment synthesis at the later stage of shooting to generate a plurality of synthesized images, and then controls the alignment means to perform the reference alignment synthesis of the plurality of synthesized images. The image processing apparatus according to any one of claims 7 to 9, characterized in that.

17. The alignment means sets three or more images as one group. The image processing apparatus according to any one of claims 1 to 16, characterized in that.

18. An imaging device having an image sensor and an imaging unit that outputs an imaged image, The number of images in one group is set according to the amount of composition deviation between each of a plurality of images continuously imaged in a time series including the same subject by the imaging unit. In the first group of the plurality of images, permutation alignment for aligning between adjacent images, and in the second group of the plurality of images, alignment means for performing reference alignment for aligning the remaining images other than the reference image with the reference image; Control means for controlling the alignment means to align the image resulting from the permutation alignment in the first group and the image resulting from the reference alignment in the second group by the reference alignment; Permutation alignment synthesis for synthesizing a plurality of images permutation-aligned by the alignment means, and reference alignment synthesis for synthesizing a plurality of images reference-aligned by the alignment means; an image synthesis means for performing; Comprising The control means controls the image synthesis means to combine the result of the permutation alignment synthesis and the result of the reference alignment synthesis to generate a synthesized image. An imaging device characterized by that.

19. Set the number of images in one group according to the amount of composition deviation between each of a plurality of images continuously captured in a time series including the same subject by an imaging means. In the first group among the plurality of images, perform permutation alignment for alignment between adjacent images, and in the second group among the plurality of images, perform reference alignment for aligning the remaining images other than the reference image with the reference image. An alignment step; Control the alignment step to align the image resulting from the permutation alignment in the first group and the image resulting from the reference alignment in the second group by the reference alignment. A control step; Perform permutation alignment synthesis for synthesizing a plurality of images permutation-aligned in the alignment step, and perform reference alignment synthesis for synthesizing a plurality of images reference-aligned in the alignment step. An image synthesis step; Control the image synthesis step to combine the result of the permutation alignment synthesis and the result of the reference alignment synthesis to generate a synthesized image. A control method characterized by this.

20. A program characterized by causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 17.

Images