Image processing equipment, camera equipment and image processing methods

Abstract

This invention relates to an image processing apparatus, a camera device, and an image processing method. The image processing apparatus is capable of obtaining high-definition images by reducing alignment errors in multiple images. The image processing apparatus includes: an alignment unit for performing sequential alignment and reference alignment, the sequential alignment being used to align adjacent images, and the reference alignment being used to align images other than a reference image to a reference image; and a control unit for controlling the alignment unit to align multiple images including the same subject, captured sequentially in a time sequence by a camera unit, by combining the sequential alignment and the reference alignment.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for aligning images, a camera device including the image processing apparatus, and a control method for the image processing apparatus. Background Technology

[0002] There is a known conventional technique called image stabilization, which combines images captured in succession over a short period of time by aligning them to obtain an image equivalent to a long-exposure image without camera shake (camera shake corrected image). There are also methods for image stabilization that align the current image with the previous image by eliminating framing discrepancies between images (hereinafter referred to as sequential alignment combination) (see, for example, Japanese Patent 4418632).

[0003] However, when using the sequence alignment synthesis disclosed above for image stabilization, if an alignment error occurs, subsequent images will be aligned with the image where the alignment error occurred, amplifying the image degradation caused by the alignment error. It should be noted that image degradation due to alignment errors occurs not only in image stabilization but also in HDR image synthesis, which generates high dynamic range images by compositing multiple images. That is, a similar problem occurs in techniques that align multiple images, and this problem becomes significant as the number of images to be aligned increases. Summary of the Invention

[0004] The present invention provides an image processing device, a camera device including the image processing device, and a control method for the image processing device, which can obtain high-definition images by reducing alignment errors of multiple images.

[0005] Therefore, an aspect of the present invention provides an image processing apparatus, comprising: an alignment unit configured to perform sequential alignment and reference alignment, the sequential alignment being used to align adjacent images, and the reference alignment being used to align images other than a reference image with the reference image; and a control unit configured to control the alignment unit to align multiple images including the same subject captured sequentially by a camera unit in a time sequence by combining the sequential alignment and the reference alignment.

[0006] A camera device includes: a camera unit having an image sensor and outputting captured images; an alignment unit configured to perform sequential alignment and reference alignment, the sequential alignment aligning adjacent images and the reference alignment aligning images other than a reference image with the reference image; and a control unit configured to control the alignment unit to align images including the same subject captured sequentially by the camera unit in a time sequence by combining the sequential alignment and the reference alignment.

[0007] An image processing method is provided for aligning multiple images including the same subject, which are captured sequentially in a time sequence by a camera unit having an image sensor. The image processing method is characterized in that the multiple images are aligned by combining sequential alignment and reference alignment, wherein the sequential alignment is used to align adjacent images, and the reference alignment is used to align images other than a reference image with the reference image.

[0008] According to the present invention, alignment errors of multiple images can be reduced, and high-definition images can be obtained.

[0009] Other features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating the hardware configuration of a camera including the image processing apparatus of the present invention.

[0011] Figure 2 This illustrates that image alignment synthesis according to the first embodiment is applied using... Figure 1 An example diagram showing 12 images captured consecutively by the camera unit.

[0012] Figure 3 This is a flowchart illustrating the alignment synthesis process according to the first embodiment.

[0013] Figure 4 This is a diagram illustrating an example of image alignment synthesis according to the second embodiment being applied to 10 images that are continuously output in a time sequence by means of a camera unit.

[0014] Figure 5 This is a flowchart illustrating the alignment synthesis process according to the second embodiment.

[0015] Figure 6 This is a diagram illustrating an example of image alignment synthesis according to a variant of the second embodiment being applied to 10 images that are output sequentially in time by using a camera unit for long-term video recording.

[0016] Figure 7This illustrates that image alignment synthesis according to the third embodiment is applied to images generated by front-curtain synchronized imaging. Figure 2 The 12 images shown are examples.

[0017] Figure 8 This illustrates that image alignment synthesis according to the third embodiment is applied to images generated by a rear-curtain synchronized camera. Figure 2 The 12 images shown are examples.

[0018] Figure 9 It is shown in Figure 2 The figure shows an example of applying image alignment synthesis according to the third embodiment when the imaging conditions of the last image of the second sequence alignment group are different.

[0019] Figure 10 This is a flowchart illustrating the alignment synthesis process according to the third embodiment.

[0020] Figure 11 It is shown in Figure 6 The diagram shows an example of alignment synthesis applied according to a variant of the third embodiment in the case of images with different imaging conditions.

[0021] Figure 12 This is a flowchart illustrating the alignment synthesis process according to a variant of the third embodiment. Detailed Implementation

[0022] In the following, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This is a block diagram illustrating the hardware configuration of a camera 101, which is an imaging device including an image processing apparatus according to a first embodiment of the present invention.

[0024] like Figure 1As shown, the camera 101 includes an image capturing unit 102, an image compositing unit 103, an image compositing controller 104, an image capturing condition output unit 105, a recording unit 106, a camera shake detection unit 107, an image stabilization system 108, an RTC 109, an operation unit 110, and a focus detection unit 111. Furthermore, a camera lens 112 is detachably attached to the camera 101. In this embodiment, the image compositing unit 103 and the image compositing controller 104 constitute an image processing device according to the present invention. Furthermore, the camera 101 of this embodiment has a controller (not shown) that implements the functions of the camera 101 by controlling corresponding components of the camera 101. The controller is, for example, a CPU, which reads a program stored in a ROM (not shown) into a RAM (not shown) and executes the program to implement the functions of the camera 101 by controlling corresponding components of the camera 101. The ROM is a rewritable non-volatile memory that stores programs that the controller can run, setting values, GUI data, etc. The RAM is used to read the programs run by the controller and to store values ​​required during program execution. Figure 1 The image compositing unit 103 and the image compositing controller 104 are functions implemented when the controller runs the program.

[0025] The operation unit 110 includes a release button, a touch sensor, and an image stabilization switch for activating the image stabilization mode. In the description, the image stabilization mode is a mode that generates a camera shake-corrected image equivalent to a long-exposure image without camera shake by aligning and combining images captured consecutively over a short period of time.

[0026] The camera unit 102 has an image sensor such as a CCD or CMOS. When the release button is fully pressed while the image stabilization switch is on, the camera unit 102 continuously generates a preset number of images (12 in this embodiment) in a time sequence and outputs the preset number of images sequentially to the image synthesis unit 103.

[0027] When a preset number of images are received from the imaging unit 102, the image synthesis unit 103 generates a camera shake-corrected image by aligning and synthesizing the acquired images (image alignment synthesis). The image synthesis unit 103 may temporarily store the images obtained from the imaging unit 102 in a memory (not shown), such as a removable memory card or internal memory, and synthesize the images read from the memory after aligning them. In the following text, the images obtained from the imaging unit 102 and the images obtained by the imaging unit 102 refer not only to the images output from the imaging unit 102, but also to the images read from the memory as described above.

[0028] The image compositing unit 103 can perform two image alignment compositing methods. The first method is sequential alignment compositing, which aligns a preceding image with a subsequent image (two consecutive images) to eliminate framing discrepancies between the two images in the plurality of consecutive images acquired by the imaging unit 102, and then composites the two images. The second method is reference alignment compositing, which selects a reference image from a plurality of consecutive images, aligns each of the remaining images with the reference image to eliminate framing discrepancies relative to the reference image, and then composites these images. The advantage of sequential alignment compositing is that the framing discrepancy between the two images to be aligned is small. This is because the two images to be aligned are adjacent images, and the time interval between the shooting times of the adjacent images is extremely short. Therefore, the range for searching for framing discrepancies can be narrowed, and alignment compositing can be performed at high speed. In contrast, in reference alignment compositing, the larger the time interval between the shooting time of the reference image and the shooting time of the aligned image, the larger the framing discrepancy. Therefore, the range for searching for framing discrepancies needs to be expanded, and alignment compositing requires more time. However, since reference alignment compositing aligns and composites all remaining images with the reference image, only the image that caused the alignment error becomes the cause of degradation in the composite image. Therefore, in the case of alignment errors occurring in a group, the degradation of the synthesized image in the baseline alignment synthesis becomes less than the degradation of the synthesized image in the sequence alignment synthesis.

[0029] With the image stabilization switch on and the release button half-pressed, the camera condition output unit 105 outputs camera conditions to the image compositing controller 104. In this embodiment, the camera condition output unit 105 outputs the camera focal length as a camera condition detected by the focal length detection unit 111 to the image compositing controller 104. Since the detection of camera focal length is a known technique, details are omitted.

[0030] When the camera conditions (camera focal length in this embodiment) of the camera 101 are obtained from the camera condition output unit 105, the image synthesis controller 104 controls the image synthesis unit 103 to perform at least one of sequence alignment synthesis and reference alignment synthesis based on the obtained camera conditions.

[0031] The recording unit 106 records the camera shake correction image generated by the image synthesis unit 103 into the memory, or temporarily records the alignment image and the synthesized image mentioned below.

[0032] The camera shake detection unit 107 has inertial sensors such as angular velocity sensors and accelerometers, and detects vibrations such as camera shake that occur in the camera 101.

[0033] The image stabilization system 108 performs optical image stabilization by coordinating the camera unit 102 and the camera lens 112.

[0034] The RTC (Real-Time Clock) 109 is an IC with time checking function, which checks the time.

[0035] It should be noted that the image compositing controller 104 and image compositing unit 103 constituting the image processing apparatus of the present invention can be located in an external device different from the camera 101. In such a configuration, the image obtained by the imaging unit 102 should be input to the external device via, for example, a recording medium of a memory card or communication with the camera 101.

[0036] Figure 2 This is a diagram illustrating an example of image alignment synthesis according to this embodiment being applied to 12 images 21a to 21l captured consecutively by the camera unit 102.

[0037] like Figure 2 As shown, with the image stabilization switch on and the release button fully pressed, the camera unit 102 continuously captures images 21a to 21l, which are the objects of image alignment and synthesis, in a time sequence, and outputs these images to the image synthesis unit 103. Figure 2 In images 21a to 21l, the position of the subject 22 differs in the vertical direction. This demonstrates that the framing of the images differs due to camera shake, i.e., it shows that framing error has occurred. Although it may actually happen... Figure 2 The horizontal framing deviation is described, but for simplicity, this embodiment describes... Figure 2 The framing deviation only occurs in the vertical direction. Furthermore, for simplicity, the subject 22 should be stationary in the description, and the image capture conditions (exposure time, aperture value, focal length, etc.) should be fixed.

[0038] The image compositing controller 104 instructs the image compositing unit 103 to perform sequential alignment compositing on a specified number of images (4 in this embodiment) in the order of the images obtained from the camera unit 102.

[0039] In response to the instruction, the image synthesis unit 103 performs sequence alignment synthesis on images 21a to 21d obtained from the camera unit 102.

[0040] Specifically, the image compositing unit 103 first detects the deviation (framing deviation) between the position of the subject 22 in image 21a and the position of the subject 22 in image 21b, and adjusts the position of image 21b to eliminate the framing deviation. Hereinafter, the image 21b after position adjustment is referred to as image 21b'. Similarly, the image compositing unit 103 detects the deviation (framing deviation) between the position of the subject 22 in image 21b' and the position of the subject 22 in image 21c, and adjusts the position of image 21c to eliminate the framing deviation. Hereinafter, the image 21c after position adjustment is referred to as image 21c'. Furthermore, the image compositing unit 103 detects the deviation (framing deviation) between the position of the subject 22 in image 21c' and the position of the subject 22 in image 21d, and adjusts the position of image 21d to eliminate the framing deviation. Hereinafter, the image 21d after position adjustment is referred to as image 21d'.

[0041] The time interval between the recording timing of two consecutively captured images (hereinafter referred to as adjacent images) in images 21a to 21l is short. Therefore, the framing deviation between adjacent images is reduced, which shortens the time period used by the image synthesis unit 103 to detect and calculate the framing deviation. In the above processing, known methods can be used to detect the framing deviation between adjacent images. For example, methods using motion vectors found by comparing the positions of feature points in the images, methods using detection results from inertial sensors (angular velocity sensors, accelerometers, etc.), or methods using both motion vectors and inertial sensor detection results can be employed.

[0042] Next, after adjusting the brightness and cropping of non-overlapping areas during the synthesis process, the image synthesis unit 103 generates a synthesized image 23a by synthesizing image 21a with aligned images 21b', 21c' and 21d'.

[0043] In this way, a number of images (four images in this embodiment) specified by the image compositing controller 104 are sequentially aligned and grouped together. The number of images belonging to a group is limited. This is intended to reduce the degree of degradation in the quality of the synthesized image. That is, once an alignment error occurs in a group synthesized through sequential alignment, the error affects all subsequent images in that group. Therefore, the more images in a group, the greater the degradation in the quality of the synthesized image. To prevent this, the number of images is limited.

[0044] If the number of synthesized images generated by the image synthesis unit 103 has not reached the specified number (3 in this embodiment), the image synthesis controller 104 repeatedly instructs the image synthesis unit 103 to perform sequence alignment synthesis on the four newly obtained images.

[0045] Therefore, the image synthesis unit 103 performs sequence alignment and synthesis on images 21e to 21h, and generates a synthesized image 23e. Furthermore, the image synthesis unit 103 performs sequence alignment and synthesis on images 21i to 21l, and generates a synthesized image 23i.

[0046] When the number of composite images generated by the image compositing unit 103 reaches a predetermined number, the image compositing controller 104 instructs the image compositing unit 103 to perform a reference alignment compositing process. This reference alignment compositing process selects the first composite image as the reference image and aligns and composites the remaining composite images with the reference image.

[0047] In response to the instruction, the image synthesis unit 103 performs reference alignment synthesis of the synthesized images 23a, 23e and 23i.

[0048] Specifically, the image compositing unit 103 first detects the deviation (framing deviation) between the position of the subject 22 in the composite image 23a and the position of the subject 22 in each of the composite images 23e and 23i, and adjusts the positions of the composite images 23e and 23i to eliminate the framing deviation. In the following text, the composite images 23e and 23i after position adjustment are referred to as composite images 23e' and 23i'.

[0049] After that, the image synthesis unit 103 adjusts the brightness and cropping of non-overlapping areas during synthesis, and then synthesizes the synthesized image 23a and the aligned synthesized images 23e' and 23i' to generate a camera shake correction image.

[0050] It should be noted that the framing deviation between adjacent images in the composite images 23a, 23e, and 23i is estimated to be greater than the framing deviation between adjacent images in images 21a to 21l. This is because the time interval between the imaging timings of the first images 21a, 21e, and 21i in the sequential alignment composite group is longer than the time interval between the imaging timings of adjacent images in images 21a to 21l. Therefore, instead of sequential alignment composite, reference alignment composite is performed on composite images 23a, 23e, and 23i, resulting in higher alignment composite accuracy.

[0051] exist Figure 2In this process, although four images are sequentially aligned and composited in each of image groups 21a to 21d, image groups 21e to 21h, and image groups 21i to 21l, the specified number of images composited after sequential alignment varies depending on the camera focal length, which is the imaging condition. Specifically, as the camera focal length increases, the specified number of images composited after sequential alignment decreases. This is because when the camera focal length of camera 101 is long, the amount of framing deviation between adjacent images tends to increase, leading to alignment errors. In this case, the lower the specified number of images composited after sequential alignment, the higher the alignment synthesis accuracy. Conversely, when the camera focal length of camera 101 is short, even if alignment errors occur, the framing deviation is small and insignificant. In this case, the speed of alignment synthesis is increased by increasing the specified number of images composited after sequential alignment.

[0052] Figure 3 This is a flowchart illustrating the alignment and compositing process according to the first embodiment. The process begins when the image stabilization switch is turned on.

[0053] First, in step S301, when the release button is half-pressed, the camera condition output unit 105 detects the camera focal length through the focal length detection unit 111 and outputs it to the image synthesis controller 104.

[0054] In step S302, the image compositing controller 104 sets the number (specified number) of images belonging to a sequence alignment compositing group to the image compositing unit 103 based on the camera focal length output from the camera condition output unit 105 in step S301. Specifically, the specified number is set to 4 when the camera focal length falls within a predetermined intermediate range. The specified number is set to 3 when the camera focal length is longer than the maximum value in the intermediate range. And the specified number is set to 6 when the camera focal length is shorter than the minimum value in the intermediate range. Although the specified number is set to 3, 4, or 6 in this embodiment, the specified number can be set more precisely based on the camera focal length. That is, it is sufficient to satisfy the condition that the specified number set for short camera focal lengths is greater than the specified number set for long camera focal lengths.

[0055] In step S303, when the release button is fully pressed, the camera unit 102 begins recording. Thus, the camera unit 102 captures a series of images in a time sequence and outputs these images to the image compositing unit 103. It should be noted that regardless of the specified number set in step S302, the camera unit 102 captures the set number of images (12 in this embodiment) in step S303. Therefore, for example, if the specified number is set to 4 in step S302, the image compositing controller 104 sets 3 as the number (specified number) of composite images to be generated for generating camera shake correction images.

[0056] In step S304, the image compositing controller 104 instructs the image compositing unit 103 to perform sequential alignment and compositing of a specified number of images. The image compositing unit 103 responds to this instruction by performing... Figure 2 The sequences of images 21a to 21d are aligned and synthesized to generate a composite image 23a.

[0057] In step S305, the image compositing controller 104 repeatedly issues the instruction of step S304 to the image compositing unit 103 until a predetermined number of composite images are generated. If the predetermined number of composite images has been generated (yes in step S305), the process proceeds to step S306. Thus, the image is generated... Figure 2 Synthetic images 23a, 23e and 23i.

[0058] In step S306, the image compositing controller 104 instructs the image compositing unit 103 to perform reference alignment compositing and completes the process. Thus, reference alignment compositing of the composite images 23a, 23e, and 23i is performed, and a camera shake-corrected image is generated.

[0059] Since the number of images synthesized by the image synthesis unit 103 in sequence alignment is changed based on the camera focal length of the camera 101 in this way (specified number), the degradation of the camera shake correction image due to alignment error is reduced.

[0060] Although this embodiment sets the number of images synthesized through sequential alignment based solely on the camera focal length (a specified number), this specified number can be set by considering other camera conditions. For example, the following camera conditions (a) to (e) can be considered. This further reduces alignment errors in image synthesis stabilization, thereby enabling the acquisition of high-definition camera shake-corrected images.

[0061] (a) Camera shake

[0062] When the camera shake detected by the camera shake detection unit 107 is large, the number of synthesized images after sequence alignment decreases. In this case, the camera condition output unit 105 outputs the camera shake detected by the camera shake detection unit 107 as the camera condition. The camera shake detected before the start of the camera processing in step S303 can be used.

[0063] (b) Optical image stabilization performance

[0064] When the image stabilization system 108 performs optical image stabilization in cooperation with the camera 101 and the camera lens 112, the optical image stabilization performance varies depending on the combination of the camera 101 and the camera lens 112. Therefore, since a combination with low image stabilization performance amplifies framing deviations caused by camera shake, the number of composite images for sequential alignment is reduced. In this case, the imaging condition output unit 105 outputs the optical image stabilization performance as imaging conditions based on the combination of the camera 101 and the camera lens 112. For example, when optical image stabilization is performed using either the camera 101 or the camera lens 112, the number of composite images for sequential alignment is reduced compared to the case where optical image stabilization is performed using both the camera 101 and the camera lens 112.

[0065] (c) Frequency characteristics of camera shake

[0066] Typically, the image stabilization accuracy of the image stabilization system 108 installed in the camera 101 is low for low-frequency camera shake. Therefore, when the camera shake detection unit 107 detects low-frequency camera shake, the number of synthesized images for sequence alignment is reduced. In this case, the imaging condition output unit 105 outputs the imaging condition based on the camera shake detected by the camera shake detection unit 107. For example, when the frequency of the principal component of the camera shake detected by the camera shake detection unit 107 is low, the number of synthesized images for sequence alignment is reduced compared to when the frequency of the principal component of the camera shake detected by the camera shake detection unit 107 is high.

[0067] (d) Time elapsed since the image stabilized

[0068] Because the image stabilization accuracy of the image stabilization system 108 installed in the camera 101 is low in the early stages of image stabilization, the number of sequentially aligned composite images is reduced in the first time period (e.g., one second) from the start of image stabilization. In this case, the shooting condition output unit 105 outputs the shooting condition as the time elapsed since the start of image stabilization, measured by the RTC 109 installed in the camera 101. For example, if shooting begins before the first time period elapsed since the start of image stabilization, the number of sequentially aligned composite images is reduced compared to if shooting begins after the first time period elapsed since the start of image stabilization. Image stabilization begins when the image stabilization switch is turned on or the release button is half-pressed.

[0069] (e) Time elapsed since the start of filming

[0070] When a user takes a picture by holding camera 101, camera shake increases with the time elapsed since the start of recording. This is because the user cannot check the subject through the viewfinder eyepiece during recording. Therefore, during recording, after a second time interval (e.g., two seconds) has elapsed since the start of recording, the number of composite images with sequence alignment decreases. In this case, the recording condition output unit 105 outputs the recording condition as the time elapsed since the start of recording, measured by the RTC 109 installed in camera 101. For example, when the time elapsed since the start of recording is longer than the second time interval, the number of composite images with sequence alignment decreases compared to when the time elapsed since the start of recording is shorter than the second time interval.

[0071] As described above, in this embodiment, the image compositing controller 104 controls the image compositing unit 103 to perform sequence alignment compositing and reference alignment compositing in combination according to the shooting conditions. This reduces the impact of alignment errors on image compositing stabilization, enabling the acquisition of high-definition camera shake-corrected images.

[0072] Next, a second embodiment of the invention will be described. The advantage of sequential alignment synthesis is the small framing deviation between the two images to be aligned. This is because the two images to be aligned are adjacent images, and the time interval between the shooting timings of adjacent images is extremely short. Therefore, the range for searching for framing deviations can be narrowed, and alignment synthesis can be performed at high speed. In contrast, in reference alignment synthesis, the larger the time interval between the shooting timing of the reference image and the shooting timing of the alignment image, the larger the framing deviation. Therefore, it is necessary to expand the range for searching for framing deviations, and alignment synthesis requires more time. However, since reference alignment synthesis aligns and synthesizes all remaining images with the reference image, only the image that caused the alignment error becomes the cause of degradation in the synthesized image. Therefore, in the case of an alignment error occurring within a group, the degree of degradation of the synthesized image in reference alignment synthesis becomes less than the degree of degradation of the synthesized image in sequential alignment synthesis.

[0073] Therefore, this embodiment adopts the following configuration: when the framing deviation between the two images to be aligned is small, reference alignment synthesis is performed, but sequence alignment synthesis is not performed.

[0074] It should be noted that the hardware configurations in this embodiment that are the same as those in the first embodiment are indicated by the same reference numerals, and their descriptions are omitted.

[0075] In this embodiment, when the release button is fully pressed while the image stabilization switch is on, the camera unit 102 continuously outputs 10 images in a time sequence as the objects for image alignment and synthesis.

[0076] Furthermore, in this embodiment, the camera condition output unit 105 outputs not only the camera focal length detected by the focal length detection unit 111, but also the camera shake amount of the camera 101 detected by the camera shake detection unit 107 as camera conditions to the image synthesis controller 104.

[0077] Figure 4 This is a diagram illustrating an example of image alignment synthesis according to the second embodiment being applied to 10 images 41a to 41j that are continuously output in a time sequence after being captured by the camera unit 102.

[0078] Figure 4 This illustrates a scenario where the camera condition output unit 105 outputs a small amount of camera shake during the first half of the video capture (the first time period from capturing images 41a to 41f) and a large amount of camera shake during the second half of the video capture (the second time period from capturing images 41g to 41j). For simplicity, the camera focal length output from the camera condition output unit 105 should be within a moderate range between not too large and not too small.

[0079] In this case, the image synthesis controller 104 instructs the image synthesis unit 103 to perform reference alignment synthesis of images 41a to 41f captured in the first time period based on the camera shake amount output from the camera condition output unit 105.

[0080] In response to the instruction, the image synthesis unit 103 aligns each of the images 41b to 41f with the image 41a, which serves as a reference image, to eliminate individual framing deviations, and synthesizes the images 41a to 41f to generate a composite image 42a.

[0081] Furthermore, the image synthesis controller 104 instructs the image synthesis unit 103 to perform sequence alignment synthesis on the images 41g to 41j captured in the second time period based on the amount of camera shake output from the camera condition output unit 105.

[0082] In response to this instruction, the image synthesis unit 103 adjusts the position of image 41h to eliminate the framing deviation of image 41h relative to image 41g. Hereinafter, the image 41h after position adjustment is referred to as image 41h'. Similarly, the position of image 41i is adjusted to eliminate the framing deviation of image 41i relative to image 41h'. Hereinafter, the image 41i after position adjustment is referred to as image 41i'. Similarly, the position of image 41j is adjusted to eliminate the framing deviation of image 41j relative to image 41i'. Hereinafter, the image 41j after position adjustment is referred to as image 41j'.

[0083] Next, after adjusting the brightness and cropping of non-overlapping areas during the synthesis process, the image synthesis unit 103 generates a synthesized image 42g by synthesizing the image 41g with the aligned images 41h', 41i' and 41j'.

[0084] In this manner, the image compositing controller 104 instructs the image compositing unit 103 to perform reference alignment compositing on a group of images captured during a first time period in which the camera shake is less than a threshold. On the other hand, the image compositing controller 104 instructs the image compositing unit 103 to perform sequence alignment compositing on a group of images captured during a second time period in which the camera shake is greater than a threshold.

[0085] As a result, the image compositing unit 103 generates composite images 42a and 42g. After this, the image compositing controller 104 instructs the image compositing unit 103 to perform reference alignment compositing of the composite images 42a and 42g.

[0086] In response to the instruction, the image synthesis unit 103 performs reference alignment synthesis of the synthesized images 42a and 42g to generate a camera shake-corrected image.

[0087] This configuration reduces the degradation of the synthesized image due to alignment errors in each group and shortens the time required for alignment synthesis.

[0088] Figure 5 This is a flowchart illustrating the alignment and compositing process according to the second embodiment. The process begins when the image stabilization switch is turned on.

[0089] It should be noted that the same step number is assigned to the execution Figure 3 The same processing steps as in [the previous section].

[0090] First, perform the processing steps S301 to S303.

[0091] In step S501, the camera condition output unit 105 uses the camera shake detection unit 107 to detect the current camera shake of the camera 101, and outputs the detection result as a camera condition to the image synthesis controller 104.

[0092] In step S502, the image compositing controller 104 determines whether the product of the camera shake amount output from the camera condition output unit 105 in step S501 and the camera focal length is less than a threshold. If the product is less than the threshold, the process proceeds to step S503. If the product is not less than the threshold, the process proceeds to step S504. The reason for calculating the product of the camera focal length and the camera shake amount will be described. Even if the camera shake amount is small, the framing deviation on the image plane of the camera unit 102 becomes large when the camera focal length is long. Conversely, even if the camera focal length is short, the framing deviation on the image plane of the camera unit 102 becomes large when the camera shake amount is large. Therefore, the framing deviation amount on the image plane is determined by calculating the product of the camera focal length and the camera shake amount. It should be noted that, as another method, the threshold can be changed according to the camera focal length. For example, a first threshold is set to correspond to a first camera focal length, and a second threshold is set to correspond to a second camera focal length. In this way, by comparing the set threshold with the amount of camera shake, the same effect is achieved as by calculating the product of the camera focal length and the amount of camera shake.

[0093] In step S503, the image compositing controller 104 instructs the image compositing unit 103 to perform reference alignment compositing and proceeds to step S505. Thus, the image compositing unit 103 can perform compositing processing that is almost unaffected by alignment errors. Furthermore, since the framing deviation between the two images to be aligned is understood to be small, the image compositing controller 104 instructs the image compositing unit 103 to narrow the search range for framing deviation. Therefore, the image compositing unit 103 can shorten the alignment compositing time.

[0094] In step S504, the image compositing controller 104 instructs the image compositing unit 103 to perform sequence alignment compositing, and then proceeds to step S505. Thus, the image compositing unit 103 can perform alignment compositing at high speed.

[0095] In step S505, the image compositing controller 104 determines whether the camera unit 102 has captured the set number of images (in...). Figure 4 In the example, image 10 is used, and it is determined whether all images have been aligned and composited. If the condition is not met, the process returns to step S501. The process from step S501 is repeated until the condition of step S505 is met. Thus, the image is generated. Figure 2 The composite images 23a, 23e and 23i.

[0096] In step S306, the image compositing controller 104 instructs the image compositing unit 103 to perform reference alignment compositing and completes the process. Thus, reference alignment compositing of the composite images 42a and 42g is performed, and a camera shake-corrected image is generated.

[0097] According to this process, during image capture, when the product of the focal length and the amount of camera shake is small, the process proceeds from step S502 to step S503, and reference alignment synthesis is performed. On the other hand, during image capture, when the product of the focal length and the amount of camera shake is large, the process proceeds from step S502 to step S504, and sequential alignment synthesis is performed. Therefore, by combining reference alignment synthesis and sequential alignment synthesis, the number of images undergoing sequential alignment synthesis is reduced. As a result, compared to generating camera shake-corrected images by performing sequential alignment synthesis on all images 41a to 41j, the quality degradation of camera shake-corrected images due to alignment errors is reduced.

[0098] If the camera shake decreases after changing from reference alignment synthesis to sequence alignment synthesis due to increased camera shake, reference alignment synthesis can be used again.

[0099] Furthermore, when reference alignment synthesis is changed to sequence alignment synthesis, the number of remaining images can be considered. For example, in Figure 4 When the camera shake of images 41a to 41i is small and the camera shake of image 41j is large, even if the method is changed to sequence alignment synthesis starting from image 41j, it essentially becomes reference alignment synthesis of images 41a and 41j. Therefore, if the number of remaining images is less than a predetermined number (e.g., less than two images), the synthesis method can remain unchanged.

[0100] Figure 6 This is a diagram illustrating an example of image alignment synthesis according to a variant of the second embodiment being applied to ten images 61a to 61j that are output sequentially in time by using the camera unit 102 for long-term video recording.

[0101] like Figure 6 As shown, in the case of long-duration video recording, sequence alignment and synthesis are performed in the early stage of recording (the time period from capturing images 61a to 61d) to generate a composite image 62a. On the other hand, reference alignment and synthesis are performed in the later stage (the time period from capturing images 61e to 61j) to generate a composite image 62e.

[0102] The reason for performing reference alignment synthesis in the later stages of long-duration video recording is that the amount of camera shake tends to become greater in the later stages than in the earlier stages, and alignment errors are more likely to occur in the later stages. Therefore, reference alignment synthesis is performed on images 61e to 61j captured in the later stages of recording, using image 61e as the reference image. Furthermore, the remaining images 61f to 61j are aligned with image 61e while expanding the search range for framing deviations, and a composite image 62e is generated.

[0103] After that, the image compositing controller 104, after completing the generation of the composite images 62a and 62e using the image compositing unit 103, instructs the image compositing unit 103 to perform reference alignment compositing of the composite images 62a and 62e.

[0104] In response to the instruction, the image synthesis unit 103 performs reference alignment synthesis of the synthesized images 62a and 62e to generate a camera shake-corrected image.

[0105] In such Figure 6 In the example, although the alignment compositing time is longer, accurate alignment compositing can be performed. Therefore, the alignment compositing method to be performed under conditions of high camera shake can be selected from reference alignment compositing and sequential alignment compositing, depending on factors such as the camera mode. For example, reference alignment compositing can be performed under conditions of high camera shake in high image quality mode, while sequential alignment compositing can be performed under conditions of high camera shake in other modes.

[0106] Next, a third embodiment of the invention will be described. The first and second embodiments assume that multiple images used as alignment objects are captured under nearly identical conditions, and the motion vectors between the images are correctly obtained to detect framing deviations. However, when the multiple images used as alignment objects include images captured under different conditions (e.g., using lighting not used in capturing other images), it may be impossible to obtain the correct motion vectors between the images.

[0107] Therefore, the third embodiment is configured to achieve stable alignment and synthesis even for images with different imaging conditions. The configuration of this embodiment is described in detail below.

[0108] It should be noted that the hardware configurations in this embodiment that are the same as those in the first embodiment are indicated by the same reference numerals, and their descriptions are omitted.

[0109] Figure 7 This demonstrates the generation of images through a front-curtain synchronized camera. Figure 2 The diagram shows an example of image alignment synthesis according to the third embodiment applied to the 12 images 21a to 21l shown. That is, Figure 2 Image 21b was taken with a flash. The other images 21a and 21c to 21l were taken without a flash.

[0110] In known front-curtain sync cameras, the flash is fired in the early stages of exposure. In this embodiment, image 21a is captured without flash firing immediately before obtaining image 21b generated by flash photography. This allows for the correction of framing deviations in image 21b by obtaining the motion vector trajectory 71 described below. This will be described in detail below.

[0111] In this embodiment, when image 21b is captured with the flash on during the early stages of image capture, images 21a and 21c are captured immediately before and after image 21b, with the flash off (under the same capture conditions). It should be noted that in front-curtain synchronized capture, image 22b is captured with an exposure time shorter than that of the other images 21a and 21c to 21l. Therefore, the framing deviation of image 21b becomes an intermediate value between the framing deviations of images 21a and 21c. Therefore, when aligning image 21b with image 21a through sequence alignment synthesis, the framing deviation of image 21c is first detected, the position of image 21c is adjusted to eliminate the framing deviation, and an aligned image 21c' is generated. Next, the motion vector trajectory 71 between images 21a and 21c' is calculated, and the calculated motion vector trajectory 71 is used to align image 21b. That is, even if no framing deviation of image 21b is detected, the framing deviation of image 21b is corrected and an aligned image 21b' is obtained.

[0112] It should be noted that image 21a, captured before the flash-fired image 21b, can be an image captured before the release button is fully pressed (a so-called live view image) and may not be used for compositing. That is, the flash is fired after the release button is fully pressed. When compositing the flash-fired image and the subsequent non-flash-fired image, the live view image 21a captured before the release button is fully pressed can be used to align image 21b. In this way, by using the live view image to align the flash-fired image, the time interval between fully pressing the release button and the flash firing is shortened, and the camera's chances of missing the target image are reduced.

[0113] Figure 8 This shows the generation via rear curtain synchronous camera. Figure 2 The diagram shows an example of image alignment synthesis according to the third embodiment applied to the 12 images 21a to 21l shown. That is, Figure 2 Image 21k was taken with a flash. Images 21a to 21j and 21l were taken without a flash.

[0114] In known rear-curtain sync shooting, the flash is fired during the later stages of exposure. In this embodiment, immediately after obtaining the image 21k generated by flash photography, image 21l is captured without flash firing. This allows for the correction of framing deviations in image 21k by obtaining the motion vector trajectory 81. This will be described in detail below.

[0115] In this embodiment, when image 21k is captured with the flash on during the later stages of video recording, images 21j and 21l are captured immediately before and after image 21k with the flash off (under the same recording conditions). It should be noted that in rear-curtain synchronized recording, image 22k is captured with an exposure time shorter than that of the other images 21a to 21j and 21l. Therefore, the framing deviation of image 21k becomes an intermediate value between the framing deviations of images 21j and 21l. Therefore, when aligning image 21k with image 21j through sequence alignment synthesis, the framing deviation of image 21l is first detected, the position of image 21l is adjusted to eliminate the framing deviation, and an aligned image 21l' is generated. Next, the motion vector trajectory 81 between images 21j and 21l' is calculated, and the calculated motion vector trajectory 81 is used to align image 21k. That is, even if no framing deviation of image 21k is detected, the framing deviation of image 21k is corrected and an aligned image 21k' is obtained.

[0116] It should be noted that image 21l, taken after the flash-fired image 21k, can be used only to correct framing errors in image 21k generated by flash photography and may not be used for compositing. This is because the subject tends to move after the flash fires. When image 21l includes a moving subject, the quality of the camera shake-corrected image may degrade.

[0117] In this way, Figure 2 When images 21a to 21l include images whose shooting conditions differ from those of other images, the image synthesis unit 103 uses the framing deviations detected in images taken immediately before and after the images to correct the framing deviations of images with different shooting conditions.

[0118] Figure 9 It is shown in Figure 2 The diagram illustrates an example of applying image alignment synthesis according to the third embodiment when the imaging conditions of the last image in the second sequence alignment group are different. In the description, a sequence alignment group refers to a group comprising images used to generate a synthesized image through sequence alignment.

[0119] The image 21h, taken at the exact time of the fireworks display, differs in brightness from the image 21g, taken immediately before it. In this situation, the framing discrepancy between images 21g and 21h cannot be accurately determined. Therefore, in this embodiment, a method using... Figure 7 and Figure 8The same method is used to correct the framing deviation of image 21h by taking images 21g and 21i immediately before and after image 21h. Specifically, the framing deviation of image 21i relative to image 21g is detected, the position of image 21i is adjusted to eliminate the framing deviation, and an aligned image 21i' is generated. Next, the motion vector trajectory 91 between image 21g and image 21i' is calculated, and the calculated motion vector trajectory 91 is used to align image 21k. That is, even if the framing deviation of image 21h is not detected, the framing deviation of image 21h is corrected, and an aligned image 21h' is obtained.

[0120] like Figure 9 As shown, images from image 21e up to image 21i, which was taken immediately after image 21h generated by the fireworks camera, are grouped into a sequence alignment group. However, image 21i is only used to calculate the motion vector trajectory 91 for correcting the framing deviation of image 21h, and is not used to generate the composite image 23e. That is, although in Figure 2 , Figure 7 and Figure 8 Images 21e to 21h belong to the sequence alignment group, but in Figure 9 Images 21e to 21i belong to the sequence alignment group. On the other hand, in Figure 2 , Figure 7 , Figure 8 and Figure 9 In each case, a composite image 23e is generated using images 21e, 21f', 21g', and 21h'. This is because the alignment accuracy is uniformized when the number of images synthesized through sequence alignment is consistent for each group. Therefore, when the fireworks image is image 21g and not the last image 21h in the group, images 21e to 21h are grouped into a sequence alignment group, and a composite image 23e is generated using images 21e, 21f', 21g', and 21h'.

[0121] Figure 9 Examples of images with different shooting conditions include those where the brightness of the subject differs from the brightness of other images in the sequence alignment group. However, shooting conditions are not limited to brightness. For example, the presence or absence of a flash at the shooting time can be a shooting condition.

[0122] Figure 10 This is a flowchart illustrating the alignment compositing process according to the third embodiment. The process begins when the image stabilization switch is turned on. The process is applied to... Figure 7 , Figure 8 and Figure 9 Each pattern shown.

[0123] It should be noted that the same step number is assigned to the execution. Figure 3 The same processing steps as in [the previous section].

[0124] First, perform the processing steps S301 to S303.

[0125] In step S1001, the image compositing controller 104 instructs the image compositing unit 103 to detect framing deviation in the image output from the camera unit 102. In response to this instruction, the image compositing unit 103 detects the framing deviation between the output image and the image immediately preceding it.

[0126] In step S1002, the image compositing controller 104 instructs the image compositing unit 103 to detect the brightness value of the image output from the camera unit 102. In response to this instruction, the image compositing unit 103 detects the brightness value of the output image.

[0127] In step S1003, the image compositing controller 104 determines whether the framing deviation and brightness values ​​have been detected for the specified number of images (set in step S302). If it is determined that these values ​​have not been detected, the image compositing controller 104 instructs the image compositing unit 103 to repeat the processing of steps S1001 and S1002. Thus, the detection of framing deviation and brightness values ​​for images belonging to the current sequence alignment group is completed.

[0128] In step S1004, the image synthesis controller 104 determines, based on the brightness values ​​detected from images belonging to the current sequence alignment group, whether there exists an image whose brightness value differs from that of other images in the current sequence alignment group. If an image with a different brightness value (image with different brightness) exists, the process proceeds to step S1005; otherwise, the process proceeds to step S1007.

[0129] In step S1005, the image compositing controller 104 determines whether the different brightness images determined in step S1004 are the last images of the current alignment sequence. If it is not the last image, the process proceeds to step S1006. If it is the last image, the process returns to step S1001 to detect the framing deviation and brightness value of the initial image of the next alignment sequence. Therefore, when the last image of the current alignment sequence is a different brightness image, a vector trajectory for correcting the framing deviation of the last image is obtained by using an image captured immediately after the last image. If there is no next alignment sequence, the image compositing controller 104 controls the camera unit 102 to capture an additional image including the subject 22, and the process proceeds to step S1006. This process is not included in... Figure 10 In the flowchart.

[0130] In step S1006, the image compositing controller 104 calculates the framing deviation of images with different brightness levels. For example, when Figure 9When the brightness values ​​of image 21h are different, the framing deviation between images 21g and 21i, which were captured immediately before and after image 21h, is obtained, and the position of image 21i is adjusted to eliminate the framing deviation. Next, the motion vector trajectory 91 of the aligned image 21i' relative to image 21g is calculated. Then, the framing deviation of image 21h is calculated based on the motion vector trajectory 91. It should be noted that the framing deviation of image 21h becomes the intermediate amount of the framing deviation of adjacent images 21g and 21i.

[0131] In step S1007, the image compositing controller 104 instructs the image compositing unit 103 to composite the images after aligning the images belonging to the current sequence alignment group based on the framing deviations detected and calculated in steps S1001 and S1006. According to this process, for example, based on... Figure 9 Images 21b', 21c', and 21d' are generated from images 21a to 21d, and a composite image 23a is obtained by synthesizing images 21a, 21b', 21c', and 21d'. Furthermore, based on... Figure 9 Images 21e to 21i are used to generate images 21f' to 21i', and composite image 23e is obtained by combining images 21e and 21f' to 21i'.

[0132] In step S1008, the image compositing controller 104 determines whether a predetermined number of composite images have been generated. The image compositing controller 104 repeats the process from step S1001 until the predetermined number of composite images has been generated. If the predetermined number of composite images has been generated (yes in step S1008), the process proceeds to step S1009. Thus, a composite image is generated. Figure 9 Synthetic images 23a, 23e and 23i.

[0133] In step S1009, the image compositing controller 104 instructs the image compositing unit 103 to perform reference alignment compositing and completes the process. In response to the instruction, the image compositing unit 103 performs reference alignment compositing of the composite images 23a, 23e, and 23i to generate a camera shake-corrected image.

[0134] In this way, when generating images of different brightness during recording with the image stabilization switch on, the image synthesis unit 103 detects the framing deviation of the images of different brightness by using the framing deviation (trajectory) between images captured immediately before and after capturing the images of different brightness. Then, if the image of different brightness is the last image in the current sequence alignment group, the framing deviation of the images of different brightness is also detected by using images from the next sequence alignment group.

[0135] Therefore, even when the image stabilization switch is on and images of different brightness are generated during recording, high-precision alignment and synthesis can be achieved.

[0136] As described above, a framing deviation detection method used when images of different brightness are in a sequence alignment group has been described. Next, a framing deviation detection method used in situations such as... Figure 4 Images 41a to 41f and Figure 6 The group of images 61e to 61j, etc., which are reference-aligned for generating a composite image (hereinafter referred to as the reference alignment group), includes images with different brightness values.

[0137] As described above, in the second embodiment, during the time period when the amount of camera shake of the camera 101 detected by the camera shake detection unit 107 is small ( Figure 4 Images captured during the long-duration video recording process, and images taken during the later stages of filming. Figure 6 The captured images are included in the reference alignment group.

[0138] Figure 11 It is shown in Figure 6 The diagram shows an example of alignment synthesis applied according to a variant of the third embodiment when images are captured under different imaging conditions. Specifically, 10 images 1101a to 1101j are continuously output in a time sequence through long-term imaging by the imaging unit 102. Six of these ten images 1101e to 1101j (corresponding to images captured in the later stages of the long-term imaging)... Figure 6 Images 61e to 61j belong to the reference alignment group. This group includes images 1101i with different brightness values.

[0139] Images 1101a to 1101d (corresponding to those captured in the early stages of long-duration video recording) Figure 6 Images 61a to 61d in the sequence are aligned and synthesized to generate a synthesized image 1002a.

[0140] The image synthesis unit 103 calculates a motion vector trajectory 1103, which shows the difference between the framing deviations of images 1101h and 1101j, captured immediately before and after images 1101i with different brightness, relative to a reference image 1101e. The framing deviation of the images 1101i with different brightness is calculated based on the motion vector trajectory 1103. The framing deviation of image 1101i becomes an intermediate amount between the framing deviations of adjacent images 1101h and 1101j. Subsequently, the images are synthesized by aligning them based on the framing deviations to obtain a synthesized image 1102e.

[0141] After that, the image compositing controller 104 instructs the image compositing unit 103 to perform reference alignment compositing of the composite images 1102a and 1102e after the image compositing unit 103 has completed the generation of the composite images 1102a and 1102e.

[0142] In response to the instruction, the image synthesis unit 103 performs reference alignment synthesis of the synthesized images 1102a and 1102e to generate a camera shake-corrected image.

[0143] Figure 12 This is a flowchart illustrating the alignment and compositing process according to a variant of the third embodiment. When the release button is fully pressed while the image stabilization switch is on, the camera unit 102 begins continuous imaging. Figure 12 The processing begins at a set time during the post-production stage of filming.

[0144] In step S1201, the image compositing controller 104 instructs the image compositing unit 103 to detect the brightness value of the image output from the camera unit 102. In response to this instruction, the image compositing unit 103 detects the brightness value of the output image.

[0145] In step S1202, the image synthesis controller 104 determines whether the brightness value of the image output from the camera unit 102 detected in step S1201 is different from the brightness value of the image output from the camera unit 102 immediately preceding it. As a result of the determination, if the brightness values ​​are different, the process skips step S1203 and proceeds to step S1204; otherwise (if not in step S1202), the process proceeds to step S1203.

[0146] In step S1203, the image compositing controller 104 instructs the image compositing unit 103 to detect framing deviation in the image output from the camera unit 102. In response to this instruction, the image compositing unit 103 detects the framing deviation between the output image and its reference image.

[0147] In step S1204, the image compositing controller 104 determines whether the number of settings captured during the post-capture stage has been detected (in... Figure 11 In the example, image 6) shows the framing deviation. The image synthesis controller 104 repeats the processing from step S1201 until the framing deviation of a set number of images has been detected. That is, the framing deviation of each image 1101f to 1101j sequentially output from the imaging unit 102 relative to the reference image 1101e is calculated. However, the framing deviation of images 1101i with different brightness relative to the reference image 1101e is not detected.

[0148] In step S1205, the image synthesis controller 104 determines whether the images with different brightness levels are the last images of the reference alignment group. If they are the last images, the process proceeds to step S1206; otherwise (if not in step S1205), the process proceeds to step S1207.

[0149] In step S1206, the image compositing controller 104 controls the camera unit 102 to capture an additional image including the subject 22, and instructs the image compositing unit 103 to detect the framing deviation of the additional image relative to the reference image. Then, the image compositing controller 104 proceeds to step S1207. Thus, even if the image with different brightness is the last image of the reference alignment group, the framing deviation of the additional image captured immediately afterward relative to the reference image is obtained.

[0150] In step S1207, the image synthesis controller 104 calculates the framing deviation of images with different brightness levels based on the framing deviation of images captured immediately before and after it relative to the reference image. Figure 11 In the example, the framing deviation of image 1101i is calculated based on motion vector trajectory 1103, which shows the difference between the framing deviations of images 1101h and 1101j.

[0151] In step S1208, the image compositing controller 104 performs reference alignment compositing of the images of the reference alignment group based on the framing deviation calculated in step S1203 and the framing deviation calculated in step S1206. Then, the image compositing controller 104 completes the process.

[0152] In this way, when images of different brightness are generated during the shooting of the reference alignment group, the framing deviation of the images of different brightness is detected by using the difference (motion vector trajectory) between the framing deviation of the images taken immediately before and after the reference image relative to the reference image. High-precision alignment synthesis can be performed even when images of different brightness are generated during the shooting of the reference alignment group.

[0153] although Figure 12 Shown in Figure 6 This applies to the post-production stage of long-duration video recording, where images with varying brightness are generated, but the same processing method is applicable to... Figure 4 The case where images of different brightness are generated during periods of low camera shake in camera 101.

[0154] As described above, in this embodiment, stable image alignment synthesis can be achieved by satisfying the following configuration.

[0155] 1. In the presence of images with different shooting conditions (such as flash illumination and changes in subject brightness), the framing deviation of the images under different conditions is calculated by using the difference in framing deviation (motion vector trajectory (alignment trajectory)) between images taken immediately before and after the images under different conditions. Then, the images under different conditions are aligned to eliminate framing deviation.

[0156] 2. When obtaining images under different conditions, obtain images with consistent shooting conditions immediately before and after capturing images under different conditions. Specifically, in the case of front-curtain sync shooting, obtain an image without flash immediately before obtaining an image with flash. In the case of rear-curtain sync shooting, obtain an image without flash immediately after obtaining an image with flash.

[0157] 3. When the image under different conditions is the last image in a group, the framing deviation of the last image is calculated using the first image of the next group, and the last image is aligned to eliminate the framing deviation. However, the first image of the next group is not used for image synthesis of groups that include images under different conditions. Furthermore, if there is no next group, the camera unit 102 captures an additional image including the subject 22, and uses this additional image to calculate the framing deviation of the last image.

[0158] 4. When images under different conditions are included in the reference alignment group, calculate the difference between the framing deviations of images taken immediately before and after the images under different conditions, and use the calculated difference to calculate the framing deviation of the images under different conditions.

[0159] Furthermore, it becomes easier to obtain the image intended by the user when calculating the framing deviation of images under different conditions without using images not used in image synthesis of groups including images under different conditions. Specifically, images without flash in front-curtain sync shooting and images without flash after flash in rear-curtain sync shooting can be excluded from image synthesis.

[0160] Although the above embodiments describe the case of compositing multiple images for the purpose of image stabilization, the present invention is applicable to other techniques for aligning and compositing multiple images, such as HDR image compositing. Furthermore, the alignment technique of the present invention is also applicable to techniques that do not involve compositing multiple images.

[0161] Other embodiments

[0162] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0163] Although the invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims shall be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0164] This application claims the benefit of Japanese Patent Application 2020-151561, filed on September 9, 2020, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus, comprising: An alignment unit is configured to perform sequential alignment and reference alignment, wherein the sequential alignment is used to align adjacent images and the reference alignment is used to align two or more images other than the reference image with the reference image. A control unit, configured to control the alignment unit to align multiple images including the same subject captured sequentially in a time sequence by the imaging unit by combining the sequence alignment and the reference alignment. A setting unit is configured to group the plurality of images for either sequence alignment synthesis or reference alignment synthesis. The control unit sets each group of images as either a reference alignment synthesis or a sequence alignment synthesis, based on the amount of framing deviation during the time period of capturing each group of images.

2. The image processing apparatus of claim 1, further comprising an image compositing unit, the image compositing unit being configured to perform sequence alignment compositing and reference alignment compositing, the sequence alignment compositing being used to composite an image that has undergone the sequence alignment using the alignment unit, and the reference alignment compositing being used to composite an image that has undergone the reference alignment using the alignment unit. in, The control unit controls the image synthesis unit to generate a synthesized image by combining the sequence alignment synthesis and the reference alignment synthesis.

3. The image processing apparatus according to claim 2 further includes a camera condition output unit, and in, The control unit controls the combination of the sequence alignment synthesis and the reference alignment synthesis in the image synthesis unit according to the first camera condition output from the camera condition output unit.

4. The image processing apparatus according to claim 2, wherein, In the case where the plurality of images include specific images that differ in terms of the second shooting conditions, the alignment unit calculates the alignment trajectory of images captured immediately before and after the specific image, and uses the alignment trajectory to align the specific image.

5. The image processing apparatus according to claim 4, wherein, Images taken immediately before and after the specific image are consistent with the second camera conditions.

6. The image processing apparatus according to claim 5, wherein, The second camera condition is the presence of a flash. In the case of generating the multiple images via front-curtain synchronous imaging, immediately before the flash-lit image, which is a specific image different in terms of the second imaging conditions, the imaging unit captures a flash-unlit image that is consistent in terms of the second imaging conditions. In the case where the multiple images are generated by rear-curtain synchronous imaging, immediately after the flash-lit image, which is different in terms of the second imaging conditions, the imaging unit captures a flash-unlit image that is consistent in terms of the second imaging conditions.

7. The image processing apparatus according to claim 6, wherein, The image synthesis unit does not use the image without flash, which is captured immediately after the image with flash illumination during the rear curtain synchronous camera.

8. The image processing apparatus according to claim 4, in, If the specific image is included as the last image in a group and there is no next group, the control unit controls the camera unit to capture an additional image including the same subject after capturing the plurality of images, and uses the additional image as the image captured immediately after the specific image.

9. The image processing apparatus according to claim 8, wherein, If the specific image is included as the last image in a group and there is a next group that includes the specific image, the control unit uses the first image of the next group as the image captured immediately after the specific image.

10. The image processing apparatus according to claim 8, wherein, For image synthesis of a group that includes the specific image as the last image, the control unit does not use images captured immediately after the specific image.

11. The image processing apparatus according to claim 4, wherein, When the specific image is included in the sequence alignment group, the control unit calculates the alignment trajectory based on the framing deviation of the image taken immediately after the specific image relative to the image taken immediately before the specific image.

12. The image processing apparatus according to claim 4, wherein, When the specific image is included in the reference alignment group, the control unit calculates the alignment trajectory based on the difference between the framing deviations of images taken immediately before and after the specific image relative to the reference image.

13. A camera device, comprising: The camera unit has an image sensor and outputs the captured images; An alignment unit is configured to perform sequential alignment and reference alignment, wherein the sequential alignment is performed between adjacent images, and the reference alignment is used to align two or more images other than the reference image with the reference image. A control unit, configured to control the alignment unit to align images including the same subject captured sequentially in a time sequence by the imaging unit by combining the sequence alignment and the reference alignment. A setting unit is configured to group the images for either sequence alignment synthesis or reference alignment synthesis. The control unit sets each group of images as either a reference alignment synthesis or a sequence alignment synthesis, based on the amount of framing deviation during the time period of capturing each group of images.

14. An image processing method for aligning multiple images including the same subject captured sequentially in a time sequence by a camera unit having an image sensor, the image processing method being characterized in that: The plurality of images are aligned by combining sequential alignment and reference alignment. Sequential alignment is used to align adjacent images, and reference alignment is used to align two or more images other than the reference image with the reference image. The multiple images are grouped for either sequence alignment synthesis or reference alignment synthesis. Based on the amount of framing deviation during the time period of capturing each group of images, each group of images is set as either a reference alignment synthesis of images or a sequence alignment synthesis of images.

15. A computer program product comprising a program that, when executed by a processor, implements the image processing method according to claim 14.

16. A computer-readable storage medium storing a program that, when executed by a processor, implements the image processing method according to claim 14.

Images