Image processing apparatus and method, program, storage medium

Abstract

To provide an image processing device capable of reducing the difference in noise characteristics between a synthesized plurality of images and a single image.SOLUTION: An image processing device includes synthesizing means for generating a first image by synthesizing a plurality of images captured at a sensitivity higher than a target sensitivity, generation means for generating a second image by performing noise reduction processing on the first image, and control means for adjusting at least one of the number of images to be synthesized by the synthesizing means and the characteristics of the noise reduction processing so that the noise characteristics of the second image approaches the noise characteristics of the image captured at the target sensitivity.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] ,

[0002] , , , , , , , , , ,

[0006] , , , , , ,

[0005] , , , , , , ,

[0003] ,

[0001] The present invention relates to an image processing apparatus that synthesizes a plurality of images to reduce noise.

Background Art

[0002] When taking a still image with an imaging device such as a digital camera, it is effective to ensure a sufficient exposure time in order to obtain an image with less noise. However, when the exposure time is lengthened, there is a problem that image blur occurs in the image due to camera movement or subject movement caused by camera shake, and the captured image becomes unclear. ' <00]00011> As a method for dealing with such image blur, electronic image blur correction has been proposed. For example, Patent Document 1 discloses a method of obtaining an image with less image blur and noise by continuously performing a plurality of short exposure time shootings with less image blur, aligning the obtained plurality of images so as to cancel the deviation between the images, and then performing a synthesis process.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the RAW image captured by the imaging device contains a plurality of types of noise with different characteristics, such as dark noise that does not depend on the amount of light from the subject, that is, the luminance of each pixel, and optical shot noise that depends on the luminance of each pixel. Therefore, depending on the type of each noise, for example, the number of synthesized images required to obtain a noise improvement effect equivalent to one step of ISO sensitivity is different.

[0006] More specifically, when the ISO sensitivity changes by one stop, meaning the gain doubles, the dark noise doubles and the optical shot noise increases by √2. Therefore, the number of composite images required to improve noise by one stop of ISO sensitivity is four for dark noise and two for optical shot noise.

[0007] In other words, if the same number of images are combined regardless of the subject's brightness, the noise reduction effect will differ depending on the subject's brightness, resulting in a combined image with different noise characteristics than a single, uncombined image taken under low ISO conditions.

[0008] Thus, if the characteristics of the composite image differ from those of a single image taken under low ISO conditions, the composite RAW image (composite RAW image) must be processed using application software with different characteristics than the single, uncomposite image. As a result, the user experience becomes poor. Furthermore, when processing images within the digital camera, it becomes necessary to maintain optimal settings for both the single image and the composite image.

[0009] The present invention has been made in view of the above-mentioned problems, and its objective is to provide an image processing apparatus that can reduce the difference in noise characteristics between a composite image obtained by combining multiple images and a single image. [Means for solving the problem]

[0010] The image processing apparatus according to the present invention is characterized by comprising: a synthesis means for generating a first image by synthesizing a plurality of images captured at a sensitivity higher than a target sensitivity; a generation means for generating a second image by performing noise reduction processing on the first image; and a control means for adjusting at least one of the number of images synthesized by the synthesis means and the characteristics of the noise reduction processing so that the noise characteristics of the second image approach the noise characteristics of an image captured at the target sensitivity. [Effects of the Invention]

[0011] According to the present invention, it is possible to reduce the difference in noise characteristics between a composite image obtained by combining multiple images and a single image. [Brief explanation of the drawing]

[0012] [Figure 1] A block diagram showing the configuration of a digital camera according to the first embodiment. [Figure 2A] A flowchart illustrating the operation of the first embodiment. [Figure 2B] A flowchart illustrating the operation of the first embodiment. [Figure 2C] A flowchart illustrating the operation of the first embodiment. [Figure 3] A block diagram showing an example configuration of the synthesis processing unit. [Figure 4] A flowchart illustrating the operation of the second embodiment. [Modes for carrying out the invention]

[0013] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0014] (First Embodiment) This embodiment describes an example of generating a noise-reduced image by combining multiple images taken within a digital camera.

[0015] Figure 1 is a block diagram showing the configuration of a digital camera 100 according to a first embodiment of the present invention.

[0016] In FIG. 1, the optical system 101 forms an image of a subject image on the imaging unit 102. The imaging unit 102 includes an imaging device 102a composed of, for example, a CCD or a CMOS sensor. The imaging device 102a photoelectrically converts the optical image formed by the optical system 101 to generate an analog image signal. Further, the imaging unit 102 performs A / D conversion on the analog image signal output from the imaging device 102a to generate digital image data.

[0017] The image processing unit 103 applies various image processes such as white balance adjustment processing, color interpolation processing, reduction / enlargement processing, filtering processing, etc. to the image data generated by the imaging unit 102.

[0018] The RAM 104 is a rewritable volatile memory and is used as a temporary storage area for data output in the operations of each block included in the digital camera 100 and as an image buffer.

[0019] The ROM 105 is a rewritable non-volatile memory and stores, in addition to the operation programs of each block included in the digital camera 100, parameters necessary for the operations of each block, etc. It also stores information such as the noise characteristics of an image taken at a target ISO sensitivity.

[0020] The control unit 106 is, for example, a CPU. It reads the operation programs of each block included in the digital camera 100 from the ROM 105, expands them in the RAM 104, and executes them to control the operations of each block included in the digital camera 100. Also, by the control unit 106 instructing the optical system 101 of the lens driving amount, it is possible to take pictures at different focusing positions.

[0021] The synthesis processing unit 107 performs the synthesis processing of an image, which will be described later, on the image data stored in the RAM 104 to generate a noise-reduced image.

[0022] The recording unit 108 records the image processed by the image processing unit 103 stored in the RAM 104 and the synthesized image processed by the synthesis processing unit 107 as a recorded image.

[0023] The display unit 109 consists of a display device such as a PC monitor, television, or smartphone, and displays images stored in the RAM 104.

[0024] The user input unit 110 inputs instructions from the user to the control unit 106. The internal bus 111 transmits data between blocks according to the instructions of the control unit 106.

[0025] Next, the operation of the digital camera 100 of the first embodiment will be explained using Figures 2A to 2C and Figure 3.

[0026] Figure 2A is a flowchart showing a series of processes for generating a noise-reduced image by combining multiple captured images.

[0027] In step S201, the control unit 106 captures multiple images, in step S202, it combines these images, and in step S203, it performs noise reduction processing (NR processing) on ​​the combined image. Each of these steps will be described in detail below.

[0028] Figure 2B is a flowchart that shows in detail the image acquisition operation in step S201 of Figure 2A.

[0029] In step S211, the control unit 106 determines the shooting conditions for the optical system 101 and the imaging unit 102. The shooting conditions are the shutter speed, aperture value, and ISO sensitivity, which can be automatically determined by the control unit 106 based on the metering results of the digital camera 100, or set arbitrarily by the photographer according to the subject being photographed. Alternatively, the photographer can determine the shutter speed and aperture value, which are related to motion and depth of field expression, while the control unit 106 automatically determines the ISO sensitivity based on the metering results of the digital camera 100.

[0030] In step S212, the control unit 106 determines the number of images (N). This value is set by the control unit 106 depending on the target noise reduction level. In other words, it sets how many stops lower in ISO sensitivity the noise characteristics equivalent to the image captured by the imaging unit 102 should be. For example, if the target is a noise reduction level of 1 stop, the number of images (N=4) that can reduce dark noise by 1 stop is set. This is because dark noise requires more composite images to reduce noise by 1 stop compared to optical shot noise. All N images are assumed to be captured under the same shooting conditions.

[0031] In step S213, the control unit 106 operates the imaging unit 102 based on the shooting conditions set in step S211 and the number of images determined in step S212, and captures multiple images.

[0032] In step S214, the control unit 106 determines whether the number of images taken has reached the number of images (N) set in step S212. If the number of images (N) has not been reached, the control unit 106 repeats steps S213 and S214 until the number of images (N) is reached. If the number of images (N) has been reached, the camera stops taking images.

[0033] Next, the image synthesis process in step S202 will be explained using Figures 2C and 3.

[0034] Figure 3 is a block diagram showing an example of the configuration of the synthesis processing unit 107 in this embodiment. The synthesis processing unit 107 is configured to include a block readout unit 301, an alignment processing unit 302, a noise characteristic setting unit 303, an image synthesis unit 304, and an NR (noise reduction) processing unit 305.

[0035] Figure 2C is a flowchart that shows in detail the image synthesis process (processing by the synthesis processing unit 107) in step S202 of Figure 2A.

[0036] In Figure 2C, in step S221, the control unit 106 uses the block readout unit 103 to select a reference image to be used as the basis for synthesis from the captured images recorded in the RAM 104 and reads it out as a block image of a predetermined size (block division). The block size is, for example, 16 pixels vertically and 16 pixels horizontally. Hereafter, in the synthesis process of this embodiment, processing is performed in units of this block image. When processing of one block is completed, processing of the next adjacent block is performed, and when processing of all blocks is completed, the synthesis process is also completed. The selection of the reference image is arbitrary, but it is preferable to select the image taken at the time the shutter was pressed.

[0037] In step S222, the control unit 106 first inputs a reference image to be combined from the captured images stored in the RAM 104 to the alignment processing unit 302. Then, it performs alignment processing on the block image of the reference image (reference block image) output from the block readout unit 301 with a block image of the same size in the reference image, and outputs a block image (reference block image) that is in the correct relative position. Because there is generally a discrepancy between the reference image and the reference image due to camera shake or local motion, the subject will appear as a multiple image if they are combined as is. Therefore, the position of the subject is aligned between the images before combining them. The alignment processing can be done using known techniques such as template matching.

[0038] In step S223, the control unit 106 uses the noise characteristic calculation unit 303 to read the ISO sensitivity of the captured image (shooting ISO sensitivity) and the noise dispersion value for each brightness level of the ISO sensitivity corresponding to the target noise reduction effect (target ISO sensitivity) from the ROM 105. Then, it calculates the number of images to be combined in the image synthesis unit 304 and the characteristics to be used when performing NR processing in the NR processing unit 305. Specifically, first, the number of images to be combined is calculated by referring to the noise dispersion value of the shooting ISO sensitivity and the noise dispersion value of the target ISO sensitivity for the brightness value (brightness information) of the brightest pixel in the reference block image.

[0039] Here, the number of composite images N_COMP is calculated using equation (1), where Ymax is the brightness value of the highest brightness pixel in the reference block image, Var_O(Ymax) is the variance value of the noise at the ISO sensitivity at brightness value Ymax, and Var_T(Ymax) is the variance value of the noise at the target ISO sensitivity.

[0040] N_COMP=Var_O(Ymax) / Var_T(Ymax) …(1) In equation (1), any decimals are truncated.

[0041] In this embodiment, the number of composite images N_COMP is calculated using the highest brightness pixel in the reference block image. However, if only the value of one pixel is referenced, it may be affected by isolated point noise. Therefore, the number of composite images may be calculated from the brightness value of the high-brightness region instead of from a single pixel. For example, the effect of isolated point noise can be reduced by performing a low-pass filter on the reference block image beforehand and calculating Ymax from the filtered image.

[0042] By calculating the number of composite images in this way, it is possible to determine the number of composite images required to obtain a noise reduction effect equivalent to the target ISO sensitivity in the high-brightness region within the block.

[0043] Next, the NR processing unit 305 calculates a parameter that indicates the strength of the noise reduction effect based on the brightness value of each pixel when performing NR processing.

[0044] Here, if the brightness value of the corresponding pixel in the reference block image is Y, the variance value of the noise at the ISO sensitivity at brightness value Y is Var_O(Y), the variance value of the noise at the target ISO sensitivity is Var_T(Y), and the number of composite images is N_COMP, the noise reduction parameter α is calculated using equation (2).

[0045] α=Var_O(Y) / (Var_T(Y)× N_COMP) …(2) In this way, by calculating the noise reduction parameter α, it is possible to calculate the filter characteristics (how many pixels of filter strength are needed to achieve a noise reduction effect equivalent to the target ISO sensitivity) required for each pixel of the image synthesized by the image synthesis unit 304, relative to the brightness value of each pixel.

[0046] In step S224, the control unit 106 uses the image synthesis unit 304 to synthesize the reference block image output from the block reading unit 301 with the reference block image output from the alignment processing unit 302. Specifically, it averages the N_COMP-1 reference block images pixel by pixel with respect to the reference block image. The synthesized block image (synthesized block image) is output to the NR processing unit 305.

[0047] In this embodiment, simple averaging was performed on each pixel. However, it is also possible to perform transformations such as Fast Fourier Transform and Discrete Cosine Transform on each of the reference block image and the reference block image to convert them into frequency images, and then perform averaging on each frequency component. Alternatively, when performing averaging on each frequency component, the difference between the reference block image and the reference block image may be calculated for each frequency component, and averaging may be performed only on frequency components where the difference is smaller than a predetermined threshold. By adding only the frequency components with small differences in this way, noise is suppressed by averaging at frequencies where noise components are dominant, and frequencies where differences in frequency components occur due to positional shifts or changes in the shape of the subject are not averaged. Therefore, it is possible to mitigate the phenomenon of the subject appearing as multiple images.

[0048] In step S225, the control unit 106 uses the NR processing unit 305 to perform NR processing on the composite block image output from the image synthesis unit 304. Specifically, when the pixel to be processed in the composite block image is the pixel of interest, the pixels surrounding the pixel of interest (for example, the two pixels above, below, left, and right) are sorted in descending order of the difference from the pixel of interest, and then the average of the α parameters calculated by the noise characteristic calculation unit 303 is calculated.

[0049] In this way, by performing a filtering process that calculates the average of the pixel of interest and surrounding pixels based on parameter α, it becomes possible to obtain a noise reduction effect equivalent to the target ISO sensitivity.

[0050] In step S226, the control unit 106 determines whether processing has been completed for all block images. If processing has been completed for all block images, the control unit 106 terminates the image synthesis process. If processing has not been completed for all block pixels, the process proceeds to step S227, where new block pixels are set, and then the processing in steps S222 to S226 is repeated. The above is a detailed description of the synthesis processing unit 107's operation. Once all operations are completed, the output image is generated and stored in the RAM 104.

[0051] In this embodiment, by performing the image synthesis process in this manner, it becomes possible to generate a composite image with little difference in noise characteristics from a single, uncomposite image taken under low ISO sensitivity conditions.

[0052] In this embodiment, the image synthesis process was performed on a newly captured image. However, the user may select an image from the images recorded in the recording unit 108 using the user input unit 110 and perform the image synthesis process. For example, images taken in succession under the same conditions as the image specified by the user are read into the RAM 104 and synthesized according to the flowchart in Figure 2C.

[0053] At this time, the noise characteristic calculation unit 303 calculates the number of ISO sensitivity stops required to reduce dark noise according to the number of images read into the RAM 104, and sets the target ISO sensitivity. For example, if the number of images read into the RAM 104 is in the range of 4 to 15, the target ISO sensitivity is set to one stop lower than the shooting ISO sensitivity.

[0054] By performing this compositing process, it becomes possible to apply the process not only to newly captured images but also to images that have been captured in advance.

[0055] (Second embodiment) In the first embodiment, when capturing images in step S213 of Figure 2B, all images were captured under the same conditions. In contrast, in the second embodiment, instead of capturing all images under the same conditions, some images are captured under different conditions depending on the difference in the number of stops between the shooting ISO sensitivity and the target ISO sensitivity, and the number of images that can be stored in RAM 104.

[0056] Figure 4 is a flowchart showing the image acquisition process in step S201 in this embodiment.

[0057] In step S401, the control unit 106 determines the shooting conditions in the same manner as in step S211.

[0058] In step S402, the control unit 106 determines the number of images (N) to be captured, similar to step S212. At this stage, as in the first embodiment, the number of images (N) is determined assuming that all N images are captured under the same shooting conditions.

[0059] In step S403, the control unit 106 determines whether N images can be stored in the RAM 104. If they can be stored, the control unit 106 proceeds to step S405. If there is no free space in the RAM 104 (image storage unit) and N images cannot be stored, the control unit 106 proceeds to step S404.

[0060] In step S404, the control unit 106 reviews the number of shots and shooting conditions. Specifically, it reviews the shooting conditions based on the difference in the number of stops between the shooting ISO sensitivity determined in step S401 and the target ISO sensitivity, and the number of images that can be stored in RAM 104. Then, it determines the number of shots to be taken without changing the shooting conditions (N) and the number of shots to be taken with changed shooting conditions (M).

[0061] For example, if the difference in ISO sensitivity between the shooting ISO and the target ISO is 2 stops, 16 images would need to be combined to reduce dark noise by 2 stops. However, if the RAM 104 can store, for example, 12 images, the shooting conditions for some of the images are changed to double the brightness, such as by doubling the shutter speed. By multiplying the images with these modified shooting conditions (high-exposure images) by a gain of 1 / 2 during the image compositing process, it becomes possible to reduce the number of images (M) required to reduce dark noise to 4. In addition, the number of images (N) to be taken with the shooting conditions determined in step S401 is set to the remaining images that can be stored in RAM 104, in this case 8 images.

[0062] In step S405, the control unit 106 operates the imaging unit 102 to capture an image using the shooting conditions set in step S401 or the shooting conditions revised in step S404.

[0063] In step S406, the control unit 106 determines whether the number of images captured has reached the number of images set in step S402, or the number of images revised in step S404. If it has reached that number, the control unit 106 terminates image capture. If the number of images captured is less than the set number, the process returns to step S405, and steps S405 and S406 are repeated.

[0064] Regarding the image synthesis process in step S202 in the second embodiment, the flowchart in Figure 2C is followed, similar to the first embodiment. However, when performing the alignment process in step S222, a gain of 1 / 2 is applied to high-exposure images before performing the alignment process.

[0065] Furthermore, in step S223, when calculating the number of images to be combined by the image synthesis unit 304 and the characteristics of the NR processing unit 305, if the brightness value Ymax of the brightest pixel in the reference block image is less than half of the maximum brightness, a high-exposure image is selected.

[0066] If the brightness value Ymax of the brightest pixel in the reference block image is 1 / 2 or greater than the maximum brightness value, the same processing as in the first embodiment is performed, but since N_COMP will be greater than the number of images taken (N), in that case the processing is performed with N_COMP=N.

[0067] By performing the composite processing as described above, it becomes possible to perform noise reduction processing within the range of the number of images that can be stored in RAM104, even when there is a large difference between the shooting ISO sensitivity and the target ISO sensitivity.

[0068] The disclosures herein include the following image processing devices, methods, programs, and storage media.

[0069] (Item 1) A synthesis means that generates a first image by combining multiple images captured at a sensitivity higher than the target sensitivity, A generation means that performs noise reduction processing on the first image to generate a second image, Control means for adjusting at least one of the number of images synthesized by the synthesis means and the characteristics of the noise reduction processing, such that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity. An image processing apparatus characterized by comprising:

[0070] (Item 2) The image processing apparatus according to item 1, characterized in that the plurality of images are images captured by an imaging means.

[0071] (Item 3 The image processing apparatus according to item 1, characterized in that the plurality of images are images stored in a storage means.

[0072] (Item 4) The image processing apparatus according to any one of items 1 to 3, further comprising a storage means for storing the noise characteristics of an image captured with the aforementioned target sensitivity.

[0073] (Item 5) The image processing apparatus according to any one of items 1 to 4, characterized in that the synthesis means aligns and synthesizes the plurality of images.

[0074] (Item 6) The image processing apparatus according to any one of items 1 to 5, characterized in that the synthesis means divides the image into blocks consisting of a plurality of pixels and determines the number of plurality of images to be synthesized based on the brightness information of the high-brightness region within the block.

[0075] (Item 7) The image processing apparatus according to item 6, characterized in that the synthesis means changes the number of images to be synthesized and the noise reduction characteristics for each block based on the target sensitivity.

[0076] (Item 8) The image processing apparatus according to any one of items 1 to 5, characterized in that the generation means changes the characteristics of the noise reduction process based on the brightness information of each pixel of the first image.

[0077] (Item 9) The image processing apparatus according to any one of items 1 to 8, characterized in that the synthesis means changes the exposure of the image to be captured when the number of images to be synthesized exceeds the capacity of the image storage unit.

[0078] (Item 10) The image processing apparatus according to any one of items 1 to 9, characterized in that the aforementioned sensitivity is ISO sensitivity.

[0079] (Item 11) A synthesis process that generates a first image by combining multiple images captured at a sensitivity higher than the target sensitivity, A generation step involves performing noise reduction processing on the first image to generate a second image, A control step that adjusts at least one of the number of images synthesized in the synthesis step and the characteristics of the noise reduction process so that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity, An image processing method characterized by having the following features.

[0080] (Item 12) A program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 10.

[0081] (Item 13) A computer-readable storage medium storing a program for causing the computer to function as one of the means of the image processing apparatus described in any one of items 1 to 10.

[0082] (Other embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0083] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0084] 100: Digital camera, 101: Optical system, 102: Imaging unit, 102a: Image sensor, 103: Image processing unit, 104: RAM, 105: ROM, 106: Control unit, 107: Synthesis processing unit, 108: Recording unit, 109: Display unit, 110: User input unit, 111: Internal bus, 301: Block readout unit, 302: Alignment processing unit, 303: Noise characteristic calculation unit, 304: Image synthesis unit, 305: NR processing unit

Claims

1. A synthesis means that generates a first image by combining multiple images captured at a sensitivity higher than the target sensitivity, A generation means that performs noise reduction processing on the first image to generate a second image, Control means for adjusting at least one of the number of images synthesized by the synthesis means and the characteristics of the noise reduction processing, such that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity. An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, characterized in that the plurality of images are images captured by the imaging means.

3. The image processing apparatus according to claim 1, characterized in that the plurality of images are images stored in the storage means.

4. The image processing apparatus according to claim 1, further comprising a storage means for storing the noise characteristics of an image captured with the aforementioned target sensitivity.

5. The image processing apparatus according to claim 1, characterized in that the synthesis means aligns and synthesizes the plurality of images.

6. The image processing apparatus according to claim 1, wherein the synthesis means divides the image into blocks consisting of a plurality of pixels and determines the number of plurality of images to be synthesized based on the brightness information of the high-brightness region within the block.

7. The image processing apparatus according to claim 6, characterized in that the synthesis means changes the number of images to be synthesized and the noise reduction characteristics for each block based on the target sensitivity.

8. The image processing apparatus according to claim 1, characterized in that the generation means changes the characteristics of the noise reduction processing based on the brightness information of each pixel of the first image.

9. The image processing apparatus according to claim 1, characterized in that the synthesis means changes the exposure of the image to be captured when the number of images to be synthesized exceeds the capacity of the image storage unit.

10. The image processing apparatus according to claim 1, characterized in that the aforementioned sensitivity is ISO sensitivity.

11. A synthesis process that generates a first image by combining multiple images captured at a sensitivity higher than the target sensitivity, A generation step involves performing noise reduction processing on the first image to generate a second image, A control step that adjusts at least one of the number of images synthesized in the synthesis step and the characteristics of the noise reduction processing so that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity, An image processing method characterized by having the following features.

12. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a program for causing the computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 10.

Images