Image processing apparatus and method, imaging apparatus, program, storage medium

Abstract

To reduce noise while suppressing deterioration in resolution in a case where a plurality of images are combined to obtain one combined image.SOLUTION: Included are: first acquisition means configured to acquire a plurality of RAW images obtained by continuously capturing a subject; combining means configured to combine the plurality of RAW images to generate a combined RAW image; second acquisition means configured to acquire first information regarding a noise signal level in any of the plurality of RAW images not combined by the combining means; third acquisition means configured to acquire second information regarding a noise signal level in the combined RAW image after the combine by the combining means; and determination means configured to determine a development parameter for developing the combined RAW image based on capturing sensitivity of a capturing element when the plurality of RAW images are captured and the first information and the second information.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus that synthesizes a plurality of RAW images to generate a single synthesized image.

Background Art

[0002] When taking a long-exposure photograph, it is generally known that image blur occurs due to shaking of the imaging device such as camera shake. As a method for correcting this image blur, there is a technique of taking a plurality of short-exposure images by dividing the total exposure time, and synthesizing them to suppress the image blur.

[0003] When the total exposure time is divided, the exposure time per shot becomes shorter. Therefore, when trying to obtain an image with appropriate brightness, it is necessary to increase the sensitivity of the imaging device for shooting. However, when the sensitivity of the imaging device is increased, the noise of the imaging element is amplified, and there is a drawback that the quality of the image deteriorates.

[0004] When developing a captured RAW image, a technique is known in which appropriate development parameters are applied to the noise increased by setting the imaging device to high sensitivity to reduce the noise. In this case, when the amount of noise increases by increasing the sensitivity of the imaging device, it is necessary to use stronger noise reduction parameters. However, generally, when using strong noise reduction parameters, high-frequency components other than noise, such as edge portions in the image, are also removed, and there is a problem that the resolution of the image is impaired.

[0005] On the other hand, as a method for reducing noise, there is also a method called additive average synthesis in which noise included in an image is made less noticeable by synthesizing and averaging a plurality of images. When the developed image is the target of additive average synthesis, since noise reduction processing is applied during development in the developed image, the impaired resolution cannot be restored. Therefore, it is desirable to perform additive average synthesis on the RAW image, reduce the noise, and then determine the noise reduction parameters according to the amount of noise reduction and develop the image. However, in this case, there is a problem that the amount of noise reduction is not known in advance.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] Patent Document 1 discloses a technique for determining the intensity of noise reduction based on the shooting sensitivity when a RAW image is acquired and a synthesis coefficient indicating the degree to which the RAW image is used for synthesis. However, in Patent Document 1, the shooting sensitivity and the synthesis coefficient are determined before shooting, and the noise reduction parameter cannot be determined according to the amount of noise reduction by actual additive average synthesis.

[0008] Patent Document 2 discloses a technique for controlling a noise reduction parameter based on the number of additions for image data generated by adding a plurality of images. However, also in this technique, the noise reduction parameter cannot be determined according to the amount of noise reduction by additive average synthesis.

[0009] The present invention has been made in view of the above-described problems, and an object thereof is to provide an image processing apparatus capable of reducing noise while suppressing a decrease in resolution when obtaining one synthesized image by synthesizing a plurality of images.

Means for Solving the Problems

[0010] The image processing apparatus according to the present invention includes: a first acquisition means for acquiring a plurality of RAW images obtained by continuously imaging a subject; a synthesis means for synthesizing the plurality of RAW images to generate a synthesized RAW image; a second acquisition means for acquiring first information regarding a noise signal level in any one of the plurality of RAW images that have not been synthesized by the synthesis means; a third acquisition means for acquiring second information regarding a noise signal level in the synthesized RAW image after synthesis by the synthesis means; the imaging sensitivity of the imaging device when the plurality of RAW images are imaged, and the first information and the second information Difference and a determination means for determining development parameters for developing the synthesized RAW image based on the above, and is characterized by comprising the same.

Advantages of the Invention

[0011] According to the present invention, when synthesizing a plurality of images to obtain one synthesized image, it is possible to reduce noise while suppressing a decrease in resolution.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] FIG. 1 is a block diagram showing the configuration of an imaging device 100 which is an embodiment of an image processing apparatus of the present invention.

[0015] In FIG. 1, light rays incident on the optical lens 101 pass through the aperture 102 and are imaged as a subject image on the imaging element 103. The imaging element 103 photoelectrically converts the subject image and outputs an analog image signal. The A / D conversion unit 104 converts this analog image signal into a digital signal and outputs digital image data. A color filter having a Bayer array configuration in which pixels of R (red), G1 (green), G2 (green), and B (blue) are regularly arranged is arranged on the imaging element 103. The digital image data (RAW image data) output from the A / D conversion unit 104 is once stored in the memory 113.

[0016] FIG. 2 is a diagram schematically showing the pixel arrangement inside the imaging element 103. As shown in FIG. 2, the imaging element 103 includes an effective pixel region 201 where a photodiode, which is a photoelectric conversion element, is irradiated with light, and an optical black region (hereinafter referred to as an OB region) 202 where the photodiode is shielded from light by an aluminum thin film or the like. That is, the effective pixel region 201 is a non-shielded region, and the OB region 202 is a shielded region (light-shielded region). The OB integration unit 105 in FIG. 1 integrates the pixel values of the OB region 202 for each of R, G1, G2, and B in the Bayer array and outputs the average value of the signals in the OB region. Using the output value of the OB integration unit 105 as the dark level (black level), OB clamping is performed by the OB clamp unit 106. By this OB clamping process, problems such as black floating and color shift can be prevented.

[0017] Returning to FIG. 1, the memory 113 stores RAW image data, image data processed by the signal processing unit 110, and the like. Also stored are control programs and the like executed by the control unit 112. The shading correction unit 107 corrects the shading caused by the aberration of the optical lens 101 and the characteristics of the imaging device 103 for the RAW image data, which is digital image data from the A / D conversion unit 104, by performing correction of the luminance level within the screen. The WB (white balance) processing unit 108 performs white balance processing to adjust the white reference within the screen to white for the output image data from the shading correction unit 107. In the present embodiment, the shading correction is a correction that multiplies each pixel by a gain according to the two-dimensional coordinates (positions) within the imaging device 103, and the white balance processing is a process that multiplies different gains for each of R, G1, G2, and B in the Bayer array.

[0018] The combining unit 109 performs an operation and combination of N RAW images taken by split exposure by an addition averaging method. Let the luminance value of each image before combination be I_i(x, y) (i = 1 to N, where x and y represent coordinates within the screen), and the luminance value of the image after combining these N images be I(x, y). At this time, in the operation by the addition averaging method, the luminance value I(x, y) of the combined image is represented by the following equation.

[0019] I(x, y)=(I_1(x, y)+I_2(x, y)+ ···+I_N(x, y)) / N Then, the addition averaging process of the luminance values of the N images for each pixel is the combined image data.

[0020] The signal processing unit 110 performs development processing including color matrix processing, gamma processing, noise reduction processing, etc. on the RAW image data and the RAW image data generated by combining by the combining unit 109. The recording unit 111 records RAW image data, combined RAW image data combined by the combining unit 109, image data developed by the signal processing unit 110, and the like. The control unit 112 performs overall control of the imaging device 100.

[0021] FIG. 3 is a flowchart showing the operation of split exposure shooting in the present embodiment. The operation of this flowchart is realized by the control unit 112 executing a program stored in the memory 113.

[0022] In step S301, the number of split exposure shots is set by the operation of the user. In the present embodiment, the total exposure time required to obtain proper exposure is divided, a plurality of short-exposure images are acquired, and then synthesized to obtain one synthesized image. Thereby, it is possible to suppress blurring of the image due to shaking of the imaging device 100 such as camera shake in long-exposure shooting.

[0023] In step S302, the user gives a shooting instruction by pressing the shutter switch.

[0024] Hereinafter, in steps S303 to S306, the control unit 112 performs continuous shooting until the number of shots set in step S301 is completed, and repeats the storage of the captured RAW image while calculating the dispersion value of the noise signal level in the OB region of the captured RAW image.

[0025] More specifically, in step S303, the control unit 112 captures a RAW image in response to the shooting instruction in step S302 and stores it in the memory 113.

[0026] In step S304, the control unit 112 calculates the dark level of the captured RAW image (black level calculation) using the OB integration unit 105.

[0027] In step S305, the control unit 112 calculates the dispersion value of the signal (first information regarding the noise signal level) from the signal in the OB region of the captured RAW image.

[0028] Here, FIG. 4 is a diagram showing the signal levels of the effective pixel region and the OB region. Signal 401 indicates the signal obtained when signal accumulation is performed with high sensitivity in the shaded OB region 202 of the imaging device 103. Signal 402 indicates the signal obtained by extracting a partial region of signal 401.

[0029] Since signal 401 is not affected by the light irradiated on the imaging device 103, the components of the random noise of the imaging device itself appear. When signal accumulation is performed with high sensitivity, granular noise appears as in signal 402. Graph 403 shows the distribution of the signal levels in a specific region in signal 402 and the number of pixels corresponding to the signal levels. In step S305, with reference to a specific region such as signal 402, a variance value (hereinafter, also referred to as the variance value of the noise signal level or the noise variance value) is calculated from the distribution of the signal levels and the number of pixels for each signal level.

[0030] Signal 402 refers to a part of the OB region arranged vertically, but a part of the OB region arranged horizontally may also be referred to. Further, not limited to a part of the OB region, the entire region may be referred to.

[0031] In step S306, the control unit 112 executes OB clamping based on the dark level calculated in step S304 using the OB clamp unit 106.

[0032] In step S307, the control unit 112 determines whether the number of captured images has reached the number of captured images set in step S301. If the control unit 112 has reached the set number of captured images, the process proceeds to step S308. Otherwise, the process returns to step S303, and steps S303 to S307 are repeated until the set number of captured images is reached.

[0033] In step S308, the control unit 112 uses the combining unit 109 to additively average and combine the N RAW images captured in steps S303 to S307. Considering the case where the signal value becomes zero or less due to the influence of noise in the negative direction by the combination, a specific dark level may be added to the output of the captured RAW image and then output. When adding a fixed dark level, after the combination in step S308, the fixed dark level is subtracted from the combined RAW image.

[0034] In step S309, the control unit 112 calculates the combined dark level from the signals in the OB region of the combined RAW image.

[0035] In step S310, the control unit 112 calculates the variance value (variance value of the noise signal level, noise variance value, second information regarding the noise signal level) of the signal levels in the OB region of the combined RAW image.

[0036] Here, FIG. 5 is a diagram showing the signal levels of the effective pixel region and the OB region of the combined RAW image. Signal 501 indicates a signal in which additive averaging is performed in the shielded OB region 202 of the imaging device 103 and the noise is reduced. Signal 502 indicates a signal obtained by extracting a partial region of signal 501. In signal 501, the random noise of the RAW data captured in step S303 is reduced by additive averaging, so the amount of granular noise is decreased. Graph 503 shows the distribution of the signal levels of a specific region in signal 502 and the number of pixels corresponding to the signal levels.

[0037] In the OB region of the combined RAW image, since the random noise is reduced like signal 502 compared to the OB region of a single RAW image like signal 402, the distribution of the variance becomes smaller than graph 403.

[0038] In step S311, the control unit 112 calculates the difference between the noise variance value of any one of the RAW images calculated in step S305 and the noise variance value of the synthesized RAW image calculated in step S310. If the difference in the variance values is small, it can be understood that the noise reduction effect by addition averaging is small for the RAW before synthesis, and if the difference in the variance values is large, it can be understood that the noise reduction effect is large.

[0039] In this embodiment, the noise variance value of any one of the RAW images calculated in step S305 is used, but this may be the variance value of the first or last one, or the variance value of any one during shooting excluding the first and the last.

[0040] In step S312, the control unit 112 determines the development parameters to be applied from the difference in the noise variance values calculated in step S311.

[0041] Here, FIG. 6 is a diagram showing an example of determining the parameters to be applied from the set imaging sensitivity and the difference in the noise variance values calculated in step S311.

[0042] The ISO sensitivity is a reference value for amplifying the signal of the imaging element in a digital camera, and the higher the ISO sensitivity, the more likely noise is to occur. 601 indicates the set value of the ISO sensitivity set at the time of shooting, and 602 shows an example of the development parameters to be applied in step S311. In this way, the smaller the noise variance value, the more the development parameters corresponding to an ISO sensitivity close to the set ISO sensitivity are applied, and the larger the noise variance value, the more the development parameters corresponding to an ISO sensitivity smaller than the set ISO sensitivity are applied. In this case, the parameters to be applied may be parameters related to noise reduction processing or sharpness parameters applied to enhance the edge portion.

[0043] In step S313, the control unit 112 performs OB clamp processing using the OB clamp unit 106 and using the synthesized dark level calculated in step S309.

[0044] In step S314, the control unit 112 performs WB processing on the combined RAW image after OB clamping using the WB processing unit 108.

[0045] In step S315, the control unit 112 performs development processing on the combined RAW image that has undergone WB processing in step S314. In performing the development processing, the development parameters determined in step S311 are applied for the development processing.

[0046] In step S316, the control unit 112 records the image developed in step S315 on an external storage medium such as an SD card. As information to be saved at this time, not only the developed image but also the combined RAW image before OB clamping in step S313 may be recorded. Further, the dark level calculated in step S309 and the noise variance value calculated in step S311 may also be recorded so that the processing in steps S312 to S315 is performed by an image processing device other than the imaging device.

[0047] As described above, in the present embodiment, when developing a RAW image in which noise is reduced by addition averaging, the noise reduction effect by addition averaging is estimated from the difference between the noise variance value of one RAW image that has not been addition averaged and the noise variance value of the RAW image in which noise is reduced by addition averaging. Then, development parameters suitable for the noise reduction effect are determined and applied to the development. Thereby, it is possible to suppress a decrease in the resolution feeling of the image after development without applying development parameters having a noise reduction effect more than necessary to the addition-averaged RAW image.

[0048] Note that the method of determining the development parameters based on the information on the noise signal level of the uncomposed RAW image and the information on the noise signal level of the composed RAW image is not limited to the method using the difference in variance values. For example, the difference in the signal level with the largest frequency in the distribution of the number of pixels for each signal level of the noise signal level may be used, or the difference in the average value of the noise signal level may be used. Further, in the present embodiment, N captured RAW images are composed, but a part of the captured RAW images may be excluded from the composition target so as not to be composed. In that case, the development parameters may be determined based on the information on the noise signal level of the RAW image that is not the composition target.

[0049] Also, in the above-described embodiment, a digital camera for personal use has been described. However, the present invention can also be applied to other devices such as portable devices, smartphones, or network cameras connected to a server, as long as the device is equipped with a RAW composition function and a RAW development function. Alternatively, a part of the above-described processing may be performed by a portable device, a smartphone, or a network camera connected to a server.

[0050] The disclosure of this specification includes the following image processing apparatus, method, program, and storage medium.

[0051] (Item 1) A first acquisition means for acquiring a plurality of RAW images obtained by continuously imaging a subject; A composition means for composing the plurality of RAW images to generate a composed RAW image; A second acquisition means for acquiring first information regarding a noise signal level in any one of the plurality of RAW images not composed by the composition means; A third acquisition means for acquiring second information regarding a noise signal level in the composed RAW image after composition by the composition means; A determination means for determining development parameters for developing the composed RAW image based on the imaging sensitivity of the imaging device when the plurality of RAW images are captured, the first information, and the second information. An image processing apparatus characterized by comprising

[0052] (Item 2) The image processing apparatus according to item 1, wherein the combining means generates the combined RAW image by adding and averaging the plurality of RAW images.

[0053] (Item 3) The imaging device has an effective pixel region and a light-shielded region that is shielded from light, The image processing apparatus according to item 1 or 2, wherein the second acquisition means and the third acquisition means acquire the first information and the second information based on signals obtained in the light-shielded region.

[0054] (Item 4) The image processing apparatus according to item 3, wherein the second acquisition means acquires the first information based on a signal in the light-shielded region in any one of the plurality of RAW images, and the third acquisition means acquires the second information based on a signal obtained by combining signals in the light-shielded region in the plurality of RAW images.

[0055] (Item 5) The second acquisition means acquires, as the first information, a first variance value of the noise signal level based on the distribution of the noise signal levels in any one of the plurality of RAW images, The image processing apparatus according to any one of items 1 to 4, wherein the third acquisition means acquires, as the second information, a second variance value of the noise signal level based on the distribution of the noise signal levels in the combined RAW image.

[0056] (Item 6) The image processing apparatus according to item 5, wherein the determination means determines the development parameter based on the imaging sensitivity and the difference between the first variance value and the second variance value.

[0057] (Item 7) The determination means determines the development parameter to a value corresponding to a sensitivity closer to the imaging sensitivity as the difference between the first variance value and the second variance value is smaller, and determines the development parameter to a value corresponding to a sensitivity farther from the imaging sensitivity as the difference between the first variance value and the second variance value is larger. The image processing apparatus according to item 6, characterized in that.

[0058] (Item 8) The determination means determines the development parameter to a value corresponding to a sensitivity lower than the imaging sensitivity as the difference between the first variance value and the second variance value is larger. The image processing apparatus according to item 7, characterized in that.

[0059] (Item 9) The second acquisition means acquires the first information based on the first one of the plurality of RAW images. The image processing apparatus according to item 5, characterized in that.

[0060] (Item 10) The second acquisition means acquires the first information based on the last one of the plurality of RAW images. The image processing apparatus according to item 5, characterized in that.

[0061] (Item 11) The second acquisition means acquires the first information based on any one of the plurality of RAW images before the synthesis by the synthesis means. The image processing apparatus according to any one of items 1 to 10, characterized in that.

[0062] (Item 12) The second acquisition means acquires the first information based on any one of the plurality of RAW images that are not the target of the synthesis by the synthesis means. The image processing apparatus according to any one of items 1 to 10, characterized in that.

[0063] (Item 13) The determination means determines, as the development parameter, a parameter for noise reduction processing. The image processing apparatus according to any one of items 1 to 12, characterized in that.

[0064] (Item 14) The determination means determines a sharpness parameter as the development parameter, and the image processing apparatus according to any one of claims 1 to 13 is characterized in that.

[0065] (Item 15) An image processing apparatus according to any one of Items 1 to 14, the imaging element, and an imaging apparatus characterized by comprising.

[0066] (Item 16) A first acquisition step of acquiring a plurality of RAW images obtained by continuously imaging a subject; A synthesis step of synthesizing the plurality of RAW images to generate a synthesized RAW image; A second acquisition step of acquiring first information regarding a noise signal level in any one of the plurality of RAW images that have not been synthesized; A third acquisition step of acquiring second information regarding a noise signal level in the synthesized RAW image after synthesis by the synthesis step; A determination step of determining development parameters for developing the synthesized RAW image based on the imaging sensitivity of the imaging element when the plurality of RAW images are imaged, the first information, and the second information; and an image processing method characterized by comprising.

[0067] (Item 17) A program for causing a computer to execute each step of the image processing method according to Item 16.

[0068] (Item 18) A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to Item 16.

[0069] (Other Embodiments) Further, the present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0070] The invention is not limited to the above embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

Explanation of Reference Numerals

[0071] 100: Imaging device, 101: Optical lens, 102: Diaphragm, 103: Image sensor, 104: A / D conversion unit, 105: OB integration unit, 106: OB clamp unit, 107: Shading correction unit, 108: WB processing unit, 109: Synthesis unit, 110: Signal processing unit, 111: Recording unit, 112: Control unit, 113: Memory

Claims

1. A first acquisition means for acquiring a plurality of RAW images obtained by continuously imaging a subject; A synthesis means for synthesizing the plurality of RAW images to generate a synthesized RAW image; A second acquisition means for acquiring first information regarding a noise signal level in any one of the plurality of RAW images not synthesized by the synthesis means; A third acquisition means for acquiring second information regarding a noise signal level in the synthesized RAW image after synthesis by the synthesis means; A determination means for determining development parameters for developing the synthesized RAW image based on the imaging sensitivity of the image sensor when the plurality of RAW images are imaged and the difference between the first information and the second information; An image processing apparatus comprising the same.

2. The image processing apparatus according to claim 1, wherein the synthesis means generates the synthesized RAW image by adding and averaging the plurality of RAW images.

3. The image sensor has an effective pixel region and a light-shielded region, The second acquisition means and the third acquisition means acquire the first information and the second information based on signals obtained in the light-shielded region. The image processing apparatus according to claim 1.

4. The second acquisition means acquires the first information based on a signal in the light-shielded region in any one of the plurality of RAW images, and the third acquisition means synthesizes signals in the light-shielded region in the plurality of RAW images. The image processing apparatus according to claim 3, wherein the second information is acquired based on the obtained signal.

5. The second acquisition means acquires, as the first information, a first variance value of the noise signal level based on the distribution of the noise signal level in any one of the plurality of RAW images, The third acquisition means acquires, as the second information, a second variance value of the noise signal level based on the distribution of the noise signal level in the synthesized RAW image. The image processing apparatus according to claim 1.

6. The determination means determines the development parameters to be values corresponding to a sensitivity closer to the imaging sensitivity as the difference between the first variance value and the second variance value is smaller, and the difference between the first variance value and the second variance value. The image processing apparatus according to claim 5, wherein the development parameters are determined to be values corresponding to a sensitivity farther from the imaging sensitivity as the value is larger.

7. The image processing apparatus according to claim 6, wherein the determination means determines the development parameter to a value corresponding to a sensitivity lower than the imaging sensitivity as the difference between the first variance value and the second variance value is larger.

8. The image processing apparatus according to claim 5, wherein the second acquisition means acquires the first information based on the first one of the plurality of RAW images.

9. The image processing apparatus according to claim 5, wherein the second acquisition means acquires the first information based on the last one of the plurality of RAW images.

10. The image processing apparatus according to claim 1, wherein the second acquisition means acquires the first information based on any one of the plurality of RAW images before synthesis by the synthesis means.

11. The image processing apparatus according to claim 1, wherein the second acquisition means acquires the first information based on any one of the plurality of RAW images that is not a synthesis target by the synthesis means.

12. The image processing apparatus according to claim 1, wherein the determination means determines a parameter of noise reduction processing as the development parameter.

13. The image processing apparatus according to claim 1, wherein the determination means determines a sharpness parameter as the development parameter.

14. An imaging apparatus, comprising: the image processing apparatus according to any one of claims 1 to 13; and the imaging element. characterized by comprising.

15. A first acquisition step of acquiring a plurality of RAW images obtained by continuously imaging a subject; A synthesis step of synthesizing the plurality of RAW images to generate a synthesized RAW image; A second acquisition step of acquiring first information regarding a noise signal level in any one of the plurality of RAW images that have not been synthesized; A third acquisition step of acquiring second information regarding a noise signal level in the synthesized RAW image after synthesis by the synthesis step; A determination step of determining a development parameter for developing the synthesized RAW image based on an imaging sensitivity of the imaging element when the plurality of RAW images are imaged and a difference between the first information and the second information; characterized by having.

16. A program for causing a computer to execute each step of the image processing method according to claim 15.

17. A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 15.

Images