Image processing device, control method thereof and program

Abstract

To generate a composite image in which a noise amount is suppressed from two images with an exposure difference.SOLUTION: An image processing device for processing first and second image data from an imaging unit which can output the first and second image data by amplifying a signal obtained with photoelectric conversion by different gains comprises: an acquisition unit which acquires an exposure difference of the first and second image data; a correction unit which decides a correction amount that suppresses a noise difference of the first and second image data on the basis of the exposure difference acquired by the acquisition unit and corrects at least one of the first and second image data on the basis of the decided correction amount; and a generation unit which generates a composite image by combining the first and second image data after correction by the correction unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for synthesizing two images having an exposure difference.

Background Art

[0002] Conventionally, a high dynamic range synthesis processing technique for synthesizing a plurality of images taken with different exposure amounts has been known. According to this technique, an image without overexposure or underexposure can be obtained by connecting signals of appropriate exposures of each image. The weighted addition ratio when synthesizing a plurality of images with different exposures is often determined based on the luminance value. For example, an under-image that is not overexposed in the bright part is used 100%, an over-image that is not underexposed in the dark part is used 100%, and an appropriate image is used 100% in the vicinity of the appropriate exposure in the central part of the luminance. In the intermediate region between the dark part and the range of the appropriate exposure part, the usage rates are determined for the over-image and the appropriate image and synthesized. The same applies to the intermediate region such as the range between the appropriate exposure part and the bright part. Thus, a Mix table in which the usage rate of each image is determined by luminance is held in advance and synthesis is performed using it.

[0003] In recent years, due to improvements in sensor performance, etc., an image sensor (Dual Gain Output: DGO) having two column circuits for an output signal from a unit pixel and having different gains for amplification parts in the column circuits and capable of outputting images with different gains has been used. This image sensor can output two images (High gain and Low gain images) with different gains by a single exposure. The synthesis of two images by DGO has the merit that alignment processing of the two images is unnecessary and good synthesis can be achieved even for a moving subject. Therefore, DGO is suitable for HDR (High Dynamic Range) synthesis for obtaining an image with an expanded dynamic range.

[0004] Image sensors have a Floating Diffusion (FD) capacitor that stores electric charge. The size of the charge capacity that this FD can store changes depending on the settings. Increasing the capacity allows it to handle more light, so it is used with a larger capacity at low sensitivity. However, increasing the capacity increases noise, which affects the composite image. In response to this, Patent Document 1 discloses a method for reducing noise when compositing RAW image data, according to the ISO sensitivity and compositing ratio of each image. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2010-124412 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, the prior art disclosed in Patent Document 1 above calculates the synthesis coefficient based on the proportion of images used during synthesis. In other words, Patent Document 1 basically deals with images of the same exposure but different sensitivities, and makes no mention of noise reduction that occurs when synthesizing images with different brightness levels.

[0007] While DGO allows for the synthesis of RAW images and the application of known image processing techniques due to the advantages mentioned above, DGO exposure introduces noise due to exposure differences when generating high-gain and low-gain images. This noise worsens as the exposure difference increases, affecting the image synthesis.

[0008] This invention has been made in view of the above problems, and aims to provide a high dynamic range image synthesis processing technique that reduces noise in image synthesis when exposures differ significantly. [Means for solving the problem]

[0009] To solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, An image processing apparatus that processes first and second image data from an imaging means capable of outputting first and second image data by amplifying signals obtained by photoelectric conversion with different gains, An acquisition means for acquiring the exposure difference of the first and second image data, A correction means that determines a correction amount to suppress the noise difference between the first and second image data based on the exposure difference acquired by the acquisition means, and corrects at least one of the first and second image data based on the determined correction amount, The system includes a generation means that combines the first and second image data after correction by the correction means to generate a composite image. [Effects of the Invention]

[0010] According to the present invention, a composite image with suppressed noise can be generated from two images with different exposure levels. [Brief explanation of the drawing]

[0011] [Figure 1] Block diagram of the imaging device according to the embodiment. [Figure 2] A block diagram showing the internal structure of a solid-state image sensor. [Figure 3] Block diagram of the image synthesis unit in the first embodiment. [Figure 4] A diagram showing the relationship between the input light intensity and the output code after AD conversion in the first embodiment. [Figure 5] This figure shows an example of a table that defines the relationship between exposure difference and the amount of noise correction. [Figure 6] A flowchart illustrating the composite image generation process in the second embodiment. [Modes for carrying out the invention]

[0012] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0013] [First Embodiment] Figure 1 is a block diagram showing the image processing device in the first implementation applied to an imaging device such as a digital camera.

[0014] The imaging device comprises an optical lens 101, an image sensor 102, an image acquisition unit 103, an image synthesis unit 104, a signal processing unit 105, an exposure control unit 107, an image sensor control unit 108, a control unit 150, and a UI 160. The control unit 150 is responsible for controlling the entire device and consists of a processor, a ROM that stores programs and various parameters executed by the processor, and RAM used as a work area. The UI 160 consists of various buttons and keys, such as a shutter button, a display unit, and a touch panel provided on the front of the display unit, and functions as a user interface between the user and the imaging device.

[0015] The optical lens 101 captures light from the subject and forms an image of the subject on the imaging surface of the image sensor 102. The image sensor 102 converts the optical image formed on the imaging surface into an electrical signal and outputs it. Typical examples include CCD image sensors (Charge Coupled Devices) and CMOS image sensors. Some image sensors directly output analog video signals, while others perform AD (analog-to-digital) conversion processing internally and output digital data, such as LVDS (Low Voltage Differential Signaling). Furthermore, the image sensor 102 in this embodiment is capable of outputting two image data with different gains by amplifying the electrical signal obtained in a single exposure with two different gains.

[0016] Here, the circuit configuration of the image pickup device 102 used in the embodiment is shown in FIG. 2.

[0017] The timing pulse control unit 201 controls the operation of the image pickup device 102 by supplying an operation clock signal CLK to each block of the image pickup device 102 and supplying a timing signal to each block.

[0018] The vertical scanning circuit 202 performs timing control for sequentially reading out the voltage represented by the pixel signals accumulated in the pixel unit 203 within one frame. Generally, the video signal is sequentially read out row by row from the upper row to the lower row within one frame.

[0019] The pixel unit 203 has a plurality of photoelectric conversion elements arranged in a matrix. Each photoelectric conversion element generates and outputs an electrical signal corresponding to the incident light amount. In this embodiment, the pixel unit 203 converts the captured light into electric charges and accumulates the electric charges in a capacitor FD (Floating Diffusion). The size of the capacitance of the FD can be changed to large or small, and by changing the capacitance according to the ISO sensitivity, the SN is improved. Basically, at low ISO sensitivity, the capacitance is set to large, and at high ISO sensitivity, the capacitance is set to small and used. When outputting two images with different gains described later, the capacitances for accumulating electric charges with the two gains are common. Also, the size of the capacitance is not limited to two types of large and small, and a setting of three or more steps may be possible.

[0020] The column AMP 204 is used to electrically amplify the signal read from the pixel section 203. By amplifying the signal with the column AMP 204, the signal level of the pixels is amplified against the noise generated by the subsequent column ADC 205, effectively improving the signal-to-noise ratio (SNR). Furthermore, the structure allows the gain of the column AMP 204 to be changed from the timing pulse control unit 201. In this image sensor 102, for HDR (High Dynamic Range) image generation, the column AMP section 204 has two input memories, making it possible to output two different gains by changing the column AMP gain. Having two input memories allows two gains to be applied to the signal read from the FD at a certain time and output, so although the amount of data increases, it is possible to obtain two images with different gains that are simultaneous. Note that although the image sensor 102 in this embodiment has two types of output, there is no particular limit to the number of types of simultaneous output.

[0021] The column ADC 205 converts the signal from the column AMP 204 from analog to digital. The column ADC 205 then supplies the digitized signal (digital image data) to the horizontal transfer circuit 206. The horizontal transfer circuit 206 outputs the line-by-line digital image data to the signal processing circuit 207. The signal processing circuit 207 is a circuit that performs signal processing digitally. In addition to adding a fixed amount of offset value through digital processing, it can easily perform gain calculations by performing shift operations and multiplication. The signal processing circuit 207 passes the processed image data to the external output circuit 208. The external output circuit 208 supplies the image data to an external device (image acquisition unit 193).

[0022] Let's return to the explanation of the configuration in Figure 1. The image acquisition unit 103 captures the image data output from the image sensor 102. If the image sensor 102 outputs an analog signal without performing AD conversion, the above analog-to-digital conversion configuration will be performed by the image acquisition unit 103.

[0023] The image synthesis unit 104 synthesizes HDR image data (image data with two different gains) output from the image sensor 102 into an HDR image using a predetermined synthesis method. For example, a high-gain image (hereinafter referred to as H image) is used for dark areas, a low-gain image (hereinafter referred to as L image) is used for bright areas, and in intermediate regions such as dark to bright areas, the usage rate of each image is determined by weighted addition of the H image and the L image. In this embodiment, the synthesis algorithm is not limited as long as it is a method of synthesizing two images with different gains.

[0024] Furthermore, the image synthesis unit 104 generates synthesis ratio information that allows for the determination of the synthesis ratio of each pixel of the H image and L image during the synthesis process, and adds the generated synthesis ratio information to the synthesized image. In this embodiment, the synthesis ratio is generated for each pixel, but it is also possible to use information that only contains synthesis information for the mixed region, and the synthesis information is not limited as long as it allows for the determination of the synthesis ratio.

[0025] The signal processing unit 105 performs various image processing functions, including pixel addition, noise reduction, gamma correction, knee correction, and digital gain, which are typical image processing functions of the imaging device, as well as scratch correction. The image acquisition unit 103 and the signal processing unit 105 also include memory circuits that store the setting values ​​necessary for each correction and image processing, which are not explicitly shown in the block diagram.

[0026] The signal recording unit 106 records the HDR composite image and video signal received from the image synthesis unit 104 and the signal processing unit 105 onto a recording medium (not shown). The recording medium can be any type of storage device, such as an HDD or a non-volatile memory card such as an SD card.

[0027] The exposure control unit 107 can calculate the exposure amount from the video signal information received from the image acquisition unit 103. In this embodiment, the exposure control unit 107 calculates the exposure amount from the video signal information, but it may also obtain it from the control exposure during imaging. Then, based on the information related to the calculated exposure amount, the exposure control unit 107 determines the operation of the image sensor control unit 108 and transmits the parameters to the image sensor control unit 108.

[0028] Next, we will describe the operation of the image sensor and the image synthesis unit 104 during HDR image generation.

[0029] Figure 3 is a block diagram of the image synthesis unit 104, showing the configuration up to the point where it performs synthesis using the H image and L image output from the image acquisition unit 103, which have undergone two types of gain. Note that the equivalent processing shown in the figure may be implemented in hardware or by a processor executing a control program.

[0030] The exposure compensation unit 301 sets the gains of the input H image and L image to be the same. This is to ensure that the output is linear to the input signal after the images are combined into a single image. Figures 4(a) to (c) are graphs with the input light intensity of the H image and L image on the horizontal axis and the output code after AD conversion on the vertical axis. The unit of light intensity on the horizontal axis is mainly candela (cd).

[0031] Figure 4(a) shows graphs of the H image (thick line) and the L image (thin line). In this state, the brightness of the two images is different, and they cannot be combined as they are.

[0032] Therefore, as shown in Figure 4(b), the exposure compensation unit 301 applies gain to the L image in order to match the brightness of the H image and the L image. Figure 4(c) shows an example where the L image is adjusted to match the brightness of the H image. The image after exposure compensation, where the L image is adjusted to match the brightness of the H image, will be referred to as the Low Gain 2 image (hereinafter referred to as the L2 image). The gain required to obtain the L2 image from the L image depends on the exposure difference between the H image and the L image. This exposure difference depends on the exposure parameters (shutter speed, aperture, etc.) used during shooting to obtain the HDR composite image.

[0033] Figure 5 shows an example of table information (stored in the ROM of the control unit 150) that defines the relationship between the exposure difference and the amount of shift to be corrected. In the table shown, the exposure difference is based on the H image, and represents the difference in exposure of the L image relative to the H image. As shown in the figure, the larger the exposure difference, the larger the amount of shift to be corrected. The exposure correction unit 301 corrects the L image to have the same exposure as the H image based on the shift amount determined by referring to this table, and generates the L2 image. For example, if the exposure difference between the H image and the L image when captured by the image sensor 102 is "-3EV", the exposure correction unit 301 generates the L2 image by performing exposure correction on the L image according to a gain shift amount of "+4 stops".

[0034] It's important to note that when creating the L2 image, if there is a large exposure difference between the H and L images, the amount of gain correction will also increase, and consequently, the amount of noise will also increase. Naturally, increased noise will degrade the image quality of the resulting HDR image.

[0035] Therefore, in this embodiment, if the exposure difference exceeds a preset threshold, or if the gain shift amount exceeds a threshold, the noise correction unit 302 performs noise suppression correction on the L2 image. For example, the noise correction unit 302 performs a filter process in which the pixel of interest to be corrected in the L2 image (one of the R, G, or B pixels in the Bayer array) and the four closest pixels of the same color are used as the corrected pixel value of the pixel of interest. As a result, if a lot of noise is superimposed on the pixel of interest, and its brightness value is extremely high (or low) compared to its surroundings, this can be made to a more natural value, thereby reducing noise. The size of the filter may be set as appropriate, or it may be selected by the user.

[0036] The synthesis processing unit 303 then performs a synthesis process using the H image and L2 image received via the noise correction unit 302 to generate and output HDR image data.

[0037] In this embodiment, the method involves brightness adjustment and noise correction before synthesis. However, if the processing order is reversed, the same processing can be achieved by using a noise correction amount that takes into account the gain of the brightness adjustment, so the order is not limited.

[0038] [Second Embodiment] A second embodiment will now be described. In this second embodiment, the photographer will use an image sensor 102 that can output multiple images with different gains, acquire two images in a single shot, and combine them. The configuration in the second embodiment is the same as the configuration in the first embodiment shown in Figure 1.

[0039] The processing of the control unit 150 will be explained below with reference to the flowchart in Figure 6. It is assumed that the processing in Figure 6 is executed when the user selects the HDR synthesis mode by operating the UI 160 and presses the shutter button.

[0040] In S601, the control unit 150 determines the ISO sensitivity of the H image and L image taken by the image sensor 102 based on the current shooting parameters. This ISO sensitivity can be determined by manual setting by the photographer or by automatic determination by the camera. In this embodiment, the gain is set by ISO sensitivity, but the gain can also be set by exposure time. Subsequently, in S602, the control unit 150 controls the image sensor control unit 108 to take a picture and generate two images with different gains (H image and L image).

[0041] Next, in step S603, the control unit 150 determines the gain shift amount from the difference in ISO sensitivity (exposure difference) between the H image and the L image. Then, the control unit 150 controls the exposure compensation unit 301 and applies the determined gain shift amount to the L image to generate an L2 image that matches the brightness of the H image.

[0042] Next, in S604, the control unit 150 determines whether the gain shift amount based on the exposure difference acquired in S603 is equal to or greater than a threshold (for example, two steps). If the control unit 150 determines that the gain shift amount based on the exposure difference acquired in S603 is equal to or greater than a threshold (for example, two steps), it proceeds to S605; otherwise, it proceeds to S606.

[0043] In step S605, the control unit 150 controls the noise correction unit 302 to perform noise correction processing on the L2 image.

[0044] In S606, the control unit 150 controls the synthesis processing unit 303 to synthesize the H image and the L2 image (or the L2 image after noise correction processing if S605 has been performed) to generate an HDR composite image. Then, in S606, the control unit 150 generates synthesis information and adds it to the HDR image.

[0045] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0046] The disclosures herein include the following image processing apparatus, control methods therefor, and programs. (Item 1) An image processing apparatus that processes first and second image data from an imaging means capable of outputting first and second image data by amplifying signals obtained by photoelectric conversion with different gains, An acquisition means for acquiring the exposure difference of the first and second image data, A correction means that determines a correction amount to suppress the noise difference between the first and second image data based on the exposure difference acquired by the acquisition means, and corrects at least one of the first and second image data based on the determined correction amount, A generation means that generates a composite image by combining the first and second image data corrected by the correction means, An image processing apparatus characterized by having (Item 2) The correction means performs a correction to match the exposure of one of the first and second image data to that of the other, according to the exposure difference acquired by the acquisition means. The generation means generates a composite image by combining the image data obtained by exposure correction by the correction means with the other image data. The image processing apparatus according to item 1, characterized in that it is a picture processing apparatus. (Item 3) Furthermore, a determination means for determining whether or not to perform noise suppression correction according to the exposure difference acquired by the acquisition means, If the determination means determines that correction should be performed, the generation means performs noise reduction correction on the image data after correction by the correction means before the generation means performs the synthesis. An image processing apparatus according to item 1 or 2, characterized by having the following features. (Item 4) The acquisition means acquires the exposure difference between the first and second image data based on the parameters or controlled exposure during shooting by the imaging means. The image processing apparatus according to item 3, characterized in that (Item 5) When the image data obtained at the higher gain among the different gains is referred to as the first image data, and the image data obtained at the lower gain is referred to as the second image data, The acquisition means acquires the exposure difference between the first and second image data by referring to a predetermined table based on the parameters during shooting by the imaging means. The correction means corrects the second image data with a shift amount corresponding to the exposure difference. The image processing apparatus according to item 1, characterized in that it is a picture processing apparatus. (Item 6) The image processing apparatus according to any one of items 1 to 5, characterized in that the generation means generates HDR (High Dynamic Range) image data. (Item 7) The generation means further generates information indicating the composite ratio of each pixel in the two image data used when generating the HDR image data. The image processing apparatus according to item 6, characterized in that (Item 8) An image processing apparatus according to any one of items 1 to 7, characterized by having the aforementioned imaging means. (Item 9) A control method for an image processing apparatus that processes first and second image data from an imaging means capable of outputting first and second image data by amplifying signals obtained by photoelectric conversion with different gains, An acquisition process to acquire the exposure difference of the first and second image data, A correction step of determining a correction amount to suppress the noise difference between the first and second image data, and correcting at least one of the first and second image data based on the determined correction amount, A generation step that generates a composite image by combining the first and second image data corrected by the correction step, A control method for an image processing apparatus, characterized by having the following features. (Item 10) A program that, when read and executed by a computer, causes the computer to function as one of the means of the image processing apparatus described in any one of items 1 to 8.

[0047] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0048] 101...Optical lens, 102...Image sensor, 103...Image acquisition unit, 104...Image synthesis unit, 105...Signal processing unit, 105, 106...Signal recording unit, 150...Control unit, 160...UI

Claims

1. An image processing apparatus that processes the first image data and the second image data output from an imaging unit capable of amplifying a signal obtained by photoelectric conversion with different gains and outputting the first image data and the second image data, an acquisition unit that acquires an exposure difference between the first image data and the second image data; a correction unit that determines a correction amount for suppressing a noise difference between the first image data and the second image data based on the exposure difference acquired by the acquisition unit, and corrects at least one of the first image data and the second image data based on the determined correction amount; a generation unit that generates composite image data by synthesizing the corrected first image data and the corrected second image data when the correction unit corrects both the first image data and the second image data, and synthesizes the other of the first image data and the second image data and the corrected one of the image data to generate composite image data when the correction unit corrects one of the first image data and the second image data; An image processing apparatus characterized by comprising the above.

2. The correction unit performs correction to match the exposure of one of the first image data and the second image data with the other according to the exposure difference acquired by the acquisition unit, and the generation unit generates composite image data by synthesizing the image data obtained by the exposure correction by the correction unit and the other image data. The image processing apparatus according to claim 1, characterized by the above.

3. Further, a determination unit that determines whether to perform noise suppression correction according to the exposure difference acquired by the acquisition unit, and a noise correction unit that performs noise suppression correction on the image data after correction by the correction unit before the generation unit performs the synthesis when the determination unit determines to perform correction. The image processing apparatus according to claim 1 or 2, characterized by comprising the above.

4. The acquisition means acquires an exposure difference between the first image data and the second image data based on parameters or control exposure during imaging by the imaging means. The image processing apparatus according to claim 3, characterized in that.

5. When the image data obtained with the higher gain among the different gains is used as the first image data and the image data obtained with the lower gain is used as the second image data, The acquisition means acquires an exposure difference between the first image data and the second image data with reference to a predetermined table based on parameters during imaging by the imaging means, The correction means corrects the second image data by a shift amount corresponding to the exposure difference. The image processing apparatus according to claim 1, characterized in that.

6. The image processing apparatus according to any one of claims 1 to 5, characterized in that the generation means generates HDR (High Dynamic Range) image data as the composite image data.

7. The generation means further generates information indicating a composite ratio of each pixel in the two image data used when generating the HDR image data. The image processing apparatus according to claim 6, characterized in that.

8. The image processing apparatus according to any one of claims 1 to 7, characterized in that it includes the imaging means.

9. An image processing apparatus that processes the first image data and the second image data output from an imaging means capable of amplifying a signal obtained by photoelectric conversion with a first gain and a second gain lower than the first gain, and outputting the first image data corresponding to the first gain and the second image data corresponding to the second gain, An acquisition means for acquiring an exposure difference between the first image data and the second image data; Correction means for correcting the exposure of the second image data so as to match the exposure of the first image data based on the exposure difference; Noise correction means for performing noise suppression correction on the second image data corrected by the correction means; Generation means for generating composite image data by combining the first image data and the second image data after the noise suppression correction; An image processing apparatus, characterized by comprising the same.

10. The acquisition means acquires the exposure difference between the first image data and the second image data based on the parameters or the controlled exposure at the time of shooting by the imaging means. The image processing apparatus according to claim 9, characterized by the above.

11. The generation means generates HDR (High Dynamic Range) image data as the composite image data. The image processing apparatus according to claim 9 or 10, characterized by the above.

12. The generation means further generates information indicating the composite ratio of each pixel in the two image data used when generating the HDR image data. The image processing apparatus according to claim 11, characterized by the above.

13. The image processing apparatus according to any one of claims 9 to 12, characterized by comprising the imaging means.

14. A control method for an image processing apparatus that processes the first image data and the second image data output from an imaging means capable of amplifying signals obtained by photoelectric conversion with different gains and outputting the first image data and the second image data, An acquisition step of acquiring the exposure difference between the first image data and the second image data; A correction step of determining a correction amount for suppressing the noise difference between the first image data and the second image data, and correcting at least one of the first image data and the second image data based on the determined correction amount; When both the first image data and the second image data are corrected in the correction process, the corrected first image data and the corrected second image data are combined to generate combined image data. When only one of the first image data and the second image data is corrected in the correction process, the other of the first image data and the second image data and the corrected one of the image data are combined to generate combined image data. A generation step; A control method for an image processing apparatus, characterized by comprising the above.

15. A control method for an image processing apparatus that processes the first image data and the second image data output from an imaging means capable of amplifying a signal obtained by photoelectric conversion with a first gain and a second gain lower than the first gain, and outputting the first image data corresponding to the first gain and the second image data corresponding to the second gain, An acquisition step of acquiring an exposure difference between the first image data and the second image data; A correction step of correcting the exposure of the second image data based on the exposure difference so as to match the exposure of the first image data; A noise correction step of performing noise suppression correction on the second image data corrected in the correction step; A generation step of generating combined image data by combining the first image data and the second image data after the noise suppression correction; A control method for an image processing apparatus, characterized by comprising the above.

16. A program for causing a computer to execute each step of the method according to claim 14 or 15 when the computer reads and executes the program.

Images