Image processing device, image processing method and program

Abstract

To provide an image processing device that can decrease noise of a synthetic image when two images with different gains which are output through one exposure are synthesized.SOLUTION: An image processing device 100 includes an imaging element 102. The imaging element 102 includes a pixel unit 203 including a photoelectric conversion unit configured to convert light to electric charge and a charge accumulating unit configured to accumulate the electric charge, and a column AMP 204 configured to amplify a signal output from the pixel unit 203 with different gains, and outputs a plurality of image signals with different gains applied thereto through one exposure. The image processing device 100 further includes an image synthesizing unit 105 configured to generate a synthetic image signal by synthesizing the plurality of image signals. An image synthesis control unit 109 configured to control the image synthesizing unit 105 determines synthesis proportions of the plurality of image signals in the synthetic image signal in accordance with a common capacity of the charge accumulating unit when the capacity of the charge accumulating unit is common with the plurality of image signals.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device having an image sensor used in a camera or the like, and more particularly to an image processing device, image processing method, and program for synthesizing a plurality of images in order to expand the dynamic range of the images. [Background technology]

[0002] There is an image sensor that has two column circuits for output signals from unit pixels, has separate gains for the amplifiers in the column circuits, and is capable of outputting images with different gains (see, for example, Patent Document 1). Such an image sensor is called a DGO (Dual Gain Output) sensor. This image sensor can output two images with different gains (high gain and low gain images) with a single exposure. Compared to the synthesis of two images obtained by time-division exposure (see, for example, Patent Document 2), the synthesis of two images by the DGO has the advantage of being strong against moving objects because it does not require alignment processing. Therefore, it is compatible with HDR (high dynamic range) synthesis, which is a means of obtaining images with an expanded dynamic range.

[0003] The image sensor has a floating diffusion (FD) that acts as a capacitor to store electric charge. The amount of charge that the FD can store changes depending on the settings. Increasing the capacity allows more light to be handled, so a larger capacity is used at low sensitivity. However, increasing the capacity of the FD has the disadvantage of increasing noise. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-178186 A [Patent Document 2] JP 2009-088928 A Summary of the Invention [Problem to be solved by the invention]

[0005] When synthesizing images using time-division exposure, it is possible to set the FD capacity appropriately for each exposure depending on the gain. On the other hand, when exposure is performed using a DGO, the FD capacity is common to both high and low gains. Therefore, depending on the combination of high and low gain sensitivities, the FD capacity is common to both high and low gains, which can cause a problem in that it is not possible to set an appropriate FD capacity for either.

[0006] As a general example, it is not practical to use a small FD capacity at an ISO sensitivity where a large FD capacity is an appropriate setting. This is because the FD capacity will be insufficient and the FD will be saturated. Therefore, basically, when the FD capacity of high gain and low gain are different, it is considered to match the FD capacity of low gain. In this case, an appropriate FD capacity cannot be set for the high gain image, and the capacity will become large, causing noise to worsen, and this effect will appear in the composite image. Thus, depending on the combination of ISO sensitivity, there is a problem that the FD capacity setting for the high gain image is not appropriate, causing noise to worsen in the composite image.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide an image processing device that can reduce noise in a composite image when combining two images with different gains output by a single exposure. [Means for solving the problem]

[0008] In order to achieve the above object, the image processing device of the present invention is characterized in that it has a pixel unit having a photoelectric conversion unit that converts light into an electric charge and a charge accumulation unit that accumulates the electric charge, and an amplifier unit that can amplify signals output from the pixel unit with different gains from each other, an imaging element that outputs multiple image signals with different gains in a single exposure, an image synthesis unit that synthesizes the multiple image signals to generate a composite image signal, and an image synthesis control unit that controls the image synthesis unit to determine the synthesis ratio of the multiple image signals in the composite image signal in accordance with the common capacity when the capacity of the charge accumulation unit is common to the multiple image signals. Effect of the Invention

[0009] According to the present invention, when a plurality of images having different gains outputted by a single exposure are synthesized, noise in the synthesized image can be reduced. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing internal blocks of a solid-state imaging element. [Diagram 3] FIG. 2 is a diagram showing a column AMP of a solid-state imaging element. [Figure 4] FIG. 2 is a block diagram showing an image synthesis unit according to the embodiment. [Diagram 5] 5 is a graph showing the relationship between the amount of input light and the output code after AD conversion according to the first embodiment. [Figure 6] FIG. 4 is a graph showing the relationship between a synthesis threshold and an FD capacity according to the first embodiment. [Figure 7] 5 is a flowchart showing a flow of determining a synthesis threshold value according to the first embodiment. [Figure 8] 5 is a flowchart showing a flow of FD capacity setting decision according to the first embodiment. [Figure 9] FIG. 4 is a graph showing the relationship between the amount of input light and the FD capacity according to the first embodiment. [Figure 10] 10 is a flowchart showing a flow of determining a synthesis threshold value according to the second embodiment. [Figure 11] FIG. 11 is a graph showing the relationship between a synthesis threshold value and the temperature of an image sensor according to the second embodiment. [Figure 12] 13 is a flowchart showing a flow of determining a synthesis threshold value according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0012] [First embodiment] Fig. 1 is a block diagram showing the configuration of an image processing device 100 according to this embodiment. Next, each block in Fig. 1 will be described in detail. Note that the image processing device 100 (imaging device) has a CPU (not shown) that controls each unit in the device, and has a recording medium or recording device such as a ROM or RAM.

[0013] The optical lens 101 is an optical lens that captures light from a subject. The light incident on the optical lens 101 forms an image on the image sensor 102.

[0014] The imaging element 102 converts the incident light received from the optical lens 101 into an electrical signal, and outputs it as a video signal. Representative examples of the imaging element 102 include a CCD image sensor (Charge Coupled Device) and a CMOS image sensor.

[0015] The image signal output from the image sensor 102 may be an analog image signal that is directly output, or a digital signal that is output through AD (analog-digital) conversion processing. The AD conversion processing is performed inside the image sensor 102. The digital signal that is output is a digital signal such as LVDS (Low voltage differential signaling). The detailed configuration of the image sensor will be described later.

[0016] The image acquisition unit 103 is composed of blocks that capture the video signal output from the image sensor 102 and perform various processes. If AD conversion is not performed inside the image sensor 102, the image acquisition unit 103 also includes an analog front end that performs AD conversion. The image acquisition unit 103 removes fixed pattern noise from the image sensor 102, clamps the black level, and performs other processes. It also has a role of separating the image to be used for video signal recording from an evaluation signal for controlling the image sensor.

[0017] The signal processing unit 104 performs various image processing such as pixel addition, noise reduction, gamma correction, knee correction, digital gain, etc., which are typical image processing functions of an image processing device, as well as defect correction, etc. Although not shown in the block diagram, the image acquisition unit 103 and the signal processing unit 104 also include memory circuits that store setting values ​​required for each correction and image processing.

[0018] The image synthesis unit 105 synthesizes an HDR image signal from the image signals for HDR generation output from the imaging element using an arbitrary synthesis method. For example, there is a synthesis method using a high-gain image signal as the main image signal (normal image) and a low-gain image signal for bright and blown-out parts of the normal image. However, the synthesis method of the HDR image signal in this embodiment is not limited to the above method as long as it is a method of synthesizing two image signals with different gains. The image synthesis unit 105 is controlled by an image synthesis control unit 109 that controls at least the synthesis ratio when generating a synthesized image signal.

[0019] The signal recording unit 106 records the video signal received from the image synthesis unit 105 in a storage device or storage medium. The exposure control unit 107 calculates an optimal exposure amount from the video signal information received from the image acquisition unit 103, and determines the operation of the image sensor control unit 108. The image sensor control unit 108 transmits a control signal to the image sensor 102 to control the image sensor 102.

[0020] Fig. 2 is a diagram showing internal blocks of a solid-state imaging device, and shows each block of the imaging device as an example of the imaging device used in this embodiment.

[0021] A timing pulse control section 201 controls the operation of the image sensor by supplying an operating clock (CLK) to each block of the image sensor and by supplying a timing signal to each block.

[0022] A vertical scanning circuit 202 performs timing control for sequentially reading out pixel signal voltages from two-dimensionally arranged pixel units 203 during one frame. In general, video signals are sequentially read out row by row from the top row to the bottom row during one frame.

[0023] The pixel unit 203 has a photoelectric conversion element (photoelectric conversion unit) that performs photoelectric conversion according to the amount of incident light and outputs it as a voltage. The pixel unit 203 converts the captured light into electric charges, and then accumulates the electric charges in a floating diffusion (FD) that serves as an electric charge accumulation unit. The capacity of the FD can be changed between large and small. That is, the FD capacity can be selected from at least a first capacity having a relatively large FD capacity and a second capacity having a capacity smaller than the first capacity. The S / N ratio (signal-to-noise ratio) is improved by changing the FD capacity according to the ISO sensitivity. Basically, the FD capacity is set to large at low ISO sensitivity, and the FD capacity is set to small at high ISO sensitivity. Note that, when outputting two image signals with different gains, which will be described later, the FD capacity for the two image signals using the two gains is common. Note that the size of the FD capacity is not limited to two levels, large and small, and may be set to three or more levels.

[0024] The column AMP 204 (amplification unit) is an amplifier used to electrically amplify a signal read from the pixel unit 203. By amplifying a signal in the column AMP 204, the signal level of the pixel is amplified against noise generated by the subsequent column ADC 205, and the S / N ratio is equivalently improved. In addition, the gain of the column AMP 204 is structured to be changeable by the timing pulse control unit 201. The image sensor 102 of this embodiment has two input memories in the column AMP 204 for generating HDR images, and is capable of outputting two types of gain by changing the gain setting in the column AMP. By having two input memories, it is possible to output a signal at a certain time output from the FD of the pixel unit by applying two different gains to the signal. The two gains are composed of a relatively large first gain and a second gain smaller than the first gain. Therefore, although the amount of data increases, it is possible to obtain images with two different gains with simultaneity. Note that, although two outputs are provided in this embodiment, the number of simultaneous outputs is not limited to two.

[0025] A column ADC (analog-digital converter) 205 performs analog-to-digital conversion on the read signals from the column AMP 204. A horizontal transfer circuit 206 sequentially reads out the signals digitized by the column ADC 205. The output of the horizontal transfer circuit 206 is input to a signal processing circuit 207.

[0026] The signal processing circuit 207 is a circuit that performs signal processing digitally. The signal processing circuit 207 adds a certain amount of offset value by digital processing, and also performs shift calculation and multiplication. This allows gain calculation to be performed easily. In addition, the pixel unit 203 may be provided with an intentionally light-shielded pixel region, and a digital black level clamp operation using the pixel region may be performed.

[0027] The external output circuit 208 receives the output of the signal processing circuit 207. The external output circuit 208 has a serializer function and converts a multi-bit input parallel signal from the signal processing circuit 207 into a serial signal. The external output circuit 208 also converts this serial signal into, for example, an LVDS signal, and outputs it as the exchange of image information with an external device. In this way, the image sensor 102 outputs multiple image signals with different gains applied in a single exposure.

[0028] Next, the operation of the image sensor when generating an HDR image and the image synthesis unit 105 will be described. As described above, the image sensor of this embodiment can change and output the column AMP gain in order to generate an HDR image. The operation of the column AMP 204 when generating an HDR image is shown in FIG. 3. FIG. 3 is a diagram showing the column AMP of a solid-state image sensor. FIG. 3 shows a circuit in which one column of the column AMP 204 is extracted. The capacitance section is represented as C, the switching element (transistor, etc.) as SW, and the operational amplifier as OP.

[0029] An input capacitance and a feedback capacitance are connected to the OP305. C303 and C304 are input capacitances, and signals read from the pixel unit 203 are connected by SW301 and SW302, respectively. C306 and C308 are feedback capacitances, and the connection of C308 can be controlled by SW307. Since capacitance is used, the amplification factor of the amplifier is the input capacitance / feedback capacitance. In this embodiment, there are two input capacitances. Therefore, first, SW301 is connected and SW302 is disconnected, and the signal is multiplied by the gains of C303 and C306 and output to the column ADC205. Next, SW301 is disconnected, SW302 and SW307 are connected, and the signal is multiplied by the gains of C304, C306, and C308 and output. This makes it possible to output two images obtained from image signals to which different gains have been applied. Then, the image synthesis unit 105 synthesizes multiple image signals to generate a synthesized image signal. In the following, when necessary, the synthesis of "image signals" will be described using the expression "synthesizing images."

[0030] Fig. 4 is a block diagram showing the image synthesis unit 105 according to this embodiment. Fig. 4 shows that two images with two types of gain applied output from the imaging element 102 pass through the processing blocks of the signal processing unit 104 and the image synthesis unit 105. Here, the two images are referred to as a high gain image (hereinafter, H image) and a low gain image (hereinafter, L image). The exposure correction unit 401 makes the gains of the H image (first image signal) and the L image (second image signal) the same. As a result, when the images are synthesized into one image in the image synthesis unit 402, the output becomes linear with respect to the input signal.

[0031] Fig. 5 is a graph showing the relationship between the amount of input light and the output code after AD conversion according to the first embodiment. In Fig. 5, the horizontal axis represents the amount of light input to the H image and the L image, and the vertical axis represents the output code after AD conversion. The unit of light amount on the horizontal axis is usually candela (cd).

[0032] In Fig. 5(a), a graph of the H image (thick line) and the L image (thin line) is shown. In the state of Fig. 5(a), the brightness of the two images is different, so they cannot be combined as they are.

[0033] For this reason, the exposure correction unit 401 applies a gain to match the brightness of the H image and the L image. FIG. 5(b) shows an example in which the L image is matched to the brightness of the H image. The L image is subjected to exposure correction and the brightness is matched to the H image to create a low gain second image (hereinafter referred to as L2 image). In the case shown in FIG. 5(b), the gain applied by the exposure correction unit 401 is greater than 1. On the other hand, when matching the brightness of the H image to that of the L image, the gain is less than 1.

[0034] Fig. 5(c) is a graph after synthesis, which is an example of synthesis of an H image and an L2 image. Synthesis is performed such that the H image is used for parts with data of an H image with a small input light amount, and the L2 image is used for parts with a large input light amount.

[0035] FIG. 6 is a diagram showing the relationship between the synthesis threshold and the FD capacity according to the first embodiment. FIG. 6 shows the synthesis threshold showing the synthesis ratio of the H image and the L image by the brightness according to the FD capacity that accumulates the electric charge. In this figure, the synthesis threshold at the time of synthesis is made different according to the size of the FD capacity that accumulates the electric charge at the time of shooting. Specifically, the synthesis threshold becomes lower brightness side as the FD capacity becomes larger. Here, when performing DGO, the FD capacity that is common to all of the multiple image signals is set to a large capacity. Here, when the FD capacity is large, the FD capacity setting is in a state suitable for the L image, so that the noise of the H image becomes worse. For this reason, the synthesis threshold is set to a lower brightness value so that the area (synthesis ratio) of the H image occupies less in the synthetic image. In other words, the synthesis threshold when the FD capacity is large is set to a lower brightness value side than the synthesis threshold when the FD capacity is small.

[0036] Next, as described above, an example of a case where a photographer captures two images using the image sensor 102 capable of outputting a plurality of images to which different gains have been applied, and then synthesizes the two images will be described. The explanation will be given with reference to the flowchart of Fig. 7. Fig. 7 is a flowchart showing the flow of determining a synthesis threshold value according to the first embodiment.

[0037] In step S701, the ISO sensitivity of the H image and the L image to be captured is determined. The ISO sensitivity may be determined by manual setting by the photographer, or may be determined automatically by the camera.

[0038] In step S702, a process of determining the setting of the FD capacity is performed. The detailed process of step S702 will be described later. After the FD capacity determination process is performed in step S702, two images with different gains are captured in step S703.

[0039] In step S704, it is determined whether or not the process of making the FD capacity settings uniform has been performed in step S702. If the process of changing the FD capacity setting has been performed in step S702, the process proceeds to step S705. On the other hand, if the process of changing the FD capacity setting has not been performed in step S702, the process proceeds to step S706.

[0040] In step S705, a synthesis threshold is determined based on the FD capacity setting of the captured image. If the process of step S702 has been performed, the FD capacity setting of the H image is set to a larger FD capacity setting than the FD capacity setting suitable for the H image. This causes the noise of the H image to worsen. In this case, as described above with reference to FIG. 6, the synthesis threshold is set to a lower brightness according to the size of the set FD capacity. This reduces the area of ​​the H image that appears in the synthesis image, and reduces the effect of noise on the synthesis image.

[0041] In step S706, the two images are composited. The composite threshold used when composite images is the composite threshold changed in step S705, or the composite threshold before change determined in step S702. When composite images are composited, the brightness of the two images is adjusted in the blocks shown in FIG. 4, and the images are composited as shown in FIG. 5. As a method for determining the composite threshold according to the FD capacity, there is a method in which the composite threshold has a fixed value for each FD capacity. However, the method for determining the composite threshold is not limited to this method.

[0042] The FD capacity setting determination process in step S702 will be described in detail with reference to FIG. 8. FIG. 8 is a flowchart showing the flow of FD capacity setting determination according to the first embodiment. There is no problem with the FD capacity setting at the time of shooting if the appropriate FD capacity setting for the set ISO sensitivity is common to the H image and the L image. However, in the combination of the H image and the L image for which appropriate FD capacity settings are not shared, it is necessary to determine which FD capacity setting to use. A specific description will be given below.

[0043] In step S801, it is determined whether the FD capacity settings of the H image and the L image are the same. If the FD capacity settings are not the same, the process proceeds to step S802. On the other hand, if the FD capacity settings are the same, the determination of the FD capacity setting is completed.

[0044] In step S802, a saturation threshold is calculated for each FD capacitance. Specifically, the saturation threshold is the output value of the image sensor after AD conversion when the amount of light is such that the FD capacitance is saturated.

[0045] In step S803, the FD capacity setting for low gain is set to the smallest capacity. For example, if the capacity setting has two levels, large and small, the FD capacity is set to the smaller one.

[0046] In step S804, it is determined whether the amount of light input at the low gain exceeds the saturation threshold of the FD capacity set to the smallest capacity. If the amount of light input at the low gain exceeds the saturation threshold of the FD capacity set to the smallest capacity, the process proceeds to step S805. On the other hand, if the amount of light input at the low gain does not exceed the saturation threshold of the FD capacity set to the smallest capacity, the process proceeds to step S307.

[0047] In step S805, the FD capacity set for the low gain is changed to a value larger than the saturation threshold value. Specifically, the set value of the FD capacity is changed to a value one level larger.

[0048] In step S806, the FD capacity of the low gain is compared with the FD capacity of the high gain. If the FD capacity of the low gain is the same as the FD capacity of the high gain, the process proceeds to step S807. On the other hand, if the FD capacity of the low gain is not the same as the FD capacity of the high gain, the process proceeds to step S804.

[0049] In step S807, the FD capacity setting for the high gain is adjusted to match the FD capacity setting for the low gain. After step S807, the process ends.

[0050] Next, the FD capacity setting determination process will be described by taking as an example a case where the FD capacity that can be set is three stages. Fig. 9 is a diagram showing the relationship between the input light amount and the FD capacity according to the first embodiment. In the figure, the horizontal axis shows the input light amount, and the vertical axis shows the output after AD conversion. The saturated light amount for each FD capacity is also shown by a dashed line.

[0051] As shown in FIG. 9, when there are three stages of FD capacity (large FD capacity, medium FD capacity, small FD capacity), the amount of input light may not exceed the saturation light amount (saturation threshold) of the FD capacity with the smallest capacity. In this case, it is not necessary to match the FD capacity used in high gain with the FD capacity used in low gain. In other words, it is possible to change the FD capacity of low gain and match the FD capacity of high gain to the FD capacity after this change. This makes it possible to reduce the effects of noise that appears in high gain.

[0052] 9, when three or more stages of FD capacity settings are possible, the above-mentioned steps S804 to S806 are repeated. Then, when the input light amount does not exceed the saturation threshold, the low-gain FD capacity setting is changed to a smaller capacity. Finally, in step S807, the high-gain FD capacity is adjusted to the low-gain FD capacity.

[0053] As described above, according to the first embodiment, after the settings of the low-gain FD capacity and the high-gain FD capacity are aligned, a synthesis threshold is determined according to the aligned FD capacity. Here, when determining the synthesis threshold, if an FD capacity with a relatively large (large) FD capacity is selected, the synthesis threshold is set so that there are fewer high-gain images than if an FD capacity with a relatively small (small) FD capacity is selected. This makes it possible to reduce the number of high-gain images when the FD capacity is large, and to reduce noise in the synthesis image.

[0054] [Second embodiment] In the second embodiment, a synthesis threshold, which is a threshold for the synthesis ratio, is determined according to the FD capacity setting and the temperature of the image sensor. The image processing device of this embodiment has a temperature sensor that measures the temperature of the image sensor 102, and the measurement result of the temperature sensor is input to an image synthesis unit 105. Descriptions of other configurations of the image processing device, etc., that have already been described in the first embodiment will be omitted.

[0055] 10 is a flowchart showing a flow of determining a synthesis threshold value according to the second embodiment. Steps S1001 to S1005 are similar to steps S701 to S705 in FIG.

[0056] In step S1006, the synthesis threshold is corrected according to the temperature of the image sensor. As the temperature of the image sensor increases, the noise of the H image increases. For this reason, the synthesis threshold is adjusted so that the area of ​​the H image appearing in the synthesis image is reduced as the temperature of the image sensor increases.

[0057] In step S1007, the two images, the image H and the image L, are synthesized. As the synthesis threshold for synthesizing the images, the synthesis threshold corrected in step S1006 or the synthesis threshold before change determined in step S1004 is used.

[0058] Next, the synthesis threshold correction in step S1006 will be described with reference to FIG. 11. FIG. 11 is a diagram showing the relationship between the synthesis threshold and the temperature of the image sensor according to the second embodiment. As shown in FIG. 11, in step S1006, as the temperature of the image sensor increases, the synthesis threshold is changed toward lower brightness from the synthesis threshold determined in step S1005 so that the area of ​​the H image is further reduced. In this way, it is possible to reduce the influence of noise on the synthesis image according to the temperature of the image sensor. Note that the method of changing the synthesis threshold according to the temperature of the image sensor is not limited to that of this embodiment. Any other method may be used as long as it is a method of changing the synthesis threshold so as to reduce the area of ​​the H image as the temperature of the image sensor increases.

[0059] As described above, according to the second embodiment, the synthesis threshold value of the H image and the L image is changed to a lower brightness value according to the FD capacity setting, and the synthesis threshold value is corrected so that the area of ​​the H image is reduced as the temperature of the image sensor increases. This makes it possible to reduce noise in the synthesized image.

[0060] [Third embodiment] In the third embodiment, in addition to the above-mentioned embodiment, a luminance noise distribution is calculated from a captured image, and a synthesis threshold is determined based on the luminance noise distribution. Then, when a luminance noise distribution of a sufficient area cannot be created in the calculation of the luminance noise distribution, the synthesis threshold is determined from the above-mentioned FD capacity setting. Descriptions of the same configuration as the above-mentioned embodiment are omitted.

[0061] Fig. 12 is a flowchart showing the flow of synthesis threshold determination according to the third embodiment. Fig. 12 shows the procedure for calculating the luminance noise distribution and determining the synthesis threshold when determining the synthesis threshold after shooting.

[0062] In step S1201, development processing is performed to calculate the luminance noise distribution. Here, the brightness of two images, the H image and the L image, is matched. In this embodiment, the H image is used as a base image, and the L2 image is subjected to exposure correction to match the brightness of the H image.

[0063] In step S1202, a uniform luminance surface is detected from the H image and the L2 image. A uniform luminance surface is a region with no edges or few edges, and in which the maximum and minimum luminance value difference is equal to or smaller than a threshold value.

[0064] In step S1203, it is determined whether a predetermined number of uniform luminance surfaces have been obtained. If the predetermined number of uniform luminance surfaces have been obtained, the process proceeds to step S1204. On the other hand, if the predetermined number of uniform luminance surfaces have not been obtained, the process proceeds to step S1202.

[0065] In step S1204, it is determined whether the area of ​​the detected uniform luminance surface (range of the uniform surface group) is equal to or larger than a predetermined range. In determining whether the area of ​​the uniform luminance surface is sufficiently large, it is determined in particular whether it includes a possible composite threshold value.

[0066] In step S1205, noise is measured in the detected uniform luminance surface area of ​​the H image, and noise is measured in the same area of ​​the L2 image. Based on the results of these noise measurements, the luminance noise distributions of the H image and the L2 image are calculated.

[0067] In step S1206, the synthesis threshold is determined based on the luminance value at which the noise of the H image and the L2 image are reversed, based on the luminance noise distribution calculated in step S1205. In this embodiment, the synthesis threshold is determined to be a luminance value at which the noise of the H image does not become greater than the noise of the L2 image, based on the luminance value at which the noise of the H image and the L2 image are reversed. After step S1206, the process ends.

[0068] In step S1207, as described in the above embodiment, a synthesis threshold is calculated based on the FD capacity setting. In this way, if a uniform luminance surface of a sufficient range cannot be detected in step S1204, the process of step S1207 is performed. After step S1207, the process ends.

[0069] As described above, according to the third embodiment, it is possible to obtain a more accurate noise reduction effect from the luminance noise distribution. Also, even in a scene where it is difficult to detect a uniform luminance surface, it is possible to obtain a certain noise reduction effect by determining the synthesis threshold from the FD capacity setting.

[0070] [Other embodiments] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. In addition, parts of the above-mentioned embodiments may be combined as appropriate.

[0071] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0072] 100...Image processing device 102...Image sensor 105...Image synthesis unit 108...imaging element control unit 109...Image synthesis control unit 203...Pixel section 204…column AMP

Claims

1. an image sensor including a pixel section having a photoelectric conversion section for converting light into electric charges and a charge accumulation section for accumulating the electric charges, and an amplifier section for amplifying signals output from the pixel section with gains different from each other, and outputting a plurality of image signals with different gains by a single exposure; an image synthesis unit that synthesizes the plurality of image signals to generate a synthesized image signal; and an image synthesis control unit that controls the image synthesis unit so as to determine a synthesis ratio of the plurality of image signals in the synthesized image signal in accordance with the common capacity of the charge storage unit when the capacity of the charge storage unit is common to the plurality of image signals.

13. An image processing device comprising:

2. the capacitance of the charge storage unit includes a first capacitance and a second capacitance that is smaller than the first capacitance, the plurality of image signals includes a first image signal amplified using a first gain and a second image signal amplified using a second gain smaller than the first gain; When the common capacitance in the charge storage unit is the first capacitance, the image synthesis control unit reduces a synthesis ratio of the first image signal in the synthesized image signal compared to when the common capacitance is the second capacitance.

2. The image processing device according to claim 1,

3. The image synthesis unit changes the synthesis ratio in response to a temperature of the image sensor.

3. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing device.

4. The image synthesis unit performs correction so that the synthesis ratio of the first image signal used for the synthesized image signal decreases as the temperature of the imaging element increases.

3. The image processing device according to claim 2.

5. an image sensor control unit that controls a gain for the signal and a capacity of the charge storage unit; The imaging element control unit calculating a threshold value at which the capacitance of the charge storage unit is saturated based on a signal voltage obtained at a relatively low gain among a plurality of gain settings for the signal; The capacitance of the charge storage unit is determined so that the capacitance of the charge storage unit is reduced as long as the amount of light input to the pixel unit does not exceed the threshold value.

5. The image processing device according to claim 1,

6. The image synthesis unit includes: A luminance noise distribution of each of the plurality of image signals is calculated, and a combination ratio of the plurality of image signals in the combined image signal is changed according to the luminance noise distribution and the capacity of the charge storage unit.

6. The image processing device according to claim 1,

7. an output step of amplifying signals output from a pixel unit having a photoelectric conversion unit that converts light into electric charges and a charge accumulation unit that accumulates the electric charges with gains different from each other, thereby outputting a plurality of image signals with different gains from an image pickup element by a single exposure; a generating step of generating a composite image signal by combining the plurality of image signals, When generating the composite image signal, if the capacitance of the charge storage unit becomes common to all of the plurality of image signals, a composite ratio of the plurality of image signals in the composite image signal is changed according to the common capacitance.

13. An image processing method comprising:

8. A program for causing a computer to execute the image processing method according to claim 7.

Images