Image processing device and method, imaging device, program, and storage medium

Abstract

To suppress occurrence of a false color, when a plurality of images having different sensitivities are synthesized to expand a dynamic range.SOLUTION: An image processing device includes: acquisition means for acquiring pixel signals of a plurality of images obtained by photographing the same subject with a plurality of different sensitivities by an imaging element having a plurality of pixels; arithmetic means for calculating a conversion ratio for converting the pixel signals of the plurality of images into the pixel signals whose sensitivities are matched and a synthesis ratio for synthesizing the pixel signals of the plurality of images whose sensitivities are matched for each pixel; and synthesis means for performing the dynamic range expansion synthesis of the pixel signals of the plurality of images for each pixel by using the conversion ratio and the synthesis ratio. The arithmetic means calculates the synthesis ratio for each division area obtained by dividing the plurality of pixels.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image processing device and method, an imaging device, a program, and a storage medium, and in particular to a dynamic range expansion synthesis technique. [Background technology]

[0002] So-called CMOS image sensors include GS sensors that have a global shutter (hereinafter referred to as "GS") function by having a memory section (charge holding section) in each pixel. The pixels of this GS sensor are equipped with a gate that transfers the signal charge accumulated in the photoelectric conversion section to the charge holding section. In a GS sensor, the charge is basically transferred from the photoelectric conversion section to the charge holding section simultaneously for all pixels, and the GS function is realized by making the timing of the start and end of charge accumulation in the photoelectric conversion section the same for all pixels. Patent Document 1 discloses the configuration of a GS pixel that has multiple charge holding sections for one photoelectric conversion section.

[0003] In addition, by configuring multiple charge storage units for one photoelectric conversion unit and transferring charges to each charge storage unit multiple times during one frame period, multiple pixel signals with different charge accumulation times can be obtained from each pixel. Then, by synthesizing the multiple obtained pixel signals for each pixel, an image with an expanded dynamic range can be obtained. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2013 / 0135486 Summary of the Invention [Problem to be solved by the invention]

[0005] When generating a single image with an expanded dynamic range using multiple pixel signals obtained at different sensitivities, such as by varying the charge accumulation period as in Patent Document 1, or by using different gains or ND filters, the luminance of the obtained pixel signals is usually corrected according to the difference in sensitivity before being combined.

[0006] However, due to the influence of the blinking cycle of the subject light source such as an LED, the brightness ratio of the obtained pixel signals may not be the theoretical ratio according to the difference in sensitivity. If the brightness ratio of the obtained pixel signals deviates from the theoretical ratio, the RGB ratio will be lost when multiple pixel signals are combined, resulting in false colors.

[0007] The present invention has been made in consideration of the above problems, and has an object to suppress the occurrence of false colors when multiple images obtained at different sensitivities are synthesized to expand the dynamic range. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the image processing device of the present invention has an acquisition means for acquiring pixel signals of a plurality of images captured of the same subject at a plurality of different sensitivities using an image sensor having a plurality of pixels, a calculation means for determining a conversion ratio for converting the pixel signals of the plurality of images into pixel signals with uniform sensitivity and a combination ratio for combining the pixel signals of the plurality of images with the uniform sensitivities on a pixel-by-pixel basis, and a combination means for expanding the dynamic range of the pixel signals of the plurality of images on a pixel-by-pixel basis using the conversion ratio and the combination ratio, and the calculation means determines the combination ratio for each divided area obtained by dividing the plurality of pixels. Effect of the Invention

[0009] According to the present invention, it is possible to suppress the occurrence of false colors when a plurality of images having different sensitivities are combined to expand the dynamic range. [Brief description of the drawings]

[0010] [Figure 1]1 is a block diagram showing a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is an equivalent circuit diagram showing the configuration of a pixel according to the first embodiment. [Diagram 3] 5 is a timing chart showing charge accumulation control in the first embodiment. [Figure 4] 5 is a timing chart showing signal read control in the first embodiment. [Diagram 5] 5A to 5C are views for explaining image synthesis processing for expanding a dynamic range according to the first embodiment. [Figure 6] FIG. 2 is a view showing an example of the arrangement of color filters according to the first embodiment. [Figure 7] FIG. 2 is a block diagram showing the configuration of an image synthesis unit according to the first embodiment. [Figure 8] 5 is a flowchart of a synthesis process according to the first embodiment. [Figure 9] FIG. 11 is a block diagram showing the configuration of an image synthesis unit according to a second embodiment. [Figure 10] 10 is a flowchart of image synthesis according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] <First embodiment> 1 is a block diagram showing a schematic configuration of an image pickup device 100 according to the first embodiment. The image pickup device 100 includes an image pickup element (CMOS image sensor) 111 and an image processing unit 112. The image pickup element 111 includes a pixel unit 101, a vertical scanning circuit 102, a column amplifier circuit 103, a horizontal scanning circuit 104, an output circuit 105, and a control circuit 106.

[0013] The pixel section 101 is a pixel array including a plurality of pixels 107 arranged two-dimensionally including a plurality of rows and a plurality of columns when viewed in a plan view of the substrate. The vertical scanning circuit 102 drives the plurality of pixels 107 by supplying control signals to a plurality of transistors included in each of the pixels 107 and controlling the on (conductive state) or off (non-conductive state) of these transistors. A signal line 108 is provided in each column of the pixel section 101, and signals from the pixels 107 are output to the signal line 108 of each column on a row-by-row basis.

[0014] The column amplifier circuit 103 includes an amplifier for amplifying the pixel signal output to the signal line 108, and a processing unit for performing correlated double sampling processing based on the signal at the time of resetting the pixel 107 and the signal at the time of photoelectric conversion. The horizontal scanning circuit 104 supplies a control signal to a switch of the column amplifier circuit 103, and controls the switch to be turned on or off, thereby outputting the pixel signal processed by the column amplifier circuit 103 on a row-by-row basis to the output circuit 105.

[0015] The control circuit 106 controls the vertical scanning circuit 102, the column amplifier circuit 103, and the horizontal scanning circuit 104. The control circuit 106 controls the vertical scanning circuit 102, thereby enabling control of the charge accumulation time of the pixels 107.

[0016] The output circuit 105 includes a buffer amplifier, a differential amplifier, etc., and outputs the pixel signal from the column amplifier circuit 103 to an image processing unit 112 outside the image sensor 111. Note that the image sensor 111 may be configured to output a digital pixel signal by providing an AD conversion unit in the image sensor 111.

[0017] FIG. 2 is an equivalent circuit diagram showing the configuration of each pixel 107 in this embodiment. 2, a photodiode (PD) 1 is an example of a photoelectric conversion unit that generates charges according to the amount of incident light. The charge transfer unit (GS_A) 20, the charge transfer unit (GS_B) 21, the charge transfer unit (TX_A) 28, the charge transfer unit (TX_B) 29, the selection unit (SEL) 17, and the reset unit (RES) 15 are each composed of, for example, MOS transistors.

[0018] When GS_A 20 is turned on, it transfers the charge generated in PD1 to the charge accumulation unit (MEM_A) 22. When GS_B 21 is turned on, it transfers the charge generated in PD1 to the charge accumulation unit (MEM_B) 23. MEM_A 22 and MEM_B 23 are configured to be able to hold the charge generated in PD1.

[0019] When TX_A28 is turned on, it transfers the charge held in MEM_A22 to the floating diffusion region (FD) 14. When TX_B29 is turned on, it transfers the charge held in MEM_B23 to the FD14.

[0020] By turning on RES 15, the FD 14 can be reset to the power supply voltage VDD. In addition, by turning on RES 15, TX_A 28, and TX_B 29 simultaneously, MEM_A 22 and MEM_B 23 can be reset to the power supply voltage VDD.

[0021] When SEL17 is turned on, the corresponding pixel 107 is selected, and a voltage corresponding to the charge of FD14 amplified by a source follower (SF) 16 is output to a signal line 108 connected to the pixel 107.

[0022] The charge drain section (OFG) 18 is configured to drain unnecessary charges from the PD 1, and may be, for example, a MOS transistor. In this case, a semiconductor region of the same polarity as the charges, which constitutes part of the PD 1, serves as the source, and a semiconductor region (overflow drain (OFD) region) to which the power supply voltage VDD is supplied serves as the drain.

[0023] Next, a method for driving the image sensor 111 in this embodiment will be described with reference to FIGS. Figures 3 and 4 show the time series of the transition of the drive pulses supplied to the control electrodes of each transistor shown in Figure 2, with Figure 3 showing the drive related to exposure and Figure 4 showing the drive related to readout. When the drive pulses shown in Figures 3 and 4 are high, each transistor is turned on.

[0024] The subscripts (n, n+1) in Fig. 4 indicate pixel rows (nth row, n+1th row). Driving two rows will be described here, but driving three or more rows is performed by repeating the drive pattern shown in Fig. 4. Meanwhile, since the image sensor 111 is driven by the GS method in this embodiment, the timing of exposure-related driving is the same for all pixels, regardless of row.

[0025] In Figure 3, Tshort i indicates the charge accumulation time corresponding to the i-th charge transfer among the charge transfers repeated multiple times, for example, Nshort times, in the N-th frame. The charge accumulation time of each round corresponds to the time from when the reset of PD1 is released by turning on and off OFG18, when GS_B21 is turned on, when the charge generated from PD1 is transferred to MEM_B23, and when GS_B21 is turned off. In the N-th frame, the total accumulation time Tshort corresponding to the charge accumulated in MEM_B23 is the charge accumulation time Tshort. i This is the sum of the time from i=1 to i=Nshort.

[0026] In addition, the Tlong iindicates the charge accumulation time corresponding to the i-th charge transfer n among the charge transfers repeated multiple times, for example Nlong times, in the N-th frame. The charge accumulation time of each round corresponds to the time from when the reset of PD1 is released by turning on and off OFG18, when GS_A20 is turned on, when the charge generated from PD1 is transferred to MEM_A22, and when GS_A20 is turned off. In the N-th frame, the total accumulation time Tlong corresponding to the charge accumulated in MEM_A22 is the charge accumulation time Tlong i This is the time added from i=1 to i=Nlong.

[0027] Next, the signal read control in the (N+1)-th frame will be described with reference to Fig. 4. Here, the electric charges transferred to MEM_A22 and MEM_B23 in the previous frame (N-th frame) are read. The case where signals are read from the pixels 107 in the nth row in a row-sequential read format will be described.

[0028] First, SEL17(n) is turned on to read out the voltage corresponding to the charge of FD14 of the pixel 107 in the nth row. Then, RES15(n) is turned off to read out the reset level voltage VRES of FD14 (time t0). Next, TX_A28(n) is turned on to transfer the charge held in MEM_A22 to FD14 and read out the signal level VSIG of FD14 (time t1). The difference between these two signal levels, |VSIG-VRES|, is a physical quantity proportional to the amount of charge held in MEM_A22. Hereinafter, this |VSIG-VRES| is referred to as the "long exposure signal Slong."

[0029] After that, RES15(n) is turned on again to reset the FD14, and the reset level voltage VRES of the FD14 is read out (time t2). Next, TX_B29(n) is turned on to transfer the charge held in MEM_B23 to the FD14, and the signal level VSIG of the FD14 is read out (time t3). The difference between these two signal levels, |VSIG - VRES|, is a physical quantity proportional to the amount of charge held in MEM_B23. Hereinafter, this |VSIG - VRES| is referred to as the "short exposure signal Sshort."

[0030] By repeating the above-mentioned driving in row sequence for all or a desired area, the charges held in MEM_A22 and MEM_B23 are read out as a long exposure signal Slong and a short exposure signal Sshort.

[0031] As described above, by configuring two charge accumulation sections for one PD1 to hold the charges to be transferred and controlling the charge accumulation time of the charges transferred to each section, it is possible to obtain an image with a high dynamic range by combining the obtained pixel signals.

[0032] Fig. 5 is a diagram for explaining an overview of the dynamic range expansion synthesis process. Fig. 5(a) shows the output (signal value) obtained for luminance at different charge accumulation times, and Fig. 5(b) shows the output (signal value after correction) obtained for luminance when the dynamic range is expanded.

[0033] As an example, the long exposure signal Slong obtained with a long charge accumulation time Tlong is basically used, and for bright, blown-out high-luminance parts, the short exposure signal Sshort obtained with a short charge accumulation time Tshort is corrected and used for synthesis. However, depending on the charge accumulation time, the short exposure signal Sshort obtained with the charge accumulation time Tshort may be used as the basis, and for dark, crushed low-luminance parts, the long exposure signal Slong obtained with the charge accumulation time Tlong may be corrected and used for synthesis. Alternatively, the charge accumulation time Tlong and the short exposure signal Sshort may be corrected and synthesized to obtain a signal equivalent to an intermediate sensitivity.

[0034] Specifically, when the signal level is lower than a predetermined level Lth, the long exposure signal Slong is used, and when the signal level is equal to or higher than the predetermined level Lth, the short exposure signal Sshort is corrected and used. In this way, in order to correct the time difference between the charge accumulation time Tlong and the charge accumulation time Tshort, the signal is corrected using a conversion ratio R (=Tlong / Tshort), which is the ratio between the charge accumulation time Tlong and the charge accumulation time Tshort, before combining. For example, when the ratio between the charge accumulation time Tlong and the accumulation time Tshort is 4:1, the short exposure signal Sshort is multiplied by 4 and combined.

[0035] In addition, in the synthesis process, by using alpha blending (weighted addition), it is possible to smoothly connect images so that the switching parts are not noticeable. Figure 5(c) is a diagram showing an example of alpha, where the synthesis ratio is changed according to the brightness.

[0036] In this embodiment, alpha blending is performed, and the composite pixel signal is expressed as S HDR Then, it can be expressed by the following formula (1). S HDR =(1-α)×Slong+α×R×Sshort (where 0≦α≦1) …(1)

[0037] The synthesis process has been described above, but as long as multiple images obtained with different sensitivities can be obtained for synthesis, the sensor does not necessarily have to have a GS function.

[0038] 6 is a diagram showing an example of the arrangement of color filters provided on the pixel unit 101. Here, a Bayer array of color filters is shown as an example. Reference numeral 601 denotes an R color filter, 602 and 603 denote G color filters, and 604 denotes a B color filter. A short exposure signal Sshort and a long exposure signal Slong are output from each of the pixels 107 covered with each color filter. In this embodiment, a composition process is performed with the same composition ratio (α) for each pixel block 600, each of which is a set of four pixels covered with color filters 601 to 604.

[0039] Fig. 7 is a block diagram showing the configuration of an image synthesis unit included in the image processing unit 112 for performing image synthesis processing in the first embodiment. As shown in Fig. 7, the image synthesis unit includes a synthesis ratio calculation unit 700, a sensitivity correction unit 701, and a synthesis processing unit 702.

[0040] The long exposure signal Slong is input to the composite ratio calculation unit 700 for each pixel block 600, and the composite ratio is calculated based on the maximum luminance of the input long exposure signal Slong. Here, α shown in formula (1) is calculated. As a method of calculating α, a table as shown in FIG. 5(c) may be stored and a value corresponding to the maximum luminance may be read out, or an approximation formula for calculating α as a function of the maximum luminance may be stored and calculated. In addition, when correcting and synthesizing the long exposure signal Slong based on the short exposure signal Sshort, the short exposure signal Sshort is input to the composite ratio calculation unit 700, and the composite ratio is calculated based on the minimum luminance of the input short exposure signal Sshort.

[0041] The short exposure signal Sshort is input to the sensitivity correction unit 701 for each pixel block 600. Then, a sensitivity correction process is performed on each input short exposure signal Sshort to convert it into a signal with the same sensitivity as the long exposure signal Slong according to a conversion ratio R (=Tlong / Tshort) which is the ratio between the charge accumulation time Tlong and the charge accumulation time Tshort. The long exposure signal Slong for each pixel block 600 and the short exposure signal S'short (=Sshort×R) whose sensitivity has been corrected by the sensitivity correction unit 701 are input to the synthesis processing unit 702. Then, using the synthesis ratio α obtained by the synthesis ratio calculation unit 700 for each pixel block 600, a synthesis process is performed on the long exposure signal Slong and the sensitivity-corrected short exposure signal S'short for each pixel 107 included in the pixel block 600.

[0042] FIG. 8 is a flowchart of the synthesis process in this embodiment described above. First, in S800, the sensitivity correction unit 701 acquires information on sensitivity from the control circuit 106. Here, the charge accumulation time Tlong and the charge accumulation time Tshort are acquired. Note that the control circuit 106 may obtain a conversion ratio R (=Tlong / Tshort) which is the ratio between the charge accumulation time Tlong and the charge accumulation time Tshort, and acquire the ratio R.

[0043] Next, in S801, the long exposure signal Slong and the short exposure signal Sshort are input from one pixel block 600, and the blending ratio calculation unit 700, the sensitivity correction unit 701, and the blending processing unit 702 each detect the luminance of the input signal.

[0044] In S802, the composite ratio calculation unit 700 detects the maximum luminance among the luminance of the input long exposure signal Slong for four pixels, and in the next S803, the composite ratio calculation unit 700 calculates a composite ratio α according to the maximum luminance detected in S802.

[0045] In S804, dynamic range expansion synthesis processing is performed according to the charge accumulation time Tlong and charge accumulation time Tshort acquired in S800 and the synthesis ratio α calculated in S803. Here, the sensitivity correction unit 701 corrects the luminance level of the short exposure signal Sshort based on the charge accumulation time Tlong and charge accumulation time Tshort, and then the synthesis processing unit 702 weights and synthesizes the long exposure signal Slong and the corrected short exposure signal S'short using the synthesis ratio α. In this embodiment, the dynamic range expansion synthesis processing is performed for each pixel included in the same pixel block 600 using the same synthesis ratio α.

[0046] In S805, it is determined whether all pixel blocks 600 have been processed. If there are unprocessed pixel blocks 600, the process returns to S801 and the above process is repeated. If all pixel blocks 600 have been processed, the synthesis process ends.

[0047] As described above, according to the first embodiment, false colors can be suppressed by performing dynamic range expansion composition processing using the same composition ratio for every four pixels covered by color filters that make up one pattern of a Bayer array.

[0048] <Second embodiment> Next, a second embodiment according to the present disclosure will be described. The second embodiment is different from the first embodiment in the configuration of the image synthesis unit shown in Fig. 7. Other configurations, pixel driving methods, and basic dynamic range expansion synthesis processing are similar to those described with reference to Figs. 1 to 6, so the same reference numbers are used and descriptions are omitted.

[0049] Fig. 9 is a block diagram showing the configuration of an image synthesis unit included in the image processing unit 112 for performing image synthesis processing in the second embodiment. A luminance comparison unit 903 is added to the configuration shown in Fig. 7, and the synthesis ratio calculation unit 900 calculates the synthesis ratio by further using the comparison result by the luminance comparison unit 903. Since the configuration other than these is the same as the configuration shown in Fig. 7, the same reference numerals are used and the description will be omitted.

[0050] The luminance comparison unit 903 receives the long exposure signal Slong for each pixel block 600 and the short exposure signal S'short whose sensitivity has been corrected by the sensitivity correction unit 901. Then, it calculates the luminance difference between the maximum luminance and the minimum luminance of the input long exposure signal Slong and the short exposure signal S'short whose sensitivity has been corrected. The combination ratio calculation unit 900 adaptively changes the calculation method of the combination ratio α using the luminance difference calculated by the luminance comparison unit 903. Specifically, when the luminance difference is equal to or greater than a predetermined threshold, the combination ratio α is calculated for the pixel block 600, and when the luminance difference is less than the predetermined threshold, the combination ratio α is calculated for each pixel.

[0051] Fig. 10 is a flowchart of the synthesis process in the second embodiment. Note that in the flowchart of Fig. 10, the same processes as those explained in Fig. 8 are given the same reference numerals and the explanation thereof will be omitted.

[0052] In S1001, as in the first embodiment, a long exposure signal Slong and a short exposure signal Sshort are input from one pixel block 600, and the blending ratio calculation unit 900, the brightness comparison unit 903, the sensitivity correction unit 701, and the blending processing unit 702 each detect the brightness of the input signals.

[0053] Next, in S1002, the luminance comparison unit 903 obtains a luminance difference between the maximum luminance and the minimum luminance of the luminance of the short exposure signal S'short and the luminance of the long exposure signal Slong, the sensitivity of which has been corrected by the sensitivity correction unit 701. Then, the blending ratio calculation unit 900 judges whether the obtained luminance difference is equal to or greater than a predetermined threshold. If the luminance difference is equal to or greater than the predetermined threshold, the process of S802 to S804 is performed.

[0054] On the other hand, if the luminance difference is less than the predetermined threshold in S1002, the process proceeds to S1003. In S1003, the blending ratio calculation unit 900 calculates a blending ratio α for each pixel 107 included in the pixel block 600. Then, in S804, the dynamic range expansion blending process is performed for each pixel 107 using the blending ratio α calculated for each pixel 107 in S1003.

[0055] As described above, according to the second embodiment, in a scene that is not affected by the blinking cycle, it is possible to calculate the blending ratio for each pixel and blend the images. This makes it possible to suppress false colors when there is an effect of the blinking cycle, and to improve the S / N ratio more than in the first embodiment when there is no effect of the blinking cycle.

[0056] <Other embodiments> In the above embodiment, four pixels covered by R, G, G, and B filters have been described as one unit of processing, but for example, the screen may be divided into a matrix of multiple pixels, and the processing of this embodiment may be performed for each divided area. Furthermore, the size and position of the multiple pixel area may be changed for each frame.

[0057] In addition, in the above-described embodiment, the combination ratio is calculated based on the maximum luminance of the input signals. However, the combination ratio may be calculated based on the maximum luminance, taking into consideration that, for example, a white balance gain or the like is applied during the development process or the like.

[0058] Furthermore, in the above embodiment, the pixels are covered with primary color filters, but it is also possible to use complementary color filters.

[0059] In the above embodiment, the case where two types of pixel signals obtained by different charge accumulation times are synthesized has been described, but the method of changing the sensitivity is not limited to the method of varying the charge accumulation time, and for example, the gain value or the transmittance of the ND filter may be changed. In this case, the ratio between sensitivities such as between gain values ​​or transmittances may be used as the conversion ratio R. In addition, the types of pixel signals (types of exposure conditions) used in the synthesis process may be two or more types.

[0060] Furthermore, the present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0061] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0062] <Summary> The disclosure of this embodiment includes the following configuration.

[0063] (Item 1) an acquisition means for acquiring pixel signals of a plurality of images obtained by photographing the same subject at a plurality of different sensitivities using an image sensor having a plurality of pixels; a calculation means for calculating a conversion ratio for converting pixel signals of the plurality of images into pixel signals having uniform sensitivity, and a synthesis ratio for synthesizing the pixel signals of the plurality of images having uniform sensitivity for each pixel; a synthesis unit for synthesizing pixel signals of the plurality of images for each pixel to expand a dynamic range by using the conversion ratio and the synthesis ratio; The image processing device according to claim 1, wherein the calculation means calculates the blending ratio for each divided area obtained by dividing the plurality of pixels. (Item 2) 2. The image processing device according to item 1, wherein the divided regions correspond to repeating units of an array of color filters covering the image sensor. (Item 3) 3. The image processing device according to item 2, wherein the color filter is a Bayer array color filter. (Item 4) 4. The image processing device according to any one of items 1 to 3, wherein the calculation means calculates the blending ratio for each of the divided regions based on a maximum luminance among luminances of pixel signals of the plurality of images. (Item 5) 5. The image processing device according to item 4, wherein the calculation means determines the conversion ratio for aligning pixel signals of the plurality of images to the highest sensitivity. (Item 6) 4. The image processing device according to any one of items 1 to 3, wherein the calculation means calculates the blending ratio for each of the divided regions based on a minimum luminance among luminances of pixel signals of the plurality of images. (Item 7) 7. The image processing device according to item 6, wherein the calculation means determines the conversion ratio for aligning pixel signals of the plurality of images to the lowest sensitivity. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the calculation means determines the conversion ratio based on a ratio between the sensitivities. (Item 9) a determining unit for determining whether a difference between a maximum luminance and a minimum luminance of the luminance of the pixel signals whose sensitivities have been made uniform by the conversion ratio is equal to or greater than a predetermined threshold value for each of the divided regions; The image processing device according to any one of items 1 to 8, characterized in that the calculation means calculates the blending ratio for the divided region when the brightness difference is equal to or greater than the threshold, and calculates the blending ratio for each pixel of the divided region when the brightness difference is less than the threshold. (Item 10) An imaging element; An image processing device according to any one of items 1 to 9, An imaging device comprising: (Item 11) 11. The imaging device according to item 10, wherein the sensitivity is controlled by varying at least one of a charge accumulation time of the imaging element, a gain value, and a transmittance of a filter that transmits incident light. (Item 12) an acquisition step of acquiring pixel signals of a plurality of images obtained by photographing the same subject at a plurality of different sensitivities using an image sensor having a plurality of pixels; a calculation step of calculating a conversion ratio for converting pixel signals of the plurality of images into pixel signals having uniform sensitivity, and a synthesis ratio for synthesizing the pixel signals of the plurality of images having uniform sensitivity for each pixel; a synthesis unit for synthesizing pixel signals of the plurality of images for each pixel to expand a dynamic range by using the conversion ratio and the synthesis ratio, an image processing method comprising the steps of: obtaining the blending ratio for each divided area obtained by dividing the plurality of pixels; (Item 13) an acquisition step of acquiring pixel signals of a plurality of images obtained by photographing the same subject at a plurality of different sensitivities using an image sensor having a plurality of pixels; a first calculation step of calculating a conversion ratio for converting pixel signals of the plurality of images into pixel signals having uniform sensitivity; a determination step of determining whether or not a luminance difference between a maximum luminance and a minimum luminance among luminances of pixel signals whose sensitivities have been made uniform by the conversion ratio is equal to or greater than a predetermined threshold value for each divided region obtained by dividing the plurality of pixels; a second calculation step of calculating a synthesis ratio for synthesizing pixel signals of the plurality of images having the same sensitivity for each pixel; a synthesis unit for synthesizing pixel signals of the plurality of images for each pixel to expand a dynamic range by using the conversion ratio and the synthesis ratio, An image processing method characterized in that in the second calculation step, when the brightness difference is equal to or greater than the threshold, the synthesis ratio for the divided area is calculated, and when the brightness difference is less than the threshold, the synthesis ratio is calculated for each pixel of the divided area. (Item 14) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 9. (Item 15) Item 15. A computer-readable storage medium storing the program according to item 14.

[0064] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0065] 100: imaging device, 101: pixel section, 102: vertical scanning circuit, 103: column amplifier circuit, 104: horizontal scanning circuit, 105: output circuit, 106: control circuit, 107: pixel, 108: signal line, 111: imaging element, 112: image processing section, 600: pixel block, 601 to 604: color filters, 700, 900: composite ratio calculation section, 701: sensitivity correction section, 702: composite processing section, 903: brightness comparison section

Claims

1. an acquisition means for acquiring pixel signals of a plurality of images obtained by photographing the same subject at a plurality of different sensitivities using an image sensor having a plurality of pixels; a calculation means for calculating a conversion ratio for converting pixel signals of the plurality of images into pixel signals having uniform sensitivity, and a synthesis ratio for synthesizing the pixel signals of the plurality of images having uniform sensitivity for each pixel; a synthesis unit for synthesizing pixel signals of the plurality of images for each pixel to expand a dynamic range by using the conversion ratio and the synthesis ratio; The image processing device according to claim 1, wherein the calculation means calculates the blending ratio for each divided area obtained by dividing the plurality of pixels.

2. 2. The image processing apparatus according to claim 1, wherein the divided regions correspond to repeating units of an array of color filters covering the image sensor.

3. 3. The image processing device according to claim 2, wherein the color filters are color filters in a Bayer array.

4. 2 . The image processing apparatus according to claim 1 , wherein the calculation means calculates the blending ratio for each of the divided regions based on a maximum luminance among luminances of pixel signals of the plurality of images.

5. 5. The image processing apparatus according to claim 4, wherein said calculation means determines said conversion ratio for unifying pixel signals of said plurality of images to the highest sensitivity.

6. 2 . The image processing apparatus according to claim 1 , wherein the calculation means calculates the blending ratio for each of the divided regions based on a minimum luminance among luminances of pixel signals of the plurality of images.

7. 7. The image processing apparatus according to claim 6, wherein said calculation means determines said conversion ratio for unifying pixel signals of said plurality of images to the lowest sensitivity.

8. 2. The image processing apparatus according to claim 1, wherein the calculation means determines the conversion ratio based on a ratio between the sensitivities.

9. a determining unit that determines whether a luminance difference between a maximum luminance and a minimum luminance of the luminance of the pixel signals whose sensitivities have been made uniform by the conversion ratio is equal to or greater than a predetermined threshold value for each of the divided regions; 2. The image processing device according to claim 1, wherein the calculation means calculates the blending ratio for the divided region when the luminance difference is equal to or greater than the threshold value, and calculates the blending ratio for each pixel of the divided region when the luminance difference is less than the threshold value.

10. An imaging element; The image processing device according to any one of claims 1 to 9, An imaging device comprising:

11. 11. The image pickup apparatus according to claim 10, wherein the sensitivity is controlled by varying at least one of a charge accumulation time of the image pickup element, a gain value, and a transmittance of a filter that transmits incident light.

12. an acquisition step of acquiring pixel signals of a plurality of images obtained by photographing the same subject at a plurality of different sensitivities using an image sensor having a plurality of pixels; a calculation step of calculating a conversion ratio for converting pixel signals of the plurality of images into pixel signals having uniform sensitivity, and a synthesis ratio for synthesizing the pixel signals of the plurality of images having uniform sensitivity for each pixel; a synthesis unit for synthesizing pixel signals of the plurality of images for each pixel to expand a dynamic range by using the conversion ratio and the synthesis ratio; an image processing method comprising the steps of: obtaining the blending ratio for each divided area obtained by dividing the plurality of pixels;

13. an acquisition step of acquiring pixel signals of a plurality of images obtained by photographing the same subject at a plurality of different sensitivities using an image sensor having a plurality of pixels; a first calculation step of calculating a conversion ratio for converting pixel signals of the plurality of images into pixel signals having uniform sensitivity; a determination step of determining whether or not a luminance difference between a maximum luminance and a minimum luminance among luminances of pixel signals whose sensitivities have been made uniform by the conversion ratio is equal to or greater than a predetermined threshold value for each divided region obtained by dividing the plurality of pixels; a second calculation step of calculating a synthesis ratio for synthesizing pixel signals of the plurality of images having the same sensitivity for each pixel; a synthesis step of synthesizing pixel signals of the plurality of images for each pixel with an expanded dynamic range by using the conversion ratio and the synthesis ratio, An image processing method characterized in that, in the second calculation step, when the brightness difference is equal to or greater than the threshold, the synthesis ratio of the divided area is calculated, and when the brightness difference is less than the threshold, the synthesis ratio is calculated for each pixel of the divided area.

14. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 9.

15. A computer-readable storage medium storing the program according to claim 14.

Images