Camera device and image processing method

Abstract

The imaging device (100) includes an imaging unit (120) and an image processing unit (140). The imaging unit (120) includes: a plurality of pixels (10), each including a photoelectric conversion unit (13) and a charge accumulation node (41) for accumulating signal charges; and a signal readout circuit for reading out a signal corresponding to the signal charges accumulated in the charge accumulation node (41). The imaging unit (120) outputs image data based on the signal, and the image processing unit (140) processes the image data output by the imaging unit (120). The imaging unit (120) reads out the image data multiple times in a non-destructive manner, thereby sequentially outputting a plurality of image data within a frame period. The image processing unit (140) obtains a difference between two image data to generate differential image data, selects output image data (IS) from the initial image data and the differential image data, and synthesizes the output image data (IS) with normal readout image data (IL) included in the plurality of image data to generate synthesized result image data (IHDR).

Description

Technical Field

[0001] The present disclosure relates to a camera device and an image processing method. Background Art

[0002] In recent years, solid-state imaging elements such as CCD (Charge Coupled Device) sensors and CMOS (Complementary Metal Oxide Semiconductor) sensors have become increasingly popular as imaging devices that capture images of subjects and generate image data. The dynamic range of these solid-state imaging elements for incident light (i.e., the brightness range from the level at which the signal cannot be extracted due to the background noise to the saturation level) has been continuously improved with the advancement of semiconductor technology. However, in situations where solid-state imaging elements are actually used, incident light that exceeds the pre-set dynamic range sometimes occurs. Therefore, technologies for expanding the dynamic range are being vigorously studied.

[0003] Patent Document 1 discloses a technique for synthesizing image data with different exposure times to obtain a wide dynamic range. Patent Document 2 discloses a technique for synthesizing image data obtained from a plurality of imaging units with different sensitivities arranged in one pixel to expand the dynamic range.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 62-108678

[0007] Patent Document 2: Japanese Patent Application Publication No. 2008-99073 Summary of the invention

[0008] Problems to be Solved by the Invention

[0009] The present disclosure provides an imaging device and the like that can effectively expand the dynamic range of an image being captured.

[0010] Means for solving problems

[0011] An imaging device according to one embodiment of the present disclosure includes an imaging unit and a processing circuit, wherein the imaging unit includes: a plurality of pixels, each including a photoelectric conversion unit for generating a signal charge and a charge accumulation unit for accumulating the signal charge; and a signal readout circuit for reading out a signal corresponding to the signal charge accumulated in the charge accumulation unit, wherein the imaging unit outputs image data based on the signal, and the processing circuit processes the image data output by the imaging unit, wherein the imaging unit reads out the image data multiple times in a non-destructive manner, thereby sequentially outputting a plurality of image data within a frame period, wherein the processing circuit calculates a difference between two image data output continuously among the plurality of image data to generate at least one differential image data, selects one image data as the first image data from the initial image data outputted first among the plurality of image data and the at least one differential image data based on their respective brightness values, and synthesizes the first image data with the second image data which is one of the image data included in the plurality of image data except the initial image data to generate the third image data.

[0012] In an image processing method involved in one embodiment of the present invention, a plurality of image data obtained by being read out multiple times in a non-destructive manner during one frame period are sequentially input, a difference between two consecutively input image data among the plurality of image data is calculated, thereby generating at least one differential image data, and one image data is selected as the first image data from among the initial image data first input among the plurality of image data and the at least one differential image data based on their respective brightness values, and the first image data is synthesized with the second image data which is one of the image data included in the plurality of image data except the initial image data, thereby generating the third image data.

[0013] Effects of the Invention

[0014] The imaging device and the like according to the present disclosure can effectively expand the dynamic range of an image being captured. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a timing chart showing an example of the operation timing of the imaging device according to the comparative example.

[0016] Figure 2 It is a diagram showing an example of an image captured by the imaging device according to the comparative example.

[0017] Figure 3 This is a block diagram showing an exemplary configuration of the imaging device according to the first embodiment.

[0018] Figure 4 This is a schematic diagram showing an exemplary circuit configuration of the imaging unit according to the first embodiment.

[0019] Figure 5 This is a schematic cross-sectional view showing an exemplary device structure of a pixel according to Embodiment 1.

[0020] Figure 6 This is a timing chart showing an example of the operation in the imaging unit according to the first embodiment.

[0021] Figure 7 This is a timing chart showing an example of control signals during a signal readout period.

[0022] Figure 8 This is a block diagram showing an exemplary structure of an image selection unit according to the first embodiment.

[0023] Fig. 9 This is a diagram for explaining an example of the operation of the image selection unit according to the first embodiment.

[0024] Fig.10 This is a diagram for explaining an example of changes in the pixel signal level of the charge accumulation node according to the first embodiment.

[0025] Fig.11 It is a diagram for explaining the change of the pixel signal level when the non-destructive readout is not performed.

[0026] Fig.12 This is a diagram for explaining an example of the operation of the comparison unit according to the first embodiment.

[0027] Fig.13 This is a diagram for explaining another example of the change in the pixel signal level of the charge accumulation node according to the first embodiment.

[0028] Fig.14 This is a block diagram showing an exemplary structure of an image selection unit according to the second embodiment.

[0029] Fig.15 This is a flowchart showing an example of the operation of the selection control unit involved in the second embodiment.

[0030] Fig.16 This is a schematic diagram showing an exemplary circuit configuration of an imaging unit according to the third embodiment. DETAILED DESCRIPTION

[0031] (Insights that form the basis of this disclosure)

[0032] Before describing the present embodiment in detail, an operation example of an imaging device in a conventional dynamic range expansion method will be described.

[0033] In addition, in this specification, an image with an expanded dynamic range is sometimes referred to as a "high dynamic range image". In addition, the acquisition of image data related to the expansion of the dynamic range and the image processing related to the expansion of the dynamic range are sometimes expressed by terms such as "high dynamic range imaging", "high dynamic range processing" or "high dynamic range synthesis", respectively.

[0034] As a method for obtaining a high dynamic range image, for example, a method of synthesizing a plurality of image data captured under different exposure conditions into one has been proposed. In this method, for example, the same scene is captured under two conditions, namely, a long exposure time and a short exposure time. In an image captured with a long exposure time, a portion of a subject with high brightness is prone to being overexposed due to an excess of light, while a portion of a subject with low brightness is not buried in noise and appears brighter. In an image captured with a short exposure time, a portion of a subject with low brightness is prone to being buried in noise due to an insufficient amount of light, while a portion of a subject with high brightness is not overexposed and can be obtained. Hereinafter, an image captured with a long exposure time is sometimes referred to as a "long exposure image". In addition, an image captured with a short exposure time is sometimes referred to as a "short exposure image". By synthesizing the image data of each image into one image data in such a manner that a portion with high brightness is represented by a short exposure image and a portion with low brightness is represented by a long exposure image, a high dynamic range image can be obtained.

[0035] A plurality of image data with different exposure conditions can be obtained, for example, by capturing images with different exposure times for each pixel within the imaging plane of an imaging element. Figure 1 1 is a timing chart showing an example of the operation timing of the imaging device when imaging is performed with different exposure times for each row of the imaging device in the imaging device according to the comparative example. Figure 1 , a rolling shutter operation is shown in which exposure and readout are performed sequentially in units of horizontal rows of pixels in a CMOS sensor.

[0036] Figure 1 (a) is a diagram showing the timing of the falling (or rising) of the vertical synchronization signal VD. Figure 1 (b) is a diagram showing the timing of the falling (or rising) of the horizontal synchronization signal HD. Figure 1 (c) is a diagram showing the timing of the readout operation and shutter operation of each horizontal pixel row from the R0th row to the R7th row of the imaging element. Figure 1 In the figure of each horizontal pixel row of (c), the pixel readout operation is performed at the timing indicated by the white rectangle, and the electronic shutter operation is performed at the timing indicated by the rectangle with the vertical line. That is, the exposure period of each pixel belonging to each horizontal pixel row becomes the period from the end of the shutter operation to the start of the next readout operation in the horizontal pixel row.

[0037] exist Figure 1 In the example shown, the exposure time of each pixel belonging to the R0th row, the R2th row, the R4th row, and the R6th row is the exposure time TeL shown in the figure. In addition, the exposure time of each pixel belonging to the R1th row, the R3th row, the R5th row, and the R7th row is the exposure time TeS shown in the figure. Figure 1 In this way, the exposure time TeL> exposure time TeS. In this way, the image is taken with different exposure times for each pixel, and the output image data of the pixel taken with the exposure time TeL is synthesized with the output image data of the pixel taken with the exposure time TeS, thereby obtaining an image with a high dynamic range. Figure 1 The pixels belonging to the R0th row, R2th row, R4th row and R6th row that are photographed with exposure time TeL are called "long exposure pixels", and the pixels belonging to the R1th row, R3th row, R5th row and R7th row that are photographed with exposure time TeS are called "short exposure pixels".

[0038] according to Figure 1 It can be understood that the long exposure pixel and the short exposure pixel have a period in which their exposure does not overlap. The length of the period in which their exposure overlaps, that is, the period in which the long exposure pixel and the short exposure pixel are simultaneously exposed, is at most the exposure time TeS of the short exposure pixel. Events occurring during the period in which the long exposure pixel and the short exposure pixel do not overlap are recorded only by the long exposure pixel, and not by the short exposure pixel.

[0039] Figure 2 It is a diagram showing an example of an image captured by the imaging device according to the comparative example. Figure 2 The left side of the "light source" is a diagram showing the lighting timing of the light source, and the right side is a diagram showing the light source pattern 200 of the light source. Figure 2 The left side of the "long exposure pixel" is a diagram showing the exposure timing of the long exposure pixel, and the right side is a diagram showing the long exposure image 201 captured by the long exposure pixel. Figure 2 The left side of the “short-time exposure pixel” is a diagram showing the exposure timing of the short-time exposure pixel, and the right side thereof is a diagram showing the short-time exposure image 202 captured by the short-time exposure pixel.

[0040] For example, Figure 2As shown, when the light source serving as the subject has both a lighting period and an off period, and further, the light source is only lit outside the exposure time TeS of the short-exposure pixel, the light from the light source is only acquired by the long-exposure pixel, and not by the short-exposure pixel. In this way, depending on the length of the lighting period of the light source, the light receiving sensitivity of each pixel, and the gain setting of the output image data, the following situation may occur: the light from the light source is not reflected in the short-exposure image 202 captured by the short-exposure pixel, while the light from the light source is reflected in an overexposed state in the long-exposure image 201 captured by the long-exposure pixel. For example, in Figure 2 In the example, although the light source has a pattern like the light source pattern 200, since the light source is only lit for a certain duration during the period when only the long-exposure pixels are exposed, the light source appears in an overexposed state in the long-exposure image 201. On the other hand, the light source itself is not shown in the short-exposure image 202. As a result, the information of the light source pattern 200 cannot be obtained in any image. Therefore, the light source pattern 200 cannot be depicted in the synthesized output image, and the desired dynamic range expansion effect cannot be obtained. That is, in the camera device involved in the comparative example, the dynamic range expansion effect of the camera device is reduced depending on the lighting conditions of the shooting scene, and the dynamic range of the captured image cannot be effectively expanded.

[0041] Therefore, the present disclosure provides an imaging device that can effectively expand the dynamic range of an image being captured. For example, the present disclosure provides an imaging device whose dynamic range expansion performance is not easily reduced even when the brightness of the object changes with time as described above.

[0042] An outline of one embodiment of the present disclosure is as follows.

[0043] An imaging device according to one embodiment of the present disclosure includes an imaging unit and a processing circuit, wherein the imaging unit includes: a plurality of pixels, each including a photoelectric conversion unit for generating a signal charge and a charge accumulation unit for accumulating the signal charge; and a signal readout circuit for reading out a signal corresponding to the signal charge accumulated in the charge accumulation unit, wherein the imaging unit outputs image data based on the signal, and the processing circuit processes the image data output by the imaging unit, wherein the imaging unit reads out the image data multiple times in a non-destructive manner, thereby sequentially outputting a plurality of image data within a frame period, wherein the processing circuit calculates a difference between two image data output continuously among the plurality of image data to generate at least one differential image data, selects one image data as the first image data from the initial image data outputted first among the plurality of image data and the at least one differential image data based on their respective brightness values, and synthesizes the first image data with the second image data which is one of the image data included in the plurality of image data except the initial image data to generate the third image data.

[0044] Thus, at least one differential image data of an exposure time equivalent to the image data photographed with an exposure time shorter than the exposure time within one frame period is generated based on the difference between the image data continuously outputted from the imaging unit. Furthermore, the first image data is selected from the initial image data and the differential image data based on the brightness value, so that even if it is the image data of the exposure time equivalent to the image data photographed with a short exposure time, the image data suitable for use in high dynamic range synthesis, for example, with a high brightness value, is selected. Then, by synthesizing the first image data with the second image data, which is the image data photographed with an exposure time longer than the first image data without taking the difference, the third image data with an expanded dynamic range can be obtained. In addition, the imaging unit reads out a plurality of image data multiple times in a non-destructive manner, so that even if the brightness of the subject of the imaging scene changes in time, the plurality of image data includes image data exposed in a time period with high brightness. Therefore, in the differential image data, even if it is the image data of the exposure time equivalent to the image data photographed with a short exposure time, the image data with a high brightness value is included. That is, in the synthesis of the third image data, the difference image data having a high brightness value as the first image data is used, so that even if the brightness of the subject in the shooting scene changes with time, the dynamic range expansion performance is not easily reduced. Thus, the camera device of this mode can effectively expand the dynamic range of the image being shot.

[0045] In addition, the first image data and the second image data are a plurality of image data output from the imaging unit, or differential image data of the plurality of image data, and therefore are image data based on signals of the same pixel. Therefore, in order to obtain image data with different exposure times, it is not necessary to divide the function of the pixel in terms of time or position. Thus, the imaging device of this mode does not cause an increase in frame rate or a decrease in resolution, and can effectively expand the dynamic range of the image being photographed.

[0046] In addition, for example, the second image data may be image data output last among the plurality of image data.

[0047] Therefore, the second image data is the image data with the longest exposure time among the multiple image data output from the camera unit. Therefore, even if there is a low-brightness part in the subject, the exposure time of the second image data is ensured, and the third image data with reduced noise can be synthesized.

[0048] In addition, for example, the processing circuit may select one of the image data from the initial image data and the initial differential image data during the one frame among the at least one differential image data based on their respective brightness values, and repeatedly select one of the image data from the selected image data and the newly generated differential image data based on their respective brightness values ​​to select the first image data.

[0049] Thus, even when the number of multiple image data output from the camera unit increases, one of the image data is repeatedly selected from the two image data, the selected image data and the newly generated difference image data. Thus, only two image data need to be retained for selection. Therefore, the memory required for storing image data can be reduced.

[0050] Furthermore, for example, when the brightness value of the newly generated difference image data is smaller than the brightness value of the previously selected image data, the processing circuit may select the previously selected image data as the first image data.

[0051] In this way, when the brightness value of the newly generated difference image data is smaller than the brightness value of the previously selected image data, the signal charge accumulated in the charge accumulation section is close to saturation, so the brightness value of the difference image data is likely to be smaller. Therefore, the difference image data generated later is also likely to have a smaller brightness value, and even if the previously selected image data is selected as the first image data without comparing with the difference image data generated later, the selected first image data is likely to remain unchanged. Therefore, by selecting the previously selected image data as the first image data, it is no longer necessary to generate the difference image data later, so that the processing speed can be improved and the power consumption can be reduced.

[0052] In addition, for example, the processing circuit may include: a first memory for storing one of two image data output continuously from the plurality of image data output by the camera unit, and a second memory for storing the other of the two image data; a difference operation circuit for calculating the difference between the two image data stored in the first memory and the second memory, thereby generating difference image data; a third memory for storing the initial image data or the difference image data generated by the difference operation circuit; and a comparison circuit for comparing the brightness value of the difference image data generated by the difference operation circuit with the brightness value of the initial image data or the difference image data stored in the third memory.

[0053] Thus, by storing two image data continuously output from the camera unit in the first memory and the second memory and storing the initial image data or the difference image data in the third memory, the comparison circuit compares the brightness values. Thus, the dynamic range of the image being captured can be effectively expanded by the camera device with a simple structure.

[0054] Furthermore, for example, the photoelectric conversion unit may include a first electrode, a second electrode, and a photoelectric conversion layer between the first electrode and the second electrode.

[0055] Thus, in an imaging device including a photoelectric conversion unit including a photoelectric conversion layer, the dynamic range of an image to be captured can be effectively expanded.

[0056] In addition, for example, the photoelectric conversion unit may be a photodiode.

[0057] Thus, in an imaging device including a photoelectric conversion unit including a photodiode, the dynamic range of an image to be captured can be effectively expanded.

[0058] In addition, in an image processing method involved in one embodiment of the present invention, a plurality of image data obtained by being read out multiple times in a non-destructive manner during one frame period are sequentially input, and the difference between two continuously input image data among the plurality of image data is calculated to generate at least one difference image data, and one image data is selected as the first image data from among the initial image data initially input among the plurality of image data and the at least one difference image data based on their respective brightness values, and the first image data is synthesized with the second image data which is one of the image data included in the plurality of image data except the initial image data to generate the third image data.

[0059] Thus, at least one differential image data of an exposure time equivalent to the image data photographed with an exposure time shorter than the exposure time within one frame period is generated based on the difference between the continuously input image data. Furthermore, the first image data is selected from the initial image data and the differential image data based on the brightness value, so that even if it is the image data of the exposure time equivalent to the image data photographed with a short exposure time, image data suitable for use in high dynamic range synthesis, for example, high brightness value is selected. By synthesizing the first image data with the second image data, which is image data photographed with an exposure time longer than the first image data without taking a difference, the third image data with an expanded dynamic range can be obtained. In addition, since a plurality of image data are input by reading out a plurality of times in a non-destructive manner, even if the brightness of the subject of the photographing scene changes in time, the plurality of image data includes image data exposed in a time period with high brightness. Therefore, in the differential image data, even if it is the image data of the exposure time equivalent to the image data photographed with a short exposure time, image data with high brightness is included. That is, in the synthesis of the third image data, the difference image data having a high brightness value as the first image data is used, so that even if the brightness of the subject in the shooting scene changes with time, the dynamic range expansion performance is not easily reduced. Therefore, the image processing method of this mode can effectively expand the dynamic range of the image being shot.

[0060] In addition, for example, the second image data may be image data output last within the one frame period among the plurality of image data.

[0061] Therefore, the second image data has the longest exposure time among the multiple image data input. Therefore, even if there is a low-brightness part in the subject, the exposure time of the second image data is ensured, and the third image data with reduced noise can be synthesized.

[0062] In addition, for example, the image processing method may select one of the image data based on the respective brightness values ​​from the initial image data and the initial differential image data during the one frame among the at least one differential image data, and repeatedly select one of the image data based on the respective brightness values ​​from the selected image data and the newly generated differential image data to select the first image data.

[0063] Thus, even when the number of input image data increases, one of the image data is repeatedly selected from the selected image data and the newly generated difference image data. Therefore, only two image data need to be retained for selection. Thus, the memory required for storing image data can be reduced.

[0064] Furthermore, for example, the image processing method may select the previously selected image data as the first image data when the brightness value of the newly generated difference image data is smaller than the brightness value of the previously selected image data.

[0065] In this way, when the brightness value of the newly generated difference image data is smaller than the brightness value of the previously selected image data, the signal charge accumulated in the charge accumulation section is close to saturation, so the brightness value of the difference image data is likely to be smaller. Therefore, the difference image data generated later is also likely to have a smaller brightness value, and even if the previously selected image data is selected as the first image data without comparing with the difference image data generated later, the selected first image data is likely to remain unchanged. Therefore, by selecting the previously selected image data as the first image data, it is no longer necessary to generate the difference image data later, so that the processing speed can be improved and the power consumption can be reduced.

[0066] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, the embodiments described below all represent general or specific examples. The numerical values, shapes, materials, constituent elements, configurations and connection methods of constituent elements, steps, the order of steps, etc. shown in the following embodiments are examples and are not intended to limit the present disclosure. The various methods described in this specification can be combined with each other as long as no contradiction arises. In addition, among the constituent elements in the following embodiments, constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. In the following description, constituent elements having substantially the same function are represented by common reference marks, and the description is sometimes omitted.

[0067] (Implementation Method 1)

[0068] First, the imaging device according to Embodiment 1 will be described.

[0069] [Overall Configuration of the Camera Device]

[0070] The camera device involved in this embodiment has the following function: based on multiple image data captured by different exposure times, image data of a high dynamic range image is generated. The camera unit included in the camera device involved in this embodiment captures images at a specified exposure time, and reads out pixel signals in a non-destructive manner multiple times during the exposure time. Furthermore, in the image processing unit included in the camera device involved in this embodiment, differential image data between the multiple image data obtained by reading out pixel signals in the non-destructive manner is generated. From the image data initially obtained and the differential image data, for example, one image data is selected based on the local brightness value in the image data, and the selected image data is synthesized with the image data obtained by the above-mentioned capture of the specified exposure time, thereby generating image data of a high dynamic range image.

[0071] Figure 3 1 is a block diagram showing an exemplary configuration of an imaging device according to the present embodiment. Figure 3 As shown, the imaging device 100 according to the present embodiment includes a lens optical system 110, an imaging unit 120, an imaging control unit 130, and an image processing unit 140. The image processing unit 140 is an example of a processing circuit.

[0072] The lens optical system 110 includes, for example, an autofocus lens, a zoom lens, and a diaphragm, and focuses light onto an imaging surface of the imaging unit 120 .

[0073] The imaging unit 120 converts the light incident through the lens optical system 110 into an electrical signal corresponding to its intensity, and outputs it as image data. Here, the imaging unit 120 has a function of reading out a signal corresponding to the intensity of the incident light without a signal reset action, in other words, reading out the signal in a non-destructive manner. That is, the imaging unit 120 performs imaging at a prescribed exposure time, and reads out the signal of each pixel in a non-destructive manner during the exposure time, thereby outputting both image data in the middle of the prescribed exposure time read out in a non-destructive manner, and normal read-out image data IL captured at the prescribed exposure time. That is, within a 1-frame period including the prescribed exposure time, the imaging unit performs readout in a non-destructive manner multiple times, thereby sequentially outputting a plurality of image data within a 1-frame period. The normal read-out image data IL is an example of the second image data. The detailed structure of the imaging unit 120 will be described later.

[0074] The camera control unit 130 generates a signal for controlling the camera unit 120 and supplies it to the camera unit 120. The signal for controlling the camera unit 120 is, for example, a vertical synchronization signal (VD) or a horizontal synchronization signal (HD), and a trigger signal for starting the above-mentioned non-destructive reading. Figure 3 In the example shown, the imaging control unit 130 is described as being separated from the imaging unit 120, but both may be fabricated on the same semiconductor substrate. In addition, part or all of the imaging control unit 130 may be included in the imaging unit 120. The imaging control unit 130 may be implemented, for example, by a dedicated circuit, a processor, a memory, or a combination thereof.

[0075] The image processing unit 140 receives a plurality of image data output from the imaging unit 120 and performs image processing on the plurality of image data. The image processing unit 140 includes an image selection unit 141 and an HDR synthesis unit 142. The image processing unit 140 is realized by, for example, a dedicated circuit, a processor, a memory, or a combination thereof.

[0076] The image selection unit 141 obtains the difference between two consecutive image data that are read out in a non-destructive manner and outputted in the image pickup unit 120 based on the control signal from the image pickup control unit 130, thereby generating at least one differential image data. Furthermore, one image data is selected from the initial image data that is outputted first among the plurality of image data and the at least one differential image data based on their respective brightness values, and is outputted as output image data IS. The output image data IS is an example of the first image data. As will be described in detail later, the output image data IS selected by the image selection unit 141 is equivalent to image data captured with an exposure time shorter than the prescribed exposure time set in the image pickup unit 120 or the image pickup control unit 130. That is, the output image data IS of the image selection unit 141 and the normal read image data IL captured with the prescribed exposure time outputted from the image pickup unit 120 are supplied to the HDR synthesis unit 142 as image data of the short-exposure image and the long-exposure image in the above-mentioned high dynamic range imaging.

[0077] The HDR synthesis unit 142 receives the output image data IS output from the image selection unit 141 and the normal read image data IL which is one of the plurality of image data output from the imaging unit 120. The HDR synthesis unit 142 synthesizes the output image data IS and the normal read image data IL to generate synthesis result image data IHDR. Thus, the imaging device 100 captures a high dynamic range image.

[0078] [Circuit Configuration of Camera Section]

[0079] Next, the circuit configuration of the imaging unit 120 will be described. Figure 4 It is a schematic diagram showing an exemplary circuit configuration of the imaging unit 120 according to the present embodiment. Figure 4 The illustrated imaging unit 120 includes a pixel array PA including a plurality of pixels 10 arranged in two dimensions. Figure 4 The example in which the pixels 10 are arranged in a matrix of 2 rows and 2 columns is schematically shown. The number and arrangement of the pixels 10 in the imaging unit 120 are not limited to Figure 4 In addition, the surface where these pixels 10 are arranged two-dimensionally is sometimes called an imaging surface.

[0080] Each pixel 10 has a photoelectric conversion unit 13 and a signal detection circuit 14. As described later with reference to the accompanying drawings, the photoelectric conversion unit 13 has a photoelectric conversion layer separated by two electrodes facing each other, receives incident light and generates signal charges. The photoelectric conversion unit 13 does not need to be an independent element for each pixel 10 as a whole, and a part of the photoelectric conversion unit 13, for example, can also span multiple pixels 10. The signal detection circuit 14 is a circuit that detects the signal charge generated by the photoelectric conversion unit 13. Specifically, the signal detection circuit 14 reads a signal corresponding to the signal charge accumulated in the charge accumulation node 41 described later. In this example, the signal detection circuit 14 includes a signal detection transistor 24 and an address transistor 26. The signal detection transistor 24 and the address transistor 26 are, for example, field effect transistors (FETs), and here, as the signal detection transistor 24 and the address transistor 26, an N-channel MOSFET (Metal Oxide Semiconductor Field Effect Transistor) is exemplified. Each transistor such as the signal detection transistor 24, the address transistor 26, and the reset transistor 28 described later has a control terminal, an input terminal, and an output terminal. The control terminal is, for example, a gate. The input terminal is one of the drain and the source, for example, the drain. The output terminal is the other of the drain and the source, for example, the source.

[0081] like Figure 4 As schematically shown in FIG. 1 , the control terminal of the signal detection transistor 24 is electrically connected to the photoelectric conversion unit 13. The signal charge generated by the photoelectric conversion unit 13 is accumulated in the charge accumulation node 41 between the gate of the signal detection transistor 24 and the photoelectric conversion unit 13. Here, the signal charge is a hole or an electron. The charge accumulation node 41 is an example of a charge accumulation unit, also called a "floating diffusion node". The details of the structure of the photoelectric conversion unit 13 will be described later.

[0082] The imaging unit 120 includes a driving unit that drives the pixel array PA and acquires image data at a plurality of timings. The driving unit includes a voltage supply circuit 32 , a reset voltage source 34 , a vertical scanning circuit 36 ​​, a column signal processing circuit 37 , and a horizontal signal reading circuit 38 .

[0083] The photoelectric conversion unit 13 of each pixel 10 is also connected to the sensitivity control line 42. Figure 4 In the illustrated configuration, the sensitivity control line 42 is connected to the voltage supply circuit 32. The voltage supply circuit 32 supplies a predetermined voltage to the counter electrode 12 described later. Alternatively, different voltages may be supplied to the counter electrode 12 between frames.

[0084] The voltage supply circuit 32 is not limited to a specific power supply circuit, and may be a circuit that generates a predetermined voltage or a circuit that converts a voltage supplied from another power supply into a predetermined voltage.

[0085] Each pixel 10 is connected to a power line 40 that supplies a power supply voltage VDD. As shown in the figure, the input terminal of the signal detection transistor 24 is connected to the power line 40. The signal detection transistor 24 amplifies and outputs the signal charge generated by the photoelectric conversion unit 13 and accumulated in the charge accumulation node 41 by the power line 40 functioning as a source follower power supply. As a result, the imaging unit 120 can output a signal corresponding to the signal charge accumulated in the charge accumulation node 41 in a non-destructive manner.

[0086] The output terminal of the signal detection transistor 24 is connected to the input terminal of the address transistor 26. The output terminal of the address transistor 26 is connected to one of the plurality of vertical signal lines 47 arranged for each column of the pixel array PA. The control terminal of the address transistor 26 is connected to the address control line 46, and the output of the signal detection transistor 24 can be selectively read to the corresponding vertical signal line 47 by controlling the potential of the address control line 46.

[0087] In the example shown in the figure, the address control line 46 is connected to the vertical scanning circuit 36. The vertical scanning circuit 36 ​​is also called a "row scanning circuit". The vertical scanning circuit 36 ​​selects a plurality of pixels 10 arranged in each row in units of rows by applying a predetermined voltage to the address control line 46. Thus, the signal of the selected pixel 10 is read out, and the reset operation of the charge accumulation node 41 described later is performed.

[0088] Furthermore, a pixel drive signal generating circuit 39 is connected to the vertical scanning circuit 36. In the illustrated example, the pixel drive signal generating circuit 39 generates a signal for driving the pixels 10 arranged in each row of the pixel array PA, and the generated pixel drive signal is supplied to the pixels 10 of the row selected by the vertical scanning circuit 36.

[0089] The vertical signal line 47 is a main signal line that transmits the pixel signal from the pixel array PA to the peripheral circuit. The column signal processing circuit 37 is connected to the vertical signal line 47. The column signal processing circuit 37 is also called a "row signal accumulation circuit". The column signal processing circuit 37 performs noise suppression signal processing represented by correlated double sampling and analog-to-digital conversion (AD conversion), etc. As shown in the figure, the column signal processing circuit 37 is arranged corresponding to each column of the pixel 10 in the pixel array PA. The horizontal signal readout circuit 38 is connected to these column signal processing circuits 37. The horizontal signal readout circuit 38 is also called a "column scanning circuit". The horizontal signal readout circuit 38 reads out signals from multiple column signal processing circuits 37 to the horizontal common signal line 49 in sequence.

[0090] exist Figure 4 In the illustrated configuration, the pixel 10 has a reset transistor 28. The reset transistor 28 can be, for example, a field effect transistor like the signal detection transistor 24 and the address transistor 26. In the following, unless otherwise specified, an example in which an N-channel MOSFET is used as the reset transistor 28 is described. As shown in the figure, the reset transistor 28 is connected between a reset voltage line 44 that supplies a reset voltage Vr and a charge accumulation node 41. The control terminal of the reset transistor 28 is connected to a reset control line 48, and by controlling the potential of the reset control line 48, the potential of the charge accumulation node 41 can be reset to the reset voltage Vr. In this example, the reset control line 48 is connected to a vertical scanning circuit 36. Therefore, by applying a predetermined voltage to the reset control line 48 through the vertical scanning circuit 36, a plurality of pixels 10 arranged in each row can be reset in units of rows.

[0091] In this example, a reset voltage line 44 that supplies a reset voltage Vr to the reset transistor 28 is connected to a reset voltage source 34. The reset voltage source 34 is also referred to as a "reset voltage supply circuit". The reset voltage source 34 only needs to have a structure capable of supplying a predetermined reset voltage Vr to the reset voltage line 44 when the imaging unit 120 is in operation, and is not limited to a specific power supply circuit as with the above-mentioned voltage supply circuit 32. The voltage supply circuit 32 and the reset voltage source 34 may each be part of a single voltage supply circuit or an independent individual voltage supply circuit. In addition, one or both of the voltage supply circuit 32 and the reset voltage source 34 may also be part of the vertical scanning circuit 36. Alternatively, the sensitivity control voltage from the voltage supply circuit 32 and / or the reset voltage Vr from the reset voltage source 34 may also be supplied to each pixel 10 via the vertical scanning circuit 36.

[0092] The power supply voltage VDD of the signal detection circuit 14 can also be used as the reset voltage Vr. In this case, the power supply voltage supplied to each pixel 10 can be set to Figure 4 The voltage supply circuit not shown in the figure is common to the reset voltage source 34. In addition, the power supply line 40 and the reset voltage line 44 can be common, so the wiring in the pixel array PA can be simplified. However, by setting the reset voltage Vr to a voltage different from the power supply voltage VDD of the signal detection circuit 14, the camera unit 120 can be controlled more flexibly.

[0093] [Pixel device structure]

[0094] Next, the device structure of the pixel 10 of the imaging unit 120 will be described. Figure 5 is a schematic cross-sectional view showing an exemplary device structure of the pixel 10 according to this embodiment. Figure 5In the illustrated configuration, the signal detection transistor 24, the address transistor 26, and the reset transistor 28 are formed on a semiconductor substrate 20. The semiconductor substrate 20 is not limited to a substrate whose entirety is a semiconductor. The semiconductor substrate 20 may also be an insulating substrate having a semiconductor layer disposed on the surface of the side where the photosensitive region is formed. Here, an example of using a P-type silicon (Si) substrate as the semiconductor substrate 20 is described.

[0095] The semiconductor substrate 20 has impurity regions 26s, 24s, 24d, 28d, and 28s, and an element isolation region 20t for electrical isolation between pixels 10. Here, the impurity regions 26s, 24s, 24d, 28d, and 28s are N-type regions. In addition, the element isolation region 20t is also provided between the impurity region 24d and the impurity region 28d. The element isolation region 20t is formed by, for example, performing acceptor ion implantation based on predetermined implantation conditions.

[0096] The impurity regions 26s, 24s, 24d, 28d, and 28s are, for example, diffusion layers of impurities formed in the semiconductor substrate 20. Figure 5 As schematically shown in FIG. 1 , the signal detection transistor 24 includes an impurity region 24s and an impurity region 24d, and a gate electrode 24g. The gate electrode 24g is formed using a conductive material. The conductive material is, for example, polysilicon that is given conductivity by doping with impurities, but may also be a metal material. The impurity region 24s functions as, for example, a source region of the signal detection transistor 24. The impurity region 24d functions as, for example, a drain region of the signal detection transistor 24. Between the impurity region 24s and the impurity region 24d, a channel region of the signal detection transistor 24 is formed.

[0097] Similarly, the address transistor 26 includes the impurity region 26s and the impurity region 24s, and an address control line 46 (see Figure 4 ) is connected to a gate electrode 26g. The gate electrode 26g is formed using a conductive material. The conductive material is, for example, polysilicon that has been given conductivity by doping with impurities, but it can also be a metal material. In this example, the signal detection transistor 24 and the address transistor 26 are electrically connected to each other by sharing an impurity region 24s. The impurity region 24s functions as, for example, a drain region of the address transistor 26. The impurity region 26s functions as, for example, a source region of the address transistor 26. The impurity region 26s has the same Figure 5 The vertical signal line 47 (see Figure 4). In addition, the impurity region 24s may not be shared by the signal detection transistor 24 and the address transistor 26. Specifically, the source region of the signal detection transistor 24 and the drain region of the address transistor 26 may be separated in the semiconductor substrate 20 and electrically connected via a wiring layer provided in the interlayer insulating layer 50.

[0098] The reset transistor 28 includes impurity regions 28d and 28s and a reset control line 48 (see Figure 4 ) is connected to the gate electrode 28g. The gate electrode 28g is formed using, for example, a conductive material. The conductive material is, for example, polysilicon that has been given conductivity by doping with impurities, but may also be a metal material. The impurity region 28s functions as, for example, a source region of the reset transistor 28. The impurity region 28s has the same Figure 5 The reset voltage line 44 (see Figure 4 The impurity region 28d functions as, for example, a drain region of the reset transistor 28.

[0099] On the semiconductor substrate 20, an interlayer insulating layer 50 is arranged in a manner covering the signal detection transistor 24, the address transistor 26, and the reset transistor 28. The interlayer insulating layer 50 is formed of an insulating material such as silicon dioxide. As shown in the figure, a wiring layer 56 is arranged in the interlayer insulating layer 50. The wiring layer 56 is formed of a metal such as copper, and can include a signal line or a power line such as the vertical signal line 47 described above in a portion thereof. The number of insulating layers in the interlayer insulating layer 50 and the number of layers included in the wiring layer 56 arranged in the interlayer insulating layer 50 can be set arbitrarily and are not limited to Figure 5 Example shown.

[0100] The above-mentioned photoelectric conversion unit 13 is arranged on the interlayer insulating layer 50. In other words, in this embodiment, the pixel array PA (see Figure 4 ) are formed in and on a semiconductor substrate 20. A plurality of pixels 10 arranged two-dimensionally on the semiconductor substrate 20 form a photosensitive region. The photosensitive region is also called a pixel region. The distance between two adjacent pixels 10, i.e., the pixel pitch, may be, for example, about 2 μm.

[0101] The photoelectric conversion unit 13 includes a pixel electrode 11, an opposing electrode 12, and a photoelectric conversion layer 15 disposed therebetween. The pixel electrode 11 is an example of a first electrode, and the opposing electrode 12 is an example of a second electrode. In the illustrated example, the opposing electrode 12 and the photoelectric conversion layer 15 are formed across a plurality of pixels 10. On the other hand, the pixel electrode 11 is provided for each pixel 10, and is electrically separated from the pixel electrodes 11 of other adjacent pixels 10 by being spatially separated from the pixel electrodes 11 of other pixels 10.

[0102] The counter electrode 12 is, for example, a transparent electrode formed of a transparent conductive material. The counter electrode 12 is arranged on the side of the photoelectric conversion layer 15 for light to enter. Therefore, the light that has passed through the counter electrode 12 enters the photoelectric conversion layer 15. In addition, the light detected by the imaging unit 120 is not limited to the light within the wavelength range of visible light. For example, the imaging unit 120 can also detect infrared rays or ultraviolet rays. Here, the wavelength range of visible light is, for example, above 380nm and below 780nm. "Transparent" in this specification means transmitting at least a part of the light in the wavelength range to be detected, and it is not necessary to transmit light in the entire wavelength range of visible light. In this specification, for convenience, the electromagnetic wave including infrared rays and ultraviolet rays is expressed as "light" as a whole. The counter electrode 12 can use transparent conductive oxides (TCO: Transparent Conducting Oxide) such as ITO, IZO, AZO, FTO, SnO2, TiO2, and ZnO2.

[0103] The photoelectric conversion layer 15 receives incident light and generates hole-electron pairs. The photoelectric conversion layer 15 is formed of, for example, an organic semiconductor material.

[0104] As reference Figure 4 As described above, the counter electrode 12 is connected to the sensitivity control line 42 connected to the voltage supply circuit 32. In addition, here, the counter electrode 12 is formed across a plurality of pixels 10. Therefore, a sensitivity control voltage of a desired magnitude can be applied from the voltage supply circuit 32 to a plurality of pixels 10 at once via the sensitivity control line 42. In addition, as long as a sensitivity control voltage of a desired magnitude can be applied from the voltage supply circuit 32, the counter electrode 12 can also be provided separately for each pixel 10. Similarly, the photoelectric conversion layer 15 can also be provided separately for each pixel 10.

[0105] The pixel electrode 11 is an electrode for collecting signal charges generated by the photoelectric conversion unit 13. There is at least one pixel electrode 11 for each pixel 10. The pixel electrode 11 is arranged opposite to the counter electrode 12. By controlling the potential of the counter electrode 12 relative to the potential of the pixel electrode 11, the pixel electrode 11 can collect one of the positively charged holes and negatively charged electrons among the hole-electron pairs generated in the photoelectric conversion layer 15 by photoelectric conversion as signal charges. For example, in the case of using holes as signal charges, by making the potential of the counter electrode 12 higher than that of the pixel electrode 11, holes can be selectively collected by the pixel electrode 11. In addition, the amount of signal charge collected per unit time varies according to the potential difference between the pixel electrode 11 and the counter electrode 12. The following is an example of using holes as signal charges. In addition, electrons can also be used as signal charges. In this case, the potential of the counter electrode 12 can be lower than that of the pixel electrode 11.

[0106] The pixel electrode 11 is formed of a metal such as aluminum or copper, a metal nitride, or polysilicon that has been given conductivity by doping with impurities.

[0107] The pixel electrode 11 may also be a light-shielding electrode. For example, by forming a TaN electrode with a thickness of 100 nm as the pixel electrode 11, sufficient light-shielding can be achieved. By setting the pixel electrode 11 as a light-shielding electrode, it is possible to suppress the light after passing through the photoelectric conversion layer 15 from entering the channel region or impurity region of the transistor formed on the semiconductor substrate 20. In the illustrated example, the transistor is at least one of the signal detection transistor 24, the address transistor 26, and the reset transistor 28. The above-mentioned wiring layer 56 may also be used to form a light-shielding film in the interlayer insulating layer 50. By using these light-shielding electrodes or light-shielding films to suppress the light from entering the channel region of the transistor formed on the semiconductor substrate 20, for example, the shift of the characteristics of the transistor, such as the change of the threshold voltage of the transistor, can be suppressed. In addition, by suppressing the light from entering the impurity region formed on the semiconductor substrate 20, the noise mixing caused by the unexpected photoelectric conversion in the impurity region can be suppressed. In this way, suppressing the light from entering the semiconductor substrate 20 helps to improve the reliability of the camera unit 120.

[0108] like Figure 5 As schematically shown in FIG. 1 , the pixel electrode 11 is connected to the gate electrode 24g of the signal detection transistor 24 via the plug 52, the wiring 53, and the contact plug 54. In other words, the gate of the signal detection transistor 24 is electrically connected to the pixel electrode 11. The plug 52 and the wiring 53 can be formed of a metal such as copper. The plug 52, the wiring 53, and the contact plug 54 constitute a charge accumulation node 41 between the signal detection transistor 24 and the photoelectric conversion unit 13 (see FIG. 1 ). Figure 4) at least a portion of the wiring layer 56. The wiring 53 can be a portion of the wiring layer 56. In addition, the pixel electrode 11 is also connected to the impurity region 28d via the plug 52, the wiring 53 and the contact plug 55. Figure 5 In the illustrated configuration, the gate electrode 24g of the signal detection transistor 24, the plug 52, the wiring 53, the contact plugs 54 and 55, and the impurity region 28d which is one of the source region and the drain region of the reset transistor 28 constitute at least a part of the charge accumulation node 41, and function as a charge accumulation region for accumulating signal charges collected by the pixel electrode 11.

[0109] A voltage corresponding to the amount of signal charges collected by the pixel electrode 11 and accumulated in the charge accumulation node 41 is applied to the gate of the signal detection transistor 24. The signal detection transistor 24 amplifies the voltage. The voltage amplified by the signal detection transistor 24 is selectively read out as a signal voltage via the address transistor 26. That is, a signal corresponding to the signal charge accumulated in the charge accumulation node 41 is read out. In this specification, the signal voltage corresponding to the signal charge accumulated in the charge accumulation node 41 in the pixel 10 is sometimes referred to as a "signal" or a "pixel signal".

[0110] The imaging unit 120 as described above can be manufactured using a general semiconductor manufacturing process. In particular, when a silicon substrate is used as the semiconductor substrate 20, the imaging unit 120 can be manufactured using various silicon semiconductor processes.

[0111] [Camera unit actions]

[0112] Next, the operation of the imaging device 100 according to the present embodiment will be described. In other words, the image processing method performed by the imaging device 100 will be described. Figures 4 to 7 An example of the operation in the imaging unit 120 will be described. For the sake of simplicity, an example of the operation in the case where the number of rows of pixels 10 included in the pixel array PA is a total of eight rows, namely, the R0th row to the R7th row, will be described.

[0113] First, the acquisition of image data in the imaging unit 120 is described. When acquiring image data, a reset operation for resetting the charge accumulation node 41 of each pixel 10 in the pixel array PA and a readout of a signal corresponding to the charge signal accumulated after the reset operation are performed. The imaging unit 120 involved in this embodiment uses a readout method called "normal readout" and a readout method called "non-destructive readout" together. Normal readout performs a non-destructive readout of a pixel signal during one readout period, and performs a reset operation for resetting the potential of the charge accumulation node 41 for charge accumulation in the next frame period. Non-destructive readout only performs a non-destructive readout of a pixel signal without a reset operation for resetting the potential of the charge accumulation node 41. The imaging unit 120 performs a plurality of readouts in a non-destructive manner without performing a reset operation through normal readout and non-destructive readout, thereby sequentially outputting a plurality of image data within one frame period.

[0114] Figure 6 This is a timing chart showing an example of the operation in the imaging unit 120 according to the present embodiment. Figure 6 (a) is a diagram showing the timing of the falling (or rising) of the vertical synchronization signal VD. Figure 6 (b) is a diagram showing the timing of the falling (or rising) of the horizontal synchronization signal HD. Figure 6 (c) is a diagram showing an example of the timing of the non-destructive readout trigger signal VD_ND for defining the timing at which the non-destructive readout is performed in the imaging unit 120 . Figure 6 (d) is a diagram schematically showing the readout timing during the signal readout period in each row of the pixel array PA. Figure 6 In the diagram (d), normal pixel reading is performed at the timing indicated by the white rectangle, and non-destructive reading is performed at the timing indicated by the dotted rectangle.

[0115] The vertical synchronization signal VD, the horizontal synchronization signal HD, and the non-destructive readout trigger signal VD_ND are supplied to the imaging unit 120 from, for example, the imaging control unit 130 .

[0116] exist Figure 6 In the example of FIG. 1 , one normal readout and three non-destructive readouts are performed during one frame period. Here, one frame period refers to Figure 6 The time TVD between two consecutive vertical synchronization signals VD shown in (a) is the period of time TVD, in other words, Figure 6 The period from time t0 to t5.

[0117] like Figure 6As shown, the camera unit 120 starts normal readout of the signals of the multiple pixels 10 belonging to the R0-th row based on the vertical synchronization signal VD at time t0. Thereafter, the first non-destructive readout starts at time t2, the second non-destructive readout starts at time t3, and the third non-destructive readout starts at time t4. Next, at time t5, based on the next vertical synchronization signal VD, the normal readout of the signals of the multiple pixels belonging to the R0-th row starts. Regarding the readout of the pixels 10 belonging to the R1-th row and thereafter, the normal readout and non-destructive readout are performed in the same manner as the above-mentioned pixels 10 belonging to the R0-th row. The difference from the pixels 10 belonging to the R0-th row is that the start time of the readout is per unit time, for example, per unit time. Figure 6 Each 1HD period shown as time THD in FIG. 1 is shifted backward.

[0118] Figure 7 : is a timing chart showing an example of the control signal in the above-mentioned signal read period. Figure 7 (a), (b) and (c) are respectively Figure 6 (a), (b) and (c) of FIG. 1 also show the timing of the falling (or rising) of the vertical synchronization signal VD and the horizontal synchronization signal HD, and the timing of the non-destructive readout trigger signal VD_ND. Figure 7 In (d), "Vsel_R0" is a graph showing the time variation of the potential Vsel_R0 of the address control line 46 in the multiple pixels 10 belonging to the R0th row, "Vrs_R0" is a graph showing the time variation of the potential Vrs_R0 of the reset control line 48 in the multiple pixels 10 belonging to the R0th row, and "VFD_R0" is a graph showing the time variation of the potential VFD_R0 of the charge storage node 41 in a specific pixel 10 belonging to the R0th row. Figure 7 In (d), "Vsel_R1", "Vrs_R1", "Vsel_R2", "Vrs_R2", "Vsel_R3" and "Vrs_R3" are graphs showing the time changes in the potential of the address control line 46 and the time changes in the potential of the reset control line 48 in multiple pixels 10 belonging to the R1th row, the R2nd row and the R3th row, respectively, in the same way as "Vsel_R0" and "Vrs_R0".

[0119] In the signal readout period, first, normal readout is performed. In normal readout, first, based on the vertical synchronization signal VD, the potential Vsel_R0 of the address control line 46 of the R0th row is set to a high level at time t0. Specifically, Figure 7The potential VH shown by "Vsel_R0" of the address control line 46 is turned on, thereby turning on the address transistor 26 whose gate is connected to the address control line 46. As a result, the potential VFD_R0 of the charge accumulation node 41, specifically, the signal voltage corresponding to the pixel signal potential Vsig_L corresponding to the amount of signal charge accumulated in the charge accumulation node 41 before time t0, is output to the vertical signal line 47 and transmitted to the column signal processing circuit 37. That is, the pixel signal of the pixel 10 in the R0th row is read out.

[0120] After the pixel signal of the above-mentioned pixel 10 is read out, for example, the AD conversion of the pixel signal in the column signal processing circuit 37 is completed, the reset operation of the pixel 10 belonging to the R0th row is performed in the same normal readout period in the following order. First, the potential Vrc_R0 of the reset control line 48 of the R0th row is set to a high level, specifically, set to Figure 7 The potential VH shown by "Vrs_R0" of the reset control line 48 is set to turn on the reset transistor 28 whose gate is connected to the reset control line 48. As a result, the charge storage node 41 is connected to the reset voltage line 44, and the reset voltage Vr is supplied to the charge storage node 41. That is, the potential VFD_R0 of the charge storage node 41 is reset to the reset voltage Vr. Thereafter, at time t1 after a time Trs has passed from time t0, the potential Vrs_R0 of the reset control line 48 is set to a low level, specifically, set to Figure 7 The reset transistor 28 is turned off by the potential VL indicated by "Vrs_R0", and the pixel signal after the reset operation is read from the pixel 10 in the R0th row via the vertical signal line 47. The pixel signal obtained at this time is a pixel signal corresponding to the magnitude of the reset voltage Vr.

[0121] After the pixel signal after the above reset operation is read, the potential Vsel_R0 of the address control line 46 is set to a low level. Specifically, it is set to Figure 7 The potential VL indicated by "Vsel_R0" turns off the address transistor 26. The pixel signal potential Vsig_L1 and the signal voltage corresponding to the reset voltage Vr read out here are respectively transmitted to the column signal processing circuit 37, and signal processing such as removing fixed pattern noise by taking the difference between the signal processing results is performed.

[0122] exist Figure 6 and Figure 7 The time indicated by time Trs represents the time from the start of normal reading of the pixel signal to the completion of the reset operation of the potential of the charge storage node 41. Time t1 after time Trs has passed from time t0 becomes the start point of exposure in the next pixel signal reading.

[0123] Next, non-destructive reading is performed. First, at time t2 after time Te1 has passed from time t1, the potential Vsel_R0 of the address control line 46 of the R0th row is set to a high level again based on the non-destructive reading trigger signal VD_ND. As a result, the address transistor 26 whose gate is connected to the address control line 46 is turned on. At this time, the potential VFD_R0 of the charge accumulation node 41 is the pixel signal potential Vsig_e1 corresponding to the amount of charge accumulated in the charge accumulation node 41 during the time Te1 between time t1 and time t2. Through the above-mentioned signal reading operation, the signal voltage corresponding to the pixel signal potential Vsig_e1 is output to the vertical signal line 47 and transmitted to the column signal processing circuit 37.

[0124] On the other hand, in the non-destructive readout starting at time t2, the potential Vrc_R0 of the reset control line 48 of the R0th row does not change from the low level. As a result, the reset transistor 28 whose gate is connected to the reset control line 48 remains off, and the potential VFD_R0 of the charge accumulation node 41 does not change to the reset voltage Vr. Since the reset operation of the potential VDF_R0 of the charge accumulation node 41 is not performed, the signal charge accumulated in the charge accumulation node 41 is not destroyed along with the readout operation at time t2, that is, non-destructive readout is performed. At the moment when the readout of the potential VFD_R0 of the charge accumulation node 41 is completed, the potential of the address control line 46 is set to a low level, and the address transistor 26 is turned off.

[0125] At time t3 after time Te2 has passed from time t2, and at time t4 after time Te3 has passed from time t3, non-destructive reading is performed in the same manner as the operation at time t1 based on the non-destructive reading trigger signal VD_ND. That is, at each time t3 and t4, the potential Vsel_R0 of the address control line 46 of the R0th row is set to a high level, so that the address transistor 26 whose gate is connected to the address control line 46 is turned on. As a result, at time t3, a signal voltage corresponding to the pixel signal potential Vsig_e2 corresponding to the amount of charge accumulated in the charge accumulation node 41 during the period of "time Te1+time Te2" between time t1 and time t3 is output to the vertical signal line 47 and transmitted to the column signal processing circuit 37. In addition, at time t4, a signal voltage corresponding to the pixel signal potential Vsig_e3 corresponding to the amount of charge accumulated in the charge accumulation node 41 during the period of "time Te1+time Te2+time Te3" between time t1 and time t4 is output to the vertical signal line 47 and transmitted to the column signal processing circuit 37. Furthermore, the potential Vrc_R0 of the reset control line 48 of the R0th row does not change from the low level, and the potential VDF_R0 of the charge accumulation node 41 is not reset to the reset voltage Vr. Similar to the readout starting from time t2, at the time when the readout of the potential VFD_R0 of the charge accumulation node 41 starting from time t3 and time t4 is completed, the potential of the address control line 46 is set to a low level, and the address transistor 26 is turned off.

[0126] Next, normal readout is performed again. First, based on the next vertical synchronization signal VD, the potential Vsel_R0 of the address control line 46 of the R0th row is set to a high level at time t5, so that the address transistor 26 whose gate is connected to the address control line 46 is turned on. The potential VFD_R0 of the charge accumulation node 41 at this time is the pixel signal potential Vsig_L2 corresponding to the amount of charge accumulated in the charge accumulation node 41 during the period of "time Te1+time Te2+time Te3+time Te4=exposure time TeL" from time t1 to time t5, that is, the length obtained by subtracting the time Trs from the time TVD of the interval of the vertical synchronization signal VD. The signal voltage corresponding to this potential is output to the vertical signal line 47 through the above-mentioned operation and transmitted to the column signal processing circuit 37. In addition, in the normal readout starting from time t5, as in the normal readout at time t0, the reset operation of the pixel 10 belonging to the R0th row is performed during the same normal readout period. That is, the potential Vrc_R0 of the reset control line 48 of the R0th row is set to a high level, so that the reset transistor 28 whose gate is connected to the reset control line 48 is turned on, and the potential VDF_R0 of the charge accumulation node 41 is reset to the reset voltage Vr. Thereafter, at time t6 after the time Trs has passed, the potential of the reset control line 48 is set to a low level, the reset transistor 28 is turned off, and on this basis, the pixel signal after the reset operation is read from the pixel 10 of the R0th row via the vertical signal line 47. After the above-mentioned pixel signal is read, the potential of the address control line 46 is set to a low level, and the address transistor 26 is turned off. In the normal reading starting from time t5, first, the signal corresponding to the pixel signal potential Vsig_L2 is read in a non-destructive manner, and then the potential VDF_R0 of the charge accumulation node 41 is reset.

[0127] The reading of the pixels 10 belonging to the R1th row and later is performed in the same manner as the reading of the pixels 10 belonging to the R0th row. The difference from the pixels 10 belonging to the R0th row is that the start time of the reading is set per unit time, for example, Figure 7 In this specification, when explaining the operation timing of the camera device 100, the operation of the camera unit 120 is described based only on the readout timing of the pixels 10 belonging to the R0th row, and the offset of the readout timing of the pixels 10 belonging to the R1th row and thereafter is sometimes omitted.

[0128] exist Figure 6 and Figure 7 In the embodiment, non-destructive readout is performed three times in one frame period, but the number of non-destructive readouts in one frame period is not particularly limited, and may be more than one time, less than two times, or more than four times. For example, the number of times may also be changed according to the brightness of the subject in the imaging scene, etc.

[0129] exist Figure 6 and Figure 7 In order to simplify the description, the electronic shutter action, that is, the action of discharging the charge accumulated in the charge accumulation node 41 at a certain timing during one frame in order to adjust the exposure time of each pixel 10, is not specifically described, but this action can also be adopted in the camera unit 120 in this embodiment. The timing of the electronic shutter action can be any timing as long as at least one non-destructive readout is performed after the electronic shutter action ends. For example, the electronic shutter action for the pixel 10 belonging to the R0th row can be Figure 6 and Figure 7 In this case, the exposure time Te1 of the pixel signal obtained by the non-destructive readout at time t2 is the length of the period from the completion of the electronic shutter operation to time t2.

[0130] pass Figure 7 In the series of operations shown, image data called "normal read image data" acquired by one normal read and image data called "non-destructive read image data" acquired by three non-destructive reads are acquired during the time TVD corresponding to the time between time t1 and time t6. In addition, hereinafter, the normal read image data is sometimes referred to as the normal read image data IL, and the three non-destructive read image data are referred to as non-destructive read image data Ind1, Ind2, and Ind3, starting from the data acquired at the earliest time.

[0131] As described above, the non-destructive read image data Ind1 at time t2 is image data based on the pixel signal corresponding to the amount of charge accumulated in the charge accumulation node 41 during the time Te1 from when the charge accumulation node 41 is reset at time t1. Similarly, the non-destructive read image data Ind2 at time t3 is image data based on the pixel signal corresponding to the amount of charge accumulated during the period of "time Te1 + time Te2" from time t1, and the non-destructive read image data Ind3 at time t4 is image data based on the pixel signal corresponding to the amount of charge accumulated during the period of "time Te1 + time Te2 + time Te3" from time t1. The normal read image data IL is image data based on the pixel signal corresponding to the amount of charge accumulated during the exposure time TeL corresponding to the period from time t1 to time t5. In this way, the imaging unit 120 does not perform a reset operation to reset the signal charge of the charge accumulation node 41, but performs a plurality of readouts in a non-destructive manner, thereby sequentially outputting the non-destructive readout image data Ind1, Ind2, and Ind3 and the normal readout image data IL as a plurality of image data to the image processing unit 140 within one frame period. In this specification, the plurality of image data output within one frame period is a plurality of image data based on pixel signals corresponding to the charge accumulated in the charge accumulation node 41 at each time within one frame period. Specifically, the plurality of image data output within one frame period is a plurality of image data formed by combining the non-destructive readout image data read out by non-destructive readout after the charge accumulation node 41 is reset in the normal readout and the normal readout image data read out in the next normal readout.

[0132] [Operation of the image selection unit]

[0133] Next, refer to Figure 8 and Fig. 9 The operation of the image selection unit 141 according to the present embodiment will be described. As described above, the image selection unit 141 sequentially receives a plurality of image data consisting of a plurality of non-destructive read image data and a normal read image data from the imaging unit 120 within one frame period, calculates differential image data between the plurality of image data, and selects one image data from the initial image data outputted first among the plurality of image data and at least one of the calculated differential image data.

[0134] Figure 814 is a block diagram showing an exemplary structure of the image selection unit 141 involved in the present embodiment. The image selection unit 141 in the present embodiment includes an input distribution unit 1401, a frame memory 1402, a frame memory 1403, a subtraction unit 1404, a comparison unit 1405, a selector 1406 and a frame memory 1407. The frame memory 1402 is an example of the first memory, and is sometimes referred to as "frame memory MEM1" hereinafter. The frame memory 1403 is an example of the second memory, and is sometimes referred to as "frame memory MEM2" hereinafter. The frame memory 1407 is an example of the third memory, and is sometimes referred to as "frame memory MEM3" hereinafter. In addition, the subtraction unit 1404 is an example of a difference operation circuit, and the comparison unit 1405 is an example of a comparison circuit.

[0135] The input distribution unit 1401 stores the plurality of image data outputted by the camera unit 120 in the frame memory MEM1 and the frame memory MEM2. The image data inputted to the image selection unit 141 is transmitted to the input distribution unit 1401 and selectively stored in one of the frame memory MEM1 and the frame memory MEM2. Specifically, the frame memory MEM1 stores one of two image data outputted continuously among the plurality of image data outputted by the camera unit 120, and the frame memory MEM2 stores the other of the two image data. For example, the input distribution unit 1401 stores the plurality of image data outputted by the camera unit 120 alternately in the frame memory MEM1 and the frame memory MEM2, and when storing new image data in the frame memory MEM1 and the frame memory MEM2, the stored image data is overwritten.

[0136] The subtracting unit 1404 calculates the difference image data between the two image data stored in the frame memory MEM1 and the frame memory MEM2. That is, the subtracting unit 1404 obtains the difference between the two image data stored in the frame memory MEM1 and the frame memory MEM2, thereby generating the difference image data. For example, the subtracting unit 1404 appropriately inverts the frame memory to be the subtrahend and the minuend in such a manner that the image data stored earlier is subtracted from the image data stored later among the two image data stored in the frame memory MEM1 and the frame memory MEM2. In addition, the subtracting unit 1404 may obtain the absolute value of the difference between the two image data stored in the frame memory MEM1 and the frame memory MEM2, thereby generating the difference image data. The subtracting unit 1404 outputs the generated difference image data to the comparing unit 1405 and the selector 1406. When only one of the frame memory MEM1 and the frame memory MEM2 stores the image data, the subtracting unit 1404 outputs the stored image data as it is.

[0137] The comparison unit 1405 compares the brightness value of the image data output from the subtraction unit 1404 with the brightness value of the image data stored in the frame memory MEM3, and outputs a selection signal for selecting one image data to the selector 1406. Specifically, the comparison unit 1405 compares the difference image data generated by the subtraction unit 1404 with the initial image data or the difference image data stored in the frame memory MEM3. When the frame memory MEM3 does not store the image data, the comparison unit 1405 outputs a selection signal for selecting the difference image data output from the subtraction unit 1404, for example.

[0138] The selector 1406 selects one of the image data output from the subtraction unit 1404 and the image data stored in the frame memory MEM3 based on the selection signal supplied from the comparison unit 1405, and stores it in the frame memory MEM3. In addition, when the selector 1406 receives the selection signal for selecting the image data stored in the frame memory MEM3, it is not necessary to store the image data in the frame memory MEM3 again.

[0139] The image data stored in the frame memory MEM3 at the moment when a series of image readouts from the reset action to the next normal readout of the pixels 10 in the camera unit 120 and the image selection action described later are all completed is output to the HDR synthesis unit 142 as the final output image data IS of the image selection unit 141.

[0140] Reference Fig. 9 The operation of the image selection unit 141 according to this embodiment will be described in detail. Fig. 9 FIG. 1 is a diagram for explaining an example of the operation of the image selection unit 141 according to the present embodiment. Fig. 9 (a) of FIG. 1 is a timing diagram showing the operation of the imaging unit 120. Fig. 9 In the action shown, by Figure 6 and Figure 7 Similarly, the operation of the imaging unit 120 is to perform a reset operation of the pixel 10 at time t1 during the one frame period starting from time t0, and then perform non-destructive readout at time t2, t3, and t4, respectively, to obtain non-destructive readout image data Ind1, Ind2, and Ind3. Thereafter, from time t5 when the next normal readout is started, the pixel signal corresponding to the charge accumulated in the charge accumulation node 41 during the one frame period starting from time t0 is read out by normal readout, to obtain normal readout image data IL. Fig. 9 The table (b) shows temporal changes in the contents stored in the three frame memories MEM1 , MEM2 , and MEM3 included in the image selection unit 141 .

[0141] First, as an initialization operation, at the timing of performing a reset operation on each pixel 10 of the imaging unit 120, the image selection unit 141 resets the values ​​stored in the frame memory MEM1, the frame memory MEM2, and the frame memory MEM3 to 0 ( Fig. 9 That is, the frame memory MEM1, the frame memory MEM2, and the frame memory MEM3 are in a state where no image data is stored.

[0142] Next, in synchronization with the first non-destructive reading of the imaging unit 120 starting at time t2, the non-destructive read image data Ind1 is input to the image selection unit 141. The input distribution unit 1401 stores the input non-destructive read image data Ind1 in the frame memory MEM1. At this time, since 0 is written in the frame memory MEM2, the subtraction unit 1404 outputs the non-destructive read image data Ind1 itself. The non-destructive read image data Ind1 is also stored in the frame memory MEM3 ( Fig. 9 (b) Step S2). In this way, in step S2, the image selection unit 141 stores the first non-destructively read image data Ind1 as provisional output image data in the frame memory MEM3.

[0143] Next, in synchronization with the second non-destructive reading of the imaging unit 120 starting at time t3, the non-destructive read image data Ind2 is input to the image selection unit 141. The input distribution unit 1401 stores the input non-destructive read image data Ind2 in the frame memory MEM2. The subtraction unit 1404 receives the non-destructive read image data Ind2 stored in the frame memory MEM2 and the non-destructive read image data Ind1 stored in the frame memory MEM1 in the previous step. The subtraction unit 1404 subtracts the non-destructive read image data Ind1 from the non-destructive read image data Ind2 to generate differential image data Ind2-Ind1. The differential image data Ind2-Ind1 is supplied to one input terminal of the comparison unit 1405. The non-destructive read image data Ind1 stored in the frame memory MEM3 in step S1 is supplied to the other input terminal of the comparison unit 1405. The comparison unit 1405 compares these image data and outputs a selection signal for selecting one image data. The two image data supplied to the comparison unit 1405 are also supplied to the selector 1406. The selector 1406 selects one of the two input image data based on the selection signal output by the comparison unit 1405, and stores the image data in the frame memory MEM3. The detailed operation of the comparison unit 1405 will be described later. As a result, the image data stored in the frame memory MEM3 is rewritten from the first non-destructive read image data Ind1 stored in step S2 to one of the first non-destructive read image data Ind1 output from the selector 1406 and the difference image data Ind2-Ind1 output by the subtraction unit 1404 ( Fig. 9 The output of the selector 1406 in step S3, that is, the image data stored in the frame memory MEM3, is referred to as output image data OUT_2. The output image data OUT_2 is the provisional output image data of the image selection unit 141 determined in step S3.

[0144] Next, in synchronization with the third non-destructive reading of the imaging unit 120 starting from time t4, the non-destructive read image data Ind3 is input to the image selection unit 141. The input distribution unit 1401 stores the input non-destructive read image data Ind3 in the frame memory MEM1. That is, the image data stored in the frame memory MEM1 is rewritten as the non-destructive read image data Ind3. The subtraction unit 1404 receives the non-destructive read image data Ind3 stored in the frame memory MEM1 and the non-destructive read image data Ind2 stored in the frame memory MEM2 in step S3. At this time, the subtraction unit 1404 makes the frame memory to be the subtrahend and the minuend object opposite to step S3, and generates the difference image data Ind3-Ind2. The difference image data Ind3-Ind2 is supplied to one input terminal of the comparison unit 1405. The other input end of the comparison unit 1405 is supplied with the output image data OUT_2 stored in the frame memory MEM3 in step S3. The comparison unit 1405 compares these image data and outputs a selection signal for selecting one image data. The two image data supplied to the comparison unit 1405 are also supplied to the selector 1406. The selector 1406 selects one of the two input image data based on the selection signal output by the comparison unit 1405 and stores the selected image data in the frame memory MEM3. As a result, the image data stored in the frame memory MEM3 is rewritten from the output image data OUT_2 stored in step S4 to one of the output image data OUT_2 of step S4 output from the selector 1406 and the differential image data Ind3-Ind2 output by the subtraction unit 1404 ( Fig. 9 (b) Step S4). The output of the selector 1406 in step S4, that is, the image data stored in the frame memory MEM3, is referred to as output image data OUT_3. The output image data OUT_3 is the provisional output image data of the image selection unit 141 determined in this step.

[0145] As described above, in steps S3 and S4, the subtraction unit 1404 generates differential image data between the Nth non-destructive read image data and the N+1th non-destructive read image data. N is an integer greater than 1. The comparison unit 1405 obtains the generated differential image data and the provisional output image data stored in the frame memory MEM3 at this moment. Then, the comparison unit 1405 compares the differential image data with the provisional output image data, and outputs a selection signal for selecting image data with a large brightness value. The selected image data is stored in the frame memory MEM3 as new provisional output image data by the selector 1406. These actions are repeatedly performed within 1 frame period until the non-destructive read of the camera unit 120 is performed a predetermined number of times.

[0146] Next, in synchronization with the normal reading of the imaging unit 120 starting from time t5, the normal read image data IL is input to the image selection unit 141. The input distribution unit 1401 stores the input normal read image data IL in the frame memory MEM2. That is, the image data stored in the frame memory MEM2 is rewritten as the normal read image data IL. The subtraction unit 1404 receives the normal read image data IL stored in the frame memory MEM2 and the non-destructive read image data Ind3 stored in the frame memory MEM1 in step S4. At this time, the subtraction unit 1404 makes the frame memory to be the subtrahend and the minuend object reverse to step S4, and generates the difference image data IL-Ind3. The difference image data IL-Ind3 is supplied to one input terminal of the comparison unit 1405. The output image data OUT_3 stored in the frame memory MEM3 in step S4 is supplied to the other input terminal of the comparison unit 1405, and the comparison unit 1405 compares these image data and outputs a selection signal for selecting one image data. The two image data supplied to the comparison unit 1405 are also supplied to the selector 1406. The selector 1406 selects one of the two input image data based on the selection signal output by the comparison unit 1405, and stores the selected image data in the frame memory MEM3. As a result, the image data stored in the frame memory MEM3 is rewritten from the output image data OUT_3 stored in step S4 to one of the output image data OUT_3 of step S4 output from the selector 1406 and the difference image data IL-Ind3 output from the subtraction unit 1404 ( Fig. 9 The output of the selector 1406 in step S5, i.e., the image data stored in the frame memory MEM3, becomes the output image data IS of the image selection unit 141. The output image data IS of step S5 becomes the image data finally output from the image selection unit 141. The image selection unit 141 outputs the output image data IS to the HDR synthesis unit 142.

[0147] In this way, in step S5, the subtracting unit 1404 generates the difference image data IL-Ind3 between the last non-destructive read image data Ind3 and the normal read image data IL within one frame period. The comparing unit 1405 obtains the generated difference image data IL-Ind3 and the provisional output image data stored in the frame memory MEM3 at this time. Then, the comparing unit 1405 compares the difference image data IL-Ind3 with the provisional output image data, and outputs a selection signal for selecting the image data with a large brightness value. The selected image data is stored in the frame memory MEM3 as the output image data IS by the selector 1406.

[0148] As described above, in steps S2 to S5, nondestructively read image data Ind1, Ind2 and Ind3, which are a plurality of image data obtained by the imaging unit 120 nondestructively reading multiple times within one frame period, and normal read image data IL are sequentially input to the input distribution unit 1401.

[0149] In addition, in steps S3 to S5, the subtracting unit 1404 obtains the difference between two image data that are continuously input among the non-destructive read image data Ind1, Ind2, and Ind3 and the normal read image data IL, thereby generating differential image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3 as at least one differential image data. The comparing unit 1405 outputs a selection signal for selecting the output image data IS from the non-destructive read image data Ind1, Ind2, and Ind3, which is the initial image data output first among the normal read image data IL, and the differential image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3 based on their respective brightness values. Specifically, in step S3, the comparing unit 1405 outputs a selection signal for selecting one of the image data from the non-destructive read image data Ind1 and the differential image data Ind2-Ind1 based on their respective brightness values. In steps S4 and S5 , the comparison unit 1405 repeatedly outputs a selection signal for selecting one of the selected image data and the newly generated difference image data based on their respective brightness values, and outputs a selection signal for selecting the output image data IS as the finally selected image data.

[0150] As will be described in detail later, the HDR synthesis unit 142 synthesizes the output image data IS and the normal read image data IL to generate synthesis result image data IHDR.

[0151] As long as Figure 8 and Fig. 9 The configuration shown in the figure does not require any change in the hardware configuration required for the image selection unit 141, regardless of the number of times the non-destructive readout is performed during one frame. The input distribution unit 1401 distributes the image data input to the image selection unit 141 alternately to the two frame memories MEM1 and MEM2 and stores them. By keeping the image data once stored until the next step and rewriting it with the newly input image data, the required frame memory resources are reduced. In addition, the selector 1406 reduces the required frame memory resources by overwriting the tentative output image data OUT_2 or OUT_3 stored in the frame memory MEM3 in each step.

[0152] In the image selection unit 141, differential image data between the normal read image data and the non-destructive read image data, or differential image data between the non-destructive read image data is generated. Figure 7 As described in the description, for example, the non-destructive read image data Ind1 at time t2 is image data based on a pixel signal corresponding to the amount of charge accumulated during the period of time Te1 from when time t1 is reset, and the non-destructive read image data Ind2 at time t3 is image data based on a pixel signal corresponding to the amount of charge accumulated during the period of "time Te1+time Te2" from when time t1 is reset. The difference image data Ind2-Ind1, which is the difference between them, is image data based on a pixel signal corresponding to the amount of charge accumulated during the period of time Te2 from time t3 to time t2. That is, by obtaining the difference of the two image data output, image data of an exposure time equivalent to the image data photographed at the exposure time of time Te2 can be obtained. Similarly, the difference image data Ind3-Ind2, which is the difference between the non-destructive read image data Ind2 at time t3 and the non-destructive read image data Ind3 at time t4, is image data of an exposure time equivalent to the image data photographed at the exposure time of time Te3. The difference image data IL-Ind3, which is the difference between the normal read image data IL at time t5 and the non-destructive read image data Ind3 at time t4, is image data of an exposure time equivalent to the image data photographed at the exposure time of time Te4. In addition, the non-destructive read image data Ind1 at time t2 is image data corresponding to a pixel signal obtained by performing exposure and charge accumulation based on the reset of the pixel 10 at time t1, and is data equivalent to the normal read image data with the period of time Te1 from time t1 to time t2 as the exposure time.

[0153] The equivalent exposure time in the above-mentioned difference image data, that is, time Te1 to time Te4, is shorter than the exposure time TeL of the normal read image data IL obtained by normal read in the imaging unit 120. Therefore, the HDR synthesis unit 142 synthesizes the output image data IS, which is one of the difference image data, as the image data of the short-time exposure image in the high dynamic range imaging, and the normal read image data IL as the image data of the long-time exposure image in the high dynamic range imaging, thereby realizing the generation of the high dynamic range image.

[0154] The sum of the time Trs and the time Te1 and the time Te2, Te3 and Te4 are, for example, the same length. That is, the actual exposure time of the non-destructive read image data Ind1 and the differential image data Ind2-Ind1, Ind3-Ind2 and IL-Ind3 is substantially the same. In addition, the sum of the time Trs and the time Te1 and the time Te2, Te3 and Te4 can be set to different lengths by introducing an electronic shutter action or changing the interval of the non-destructive read time. In this case, in order to adjust the difference in exposure time, the image data obtained by appropriately applying the gain can also be input to the subtraction unit 1404 or the comparison unit 1405.

[0155] As described above, the imaging device 100 according to the present embodiment generates output image data IS of a short-time exposure image in high dynamic range imaging by performing non-destructive readout during the exposure time for capturing a long-time exposure image. That is, this means that both the long-time exposure image and the short-time exposure image are captured using the same pixels 10 in the imaging unit 120. For example, in the case of Figure 1 In the case of the method of photographing by changing the exposure time for each pixel on the imaging surface as shown, the resolution of the image at each exposure time is reduced because the pixels on the imaging surface are divided into long-exposure pixels and short-exposure pixels. In the imaging device 100 involved in this embodiment, as described above, both the long-exposure image and the short-exposure image are photographed from the same pixel 10, so that all the pixels 10 on the imaging surface can be used in the photographing of the images of both exposure times, and the resolution of the image will not be degraded.

[0156] On the other hand, the difference image data generated from the non-destructive read image data as described above cannot always be used in place of the image data captured with an equivalent exposure time, for example, the normal read image data acquired at the same timing. Fig.10 and Fig.11 This will be described in more detail.

[0157] Fig.10 4 is a diagram for explaining an example of a change in the pixel signal level of the charge accumulation node 41. Fig.10 The following situations are expressed in Figure 6 or Figure 7 The operation of the imaging unit 120 in the same manner is that the exposure and charge accumulation of each pixel 10 of the imaging unit 120 are performed from the time when the pixel 10 is reset at time t1 to the time when t5, non-destructive reading is performed at time t2, time t3 and time t4, respectively, and normal reading is performed at time t5. In addition, the imaging scene is set as a scene in which a circular light source is lit in a dark background. Fig.10The upper side of (a) is a diagram showing the level of the amount of charge accumulated in the charge accumulation node 41 of the pixel 10 in the area corresponding to the light source on the imaging plane, that is, the pixel signal level. Fig.10 The lower side of (a) is a diagram showing an example of an image corresponding to the non-destructive read image data Ind1, Ind2, and Ind3 and the normal read image data IL. Fig.10 (b) is a diagram showing an example of an image corresponding to the non-destructive read image data Ind1 and the difference image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3.

[0158] like Fig.10 As shown in (a), in a state where there is almost no charge in the charge accumulation node 41 of each pixel 10 of the camera unit 120 just after the pixel has been reset, the pixel signal level of the light source part increases approximately linearly with a slope of a certain size as the exposure time passes. Thereafter, if the amount of charge accumulated in the charge accumulation node 41 increases and approaches the pixel signal level of the upper limit that can be accumulated (i.e., the saturation charge amount), the rate of increase becomes gentle. Thereafter, when the charge amount of the charge accumulation node 41 of the camera unit 120 reaches the upper limit of the pixel signal level and no more charge can be accumulated, the pixel signal level no longer changes. This state is referred to as the signal being in a saturated state. Fig.10 , the upper limit of the pixel signal level is the pixel signal level at time t4 and t5. In addition, not only the accumulated charge of the charge accumulation node 41 of the camera unit 120 has an upper limit, but also the signal level that can be detected in the column signal processing circuit 37 generally has an upper limit. Sometimes, the output from the camera unit 120 becomes saturated due to the signal level in the column signal processing circuit 37 reaching its upper limit. Here, in order to simplify the description, it is assumed that the saturated signal level in the column signal processing circuit 37 is the same as the saturated pixel signal level of the charge accumulation node 41, and the brightness value of the output image data in the state where the accumulated charge of the charge accumulation node 41 reaches the saturated pixel signal level is equal to the maximum brightness value that can be output from the camera unit 120 (that is, it becomes an overexposed state). At this time, the image corresponding to the non-destructive read image data Ind1, Ind2 and Ind3 obtained by non-destructive readout at time t2, t3 and t4, and the normal read image data IL obtained by normal readout at time t5 is, for example, Fig.10 The image shown on the lower side of (a).

[0159] At this time, inside the image selection unit 141, the images corresponding to the difference image data Ind2-Ind1, Ind3-Ind2 and IL-Ind3 between the non-destructive read image data Ind1, Ind2 and Ind3 and the normal read image data IL, and the image corresponding to the non-destructive read image data Ind1, for example, Fig.10 The brightness value of the light source part I21 in the non-destructive readout image data Ind1 corresponds to Fig.10 The luminance value of the light source part I32 in the difference image data Ind2-Ind1 between the non-destructive read image data Ind2 and the non-destructive read image data Ind1 corresponds to Fig.10 The luminance value of the light source part I42 in the difference image data Ind3-Ind2 between the non-destructive readout image data Ind3 and the non-destructive readout image data Ind2 corresponds to Fig.10 Finally, the brightness value of the light source part I54 in the difference image data IL-Ind3 between the normal read image data IL and the non-destructive read image data Ind3 corresponds to Fig.10 The pixel signal level LI54 in (a) is Fig.10 It can be understood that the magnitude relationship of the pixel signal levels of the non-destructive read image data Ind1 and the difference image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3 is pixel signal level LI21>LI32>LI43>LI54. Therefore, in the non-destructive readout performed in a state where a long exposure time has passed since the reset operation of the pixel and the pixel signal level is close to the saturation state, the rise of the pixel signal level accompanying the exposure time becomes smaller, so that in the difference image data of the non-destructive read image data after a certain exposure time or more has passed since the reset operation of the pixel, the position appears darker.

[0160] Fig.11 2 is a diagram for explaining changes in pixel signal levels when non-destructive readout is not performed. Fig.11 In the Fig.10 The same camera scene, through Fig.10 The images are shown in the case where the image is captured by non-destructive reading at the same timing, and in the case where the image is captured by normal reading at the same timing as the non-destructive reading. Fig.11 (a) is a diagram schematically showing the temporal change of the pixel signal level in each readout method. Fig.11The pixel signal level in the case of imaging by non-destructive reading shown by the solid line plot of (a) starts to increase from time t1 and reaches a saturated state at a certain time. Fig.11 The pixel signal level in the case of imaging by normal readout shown by the dotted line plot in (a) is reset at each normal readout of the pixel 10 , and therefore returns to the reset level at each normal readout in accordance therewith.

[0161] exist Fig.11 (b) shows an example of an image based on image data obtained from time t2 to time t5. Fig.10 As explained in the case of Fig.11 In the image corresponding to the difference image data Ind2-Ind1, Ind3-Ind2 and IL-Ind3 between the non-destructive read image data shown in the "Difference Image" on the upper side of (b) and the non-destructive read image data Ind1 as the initial image data, as the acquisition time of the non-destructive read image data used for difference calculation moves backward, the brightness value of the difference image data becomes smaller. In contrast, the image data It2, It3, It4 and It5 in the case of photographing by normal readout at times t2, t3, t4 and t5 are image data obtained by always accumulating charges at the pixel signal level starting from the reset level. Therefore, at any readout timing, the pixel signal level increases with the same slope as the increase in the pixel signal level from time t1 to time t2. Thus, for example, the image Fig.11 If the exposure time is the same, the image data obtained as a result will also have the same level of brightness value, just like the images corresponding to the image data It2, It3, It4 and It5 shown in the "normal read image" on the lower side of (b). That is, in the method of obtaining image data of a specific exposure time by calculating the differential image data of the non-destructive read image data, a large amount of charge is accumulated in the charge accumulation node 41 of the camera unit 120. When the differential image data is calculated based on the non-destructive read image data in a state where the accumulated charge is close to the saturation level, a difference occurs in the brightness value from the image data obtained by normal reading with the same exposure time. In such a situation, even if the brightness of the subject does not change, the brightness value becomes low in the differential image data of the non-destructive read image data, and the subject is reflected too darkly. Therefore, the differential image data of the non-destructive read image data cannot be said to correctly express the brightness of the subject, and there is also image data that is not suitable for use as a short-time exposure image in high dynamic range imaging.

[0162] The image selection unit 141 according to the present embodiment selects image data of a short-exposure image more suitable for high dynamic range imaging, rather than selecting an image that does not accurately represent the brightness of the subject, in which the brightness level of the subject is too dark. More specifically, this operation is implemented by the comparison unit 1405 described below.

[0163] Fig.12 1405 of the image selection unit 141 according to the present embodiment. Fig.12 The processing shown selects the image data more suitable for high dynamic range synthesis from the two image data. Fig.12 In the following, a case is described where the comparison unit 1405 selects one of the image data A and the image data B. For example, the image data A is the image data stored in the frame memory MEM3, and the image data B is the difference image data outputted from the subtraction unit 1404.

[0164] first, Fig.12 The image data A and the image data B of the two images shown in (a) are input to the comparison unit 1405. Next, Fig.12 As shown in (b) of FIG. 14 , the comparison unit 1405 divides the two input image data into a plurality of blocks of a predetermined size. Here, the predetermined size is, for example, 16 pixels horizontally × 16 pixels vertically. Fig.12 In the example, the horizontal direction is divided into 8 blocks and the vertical direction is divided into 5 blocks. Fig.12 As shown in (c) of FIG. 1 , the comparison unit 1405 calculates the average value of the brightness value for each block divided into a predetermined size. Fig.12 As shown in (d), for each block of the two input image data, the average values ​​of the brightness values ​​of the blocks located at the same position are compared. If the average value of the image data of one block is greater than the average value of the image data of the other block by more than a predetermined threshold, a selection signal for selecting the image data is output as the comparison unit 1405. Fig.12 In the example (d) of FIG. 14 , the average brightness value of image data A is larger than that of image data B in the block in the central part of the image, so image data A is selected as the output of the comparison unit 1405 .

[0165] exist Fig.12 In the operation of the comparison unit 1405 shown in FIG. 1 , for each block of the two input image data, the image data of the block with the larger brightness value is selected as the output. Fig.10That is, in the differential image data of the non-destructive read image data, when the pixel signal level of the pixel 10 at the position corresponding to the light source in the camera unit 120 has not reached the saturation level, and the slope of the increase in the pixel signal level relative to the exposure time is large, the brightness value of the image data is large. In addition, when the pixel signal level of the pixel 10 is close to or reaches the saturation level, and the slope of the increase in the pixel signal level relative to the exposure time is small or approximately 0, the brightness value of the differential image data of the non-destructive read image data is small. In addition, of course, when the light source is turned off, even if the exposure time is extended, the brightness value of the corresponding position (excluding the influence of noise, etc.) does not change, and in the differential image data of the non-destructive read image data, the brightness value of the corresponding position is approximately 0.

[0166] By utilizing the above, it is possible to detect the lighting state of the light source of the subject or the saturation of the accumulated charge of the charge accumulation node 41 of the imaging unit 120 based on the brightness value of the differential image data of the non-destructive read image data. When the brightness value of the non-destructive read image data is large, the light source is lit and the charge accumulated in the charge accumulation node 41 of the imaging unit 120 has not reached the saturated charge amount. On the other hand, when the brightness value of the differential image data of the non-destructive read image data is small, it can be considered that the charge accumulated in the charge accumulation node 41 of the imaging unit 120 is close to the saturated charge amount or the lighting time of the light source during the exposure time is short. If the brightness value of the light source part of the differential image data is approximately 0, it is considered that the light source is off or the accumulated charge of the charge accumulation node 41 of the imaging unit 120 has reached the saturated charge amount. As described above, in high dynamic range photography using both long exposure photography and short exposure photography, a bright subject that is easily overexposed in long exposure photography, that is, the charge accumulation node 41 is saturated, is represented by image data that is photographed by short exposure so that it is not overexposed. Therefore, for the image data with short exposure used therein, it is required to obtain a high brightness portion without saturating the charge accumulation node 41. The closer the accumulated charge of the charge accumulation node 41 is to the saturated charge amount, the smaller the brightness value of the differential image data between the non-destructive read image data and the normal read image data. Therefore, the comparison unit 1405 regards the image data with a larger brightness value among these differential image data and the initial image data as image data with a light source turned on and far from the saturation state of the accumulated charge, that is, image data more suitable for high dynamic range synthesis. That is, the comparison unit 1405 outputs a selection signal for selecting the image data with a larger brightness value from the two input image data to the selector 1406.

[0167] In addition, Fig.12In (b), the input image data is divided into blocks of a specified size, but it is also possible not to divide into blocks as described above, and perform the above-mentioned comparison operation based on the average value of the brightness value of the entire image data. However, in a case where the area of ​​the subject as the light source is small relative to the entire screen, the proportion of the signal change caused by the lighting state of the light source or the saturation state of the charge accumulation node 41 of the camera unit 120 in the average value of the entire image data is small, and it may be difficult to determine the magnitude relationship of the brightness value as intended. From the perspective of effectively detecting more local brightness information, for example, as described above, the comparison unit 1405 divides each input image data into multiple blocks and compares the brightness values ​​in units of blocks. In addition, the comparison unit 1405 can also replace multiple blocks and compare the brightness values ​​of pixels at the same position in the image data A and the image data B.

[0168] In addition, the block size when dividing the input image data into blocks can be set to a fixed size, or can be variable for example, during each frame. In addition, before calculating the average value of each block, a smoothing process such as a median filter can be added to remove defective components such as pixel defects of the camera unit 120.

[0169] In addition, when there are blocks with a higher average brightness value in image data A and blocks with a higher average brightness value in image data B among the divided multiple blocks according to the positions of the blocks, the comparison unit 1405 may also use the following method for comparison. For example, the comparison unit 1405 may select image data with a large number of blocks with a higher average brightness value. In addition, for example, the comparison unit 1405 may select image data including blocks with the highest average brightness value. In addition, for example, the comparison unit 1405 may compare a value obtained by averaging the average brightness values ​​of the blocks of image data A with a value obtained by averaging the average brightness values ​​of the blocks of image data B, and select image data with a higher average brightness value. The comparison method of the comparison unit 1405 is not limited to these methods, and any method may be used as long as it can select image data with a region with a high brightness value suitable for image data used in high dynamic range imaging.

[0170] When the difference in the average brightness values ​​of image data A and image data B is small and equal to or less than the threshold value in all blocks, the comparison unit 1405 selects image data A stored in the frame memory MEM3 as the previously selected image data, for example.

[0171] Fig.13 4 is a diagram for explaining another example of the change in the pixel signal level of the charge accumulation node 41. Fig.13 In, as Fig.10The imaging scene different from the imaging scene in shows an example in which the light source is turned on only during a part of the exposure time TeL during which the image data IL is normally read. Fig.13 (a) is a diagram showing the lighting timing of the light source. Fig.13 The upper side of (b) is a diagram showing the pixel signal level of the charge storage node 41 of the pixel 10 in the area corresponding to the light source on the imaging plane. Fig.13 The lower side of (b) is a diagram showing an example of an image corresponding to the non-destructive read image data Ind1, Ind2, and Ind3 and the normal read image data IL. Fig.13 (c) is a diagram showing an example of an image corresponding to the non-destructive read image data Ind1 and the difference image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3.

[0172] exist Fig.13 In the process, the timing of the normal readout operation and the non-destructive readout operation of the pixel 10 is Figure 6 and Figure 7 Same as Fig.13 As shown in (a) and (b), the light source is turned on at time tON and turned off at time tOFF.

[0173] like Fig.13 As shown in (b), the pixel signal level starts to increase from time tON and does not increase after time tOFF. The image corresponding to the non-destructive readout image data Ind1, Ind2 and Ind3 obtained by non-destructive readout at each time based on the pixel signal level and the normal readout image data IL obtained by normal readout at the exposure time TeL is, for example, Fig.13 The image shown in the lower side of (b) is shown in FIG. 1. Furthermore, the non-destructive read image data Ind1, and the difference image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3 obtained in this way are, for example, Fig.13 The image shown in (c) is outputted through the operation of the image selection unit 141, where the light source part is brightest, as the output image data IS of the short-time exposure image outputted to the subsequent HDR synthesis unit 142. Fig.13 In the graph of pixel signal levels (b), the differential image data Ind2-Ind1 also obtains data for the time period in which the pixel signal level increases with the largest slope relative to the exposure time, and is regarded as image data that best obtains information on the subject when the light source is turned on and is farthest from the saturation state of the charge accumulation node 41 of the camera unit 120.

[0174] [HDR synthesis unit operation]

[0175] The HDR synthesis unit 142 synthesizes the output image data IS of the short-time exposure image of the image selection unit 141 and the normal read image data IL of the long-time exposure image output from the camera unit 120 to generate the synthesis result image data IHDR of the high dynamic range image. As for the means for generating the synthesis result image data IHDR of the high dynamic range image, a well-known method can be used. For example, for each pixel value of the synthesis result image data IHDR, the result of weighted addition of the corresponding pixel values ​​of the output image data IS of the image selection unit 141 and the normal read image data IL can be used. That is, when the pixel value at the position x of the synthesis result image data IHDR is set to IHDR(x), it can be obtained by:

[0176] IHDR(x)=α(x)×IL(x)+(1-α(x))×IS(x) is calculated by formula (1). Here, α(x) is a real number satisfying 0≤α(x)≤1. In addition, IL(x) is the pixel value at position x of the normally read image data IL, and IS(x) is the pixel value at position x of the output image data IS. In order to compensate for the difference in each pixel value of IS(x) and IL(x) caused by the difference in exposure time, the calculation of formula (1) can also be performed after appropriately applying a gain to the pixel value of each image data. In addition, the coefficient α(x) in formula (1) is a value that varies for each pixel or for each block in which a plurality of pixels are aggregated, and can be determined, for example, based on both the brightness value and the noise level of the output image data IS and the normally read image data IL. For example, in a pixel whose brightness value of the normally read image data IL is close to the maximum value, α(x) is made smaller. Furthermore, for example, among the pixels of the output image data IS, in a pixel having a large variance with surrounding pixels and a high noise level, α(x) is made larger.

[0177] As described above, the camera device 100 according to the present embodiment performs one normal readout and at least one non-destructive readout within one frame period. Through one normal readout and at least one non-destructive readout, a plurality of image data obtained by performing readout in a non-destructive manner for a plurality of times are output. Furthermore, the camera device 100 generates at least one image data of an exposure time equivalent to the image data captured with an exposure time shorter than the exposure time in the normal readout, based on the difference between at least one non-destructive readout image data or the difference between the normal readout image data and the non-destructive readout image data. Furthermore, from the image data that is initially read out and output in a non-destructive manner, i.e., the initial image data, and at least one difference image data, the output image data IS that is most suitable as a short-exposure image used in high dynamic range synthesis is selected, for example, based on the local brightness value, and the synthesized result image data IHDR of the high dynamic range image is generated by high dynamic range processing based on the output image data IS selected in this way and the normal readout image data IL. In the imaging device 100 according to the present embodiment, the state of the subject is continuously observed by performing readout multiple times in a non-destructive manner within one frame period. Therefore, even when the state of the subject changes with time, such as a flickering light source, image data of a short-exposure image that is most suitable for high dynamic range imaging can be generated in accordance with this. In the imaging device 100, for example, the state of the subject does not change with time. Figure 2 In the case where neither the long exposure image nor the short exposure image can properly capture the state of the subject as described in the description of , the imaging device 100 can effectively perform high dynamic range imaging for any subject. In other words, the imaging device 100 can effectively expand the dynamic range of the captured image.

[0178] (Implementation Method 2)

[0179] Next, the second embodiment will be described. The second embodiment is different from the first embodiment described above in that an image selection unit 141B is provided instead of the image selection unit 141 provided in the imaging device 100 .

[0180] Fig.14 1 is a block diagram showing an exemplary configuration of the image selection unit 141B according to the present embodiment. The image selection unit 141B differs from the image selection unit 141 according to the first embodiment in that it has a selection control unit 1408. Other components common to the image selection unit 141 according to the first embodiment are modified in accordance with the present embodiment. Figure 8 The same reference numerals are used and detailed description is omitted. Fig. 9 The operations of the image selection unit 141 according to the first embodiment shown are mostly the same, and only the differences will be described below.

[0181] The selection control unit 1408 controls the start and stop of the overall operation of the image selection unit 141B based on the output of the subtraction unit 1404 in each operation step of the image selection unit 141B, that is, the brightness value of the difference image data between the non-destructive read image data. Fig.14 Although not shown in the figure, the start and stop of the overall operation of the image selection unit 141B is achieved by, for example, setting the output end of the camera unit 120 and the input end of the input distribution unit 1401 to a connected state or a disconnected state. When set to a disconnected state, the output of the image selection unit 141B is fixed to the image data stored in the frame memory MEM3 at that moment, and the image selection operation in the image selection unit 141B is stopped. In addition, it is also possible to stop not only the image selection operation in the image selection unit 141B, but also the non-destructive readout itself in the camera unit 120 and the camera control unit 130. However, the readout stopped at this time is only in Fig. 9 One of the non-destructive readouts performed at times t2 to t4 is performed, and the normal readout at time t5 is implemented.

[0182] Fig.15 1408 is a flowchart showing an example of the operation of the selection control unit 1408. First, when the non-destructive reading of the imaging unit 120 in one frame period is started, the selection control unit 1408 obtains the comparison result of the brightness value in the comparison unit 1405 (step S11). Fig. 9 In each of the illustrated operation steps, for example, local brightness values ​​of the initial image data and the difference image data supplied from the subtraction unit 1404 to the comparison unit 1405 are monitored.

[0183] Next, the selection control unit 1408 determines whether the brightness value of the difference image data newly generated by the subtraction unit 1404 is smaller than the brightness value of the image data previously selected by the comparison unit 1405, that is, the provisional output image data at this moment (step S12). If the brightness value of the newly generated difference image data is smaller than the brightness value of the previously selected image data (step S12: yes), the selection control unit 1408 stops the entire operation of the image selection unit 141B (step S13). If Fig. 9 If the brightness value of the difference image data generated in each of the action steps shown is lower than the brightness value of the initial image data or the difference image data in the previous step, the image selection operation of the image selection unit 141B is stopped by the above method at the time when the action step is completed, and the image selection unit 141B does not perform subsequent image selection operations until the next frame period starts. As a result, the image selection unit 141B selects the previously selected image data as the output image data IS and outputs it to the HDR synthesis unit 142.

[0184] Thereafter, for example, the selection control unit 1408 starts the entire operation of the image selection unit 141B in synchronization with the start of the next frame period.

[0185] When the brightness value of the newly generated difference image data is greater than the brightness value of the previously selected image data (step S12: No), the selection control unit 1408 determines whether the non-destructive readout of the imaging unit 120 has been performed a predetermined number of times within one frame period (step S14). When the non-destructive readout of the imaging unit 120 has been performed a predetermined number of times within one frame period (step S14: Yes), the selection control unit 1408 ends the operation. When the non-destructive readout of the imaging unit 120 has not been performed a predetermined number of times within one frame period (step S14: No), the operation from step S11 is repeated.

[0186] Generally speaking, the pixel signal level obtained in the camera unit 120 is as follows: Fig.10 As shown in FIG. 1 , the pixel signal level does not increase infinitely as the exposure time passes, but reaches saturation at a certain pixel signal level. If a certain amount of light is obtained from the subject in the imaging unit 120, the possibility of such a saturated state of the pixel signal level will increase the later the non-destructive readout is. Fig.10 As described in the description of , when the pixel signal level is close to or reaches saturation, and the slope of the increase in the pixel signal level relative to the exposure time is small or approximately 0, the brightness value of the difference image data is small. Therefore, it can be considered that when the brightness value of the difference image data read out at a later time is small, the possibility of a decrease in the brightness value of the difference image data due to the saturation of the above-mentioned pixel signal level is high. Therefore, if the brightness value in the difference image data obtained at a later time is smaller than the brightness value of the previous difference image data, the image selection unit 141B involved in this embodiment regards that the pixel signal level of the area corresponding to the light source part in the camera unit 120 is close to the saturation state, and it is difficult to obtain image data of a short-exposure image suitable for high dynamic range processing in the image selection using the difference image data generated thereafter, and the subsequent image selection action is stopped. Therefore, by reducing the number of times the image selection action or non-destructive reading is performed, the effect of improving the processing speed and reducing power consumption can be obtained.

[0187] In addition, the criterion for determining the brightness value of the selection control unit 1408 in step S12 is not limited to the above example. The selection control unit 1408 may also determine whether the brightness value of the difference image data newly generated by the subtraction unit 1404 is smaller than the brightness value of the image data previously selected by the comparison unit 1405 by a predetermined threshold or more, etc. Fig.12For example, when the temporary output image data, which is the image data previously selected by the comparison unit 1405, is selected, the selection control unit 1408 may stop the entire operation of the image selection unit 141B.

[0188] (Implementation method 3)

[0189] Next, the third embodiment is described. The third embodiment is different from the first embodiment in that an imaging unit 120B is provided instead of the imaging unit 120 provided in the imaging device 100. The photoelectric conversion unit 13 is not limited to the one having Figure 4 and Figure 5 The photoelectric conversion layer 15 shown in FIG. 1 may be a photodiode (PD: Photo Diode) as the photoelectric conversion unit, for example.

[0190] Fig.16 1 is a schematic diagram showing an exemplary circuit configuration of the imaging unit 120B according to the present embodiment. Figure 4 The common components of the imaging unit 120 in the first embodiment shown in the figure are added. Figure 4 The same symbols are used and descriptions are omitted.

[0191] like Fig.16 As shown, the imaging unit 120B has a pixel array PAB including a plurality of pixels 10B arranged in two dimensions. Each pixel 10B has a photodiode 1501 instead of the photoelectric conversion unit 13 of the pixel 10. In addition, the pixel 10B has a capacitor 1502 connected to the anode of the photodiode 1501 and storing the electric energy obtained by the photodiode 1501 as charge. The charge accumulated in the capacitor 1502 is detected as a voltage on the capacitor 1502, and the voltage is amplified by the source follower circuit composed of the signal detection transistor 24 and output to the vertical signal line 47. The source follower circuit composed of the signal detection transistor 24 has a high input impedance and therefore has a feature of not consuming the charge accumulated in the capacitor 1502. Therefore, the voltage can be taken out multiple times in a manner that does not destroy the charge accumulated in the capacitor 1502, so non-destructive reading can also be performed in the imaging unit 120B.

[0192] The pixel 10B of the imaging unit 120B involved in this embodiment has a reset transistor 28 similarly to the imaging unit 120 involved in Embodiment 1. In the reset transistor 28, the source terminal and the drain terminal of the reset transistor 28 are respectively connected to the two ends of the capacitor 1502. By controlling the potential of the reset control line 48, the charge accumulated in the capacitor 1502 can be reset and discharged. Thus, the imaging unit 120B realizes the reset action of the pixel 10B accompanied by the normal readout. In addition, in this example, a PD voltage line 1503 is connected to the cathode of the photodiode 1501, and the PD voltage line 1503 is connected to the PD voltage supply circuit 1504. The PD voltage supply circuit 1504 supplies the same voltage as the power supply voltage VDD to the cathode of the photodiode 1501, for example.

[0193] The operation of the imaging unit 120B according to this embodiment can be Figure 7 The operation mode of the imaging unit 120 involved in the embodiment 1 shown is performed in exactly the same manner. That is, by setting the potential of the address control line 46 to a high level, the potential of the charge accumulation node 41 is output to the vertical signal line 47 via the address transistor 26 whose gate is connected to the address control line 46, and is transmitted to the column signal processing circuit 37. In addition, by setting the potential of the reset control line 48 to a high level, the accumulated charge of the capacitor 1502 can be reset via the reset transistor 28 whose gate is connected to the reset control line 48, so that the capacitor 1502 returns to the initial state where the charge can be accumulated. When performing non-destructive reading, only the potential of the address control line 46 is set to a high level, and the potential of the reset control line 48 is maintained at a low level, thereby realizing non-destructive reading.

[0194] According to the present embodiment, when the imaging unit 120B includes the photodiode 1501 , high dynamic range imaging can be performed by the same method as that of the first embodiment.

[0195] (Other embodiments)

[0196] As mentioned above, although the imaging device according to one or more aspects has been described based on each embodiment, the present disclosure is not limited to these embodiments.

[0197] In the above-mentioned embodiment, the normal read image data IL as an example of the second image data is used as the image data of the long exposure image used in the synthesis of the HDR synthesis unit 142, but it is not limited to this. It is also possible to use a certain image data other than the non-destructive read image data Ind1 (i.e., the initial image data) included in the plurality of image data outputted in a non-destructive manner by the camera unit 120 or 120B during one frame as the second image data of the long exposure image. For example, the image processing unit 140 may select image data whose lowest value of the brightness value among the pixels of the image data is within a prescribed range from the non-destructive read image data Ind2 and Ind3 and the normal read image data IL, and use it for synthesis. The prescribed range is set, for example, according to the purpose of use of the captured image. In addition, for example, from the perspective of repeating the exposure time, the image processing unit 140 may select the second image data of the long exposure image from among the non-destructive read image data Ind2 and Ind3 and the normal read image data IL, from the non-destructive read image data or the normal read image data whose exposure time includes the substantial exposure period of the finally selected output image data IS.

[0198] In the above-mentioned embodiment, the output image data IS is selected from the non-destructive read image data Ind1 as the initial image data, and the differential image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3, but the present invention is not limited thereto. For example, when the exposure time of the non-destructive read image data Ind1 is significantly shorter than the actual exposure time of the differential image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3, or when the reset noise in the reset operation is large, the non-destructive read image data Ind1 may not be included in the candidates of the image data selected as the output image data IS, and the non-destructive read image data Ind1 may not be input to the image processing unit. In other words, the output image data IS may be selected from the differential image data Ind2-Ind1, Ind3-Ind2, and IL-Ind3.

[0199] In addition, various modifications that can be conceived by those skilled in the art to the present embodiment and combinations of components in different embodiments are also included in the scope of the present disclosure as long as they do not depart from the gist of the present disclosure.

[0200] Industrial Applicability

[0201] The imaging device according to the present disclosure can be applied to various camera systems and sensor systems such as digital cameras, medical cameras, surveillance cameras, vehicle-mounted cameras, digital SLR cameras, or digital mirrorless single-lens cameras.

[0202] Description of reference numerals:

[0203] 10, 10B pixels

[0204] 11 Pixel electrode

[0205] 12 Counter electrode

[0206] 13 Photoelectric conversion unit

[0207] 14 Signal detection circuit

[0208] 15 Photoelectric conversion layer

[0209] 20 Semiconductor substrate

[0210] 20t Component separation area

[0211] 24 Signal detection transistor

[0212] 24d, 24s, 26s, 28d, 28s impurity region

[0213] 24g, 26g, 28g Gate electrode

[0214] 26 Address transistors

[0215] 28 Reset transistor

[0216] 32 Voltage supply circuit

[0217] 34 Reset voltage source

[0218] 36 Vertical scanning circuit

[0219] 37 columns signal processing circuit

[0220] 38 Horizontal signal readout circuit

[0221] 39 Pixel drive signal generation circuit

[0222] 40 Power cord

[0223] 41 Charge accumulation node

[0224] 42 Sensitivity control line

[0225] 44 Reset voltage line

[0226] 46 address control lines

[0227] 47 vertical signal line

[0228] 48 Reset control line

[0229] 49 Horizontal common signal line

[0230] 50 interlayer insulation layer

[0231] 52 Plug

[0232] 53 Wiring

[0233] 54, 55 Contact plug

[0234] 56 wiring layers

[0235] 100 Camera Device

[0236] 110 lens optical system

[0237] 120, 120B Camera Department

[0238] 130 Camera Control Unit

[0239] 140 Image Processing Department

[0240] 141, 141B Image selection unit

[0241] 142 HDR Compositing Department

[0242] 1401 Input Distribution Unit

[0243] 1402, 1403, 1407 frame memory

[0244] 1404 Subtraction Department

[0245] 1405 Comparative Department

[0246] 1406 Selector

[0247] 1408 Selection Control Unit

[0248] 1501 Photodiode

[0249] 1502 Capacitor

[0250] 1503 PD voltage line

[0251] 1504 PD voltage supply circuit

[0252] PA, PAB pixel array

Claims

1. A camera device comprising: Camera department, including: A plurality of pixels each including a photoelectric conversion unit that generates signal charges and a charge accumulation unit that accumulates the signal charges; and a signal readout circuit for reading out a signal corresponding to the signal charge accumulated in the charge accumulation section, wherein the imaging section outputs image data based on the signal; as well as A processing circuit processes the image data output by the camera unit, The imaging unit reads out a plurality of images in a non-destructive manner, thereby sequentially outputting a plurality of image data within a frame period. the processing circuit, obtaining a difference between two consecutively output image data among the plurality of image data, thereby generating at least one differential image data; selecting one image data as first image data from among the plurality of image data the initial image data outputted first and the at least one difference image data based on the respective brightness values, The first image data is synthesized with second image data which is any image data other than the initial image data included in the plurality of image data to generate third image data.

2. The imaging device according to claim 1, The second image data is image data output last among the plurality of image data.

3. The imaging device according to claim 1 or 2, the processing circuit, selecting one of the initial image data and the first difference image data in the one frame period among the initial image data and the at least one difference image data based on their respective brightness values; The first image data is selected by repeatedly selecting one of the selected image data and the newly generated difference image data based on their respective brightness values.

4. The imaging device according to claim 3, When the brightness value of the newly generated difference image data is smaller than the brightness value of the previously selected image data, the processing circuit selects the previously selected image data as the first image data.

5. The imaging device according to claim 1 or 2, The processing circuit comprises: a first memory storing one of two image data outputted continuously among the plurality of image data outputted by the imaging unit, and a second memory storing the other of the two image data outputted continuously; a difference calculation circuit for calculating a difference between two image data stored in the first memory and the second memory, thereby generating difference image data; a third memory for storing the initial image data or the difference image data generated by the difference calculation circuit; as well as The comparison circuit compares the brightness value of the difference image data generated by the difference calculation circuit with the brightness value of the initial image data or the difference image data stored in the third memory.

6. The imaging device according to claim 1 or 2, The photoelectric conversion unit includes a first electrode, a second electrode, and a photoelectric conversion layer between the first electrode and the second electrode.

7. The imaging device according to claim 1 or 2, The photoelectric conversion unit includes a photodiode.

8. An image processing method, applied to a camera device, comprising: sequentially inputting a plurality of image data, wherein the plurality of image data is read out a plurality of times in a non-destructive manner and sequentially output within a one-frame period, obtaining a difference between two consecutively input image data among the plurality of image data, thereby generating at least one difference image data; selecting one image data as first image data from among the plurality of image data the initial image data inputted first and the at least one difference image data based on the respective brightness values, The first image data is synthesized with second image data which is any image data other than the initial image data included in the plurality of image data to generate third image data.

9. The image processing method according to claim 8, The second image data is image data output last within the one frame period among the plurality of image data.

10. The image processing method according to claim 8 or 9, selecting one of the initial image data and the first difference image data in the one frame period among the initial image data and the at least one difference image data based on their respective brightness values; The first image data is selected by repeatedly selecting one of the selected image data and the newly generated difference image data based on their respective brightness values.

11. The image processing method according to claim 10, When the brightness value of the newly generated difference image data is smaller than the brightness value of the previously selected image data, the previously selected image data is selected as the first image data.

Images