Photoelectric conversion devices, photoelectric conversion systems and equipment

Abstract

This technology contributes to the miniaturization and / or high-speed operation of photoelectric conversion devices or systems. [Solution] The photoelectric converter has multiple pixel blocks in which multiple pixels are arranged in a matrix, and performs a first imaging operation that includes an exposure time based on a preset first exposure condition and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block during one frame period, and a storage unit, wherein the storage unit holds a first image signal whose data amount is smaller than the data amount of the image signal acquired by the photoelectric converter during the first imaging operation.

Description

[Technical Field]

[0001] This disclosure relates to photoelectric conversion devices, photoelectric conversion systems, and equipment. [Background technology]

[0002] To broaden the dynamic range of an imaging device, a method has been proposed to change the exposure conditions of the imaging sensor for each region. Patent Document 1 discloses dividing the entire light-receiving area of ​​the imaging sensor into multiple regions and setting the exposure time for each region. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-129144 [Overview of the project] [Problems that the invention aims to solve]

[0004] When performing a first imaging operation with exposure times set commonly across multiple regions and a second imaging operation with exposure times set separately for each region, the amount of signal data generated increases compared to performing only one of the two imaging operations. This increases the memory area required to hold this data, potentially hindering the miniaturization of the photoelectric converter or photoelectric conversion system. Alternatively, the increased amount of generated data lengthens the data processing time or data output time of the photoelectric converter or photoelectric conversion system. This can hinder the acceleration of the photoelectric converter or photoelectric conversion system. This disclosure provides a technology that contributes to either or both miniaturization and acceleration of the photoelectric converter or photoelectric conversion system. [Means for solving the problem]

[0005] One aspect of the present disclosure is a photoelectric conversion system including a photoelectric conversion device and a storage unit. The photoelectric conversion device includes a plurality of pixel blocks each having a plurality of pixels arranged in a matrix. In one frame period, the photoelectric conversion device performs a first imaging operation including an exposure time based on a preset first exposure condition, and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block. The storage unit holds a first image signal having a smaller amount of data than the amount of data of the image signal acquired by the photoelectric conversion device through the first imaging operation.

Effect of the Invention

[0006] According to the present disclosure, it is possible to provide a technology that contributes to miniaturization and / or speed increase of a photoelectric conversion device or a photoelectric conversion system.

Brief Description of the Drawings

[0007] [Figure 1] A diagram showing a schematic configuration example of an imaging device and a connection to an external controller. [Figure 2] A diagram for explaining an imaging element unit. [Figure 3] A diagram showing a configuration example of an exposure correction unit. [Figure 4] A diagram showing the relationship between the control of the exposure time and the light emission cycle of an LED light source of a shooting target. [Figure 5] A diagram for explaining the configuration of an exposure image and the state of holding the exposure image in a line buffer. [Figure 6] A diagram for explaining the relationship between the exposure condition, the exposure time, and the analog gain in area-by-area exposure. [Figure 7] A diagram showing a configuration example of an exposure correction unit. [Figure 8] A flowchart showing the process flow. [Figure 9] A flowchart showing the process flow. [Figure 10] A diagram for explaining the configuration of an exposure image and the state of holding the exposure image in a line buffer. [Figure 11] A diagram showing a configuration example of an exposure correction unit. [Figure 12]A diagram showing an example of the configuration of the exposure compensation unit. [Figure 13] A diagram illustrating the structure of the exposed image and the state in which the exposed image is held in the line buffer. [Figure 14] A diagram showing an example of the configuration of the exposure compensation unit. [Figure 15] A diagram illustrating the state of image information retention in the line buffer. [Figure 16] A flowchart showing the processing flow. [Figure 17] A flowchart showing the processing flow. [Figure 18] An example of applying the apparatus according to the embodiment to equipment. [Modes for carrying out the invention]

[0008] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0009] [Embodiment 1] Figure 1 is a block diagram showing a schematic configuration example of an imaging device 100 to which the image processing apparatus according to this embodiment is applied, and its connection to an external controller. The imaging device 100 of this embodiment also has various configurations that are common to general imaging devices, but for the sake of simplicity in illustration and explanation, only the main components according to this embodiment are shown in Figure 1. The imaging operation performed by setting exposure conditions for each region including pixel blocks (hereinafter referred to as "region-specific exposure") will be explained using Figures 1 to 3. Note that each component described below is an example, and the functions of multiple components described later may be combined into one or separated. Alternatively, one component may also perform other functions.

[0010] The imaging device 100 according to this embodiment includes a synchronization control unit 101, an image sensor unit 103, an analog-to-digital (A / D) conversion unit 104, an exposure compensation unit 105, a grayscale conversion unit 106, an image output unit 108, an exposure time control unit 109, a gain control unit 110, and an exposure condition determination unit 111. It also has a serial interface (SIO I / F) 141 and a register 142 for storing setting values ​​in order to reflect settings provided by an external controller 10. The imaging device 100 is connected to the external controller 10 by a serial communication line 11 and an output signal line 12.

[0011] The imaging sensor 102 of this embodiment may include an imaging sensor section 103 on which pixels including photoelectric conversion elements are arranged, and an A / D conversion section 104 that performs analog-to-digital (A / D) conversion of the photoelectric conversion signals from the pixel section. The pixel section has a plurality of pixels arranged across multiple rows and multiple columns. Each of these pixels includes a photoelectric conversion element. Each of the plurality of pixels may also be provided with a pixel output circuit that outputs the signal from the photoelectric conversion element, if necessary. The entire or a part of the imaging sensor section 103 including the photoelectric conversion elements may be called a photoelectric conversion device. The entire or a part of the imaging sensor 102 including the A / D conversion section 104 and other processing circuits may also be called a photoelectric conversion device. Furthermore, the entire or a part of the imaging device 100 may be called a photoelectric conversion system. In the following description, an imaging sensor is used as an example of a photoelectric conversion device, but the invention is not limited to this form. For example, the contents of this disclosure can also be applied to a sensor that performs at least one of distance measurement and photometry.

[0012] The synchronization control unit 101, exposure time control unit 109, gain control unit 110, and exposure condition determination unit 111 can each be considered an image control device that controls imaging. The image control device may include at least one of the synchronization control unit 101, exposure time control unit 109, gain control unit 110, and exposure condition determination unit 111. The exposure correction unit 105 and grayscale conversion unit 106 each function as an image processing device that performs image processing on the exposed image 122 or a signal generation unit 130 that generates a signal. The image processing device may include at least one of the exposure correction unit 105 and grayscale conversion unit 106.

[0013] Furthermore, the imaging device 100 may include a control unit 150 that controls the imaging device 100. The control unit 150 may be included in each of the parts of the imaging device 100, such as the synchronization control unit 101, exposure compensation unit 105, tone conversion unit 106, exposure time control unit 109, gain control unit 110, and exposure condition determination unit 111. The control unit 150 can control each of the parts of the imaging device 100, such as the synchronization control unit 101, exposure time control unit 109, gain control unit 110, exposure compensation unit 105, and tone conversion unit 106. The control unit 150 can also control part or all of the imaging device 100.

[0014] The imaging device 100 will be described in detail, starting with the image sensor unit 103. The image sensor unit 103 has an imaging area (light-receiving area) for imaging. Multiple light-receiving elements are arranged in the imaging area corresponding to pixels. The light-receiving elements may be photoelectric conversion elements. Furthermore, the imaging area is divided into multiple areas called pixel blocks 201, which contain multiple pixels, as shown in Figure 2. In other words, multiple pixel blocks 201 may be arranged in the image sensor unit 103. Multiple pixels 202 may be arranged in a matrix within each pixel block 201.

[0015] The image sensor unit 103 can drive imaging operations on a pixel block (region) basis. The image sensor unit 103 can determine exposure conditions for each region and perform exposure operations with different exposure times for each region. The exposure time corresponds to the charge storage time during which the photoelectric conversion element contained in the pixel can store charge. The pixel block (region) will be explained later with reference to Figure 2. Here, the photoelectric conversion element is described as having a configuration that uses a PN junction, in which a P-type semiconductor region and an N-type semiconductor region are joined, to store charge. However, the photoelectric conversion element is not limited to this form. For example, the photoelectric conversion element may be an avalanche photodiode (APD), and each pixel may operate as a single-photon avalanche diode (SPAD). In this case, the pixel includes an APD, a quench element connected to the output node of the APD, and a waveform shaping circuit connected to the output node that shapes the output of the APD into a pulse waveform. Furthermore, the pixel includes a counter that counts the pulse signals output from the waveform shaping circuit. The exposure time described below can be defined as the period from when the counter starts counting until it stops.

[0016] In this embodiment, the image sensor unit 103 can have its exposure time set for each region by an exposure control signal 117 supplied from the exposure time control unit 109. Exposure can be performed for each region with the exposure time set for each region. The exposure control signal 117 is a signal for setting the exposure time for each region of the image sensor unit 103. The image sensor unit 103 performs exposure for each region with the exposure time set by the exposure control signal 117, reads out the charge accumulated in each pixel as a pixel potential 118 from each pixel and outputs it to the A / D conversion unit 104.

[0017] The A / D conversion unit 104 converts the pixel potential 118 read from the image sensor unit 103 from analog to digital and converts it into a digital value. The A / D conversion unit 104 can be set by the gain control unit 110 to have analog gains 121 corresponding to each region. The A / D conversion unit 104 amplifies the signal from the pixel potential 118 output from the image sensor unit 103 using the analog gains 121 set for each region, and then converts it from analog to digital and converts it into a digital value.

[0018] The image signal, which is a digital signal that has been amplified by the analog gain 121 set for each region in the A / D conversion unit 104 and then converted from analog to digital, is called the exposed image 122. The exposed image 122 output from the A / D conversion unit 104 is sent to the exposure condition determination unit 111 and the exposure correction unit 105.

[0019] The exposure condition determination unit 111 can determine the exposure time 112 and analog gain value 113 based on the input exposure image 122 so that the imaging conditions for each region are optimal, and can update the exposure time 112 and analog gain value 113. The exposure condition determination unit 111 can determine the exposure conditions, including the exposure time 112 and analog gain value 113. As an example of the update, the exposure condition determination unit 111 acquires a histogram of the signal level values ​​(pixel values) of pixels for each pixel block based on the brightness distribution of the exposure image 122. If the histogram of pixel values ​​in a pixel block (region) shows that the pixel values ​​are distributed towards the brighter side, the exposure condition determination unit 111 changes and updates the exposure time 112 and analog gain value 113 for that pixel block to settings that allow for darker imaging. The next imaging can be performed based on the updated values.

[0020] Furthermore, if the pixel values ​​are distributed on the dark side, the exposure condition determination unit 111 can change and update the exposure time 112 and analog gain value 113 of that pixel block to settings that will produce a brighter image. The exposure time 112 value for each region is sent to the exposure time control unit 109 and the exposure compensation unit 105. The analog gain value 113 for each region is sent to the gain control unit 110 and the exposure compensation unit 105.

[0021] The synchronous control unit 101 generates an exposure time output pulse 120 and a gain output pulse 114. The synchronous control unit 101 outputs the exposure time output pulse 120 to the exposure time control unit 109 and the gain output pulse 114 to the gain control unit 110. This allows the synchronous control unit 101 to synchronize the timing of the processing of the exposure time control unit 109 and the processing of the gain control unit 110.

[0022] The exposure time output pulse 120 is a signal used to control the timing at which the exposure time control unit 109 outputs the exposure control signal 117 to the image sensor unit 103. Based on the exposure time output pulse 120, the exposure time control unit 109 outputs the exposure control signal 117 to the image sensor unit 103, thereby setting the exposure time for each arbitrary pixel block of the image sensor unit 103.

[0023] Furthermore, the gain output pulse 114 is a signal for controlling the timing at which the gain control unit 110 outputs the analog gain 121 to the A / D conversion unit 104. Based on the gain output pulse 114, the gain control unit 110 outputs the analog gain 121 to the A / D conversion unit 104, thereby setting the gain applied to the pixel potential for each arbitrary pixel block. In this embodiment, the synchronization control unit 101 synchronizes the exposure time control unit 109 and the gain control unit 110 to control their operation. By applying the exposure time and analog gain in sync to the pixel potential from each pixel in each pixel block of the image sensor unit 103, an exposed image 122 can be output from the image sensor 102.

[0024] The exposure time control unit 109 generates an exposure control signal 117 for each region based on the exposure time output pulse 120 and the exposure time value 112 for each region, and outputs it to the image sensor unit 103. As a result, the exposure time corresponding to the exposure time 112 for each region is set to the image sensor unit 103 at an appropriate timing.

[0025] The gain control unit 110 outputs the region-specific analog gain value 113 to the A / D conversion unit 104 as region-specific analog gain 121 for the region-specific pixel potential 118 of the image sensor unit 103, in accordance with the timing of the gain output pulse 114. As a result, the A / D conversion unit 104 applies the corresponding region-specific analog gain 121 to the region-specific pixel potential 118 before performing analog-to-digital conversion. The converted analog-to-digital data is then sent to the exposure correction unit 105 and the exposure condition determination unit 111 as region-specific exposure images 122 under the control of the control unit 150.

[0026] The exposure correction unit 105 receives exposure images 122 for each region from the A / D conversion unit 104, accumulates exposure images 122 taken under different exposure conditions within the same frame, performs necessary processing, and then performs an addition process for each pixel. This process will be described later. Furthermore, the added images are subjected to a gradation expansion process based on the exposure time 112 and the analog gain value 113 to generate a gradation expanded image 123.

[0027] The exposure compensation unit 105 can, for example, generate a tonally expanded image 123 represented by 23 bits from an exposure image 122 for each region, which is represented by 10 bits. The detailed operation of the exposure compensation unit 105 will be described later. The generated tonally expanded image 123 is then sent to the tonal conversion unit 106.

[0028] The tone conversion unit 106 performs tone conversion on the tone-extended image 123 and outputs the tone-converted image 124 to the image output unit 108. In this embodiment, tone conversion is a process that converts, for example, the 23-bit tone-extended image 123 into a 12-bit signal by gamma conversion to generate the tone-converted image 124. Note that the tone conversion process in this embodiment is performed to reduce the data rate in subsequent processing. In this embodiment, the bit lengths of the exposed image 122 and the tone-converted image 124 are set to 10 bits and 12 bits, respectively, but these bit lengths are just examples and are not limited thereto.

[0029] The image output unit 108 outputs the grayscale-converted image 124 to a downstream configuration of the imaging device 100 or to an external source. In this embodiment, a controller 10 is connected as a processing module that receives image signals from the imaging device 100. The output signal line 12 connecting the image output unit 108 and the controller 10 may be an LVDS signal line with 16 data channels. However, the type of signal line and the data channel width are not limited by this embodiment and can be selected according to the amount of data to be transmitted and the data transmission speed.

[0030] Furthermore, the controller 10 is connected to the serial I / O (SIO) interface 141 of the imaging device 100 via a serial communication line 11. The SIO interface 141 is connected to a register 142, and the controller 10 can set the necessary information in the register 142 located inside the imaging device 100 via the SIO interface 141. The information set in the register 142 is transmitted to the exposure condition determination unit 111. The exposure condition determination unit 111 is able to determine the exposure conditions using the information previously set in the register 142.

[0031] An example of the configuration of the image sensor unit 103 is illustrated in Figure 2. The imaging area of ​​the image sensor unit 103 has multiple areas, which are shown as pixel blocks 201. Furthermore, multiple pixels 202 can be arranged in a matrix within each pixel block 201. In this embodiment, the number of pixels in the width direction 206 (horizontal line direction) of the imaging area of ​​the image sensor unit 103 is assumed to be 2000 pixels, and the number of pixels in the height direction 205 is assumed to be 1000 pixels (i.e., the number of horizontal lines in the vertical direction is 1000 lines). Also, the number of pixels in the width direction 204 (horizontal line direction) of each pixel block 201 is assumed to be 100 pixels, and the number of pixels in the height direction 203 is assumed to be 100 pixels (equivalent to 100 horizontal lines in the vertical direction). In this case, the number of pixel blocks 201 within the imaging area of ​​the image sensor unit 103 is 20 in the horizontal direction and 10 in the vertical direction. These pixel and line counts are examples for illustrative purposes and are not limited thereto.

[0032] The notation "Pixel Block [0,0]~[19,9]" written within each pixel block 201 shown in Figure 2 represents the position of each pixel block 201 within the imaging area. The value in brackets [ ] represents the horizontal and vertical index of each pixel block within the imaging area. In Figure 2, for example, the pixel block 201 located in the upper right of the image sensor 103 is represented as Pixel Block [19,0].

[0033] Furthermore, a set of pixel blocks represented by the same vertical index will be called a block row. Block row N consists of pixel blocks [0,N] to [19,N], where N is from 0 to 9 in Figure 2. For example, block row 5 consists of pixel blocks [0,5] to [19,5]. Note that the sizes (number of pixels in the vertical and horizontal directions) of the image sensor unit 103 and the pixel blocks 201 are not limited to the examples described above. Also, the shape and aspect ratio of the pixel 202 are not limited. The shape of a pixel block may be a rectangle, for example, instead of a square. Furthermore, a pixel block 201 may consist of only one pixel 202. In this embodiment, the exposure time and analog gain can be controlled for each pixel block 201 unit.

[0034] Here, the exposure time corresponds to the charge accumulation time during which charge accumulates in the pixels (photodetectors) of the image sensor 103. Therefore, for example, if the amount of incident light to the image sensor 103 is the same and the pixels do not saturate, the longer the exposure time, the higher the pixel potential 118 becomes, and a brighter image can be captured. In other words, if the amount of incident light is the same and pixel saturation is not considered, comparing, for example, an exposure time of (1 / 480) seconds with an exposure time of (1 / 30) seconds, the image can be captured brighter in the case of (1 / 30) seconds.

[0035] The analog gain is the gain value applied to the pixel potential 118 in the A / D conversion unit 104 during image capture. Therefore, the larger the analog gain value, the larger the digitized pixel value (the digital value converted from analog to digital after gain is applied) output from the A / D conversion unit 104.

[0036] Returning to Figure 1, the configuration and operation of the imaging device 100 in this embodiment will be described. Based on the exposure control signal 117, the image sensor unit 103 controls the exposure time for each region, i.e., in units of pixel blocks 201, and performs imaging. The image sensor unit 103 then outputs a pixel potential 118 corresponding to the charge accumulated for each pixel.

[0037] The A / D conversion unit 104 applies an analog gain 121, which is set to correspond to each pixel block of the image sensor unit 103, to the pixel potential 118 output from the image sensor unit 103, and then performs digital conversion to output the exposed image 122. In this embodiment, for explanatory purposes, the exposed image 122 is assumed to be a 10-bit digital value. The analog gain 121 can take on four values, for example, ×1, ×2, ×4, and ×8.

[0038] The exposure correction unit 105 performs gradation expansion processing on the exposed image 122 input from the A / D conversion unit 104 based on the exposure time 112 and analog gain value 113 set for each region, and outputs a gradation-expanded image 123. The exposure correction unit 105 recognizes the conditions under which each region's exposed image 122 was captured based on the region's exposure time 112 and region's analog gain value 113. Then, the exposure correction unit 105 corrects each region's exposed image 122 based on the conditions under which each region's exposed image 122 was captured.

[0039] The exposure correction unit 105 applies a tonal expansion process to the region-specific exposure images 122 sent from the A / D conversion unit 104, based on the exposure time 112 and analog gain value 113 applied to the imaging, to generate a tonally expanded image 123. For example, the exposure correction unit 105 recognizes the conditions under which the input region-specific exposure images 122 were captured based on the region-specific exposure time 112 and region-specific analog gain value 113, and corrects the region-specific exposure images 122 according to those conditions.

[0040] Furthermore, the exposure correction unit 105 performs gradation expansion processing on each exposed image 122, for example, which is represented by 10 bits, to generate a gradation-expanded image 123 represented by 23 bits. The generated gradation-expanded image 123 is then sent to the gradation conversion unit 106.

[0041] Next, the operation of the exposure compensation unit 105 will be described. Figure 3 is a block diagram showing an example of the configuration of the exposure compensation unit 105. The exposure compensation unit 105 has a line buffer 301 as a storage unit, an addition ratio calculation unit 302, an image addition unit 303, and a gradation expansion unit 304.

[0042] The operation of each component of the exposure compensation unit in this embodiment will be described with reference to Figures 4 and 5. In this embodiment, the image output frame rate when the imaging device 100 captures a video will be described as 30 frames / second. The length of one frame λ (frame duration) for this frame rate is 1 / 30 second (33.3333 ms (rounded to the fifth decimal place, the same applies hereinafter)). A frame can also be defined as the period from when the synchronization signal that controls the scanning circuit that scans multiple pixels in row units or column units becomes active until it becomes active again. When this synchronization signal becomes active, the scanning circuit can start scanning multiple pixels in row units or column units.

[0043] Furthermore, regarding the source of flicker, it is assumed that the maximum emission frequency of the LED light source included in the subject of imaging is 90 Hz and that it emits light at a constant period with a duty cycle of 50%. In this case, the longest emission period T of a light source that can generate flicker is 1 / 90 seconds (11.1111 ms). For explanatory purposes, in Figure 4, the periods are written as 33.3 ms and 11.1 ms.

[0044] In this embodiment, the imaging period of each frame is divided into two parts: a first period 401-1, 401-2 in the first half of the frame, and a second period 402-1, 402-2 in the second half of the frame that follows the first period, with each period defined as shown in Figure 4. In the following, when describing the first period, whether one or the other, it will be referred to as "first period 401," and when describing the second period, whether one or the other, it will be referred to as "second period 402."

[0045] In the example shown in Figure 4, one frame period includes a first period 401 and a second period 402. The first period 401 has a predetermined length that is longer than or equal to the LED emission period. In this embodiment, the predetermined period is the same length as 1 / 90 of the LED emission period T (approximately 11.1111 ms). Period 403 is the first exposure time of the image sensor in the first period. In the first period 401, the exposure time of the image sensor is always set for the entire first period.

[0046] In Figure 4, the first exposure times of the first period 401 within the two frames are represented as first exposure times 403-1 and 403-2, and the second exposure times of the second period 402 are represented as second exposure times 404-1 and 404-2. In the following, when the first exposure time is explained in a manner that does not refer to only one of the two exposure times, it will be referred to as first exposure time 403, and when the second exposure time is explained in a manner that does not refer to only one of the two exposure times, it will be referred to as second exposure time 404.

[0047] As shown in Figure 4, the first period 401 is set to be longer than the LED's light emission period, and therefore can include the period during which the LED is emitting light (the convex portion of the LED light emission in Figure 4). During the first period 401, charge accumulation (photoelectric conversion) is performed in the photoelectric conversion element over an exposure time based on a preset first exposure condition. The photoelectric conversion during the first period is referred to as the first imaging operation.

[0048] The second period 402-1 is the period of imaging for frame 1 (33.3 ms) excluding the first period 401-1. The second period 402-1 is the period during which region-specific exposure is performed according to the brightness of the object being imaged. The second exposure time 404-1 indicates the exposure time in the second period 402-1 set by region-specific exposure. In this embodiment, the second exposure time 404-1 is half the time of the second period 402-1. Similar control is performed for frame 2 following frame 1. In the second period 402, charge accumulation is performed over the exposure time based on the second exposure conditions determined for each pixel block. The photoelectric conversion in the second period is referred to as the second imaging operation.

[0049] Frame 2 shows an example where the exposure time 404-2 for the second period 402-2 is set to 1 / 8 of the duration of the second period 402-2 as a result of region-specific exposure. Therefore, during the second period 404-2, the LED light emission period is not captured by the image sensor, as shown in Figure 4. When the LED light emission period is not captured by the image sensor, flicker may occur in the captured video.

[0050] The exposure time control for the first period 401 and second period 402 of this image sensor is performed by the exposure time control unit 109 based on the exposure time 112 determined by the exposure condition determination unit 111 in Figure 1. The light emission period T of the LED light source to be flicker reduced is set in the register 142 from the external controller 10 and can be distributed from the register setting value 143. For the first imaging operation, the exposure condition determination unit 111 uses the light emission period T supplied as the register setting value 143 to determine the length of the first exposure time 403 of the first period 401. For the second imaging operation, the exposure condition determination unit 111 determines the exposure conditions for each region before the start of the second imaging operation. For example, the exposure time 404 in the second period 402 may be calculated based on the exposure image 122 in the second period 402 of the previous frame, based on an algorithm that calculates exposure conditions for each region.

[0051] The lengths of the first period 401 and the first exposure time 403 can be determined according to the destination where the imaging device will be used as a product. The lengths of the first period 401 and the first exposure time 403 can be set in advance as data read by the external controller 10. Alternatively, the destination can be determined using location information such as GPS, and when the imaging device is started, the external controller 10 can refer to a table corresponding to the location information and set it in the register 142. In addition, the lengths of the first period 401 and the first exposure time 403 can be set according to the light emission period of the LED for which flicker is to be avoided.

[0052] In Figure 3, the exposure compensation unit 105 first receives the exposure image 122 of the first period 401-1 of frame 1 in Figure 4. In this embodiment, the image signal of the exposure image 122 is explained as being input to the exposure compensation unit 105 as a Bayer array, with each set of four pixels consisting of R pixels corresponding to red, Gr pixels corresponding to green, Gb pixels corresponding to green, and B pixels corresponding to blue, as shown in Figure 5. Note that the set of pixels is not limited to a Bayer array. Alternatively, the pixels may simply be arranged in a matrix. Here, the image signal of the exposure image 122 of the first period of a certain frame at a certain pixel position will be called the image signal O1.

[0053] As mentioned above, the image signal O1 is assumed to contain four pixels: R pixels, Gr pixels, Gb pixels, and B pixels. The operations on the image signal O1 described below are applied independently to each of the four pixels, R pixels, Gr pixels, Gb pixels, and B pixels, for each pixel component. The data input as the data value of the image signal O1 is held in the line buffer 301. When the data is held in the line buffer 301, the data is periodically thinned out in units of a Bayer array arranged in rows (horizontally), and the remaining thinned-out data is held in the line buffer 301. In this embodiment, the image signal O1 captured in the first period 401 is thinned out and the remainder is held in the line buffer 301. Here, as shown in Figure 5, the data is thinned out every other pixel in the horizontal direction shown in Figure 5 (horizontal line direction in Figure 2) in units of a Bayer array, and the remaining data is held in the line buffer 301. In the example in Figure 5, the data of pixels of the same color is thinned out every other pixel. The amount of data held in the line buffer 301 can be reduced by approximately half.

[0054] In the example in Figure 5, multiple sets of Bayer arrays are arranged in rows. The R, Gr, Gb, and B pixels of the even-numbered sets of image signals O1 in the row are assigned even numbers (0, 2, 4, ...) from the beginning of the row. Each pixel with an even number (hereinafter referred to as "even pixel") is held in the line buffer 301. On the other hand, the R, Gr, Gb, and B pixels of the odd-numbered sets of image signals O1 (1, 3, ...) are assigned odd numbers. In this embodiment, the image signals O1 from each pixel with an odd number (hereinafter referred to as "odd pixel") are decimated and not held in the line buffer 301. Hereafter, an even pixel in a line will be referred to as the 2n-th pixel (n is 0 or a natural number), and an odd pixel as the 2n+1-th pixel. The output data corresponding to each pixel output during the first period is the image signal O1. 2n , O1 2n+1 We will use the notation shown here. Image signal O1 2n This represents the image signal for the first period, which is the 2nth pixel in a row and is an even-numbered pixel image signal.

[0055] In the example of FIG. 4, after frame 1 starts and after the elapse of the first period 401-1, the data of the image signal O1 of frame 1 is first held in the line buffer 301. After the elapse of the time of the second period 402-1, the exposed image 122 of the second period of frame 1 is input. The image signal of the exposed image 122 of the second period of the same frame at this time shall be referred to as image signal O2. Also, the composite output data generated by the image addition unit 303 based on image signal O1 and image signal O2 shall be referred to as image signal O3.

[0056] Image signal O3 is an image signal obtained by correcting and adding image signal O1 and image signal O2 at a predetermined ratio. Similar to image signal O1, image signal O2 and image signal O3 also include four pixels: R pixel, Gr pixel, Gb pixel, and B pixel. In the following description, the operations related to image signal O2 are independently applied to each of the four pixels of R pixel, Gr pixel, Gb pixel, and B pixel for each pixel component.

[0057] When image signal O2 is supplied to the exposure correction unit 105 as the exposed image 122 and the image signal O1 of the same frame is held in the line buffer 301, the image addition unit 303 reads image signal O1 from the line buffer 301 as the delayed image 306. The case where image signal O1 is held in the line buffer 301 is for even-numbered pixels. The data holding period in the line buffer 301 at this time is the length of the second period 402. For the data of the 2n-th even-numbered pixel, based on the addition ratio k calculated by the addition ratio calculation unit 302, image signal O2 2n and image signal O1 2n and the composite output, which is the composite image signal of the two, is calculated as image signal O3 2n based on the following formula (1). Here, the addition ratio k is a value for weighted addition and correction of image signal O2 2n and image signal O1 2n based on their respective imaging conditions. The addition ratio k will be described later. In the following formula, for example, the value of the data of image signal O1 2n is shown as O1 2n .

[0058] O32n =O2 2n +k×O1 2n ...(Formula 1) On the other hand, when the image signal O2 is supplied to the exposure correction unit 105 as the exposed image 122, if there are odd-numbered pixels, the data for the image signal O1 at the same pixel position is not held in the line buffer 301. Therefore, the image summing unit 303 cannot read the data for the image signal O1 from the line buffer 301 as a delayed image 306. In this case, the image signal O3 corresponds to one frame. 2n+1 The data values ​​are calculated based on the following formula (2).

[0059] O3 2n+1 =G×O2 2n+1 ...(Formula 2) Here, G is the image signal O2 2n+1 Corresponding signal O3 2n+1 The conversion ratio G 307 is used to convert to [the specified value]. The conversion ratio G 307 can be calculated as the ratio of the length of the second period to the frame period λ using the following equation (3).

[0060] G = 1 frame duration λ ÷ 2nd period length ... (Equation 3) In this embodiment, as shown in Figure 4, G is G can be calculated as G = 33.3 ms ÷ (11.1 ms + 11.1 ms) = 33.3 ÷ 22.2 = 1.5.

[0061] The meaning of equation (2) is the image signal O2 of the pixels obtained as a result of exposure in the second period. 2n+1 The purpose of multiplying by G is to obtain the pixel data value when the exposure conditions of the second period are applied to the first frame period. The value of this conversion ratio G 307 is also calculated by the addition ratio calculation unit 302 based on the register setting value 143 and provided to the image addition unit 303.

[0062] Next, the method for calculating the addition ratio k of the addition ratio calculation unit 302 will be explained using Figure 6. Figure 6 is a table showing the relationship between exposure conditions, exposure time, and analog gain in region-specific exposure. This table is applicable when the frame rate when capturing video is 30fps, and shows the relationship between exposure time and analog gain for the entire duration of one frame.

[0063] In Figure 6, the number at the intersection of the horizontal analog gain and exposure time in the table is referred to here as the EV value. The EV value is a power of 2 that represents the ratio of the brightness of the imaged object based on the difference in exposure conditions, assuming that the pixel signal level (pixel value) obtained by imaging is the same. If the brightness of the imaged object when imaged with an analog gain of ×8 and an exposure time of 1 / 30 second is L0, then the brightness LE of the imaged object when imaged with an EV value E and the same pixel value is obtained can be expressed by the following equation 4.

[0064] LE = L0 × 2^E ... (Equation 4) Here, 2^E represents 2 to the power of E (for example, 2^3 = 8). For example, the EV value is 3 when the analog gain is ×1 and the exposure time is 1 / 30 second. Similarly, the EV value is 4 when the analog gain is ×1 and the exposure time is 1 / 60 second. In this case, assuming the pixel values ​​of the image signals obtained by imaging are the same, the object imaged under the EV value of 4 condition was twice as bright as the object imaged under the EV value of 3 condition.

[0065] Furthermore, in region-specific exposure, a table like the one shown in Figure 6 can be used to estimate the brightness of the object being imaged in a given region based on the pixel values ​​captured in the previous frame and the exposure conditions used for that image. This allows for the calculation of the exposure conditions to be used for the next frame.

[0066] Here, we assume that the exposure conditions used in the first period 401-1 of frame 1 in Figure 4 are an analog gain of ×1 and a fixed exposure time for the entire first period. If we consider these exposure conditions to be applied to the entire frame, the exposure time will be 1 / 30th of the entire frame, and the EV value in this case is 3 according to Figure 6. On the other hand, the exposure conditions used in the second period 402-1 of frame 1 are such that the exposure time (second exposure time 404-1) is half that of the second period 402-1. Similarly, if we consider these exposure conditions to be applied to the entire frame, the exposure time will be 1 / 60th of the entire frame, or half of the entire frame. If the analog gain is ×4 at this time, the EV value in this case is 2 according to Figure 6.

[0067] This means that, under conditions where it was judged appropriate to use an EV value of 2 for region-specific exposure, the image was captured with an EV value of 3 during the first period 401-1. In the gradation expansion unit 304, which will be described later, the brightness of the pixel value is converted based on the EV value of 2 for the region-specific exposure conditions of the second period. Therefore, the value of the image signal O1 captured with an EV value of 3 is converted to a value equivalent to when it was captured with an EV value of 2, and then added to the image signal O2 to obtain the value of the image signal for the entire frame period. The coefficient for this conversion is the addition ratio k. The addition coefficient k can be calculated using the following equation 5.

[0068] k = 2^(EV value of period 1) ÷ 2^(EV value of period 2) = 2^(EV value of period 1 - EV value of period 2) ... (Equation 5) From Equation 5, the addition ratio k1 of frame 1 in this embodiment is: k1 = 2^(3-2) = 2^1 = 2 This is how it is found.

[0069] Similarly, consider frame 2. The second exposure time 404-2 of frame 2 is 1 / 8 the length of the second period 402. This corresponds to an exposure time of 1 / 240 in Figure 6. If the analog gain at this time is ×1, the EV value of the second period 402-2 is 6. The EV value of the first period 401-2 is 3, similar to 401-1. This is because the exposure conditions for the first period 401 are set uniformly under the same conditions. The addition ratio k2 of frame 2 at this time is k² = 2^(3-6) = 2^(-3) = 1 / 8 This is what is required.

[0070] In Figure 3, the image signal O3, which is the combined output of image signals O2 and O1 calculated in the image summing unit 303 based on Equation 1, is transmitted to the tone expansion unit 304. The tone expansion unit 304 determines the EV value shown in Figure 6 from the exposure time 112 and analog gain value 113 of the corresponding pixel block. If the EV value at that time is E, the pixel value after tone expansion can be determined by Equation 4.

[0071] For example, the exposure conditions in the second period of frame 1 shown in Figure 4 were as follows: the exposure time, when the length of the second period is converted to the entire frame 1, is equivalent to 1 / 60 second, and the analog gain was ×4. Since the EV value at this time is 2, if the composite output after summing of frame 1 is image signal O31, and the pixel values ​​after grayscale expansion are image signal O41, then, O41 = O31 × 2^2 = 4 × O31 This is the result.

[0072] Similarly, the exposure conditions for the second period of frame 2 shown in Figure 4 were as follows: As mentioned above, the exposure time, when the length of the second period is converted to the entire frame 2, is equivalent to 1 / 240 seconds, and the analog gain was ×1. Since the EV value at this time is 6, if the composite output after the addition process of frame 2 is O32 and the data value of the pixels after grayscale expansion is O42, O42 = O32 × 2^6 = 64 × O32 This is the result.

[0073] At this time, if the original exposure image of image signal O32 had a bit width of 10 bits, this calculation results in image signal O42 having a bit width of 16 bits. In this embodiment, as shown in the example in Figure 6, the EV value can take up to 13. In that case, when the exposure compensation unit 105 performs grayscale expansion, the bit width of the data increases. In this embodiment, the grayscale-expanded image 123 may take up to 23 bits. Subsequently, as described above, the grayscale conversion unit 106 generates, for example, a 12-bit grayscale-converted image 124 from the 23-bit grayscale-expanded image 123 by gamma conversion.

[0074] In this embodiment, the data input as the data of the image signal O1 is held in the line buffer 301 while being thinned out laterally in units of a Bayer array, as shown in Figure 5. However, it is also possible to output the image signal O1 with thinning when outputting it from the imaging sensor 102. In this case, the image signal O1 output from the imaging sensor 102 has a smaller amount of data when averaged across multiple pixels than the amount obtained when the image was captured by the imaging sensor 102. The subsequent line buffer 301 only needs to hold the smaller amount of data that is output without thinning. The control to thin out the data before output may be performed by providing the imaging sensor 102 with a processing circuit (not shown) used for imaging control.

[0075] Even when the image sensor 102 outputs a reduced image signal, the data output from the image sensor 102 is stored in the subsequent line buffer 301, as described above. The same processing based on equations (1) and (2) can be performed on the image signal stored in the line buffer 301. Therefore, by reducing the amount of data output from the image sensor 102, the storage capacity of the subsequent line buffer 301 can be reduced.

[0076] In this case, the amount of data from the first imaging operation from the imaging sensor 102 is reduced, while the amount of data from the second imaging operation is not reduced. Since the bit width of the data output by the imaging sensor 102 does not change between the first and second imaging operations, the amount of image signal data output by the imaging sensor 102 based on the first imaging operation can be reduced to less than the amount of image signal data based on the second imaging operation.

[0077] According to this embodiment, image processing is performed to appropriately combine the image signal after downsampling the image signal from the first period with the image signal from the second period within one frame period. This makes it possible to perform wide dynamic range (WDR) imaging that takes advantage of the characteristics of region-specific exposure while suppressing flicker, even if a light source with a blinking period, such as an LED, is present in the imaging range. For example, as shown in frame 2 of Figure 4, even if the LED light emission period cannot be captured in the second period where region-specific exposure is performed, the LED light emission is captured in the first period. Therefore, even in the image of frame 2, which is a composite of the outputs of the second and first periods, the LED light emission is captured in the even-numbered pixels, making it possible to prevent the image from appearing as if the LED light source is off. Furthermore, by appropriately downsampling and retaining the data from the first period, it is possible to reduce the capacity of the memory unit required to hold the image signal from the first period, which contributes to miniaturization and cost reduction of the imaging device.

[0078] [Embodiment 2] In Embodiment 1, for the output image signal, even-numbered pairs of pixels in the row-arranged Bayer array were composite image signals of the second and first periods, while odd-numbered pairs of pixels in the Bayer array were image signals generated only from the output of the second period. In this case, under conditions where flicker occurs, the data from the first period (401) is omitted in the odd-numbered pairs of pixels, so the presence of a light source causing the flicker is not represented, and the smoothness of the image display in the flickered area is lost. In this embodiment, an example of determining whether or not there is a light source that causes flicker in the image target and adaptively changing the synthesis method for odd-numbered pairs of pixels will be explained with reference to Figures 7 to 9. Except for the points described below, this embodiment is the same as Embodiment 1 described above.

[0079] Figure 7 is a block diagram showing an example of the configuration of the exposure compensation unit 105 in this embodiment. The exposure compensation unit 105 includes a line buffer 301, an addition ratio calculation unit 706, a flicker determination unit 701, an image addition unit 702, and a grayscale extension unit 304. Components having the same functions as in Embodiment 1 are given the same reference numerals as in Embodiment 1.

[0080] The operation of each component of the exposure compensation unit 105 shown in Figure 7 will be explained with reference to the example shown in Figure 4. Note that the operation of the exposure compensation unit 105 that is the same as in Embodiment 1 will be omitted from the explanation; only the differences from Embodiment 1 will be explained here.

[0081] In Figure 7, the exposure image 122 of the first period 401-1 of frame 1 in Figure 4 is input to the exposure compensation unit 105 as an example, and the following explanation will be provided. In this embodiment as well, the image signal of the exposure image 122 of the first period of a certain frame at a certain pixel position will be called the image signal O1. The image signal O1 will include four pixels in a Bayer array: R pixels, Gr pixels, Gb pixels, and B pixels, as in Embodiment 1. The following calculations relating to the image signal O1 will be applied independently to each of the four pixels: R pixels, Gr pixels, Gb pixels, and B pixels, for each pixel component.

[0082] The image signal O1 is input to the exposure correction unit 105 as the exposed image 122 and held in the line buffer 301. When held in the line buffer 301, it is assumed that the images are thinned out laterally in units of a Bayer array, as shown in Figure 5. After the elapsed time of the second period, the exposed image 122 of the second period 402-1 of frame 1 is input. Here, the image signal of the exposed image 122 of the second period 402 of the same frame as the image signal O1 will be called the image signal O2. Furthermore, the composite output data generated by the image summing unit 702 based on the image signals O1 and O2 will be called the image signal O3.

[0083] Image signals O2 and O3 also include four pixels, R, Gr, Gb, and B, similar to image signal O1. In the following description, the calculations relating to image signal O2 are also applied independently to each of the four pixels, R, Gr, Gb, and B, for each pixel component, as described in Embodiment 1.

[0084] When the image signal O2 is supplied to the exposure correction unit 105 as the exposed image 122, if the image signal O1 at the same pixel position is held in the line buffer 301, the flicker determination unit 701 reads the image signal O1 from the line buffer 301. This image signal O1 is read as the delayed image 306. Then, based on the period ratio α 704 and addition ratio k 305 calculated by the addition ratio calculation unit 706, it is determined whether flicker is occurring at that pixel. The determination result F 703 regarding the occurrence of flicker is transmitted to the image addition unit 702. The operation of the flicker determination unit 701 will be described later.

[0085] The presence or absence of flicker is determined at even pixels. Below, the exposure image 122 of the first period of even pixels is used as the image signal O1 2n The image 122 of the second exposure period for even-numbered pixels is converted into image signal O2 2n The determination of whether or not flicker is present will be explained using this notation. In the image addition unit 702, the image signal O1 is processed for the time necessary to calculate the flicker determination result F 703 in the flicker determination unit 701. 2n306 data and image signal O2 supplied as exposed image 122 2n The data is delayed. Furthermore, the addition ratio k is 305, and the image signal O1 2n and image signal O1 2n The timing is adjusted. After adjusting the timing, the flicker detection result F 703 of the flicker detection unit 701 is used to synthesize the image signal O3. 2n The image summing unit 702 uses the flicker determination result F 703 for even pixels to perform a synthesis process on odd pixels that are consecutive to the even pixels, and outputs the image signal O3 as the synthesis output. 2n+1 This calculates the result. The operation of the image addition unit 702 will be described later.

[0086] An example of the process for calculating the flicker determination result F 703 in the flicker determination unit 701 is explained using the flowchart shown in Figure 8. When the process starts, the period ratio α 704, which is the ratio of the second period to the first period, is received from the addition ratio calculation unit 706 (S801). The ratio of the second period to the first period is calculated in the addition ratio calculation unit 706 based on the register setting value 143 input from register 142. The period ratio α can be obtained by the following equation 6.

[0087] α = (length of the second period) ÷ (length of the first period) ... (Equation 6) In this embodiment, as shown in Figure 4, the length of the second period is 22.2 ms and the length of the first period is 11.1 ms, so the period ratio α = 2. The period ratio α may be calculated at the beginning of each frame, or it may be calculated and set at the beginning of video capture. The period ratio α is input to the flicker detection unit 701 (S801). As for the addition ratio k 305, the value is determined by the addition ratio calculation unit 706 based on equation 5 of Embodiment 1.

[0088] Using the period ratio α and addition ratio k obtained as described above, the image signal O2 2n and image signal O1 2n Correct the values ​​of each. Image signal O2 2n The corrected second correction signal and image signal O1 2nUsing the first correction signal which has been corrected, a flicker evaluation value D is calculated for each pixel (S802). The flicker evaluation value D is calculated using the image signal O2 2n and image signal O1 2n Furthermore, using the addition ratio k calculated by the addition ratio calculation unit 706 and the period ratio α, it can be calculated using the following equation 7.

[0089] D=O2 2n ÷k-α×O1 2n ...(Formula 7) As shown in Equation 7, in this embodiment, the image signal O2 2n The second corrected signal obtained by correcting with the addition ratio k and the image signal O1 2n The difference between the first corrected signal, which is corrected by the period ratio α, and the flicker evaluation value D is calculated. Based on the flicker evaluation value D, it is then determined whether flicker is occurring for each pixel (S803). Here, the determination of whether or not flicker is occurring is made based on a threshold β. If the absolute value of the flicker evaluation value D obtained in S802 is smaller than the threshold β, it is determined that no flicker is occurring ("Yes" in S803), and the process proceeds to S804. Otherwise, it is determined that flicker is occurring ("No" in S803), and the process proceeds to S805.

[0090] In this embodiment, the threshold β is assumed to be a predetermined value set within the image summing unit 702. However, the method of setting the threshold β is not limited to this; for example, it can be set in the register 142 from an external controller 10, allowing its value to be changed as needed. The threshold β is used to determine whether or not flicker occurs by comparing the flicker evaluation value D with the threshold β and checking whether the flicker evaluation value D is less than or equal to the threshold β. Therefore, the threshold β can be changed to an appropriate value as needed.

[0091] When it is determined that no flicker has occurred, the value F=1 is set as the value indicating that no flicker has occurred, where F is the flicker detection result (S804). This value of F is transmitted to the image addition unit 702 as the flicker detection result 703. On the other hand, when it is determined that flicker has occurred, the value of F indicating the flicker detection result is set to F=0 as the value indicating that flicker has occurred (S805). In this case as well, the value of F is similarly transmitted to the image addition unit 702 as the flicker detection result 703.

[0092] Next, it is determined whether the calculation process for the flicker evaluation value D has been completed for all pixels in one frame (S806). If the calculation process for all pixels in one frame has not been completed (No in S806), the process from S802 onwards is repeated. If the calculation process has been completed (Yes in S806), the process is terminated.

[0093] Figure 9 is a flowchart showing an example of image addition processing in the image addition unit 702 according to the determination of whether or not flicker occurs. When processing starts, the conversion ratio G used when adding the image signals is calculated (S901). The conversion ratio G is calculated when it is determined that there is no flicker, and the image signal O2 of even pixels obtained by region-specific exposure is calculated. 2n This is a coefficient for converting the value to the pixel value of the frame in question. The conversion ratio G can be calculated using Equation 3, similar to Embodiment 1. In this embodiment, the conversion ratio G is G = 1.5, similar to Embodiment 1. The calculation of the conversion ratio G may be performed at the beginning of each frame.

[0094] Next, the flicker detection result F for each pixel is referenced to determine how to calculate the image signal O3 to be output (S902). If F=1 and no flicker has occurred (Yes in S902), proceed to S903. On the other hand, if F=0 and flicker has occurred in that pixel (No in S902), proceed to S904.

[0095] In S903, the image signal O3 should be output as an even pixel if no flicker occurs. 2nHowever, this is calculated using equation 8 below. Then, proceed to S905.

[0096] O3 2n =G×O2 2n ...(Formula 8) In S904, the image signal O3 should be output as an even-numbered pixel if flicker occurs. 2n However, this is calculated using Equation 1 shown in Embodiment 1. After processing S903 or S904 is performed, the output values ​​of the odd-numbered pixels following the even-numbered pixels are combined.

[0097] In this embodiment, as shown in Figure 7, the exposed image 122 and the delayed image 306 are supplied to the image summing unit in a pipeline manner, row by row. In one example of pipeline processing, when the image summing unit performs synthesis processing on the previous pixel, the exposed image 122 of the next pixel may be supplied to the line buffer 301. Therefore, synthesis processing for the odd-numbered pixel 2n+1 following the even-numbered pixel 2n on which flicker detection was performed can be carried out smoothly. As described above, the flicker detection result F used for synthesizing the output value of the odd-numbered pixel 2n+1 is the result of the detection for the immediately preceding even-numbered pixel 2n. If no flicker occurred, the image signal O3 should be output as an odd-numbered pixel. 2n+1 This is calculated using Equation 2 (S905). Then proceed to S907.

[0098] If flicker occurs, the image signal O1 of the first period of the immediately preceding even-numbered pixel 2n Using this, the image signal O3 should be output as an odd pixel according to the following equation 9. 2n+1 This is calculated (S906). Then proceed to S907.

[0099] O3 2n+1 =O2 2n+1 +k×O1 2n ...(Formula 9) In S907, it is determined whether the calculation process for the image signal O3 has been completed for one frame's worth of pixels. If it has not been completed (S907 is "No"), the process from S901 onwards is repeated. If the calculation process has been completed (S907 is "Yes"), this process is completed.

[0100] In this embodiment, when flicker occurred, the image signal O1 of the preceding even pixel during the first period was used to synthesize the output values ​​of odd-numbered pixels. As another example, the average value of the even pixels before and after the odd-numbered pixel to be synthesized may be used. In that case, it is necessary to read the even pixels following the odd-numbered pixel to calculate the odd-numbered pixel, and the overall image signal output timing will be shifted by one pixel compared to the case where the value of the preceding even pixel is used. Also, at the right edge of the image, edge processing can be performed using only the value of the preceding pixel. It goes without saying that this pixel interpolation method is not limited to these methods, and various interpolation methods can be used.

[0101] Furthermore, the coordinates on the image sensor unit 103 of the pixels where flicker occurs can be identified. Therefore, there is a high probability that flicker will occur in pixels at coordinates surrounding the image sensor unit 103 where flicker occurred when capturing the next frame. Thus, the image sensor unit 103 may be driven to perform the first imaging operation only on pixels around coordinates where flicker may occur, and only perform the second imaging operation on pixels in locations where the probability of flicker occurrence is low. By driving in this way, the image sensor 102 outputs image signals O1 only from coordinates where flicker may occur, so the amount of data in image signals 01 output by the image sensor 102 can be reduced to the amount of data in image signals O2.

[0102] According to this embodiment, by determining whether or not there is a light source that generates flicker in the image target and adaptively changing the composite ratio of the second period and the first period, it becomes possible to obtain an image that is more appropriate as a WDR image. Furthermore, when flicker is present, the composite image can be generated by appropriately using the image signal O1 of even pixels as an interpolation value. Therefore, by appropriately downsampling and retaining the image signal of the first period, the required memory capacity can be reduced, and the image in the part corresponding to the flicker light source can be displayed smoothly. According to this embodiment, it becomes possible to miniaturize the imaging device and reduce costs.

[0103] [Embodiment 3] In Embodiment 1, the data input as the image signal O1 was stored in the line buffer 301, and as shown in Figure 5, it was described in an example where the data was thinned out laterally in units of a Bayer array. For the output image signal, even pixels were a composite image signal of the second and first periods, while odd pixels were an image signal generated only from the output of the second period. In this case, under conditions where flicker occurs, the image signal of the first period is thinned out in the odd pixels, so the presence of the light source causing the flicker is not represented, and the smoothness of the image display is sometimes lost.

[0104] In this embodiment, by changing the method of decimation of the image signal when it is held in the line buffer 1101, it is possible to always generate a composite signal of the image signal captured in the first period and the image signal captured in the second period of one frame period. This embodiment makes it possible to reduce the amount of memory held while maintaining the smoothness of the image display in the parts where flicker occurs. This embodiment will be described with reference to Figures 10 and 11. Note that the parts that differ mainly from Embodiment 1 will be described below, and the parts that are the same as Embodiment 1 will not be described. Also, parts that can be explained using the same figures as Embodiment 1 will be explained using the figures of Embodiment 1.

[0105] Figure 11 is a block diagram showing an example of the configuration of the exposure compensation unit 105 in this embodiment. In Figure 11, the exposure compensation unit 105 is first input with the exposure image 122 of the first period 401 of frame 1 in Figure 4. This embodiment will be described assuming this configuration is used as an example. In this embodiment, the image signal of the exposure image 122 is input to the exposure compensation unit 105 in a Bayer array consisting of four pixels: R pixels, Gr pixels, Gb pixels, and B pixels, as shown in Figure 10. Here, the image signal of the exposure image 122 of the first period 401 of a frame at a certain pixel position will be called the image signal O1.

[0106] As mentioned above, the image signal O1 contains four pixels: R, Gr, Gb, and B pixels. In the following explanation, operations on the image signal O1 are applied independently to each of the four pixels, R, Gr, Gb, and B pixels, for each pixel component. When the data input as data corresponding to the image signal O1 is held in the line buffer 1101, the data of one color from the RGB pixels of the Bayer array is periodically decimated, and the remaining data is held in the line buffer 301. In the example shown in Figure 10, only the image signals of the odd-numbered Gb pixels are held while being decimated every other pixel.

[0107] After frame 1 begins and the first period 401-1 has elapsed, the data of image signal O1 is first held in line buffer 1101. After the second period 402-1 has elapsed, the exposed image 122 of the second period of frame 1 is input. The image signal of the exposed image 122 of the second period 402-1 of the same frame 1 at this time will be called image signal O2. The composite output data generated by the image summing unit 303 based on image signals O1 and O2 will be called image signal O3.

[0108] Image signals O2 and O3, like image signal O1, also contain four pixels: R, Gr, Gb, and B. In the following explanation, the operations on image signal O2 are applied independently to each of the four pixels (R, Gr, Gb, and B) according to their respective components. This is the same as in the case of image signal O1.

[0109] When the image signal O2 is supplied to the exposure correction unit 105 as the exposed image 122, the image summing unit 303 reads the image signal O1 from the line buffer 1101 as a delayed image 1103. Here, the line buffer 1101 is assumed to have a data copy unit 1102 at its output stage. The data copy unit 1102 has the function of copying the data of the odd-numbered Gr pixels of the same Bayer array set to the image position of the downsampled pixel Gb when the output pixel is an odd-numbered pixel. Therefore, as shown in Figure 10, the image signal sent to the image summing unit 303 as a delayed image 1103 will have the data of the same odd-numbered Gr pixels at the image position of the Gb pixels in the odd-numbered pixels of the Bayer array set. The data of the Gr1 pixel is copied in place of the data of the downsampled Gb1 pixel as indicated by the arrow, and the data of the Gr3 pixel is copied in place of the downsampled Gb3 pixel as indicated by the arrow.

[0110] Then, the image summing unit 303 calculates the image signal O3 as a composite output, which is a composite image signal of image signal O2 and image signal O1, based on the summing ratio k(305) calculated by the summing ratio calculation unit 302, using the following equation 10. In this embodiment, equation 10 is used for both odd-numbered and even-numbered pixels to determine the image signal O3. Equation 10 is a modification of equation 1 in Embodiment 1 so that it can also be applied to odd-numbered pixels.

[0111] O3=O2+k×O1...(Equation 10) Other operations are the same as in Embodiment 1, so their explanation will be omitted. In this embodiment, the interpolation method for odd-numbered Gb pixels in the data copy unit 1102 is to copy the data of the Gr pixels of the same odd-numbered pixels, but the interpolation method is not limited to the method disclosed herein. In other examples, when creating the data for the position of the Gb1 pixel, a method can be adopted in which the data of nearby Gb0, Gr0, Gr1, and Gb2 pixels is referenced for interpolation.

[0112] In this embodiment, the data input as the image signal O1 is held in the line buffer 1101 while only the odd-numbered Gb pixels are thinned out, as shown in Figure 10. However, it is also possible to output the image signal O1 from the imaging sensor 102 in a thinned-out state within the imaging sensor 102. The thinning process can be performed using the processing circuit (not shown) used for imaging control in the imaging sensor 102. In that case as well, the data that has not been thinned out is held in the line buffer 1101 as described above, and subsequent processing can be carried out, so the same effect as in this embodiment can be obtained.

[0113] According to this embodiment, it is possible to simultaneously reduce the required storage capacity by appropriately downsampling and retaining data from the first period, and to display flickered images smoothly. This contributes to miniaturization and cost reduction of the imaging device.

[0114] [Embodiment 4] An example in which only a predetermined number of the higher bits of the image signal at each pixel are stored in the line buffer 1201 will be explained with reference to Figures 12 and 13. Except for the points described below, this is the same as Embodiments 1 and 3 described above, and parts that can be explained using the same figures as Embodiment 1 will be explained using the figures of Embodiment 1.

[0115] Figure 12 is a block diagram showing an example of the configuration of the exposure compensation unit 105 in this embodiment. In Figure 12, the exposure compensation unit 105 first receives the exposure image 122 of the first period 401 of frame 1 in Figure 4. In this embodiment, the image signal of the exposure image 122 is assumed to be 12-bit wide data, as shown in Figure 13. In other words, in this example, the data width of each of the four pixels, R pixel, Gr pixel, Gb pixel, and B pixel, is assumed to be 12 bits.

[0116] Here, we will refer to the image signal of the exposed image 122 for the first period 401 of a frame at a certain pixel position as image signal O1. As mentioned above, image signal O1 includes four pixels: R pixels, Gr pixels, Gb pixels, and B pixels. In the following explanation, operations on image signal O1 are applied independently to each of the four pixels: R pixels, Gr pixels, Gb pixels, and B pixels, for each pixel component.

[0117] This embodiment will be explained using the example shown in Figure 13. The data input as the image signal O1 is 12 bits, represented by bits 11 to 0, from the most significant bit to the least significant bit, as shown in the exposed image 122. When this data is held in the line buffer 1201, only the 8 bits from the most significant bits 11 to 4 of the image signal are held, as shown in Figure 13. Only the most significant 8 bits of each of the four pixels, R pixel, Gr pixel, Gb pixel, and B pixel, are held in the line buffer 1201.

[0118] After frame 1 begins and the first period 401-1 has elapsed, as described above, only the upper 8 bits of the image signal O1 data are initially held in the line buffer 1201. After the second period 402-1 has elapsed, the exposed image 122 of frame 1 for the second period 402-1 (corresponding to the second exposure time 404-1) is input. The image signal of the exposed image 122 of the same frame 1 for the second period 402-1 at this time will be called image signal O2. Image signal O2 is 12 bits wide and is input to the image conversion unit 303. Here, the composite output data generated by the image summing unit 303 based on image signal O1 and image signal O2 will be called image signal O3.

[0119] Image signals O2 and O3, like image signal O1, also contain four pixels: R pixels, Gr pixels, Gb pixels, and B pixels. In the following explanation, the operations on image signal O2 are applied independently to each of the four pixels (R pixels, Gr pixels, Gb pixels, and B pixels) according to their respective pixel components, just as in the case of image signal O1.

[0120] When the image signal O2 is supplied to the exposure correction unit 105 as the exposed image 122, the image summing unit 303 reads the data of the image signal O1 from the line buffer 1201 as a delayed image 1202. At this time, the image read from the line buffer as the delayed image 1202 is 8-bit data. When this delayed image 1202 is input to the image summing unit 303, the lower 4 bits are supplemented with "1000" (also written as "^b1000") in binary notation, as shown in Figure 13. This lower 4-bit value can be achieved by fixing bit 3 to H and bits 2-0 to L on the lower 4-bit input terminal of the input signal line of the image summing unit 303.

[0121] Then, the image addition unit 303 calculates the image signal O3, which is a composite image signal of image signal O2 and image signal O1, based on the addition ratio k 305 calculated by the addition ratio calculation unit 302, using equation 10 shown in Embodiment 3. Similar to Embodiment 3, in this embodiment as well, equation 10 is used for determining the image signal O3 for both odd and even pixels. The subsequent operations are the same as in Embodiments 1 and 3 described above, so the explanation will be omitted.

[0122] In this embodiment, the bit width of the exposed image 122 is set to 12 bits, and the upper 8 bits are held in the line buffer 1201. This data bit width and the bit width held in the line buffer are merely examples, and the appropriate bit width can be selected according to the system configuration and data requirements.

[0123] In this embodiment, "1000" in binary notation was used as the value to fill in the lower bits of the delayed image 1202, but it is also possible to use "0111" in some cases. These two values ​​are intermediate values ​​between 0 and 15, which are represented by 4 bits. By using an intermediate value as the lower bits, the average expected error between the image signal O3 and the image signal O3 obtained if the original lower bits were present can be minimized during the subsequent calculation of the image signal O3.

[0124] Furthermore, in this embodiment, when the data input as the image signal O1 is held in the line buffer 1201, only the pixel values ​​of the higher bits are held, as shown in Figure 13. When the imaging sensor 102 outputs the image signal O1, it is also possible to output only a predetermined number of higher bits of the pixels. Specifically, it is possible to output the data in a state where only the upper 8 bits are included. In that case as well, the effects of this embodiment can be obtained by holding the data of the higher bits in the line buffer 1201 as described above and performing subsequent processing.

[0125] According to this embodiment, by appropriately downsampling and retaining the image signal data for the first period, it becomes possible to reduce the required memory capacity while suppressing flicker and performing wide dynamic range (WDR) imaging that takes advantage of the characteristics of region-specific exposure. This contributes to miniaturization and cost reduction of the imaging device.

[0126] [Embodiment 5] In this embodiment, the data held in the line buffer 1403 is the difference data between adjacent pixels located in the same row, thereby reducing the amount of memory held while further minimizing the error in the image before and after decimation. This example will be explained with reference to Figures 14 to 17. Except for the points described below, this embodiment is the same as Embodiments 1, 3, and 4 described above, and parts that can be explained using the same figures as Embodiments 1, 3, and 4 will be explained using the figures of each respective embodiment.

[0127] Figure 14 is a block diagram showing an example of the configuration of the exposure compensation unit 105 in this embodiment. In Figure 14, the exposure compensation unit 105 first receives the exposure image 122 of the first period 401-1 of frame 1 in Figure 4. In this embodiment, the image signal of the exposure image 122 is assumed to be 12-bit wide data, similar to that shown in Figure 13 of Embodiment 4. In the example where the pixels are arranged in a Bayer array, this means that each of the four pixels, R pixel, Gr pixel, Gb pixel, and B pixel, is 12-bit wide data. Here, the image signal of the exposure image 122 of the first period 401-1 of a frame at a certain pixel position will be called the image signal O1. As mentioned above, the image signal O1 includes four pixels: R pixel, Gr pixel, Gb pixel, and B pixel. In the following description, calculations relating to the image signal O1 are applied independently to each of the four pixels, R pixel, Gr pixel, Gb pixel, and B pixel, for each pixel component.

[0128] The data input as image signal O1 is stored in the line buffer 1403 inside the data storage unit 1401 in the format shown in Figure 15. This means that each of the four pixels, R pixel, Gr pixel, Gb pixel, and B pixel, is stored in the format shown in Figure 15.

[0129] When the data of the exposed image 122 is input to the data holding unit 1401 as the image signal O1, the data is processed by the data holding calculation unit 1402 and input to the line buffer 1403. The operation of the data holding calculation unit 1402 will be described later.

[0130] Figure 16 is a flowchart showing an example of the process in the data retention calculation unit 1402 for calculating data to be stored in the line buffer 1403 from the exposed image 122 of the image signal O1. This operation is performed in parallel for each of the four pixels, R, Gr, Gb, and B pixels, which are input in Bayer format. In S1601, it is determined whether the input pixel is the first pixel (leftmost data) in the row of the image. If it is the first pixel of the row (S1601-Yes), the process proceeds to S1608 without processing the pixel value from the pixel. In S1608, the pixel value is written directly to the beginning of the line buffer. Then the process proceeds to S1609.

[0131] If it is not the first pixel of the row (S1601-No), proceed to S1602. In S1602, calculate the difference value δ with the previous pixel value of the same color component, and proceed to S1603. In S1603, determine whether the magnitude of the difference value δ obtained in S1602 is within a predetermined bit width range. For example, as shown in Figure 15, assume that the difference data held in the line buffer 1403 is represented by 7 bits. In this case, if the difference data is represented using two's complement, the range of the number represented by 7 bits is -64 to +63 (7F to 3F). Therefore, if -64 ≤ δ ≤ +63 (S1603-Yes), proceed to S1604. If δ < -64 or 63 < δ (S1603-No), proceed to S1605.

[0132] In S1604 and S1605, a flag FL is set to indicate whether the data held in the line buffer 1403 from the second pixel onward in a row is difference data or update data. Here, difference data is the difference value δ between the previous pixel value of the same color component and the current data, and is within a predetermined bit width. Update data is data where the difference value δ exceeds the predetermined bit width. The holding of update data will be described later. In S1604, FL=0 is set to indicate that the data to be stored in the line buffer 1403 is difference data. In S1604, FL=1 is set to indicate that the data to be stored in the line buffer 1403 is update data. After that, S1604 proceeds to S1606, and S1605 proceeds to S1607.

[0133] In S1606, since the difference value δ was within the predetermined width range, the difference value obtained in S1602 and the flag FL=0 are stored in the line buffer 1403 in the format shown in Figure 15. Then the process proceeds to S1609. In S1607, since the difference value δ was outside the predetermined width range, instead of the difference value δ, the upper 7 bits of the input pixel value and the flag FL=1 are stored in the line buffer 1403 in the format shown in Figure 15. Then the process proceeds to S1609. In S1609, it is checked whether the processing of one frame has been completed. If it has not been completed (S1609-No), the process returns to S1601 and the operation is repeated. If it has been completed (S1609-Yes), this process is terminated.

[0134] Figure 17 is a flowchart showing an example of the process in the delayed image calculation unit 1404 for calculating a delayed image 1405 by reading data stored in the line buffer 1403. This operation is performed in parallel for each of the four pixels, R, Gr, Gb, and B pixels, which are input in Bayer format. In S1701, it is determined whether the pixel to be read is the first (leftmost data) of a row in the image. If it is the first pixel of the row (S1701-Yes), the process proceeds to S1706. In S1706, the first pixel of the row is read as 12 bits of data, and then the process proceeds to S1707. In S1707, the read 12 bits of data are output as the first pixel value of the delayed image 1405 and are held for use in the calculation of the next pixel.

[0135] If it is not the first pixel of the row (S1701-No), proceed to S1702. In S1702, since it is not the first pixel of the row, the pixel value is read as 8-bit data, and proceed to S1703. In S1703, it is determined whether the value of flag FL is 0 or 1. If FL=0 (S1703-Yes), the read data is 7-bit difference data, so proceed to S1704. In S1704, the data to be output is generated by adding the difference data with the data of the previous pixel held, and then proceed to S1707. On the other hand, if FL=1 (S1703-No), the read data is 7-bit update data, so proceed to S1705.

[0136] In S1705, the read 7-bit update data is set as the upper 7 bits of the 12-bit delayed image signal. At the same time, the lower 5 bits are supplemented with "10000" in binary notation. The delayed image signal to be output is prepared in this way, and the process proceeds to S1707. As mentioned earlier, in S1707, the pixel values ​​prepared as data for the delayed image 1405 to be output are output and retained for use in the calculation of the next pixel. After that, the process proceeds to S1708. In S1708, it is checked whether the processing of one frame has been completed. If it has not been completed (S1708-No), the process returns to S1701 and the operation is repeated. If it has been completed (S1708-Yes), this process is terminated. The output delayed image 1405 is input to the image summing unit 303, and the composite output O3 is generated in the manner described in Embodiment 4. The following operations are the same as in Embodiment 4, so the explanation is omitted.

[0137] In this embodiment, the bit width of the exposed image 122 is set to 12 bits, and the data from the second pixel onward is stored in the line buffer 1403 as differential data or update data, consisting of 7 bits + 1 bit flag data. This data bit width and the bit width stored in the line buffer are merely examples, and the appropriate bit width can be selected depending on the system configuration.

[0138] In this embodiment, "10000" in binary notation was used as the value to fill in the lower bits when using update data, but it is also possible to use "01111" in some cases. These two values ​​are the midpoint of the values ​​from 0 to 31 represented by 5 bits, and they can minimize the average expected error of the O3 value obtained when the original lower bits were present during the subsequent O3 calculation.

[0139] Furthermore, in this embodiment, the difference value δ between the current pixel value and the previous pixel value of the same color component was calculated as difference data, and this value of δ was used as is. However, it is also possible to apply an offset to the difference value in some cases. For example, if the data to be held as the difference value is δ / 4 (rounded down to the nearest whole number), and the difference data is expressed in two's complement, then -64 ≤ δ / 4 ≤ +63, or -256 ≤ δ ≤ +252, can be used. In this case, if δ does not fall within this range, the updated data will be stored. Also, if δ is within this range, the calculation in the delayed image calculation unit will be performed taking the offset (in this case, δ divided by 4, i.e., the difference value shifted by 2 bits) into consideration.

[0140] Furthermore, although the delayed image calculation unit 1404 is located inside the data holding unit 1401 in this embodiment, depending on the configuration, this function can also be provided in the image addition unit 303. In addition, when outputting the image signal O1 from the imaging sensor 102, the difference of the image signal O1 is output, and the image signal is held in the line buffer 301 and the same processing is performed, and the same effect as in this embodiment can be obtained.

[0141] According to this embodiment, by storing the data held in the line buffer 1403 based on the difference data with the previous pixel, it becomes possible to reduce the amount of memory held while further reducing the error in the image before and after downsampling. As a result, it becomes possible to perform wide dynamic range (WDR) imaging that takes advantage of the characteristics of region-specific exposure while suppressing flicker, and it also contributes to miniaturization and cost reduction of the imaging device.

[0142] [Application of the apparatus according to the embodiment to equipment] The following describes a device 2000 comprising a semiconductor device 2100, which includes a package 2020 on which a semiconductor chip 2110, including a semiconductor integrated circuit, is mounted, as shown in Figure 18. The semiconductor chip 2110 is housed in the package 2020 and mounted on the device 2000. In the configuration shown in Figure 18, the semiconductor chip 2110 includes an imaging device according to the above-described embodiment. The semiconductor device 2100 may include a package 2020 which includes a base 2010 on which the semiconductor chip 2110 is fixed, and a light-transmitting member 2030 such as glass facing the semiconductor chip 2110. The package 2020 may include connecting members such as wires and bumps that connect inner leads provided on the base 2010 to terminals such as pad electrodes provided on the semiconductor chip 2110.

[0143] The device 2000 may include at least one of the following: an optical device 2040, a control device 2050, a processing device 2060, a display device 2070, a storage device 2080, and a mechanical device 2090. The optical device 2040 is a lens, shutter, mirror, etc. The control device 2050 controls the semiconductor chip 2110. The control device 2050 may be composed of a semiconductor device such as an ASIC.

[0144] The processing unit 2060 processes the output signal from the imaging device included in the semiconductor chip 2110. The processing unit 2060 is a semiconductor device such as a CPU or ASIC for configuring the AFE analog front end or DFE digital front end. The processing unit 2060 may generate an image based on the event signal. The display device 2070 is an EL display device or liquid crystal display device that displays the information image obtained from the semiconductor chip 2110. The storage device 2080 is a magnetic device or semiconductor device that stores the information image obtained from the semiconductor chip 2110. The storage device 2080 is a volatile memory such as SRAM or DRAM, or a non-volatile memory such as flash memory or a hard disk drive.

[0145] The mechanical device 2090 may have moving parts or propulsion parts such as motors or engines. The device 2000 can display signals output from the semiconductor chip 2110 on the display device 2070 or transmit them externally using a communication device (not shown) provided by the device 2000. For this purpose, the device 2000 may further include a memory device 2080 and a processing device 2060, separate from the memory circuits and arithmetic circuits of the semiconductor chip 2110. The mechanical device 2090 may be controlled based on signals output from the semiconductor chip 2110.

[0146] Furthermore, the device 2000 may be an information terminal with imaging capabilities, or it may be an electronic device such as a smartphone or a wearable device. The device 2000 may also be a camera, and the camera may include interchangeable lens cameras, compact cameras, video cameras, surveillance cameras, etc. In the camera, the mechanical device 2090 can drive components of the optical device 2040 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 2090 in the camera can move the optical device 2040 for vibration damping.

[0147] Furthermore, the device 2000 may be a transport device such as a vehicle, ship, or aircraft. The mechanical device 2090 in the transport device may be used as a mobile device. The device 2000 as a transport device is suitable for transporting semiconductor chips 2110 or for assisting and / or automating driving operations through imaging functions. The processing device 2060 for assisting and / or automating driving operations can perform processing to operate the mechanical device 2090 as a mobile device based on information obtained from the semiconductor chip 2110. Alternatively, the device 2000 may be a medical device such as an endoscope, a measuring instrument such as a distance sensor, an analytical instrument such as an electron microscope, an office machine such as a copier, or an industrial machine such as a robot.

[0148] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0149] (Other embodiments) The disclosure of the present invention includes the following photoelectric converters, photoelectric converter systems, and devices to which photoelectric converters are applied. (Item 1) A photoelectric conversion system comprising: a photoelectric conversion device having multiple pixel blocks arranged in a matrix, which perform a first imaging operation including an exposure time based on a preset first exposure condition and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block during one frame period; and a storage unit, The photoelectric conversion system is characterized in that the storage unit holds a first image signal whose data amount is smaller than the data amount of the image signal acquired by the photoelectric conversion device by the first imaging operation. (Item 2) The photoelectric conversion system according to item 1, characterized in that it has a signal generation unit that generates a third image signal corresponding to the one frame period based on the first image signal held in the storage unit and the second image signal acquired by the second imaging operation. (Item 3) The photoelectric conversion system according to item 2, characterized in that the signal generation unit generates the third image signal by weighting the first image signal and the second image signal held in the storage unit at a predetermined ratio for each pixel. (Item 4) The photoelectric conversion system according to any one of items 1 to 3, characterized in that the storage unit holds the first image signal until the second image signal acquired by the second imaging operation is output from the photoelectric conversion device. (Item 5) The photoelectric conversion system according to any one of items 1 to 4, characterized in that the pixel block includes pixels that output signals corresponding to multiple different colors, and the storage unit periodically thins out the image signals of pixels of the same color in the same row from the image signal acquired by the first imaging operation and stores the remaining image signals in the storage unit. (Item 6) The photoelectric conversion system according to any one of items 1 to 4, characterized in that the pixel block includes pixels that output signals corresponding to a plurality of different colors, and the storage unit periodically thins out the image signals of pixels of at least one of the plurality of different colors among the image signals acquired by the first imaging operation and stores the remaining image signals. (Item 7) The photoelectric conversion system according to any one of items 1 to 6, characterized in that the storage unit holds a predetermined number of bits of the image signal acquired by the first imaging operation, and does not hold the signals of bits lower than the predetermined number of bits. (Item 8) The photoelectric conversion system according to any one of items 1 to 7, characterized in that the storage unit stores the difference value between image signals of pixels of the same color that are adjacent to each other in the same row of the plurality of pixels arranged in a matrix, among the image signals acquired by the first imaging operation. (Item 9) The photoelectric conversion system according to any one of items 1 to 8, characterized in that it has a signal generation unit that generates a third image signal corresponding to the 1 frame period from the second image signal acquired by the second imaging operation when it is determined that there is no flicker based on the determination of whether or not flicker occurs. (Item 10) A photoelectric converter comprising multiple pixel blocks, each containing multiple pixels arranged in a matrix, which performs a first imaging operation during one frame period, including an exposure time based on a preset first exposure condition, and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block, and outputs an image signal based on the first imaging operation and the second imaging operation, A photoelectric converter characterized in that the amount of image signal data output by the photoelectric converter based on the first imaging operation is less than the amount of image signal data output by the photoelectric converter based on the second imaging operation. (Item 11) The photoelectric converter according to item 10, wherein the pixel block includes pixels that output signals corresponding to multiple different colors, and the photoelectric converter periodically thins out the image signals of pixels of the same color in the same row from the image signal acquired by the first imaging operation and outputs the remaining image signal. (Item 12) The photoelectric converter according to item 10, wherein the pixel block includes pixels that output signals corresponding to a plurality of different colors, and the photoelectric converter periodically decimates the image signals of pixels of at least one of the plurality of different colors among the image signals acquired by the first imaging operation and outputs the remaining image signals. (Item 13) The photoelectric converter according to any one of items 10 to 12, characterized in that the photoelectric converter outputs a predetermined number of bits of the image signal acquired by the first imaging operation, and does not output signals of bits lower than the predetermined number of bits. (Item 14) A photoelectric conversion system comprising a photoelectric conversion device described in any one of items 10 to 13, and a storage unit that holds an image signal based on the first imaging operation output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the storage unit holds the image signal based on the first imaging operation until the image signal based on the second imaging operation is output from the photoelectric conversion device. (Item 15) A photoelectric conversion system comprising a photoelectric conversion device described in any one of items 10 to 13, and an image processing device for processing an image signal output from the photoelectric conversion device, The image processing device is a photoelectric conversion system characterized by generating an image signal corresponding to the 1 frame period based on the image signal based on the first imaging operation and the image signal based on the second imaging operation output from the photoelectric conversion device. (Item 16) A photoelectric conversion system comprising a photoelectric conversion device described in any one of items 10 to 13, and an image processing device for processing an image signal output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the image processing device determines whether or not flicker occurs, and when it is determined that there is no flicker, it generates an image signal corresponding to the 1 frame period from the image signal based on the second imaging operation. (Item 17) A photoelectric converter comprising multiple pixel blocks, each containing multiple pixels arranged in a matrix, which performs a first imaging operation during one frame period, including an exposure time based on a preset first exposure condition, and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block, and outputs an image signal based on the first imaging operation, The photoelectric conversion device is characterized in that it outputs an image signal with a data amount smaller than the data amount of the image signal acquired by the first imaging operation as an image signal based on the first imaging operation. (Item 18) The photoelectric converter according to item 17, wherein the pixel block includes pixels that output signals corresponding to multiple different colors, and the photoelectric converter periodically decimates the image signals of pixels of the same color in the same row from the image signal acquired by the first imaging operation and outputs the remaining image signal. (Item 19) The photoelectric converter according to item 17, wherein the pixel block includes pixels that output signals corresponding to multiple different colors, and the photoelectric converter periodically decimates the image signals of pixels of at least one of the multiple different colors among the image signals acquired by the first imaging operation and outputs the remaining image signals. (Item 20) The photoelectric converter according to any one of items 17 to 19, characterized in that the photoelectric converter outputs a predetermined number of bits of the image signal acquired by the first imaging operation, and does not output signals of bits lower than the predetermined number of bits. (Item 21) A photoelectric converter according to any one of items 17 to 20, characterized in that it outputs an image signal based on the second imaging operation. (Item 22) A photoelectric conversion system comprising a photoelectric conversion device described in any one of items 17 to 21, and a storage unit that holds an image signal based on the first imaging operation output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the storage unit holds the image signal based on the first imaging operation until the image signal based on the second imaging operation is output from the photoelectric conversion device. (Item 23) A photoelectric conversion system comprising a photoelectric conversion device described in any one of items 17 to 21, and an image processing device for processing an image signal output from the photoelectric conversion device, The image processing device is characterized by generating an image signal corresponding to the 1 frame period based on the image signal based on the first imaging operation and the image signal based on the second imaging operation output from the photoelectric conversion device. (Item 24) A photoelectric conversion system comprising a photoelectric conversion device described in any one of items 17 to 21, and an image processing device for processing an image signal output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the image processing device determines whether or not flicker occurs, and when it is determined that there is no flicker, it generates an image signal corresponding to the 1 frame period from the image signal based on the second imaging operation. (Item 25) A photoelectric conversion system described in any one of items 1 to 9, 14 to 16, and 22 to 24, The apparatus is characterized by comprising a processing device for processing the output signal from the aforementioned photoelectric conversion system.

[0150] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0151] 100: Imaging device 102: Imaging sensor 103: Image sensor unit 104: A / D conversion unit 105: Exposure compensation unit 109: Exposure time control unit 110: Gain control unit 111: Exposure condition determination unit 112: Exposure time 113: Analog gain value 122: Exposed image 301: Line buffer 302: Addition ratio calculation unit 303: Image addition unit 304: Grayscale extension unit 305: Addition ratio k 306: Delayed image 307: Conversion ratio G

Claims

1. A photoelectric conversion system comprising: a photoelectric conversion device having multiple pixel blocks arranged in a matrix, which perform a first imaging operation including an exposure time based on a preset first exposure condition and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block during one frame period; and a storage unit, The photoelectric conversion system is characterized in that the storage unit holds a first image signal whose data amount is smaller than the data amount of the image signal acquired by the photoelectric conversion device by the first imaging operation.

2. The photoelectric conversion system according to claim 1, further comprising a signal generation unit that generates a third image signal corresponding to the one-frame period based on the first image signal held in the storage unit and the second image signal acquired by the second imaging operation.

3. The photoelectric conversion system according to claim 2, characterized in that the signal generation unit generates the third image signal by weighting the first image signal and the second image signal held in the storage unit by a predetermined ratio for each pixel.

4. The photoelectric conversion system according to claim 1, characterized in that the storage unit holds the first image signal until the second image signal acquired by the second imaging operation is output from the photoelectric conversion device.

5. The photoelectric conversion system according to claim 1, wherein the pixel block includes pixels that output signals corresponding to multiple different colors, and the storage unit periodically thins out the image signals of pixels of the same color in the same row from the image signal acquired by the first imaging operation and stores the remaining image signals in the storage unit.

6. The photoelectric conversion system according to claim 1, wherein the pixel block includes pixels that output signals corresponding to a plurality of different colors, and the storage unit periodically thins out the image signals of pixels of at least one of the plurality of different colors among the image signals acquired by the first imaging operation and stores the remaining image signals.

7. The photoelectric conversion system according to claim 1, characterized in that the storage unit holds a predetermined number of bits of the image signal acquired by the first imaging operation, and does not hold the signals of bits lower than the predetermined number of bits.

8. The photoelectric conversion system according to claim 1, characterized in that the storage unit stores the difference value between image signals of pixels of the same color that are adjacent to each other in the same row of the plurality of pixels arranged in a matrix, among the image signals acquired by the first imaging operation.

9. The photoelectric conversion system according to claim 1, further comprising a signal generation unit that, when it is determined that there is no flicker based on the determination of whether or not flicker occurs, generates a third image signal corresponding to the one-frame period from the second image signal acquired by the second imaging operation.

10. A photoelectric converter comprising a plurality of pixel blocks, each containing multiple pixels arranged in a matrix, which performs a first imaging operation during one frame period, including an exposure time based on a preset first exposure condition, and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block, and outputs an image signal based on the first imaging operation and the second imaging operation, A photoelectric converter characterized in that the amount of data of the image signal based on the first imaging operation output by the photoelectric converter is less than the amount of data of the image signal based on the second imaging operation output by the photoelectric converter.

11. The photoelectric converter according to claim 10, wherein the pixel block includes pixels that output signals corresponding to multiple different colors, and the photoelectric converter periodically thins out the image signals of pixels of the same color in the same row from the image signal acquired by the first imaging operation and outputs the remaining image signal.

12. The photoelectric converter according to claim 10, wherein the pixel block includes pixels that output signals corresponding to a plurality of different colors, and the photoelectric converter periodically thins out the image signals of pixels of at least one of the plurality of different colors among the image signals acquired by the first imaging operation and outputs the remaining image signals.

13. The photoelectric converter according to claim 10, characterized in that the photoelectric converter outputs a predetermined number of bits of the image signal acquired by the first imaging operation, and does not output signals of bits lower than the predetermined number of bits.

14. A photoelectric conversion system comprising: a photoelectric conversion device according to claim 10; and a storage unit that holds an image signal based on the first imaging operation output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the storage unit holds the image signal based on the first imaging operation until the image signal based on the second imaging operation is output from the photoelectric conversion device.

15. A photoelectric conversion system comprising a photoelectric conversion device according to claim 10 and an image processing device for processing an image signal output from the photoelectric conversion device, The image processing device is a photoelectric conversion system characterized by generating an image signal corresponding to one frame period based on an image signal based on a first imaging operation and an image signal based on a second imaging operation, which are output from the photoelectric conversion device.

16. A photoelectric conversion system comprising a photoelectric conversion device according to claim 10 and an image processing device for processing an image signal output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the image processing device determines whether or not flicker occurs, and when it is determined that there is no flicker, it generates an image signal corresponding to the one-frame period from the image signal based on the second imaging operation.

17. A photoelectric converter comprising multiple pixel blocks, each containing multiple pixels arranged in a matrix, which performs a first imaging operation during one frame period, including an exposure time based on a preset first exposure condition, and a second imaging operation in which the exposure time is controlled based on a second exposure condition determined for each pixel block, and outputs an image signal based on the first imaging operation, The photoelectric conversion device is characterized in that it outputs an image signal with a data amount smaller than the data amount of the image signal acquired by the first imaging operation as an image signal based on the first imaging operation.

18. The photoelectric conversion device according to claim 17, wherein the pixel block includes pixels that output signals corresponding to multiple different colors, and the photoelectric conversion device periodically thins out the image signals of pixels of the same color in the same row from the image signal acquired by the first imaging operation and outputs the remaining image signal.

19. The photoelectric converter according to claim 17, wherein the pixel block includes pixels that output signals corresponding to a plurality of different colors, and the photoelectric converter periodically decimates the image signals of pixels of at least one of the plurality of different colors among the image signals acquired by the first imaging operation and outputs the remaining image signals.

20. The photoelectric converter according to claim 17, characterized in that the photoelectric converter outputs a predetermined number of bits of the image signal acquired by the first imaging operation, and does not output signals of bits lower than the predetermined number of bits.

21. The photoelectric converter according to claim 17, characterized in that it outputs an image signal based on the second imaging operation.

22. A photoelectric conversion system comprising a photoelectric conversion device according to claim 17, and a storage unit that holds an image signal based on the first imaging operation output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the storage unit holds the image signal based on the first imaging operation until the image signal based on the second imaging operation is output from the photoelectric conversion device.

23. A photoelectric conversion system comprising a photoelectric conversion device according to claim 17 and an image processing device for processing an image signal output from the photoelectric conversion device, The image processing device is a photoelectric conversion system characterized by generating an image signal corresponding to one frame period based on an image signal based on a first imaging operation and an image signal based on a second imaging operation output from the photoelectric conversion device.

24. A photoelectric conversion system comprising a photoelectric conversion device according to claim 17 and an image processing device for processing an image signal output from the photoelectric conversion device, The photoelectric conversion system is characterized in that the image processing device determines whether or not flicker occurs, and when it is determined that there is no flicker, it generates an image signal corresponding to the one-frame period from the image signal based on the second imaging operation.

25. A photoelectric conversion system according to any one of claims 1 to 9, claims 14 to 16, and claims 22 to 24, The apparatus is characterized by comprising a processing device for processing the output signal from the aforementioned photoelectric conversion system.

Images