Imaging control device, image processing device, imaging control method, program, imaging device, and device using the imaging device

Abstract

To provide a technique advantageous for reducing a storage capacity used for processing of imaging in a wide dynamic range while suppressing an influence on imaging due to flicker.SOLUTION: An imaging control device includes a plurality of pixel blocks, which makes an imaging element having a plurality of pixels to perform an imaging of a moving image. The imaging control device includes: a holding part configured to hold a light emission period of a light source that periodically emits light; and a control part configured to control a first period and a second period that is shorter than the light emission period following the first period in one frame period in imaging of the moving image. The control part sets a first exposure time including a charge accumulation time equal to or longer than the light emission period in the first period and sets a second exposure time in the second period based on exposure conditions determined for each of the plurality of pixel blocks.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an imaging control device, an image processing device, an imaging device, an imaging control method, a program, and equipment using the imaging device. [Background technology]

[0002] In order to widen the dynamic range of an imaging device, a method has been proposed in which the exposure conditions of an image sensor are changed for each region. Patent Document 1 discloses that the entire light receiving region of the image sensor is divided into multiple regions, and the exposure time and the amplification gain for the analog signal from the sensor are set for each region.

[0003] On the other hand, if the exposure time of a certain area is set to be short, and if that area includes a light source that causes flicker, flicker may occur if the exposure time is shorter than the blinking period of the light source. Patent Document 2 describes a technology for dealing with flicker and widening the dynamic range. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-129144 [Patent Document 2] Patent application 2024-7546 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology of Patent Document 2 requires storing images in a buffer memory for a predetermined period of time in order to combine the images. Given the demand for high-resolution imaging, having a large-capacity buffer memory not only increases the chip area, hinders chip miniaturization, but can also lead to increased chip costs. The present disclosure aims to provide a technology that is advantageous for reducing the storage capacity used to process wide-dynamic-range imaging while suppressing the effects of flicker on imaging. [Means for solving the problem]

[0006] One aspect of the present disclosure is an imaging control device that causes an imaging element having a plurality of pixel blocks, each pixel block having a plurality of pixels, to capture a moving image, the imaging control device comprising: a holding unit that holds the light emission period of a light source that emits light periodically; and a control unit that controls a first period and a second period that follows the first period and is shorter than the light emission period, within one frame period of capturing the moving image, wherein the control unit sets a first exposure time in the first period that includes a charge accumulation time equal to or longer than the light emission period, and sets a second exposure time in the second period, based on exposure conditions determined for each of the plurality of pixel blocks. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide a technique that is advantageous for reducing the storage capacity used for processing wide dynamic range imaging while suppressing the influence of flicker on imaging. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram showing an example of a schematic configuration of an imaging apparatus and a connection to an external controller. [Figure 2] FIG. 2 is a diagram illustrating an image sensor unit. [Figure 3] FIG. 2 is a diagram showing an example of the arrangement of an exposure correction unit according to the first embodiment. [Figure 4] 10 is a diagram showing the relationship between exposure time control and the light emission cycle of an LED light source of a subject to be photographed. [Figure 5]5A and 5B are diagrams illustrating exposure times in area-specific exposure control. [Figure 6] FIG. 10 is a diagram illustrating a reduction in storage capacity. [Figure 7] FIG. 10 is a diagram showing the relationship between exposure time and analog gain in area-by-area exposure control. [Figure 8] FIG. 10 is a diagram showing an example of the arrangement of an exposure correction unit according to the second embodiment. [Figure 9] 10 is a flowchart showing the flow of processing in the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an exposure time in the second embodiment. [Figure 11] 10 is a flowchart showing the flow of processing in the second embodiment. [Figure 12] 10A and 10B are diagrams illustrating exposure times in area-by-area exposure control according to the second embodiment. [Figure 13] 10A and 10B are diagrams illustrating exposure times in area-by-area exposure control according to the second embodiment. [Figure 14] 11 is a diagram showing the relationship between exposure time and analog gain in area-by-area exposure control according to the third embodiment. [Figure 15] Application examples of imaging devices according to embodiments to devices DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [Embodiment 1] Fig. 1 is a block diagram showing a schematic configuration example of an imaging device 100 to which an image processing device according to this embodiment is applied, and connection with an external controller. The imaging device 100 of this embodiment also has various components that a typical imaging device has, but for simplicity of illustration and explanation, Fig. 1 shows only the main components according to this embodiment. Note that the components described below are merely examples, and the functions of multiple components described below may be combined into one or separated. Alternatively, one component may also have other functions.

[0011] The imaging device 100 according to this embodiment includes a synchronization control unit 101, an imaging element unit 103, an analog-to-digital (A / D) conversion unit 104, an exposure correction unit 105, a gradation conversion unit 106, an image output unit 108, an exposure time control unit 109, a gain control unit 110, and an exposure condition determination unit 111. The imaging device 100 also includes a serial interface (SIO I / F) 141 and a register 142 for storing setting values ​​in order to reflect settings provided from an external controller 10. The imaging device 100 is connected to the external controller 10 via a serial communication line 11 and an output signal line 12.

[0012] The image sensor 102 of this embodiment can include an image sensor section 103 in which pixels including photoelectric conversion elements are arranged, and an A / D conversion section 104 that performs analog-to-digital (A / D) conversion on signals from the pixel section.

[0013] The synchronization control unit 101, the exposure time control unit 109, the gain control unit 110, and the exposure condition determination unit 111 can each be considered an imaging control device that controls imaging. The imaging control device may include at least one of the synchronization control unit 101, the exposure time control unit 109, the gain control unit 110, and the exposure condition determination unit 111. The exposure correction unit 105 and the gradation conversion unit 106 each function as an image processing device that performs image processing on the exposure image 122. The image processing device may include at least one of the exposure correction unit 105 and the gradation conversion unit 106. The imaging device 100 may also include a control unit 150 that controls the imaging device 100. The control unit 150 may be included in each unit of the imaging device 100. The control unit 150 can control each unit of the imaging device 100, such as the synchronization control unit 101, the exposure time control unit 109, the gain control unit 110, the exposure correction unit 105, and the gradation conversion unit 106. The control unit 150 can also control a part of or the entire imaging device 100.

[0014] An overview of each component of the imaging device 100 will be explained, starting with the imaging element unit 103. The imaging element unit 103 has an imaging area (light-receiving area). A plurality of pixels are arranged in the imaging area. The imaging area is further divided into a plurality of areas called pixel blocks, each of which includes a plurality of pixels. The imaging element unit 103 can be driven in pixel block (area) units, and has the function of determining exposure conditions for each area and performing exposure operations with different exposure times. The exposure time corresponds to the charge accumulation time during which the photoelectric conversion elements included in the pixels can accumulate charge. Note that pixel blocks will be explained later with reference to Figure 2.

[0015] In this embodiment, the image sensor unit 103 has an exposure time set for each region by an exposure control signal 117 supplied from the exposure time control unit 109, and performs exposure for the exposure time set for each region. The exposure control signal 117 is a signal for setting an exposure time for each region of the image sensor unit 103. The image sensor unit 103 then reads out the charge accumulated in each pixel for the exposure time set for each region by the exposure control signal 117 as a pixel potential 118 and outputs it to the A / D conversion unit 104. The A / D conversion unit 104 performs analog-to-digital conversion on the pixel potential 118 read out from the image sensor unit 103, converting it into a digital value. The gain control unit 110 can set an analog gain 121 corresponding to each region in the A / D conversion unit 104. The A / D conversion unit 104 amplifies the pixel potential 118 output from the image sensor unit 103 by the analog gain 121 set for each region, and then performs analog-to-digital conversion on the signal to convert it into a digital value.

[0016] Hereinafter, an image made up of a digital signal that has been amplified by analog gain 121 for each region in A / D conversion unit 104 and then analog-to-digital converted will be referred to as an exposure image 122. The exposure image 122 output from the A / D conversion unit 104 is sent to exposure condition determination unit 111 and exposure correction unit 105.

[0017] The exposure condition determination unit 111 can determine the exposure time 112 and analog gain value 113 for each region based on the exposure image 122 so as to achieve optimal imaging conditions, and update the previous values. For example, the exposure condition determination unit 111 obtains a histogram of pixel values ​​for each pixel block based on the luminance distribution of the exposure image 122. Then, if the pixel values ​​are distributed toward the bright area side, the exposure condition determination unit 111 can change and update the exposure time 112 and analog gain value 113 of that pixel block (region) to set values ​​that will result in a darker image.

[0018] Furthermore, if the pixel values ​​are distributed on the dark side, the exposure condition determination unit 111 can change and update the exposure time 112 and analog gain value 113 of that pixel block (area) to set values ​​that will result in a brighter image. The exposure time 112 value for each area is then sent to the exposure time control unit 109 and exposure correction unit 105. The analog gain value 113 for each area is sent to the gain control unit 110 and exposure correction unit 105.

[0019] The synchronization control unit 101 generates a synchronized exposure time output pulse 120 and a gain output pulse 114. The synchronization control unit 101 outputs the exposure time output pulse 120 to the exposure time control unit 109. The synchronization control unit 101 outputs the gain output pulse 114 to the gain control unit 110. In this way, the synchronization control unit 101 controls the synchronization of the processing of the exposure time control unit 109 and the processing of the gain control unit 110.

[0020] The exposure time output pulse 120 is a signal for controlling the timing at which the exposure time control unit 109 outputs the exposure control signal 117 to the image sensor unit 103. The exposure time control unit 109 outputs the exposure control signal 117 to the image sensor unit 103 based on the exposure time output pulse 120, thereby setting the exposure time for each arbitrary pixel block of the image sensor unit 103.

[0021] Furthermore, the gain output pulse 114 is a signal for controlling the timing at which the gain control unit 110 outputs the analog gain 121 to the A / D conversion unit 104. The gain control unit 110 sets the gain to be applied to the pixel potential for each arbitrary pixel block by outputting the analog gain 121 to the A / D conversion unit 104 based on the gain output pulse 114. In this manner, in this embodiment, the synchronization control unit 101 controls the operations of the exposure time control unit 109 and the gain control unit 110 in synchronization with each other, thereby making it possible to output an exposure image 122 to which an exposure time and analog gain have been applied for each pixel block of the image sensor unit 103.

[0022] The exposure time control unit 109 generates an exposure control signal 117 for each region based on the exposure time output pulse 120 and the value of the exposure time 112 for each region, and outputs it to the image sensor unit 103. As a result, an exposure time according to the exposure time 112 for each region is set for the image sensor unit 103 at an appropriate timing.

[0023] The gain control unit 110 outputs the analog gain value 113 for each region to the A / D conversion unit 104 in synchronization with the timing of the gain output pulse 114 as an analog gain 121 for each region for the pixel potential 118 for each region of the image sensor unit 103. As a result, the A / D conversion unit 104 performs analog-to-digital conversion after multiplying the pixel potential 118 for each region by the analog gain 121 for each region corresponding to the region. The data that has undergone analog-to-digital conversion is sent to the exposure correction unit 105 and the exposure condition determination unit 111 as an exposure image 122 for each region.

[0024] The exposure correction unit 105 accumulates exposure images 122 captured under different exposure conditions within the same frame for the exposure images 122 for each region sent from the A / D conversion unit 104, performs necessary processing, and then performs addition processing on each pixel data. The added image is then subjected to gradation expansion processing based on the exposure time 112 and analog gain value 113 to generate a gradation-extended image 123. The exposure correction unit 105 can generate a gradation-extended image 123 represented by a 23-bit number from the exposure images 122 for each region represented by a 10-bit number, for example. The detailed operation of the exposure correction unit 105 will be described later. The generated gradation-extended image 123 is then sent to the gradation conversion unit 106.

[0025] The gradation conversion unit 106 performs gradation conversion on the gradation-extended image 123 and outputs the gradation-converted image 124 to the image output unit 108. In this embodiment, the gradation conversion is a process of converting, for example, the 23-bit gradation-extended image 123 into, for example, a 12-bit signal using gamma conversion to generate the gradation-converted image 124. Note that the gradation conversion process in this embodiment is performed to reduce the data rate in subsequent processing. In this embodiment, the bit lengths of the exposed image 122 and the gradation-converted image 124 are 10 bits and 12 bits, respectively, but these bit lengths are merely examples and are not limited to these.

[0026] The image output unit 108 outputs the tone-converted image 124 to a downstream configuration of the imaging device 100 or to the outside. In this embodiment, a controller 10 is connected as a processing module that receives image data from the imaging device 100. Here, the output signal line 12 connecting the image output unit 108 and the controller 10 may be an LVDS signal line with 16 data channels. However, the type of this signal line and the data channel width are not limited by this embodiment and can be selected depending on the amount of data and the data speed.

[0027] The controller 10 is also connected to a serial I / O (SIO) I / F 141 of the imaging device 100 via a serial communication line 11. The SIO I / F 141 is connected to a register 142, and the controller 10 can set necessary information in the register 142 inside the imaging device 100 via the SIO I / F 141. The information set in the register 142 is transmitted to an exposure condition determination unit 111, and can be used to determine the exposure conditions.

[0028] FIG. 2 is a diagram illustrating an example configuration of the image sensor unit 103. The imaging region of the image sensor unit 103 has a plurality of pixel blocks 201. Each pixel block 201 has a plurality of pixels 202. In this embodiment, the number of pixels in the width 206 direction (horizontal line direction) of the imaging region of the image sensor unit 103 is assumed to be 2000 pixels, and the number of pixels in the height 205 direction is assumed to be 1000 pixels (i.e., 1000 horizontal lines in the vertical direction). The number of pixels in the width 204 direction (horizontal line direction) of the pixel block 201 is assumed to be 100 pixels, and the number of pixels in the height 203 direction is assumed to be 100 pixels (100 horizontal lines in the vertical direction). In this case, the number of pixel blocks 201 in the imaging region of the image sensor unit 103 is 20 in the horizontal direction and 10 in the vertical direction. These numbers of pixels and lines are merely examples for illustrative purposes and are not limiting.

[0029] The "pixel blocks [0,0] to [19,9]" written in each pixel block 201 shown in Figure 2 indicate the position of each pixel block 201 within the imaging area. The values ​​in brackets [ ] indicate the horizontal and vertical indexes of each pixel block within the imaging area. In Figure 2, for example, the pixel block 201 located in the upper right corner of the imaging element unit 103 is pixel block [19,0].

[0030] A set of pixel blocks represented by the same vertical index is referred to as a block row. Block row N consists of pixel blocks [0,N] to [19,N], where N ranges from 0 to 9 in FIG. 2. For example, block row 5 consists of pixel blocks [0,5] to [19,5]. The sizes (number of pixels in the vertical and horizontal directions) of the image sensor unit 103 and the pixel blocks 201 are not limited to those described above. The shape and aspect ratio of the pixel 202 are also not limited; for example, it may be rectangular rather than square. Furthermore, the pixel block 201 may consist of only one pixel 202. In this embodiment, the exposure time and analog gain can be controlled for each pixel block 201.

[0031] Here, the exposure time corresponds to the charge accumulation time during which charge is accumulated in the pixels (light receiving elements) of the image sensor unit 103 during image capture. Therefore, for example, assuming that the amount of light incident on the image sensor unit 103 is the same and the pixels are not saturated, the longer the exposure time, the higher the pixel potential 118, and the brighter the image can be captured. In other words, if the amount of incident light is the same and pixel saturation is not taken into consideration, comparing an exposure time of (1 / 480) seconds with an exposure time of (1 / 30) seconds, for example, the brighter the image can be captured with an exposure time of (1 / 30) seconds.

[0032] The analog gain is a gain applied to the pixel potential 118 in the A / D conversion unit 104 during imaging. Therefore, the larger the analog gain value, the larger the digital pixel value (digital value obtained by analog-to-digital conversion after gain application) output from the A / D conversion unit 104.

[0033] 1, the configuration and operation of the imaging device 100 of this embodiment will be described. The imaging element unit 103 captures an image while the exposure time is controlled for each region, i.e., for each pixel block 201, based on the exposure control signal 117. The imaging element unit 103 then outputs a pixel potential 118 corresponding to the charge accumulated in each pixel.

[0034] The A / D conversion unit 104 multiplies the pixel potential 118 output from the image sensor unit 103 by an analog gain 121 set for each pixel block of the image sensor unit 103, then performs digital conversion and outputs an exposed image 122. For the sake of explanation, in this embodiment, the exposed image 122 is assumed to be a 10-bit digital value. The analog gain 121 can take four gain values, for example, ×1, ×2, ×4, and ×8.

[0035] The exposure correction unit 105 performs gradation expansion processing on the exposure image 122 input from the A / D conversion unit 104 based on the exposure time 112 and analog gain value 113 set for each region, and outputs a gradation-expanded image 123. The exposure correction unit 105 recognizes the conditions under which the exposure image 122 for each region was captured based on the exposure time 112 for each region and the analog gain value 113 for each region. Then, the exposure correction unit 105 corrects the exposure image 122 for each region based on the conditions under which the exposure image 122 for each region was captured.

[0036] The exposure correction unit 105 performs gradation expansion processing based on the exposure time 112 and analog gain value 113 applied to the image capture on the exposure image 122 for each region sent from the A / D conversion unit 104, to generate a gradation-expanded image 123. For example, the exposure correction unit 105 recognizes the conditions under which the input exposure image 122 for each region was captured based on the exposure time 112 for each region and the analog gain value 113 for each region, and corrects the exposure image 122 for each region according to those conditions.

[0037] The exposure correction unit 105 also performs gradation extension processing on the exposed image 122 for each region, which is expressed by, for example, 10 bits, to generate a gradation-extended image 123 expressed by 23 bits. The generated gradation-extended image 123 is then sent to the gradation conversion unit 106.

[0038] Next, we will explain the operation of the exposure correction unit 105. Fig. 3 is a block diagram showing an example configuration of the exposure correction unit 105. The exposure correction unit 105 has a line buffer 301 as a storage unit, an addition ratio determination unit 302, an image addition unit 303, and a gradation expansion unit 304.

[0039] The operation of each component of the exposure correction unit shown in Fig. 3 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram illustrating an example of the relationship between exposure time control and the light emission cycle of the LED light source of the subject in this embodiment.

[0040] In this embodiment, the image output frame rate when the imaging device 100 captures a moving image will be described as 30 frames per second. The frame period λ for this frame rate is 1 / 30 seconds (33.3333 ms (rounded to the fifth decimal place, same below)). Furthermore, regarding the source of flicker, it is assumed that the LED light source that may be included in the subject of imaging has a maximum light emission frequency of 90 Hz and emits light at a constant cycle with a duty of 50%. In this case, the longest light emission period T of the light source that may cause flicker is 1 / 90 seconds (11.1111 ms).

[0041] In this embodiment, the imaging period of each frame is divided into two periods: a first period 401-1, 401-2 in the first half of the frame, and a second period 402-1, 402-2 in the second half of the frame following the first period, and each period is defined as shown in Fig. 4. Note that, hereinafter, when describing the first period in any way, it will be referred to as "first period 401," and when describing the second period in any way, it will be referred to as "second period 402."

[0042] In the example of FIG. 4 , the second period 402 in the latter half has a length t2 determined by 1 / n of T (n is an integer equal to or greater than 2, but in the example of FIG. 4 , n=4). In this case, t2 is 2.7778 ms, which is 1 / 4 of 11.1111 ms. In this embodiment, the first period 401 can be the period remaining after subtracting the second period 402 from one frame period λ. The length of the first period 401 is represented as t1. Here, t1=λ-t2, and in this embodiment, t1=30.5556 ms. In FIG. 4 , the first exposure time of the first period 401 is represented as 403-1 and 403-2, and the second exposure time of the second period is represented as 404-1 and 404-2. Note that, hereinafter, the first exposure period will be referred to as the first exposure period 403 when describing either one of them, and the second exposure period will be referred to as the second exposure period 404 when describing either one of them.

[0043] The exposure times of the image sensor in the first period 401 and the second period 402 are controlled by the exposure time control unit 109 based on the exposure time 112 determined by the exposure condition determination unit 111 in FIG. 1. The light emission cycle T of the LED light source for which flicker is to be reduced and the division number n that determines the length of the second period are set in the register 142 by the external controller 10, and can be distributed from the register setting value 143. The exposure condition determination unit 111 determines the length of the second period using the light emission cycle T and the division number n supplied as the register setting value 143. In this embodiment, the length of the first period is determined depending on the length of the second period, as will be described later. The exposure time control unit determines the first period and the second period based on the length of the second period, and sets the exposure time for each period.

[0044] The exposure time in the first period is defined as a first exposure time T1, and the exposure time in the second period is defined as a second exposure time T2. Each exposure time can be determined based on an exposure image 122 in the second period of the previous frame, in accordance with an algorithm for determining exposure conditions for area-specific exposure. The exposure conditions may also be determined using a table. Here, when the length of the frame period is λ, the appropriate exposure time determined by the exposure condition determination unit 111 to be set for area-specific exposure based on the exposure image 122 is defined as λ1. Here, the appropriate exposure time λ1 is the exposure time that should be set as an appropriate exposure time for one frame period λ when one frame period is not considered as being divided into a first period and a second period.

[0045] Because one frame period is divided into two periods, a first period and a second period, the image captured during the first period is converted as follows when combining the images captured during the first and second periods. This conversion converts the image signals captured during the first and second imaging periods into image signals captured at the appropriate exposure time λ1. The image signal captured during the first imaging period is designated the first image signal O1, the image signal captured during the second imaging period is designated the second image signal O2, and a composite image signal of these two signals is designated the composite image signal O3. The first image signal O1 is multiplied by the addition ratio k shown in Figure 3 and added to the second image signal O2 to obtain the composite image signal O3. This relationship is expressed as O3 = k × O1 + O2, where k is the value of the addition ratio k. The addition ratio k will be explained below for three cases.

[0046] Case 1 The first exposure time is T1, the second exposure time is T2, and the proper exposure time is λ1. The longest light emission period of a light source that emits light periodically and that is most susceptible to flicker is T, and the imaging period of a moving image, i.e., one frame period, is λ. The second period T2 is T / n. The ratio of the frame period λ to the proper exposure time λ1 is α=λ / λ1 (1) Let's say.

[0047] Condition 1: λ1≧(1+1 / n)×T. This is the case when the proper exposure time λ1 is equal to or longer than the sum of the length T / n of the second period and the light emission period T. In this case, the second exposure time is equal to the second period, and the first exposure period is longer than the light emission period T.

[0048] Therefore, the following relationship can be established: T1=λ1-T2 (2) T2=T / n (3) T2 is equal to the length of the second period. At this time, T1>T, so the first image signal O1 can be used as is. Therefore, the value k of the addition ratio k is as follows: k=1 (4) In this way, in case 1, λ1 is greater than the sum of the length of the second period and the length of the light emission cycle T, so imaging can be performed with the appropriate exposure time λ1.

[0049] Case 2 The appropriate exposure time λ1 is greater than the light emission cycle T. In the second period, exposure is performed for a length equal to the second period's length t2, and in the first period, exposure is performed for the length of the light emission cycle T, so in this case, the combined exposure time of the first period and the second period may exceed the appropriate exposure time λ1. The conditions for Case 2 are as follows:

[0050] Condition 2: (1+1 / n)×T>λ1≧T. At this time, T1=T (5) T2=T / n (3) T2 is equal to the length of the second period t2. Furthermore, T1 is equal to T. In this embodiment, n=4 as described above.

[0051] In this case, the first exposure time T is longer than the appropriate exposure time λ1, so correction is made using the addition ratio k. Specifically, the following equation holds: λ1=T×k+T2 (6) Substituting equation (3) into the above equation, we get k: k=λ1 / T−1 / n (7) This becomes:

[0052] Case 3 When the appropriate exposure time λ1 is smaller than T, that is, Condition 3 λ1 <Tのとき。

[0053] In this case, the second exposure time T2 is set to T / α, and the exposure time T of the first period is corrected by the addition ratio k. In this case, the following equation holds: λ1=T×k+T2 Find the above equation for k. α=λ / λ1 (1) T1=T (5) T2 = T / α ··· (8) From the relationship of k = ((λ / T) - 1) / α ··· (9) it can be expressed as

[0054] In this embodiment, since λ = 33.3333 ms and T = 11.1111 ms, k = ((33.3333 / 11.1111) - 1) / α = (3 - 1) / α = 2 / α is obtained.

[0055] When λ1 ≥ (1 + 1 / n) × T, substituting (3) into (2) gives T1 = λ1 - T2 = λ1 - T / n ≥ (1 + 1 / n) × T - T / n = T From the relationship of it can be said that T1 is a value greater than or equal to T. Also, when (1 + 1 / n) × T > λ1 from formula (5), T1 = T, so it can be said that T1 in the first period always has a period greater than or equal to T.

[0056] The above relationships regarding the shutter speed are summarized in FIG. 5. FIG. 5 is a table showing the relationships of T1, T2, λ1, and k in this embodiment. At this time, the length of one frame period is 33.3333 ms, T = 11.1111 ms, and n = 4. For example, if the shutter speed to be set as one frame period in area - by - area exposure is 1 / 30 second, then λ1 at that time is 33.3333 ms. Since λ1 ≥ (1 + 1 / n) × T, k = 1. When the shutter speed is 1 / 60 second, k = 1 as well.

[0057] If the shutter speed to be set as one frame period in area - by - area exposure is 1 / 960 second, then λ1 at that time is 1.0417 ms. Since λ1 < T, it corresponds to Case 3. According to formulas (1), (5), (6), and (8), α = λ / λ1 = 32 T1 = T = 11.1111 ms T2 = T / α = 11.1111 / 32 = 0.3472 ms are obtained. Also, k = 2 / α = 1 / 16.

[0058] As shown in Figure 5, within the normal shutter speed setting range, the shutter speed is not set to a value that satisfies Case 2, (1 + 1 / n) × T > λ1 ≥ T, and therefore the case of equation (7) is not shown in Figure 5.

[0059] Such calculations are performed by the exposure condition determination unit 111, and the values ​​of T1 and T2 are set as the exposure time 112. It is preferable that the analog gain value 113 at this time be 1x except when the shutter speed is 1 / 30 seconds. As will be described in detail in the explanation of FIG. 7, there are multiple possible combinations of shutter speed and analog gain when setting the same exposure conditions. However, from the viewpoint of noise, it is preferable to use as long a shutter speed as possible and to make the analog gain as small as possible when realizing the same exposure conditions. For this reason, in this embodiment, the analog gain is set to the minimum of 1x except when the shutter speed is 1 / 30 seconds. When the shutter speed is 1 / 30 seconds, a predetermined analog gain value determined by the area-specific exposure can be set.

[0060] The length of the light emission cycle T can be determined depending on the destination where the imaging device will be used as a product. Alternatively, it can be set in advance as data to be read by the external controller 10 and set in register 142, which is a storage unit of the imaging device. Alternatively, the destination can be determined using location information such as GPS, and when the imaging device is started, the external controller 10 can refer to a table corresponding to the location information and set the light emission cycle T or information that can determine the light emission cycle T in register 142.

[0061] The operation of the exposure correction unit 105 will be described with reference to Figure 3. First, an exposure image 122 for a first period of frame 1 in Figure 4 is input to the exposure correction unit 105. Here, the image signal of the exposure image 122 for a first period of a certain frame at a certain pixel position is a first image signal O1. The data of the first image signal O1 is first stored in a line buffer 301. After a second period has elapsed, an exposure image 122 for a second period of frame 1 is input. At this time, the image signal of the exposure image 122 for a second period of the same frame is a second image signal O2. When the second image signal O2 is supplied as the exposure image 122, the image adder 303 reads the data of the first image signal O1 from the line buffer 301 as a delayed image 306.

[0062] At this time, the memory unit that held the data of the first image signal O1 for the first period of a certain frame in the line buffer 301 will hold the data of the first image signal O1 for the length of the second period. After the data of the first image signal O1 has been read out from the memory unit, the memory unit is reused to hold the first image signal O1 of the first period exposure image 122 for another pixel position that is input after the data has been read out. By shortening the length of the second period, the memory unit is used more frequently, which results in a reduction in the capacity of the memory unit built into the sensor.

[0063] This relationship will be explained using Figure 6, which shows the data readout timing. For the sake of explanation, Figure 6 shows that the second period of frame R1 is set longer than the second period of frame R2. The period between times t11 and t12 in Figure 6 is the second period of frame R1, and the period between times t21 and t22 is the second period of frame R1. The period between times t11 and t12 is longer than the period between times t21 and t22 in Figure 6. The periods indicated by arrows on each line in Figure 6 indicate the period during which the pixels of each line are exposed. The vertical direction in the figure indicates the offset between the exposure timing of each row read out using the rolling shutter method and the data readout timing. In this embodiment, the vertical direction is 1000 lines, so there are lines from line 0 to line 999. After line 999, a blank period passes, and then line 0 of the next frame begins. The frame R2 at line 0 at the top of the figure and the frame R2 at line 0 at the bottom are the same. Also, N1 in the figure indicates the offset between line 0 and line 0. <N2<M1<M2である。

[0064] Now, the data for the first period of frame R1 of the line indicated by line 0 needs to be stored at least from time t11 to time t12. This is because the data for the first period of frame R1 of line 0 needs to be combined with the data for the second period. As shown in FIG. 6, the time when reading of the data for the second period of frame R1 ends is time t12. At this point, the data for the first period of line 0 stored in the memory unit is read out and combined.

[0065] On the other hand, when the data of the first period of the line indicated by line M1 is read out, the time is also time t12. In this case, if it is assumed that the data reading of the first period of line 0 and the data writing of the first period of line M1 cannot be performed simultaneously in the storage unit, the storage unit needs to have a storage capacity for holding the data from line 0 to line M1. In this case, the storage capacity of M1 is required for the amount of data to be held per line. When the data of the first period of the line indicated by line M2 is read out, it is after the data reading of the first period of line 0 has been performed in the storage unit. Therefore, it becomes possible to use the storage unit that has been used to hold the data of line 0 to hold the data of line M2.

[0066] On the other hand, in frame R2, since the exposure time of the data in the second period is shortened, the data of the first period of the frame R2 of the line indicated by line 0 only needs to be stored from time t21 to time t22. In this example, the time when the data of the first period of the line indicated by line N1 is read out is time t22. Therefore, as the storage unit, similarly to frame R1, it only needs to have a storage capacity for holding the data from line 0 to line N1. As described above, since N1 < M1, if the second period of the frame is shortened, the capacity of the line buffer 301 of the exposure correction unit 105 can be reduced.

[0067] Next, a composite image signal O3 of the first image signal O1 and the second image signal O2 can be obtained based on the following equation (10) according to the addition ratio k305 determined by the addition ratio determination unit 302. O3 = kO1 + O2 ··· (10) The determination of the addition ratio k is performed by the addition ratio determination unit 302. The addition ratio determination unit 302 obtains the value of λ1 as the exposure time 112. The addition ratio determination unit 302 determines the values of the first exposure time T1 and the second exposure time T2 from the value of λ1, the values of T and n distributed as the register setting value 143, and the value of λ held as the system. From the result, the addition ratio determination unit 302 obtains the addition ratio k according to the aforementioned equation (4) or equation (7) or equation (9).

[0068] When the addition ratio k is less than 1, the portion of the value less than the significant digit is discarded when calculating the value of the composite image signal O3 in equation (10). As shown in FIG. 5, when the value of λ1 determined in the area-specific exposure is small, k also becomes small, and the contribution of the first image signal O1 to the composite image signal O3 becomes smaller than that of the second image signal O2. Furthermore, depending on the exposure time, the first image signal O1 may be saturated. In this embodiment, O1 is a period equal to or longer than the light emission cycle of the LED light source, preventing the LED light source from being turned off if it is included in the subject being photographed. Although the contribution of the LED light source to the composite image signal O3 becomes small, the configuration of this embodiment makes it possible to prevent the LED light source from being turned off.

[0069] Next, a method of implementing gradation expansion in the gradation expansion unit 304 will be described with reference to FIG. 7. FIG. 7 is a table showing the relationship between exposure time and analog gain in regional exposure control. This table is applied when the frame rate is 30 fps, and shows the relationship between exposure time and analog gain for the entire frame period. In FIG. 7, the number at the intersection of the horizontal analog gain and exposure time in the table is referred to as the EV value. The EV value represents the ratio of the brightness of the object to be imaged, expressed as a power of 2, when the pixel values ​​obtained by imaging are the same. If the brightness of an object captured with an analog gain × 8 and an exposure time of 1 / 30 seconds is L0, the brightness LE of the object when the same pixel values ​​are obtained when captured with an EV value E can be expressed by the following equation (11):

[0070] LE=L0×2^E (11) In this specification, the symbol "^" indicates a power. In other words, the notation A^B indicates A to the B power. For example, when the analog gain is 1 and the exposure time is 1 / 30 seconds, the EV is 3. Similarly, when the analog gain is 1 and the exposure time is 1 / 60 seconds, the EV is 4. In this case, if the pixel values ​​obtained by shooting are the same, an object photographed under an EV value of 4 will be twice as bright as an object photographed under an EV value of 3.

[0071] In area-specific exposure, a table like the one shown in Figure 7 is used to estimate the brightness of the subject in that area from the pixel values ​​captured in the previous frame and the exposure conditions used for capturing that frame, and then the exposure conditions to be used for capturing the next frame can be determined.

[0072] 3, the synthesized image signal O3 of O1 and O2 determined by the image adder 303 based on equation (10) is transmitted to the gradation expansion unit 304. The gradation expansion unit 304 calculates the EV value shown in FIG. 7 from the exposure time 112 and analog gain value 113 of the corresponding pixel block. If the EV value at that time is E, the pixel value after gradation expansion can be calculated based on equation (11).

[0073] For example, suppose the exposure condition for frame 1 shown in FIG. 4 is λ1=33.3333 ms, and the analog gain at this time is ×2. In this case, as shown in FIG. 4, the exposure time 403-1 is 30.5556 ms, the same as in the first period, and the exposure time 404-1 in the second period is 2.7778 ms, the same as in the second period. Under these exposure conditions, the EV value is calculated to be 2 from FIG. 7. In this case, if the composite image signal after addition processing of frame 1 is O31 and the value of the signal after gradation expansion is O41, then, using equation (11), O41=O31×2^2=4×O31.

[0074] Similarly, for example, assume that the exposure condition for frame 2 shown in FIG. 4 is λ1 = 4.1667 ms (1 / 240 S), and the analog gain at this time is ×1. In this case, as shown in FIG. 4, exposure time 403-2 is 11.1111 ms, the same as T, and exposure time 404-2 in the second period is 1.3889 ms. Under these exposure conditions, the EV is calculated to be 6 from FIG. 7. In this case, if the composite image signal after addition processing for frame 2 is O32 and the value of the signal after gradation expansion is O42, then using equation (11), O42 = O32 × 2^6 = 64 × O32.

[0075] In this case, if the bit width of the original exposure image O32 is 10 bits, this calculation results in O42 being a 16-bit number. In this embodiment, it can be seen from FIG. 7 that the EV value can be up to 13. In this case, if the exposure correction unit 105 performs gradation extension, the bit width of the data will increase. In this embodiment, the gradation-extended image 123 can potentially be up to 23 bits. Then, as described above, the gradation conversion unit 106 performs gamma conversion from the 23-bit gradation-extended image 123 to generate, for example, a 12-bit gradation-converted image 124.

[0076] According to this embodiment, by appropriately combining the image signal of the first period and the image signal of the second period, flicker can be suppressed even if a light source with a blinking cycle, such as an LED light source, is present in the imaging area. Furthermore, wide dynamic range (WDR) imaging can be performed by taking advantage of the characteristics of area-specific exposure. Furthermore, by appropriately shortening the length of the second period, the capacity of the memory unit required to hold the image signal of the first period can be reduced, contributing to the miniaturization and cost reduction of the imaging device.

[0077] [Embodiment 2] In the first embodiment, the output image was always a composite image of the image signals from the first and second periods. However, since the length of the exposure period for the first period is fixed regardless of the brightness of the subject, the output is likely to become saturated under conditions that are determined to be bright by area-specific exposure. In the second embodiment, to deal with such cases, it is determined whether or not there is a light source in the subject that may cause flicker, and the composite ratio of the first and second periods is adaptively changed. This embodiment will be described with reference to Figures 8 to 11. Note that the present embodiment is the same as the first embodiment described above, except for the points described below.

[0078] 8 is a block diagram showing an example of the configuration of the exposure correction unit 105 in embodiment 2. The exposure correction unit 105 has a line buffer 301, an addition ratio etc. determination unit 706, a flicker determination unit 701, an image addition unit 702, and a gradation expansion unit 304. In addition, the longest light emission period T of the LED light source to be flicker prevented and the number of divisions n to be set are input from a register 142 as register setting values ​​143, and are used in the addition ratio etc. determination unit 706. Components equivalent to those in embodiment 1 are assigned the same numbers as those in FIG. 3 of embodiment 1.

[0079] The operation of each component of the exposure correction unit 105 shown in Fig. 8 will be described with reference to the example in Fig. 4. Most of the operation of the exposure correction unit 105 in the second embodiment is the same as in the first embodiment, and differences will be mainly described here.

[0080] In FIG. 8, the exposure correction unit 105 first receives the exposure image 122 for the first period of frame 1 in FIG. 4. In this embodiment, too, the image signal of the exposure image 122 for the first period of a certain frame at a certain pixel position is referred to as the first image signal O1. The data of the first image signal O1 is first stored in the line buffer 301. After the second period has elapsed, the exposure image 122 for the second period of frame 1 is input. The image signal of the exposure image 122 for the second period of the same frame at this time is referred to as the second image signal O2. When the second image signal O2 is supplied as the exposure image 122, the flicker determination unit 701 reads the data of the first image signal O1 from the line buffer 301 as the delayed image 306. The flicker determination unit 701 then uses the conversion factor γ (704) determined by the addition ratio determination unit 706 to determine whether flicker has occurred at the pixel in question, and transmits the flicker determination result 703 to the image addition unit 702.

[0081] The operation of the flicker determination unit 701 will be described later. In the image addition unit 702, the data of the first image signal O1 read from the delayed image 306 and the data of the second image signal O2 supplied as the exposure image 122 are delayed for the time required to determine the flicker determination result 703 in the flicker determination unit 701. Similarly, the addition ratio k305 and the conversion ratio G705 provided from the addition ratio determination unit 706 are also delayed. After aligning the timings, the synthesized image signal O3 is determined using the flicker determination result 703. The operation of the image addition unit 702 will be described later.

[0082] FIG. 9 is a flowchart showing an example of a process for determining the flicker determination result 703 in the flicker determination unit 701 and the addition ratio determination unit 706. In S801, the conversion ratio γ (704) for determining flicker by comparing O1 and O2 is determined for each pixel in the addition ratio determination unit 706. The conversion ratio γ corrects the difference in exposure time.

[0083] The conversion ratio γ can be obtained by the following formula (12). γ = T1 / T2 ··· (12) Here, when λ1 ≧ T, substituting formulas (2) and (3) into formula (12), γ = T1 / T2 = (λ1 - T2) / T2 = λ1 / T2 - 1 = nλ1 / T - 1 ··· (13) It becomes.

[0084] On the other hand, when λ1 < T, substituting formulas (5) and (8) into formula (12), γ = T1 / T2 = T / T2 = T / (T / α) = α ··· (14) It becomes.

[0085] FIG. 10 is a table showing an example of the relationship between T1, T2, and λ1 and the exposure time ratio α, addition ratio k, conversion ratio γ, and conversion ratio G (described later) in this embodiment. In FIG. 10, the length of one frame period λ = 33.3333 ms, T = 11.1111 ms, and n = 4 are assumed.

[0086] For example, assuming that the shutter speed to be set as one frame period in area-by-area exposure is 1 / 30 seconds, λ1 at that time is 33.3333 ms. Since λ1≧T, according to Equation (13), γ=nλ1 / T -1=4×33.3333 / 11.1111-1=12-1=11 This shows that it becomes so.

[0087] Similarly, assuming that the shutter speed to be set as one frame period in area-by-area exposure is 1 / 960 seconds, λ1 at that time is 1.0417 ms. Since λ1<T, according to Equation (1) and Equation (14), γ=α=λ / λ1=32, which is obtained. Such calculations are performed by the addition ratio determination unit 706 and output to the flicker determination unit 701 as the conversion ratio γ(704).

[0088] Next, in S802, the flicker evaluation value D is determined for each pixel. The flicker evaluation value D is the difference between the image signal O1 corrected by the conversion ratio γ and the image signal O2, and evaluates the magnitude of this difference. The subsequent processes of S802 to S805 are performed by the flicker determination unit 701.

[0089] The flicker evaluation value D can be obtained by the following Equation (15) using the data of the first image signal O1, the data of the second image signal O2, and the conversion ratio γ determined by the addition ratio determination unit 706. D=O1×γ-O2···(15) Next, in S803, it is determined whether flicker has occurred for each pixel. Here, β is the threshold for determining whether flicker has occurred. If the absolute value of the flicker evaluation value D calculated in S802 is smaller than β, it is determined that flicker has not occurred, the result is Yes in S803, and the process proceeds to S804. Otherwise, it is determined that flicker has occurred, the result is No in S803, and the process proceeds to S805. In the second embodiment, β is a predetermined threshold value that is set inside the image adder 702. However, the method for setting β is not limited to this. For example, like the division number n, it is also possible to change and use the value as needed by setting it in the register 142 from the external controller 10.

[0090] In S804, when the flicker determination result is F, F=1 is set, meaning that no flicker has occurred. This value of F is transmitted to the image adder 702 as the flicker determination result 703. On the other hand, in S805, F=0, meaning that a flicker has occurred, is set as the value of F indicating the flicker determination result. In this case as well, the value of F is transmitted to the image adder 702 as the flicker determination result 703.

[0091] In S806, it is determined whether the process of determining the flicker evaluation value D has been completed for one frame's worth of pixels. If it has not been completed, that is, if the answer is No in S806, the processes from S801 onwards are repeated. If it has been completed, that is, if the answer is Yes in S806, this process is completed. The processes from S801 to S806 are performed in a pipelined manner, synchronously for each pixel, in the addition ratio etc. determination unit 706 and the flicker determination unit 701.

[0092] FIG. 11 is a flowchart showing an example of processing by an image addition unit 702 and a conversion ratio determination unit 706 that obtains a conversion ratio G705 and obtains a composite image signal O3. In S1001, the conversion ratio G is determined by the addition ratio determination unit 706. The conversion ratio G is a coefficient for converting the image signal value O2 obtained by area-by-area exposure control in the second half second period 402 into the pixel value of the frame when it is determined that there is no flicker (F = 1). That is, the conversion ratio G is a coefficient for converting the second image signal captured at the second exposure time T2 into the one captured at the appropriate exposure time λ1.

[0093] The conversion ratio G can be obtained by the following equations 16 and 17. When λ1≧T G = λ1 / T2 ···(16) When λ1<T G = λ / T ···(17) It becomes like this.

[0094] Referring to FIG. 10 again. FIG. 10 is a table showing an example of the relationship between T1, T2, and λ1 and the exposure time ratio α, addition ratio k, conversion rate γ, and conversion ratio G in the present embodiment. In FIG. 10, the length of one frame period λ = 33.3333 ms, T = 11.1111 ms, and n = 4 are set.

[0095] For example, assuming that the shutter speed to be set as one frame period in area-by-area exposure is 1 / 30 second, then λ1 at that time is 33.3333 ms. Since λ1≧T, according to equation (16), G = λ1 / T2 = 33.3333 / 2.7778 = 12 It shows that it becomes like this.

[0096] Similarly, assuming that the shutter speed to be set as one frame period in area-by-area exposure is 1 / 960 second, then λ1 at that time is 1.0417 ms. Since λ1<T, according to equation (17), G = λ / T = 33.3333 / 11.1111 = 3 Such calculations are performed for each pixel in the addition ratio etc. determining unit 706, and are output to the image addition unit 702 as the conversion ratio G (705).

[0097] Next, in S1002, the flicker determination result F is referenced for each pixel to determine the calculation method for the pixel value O3 to be output. If F=1 and no flicker has occurred (S1002-Yes), the process proceeds to S1003. On the other hand, if F=0 and flicker has occurred in the pixel in question (S1002-No), the process proceeds to S1004.

[0098] In S1003, the pixel value O3 to be output when no flicker occurs is determined using the following equation 18. Then, the process proceeds to S1005. O3 = G × O2 (18) In S1004, the pixel value O3 to be output when flickering has occurred is calculated using the equation shown in equation (10) described in the first embodiment. Then, the process proceeds to S1005. In S1005, it is determined whether processing of pixel values ​​O3 has been completed for one frame's worth of pixels. If not, that is, if the answer is No in S1005, the processes from S1001 onwards are repeated. If completed, that is, if the answer is Yes in S1005, this process is completed.

[0099] The composite image signal O3 of O1 and O2 generated in this manner is transmitted to the gradation expansion unit 304. Details of the subsequent processing are the same as in the first embodiment, so a description thereof will be omitted here. Note that although the present embodiment has been described using an example in which n=4, the value of n can also be 2 or 3. Fig. 12 is a table showing the relationship between T1, T2, α, k, γ, and G corresponding to each exposure condition (shutter speed) in the case of n=3, λ=33.3333 ms, and T=11.1111 ms.

[0100] FIG. 13 shows a table showing the relationship between T1, T2, α, k, γ, and G for each exposure condition (shutter speed) when n=2, λ=33.3333 ms, and T=10.0000 ms. In FIG. 13, the light emission period T is assumed to be 1 / 100 seconds, which is the longest light emission period of the light source for which flicker prevention is desired. Generally, the larger the value of n, the shorter the second period can be, so a larger value of n is advantageous from the perspective of the amount of storage required to hold the O1 image signal. However, if n is 5 or greater, the relationship shown in this embodiment may not hold, so it is recommended to use a value of 2, 3, or 4 for n.

[0101] According to the second embodiment, it is possible to obtain an appropriate image (WDR image) with an expanded dynamic range by determining whether or not there is a light source that generates flicker in the subject and adaptively changing the composition ratio of the first period and the second period. Furthermore, by appropriately shortening the length of the second period, it is possible to reduce the capacity of the memory unit required to hold the image signal of the first period, which contributes to miniaturization of the sensor element and cost reduction.

[0102] [Embodiment 3] In the first and second embodiments described above, the analog gain value 113 is set to 1 except when the shutter speed is 1 / 30 seconds. This is an example in which all 11 exposure time settings are possible, from 1 / 30 seconds to 1 / 30720 seconds, as shown in Fig. 7.

[0103] However, there are cases where it is not possible to use all of these shutter speeds. For example, this may be the case when it is not possible to wire the signal wiring for transmitting the shutter speed in the number appropriate for the type of shutter speed. In such cases, it is necessary to use analog gain to use equivalent exposure conditions as EV values. Such cases are explained using Figure 14.

[0104] In FIG. 14, available combinations of exposure time and analog gain for displaying EV values ​​are limited by white borders along with available shutter speeds in this embodiment. In this embodiment, the flash period T is assumed to be 11.1111 ms, the frame length λ is assumed to be 33.3333 ms, and the number of divisions n is assumed to be 4. This embodiment shows that only five exposure times are available: 1 / 30 sec, 1 / 60 sec, 1 / 480 sec, 1 / 3840 sec, and 1 / 30720 sec. For example, in FIG. 7, a combination of analog gain × 1 and an exposure time of 1 / 240 sec could be used to obtain an EV value of 6. In this embodiment, as shown in FIG. 14, it is necessary to use a combination of analog gain × 2 and an exposure time of 1 / 480 sec.

[0105] Furthermore, even within usable exposure times, the effect of noise becomes greater when the analog gain is large, so if the same EV can be achieved under different exposure conditions with a smaller analog gain, priority will be given to using the smaller analog gain.For this reason, combinations of analog gain and exposure time that are not given priority for use, even within usable exposure times, are indicated by hatching with diagonal lines in the table.

[0106] The output values ​​of each pixel captured at an available shutter speed and analog gain are calculated using the table in Fig. 14 as if the image was captured at a shutter speed with an analog gain of 1x1 and the same EV value. The formulas presented in the first and second embodiments are applied. For example, if an image is captured under an exposure condition of EV 6, which is a combination of an analog gain of 2x and an exposure time of 1 / 480 seconds, the image is considered to have been captured under a combination of an analog gain of 1x1 and an exposure time of 1 / 240 seconds, both of which have the same EV value of 6. The formulas presented in the first and second embodiments are then applied to determine values ​​such as γ and G. The conversion of these values ​​is performed by the addition ratio determination unit 706.

[0107] In the first and second embodiments, the exposure time λ1 to be set differs depending on its relationship with the longest light emission period T of the LED light source for which flicker prevention is to be performed. When λ1≧T, from Equation 2 T1 = λ1 - T2 ··· (2) However, when λ1 < T, from Equation 5 T1 = T ··· (5) Therefore, when actually converting the relationship between the exposure time and the available shutter speed, it is necessary to consider whether λ1 is greater than or less than T.

[0108] For example, in Figure 14, when the EV values are 7, 6, and 5, even when the analog gain is ×1, under the condition of 1 / 120 ms with the longest λ1 λ1 = 1 / 120 ms = 8.3333 ms < T Since this holds, from Equation 5 T1 = T = 11.1111 ms On the other hand, when the EV value is 4, when the analog gain is ×1 λ1 = 1 / 60 ms = 16.6667 ms ≧ T Since this holds, from Equation 2 T1 = λ1 - T2 = 11.6667 ms At this time, when trying to shoot with an analog gain of ×8 and a shutter speed of 1 / 480 when shooting with an EV value of 4 λ1 = 1 / 480 ms = 2.0833 ms < T1 So, the pixels will be exposed for T1 = 11.1111 ms.

[0109] On the other hand, since the shutter speed at an analog gain of ×1 for an EV value of 4 is 1 / 60 ms, the exposure time at this time is λ1 = 16.6667 ms, and since T1 > λ1, it can be seen from Figure 10 that T1 = 13.8889 ms. Thus, when the exposure time varies with the same EV value, the processing in the addition ratio determination unit 706 becomes complicated.

[0110] Therefore, in this embodiment, an example is shown in which the shutter speed is selected using λ1=1 / 90 ms, which is equivalent to T=11.1111 ms, as the threshold value. That is, the shutter speed is selected so that the shutter speed does not need to be converted across the threshold value when the exposure time λ1 at an analog gain of ×1 is smaller or larger than the threshold value.

[0111] In reality, when the exposure time λ1 at an analog gain of ×1 is smaller than the threshold value at λ1=1 / 120 ms and λ1=1 / 240 ms, shooting is performed by changing the analog gain using a shutter speed of λ1=1 / 480 ms to achieve EV values ​​of 5 and 6. On the other hand, when the exposure time λ1 at an analog gain of ×1 is larger than the threshold value at λ1=1 / 60 ms, a shutter speed is assigned, making it possible to set an analog gain of ×1.

[0112] According to the third embodiment, even when the number of settable shutter speeds is limited, it is possible to obtain an appropriate image as a WDR image by using an analog gain and equivalent exposure conditions as an EV value. Furthermore, by appropriately shortening the length of the second period, it is possible to reduce the capacity of the memory unit required to hold the image signal of the first period, which contributes to miniaturization of the sensor element and cost reduction.

[0113] [Embodiment 4: Application of imaging device to equipment] Hereinafter, a device 1000 will be described that includes a semiconductor device 1100 including a package 1020 on which a semiconductor chip 1110 including a semiconductor integrated circuit is mounted, as shown in FIG. 15. The semiconductor chip 1110 is housed in the package 1020 and mounted on the device 1000. In the configuration shown in FIG. 15, the semiconductor chip 1110 includes the imaging device according to the above-described embodiment. The semiconductor device 1100 can include a package 1020 that includes a base 1010 to which the semiconductor chip 1110 is fixed, and a light-transmitting member 1030 such as glass that faces the semiconductor chip 1110. The package 1020 can include bonding members such as wires and bumps that connect inner leads provided on the base 1010 to terminals such as pad electrodes provided on the semiconductor chip 1110.

[0114] The device 1000 may include at least one of an optical device 1040, a control device 1050, a processing device 1060, a display device 1070, a storage device 1080, and a mechanical device 1090. The optical device 1040 is, for example, a lens, a shutter, or a mirror. The control device 1050 controls a semiconductor chip 1110. The control device 1050 is, for example, a semiconductor device such as an ASIC.

[0115] The processing device 1060 processes an output signal from the imaging device included in the semiconductor chip 1110. The processing device 1060 is a semiconductor device such as a CPU or ASIC for configuring an AFE analog front end or a DFE digital front end. For example, an image may be generated based on an event signal. The display device 1070 is an EL display device or a liquid crystal display device that displays an information image obtained by the semiconductor chip 1110. The storage device 1080 is a magnetic device or a semiconductor device that stores the information image obtained by the semiconductor chip 1110. The storage device 1080 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0116] The mechanical device 1090 has a moving part or a propulsion part such as a motor or an engine. In the device 1000, the signal output from the semiconductor chip 1110 is displayed on the display device 1070, or transmitted to the outside by a communication device (not shown) included in the device 1000. For this purpose, the device 1000 may further include a storage device 1080 and a processing device 1060 in addition to the memory circuit and arithmetic circuit included in the semiconductor chip 1110. The mechanical device 1090 may be controlled based on the signal output from the semiconductor chip 1110.

[0117] The device 1000 is also suitable for electronic devices such as information terminals with imaging capabilities, such as smartphones, wearable devices, and cameras, such as interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras. The mechanical device 1090 in the camera can drive components of the optical device 1040 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 1090 in the camera can move the optical device 1040 for vibration isolation.

[0118] Furthermore, the device 1000 may be transportation equipment such as a vehicle, a ship, or an aircraft. The mechanical device 1090 in the transportation equipment may be used as a moving device. The device 1000 as transportation equipment is suitable for transporting semiconductor chips 1110 or for assisting and / or automating driving by using an imaging function. The processing device 1060 for assisting and / or automating driving can perform processing for operating the mechanical device 1090 as a moving device based on information obtained by the semiconductor chip 1110. Alternatively, the device 1000 may be a medical device such as an endoscope, a measuring device such as a distance measuring sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0119] The present invention can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that implements one or more functions.

[0120] (Other Embodiments) The disclosure of the present invention includes the following imaging devices, image processing circuits, imaging devices, and devices to which the imaging device is applied. (Item 1) An imaging control device that causes an image pickup device having a plurality of pixel blocks, each pixel block having a plurality of pixels, to capture a moving image, a holding unit that holds a light emission cycle of a light source that emits light periodically, and a control unit that controls a first period and a second period shorter than the light emission cycle following the first period in one frame period in the imaging of the moving image, wherein the control unit sets a first exposure time including a charge accumulation time longer than the light emission cycle in the first period and a second exposure time in the second period based on exposure conditions determined for each of the plurality of pixel blocks. The imaging control device is characterized by this. (Item 2) When the light emission cycle is T, the first exposure time is T1, the second exposure time is T2, the one frame period is λ, the appropriate exposure time set based on the exposure conditions with respect to the one frame period is λ1, α = λ / λ1, and n is an integer of 2 or more, the first exposure time T1 and the second exposure time T2 are When λ1≧(1 + 1 / n)×T, set T1 = λ1 - T2 and T2 = T / n, When (1 + 1 / n)×T>λ1≧T, set T1 = T and T2 = T / n When λ1<T, set T1 = T and T2 = T / α. The imaging control device according to Item 1, characterized by this. (Item 3) The imaging control device according to Item 2, characterized in that n is 2, 3, or 4. (Item 4) An image processing apparatus that processes an image signal captured by the image sensor controlled by the imaging control apparatus according to Item 1, wherein the image processing apparatus generates a composite image by weighted addition of a first image captured at the first exposure time and a second image captured at the second exposure time for each pixel at a predetermined ratio. (Item 5) The image processing apparatus according to Item 4, comprising a storage unit that holds the first image, and the storage unit holds the first image until the process of generating the composite image ends. (Item 6) The image processing apparatus according to Item 4 or 5, wherein the predetermined ratio is based on the exposure conditions. (Item 7) Let the emission cycle be T, the first exposure time be T1, the second exposure time be T2, the one-frame period be λ, the appropriate exposure time set based on the exposure conditions with respect to the one-frame period be λ1, α = λ / λ1, n be an integer of 2 or more, the image signal captured at the first exposure time be the first image signal O1, the image signal captured at the second exposure time be the second image signal O2, and when the composite image is O3, O3 = k×O1 + O2. The value k of the predetermined ratio is, when λ1≧(1 + 1 / n)×T, set k = 1, (1 + 1 / n)×T > λ1≧T, set k = λ1 / T - 1 / n, when λ1 < T, set k = ((λ / T) - 1) / α. The image processing apparatus according to any one of Items 4 to 6. (Item 8) The image processing apparatus according to any one of Items 4 to 7, wherein the difference between the value of a pixel at a predetermined position of the first image corrected by a conversion rate based on the exposure conditions and the pixel at the predetermined position of the second image is compared with a threshold value to determine the predetermined ratio. (Item 9) 9. The image processing device according to item 8, wherein when the difference between the signals is lower than the threshold, the predetermined ratio for the first image is set to 0. (Item 10) 10. The image processing device according to any one of items 4 to 9, wherein the composite image is corrected based on the exposure conditions. (Item 11) 11. The image processing device according to any one of items 4 to 10, wherein the exposure conditions include a gain for amplifying the image signal obtained from the imaging element. (Item 12) Item 12. The image processing device according to item 11, wherein the gain for the image signal captured during the first period is multiplied by 1. (Item 13) An imaging control method for causing an imaging element having a plurality of pixel blocks, each pixel block having a plurality of pixels, to capture a moving image, comprising: determining an exposure condition for each of the plurality of pixel blocks; a step of maintaining the light emission cycle of the light source that emits light periodically; an imaging control method comprising: a step of setting a first exposure time in a first period and a second period following the first period, the first period being shorter than the light emission period, based on the exposure conditions, and a step of setting a second exposure time in the second period, based on the exposure conditions, during one frame period of imaging the moving image. (Item 14) A program executed by a computer, causing the computer to execute each step of the imaging control method described in item 13. (Item 15) The imaging control device according to item 1, Item 4. The image processing device according to item 4, An imaging device comprising the imaging element. (Item 16) Item 15: The imaging device according to item 15; and a processing device that processes an output signal from the imaging device.

[0121] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0122] 100: Imaging device 102: Imaging sensor 103: Imaging element section 104: A / D conversion section 105: Exposure correction section 109: Exposure time control section 110: Gain control section 111: Exposure condition determination section 112: Exposure time 113: Analog gain value 122: Exposure image 301: Line buffer 302: Addition ratio determination section 303: Image addition section 304: Gradation expansion section 305: Addition ratio k 306: Delayed image

Claims

1. An imaging control device that causes an imaging element having a plurality of pixel blocks, each pixel block having a plurality of pixels, to capture a moving image, a storage unit that stores a light emission cycle of a light source that periodically emits light; and a control unit that controls a first period and a second period that follows the first period and is shorter than the light emission cycle, in one frame period of capturing the moving image, the control unit sets a first exposure time in the first period that includes a charge accumulation time equal to or longer than the light emission cycle, and sets a second exposure time in the second period, based on exposure conditions determined for each of the plurality of pixel blocks.

2. When the light emission period is T, the first exposure time is T1, the second exposure time is T2, the one frame period is λ, the appropriate exposure time set based on the exposure conditions for the one frame period is λ1, α=λ / λ1, and n is an integer of 2 or more, the first exposure time T1 and the second exposure time T2 are defined as follows: When λ1≧(1+1 / n)×T, set T1=λ1−T2, T2=T / n, When (1+1 / n)×T>λ1≧T, set T1=T and T2=T / n. When λ1<T, set T1=T and T2=T / α.

2. The imaging control device according to claim 1.

3. 3. The imaging control device according to claim 2, wherein n is 2, 3, or 4.

4. 2. An image processing device that processes an image signal captured by the image sensor controlled by the image capture control device according to claim 1, The image processing device is characterized in that it generates a composite image by weighting and adding a first image captured during the first exposure time and a second image captured during the second exposure time at a predetermined ratio for each pixel.

5. The image processing device according to claim 4 , further comprising a storage unit for storing the first image, the storage unit storing the first image until the process of generating the composite image is completed.

6. 5. The image processing apparatus according to claim 4, wherein the predetermined ratio is based on the exposure conditions.

7. Let T be the light emission period, T1 be the first exposure time, T2 be the second exposure time, λ be the one frame period, λ1 be the appropriate exposure time set based on the exposure conditions for the one frame period, α=λ / λ1, n be an integer of 2 or more, an image signal captured during the first exposure period be a first image signal O1, an image signal captured during the second exposure period be a second image signal O2, and the composite image be O3, then O3=k×O1+O2, and the value k of the predetermined ratio is If λ1≧(1+1 / n)×T, set k=1; When (1+1 / n)×T>λ1≧T, set k=λ1 / T−1 / n; 5. The image processing apparatus according to claim 4, wherein when λ1<T, k is set to ((λ / T)−1) / α.

8. 5. The image processing device according to claim 4, wherein the difference between the signal of a pixel at a predetermined position of the first image corrected by a conversion rate based on the exposure conditions and the signal of a pixel at the predetermined position of the second image is compared with a threshold value to determine the predetermined ratio.

9. 9. The image processing apparatus according to claim 8, wherein the predetermined ratio for the first image is set to 0 when the signal difference is lower than the threshold value.

10. 5. The image processing apparatus according to claim 4, wherein the composite image is corrected based on the exposure conditions.

11. 5. The image processing apparatus according to claim 4, wherein the exposure conditions include a gain for amplifying the image signal obtained from the image sensor.

12. 12. The image processing device according to claim 11, wherein the gain for the image signal captured during the first period is set to 1.

13. An imaging control method for causing an imaging element having a plurality of pixel blocks, each pixel block having a plurality of pixels, to capture a moving image, comprising: determining an exposure condition for each of the plurality of pixel blocks; a step of maintaining the light emission cycle of the light source that emits light periodically; an imaging control method comprising: a step of setting a first exposure time in a first period and a second period following the first period, the first period being shorter than the light emission period, based on the exposure conditions, the first exposure time including a charge accumulation time equal to or longer than the light emission period, and a step of setting a second exposure time in the second period, based on the exposure conditions, during one frame period of imaging the moving image.

14. A program executed by a computer, causing the computer to execute each step of the imaging control method according to claim 13.

15. The imaging control device according to claim 1 ; The image processing device according to claim 4; An imaging device comprising the imaging element.

16. The imaging device according to claim 15; and a processing device that processes an output signal from the imaging device.

Images