Imaging control apparatus, image processing apparatus, imaging apparatus, method for driving imaging element, device, and program

Abstract

This imaging control apparatus causes an imaging element having a plurality of pixel blocks each including a plurality of pixels to perform imaging of a moving image. The imaging control apparatus comprises a control unit that controls the imaging element so as to include a first imaging period in which a charge accumulation time is controlled on the basis of a first exposure condition in which one frame period in the moving image is determined for each of the plurality of pixel blocks, and a second imaging period including a charge accumulation time based on a preset second exposure condition.

Description

Imaging control device, image processing device, imaging device, imaging element driving method, device, and program

[0001] The present disclosure relates to an imaging control device, an image processing device, an imaging device, a driving method for an imaging element, a device, and a program.

[0002] In order to widen the dynamic range of an imaging device, a method of partially changing the exposure of an imaging sensor has been proposed. Patent Document 1 discloses dividing the entire light-receiving area of ​​an imaging sensor into multiple regions, and setting an exposure time (charge accumulation time in the sensor) and an amplification gain for an analog signal from the sensor for each region. Patent Document 2 describes a technique of setting the sensitivity and exposure for each region and lengthening the exposure time when capturing an image of a flickering light source.

[0003] JP 2021-129144 A JP 2023-14831 A

[0004] To address issues (such as LED flicker) caused by periodically emitting light sources, such as light-emitting diodes (LEDs), an exposure time longer than the LED's light emission period is required. When the exposure time is long, the sensitivity of the image sensor can be adjusted to prevent saturation of the image sensor. However, since there is a limit to the sensitivity adjustment range, the effects of LED flicker may occur. The present disclosure aims to provide a technology that enables wide dynamic range imaging by setting exposure conditions for each region while reducing the effects of flicker on imaging.

[0005] An imaging control device according to one aspect of the present disclosure is an imaging control device that causes an imaging element having a plurality of pixel blocks, each including a plurality of pixels, to capture a moving image, and is characterized by including a control unit that controls the imaging element so that one frame period of the moving image includes a first imaging period in which a charge accumulation time is controlled based on a first exposure condition determined for each of the plurality of pixel blocks, and a second imaging period that includes a charge accumulation time based on a predetermined second exposure condition.

[0006] According to the present disclosure, it is possible to provide a technology that can enable wide dynamic range imaging in which exposure conditions are set for each region while reducing the effect of flicker on imaging.

[0007] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0008] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and are used, together with the description, to explain the principles of the present invention.

[0023] Figure 1 is a diagram showing an example of the schematic configuration of an imaging device and its connection with an external controller. Figure 2 is a diagram explaining an imaging element. Figure 3 is a diagram showing an example of the configuration of an exposure correction unit. Figure 4 is a diagram showing the relationship between exposure time control and the light emission cycle of a light source included in an imaging target. Figure 5 is a table showing the relationship between exposure conditions, exposure time, and analog gain in area-specific exposure control. Figure 6 is a diagram showing the relationship between exposure time control and the light emission cycle of a light source included in an imaging target in embodiment 2. Figure 7 is a diagram showing an example of the configuration of an exposure correction unit in embodiment 3. Figure 8 is a flowchart showing the processing flow of a flicker determination unit in embodiment 3. Figure 9 is a diagram showing an example of the schematic configuration of an imaging device in embodiment 3 and its connection with an external controller. Figure 9 is a flowchart showing the processing flow of an image addition unit in embodiment 3. Figure 10 is a diagram showing the relationship between exposure time control and the light emission cycle of a light source included in an imaging target in embodiment 4. Figure 11 is a diagram showing an example of the schematic configuration of an imaging device in embodiment 4 and its connection with an external controller. Figure 12 is a diagram showing an example of the application of an imaging device to equipment.

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [Embodiment 1] Fig. 1 is a block diagram showing a schematic configuration example of an imaging device 100 to which an imaging control device according to this embodiment is applied, and the connection with an external controller 10. The imaging device 100 of this embodiment also includes various components that a typical imaging device has, but for simplicity of illustration and explanation, Fig. 1 shows only the main components according to this embodiment. Furthermore, the components described below are merely examples, and the functions of the components described below may be appropriately combined into one or separated. Alternatively, one component may also serve as another component. Furthermore, the components may be provided outside the imaging device 100, for example, in the controller 10.

[0011] The imaging device 100 according to this embodiment may include a synchronization control unit 101, an image sensor unit 103, an A / D conversion unit 104, and an exposure correction unit 105. The imaging device 100 may further include a tone conversion unit 106, an image output unit 108, an exposure time control unit 109, a gain control unit 110, an exposure condition calculation unit 111, and a control unit 150.

[0012] The synchronization control unit 101, the exposure time control unit 109, and the gain control unit 110 can each be considered an imaging control device that controls imaging. The imaging control device can include at least one of the synchronization control unit 101, the exposure time control unit 109, and the gain control unit 110. The exposure correction unit 105 and the gradation conversion unit 106 each function as an image processing device that performs image processing on the exposure image 122. The image processing device can include at least one of the exposure correction unit 105 and the gradation conversion unit 106. The imaging device 100 can also include a control unit 150 that controls the imaging device 100. The control unit 150 may be provided in each unit of the imaging device 100, or may control part of or the entire imaging device 100.

[0013] An exposure time 112 and an analog gain value 113 can be set in the imaging device 100. The imaging element unit 103 is equipped with a photoelectric conversion element that converts light into an electric charge and stores the electric charge. The imaging device 100 also has a serial input / output interface (SIO I / F) 141 and a memory or register 142 to reflect settings provided from an external controller 10. The imaging sensor 102 can include the imaging element unit 103 and an A / D conversion unit 104. In this embodiment, the imaging device 100 can be connected to the external controller 10 via a serial communication line 11 and an output signal line 12.

[0014] An overview of each component of the imaging device 100 will be described, starting with the imaging element unit 103. The imaging element unit 103 has an imaging area (light-receiving area). The imaging area is divided into multiple areas called pixel blocks. The imaging element unit 103 can be driven in pixel block (area) units, and has the function of performing exposure operations (accumulation of charge) with different exposure times for each area. Here, the exposure time corresponds to the charge accumulation time during which the photoelectric conversion element accumulates charge. Furthermore, the exposure operation corresponds to accumulating charge.

[0015] Pixel blocks will be described later with reference to FIG. 2. A pixel block includes multiple pixels. Multiple pixel blocks are arranged in the image sensor unit 103. In this embodiment, the image sensor unit 103 sets a charge accumulation time for accumulating charge for each region of each pixel block using an exposure control signal 117 supplied from the exposure time control unit 109. The image sensor performs exposure for the charge accumulation time set for each region. The exposure time control unit 109 controls the image sensor. Note that this charge accumulation time can be called exposure time, as it is a time required for image capture.

[0016] The exposure control signal 117 is a signal for setting an exposure time for each region of the image sensor unit 103. The image sensor unit 103 reads out the charge accumulated in each pixel during the exposure time set for each region by the exposure control signal 117 as a pixel potential 118 and outputs it to the A / D conversion unit 104.

[0017] The A / D conversion unit 104 performs analog-to-digital conversion on the pixel potentials 118 read out from the image sensor unit 103, converting them into digital values. In this embodiment, an analog gain 121 corresponding to each region is set in the A / D conversion unit 104 by the gain control unit 110. The analog gain 121 is an amplification gain value for the pixel potentials 118. The A / D conversion unit 104 amplifies the pixel potentials 118 output from the image sensor unit 103 by the analog gain 121 determined for each region, and then performs analog-to-digital conversion to convert them into digital values. Hereinafter, an image formed of a digital signal that has been amplified by the analog gain 121 for each region by the A / D conversion unit 104 and then analog-to-digital converted is referred to as an exposure image 122. The exposure image 122 output from the A / D conversion unit 104 is sent to the exposure condition calculation unit 111 and the exposure correction unit 105.

[0018] The exposure condition calculation unit 111 calculates and updates the exposure time 112 and analog gain value 113 for each region based on the exposure image 122 so as to optimize imaging conditions. For example, the exposure condition calculation unit 111 obtains a histogram of pixel values ​​for each pixel block based on the luminance distribution of the exposure image 122. If the pixel values ​​are distributed toward the bright area side, the exposure condition calculation unit 111 changes (updates) the exposure time 112 and analog gain value 113 for that pixel block (region) to set values ​​that will result in a darker image. If the pixel values ​​are distributed toward the dark area side, the exposure condition calculation unit 111 changes (updates) the exposure time 112 and analog gain value 113 for that pixel block (region) to set values ​​that will result in a brighter image. The value of the exposure time 112 for each region is then sent to the exposure time control unit 109 and the exposure correction unit 105. The analog gain value 113 for each region is sent to the gain control unit 110 and the exposure correction unit 105.

[0019] The synchronization control unit 101 generates a synchronized exposure time output pulse 120 and a gain output pulse 114. The synchronization control unit 101 outputs the generated exposure time output pulse 120 to the exposure time control unit 109. The synchronization control unit 101 outputs the generated gain output pulse 114 to the gain control unit 110. In this way, the synchronization control unit 101 performs control to synchronize the processing of the exposure time control unit 109 and the processing of the gain control unit 110.

[0020] The exposure time output pulse 120 is a signal for controlling the timing at which the exposure time control unit 109 outputs the exposure control signal 117 to the image sensor unit 103. The exposure time control unit 109 outputs the exposure control signal 117 to the image sensor unit 103 based on the exposure time output pulse 120, thereby changing the exposure time for each arbitrary pixel block of the image sensor unit 103.

[0021] Furthermore, the gain output pulse 114 is a signal for controlling the timing at which the gain control unit 110 outputs the analog gain 121 to the A / D conversion unit 104. The gain control unit 110 outputs the analog gain 121 to the A / D conversion unit 104 based on the gain output pulse 114, thereby changing the gain for amplifying the pixel potential of the signal obtained from each pixel for each pixel block. As described above, in this embodiment, the synchronization control unit 101 controls the operation of the exposure time control unit 109 and the gain control unit 110 in synchronization with each other, thereby making it possible to output an exposure image 122 in which the exposure time and analog gain are appropriately changed for each pixel block of the image sensor unit 103.

[0022] The exposure time control unit 109 generates an exposure control signal 117 for each region based on the exposure time output pulse 120 and the value of the exposure time 112 for each region, and outputs it to the image sensor unit 103. As a result, the exposure time according to the exposure time 112 for each region is set for the image sensor unit 103 at an appropriate timing.

[0023] The gain control unit 110 outputs the analog gain value 113 for each region to the A / D conversion unit 104 in synchronization with the timing of the gain output pulse 114 as an analog gain 121 for each region corresponding to the pixel potential 118 for each region of the image sensor unit 103. As a result, the A / D conversion unit 104 performs analog-to-digital conversion after amplifying the pixel potential 118 for each region by the analog gain 121 for each region corresponding to the region. The data that has undergone analog-to-digital conversion is sent to the exposure correction unit 105 and the exposure condition calculation unit 111 as an exposure image 122 for each region.

[0024] The exposure correction unit 105 accumulates exposure images 122 captured under different exposure conditions within the same frame for the exposure images 122 for each region sent from the A / D conversion unit 104, performs necessary processing, and then performs addition processing on each pixel data. The added image is then subjected to gradation expansion processing based on the exposure time 112 and analog gain value 113 to generate a gradation-expanded image 123. The exposure correction unit 105 generates a gradation-expanded image 123 represented by a 23-bit number from the exposure images 122 for each region represented by a 10-bit number, for example. The detailed operation of the exposure correction unit 105 will be described later. The generated gradation-expanded image 123 is then sent to the gradation conversion unit 106.

[0025] The gradation conversion unit 106 performs gradation conversion on the gradation-extended image 123 and outputs the gradation-converted image 124 to the image output unit 108. In this embodiment, the gradation conversion is a process of generating, for example, a 12-bit gradation-converted image 124 from the 23-bit gradation-extended image 123 by gamma conversion. Note that the gradation conversion process in this embodiment is performed to reduce the data rate in subsequent processing. In this embodiment, the bit lengths of the exposed image 122 and the gradation-converted image 124 are 10 bits and 12 bits, respectively, but these bit lengths are merely examples and are not limited to these.

[0026] The image output unit 108 outputs the gradation-converted image 124 to a downstream configuration of the imaging device 100 or to the outside. In this embodiment, a controller 10 is connected as a processing module that receives image data from the imaging device 100. Here, an LVDS signal line with 16 data channels can be used as the output signal line 12 connecting the image output unit 108 and the controller 10. However, the type and data channel width of this output signal line 12 are not limited by this embodiment.

[0027] The controller 10 is also connected to a serial input / output interface (SIOI / F) 141 of the image capturing apparatus 100 via a serial communication line 11. The SIOI / F 141 is connected to a register 142, and the controller 10 can set necessary information in the register 142 inside the image capturing apparatus 100 via the SIOI / F 141. The information set in the register 142 is transmitted to an exposure condition calculation unit 111, making it possible to control the calculation of the exposure conditions.

[0028] FIG. 2 is a diagram illustrating an example configuration of the image sensor unit 103. The imaging area of ​​the image sensor unit 103 includes multiple pixel blocks 201. Each pixel block 201 includes multiple pixels 202. In this embodiment, the imaging area of ​​the image sensor unit 103 has 2,000 pixels in the width 206 direction (horizontal line direction) and 1,000 pixels in the height 205 direction (i.e., 1,000 horizontal lines in the vertical direction). Also, each pixel block 201 has 100 pixels in the width 204 direction (horizontal line direction) and 100 pixels in the height 203 direction (100 horizontal lines in the vertical direction). In this case, the number of pixel blocks 201 in the imaging area of ​​the image sensor unit 103 is 20 in the horizontal direction and 10 in the vertical direction. Note that these numbers of pixels and lines are merely examples and are not limiting.

[0029] Furthermore, the pixel block numbers [0,0] to [19,9] written within each pixel block 201 in Figure 2 represent the position of each pixel block 201 within the imaging area, and the values ​​in brackets [ ] represent the horizontal and vertical indexes of each pixel block within the imaging area. In Figure 2, for example, the pixel block 201 located in the upper right corner of the imaging element unit 103 is pixel block [19,0]. Furthermore, a set of pixel blocks represented by the same vertical index will be called a block row.

[0030] That is, block row N consists of pixel blocks [0,N] to [19,N]. For example, block row 5 consists of pixel blocks [0,5] to [19,5]. Note that the respective sizes (number of pixels in the vertical and horizontal directions) of the image sensor unit 103 and the pixel blocks 201 are not limited to those described above. Furthermore, the shape and aspect ratio of the pixel 202 are also not limited; for example, it may be rectangular rather than square. Furthermore, the pixel block 201 may consist of only one pixel 202. Furthermore, in this embodiment, the exposure time and analog gain are controllable for each pixel block 201.

[0031] Here, the exposure time corresponds to the time it takes for charge to accumulate in the pixels (light receiving elements) of the image sensor unit 103 during image capture. Therefore, for example, assuming that the amount of light incident on the image sensor unit 103 is the same and the pixels are not saturated, the longer the exposure time, the higher the pixel potential 118, and the brighter the image can be captured. In other words, if the amount of incident light is the same and pixel saturation is not taken into consideration, comparing an exposure time of 1 / 480 seconds with an exposure time of 1 / 30 seconds, for example, the brighter the image can be captured with an exposure time of 1 / 30 seconds.

[0032] The analog gain represents the gain by which pixel potential 118 is amplified before digital conversion in A / D conversion unit 104 during image capture. Therefore, the larger the analog gain value, the larger the input to A / D conversion unit 104, and therefore the larger the digital value output from A / D conversion unit 104.

[0033] 1, the configuration and driving method of the imaging device 100 of this embodiment will be described. The imaging element unit 103 captures an image while controlling the exposure time for each pixel block 201 based on the exposure control signal 117. The imaging element unit 103 then outputs a pixel potential 118 corresponding to the charge accumulated in each pixel.

[0034] The A / D converter 104 amplifies the pixel potential 118 output from the image sensor unit 103 by an analog gain 121 set corresponding to each pixel block of the image sensor unit 103, then performs digital conversion and outputs an exposure image 122. In this embodiment, the exposure image 122 is a 10-bit digital value. The analog gain 121 can take four gain values, for example, ×1, ×2, ×4, and ×8.

[0035] The exposure correction unit 105 performs gradation expansion processing based on the exposure time 112 and analog gain value 113 on the exposure image 122 for each region input from the A / D conversion unit 104, and outputs a gradation-expanded image 123. The exposure correction unit 105 recognizes the conditions under which the exposure image 122 for each region was captured based on the exposure time 112 for each region and the analog gain value 113 for each region. Then, the exposure correction unit 105 corrects the exposure image 122 for each region based on the conditions under which the exposure image 122 for each region was captured.

[0036] The exposure correction unit 105 performs gradation expansion processing based on the exposure time 112 and analog gain value 113 on the exposure image 122 for each region sent from the A / D conversion unit 104 to generate a gradation-expanded image 123. For example, the exposure correction unit 105 recognizes the conditions under which the input exposure image 122 for each region was captured based on the exposure time 112 for each region and the analog gain value 113 for each region, and corrects the exposure image 122 for each region in accordance with those conditions.

[0037] Furthermore, the exposure correction unit 105 performs gradation expansion processing on the exposure image 122 for each region, which is expressed by, for example, 10 bits, to generate a gradation-expanded image 123, which is expressed by 23 bits. The generated gradation-expanded image 123 is then sent to the gradation conversion unit 106. For example, the exposure correction unit 105 recognizes the conditions under which the input exposure image 122 for each region was captured based on the exposure time 112 for each region and the analog gain value 113 for each region, and corrects the exposure image 122 for each region in accordance with those conditions.

[0038] Next, the operation of the exposure correction unit 105 will be described. Fig. 3 is a block diagram showing an example configuration of the exposure correction unit 105. The exposure correction unit 105 has a line buffer 301, an addition ratio calculation unit 302, an image addition unit 303, and a gradation expansion unit 304. The line buffer 301 can be used to delay data to match the timing when images are later synthesized.

[0039] The operation of each component of the exposure correction unit shown in Fig. 3 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the relationship between exposure time control and the light emission period of an LED, which is the light emission source of the imaged object, in the case where a light emitting diode (LED) is assumed to be the periodic light emission source in this embodiment.

[0040] In this embodiment, the image output frame rate when the imaging device 100 captures video is 30 frames per second. In this case, the frame period of the video is 1 / 30 second, and one frame period is approximately 33.3 ms (hereinafter referred to as 33.3 ms). Furthermore, the LED included in the imaging target emits light periodically at a frequency of 90 Hz and a duty cycle of 50%. For explanatory purposes, FIG. 4 shows frames 1 and 2, which are consecutive in time. The imaging period of frame 1 is divided into 402-1 and 401-1. 402-1 has a length of period 2, which is a predetermined period set equal to or longer than the LED light emission period. In this embodiment, the predetermined period is the same length as 1 / 90 second of the LED light emission period (approximately 11.1 ms, hereinafter referred to as 11.1 ms). 404-1 is the exposure time of the image sensor during period 2. During period 2, the exposure time of the image sensor is always set to the entire period 2.

[0041] As can be seen from FIG. 4 , period 2 is set to be equal to or longer than the LED light emission cycle, and therefore includes the period during which the LED is emitting light (the convex portion of the LED light emission in FIG. 4 ). Period 1 of 401-1 is the imaging period of frame 1 excluding 402-1. Period 1 of 401-1 is the period during which per-area exposure control is performed according to the brightness of the object being imaged. 403-1 is the exposure time for 401-1 set by per-area exposure control, and in this embodiment, it is set to half the time of 401-1. Similar control is performed for frame 2, which follows frame 1. In frame 2, exposure time 403-2 for period 1 is set to 1 / 8 of the time of 401-2 as a result of per-area exposure control. Therefore, in FIG. 4 , the LED light emission period during period 403-2 is not captured by the image sensor. If the LED light emission period is not captured by the image sensor, flickering due to flashing may occur in the captured video.

[0042] The exposure times of the image sensor during periods 1 and 2 are controlled by the exposure time control unit 109 based on the exposure time 112 determined by the exposure condition calculation unit 111 in FIG. 1 . The exposure condition calculation unit 111 can use the length of period 2, which is set in register 142 by the external controller 10 as the light emission cycle of the LED for which flicker prevention is to be performed. The exposure time during period 1 can be calculated based on the exposure image 122 during period 1 of the previous frame, in accordance with an exposure condition calculation algorithm for area-specific exposure. The length of period 2 can be determined based on the destination of the image capture device to be used as a product, and can be set in advance as data to be read by the external controller 10. Alternatively, the destination can be determined using location information such as GPS, and the external controller 10 can set the length of period 2 in register 142 by referring to a table corresponding to the location information when the image capture device is started up.

[0043] 3, an exposure image 122 in period 2 of frame 1 in FIG. 4 is first input to the exposure correction unit 105. Here, image data of the exposure image 122 in period 2 of a certain frame at a certain pixel position is O ２ After the period 2 has elapsed, O ２The data is first stored in the line buffer 301. After the time of period 1 has elapsed, the exposure image 122 of period 1 of frame 1 is input. At this time, the image data of the exposure image 122 of period 1 of the same frame 1 is stored in O １ Let us call it this.

[0044] O １ is supplied to the exposure correction unit 105 as the exposure image 122, the image addition unit 303 ２ The data is read out from the line buffer 301 as a delayed image 306. At this time, the data retention period in the line buffer 301 is the length of period 1. Then, O is calculated based on the addition ratio k 305 calculated by the addition ratio calculation unit 302. １ and O ２ The composite output O is a composite image signal ３ is calculated based on the following formula (1): １ and image data O ２ and are weighted and added based on the respective imaging conditions to perform correction.

[0045] O ３ =O １ + k × O ２ ... (Equation 1) Next, a method for calculating the addition ratio k305 of the addition ratio calculation unit 302 will be described with reference to FIG. 5 . FIG. 5 is a table showing the relationship between exposure conditions, exposure time, and analog gain in area-specific exposure control. This table is applied when the frame rate is 30 fps and shows the relationship between exposure time and analog gain for the entire frame period. In FIG. 5 , the number at the intersection of the horizontal analog gain and exposure time in the table is referred to as the EV value. The EV value represents the ratio of brightness of an object to be captured based on differences in exposure conditions, expressed as a power of 2, when the pixel signal level (pixel value) obtained by capturing is the same. If the brightness of an object captured with an analog gain × 8 and an exposure time of 1 / 30 seconds is L0, the brightness LE of the object to be captured when the same pixel value is obtained by capturing an image with an EV value E can be expressed by the following Equation 2.

[0046] LE = L0 x 2^E (Equation 2) For example, when the analog gain is 1 and the exposure time is 1 / 30 seconds, the EV value is 3. Similarly, when the analog gain is 1 and the exposure time is 1 / 60 seconds, the EV value is 4. In this case, if the pixel values ​​obtained by imaging are the same, this indicates that the subject captured under EV value 4 conditions is twice as bright as the subject captured under EV value 3 conditions.

[0047] In the case of area-specific exposure, a table such as that shown in FIG. 5 can be used to estimate the brightness of the object to be imaged in that area from the pixel values ​​captured in the previous frame and the exposure conditions used for capturing the image at that time, and the exposure conditions to be used for capturing the next frame can be calculated.

[0048] Here, the exposure conditions used in period 402-1 in Figure 4 are analog gain x1, and the exposure time is fixed for the entire period 2. If we consider that these exposure conditions are applied to the entire frame, the exposure time will be 1 / 30 seconds for the entire frame, and the EV value in this case is 3 from Figure 5. On the other hand, the exposure conditions used in 401-1 are exposure times half that of period 1. Similarly, if we consider that these exposure conditions are applied to the entire frame, the exposure time will be 1 / 60 seconds, which is half the length of the entire frame. If the analog gain is x4 at this time, the EV value at this time will be 2 from Figure 5.

[0049] This means that 402-1 was captured with an EV value of 3 under conditions where it was determined that an EV value of 2 was appropriate for the area-specific exposure. The gradation expansion unit 304, which will be described later, converts the brightness of the pixel value based on the EV value of 2 in the area-specific exposure conditions. Therefore, the O captured with an EV value of 3 ２ The value is converted to the value equivalent to when the image is taken at EV 2, and then １ By adding these, the pixel value for one frame period as a whole can be obtained. The coefficient for this conversion is k. k can be calculated using the following equation 3.

[0050] k = 2^ (EV value of period 2) ÷ 2^ (EV value of period 1) = 2^ (EV value of period 2 - EV value of period 1) (Equation 3) From Equation 3, the conversion coefficient k1 of frame 1 in this embodiment is calculated as follows: k1 = 2^ (3 - 2) = 2^1 = 2

[0051] Similarly, consider frame 2. The exposure time 403-2 of frame 2 is 1 / 8 the length of period 1. This corresponds to an exposure time of 1 / 240 in Figure 5. If the analog gain at this time is x1, the EV value for exposure time 401-2 is 6. The EV value for 402-2 is 3, just like 402-1. This is because the exposure conditions for period 2 are set uniformly under the same conditions. The conversion coefficient k2 for frame 2 at this time is calculated as k2 = 2^(3-6) = 2^(-3) = 1 / 8.

[0052] In FIG. 3, the image adder 303 calculates O based on Equation 1. １ and O ２ The composite output of ３ is transmitted to the gradation expansion unit 304. The gradation expansion unit 304 calculates the EV value shown in Fig. 5 from the exposure time 112 and analog gain value 113 of the corresponding pixel block. If the EV value at that time is E, the pixel value after gradation expansion can be calculated by Equation 2.

[0053] For example, the exposure conditions for period 1 of frame 1 shown in Figure 4 are, as mentioned above, when the length of period 1 is converted to the entire frame 1, the exposure time is equivalent to 1 / 60 seconds and the analog gain is x4. Since the EV value at this time is 2, the composite output after the addition process of frame 1 is O ３１ , the pixel value after gradation expansion is O ４１ Then, O ４１ =O ３１ ×2^2=4×O ３１ This becomes:

[0054] Similarly, the exposure conditions for period 1 of frame 2 shown in Figure 4 are, as mentioned above, when the length of period 1 is converted to the entire frame 2, the exposure time is equivalent to 1 / 240 seconds and the analog gain is x1. Since the EV value at this time is 6, the composite output after addition processing of frame 2 is O ３２ , the pixel value after gradation expansion is O ４２ Then, O ４２ =O ３２ ×2^6=64×O ３２ This becomes:

[0055] At this time, O ３２If the bit width of the original exposure image is 10 bits, this calculation ４２ is a 16-bit number. In this embodiment, it can be seen from FIG. 5 that the EV value can be up to 13. In this case, if the exposure correction unit 105 performs gradation expansion, the bit width of the data increases. In this embodiment, the gradation-expanded image 123 can be up to 23 bits. Thereafter, as described above, the gradation conversion unit 106 generates, for example, a 12-bit gradation-converted image 124 by gamma conversion from the 23-bit gradation-expanded image 123.

[0056] According to this embodiment, image processing is performed to appropriately combine image data from period 2 and image data from period 1 within one frame period. This makes it possible to perform wide dynamic range (WDR) imaging by taking advantage of the characteristics of area-specific exposure while suppressing flicker, even when a light source with a blinking cycle, such as an LED, is present within the imaging range. For example, as shown in frame 2 of Figure 3, even though the LED light emission period cannot be captured during period 1 when area-specific exposure is performed, the LED light emission is captured during period 2. Therefore, the LED light emission is captured in the image of frame 2, which is a combination of the outputs from periods 1 and 2, and therefore it is possible to prevent the image from appearing as if the LED light source is off.

[0057] [Embodiment 2] In the control of exposure time within one frame period in embodiment 1, period 2, which is set to be equal to or longer than the LED light emission cycle, is in the first half of the frame, and period 1, in which area-specific exposure control is performed according to the brightness of the object being imaged, is in the second half of the frame, as shown in Fig. 4. However, when period 1 comes after period 2 within one frame period, if period 2 is shorter than period 1 as shown in Fig. 4, the period for which data from period 2 must be held in line buffer 301 in Fig. 3 becomes longer, and the amount of memory required for line buffer 301 increases.

[0058] In the second embodiment, a method for achieving the same effect as in the first embodiment while reducing the memory capacity to be held by the line buffer 301 when the period 2 is shorter than the period 1 will be described with reference to Fig. 6. Note that the second embodiment is the same as the first embodiment described above except for the points described below.

[0059] Fig. 6 is a diagram showing the relationship between exposure time control and the LED light emission cycle of the image capture target in embodiment 2. Unlike Fig. 4, in Fig. 6, period 2, which is set to be equal to or longer than the LED light emission cycle, is in the latter half of the frame, and period 1, in which area-specific exposure control is performed according to the brightness of the image capture target, starts from the beginning of the frame.

[0060] In FIG. 6, the imaging period of frame 1 is divided into 601-1 and 602-1. 602-1 has a length of period 2 that is determined to be equal to or longer than the LED light emission cycle. In this example, period 2 is set to 11.1 ms, as in embodiment 1. 604-1 is the exposure time of the imaging element in period 2. In period 2, the exposure time of the imaging element is always set to the entire period 2. As can be seen from FIG. 6, period 2 is determined to be equal to or longer than the LED light emission cycle, and therefore it can always include the period during which the LED is emitting light (the convex portion of the LED light emission in FIG. 6). Period 1 of 601-1 is the period of the imaging period of frame 1 excluding 602-1.

[0061] Period 1 of 601-1 is a period during which per-area exposure control is performed according to the brightness of the object being imaged. 603-1 is the exposure time for 601-1 set by per-area exposure control, and in this embodiment, it is set to half the time of 601-1. As described above, in embodiment 2, period 1 601-1 exists in the first half of the frame, and period 2 602-1 exists after that. Similar control is performed for frame 2. In this example, it is assumed that exposure time 603-2 for period 1 in frame 2 is set to 1 / 8 of the time of 601-2 as a result of per-area exposure control. Therefore, as shown in FIG. 6, during period 601-2, the LED light emission period is not captured by the image sensor.

[0062] As in the first embodiment, image data of an exposure image 122 in a certain frame during period 2 at a certain pixel position is O ２ In addition, the image data of the exposure image 122 in the period 1 of the same frame will be called O １ In the second embodiment, as can be seen from FIG. 6, the exposed image 122 in FIG. 3 is first referred to as O １Therefore, in FIG. 3, the line buffer 301 first receives the data O １ is input and stored. １ The data of period 2 is held in the line buffer, and the data of period 2 is ２ When input, the composite output O ３ is calculated based on the above-mentioned formula 1. The subsequent operations are the same as those in the first embodiment.

[0063] According to the second embodiment, the data O in the line buffer 301 １ The retention period of period 2 is the length of period 2. In the first embodiment, the retention period is the length of period 1. If period 2 is shorter than period 1, the amount of data to be retained in the line buffer 301 can be reduced by the difference in length between period 2 and period 1, and it is possible to obtain the same effect as in the first embodiment with a smaller memory amount.

[0064] [Embodiment 3] In embodiment 1, the output image signal was always a composite image signal of periods 1 and 2, but because period 2 is determined regardless of the brightness of the subject being imaged, its output is likely to saturate under conditions that are determined to be bright by area-specific exposure. In embodiment 3, an example will be described with reference to Figures 7 to 10 in which, in response to such a case, it is determined whether or not the subject being imaged has a light source that may cause flicker, and the composite ratio of periods 1 and 2 is adaptively changed. Note that this embodiment is the same as embodiment 1 described above, except for the points described below.

[0065] Fig. 7 is a block diagram showing an example of the configuration of the exposure correction unit 105 in the third embodiment. The exposure correction unit 105 has a line buffer 301, an addition ratio calculation unit 302, a flicker determination unit 701, an image addition unit 702, and a gradation expansion unit 304. Also, as shown in Fig. 9, period information 704 is input from the register 142 and is used in the flicker determination unit 701 and the image addition unit 702. Components equivalent to those in the first embodiment are assigned the same numbers as those in Fig. 3 of the first embodiment.

[0066] The operation of each component of the exposure correction unit 105 shown in Fig. 7 will be described with reference to the example of the imaging period shown in Fig. 4. Much of the operation of the exposure correction unit 105 in the third embodiment is the same as in the first embodiment, and only differences will be described here.

[0067] 7, the exposure image 122 of the period 2 of the frame 1 in FIG. 4 is first input to the exposure correction unit 105. In this embodiment, too, the image data of the exposure image 122 of the period 2 of a certain frame at a certain pixel position is O ２ I will call it O. ２ The data is first stored in the line buffer 301. After the time of period 1 has elapsed, the exposure image 122 of period 1 of frame 1 is input. At this time, the image data of the exposure image 122 of period 1 of the same frame is stored in O １ Let's call it O. １ is supplied as the exposure image 122, the flicker determination unit 701 ２ The data is read from the line buffer 301 as a delayed image 306. Then, it is determined whether flicker has occurred in the pixel based on the addition ratio k 305 calculated by the addition ratio calculation unit 302, and a flicker determination result 703 is transmitted to the image addition unit 702. The operation of the flicker determination unit 701 will be described later.

[0068] The image adder 702 delays the image by the time required for the flicker determination unit 701 to calculate the flicker determination result 703, and reads out the delayed image 306. ２ data and O provided as exposure image 122 １ The data and the addition ratio k 305 are then delayed. Then, the flicker determination result 703 is used to calculate the composite output O ３ The operation of the image adder 702 will be described later.

[0069] 8 is a flowchart showing an example of processing for calculating the flicker determination result 703 in the flicker determination unit 701. In S801, a period ratio α, which is the ratio between period 1 and period 2, is calculated. The ratio between period 1 and period 2 is calculated based on period information 704 input from the register 142. The period ratio α can be found by the following equation 4.

[0070] α=(length of period 1)÷(length of period 2) (Equation 4) In the third embodiment, in the example of Fig. 4, the length of period 1 is 22.2 ms and the length of period 2 is 11.1 ms, so the period ratio α = 2. The calculation of the period ratio α in S801 is performed initially for each frame.

[0071] The image data O is calculated using the period ratio α and the addition ratio k calculated as above. １ and image data O ２ The values ​​of the image data O are corrected. １ The first corrected signal obtained by correcting the image data O ２ In step S802, a flicker evaluation value D is calculated for each pixel using the second corrected signal obtained by correcting the image data O. １ and image data O ２ Using the addition ratio k and the period ratio α calculated by the addition ratio calculation unit 302, it can be calculated by the following equation 5: D=O １ ÷k-α×O ２ ...(Formula 5)

[0072] According to Equation 5, the image data O １ A first corrected signal obtained by correcting the image signal O ２ The difference between this signal and the second corrected signal corrected by the period ratio α is calculated as the flicker evaluation value D. Next, in S803, it is determined whether flicker is occurring for each pixel. Here, the threshold value for determining whether flicker is occurring is set to β. If the absolute value of the flicker evaluation value D calculated in S802 is smaller than β, it is determined that flicker is not occurring (Yes in S803) and the process proceeds to S804. In all other cases, it is determined that flicker is occurring (No in S803) and the process proceeds to S805.

[0073] In the third embodiment, β is a predetermined threshold value that is set in the image adder 702. However, the method for setting β is not limited to this, and for example, similar to the period information 704, it is also possible to change the value as needed and use it by setting it in the register 142 from the external controller 10. D is compared with the threshold value β to determine whether D is smaller than the threshold value β.

[0074] In S804, when the flicker determination result is F, F=1 is set, meaning that no flicker has occurred. This value of F is transmitted to the image adder 702 as the flicker determination result 703. On the other hand, in S805, F=0, meaning that a flicker has occurred, is set as the value of F indicating the flicker determination result. In this case as well, the value of F is transmitted to the image adder 702 as the flicker determination result 703.

[0075] In S806, it is determined whether the calculation process of the flicker evaluation value D has been completed for one frame's worth of pixels. If it has not been completed (No in S806), the process from S802 onwards is repeated. If it has been completed (Yes in S806), this process is completed.

[0076] 10 is a flowchart showing an example of image addition processing in the image adder 702 depending on whether or not flicker occurs. In S1001, a conversion ratio G to be used in S1003 is calculated. The conversion ratio G is the ratio of the output value O obtained by the area-specific exposure control when it is determined that no flicker occurs. １ The conversion ratio G can be calculated using the lengths of Period 1 and Period 2 based on the period information 704 input from the register 142, using the following equation 6:

[0077] G = {(length of period 1) + (length of period 2)} ÷ (length of period 1) (Equation 6) In this embodiment 3, as shown in Figure 4, the length of period 1 is 22.2 ms and the length of period 2 is 11.1 ms, so G = (22.2 + 11.1) ÷ 22.2 = 1.5. The calculation of the conversion ratio G in S1001 may be performed initially for each frame.

[0078] Next, in step S1002, the flicker determination result F is referenced for each pixel, and the pixel value O to be output is calculated. ３ If F=1 and no flicker has occurred (Yes in S1002), the process proceeds to S1003. On the other hand, if F=0 and flicker has occurred in the pixel (No in S1002), the process proceeds to S1004.

[0079] In S1003, the pixel value O to be output when no flicker occurs is calculated.３ is calculated using the following formula 7. Then, the process proceeds to S1005.

[0080] O ３ = G x O １ In S1004, the pixel value O to be output when flicker occurs is calculated. ３ is calculated using Equation 1 shown in the first embodiment. Then, the process proceeds to S1005. In S1005, the pixel value O ３ It is determined whether the calculation process for has been completed for one frame's worth of pixels. If it has not been completed (No in S1005), the process from S1002 onwards is repeated. If it has been completed (Yes in S1005), this process is completed.

[0081] According to the third embodiment, it is possible to obtain an image more suitable as a WDR image by determining whether or not there is a light source that generates flicker in the image capture target and adaptively changing the composition ratio of period 1 and period 2. It goes without saying that the technique presented in the third embodiment is also applicable to the case where period 2 is located in the latter half of the frame, as shown in the second embodiment.

[0082] [Fourth Embodiment] In the first to third embodiments, the outputs of the periods 1 and 2 are combined inside the imaging device 100, but the location of the combination does not have to be limited to inside the imaging device 100. For example, the combination can also be performed by the controller 10 connected to the imaging device 100. Such a case will be described with reference to Figs. 11 and 12.

[0083] FIG. 11 shows the relationship between image output and generated frame images in this embodiment. Here, it is assumed that the image required by the user is a moving image at 30 fps, and the LED light emission cycle is 90 Hz. In this example, the imaging device 100 divides one frame period into three to generate three subframes, which are output to the downstream controller 10. In FIG. 11, the subframes are represented as Period 1, Period 1', and Period 2. From the perspective of the imaging device 100, the output of these subframes is operationally no different from capturing images at 90 fps.

[0084] 12 is a block diagram showing a schematic configuration example of an image capture device 100 to which an image capture control device according to this embodiment is applied, and a connection to an external controller. The difference from the first embodiment is that the exposure correction unit 105 and the gradation conversion unit 106 are not included, and instead a data superposition unit 1201 is included. The data superposition unit 1201 superposes information on the exposure time 112 and analog gain value 113 used to capture each pixel block during a blanking period of image data captured by the image sensor 102, and transmits the superposition information to the controller 10 via an LVDS signal line. It is also possible to perform information communication between the controller 10 and the image capture device 100 via a separately provided wired or wireless communication line, without superposing the information on the image data.

[0085] In this embodiment, the unit of region-specific exposure control is preferably each subframe. The exposure conditions used for the same subframe as the exposure image data are output to the controller 10. The controller 10 references the exposure conditions superimposed on the image of each subframe, corrects the image, and then synthesizes the images to generate a 30 fps frame image.

[0086] Returning to FIG. 11 , the three subframes that make up one frame in FIG. 11 are composed of subframe period 1 and subframe period 1', in which region-by-region exposure control is performed, and subframe period 2, in which the entire subframe period is the exposure time. Subframe period 1 of frame 1 is indicated by 1101-1, and subframe period 1 of frame 2 is indicated by 1101-2. Subframe period 1' of frame 1 is indicated by 1101-1', and subframe period 1' of frame 2 is indicated by 1101-2'. Similarly, subframe period 2 of frame 1 is indicated by 1102-1, and subframe period 2 of frame 2 is indicated by 1102-2. Here, it is assumed that the exposure times of subframe period 1 and subframe period 1' are equal for each frame. In other words, period 1103-1 and period 1103-1' are the same length, and period 1103-2 and period 1103-2' are the same length.

[0087] The exposure times for subframe period 1 and subframe period 1' of frame 1 and frame 2 may differ because exposure conditions are calculated for each frame. FIG. 11 relating to this embodiment illustrates a case in which the exposure times for frame 1 and frame 2 are different. Meanwhile, regardless of the frame, the exposure times for period 2 (1104-1, 1104-2) are the same length as period 2. Each module within the imaging device 100 performs control so that the charges exposed and accumulated in each subframe are output based on these exposure conditions. The details are the same as those described in embodiment 1, so further explanation is omitted. The controller 10 receives image data for subframes divided into three for each frame and with different exposure conditions, and adds the exposure data for period 1 and period 1' for each pixel. Then, by correcting the data for period 1+period 1' and the data for period 2 while referring to the exposure condition data superimposed on each frame, as described in embodiments 1 to 3, it becomes possible to generate a high-quality WDR image while reducing the effects of flicker on the captured image.

[0088] As described above, according to the fourth embodiment, the outputs of the periods 1 and 2 can be combined by an external controller or the like, rather than being combined inside the imaging device 100. Furthermore, the period 1 can be divided into two sub-frame periods, and the exposure conditions can be controlled for each region in the two sub-frame periods.

[0089] <Application of Imaging Device to Equipment> Hereinafter, an equipment 1000 will be described that includes a semiconductor device 1100 including a package 1020 on which a semiconductor chip 1110 including a semiconductor integrated circuit is mounted, as shown in FIG. 13 . The semiconductor chip 1110 is housed in the package 1020 and mounted on the equipment 1000. In the configuration shown in FIG. 13 , the semiconductor chip 1110 includes the imaging device according to the above-described embodiment. The semiconductor device 1100 can include a package 1020 that includes a base 1010 to which the semiconductor chip 1110 is fixed and a light-transmitting member 1030 such as glass that faces the semiconductor chip 1110. The package 1020 can include bonding members such as wires and bumps that connect inner leads provided on the base 1010 to terminals such as pad electrodes provided on the semiconductor chip 1110.

[0090] The device 1000 may include at least one of an optical device 1040, a control device 1050, a processing device 1060, a display device 1070, a storage device 1080, and a mechanical device 1090. The optical device 1040 is, for example, a lens, a shutter, or a mirror. The control device 1050 controls the semiconductor chip 1110. The control device 1050 is, for example, a semiconductor device such as an ASIC.

[0091] The processing device 1060 processes an output signal from the imaging device included in the semiconductor chip 1110. The processing device 1060 is a semiconductor device such as a CPU or ASIC for configuring an AFE analog front end or a DFE digital front end. For example, an image may be generated based on the event signal E. The display device 1070 is an EL display device or a liquid crystal display device that displays an information image obtained by the semiconductor chip 1110. The memory device 1080 is a magnetic device or a semiconductor device that stores the information image obtained by the semiconductor chip 1110. The memory device 1080 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0092] The mechanical device 1090 has a moving part or a propulsion part such as a motor or an engine. In the device 1000, a signal output from the semiconductor chip 1110 is displayed on the display device 1070, or transmitted to the outside via a communication device (not shown) included in the device 1000. For this purpose, the device 1000 may further include a memory device 1080 and a processing device 1060 in addition to the memory circuit and arithmetic circuit included in the semiconductor chip 1110. The mechanical device 1090 may be controlled based on the signal output from the semiconductor chip 1110.

[0093] The device 1000 is also suitable for electronic devices such as information terminals with imaging capabilities, such as smartphones, wearable devices, and cameras, such as interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras. The mechanical device 1090 in the camera can drive components of the optical device 1040 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 1090 in the camera can move the optical device 1040 for vibration isolation.

[0094] Furthermore, the device 1000 may be transportation equipment such as a vehicle, a ship, or an aircraft. The mechanical device 1090 in the transportation equipment may be used as a moving device. The device 1000 as transportation equipment is suitable for transporting semiconductor chips 1110 or for assisting and / or automating driving operations using an imaging function. The processing device 1060 for assisting and / or automating driving operations can perform processing to operate the mechanical device 1090 as a moving device based on information obtained by the semiconductor chip 1110. Alternatively, the device 1000 may be a medical device such as an endoscope, a measuring device such as a distance sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0095] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0096] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0097] This application claims priority based on Japanese Patent Application No. 2024-007546, filed January 22, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An imaging control device for causing an imaging device having a plurality of pixel blocks each including a plurality of pixels to capture a moving image, the imaging control device comprising a control unit that controls the imaging device to include: a first imaging period in which a charge accumulation time is controlled based on a first exposure condition determined for each of the plurality of pixel blocks during one frame period of the moving image; and a second imaging period including a charge accumulation time based on a preset second exposure condition.

2. The imaging control device according to claim 1, wherein the second imaging period is sandwiched by the first imaging periods respectively included in temporally consecutive frames.

3. The imaging control device according to claim 1 or 2, wherein the second imaging period is set after the first imaging period in the one frame period.

4. The imaging control device according to any one of claims 1 to 3, wherein the second imaging period is set to be longer than a light emission period of a light source that emits light periodically and is included in an imaging target.

5. The imaging control device according to any one of claims 1 to 4, wherein the first exposure condition and the second exposure condition include the charge accumulation time and a gain for amplifying an image signal obtained from the imaging device.

6. The imaging control device according to any one of claims 1 to 5, wherein two first imaging periods are included in the one frame period.

7. An image processing device that processes an image signal captured by the imaging device controlled by the imaging control device according to any one of claims 1 to 6, the image processing device generating a synthesized image signal by weighted addition of a first image signal captured in the first imaging period and a second image signal captured in the second imaging period at a predetermined ratio for each pixel in the one frame period.

8. The image processing device according to claim 7, wherein the predetermined ratio is based on the first exposure condition and the second exposure condition.

9. The image processing device according to claim 7 or 8, wherein the predetermined ratio is determined by comparing a difference between a first correction signal obtained by correcting the first image signal of pixels at a predetermined position according to the first exposure condition and a second correction signal obtained by correcting the second image signal of pixels at the predetermined position according to the second exposure condition with a threshold value.

10. The image processing apparatus according to claim 9, wherein when the difference is smaller than the threshold value, the ratio of the second image signal in the weighted addition is set to 0.

11. An imaging device comprising the imaging control device according to any one of claims 1 to 6, the image processing apparatus according to any one of claims 7 to 10, and the imaging element.

12. A method for driving an imaging element that drives the imaging element including a plurality of pixel blocks each including a plurality of pixels to perform video imaging, the method comprising: driving the imaging of the imaging element in a first imaging period including a charge accumulation time controlled based on exposure conditions for each of the plurality of pixel blocks during one frame period of the video; and driving the imaging of the imaging element in a second imaging period including a charge accumulation time preset in the one frame period.

13. A program executed by a computer, the program for causing the computer to execute each step included in the method for driving an imaging element according to claim 12.

14. An apparatus comprising the imaging device according to claim 11 and a processing device that processes an output signal from the imaging device.

Images