Imaging device and method for controlling imaging device

Abstract

An image processing device according to the present disclosure comprises: a division unit that divides a detection area provided in a pixel array unit which is included in an imaging unit in order to measure a reference level of pixel data based on a signal output from a pixel included in the pixel array unit; and a detection unit that detects the signal output from the pixel included in a partial area, into which the detection area has been divided by the division unit, to generate detection data. The division unit is configured so that a division pattern for dividing the detection area into the partial areas is programmable.

Description

Image capture device and image capture device control method

[0001] The present disclosure relates to an imaging device and a control method for the imaging device.

[0002] In recent years, CMOS (Complementary Metal Oxide Semiconductor) image sensors for mobile devices have become increasingly miniaturized along with an increase in the number of pixels, and products with pixel sizes of 0.6 μm or less have been announced.

[0003] Japanese Patent Application Laid-Open No. 2008-067061

[0004] On the other hand, with the miniaturization of pixels, various cases of periodic black shift have been occurring due to factors such as pixel layout and the readout order of AD (Analog to Digital) conversion. This periodic black shift poses challenges, as its causes are complex and unclear, and the amount of black shift is extremely small, making it difficult to address at the design stage. Furthermore, because this black shift is periodic, it is highly visible in terms of image quality, and improvements are desired from the perspective of image quality.

[0005] Therefore, an object of the present disclosure is to provide an imaging device that can suppress periodic black shift and a control method for the imaging device.

[0006] The image processing device according to the present disclosure comprises a division unit that divides a detection area provided in a pixel array unit of an imaging unit in order to measure a reference level of pixel data based on signals output from pixels included in the pixel array unit, and a detection unit that detects the signals output from the pixels included in the divided areas into which the detection area is divided by the division unit to generate detection data, and the division unit is set so that a division pattern for dividing the detection area into the divided areas is programmable.

[0007] 1 is a block diagram showing a configuration of an example of an imaging device as an electronic device applicable to each embodiment of the present disclosure. FIG. 2 is a block diagram showing a configuration of an example of an image sensor applicable to each embodiment. FIG. 3 is a schematic diagram showing an example in which one floating diffusion layer is shared by multiple pixels, applicable to each embodiment. FIG. 4 is a schematic diagram showing an example of connection of each pixel group to a VSL when a floating diffusion layer is shared by multiple pixels, applicable to each embodiment. FIG. 5 is a schematic diagram for explaining periodic black shift correction according to existing technology. FIG. 6 is a schematic diagram for explaining a detection area. FIG. 7 is a schematic diagram for explaining occurrence of black shift. FIG. 8 is a schematic diagram for explaining black shift correction using correction data by a black shift correction unit. FIG. 9 is a functional block diagram of an example for explaining functions of an imaging device according to a first embodiment. FIG. 10 is a block diagram showing an example of a hardware configuration of an image sensor according to the first embodiment. FIG. 11 is a block diagram showing an example of a configuration of an OPB division detection unit according to the first embodiment. FIG. 12 is a schematic diagram for explaining a basic pattern of a division pattern according to the first embodiment. FIG. 13 is a schematic diagram for explaining a method of selecting a division pattern according to the first embodiment. FIG. 14 is a schematic diagram for explaining a communication method between the image sensor according to the first embodiment and an application processor. FIG. 15 is a schematic diagram for explaining an OPB detection area. FIG. 1 is a schematic diagram for explaining detection of a divided region according to an existing technique. FIG. 2 is a schematic diagram for explaining detection of a divided region according to a first embodiment. FIG. 3 is a functional block diagram of an example for explaining the function of an imaging device according to a second embodiment. FIG. 4 is a block diagram of an example showing the hardware configuration of an image sensor according to the second embodiment. FIG. 5 is a timing chart for explaining the process of reading a pattern setting signal from an OTP according to the second embodiment. FIG. 6 is a functional block diagram of an example for explaining the function of an imaging device according to a third embodiment. FIG. 7 is a block diagram showing an example of a schematic configuration of a vehicle control system. FIG. 8 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0009] Hereinafter, embodiments of the present disclosure will be described in the following order: 1. Configuration applicable to each embodiment of the present disclosure 2. Regarding existing technology 3. First embodiment of the present disclosure 3-1. Configuration according to the first embodiment 3-2. More specific description of the first embodiment 3-3. Comparison of the technology according to the present disclosure with existing technology 4. Second embodiment of the present disclosure 5. Third embodiment of the present disclosure 6. Application examples of each embodiment of the present disclosure

[0010] (1. Configuration applicable to each embodiment of the present disclosure) First, a configuration applicable to each embodiment of the present disclosure will be described. Fig. 1 is a block diagram showing an example configuration of an imaging device as an electronic device applicable to each embodiment of the present disclosure.

[0011] In FIG. 1, the imaging device 1 includes an image sensor 10, an optical unit 11, an image processing unit 12, an output processing unit 13, and a control unit 14.

[0012] The image sensor 10 has a pixel array in which a plurality of pixels, each of which includes one or more light receiving elements, are arranged in a matrix. The optical unit 11 includes a lens, an aperture mechanism, a focus mechanism, etc., and guides light from a subject to the light receiving surface of the pixel array.

[0013] The image sensor 10 reads out a signal (pixel signal) from each pixel exposed for a specified exposure time, performs signal processing such as noise removal and gain adjustment on the read out signal, converts the processed signal into a digital signal, and outputs pixel data.

[0014] The image processing unit 12 performs predetermined image processing on the pixel data output from the image sensor 10, and outputs the result as image data for each frame, for example. The image processing performed by the image processing unit 12 on the pixel data includes, for example, development processing, pixel data synthesis, conversion processing, and color adjustment processing such as white balance processing.

[0015] The output processing unit 13 converts the image data output from the image processing unit 12 into a format suitable for output from the imaging device 1. The output image data output from the output processing unit 13 may be supplied to a display (not shown), for example, and displayed as an image. However, the output image data may also be supplied to another device, for example, a device that performs recognition processing on the output image data, or a control device that performs control based on the output image data.

[0016] The control unit 14 controls the overall operation of the imaging device 1. The control unit 14 includes, for example, a CPU (Central Processing Unit) and an interface circuit for communicating with each part of the imaging device 1, and the CPU operates according to a predetermined program to generate various control signals and control each part of the imaging device 1 using the generated control signals.

[0017] The image processing unit 12 and output processing unit 13 may be configured by a processor such as a CPU, DSP (Digital Signal Processor), or ISP (Image Signal Processor) that operates according to a predetermined program. Alternatively, one or both of the image processing unit 12 and the output processing unit 13 may be realized by a program that runs on the CPU together with the control unit 14. These programs may be stored in advance in a nonvolatile memory included in the imaging device 1, or may be supplied to the imaging device 1 from an external device and written into the memory.

[0018] Each pixel included in the pixel array of the image sensor 10 is provided with an optical filter that transmits light in a predetermined wavelength range. Unless otherwise specified, the optical filter that transmits light in the predetermined wavelength range will be described as a color filter. When obtaining full-color image data, three types of color filters are generally used: a color filter that transmits light in the R (red) wavelength band (R color filter), a color filter that transmits light in the G (green) wavelength band (G color filter), and a color filter that transmits light in the B (blue) wavelength band (B color filter). Hereinafter, pixels provided with R color filters will be referred to as R pixels, pixels provided with G color filters will be referred to as G pixels, and pixels provided with B color filters will be referred to as B pixels.

[0019] Although various arrangements of these R, G, and B pixels are conceivable, an arrangement called a Bayer arrangement is generally used. In the Bayer arrangement, for example, when a block consisting of four pixels (two pixels horizontally and two pixels vertically) is used as a unit, two G pixels and one R pixel and one B pixel are arranged so that pixels having color filters of the same color are not adjacent to each other. Alternatively, for example, pixels of the same color may be grouped into a block of two pixels horizontally by two pixels vertically, and this block may be regarded as one pixel, forming a Bayer arrangement (called a Quad Bayer arrangement) using the block.

[0020] 2 is a block diagram showing an example of the configuration of an image sensor 10 applicable to each embodiment. In FIG. 2, the image sensor 10 includes a pixel array unit 110, a vertical scanning unit 20, a horizontal scanning and AD conversion unit 21, and a control unit 25.

[0021] The pixel array unit 110 includes a plurality of pixels 100 each having a light-receiving element that generates a signal (voltage) in response to received light. A photodiode can be used as the light-receiving element. In the pixel array unit 110, the plurality of pixels 100 are arranged in a matrix in the horizontal direction (row direction) and the vertical direction (column direction). In the pixel array unit 110, the row direction arrangement of the pixels 100 is called a line. One frame of image (image data) is formed based on pixel signals read from a predetermined number of lines in the pixel array unit 110. For example, if one frame of image is formed with 3000 pixels x 2000 lines, the pixel array unit 110 includes at least 2000 lines, each including at least 3000 pixels 100.

[0022] In addition, in the pixel array section 110, pixel signal lines HCTL are connected to the rows and columns of the pixels 100 for each row, and vertical signal lines VSL are connected to the columns.

[0023] The ends of the pixel signal lines HCTL that are not connected to the pixel array unit 110 are connected to the vertical scanning unit 20. The vertical scanning unit 20 transmits a plurality of control signals, such as drive pulses used to read pixel signals from the pixels 100, to the pixel array unit 110 via the pixel signal lines HCTL in accordance with control signals supplied from, for example, the control unit 14. The ends of the vertical signal lines VSL that are not connected to the pixel array unit 110 are connected to the horizontal scanning and AD conversion unit 21.

[0024] The horizontal scanning and AD conversion unit 21 includes an AD (Analog to Digital) conversion unit, an output unit, and a signal processing unit. Pixel signals read from the pixels 100 are transmitted to the AD conversion unit of the horizontal scanning and AD conversion unit 21 via vertical signal lines VSL.

[0025] The following provides an overview of the control of reading out pixel signals from the pixels 100. Reading out pixel signals from the pixels 100 is performed by transferring charges accumulated in the light-receiving elements upon exposure to a floating diffusion layer (FD), and converting the transferred charges into a voltage in the floating diffusion layer. The voltage into which the charges are converted in the floating diffusion layer is output to the vertical signal line VSL via an amplifier. Note that the floating diffusion layer may be referred to as the FD below.

[0026] More specifically, in the pixel 100, during exposure, the connection between the light-receiving element and the floating diffusion layer is turned off (open), allowing the light-receiving element to accumulate charge generated in response to incident light through photoelectric conversion. After exposure is completed, the floating diffusion layer is connected to the vertical signal line VSL in response to a selection signal supplied via the pixel signal line HCTL. Furthermore, the floating diffusion layer is briefly connected to a power supply voltage VDD or a black level voltage supply line in response to a reset pulse supplied via the pixel signal line HCTL, resetting the floating diffusion layer. A voltage (referred to as voltage P) at the reset level of the floating diffusion layer is output to the vertical signal line VSL. Thereafter, a transfer pulse supplied via the pixel signal line HCTL turns on (closes) the connection between the light-receiving element and the floating diffusion layer, transferring the charge accumulated in the light-receiving element to the floating diffusion layer. A voltage (referred to as voltage Q) corresponding to the amount of charge in the floating diffusion layer is output to the vertical signal line VSL.

[0027] In the horizontal scanning / AD conversion unit 21, the AD conversion unit includes, for example, an AD converter provided for each vertical signal line VSL, and pixel signals supplied from the pixels 100 via the vertical signal lines VSL are subjected to AD conversion processing by the AD converter, and two digital values ​​(values ​​corresponding to the voltage P and the voltage Q, respectively) are generated for correlated double sampling (CDS) processing, which performs noise reduction.

[0028] The two digital values ​​generated by the AD converter are subjected to CDS processing by the signal processing unit, and a pixel signal (pixel data) is generated as a digital signal. The generated pixel data is output from the pixel array unit 110.

[0029] The horizontal scanning / AD conversion unit 21 performs selective scanning to select the AD converters for each vertical signal line VSL in a predetermined order under the control of the control unit 25, thereby causing each digital value temporarily held by each AD converter to be output sequentially to the signal processing unit. The horizontal scanning / AD conversion unit 21 achieves this operation by using a configuration including, for example, a shift register, an address decoder, etc.

[0030] The control unit 25 controls the driving of the vertical scanning unit 20, the horizontal scanning / AD conversion unit 21, etc. The control unit 25 generates various driving signals that serve as references for the operation of the vertical scanning unit 20 and the horizontal scanning / AD conversion unit 21. Based on a vertical synchronization signal or an external trigger signal supplied from the outside (for example, the control unit 14) and a horizontal synchronization signal, the control unit 25 generates control signals that the vertical scanning unit 20 supplies to each pixel 100 via the pixel signal line HCTL. The control unit 25 supplies the generated control signals to the vertical scanning unit 20.

[0031] Based on a control signal supplied from the control unit 25, the vertical scanning unit 20 supplies various signals including drive pulses to the pixel signal lines HCTL of a selected pixel row of the pixel array unit 110, to each pixel 100 for each line, and causes each pixel 100 to output a pixel signal to a vertical signal line VSL. The vertical scanning unit 20 is configured using, for example, a shift register, an address decoder, etc.

[0032] The image sensor 10 configured in this manner is a column AD type CMOS (Complementary Metal Oxide Semiconductor) image sensor in which AD converters are arranged for each column.

[0033] In the above description, each pixel 100 has been described as having a floating diffusion layer, but this is not limited to this example. For example, multiple pixels 100 may share one floating diffusion layer. FIG. 3 is a schematic diagram showing an example in which multiple pixels 100 share one floating diffusion layer, which is applicable to each embodiment. The example in FIG. 3 shows an example in which four adjacent pixels 100a, 100b, 100c, and 100d included in a pixel block 150 share one floating diffusion layer (FD). Each of the pixels 100a to 100d is connected to VSL via an adder 140.

[0034] The pixel signals of each of the pixels 100a to 100d are read out in either an addition mode or an individual mode.

[0035] In the addition mode, the charges accumulated in each pixel 100a to 100d are added together within the pixel block 150 and output. For example, under the control of the control unit 25, the vertical scanning unit 20 resets the floating diffusion layer in that pixel block 150 at a predetermined timing, then reads out charges from each light-receiving element in the pixels 100a to 100d and transfers the read-out charges to the floating diffusion layer. In the floating diffusion layer, the charges transferred from each light-receiving element are added together in the adder 140. In this case, the adder 140 corresponds to a floating diffusion layer common to the pixels 100a to 100d. In the floating diffusion layer, the added charges transferred from each light-receiving element are converted into a voltage corresponding to the amount of charge, and the voltage is output to the vertical signal line VSL as a pixel signal representing the sum of the charges for each pixel 100a to 100d.

[0036] In the individual mode, pixel signals read from each of the pixels 100a to 100d are output. For example, under the control of the control unit 25, the vertical scanning unit 20 resets the floating diffusion layer in the pixel block 150 at a predetermined timing, and then reads charge from the light-receiving element in, for example, pixel 100a and transfers the read charge to the floating diffusion layer. In the floating diffusion layer, the transferred charge is converted into a voltage corresponding to the amount of charge, and is output to the vertical signal line VSL as a pixel signal read from pixel 100a. Note that addition processing by the adder 140 is not performed.

[0037] Next, the vertical scanning unit 20 resets the floating diffusion layer in the pixel block 150 at a predetermined timing after the timing at which the floating diffusion layer for pixel 100a is reset. Then, for example, in pixel 100b, the vertical scanning unit 20 reads out charge from the light-receiving element and transfers the read-out charge to the floating diffusion layer. In the floating diffusion layer, the transferred charge is converted into a voltage corresponding to the amount of charge and output to the vertical signal line VSL as a pixel signal read out from pixel 100b. Note that the addition process by the adder 140 is not performed.

[0038] The vertical scanning unit 20 also reads pixel signals from the pixels 100c and 100d, resetting the floating diffusion layers of each pixel, and then reads charges from the light-receiving elements and transfers the read charges to the floating diffusion layers. Note that addition processing by the adder 140 is not performed in either case.

[0039] FIG. 4 is a schematic diagram showing an example of connection of each pixel group to a VSL when a floating diffusion layer is shared by multiple pixels, which is applicable to each embodiment.

[0040] 4, pixel block 1501 includes eight pixels (four pixels horizontally and two pixels vertically), and these eight pixels share one floating diffusion layer FD1. Similarly, pixel blocks 1502 to 1504 each include eight pixels (four pixels horizontally and two pixels vertically), and each eight pixels share one floating diffusion layer FD2 to FD4.

[0041] 4, each of the pixel blocks 150 to 150 is connected to a different VSL. Each VSL is further connected to a number of pixel blocks 150. Each VSL is connected to a respective AD converter 22 to 22 included in the horizontal scanning / AD conversion unit 21. For example, a pixel signal read out from the pixel block 150 is supplied to the AD converter 22 via the VSL, converted into pixel data, which is a digital signal, and output from the horizontal scanning / AD conversion unit 21.

[0042] (2. Existing Technology) Here, before describing the embodiments of the present disclosure, correction processing for periodic black shift according to existing technology will be described for ease of understanding.

[0043] In recent years, CMOS (Complementary Metal Oxide Semiconductor) image sensors for mobile devices have become increasingly miniaturized along with an increase in the number of pixels, and products with pixel sizes of 0.6 μm or less have been announced.

[0044] On the other hand, with the miniaturization of pixels, various cases of periodic black shift have been occurring due to factors such as pixel layout and the readout order of AD (Analog to Digital) conversion. Periodic black shift refers to the periodic fluctuation of the black level of an image signal relative to the position on the screen (pixel array). This periodic black shift poses challenges, such as its complex and unclear causes and the extremely minute amount of black shift, making it difficult to address at the design stage. Furthermore, because this black shift is periodic, it is highly visible in terms of image quality, and improvements in terms of image quality are desired.

[0045] For example, in order to address the above-mentioned problems, Patent Document 1 discloses a technique for splitting and detecting a detection region in an OPB (Optical Black) region and correcting an image signal based on the results of this split detection.

[0046] However, while existing technology can handle known black shift, it is difficult to predict and address cases where the causes are complex, such as those identified in post-design evaluations, such as differences in the readout order depending on the drive mode, or temperature-dependent dark current causes. Even when addressing black shift, it is necessary to prepare a large number of detection circuits (possibly several tens of types) that perform split detection of the OPB region for each expected black shift pattern, which not only increases both the circuit size and power consumption, but also often results in deviations from expectations.

[0047] FIG. 5 is a schematic diagram illustrating periodic black shift correction according to an existing technique. In FIG. 5, section (a) is an exemplary functional block diagram illustrating the function of an image capture device 600 that corrects periodic black shift according to an existing technique. In section (a), the image capture device 600 includes an image sensor 10 and an application processor 30. The image sensor 10 and the application processor 30 are connected via a communication line using a predetermined communication interface. For example, a Mobile Industry Processor Interface (MIPI) may be applied as the communication interface for communication between the image sensor 10 and the application processor 30.

[0048] The image sensor 10 includes a pixel array unit 110, an OPB division detection unit 120, and an image signal processing unit 130. The application processor (AP) 30 also includes a black shift correction unit 300. The application processor 30 may include, for example, the functions of the image processing unit 12 and the control unit 14 in FIG. 1 .

[0049] The pixel array unit 110 converts pixel signals output from the pixels 100 into pixel data, which is a digital signal, using an AD converter 22 (shown as ADC 22 in the figure), and outputs the pixel data. The pixel data output from the pixel array unit 110 is supplied to the OPB division detection unit 120 and the image signal processing unit 130.

[0050] The image signal processing unit 130 performs signal processing such as noise removal and gain adjustment on the pixel data output from the pixel array unit 110, performs detection, and outputs the processed pixel data as image data, for example, in units of frames. The image data output from the image signal processing unit 130 is supplied to the application processor 30 via a communication line.

[0051] The OPB division detection unit 120 detects pixel data included in each divided region obtained by dividing the OPB region as the detection region, among the pixel data output from the pixel array unit 110. The OPB region is a region in which the pixels 100 included in a predetermined region in the pixel array unit 110 are, for example, shielded from light to prevent light from entering, and optically outputs a black pixel signal.

[0052] Fig. 6 is a schematic diagram for explaining a detection region. In the example of Fig. 6, an OPB region is provided at the upper end of the pixel array unit 110, and this OPB region is set as a detection region 1100 that is the detection target by the OPB division detection unit 120. In the pixel array unit 110, the region other than the detection region 1100 is set as an effective region 1110 that the image signal processing unit 130 uses to generate image data. In other words, image data for one frame is generated based on pixel signals output from each pixel 100 included in the effective region 1110.

[0053] 5, section (b) schematically illustrates an example of detection data output from the OPB division detection unit 120. Here, in the pixel array unit 110, each pixel 100 is assumed to be in a Bayer array (quad-Bayer array) in which four adjacent pixels, 2 pixels horizontally and 2 pixels vertically, are regarded as one pixel.

[0054] In the example of section (b), the unit is 4 squares horizontally by 8 squares vertically, and each square contains pixel data of multiple pixels 100 whose positions correspond to each unit in the detection area 1100. The detection data contains pixel data of each pixel 100 in an array corresponding to this quad Bayer array. In this example, the OPB division detection unit 120 divides the detection area 1100 into groups of multiple pixels 100 whose positions correspond to each other in the quad Bayer array. That is, each divided area may be composed of multiple pixels 100 (also called subpixels) that are not adjacent to each other in the pixel array unit 110 and are arranged according to a predetermined rule. That is, each square in section (b) represents a subpixel.

[0055] For each divided region, the OPB division detection unit 120 performs detection on the pixel data of each pixel 100 included in the divided region. Note that detection here refers to a process of calculating the average value of pixel values ​​of pixel data at the same position in each divided region (average value detection).

[0056] For example, consider a block consisting of an upper quad Bayer array and a lower quad Bayer array, with two quad Bayer arrays arranged vertically in the figure. Subpixel R00 shown in section (b) of Figure 5 corresponds to a divided area based on pixel 100 in the upper left corner of the upper quad Bayer array in each block. The OPB division detection unit 120 extracts the pixel values ​​of pixel 100 in each subpixel R00 from each block, calculates the average value, and performs detection.

[0057] Also, in the example of section (b) in Figure 5, each of the sub-pixels R00 to R03, Gr00 to Gr03, Gb00 to Gb03 and B00 to B03, as well as R10 to R13, Gr10 to Gr13, Gb10 to Gb13 and B10 to B13, which are divided regions, is linked to the color of the corresponding pixel 100 (the color of the provided color filter) and the position of that pixel 100 in the block.

[0058] The OPB division detection unit 120 supplies the detection data (average value data) to the application processor 30 via a communication line. In the application processor 30, the black shift correction unit 300 generates correction data for correcting black shift based on the detection data supplied from the OPB division detection unit 120. Section (c) of Figure 5 schematically shows the target for which the OPB division detection unit 120 calculates the average value.

[0059] (1) The OPB division detection unit 120 calculates the average value of pixel data (pixel values) based on all colors, i.e., all detection data. (2) to (17) The OPB division detection unit 120 further calculates the average value for each color (each pixel position) based on the pixel data of each pixel 100 included in each divided region corresponding to the upper Quad Bayer arrangement in the block described above, i.e., each subpixel R00 to R03, Gr00 to Gr03, Gb00 to Gb03, and B00 to B03. (18) to (33) The OPB division detection unit 120 similarly calculates the average value for each color (each pixel position) based on the pixel data of each pixel 100 included in each divided region corresponding to the lower Quad Bayer arrangement in the block described above, i.e., each subpixel R10 to R13, Gr10 to Gr13, Gb10 to Gb13, and B10 to B13.

[0060] In the existing technology, the divided regions obtained by dividing the detection region 1100 are fixed, and in order to detect periodic black shift, it is necessary to address all factors that are assumed to cause periodic black shift. Therefore, in the existing technology, detection data is obtained by setting divided regions in a brute-force manner.

[0061] 5, for the sake of explanation, the detection data includes 33 color modes consisting of all colors, subpixels R00 to R03, Gr00 to Gr03, Gb00 to Gb03, and B00 to B03, and subpixels R10 to R13, Gr10 to Gr13, Gb10 to Gb13, and B10 to B13. In other words, the OPB division detection unit 120 includes 33 detection circuits corresponding to these 33 color modes. The OPB division detection unit 120 calculates 33 average values ​​AVEOPB according to the respective color modes.

[0062] It should be noted that the OPB division detection unit 120 is not limited to the example of section (c) in Figure 5, and may include a larger number of detection circuits depending on, for example, the configuration of the AD converter 22 and the layout of the pixel 100.

[0063] The black shift correction process will be described with reference to FIGS.

[0064] 7 is a schematic diagram for explaining the occurrence of black shift. FIG. 7 shows an example in which there is no periodic black shift in the horizontal direction, but periodic black shift occurs in the vertical direction at a cycle of four rows. As described above, the OPB division detection unit 120 performs detection for each division area obtained by dividing the detection area 1100 based on the pixels 100 whose positions correspond in each of the above-described blocks. In section (a) of FIG. 7, the detection area 1100 is divided into division areas according to the category of each pixel 100.

[0065] In the figure, "Category #x-y-z" indicates that the pixels 100 are classified based on, for example, structural parameters x, y, and z. For example, assuming x=2, y=4, and z=2, the detection region 1100 is divided into 16 divided regions based on three parameters associated with each pixel 100. Note that periodic black shift is thought to occur due to the layout of each pixel 100 and the order in which pixel signals are read out, and the color filter colors R, Gr, Gb, and B are essentially unrelated to periodic black shift. The color filter colors R, Gr, Gb, and B may be used as indexes.

[0066] In the example of section (a) in Figure 7, 16 divided areas are arranged in a 4x4 array, with each divided area being a 2x2 area according to the Bayer array. The top row, or first row, is arranged from left to right in the figure in the following order: divided area B (category #1-1-2), divided area Gb (category #1-1-3), divided area B (category #1-2-2), and divided area Gr (category #1-2-3). The second row is arranged in the following order: divided area Gr (category #2-1-3), divided area R (category #2-1-2), divided area Gr (category #2-2-3), and divided area R (category #2-2-2). The third row is arranged in the following order: divided area B (category #1-1-0), divided area Gb (category #1-1-1), divided area B (category #1-2-0), and divided area Gr (category #1-2-1). In addition, the fourth row is arranged in the order of divided area Gr (category #2-2-1), divided area R (category #2-1-0), divided area Gr (category #2-2-1), and divided area R (category #2-2-0).

[0067] 7 is a graph showing an example of the average pixel value of the pixel data of each pixel 100 included in each divided area of ​​section (a). In this graph, the vertical axis represents the black level (LSB) and the horizontal axis represents each divided area.

[0068] The average pixel values ​​of each divided region are lower than 64 LSB, which is the standard for the black level, but a certain trend is observed. That is, according to sections (a) and (b) of Figure 7, the average values ​​are approximately the same in the horizontal direction, but there are large differences in the average values ​​in the vertical direction between the first, second, third, and fourth rows.

[0069] More specifically, the average pixel values ​​of divided area B (category #1-1-2) and divided area B (category #1-2-2) in the first row of section (a) of Figure 7, and divided area Gb (category #1-1-3) and divided area Gb (category #1-2-3) are lower than the average pixel values ​​of divided area B (category #1-1-0) and divided area B (category #1-2-0) in the third row, and divided area Gb (category #1-1-1) and divided area Gb (category #1-2-1).

[0070] In addition, the average pixel values ​​of the divided areas B (category #1-1-2) and B (category #1-2-2) in the second row of section (a) of Figure 7, and the divided areas R (category #2-1-3) and R (category #2-2-3) are lower than the average pixel values ​​of the divided areas Gr (category #2-2-1) and Gr (category #2-2-1) in the fourth row, and the divided areas R (category #2-1-0) and R (category #2-2-0).

[0071] The tendency of this difference in average values ​​is thought to appear as periodic black shift. The OPB division detection unit 120 generates correction data for each divided region. The black shift correction unit 300 uses the generated correction data to correct the pixel data of the pixels 100 in the effective region 1110 that correspond to each divided region.

[0072] 8 is a schematic diagram illustrating black shift correction based on correction data by the black shift correction unit 300. The OPB division detection unit 120 generates correction data 401 to 408 for each divided region based on the difference between the average luminance value of each divided region and the reference level (64 LSB). Each correction data 401 to 408 is offset data for adjusting the average luminance value of the corresponding divided region to 64 LSB. The black shift correction unit 300 adds or subtracts the correction data 401 to 408 for the divided region to which each pixel data corresponds to each pixel data in the image data supplied from the image signal processing unit 130. This corrects the value of each pixel data, making it possible to suppress black shift.

[0073] (3. First Embodiment of the Present Disclosure) Next, a first embodiment of the present disclosure will be described. In the existing technology described above, the division pattern for dividing the detection area 1100 is fixed, requiring many detection circuits. In contrast, in the first embodiment of the present disclosure, the number of detection circuits for detecting the OPB detection area is limited to a maximum of N (N is an integer equal to or greater than 2), and the division pattern for dividing the detection area 1100 to be assigned to each detection circuit is programmably selectable. By selecting the division pattern based on the shooting conditions of the image sensor 10, the temperature within the image sensor 10, and the like, it is possible to reduce the number of detection circuits for detecting the OPB detection area and further to correct periodic black shift that occurs due to unforeseen circumstances.

[0074] (3-1. Configuration According to First Embodiment) A configuration according to a first embodiment of the present disclosure will be described. Fig. 9 is a functional block diagram of an example for explaining the function of the imaging device 1 according to the first embodiment.

[0075] 9, the imaging device 1a includes an image sensor 10a and an application processor 30a connected to the image sensor 10a via a predetermined communication interface. The communication interface for communication between the image sensor 10a and the application processor 30 may be, for example, MIPI.

[0076] The image sensor 10a is configured by adding a thermometer 160 and an interface (I / F) 161 to the image sensor 10 described in Fig. 5. The application processor 30a is configured by adding an OPB pattern change determination unit 310 and a camera control unit 320 to the application processor 30 described in Fig. 5.

[0077] In the application processor 30a, the camera control unit 320 controls the image capturing operation of the image sensor 10a. The camera control unit 320 may control the image capturing operation in response to, for example, a user operation on a user interface (not shown), or in response to an external instruction via a communication interface (not shown). The camera control unit 320 passes image capturing information to the OPB pattern change determination unit 310 in response to image capturing. The image capturing information includes information related to image capturing, such as the drive mode of the image sensor 10a, the analog gain for pixel signals, and the exposure time.

[0078] The OPB pattern change determination unit 310 includes a pattern RAM (Random Access Memory) 311 that stores multiple pattern setting signals indicating division patterns. The pattern setting signal may include an index indicating the division pattern, or may include information indicating the division pattern itself. The pattern setting signal may also include combination information indicating a combination of multiple division patterns. In the following description, the pattern setting signal is assumed to include an index and combination information. The pattern RAM 311 may store the pattern setting signal in a non-volatile manner, or may obtain and store the pattern setting signal from outside the application processor 30a.

[0079] The OPB pattern change determination unit 310 reads a pattern setting signal indicating a division pattern from the pattern RAM 311 based on at least one of the shooting information passed from the camera control unit 320 and temperature information inside the image sensor 10a measured by the thermometer 160 provided in the image sensor 10a. The OPB pattern change determination unit 310 passes the pattern setting signal read from the pattern RAM 311 to the camera control unit 320.

[0080] The camera control unit 320 transmits the pattern setting signal received from the OPB pattern change determination unit 310 to the image sensor 10a, and instructs the image sensor 10a to set the division pattern indicated in the pattern setting signal. The pattern setting signal may include an area control signal that specifies the detection area 1100 to be used as the OPB area.

[0081] The image sensor 10a receives the pattern setting signal and the division pattern setting instruction transmitted from the camera control unit 320 via the interface 161. The communication between the camera control unit 320 and the interface 161 is performed, for example, by an I / F. 2 C (Inter Integrated Circuit) may be applied.

[0082] The image sensor 10a passes the division pattern setting signal and setting instruction received by the interface 161 to the OPB division detection unit 120. The OPB division detection unit 120 divides the detection area 1100 in accordance with the passed division pattern setting signal, performs detection on the pixel data of each divided area, and transmits the detection data (average value data) to the application processor 30a.

[0083] The application processor 30a uses correction data based on the detection data transmitted from the image sensor 10a to correct the periodic black shift using the black shift correction unit 300. The method of periodic black shift correction by the black shift correction unit 300 can be the method described using FIG. 8 , and therefore detailed description thereof will be omitted here.

[0084] FIG. 10 is a block diagram illustrating an example of a hardware configuration of the image sensor 10a according to the first embodiment.

[0085] 10 , the image sensor 10 a includes a CPU (Central Processing Unit) 1300, a ROM (Read Only Memory) 1301, a RAM 1302, a pixel array unit 110, an OPB detection circuit 1303, an image signal processing circuit 1304, and a communication I / F 1305, which are communicatively connected to each other via a bus 1310. The bus 1310 may be, for example, an APB (Advanced Peripheral Bus) bus.

[0086] A CPU 1300 controls the overall operation of the image sensor 10a in accordance with a program stored in a ROM 1301 and using a RAM 1302 as a work memory.

[0087] The OPB detection circuit 1303 corresponds to the above-mentioned OPB division detection unit 120, and performs detection on pixel data from the pixels 100 in the detection area 1100 output from the pixel array unit 110. The OPB detection circuit 1303 includes a maximum of N detection circuits. The OPB detection circuit 1303 passes the detection data resulting from the detection to the communication I / F 1305. The image signal processing circuit 1304 corresponds to the above-mentioned image signal processing unit 130, and performs predetermined image processing on image data from the pixels 100 in the effective area 1110 output from the pixel array unit 110, and outputs the image data for each frame to the communication I / F 1305.

[0088] The communication I / F 1305 transmits to the application processor 30a the detected data output from the OPB detection circuit 1303 and the image data output from the image signal processing circuit 1304. The communication I / F 1305 may be compliant with the MIPI standard.

[0089] 11 is a block diagram showing an example configuration of the OPB division detection unit 120 according to the first embodiment. In FIG. 11, the OPB division detection unit 120 includes a division area selection unit 1200 and an integrating unit 1201. The integrating unit 1201 includes integrators 1210, the number of which corresponds to the number N of division areas. Here, N=16, and the integrating unit 1201 includes 16 integrators 1210#0 to 1210#15. Each of the integrators 1210#0 to 1210#15 functions as a detection circuit that integrates pixel data specified by an enable signal to calculate an average value.

[0090] The OPB division detection unit 120 inputs the pixel data of the detection region 1100 output from the pixel array unit 110 to the integration unit 1201. The OPB division detection unit 120 also inputs the region selection signal included in the pattern setting signal transmitted from the application processor 30a to the division region selection unit 1200.

[0091] Meanwhile, the image sensor 10a associates an index indicating a division pattern with information about the division pattern and stores the associated information in advance in a storage unit (not shown) included in the image sensor 10a. In the following description, it is assumed that the storage unit storing the index and division information is a register included in the image sensor 10a. The image sensor 10a references the register based on the index included in the pattern setting signal transmitted from the application processor 30a, and acquires a pattern setting signal (pattern setting information) indicating the division pattern associated with the index. The image sensor 10a inputs the pattern setting signal of the acquired division pattern to the division area selector 1200.

[0092] Based on the pattern setting signal, the divided area selection unit 1200 passes an enable signal for selecting pixels to be used for calculating the average value of pixel values ​​included in the divided area to the accumulator 1201. Based on the enable signal, the accumulator 1201 assigns pixel data to be accumulated for calculating the average value, from the pixel data of the detection area 1100 input to the accumulator 1201, to each of the 16 accumulators 1210#0 to 1210#15 included in the accumulator 1201.

[0093] In the accumulator 1201, each accumulator 1210#0 to 1210#15 accumulates the pixel values ​​of the pixel data assigned to it based on the enable signal, and calculates an average value based on the accumulation results. The calculation results Average#0 to Average#15 calculated by each accumulator 1210#0 to 1210#15 are output data 1202 of the OPB division detection unit 120, and are output as average value data AVEOPB#1 to #15. The average value data AVEOPB#1 to #15 correspond to the detection data shown in FIG. 9 and have a data length of, for example, 16 bits.

[0094] (3-2. More Specific Description of the First Embodiment) Next, the processing according to the first embodiment will be described in more detail. As described above, in the first embodiment, the division pattern for dividing the detection region 1100, which is to be assigned to each detection circuit (integrators 1210#0 to 1210#15), can be programmably selected. The assignment of the division patterns to each detection circuit according to the first embodiment will be described using FIGS. 12A and 12B.

[0095] 12A is a schematic diagram illustrating a basic pattern of the division pattern according to the first embodiment. In the example shown in the figure, for each pixel 100 in the pixel array unit 110 (detection region 1100), R, Gr, Gb, and B color filters are provided by regarding four adjacent pixels, 2 pixels horizontally by 2 pixels vertically, as one pixel in the above-mentioned Quad Bayer arrangement.

[0096] In such an arrangement of pixels 100, four pixels (four pixels horizontally by one pixel vertically) are defined as a minimum unit 1101, and a pattern in which 16 such minimum units 1101 are arranged vertically is defined as a minimum pattern period 1102.

[0097] 12B is a schematic diagram for explaining a method for setting a division pattern according to the first embodiment. In the first embodiment, a division pattern for dividing the detection area 1100 can be programmably selected according to the procedure shown in steps S1 and S2 in FIG.

[0098] The division region selection unit 1200 first selects a pixel 100 (target pixel) to be integrated for calculating the average value based on an index included in the pattern setting signal (step S1). For example, a pattern consisting of eight pixels (four pixels horizontally and two pixels vertically) including two minimum units 1101 is set as a unit pattern of the division pattern, and the pixel data of which pixel 100 included in the unit pattern is to be integrated is represented by an 8-bit bit string. In the example shown in the figure, unit pattern #00 = 8'b10100100, unit pattern #01 = 8'b01101001, unit pattern #02 = 8'b00000000, and unit pattern #03 = 8'b11111111. Here, "8'b" indicates that the following character string represents each bit of the 8 bits, with bit "1" indicating a target pixel and bit "0" indicating a non-target pixel.

[0099] The division region selection unit 1200 then generates a division pattern by combining unit patterns #00 to #03 based on the pattern setting signal included in the pattern setting signal (step S2). For example, the division pattern is represented by a 16-bit bit string that combines eight unit patterns, each represented by two bits. In the example shown in the figure, the combination information is 16'b0001101001100000, and the division pattern is configured according to the target pattern, with each unit pattern arranged from top to bottom in the figure as unit pattern #00, unit pattern #01, unit pattern #01, unit pattern #10, unit pattern #11, unit pattern #01, unit pattern #10, unit pattern #00, and unit pattern #00.

[0100] The divided area selection unit 1200 passes a division pattern in which each unit pattern is arranged according to the division pattern = 16'b0001101001100000 to, for example, an accumulator 1210#0. Here, "16'b" indicates that the following character string represents each bit of 16 bits. The accumulator 1210#0 accumulates the pixel data (pixel value) of the pixel 100 at a position corresponding to bit "1" from the pixel data of the detection area 1100 input to the OPB division detection unit 120, according to the division pattern passed from the divided area selection unit 1200. The accumulator 1210#0 calculates an average value based on the accumulation result and outputs average value data AVEOPB#0.

[0101] The OPB division detection section 120 performs the process of step S2 by the division region selection section 1200 for each of the integrators 1210#0 to #15, and outputs each of the average value data AVEOPB#0 to #15.

[0102] In the above description, the division patterns are generated in two stages: a unit pattern and a combination thereof. However, this is not limited to this example. For example, it is also possible to directly specify 64-bit data (8 bits x 8 patterns) as the division patterns. However, by generating the division patterns in two stages as described above, the amount of data in the pattern setting signal can be reduced and scalability is also improved.

[0103] Next, a method for transmitting image data and detection data from the image sensor 10a to the application processor 30a according to the first embodiment will be described. Fig. 13 is a schematic diagram for explaining a communication method between the image sensor 10a and the application processor 30a according to the first embodiment.

[0104] As described above, in the first embodiment, as shown in section (a) of Figure 13, the transmitter Tx of the image sensor 10a and the receiver Rx of the application processor 30a are connected by a communication line, and MIPI can be applied as the communication method for communication via this communication line.

[0105] Section (b) of Fig. 13 shows a schematic example of a data format defined by MIPI. In Fig. 13, the top left corner of the figure is the start position of data transmission, and data is transmitted in order from left to right on the figure, and further in order from top to bottom on the figure.

[0106] In section (b) of Figure 13, image data for one frame is divided into two parts in the column direction, and one of the divided image data is placed in the upper area 500a of the format as image data for a first sub-frame, and the other divided image data is placed in the lower area 500b of the format as image data for a second sub-frame.

[0107] In the upper part of the format, area 503b at the left end of area 500a and areas 503a and 503c at the top and bottom of areas 500a and 503b are areas where embedded data is placed. Area 501a at the right end of areas 500a, 503a, and 503c is a horizontal blanking period for the image data stored in area 500a.

[0108] Similarly, in the lower part of the format, area 503e at the left end of area 500b and areas 503d and 503f at the top and bottom of areas 500b and 503e are areas where embedded data is placed, and area 501b at the right end of areas 500b, 503d and 503f is a horizontal blanking period for the image data stored in area 500b.

[0109] Furthermore, an area 502a between areas 501a and 503c and areas 501b and 503d is a vertical blanking period for the image data arranged in area 500a, for example. Area 502a at the bottom of the format is a vertical blanking period common to the image data of the first subframe and the image data of the second subframe.

[0110] In such a format, one frame of image data output from the image signal processing unit 130 is divided into two in the column direction, one of which is placed in region 500a and the other in region 500b. Also, the detected data (average value data AVEOPB#1 to #16) output from the OPB division detection unit 120 may be placed in one or more of the regions where embedded data is placed in section (b) of Figure 13, i.e., regions 503a, 503b, 503d, 503e, and 503f.

[0111] Next, a method for determining the value N that limits the number of detection circuits for detecting the OPB detection area will be described. Fig. 14 is a schematic diagram for explaining the OPB detection area (detection area 1100). As shown in the figure, the number of pixels in the horizontal direction (number of horizontal detection pixels) of the detection area 1100 is set to H, and the number of rows in the vertical direction (number of vertical detection rows) is set to V. The size of the detection area 1100 is determined by restrictions on the aperture of the effective area 1110, etc.

[0112] Consider the variation in pixel values ​​in the detection area 1100. First, as shown in the following equation (1), the variation caused by random noise can be found by dividing the random noise (RN) by the square root of the number of samples (number of pixels = H × V) in the detection area 1100.

[0113]

[0114] The image sensor 10a performs AD conversion of pixel signals using a column AD system in which AD converters are arranged on each column. This makes it easy for horizontal noise (called horizontal random noise) to occur. Since horizontal noise is the same noise in the horizontal direction, the variation caused by horizontal random noise can be calculated by dividing the horizontal random noise (HRN) by the square root of the number of vertical detection rows, as shown in the following equation (2).

[0115]

[0116] The estimated variation (σ) of the average pixel values ​​in the detection area 1100 can be obtained by squaring the variation caused by the random noise and the variation caused by the horizontal random noise, adding them together, and then calculating the square root, as shown in the following equation (3):

[0117]

[0118] As can be seen from equation (3), for example, if the detection area 1100 is divided into two in the horizontal direction, the number of horizontal detection pixels H will be halved, and the estimated value of the variation (σ) will increase. The same is true for the number of vertical detection rows V. In other words, when the detection area 1100 is divided, the variation in the average value per divided area will increase. For this reason, it is necessary to set an appropriate number of divisions N (maximum number of detection circuits N) when dividing the detection area 1100.

[0119] A method for setting the number of divisions N will now be described. Assuming that the estimated variation (σ) obtained by equation (3) is normally distributed, a criterion (k) LSB is set for 4σ as shown in the following equation (4). The predetermined variation (4σ in this example) is set to be less than this criterion (k) LSB, and the number of divisions N (maximum number N) that satisfies equation (4) is set. Note that the value k may be a value that is experimentally determined depending on, for example, the use of the image sensor 10a or imaging device 1a, the required specifications, etc. 4σ<(k) LSB ... (4)

[0120] (3-3. Comparison of Technology According to the Present Disclosure with Existing Technology) Here, the technology according to the present disclosure will be compared with existing technology, and the effects of the technology according to the present disclosure will be described.

[0121] 15A is a schematic diagram illustrating detection of a divided region according to an existing technique. In the existing technique, the selectors 50a, 50b, 50c, ..., 50n that select the pixels 100 for which an average pixel value is to be calculated from the pixels 100 included in the detection region 1100, and the integrator 1210 that calculates the average pixel value are fixed. Therefore, in order to detect periodic black shift and generate correction data, it is necessary to address all factors that are expected to cause periodic black shift.

[0122] 15A shows selectors 50a, 50b, 50c, ..., 50n that select divided regions from the detection region 1100. Each of the selectors 50a, 50b, 50c, ..., 50n passes pixel data of the selected divided region to the accumulator 1210.

[0123] 15A , the selector 50a selects all pixels 100 (all colors) in the detection area 1100. The selector 50a inputs the selected pixel data to one accumulator 1210. The selector 50b selects each subpixel in the detection area 1100. The selector 50b inputs the pixel data of each selected subpixel to, for example, 16 accumulators 1210 for each subpixel. The selector 50c selects pixel data in the detection area 1100, for example, in accordance with the readout order of the AD converter 22. The selector 50c inputs each selected pixel data to, for example, four accumulators 1210, respectively, in accordance with the readout order of the AD converter 22. Furthermore, the selector 50n is dedicated to, for example, a group of pixels 100 that satisfy a specific condition, and inputs the selected pixel data to, for example, the accumulator 1210 dedicated to the selector 50n.

[0124] In the existing technology, a large number of integrators 1210 (67 in the example of FIG. 15A) are required to be able to deal with all of the factors that are assumed to cause periodic black shift.

[0125] FIG. 15B is a schematic diagram illustrating detection of a divided region according to the first embodiment. As described above, in the first embodiment, the division pattern for dividing the detection region 1100 can be flexibly selected by register settings. This makes it possible to narrow down the factors that are assumed to cause periodic black shift and limit the number of accumulators 1210 to N, the number of accumulators 1210 that actually require simultaneous output. In the example of FIG. 15B , N=16, so the number of accumulators 1210 is limited to 16. In response to a selection instruction set by the register settings, the common selector 51 selects pixel data in the detection region 1100 according to the division pattern assigned to each accumulator 1210 and inputs the selected data to each accumulator 1210.

[0126] In the first embodiment, the number of integrators 1210 for calculating the average value of pixel values ​​in a divided region is limited to N, so that it is possible to optimize the gate size in the OPB division detection unit 120.

[0127] (4. Second Embodiment of the Present Disclosure) Next, a second embodiment of the present disclosure will be described. Fig. 16 is a functional block diagram of an example for describing the function of an imaging device 1b according to the second embodiment. In Fig. 16, the imaging device 1b according to the second embodiment includes an image sensor 10b and an application processor 30b.

[0128] 16, the image sensor 10b is configured by adding an OTP (Onetime Programmable ROM) 170, which is a nonvolatile memory, to the image sensor 10a according to the first embodiment shown in FIG. 9. The OTP 170 is a ROM that can be written to only once. In addition, the application processor 30b is configured by deleting the OPB pattern change determination unit 310 from the application processor 30a according to the first embodiment shown in FIG. 9.

[0129] In the imaging device 1a according to the first embodiment described above, the pattern setting of the division pattern for dividing the detection area 1100 is held in the application processor 30a, and the division pattern is transmitted from the application processor 30a to the image sensor 10a in accordance with the shooting information and temperature information.

[0130] In contrast, in the second embodiment, the pattern settings of the division pattern are held on the image sensor 10b side by the OTP 170. That is, the pattern settings of the division pattern determined in advance by the preliminary evaluation are written in advance to the OTP 170. The image sensor 10b selects and reads the pattern settings stored in the OTP 170 based on the shooting information (drive mode, analog gain, exposure time, etc.) transmitted from the camera control unit 320 included in the application processor 30b and the internal temperature of the image sensor 10b measured by a thermometer (not shown), and passes them to the OPB division detection unit 120.

[0131] 12B, the pattern setting signal for setting the division pattern may contain a large amount of data. Therefore, transmission of the pattern setting signal from the application processor 30b to the image sensor 10b via a communication line may impose a large communication load. Therefore, in the second embodiment, the pattern setting signal is held on the image sensor 10b side, thereby reducing the communication load between the image sensor 10b and the application processor 30b.

[0132] Fig. 17 is a block diagram showing an example of the hardware configuration of an image sensor 10b according to the second embodiment. In Fig. 17, the image sensor 10b has an OTP 170 added to the configuration of the image sensor 10a shown in Fig. 10. The pattern setting signal read from the OTP 170 is written to and expanded in a RAM 1302.

[0133] 18 is a timing chart illustrating a process for reading a pattern setting signal from the OTP 170 according to the second embodiment. Here, the OTP 170 holds two pattern setting signals, group A and group B, and switches the pattern setting signals in response to a change in the drive mode of the image sensor 10b. Groups A and B each include information indicating a target pattern that indicates a target division pattern, and information indicating target pixels #1 to #4, which are the pixels 100 to be integrated for calculating an average value.

[0134] Here, the detection region 1100 is provided in each of the first subframe (Subframe #1) and the second subframe (Subframe #2) described with reference to Fig. 13. For example, the detection region 1100 may be located at the top end of the first subframe and the top end of the second subframe.

[0135] 18, in the initial state, the signal XCLR is low, and the hardware (HW) is in a standby state. At time t0, the signal XCLR is set to a high state, the image sensor 10b is started, and the software (SW) including the firmware (FW) described below is set to a standby state. When the image sensor 10b is started, at time t1, the OPB division detection unit 120 accesses the OTP 170, and the pattern setting signals for groups A and B are read from the OTP 170 and written to a register (e.g., RAM 1302) of the image sensor 10b. Thereafter, at time t2, the signal STREAMING is set to a high state.

[0136] At time t3, the firmware of the image sensor 10b instructs the OPB division detection unit 120 to use the pattern setting signal of group A. The firmware is, for example, a program stored in the ROM 1301. In response to the instruction from the firmware, the CPU 1300 reads out the pattern setting signal of group A from the pattern setting signals of groups A and B written to the register at time t1, and writes it to, for example, an internal register of the OPB division detection unit 120.

[0137] The OPB division detection unit 120 selects pixel data output from the detection region 1100 of the first subframe in the pixel array unit 110 in accordance with the pattern setting signal of group A written to the internal register, and calculates an average value #1 of the pixel values ​​by accumulating the data. 10 Then, in accordance with the calculation completion pulse output from the integrator 1210 indicating the completion of calculation of the average value #1 for the first subframe, the calculated average value #1 is written into an area corresponding to the average value #1 in the output register of the OPB division detection unit 120. 11 Then, in accordance with the calculation completion pulse indicating the completion of calculation of average value #2 for the second subframe, the calculated average value #2 is written into the area corresponding to average value #2 in the output register of the OPB division detection unit 120.

[0138] Here, it is assumed that at time t4, the firmware is notified of a change in the drive mode of the image sensor 10b to the OPB division detection unit 120 and is instructed to switch the pattern setting signal to be used from group A to group B. Of the pattern setting signals for groups A and B written to the register at time t4, the CPU 1300 reads out the pattern setting signal for group B and overwrites it into the internal register of the OPB division detection unit 120.

[0139] The OPB division detection unit 120 selects pixel data output from the detection region 1100 in the first subframe in the pixel array unit 110 in accordance with the pattern setting signal for group B written to the internal register, and accumulates the data to calculate an average value #1 of the pixel values. 12 Then, in accordance with the calculation completion pulse indicating the completion of calculation of the average value #1 for the first subframe, the calculated average value #1 is written into an area corresponding to the average value #1 in the output register of the OPB division detection unit 120. Similarly, the OPB division detection unit 120 writes the calculated average value #1 into an area corresponding to the average value #1 in the output register of the OPB division detection unit 120 at time t 13 Then, in accordance with the calculation completion pulse indicating the completion of calculation of average value #2 for the second subframe, the calculated average value #2 is written into the area corresponding to average value #2 in the output register of the OPB division detection unit 120.

[0140] In this way, the imaging device 1b according to the second embodiment is provided with the OTP 170 in the image sensor 10b, and the pattern setting signal is stored in the OTP 170. Therefore, the imaging device 1b according to the second embodiment does not increase the communication load between the image sensor 10b and the application processor 30b even if the data amount of the pattern setting signal is large.

[0141] (5. Third Embodiment of the Present Disclosure) Next, a third embodiment of the present disclosure will be described. In the first and second embodiments described above, the application processor 30 a or 30 b includes the black shift correction unit 300, and the correction process for periodic black shift is performed in the application processor 30 a or 30 b. In contrast, in the third embodiment, the black shift correction unit 300 is provided within the image sensor 10.

[0142] 19 is a functional block diagram illustrating an example of functions of an imaging device 1c according to the third embodiment. In FIG. 19, the imaging device 1c according to the third embodiment includes an image sensor 10c and an application processor 30c.

[0143] 16, the image sensor 10c is configured by adding a black shift correction unit 300, an OPB pattern change determination unit 310, and a subsequent image signal processing unit 330 to the image sensor 10a according to the first embodiment shown in Fig. 9. Note that the pattern RAM 311 included in the OPB pattern change determination unit 310 is omitted from the figure. On the other hand, the application processor 30c is configured by removing the black shift correction unit 300 and the OPB pattern change determination unit 310 from the application processor 30a according to the first embodiment shown in Fig. 9.

[0144] In the application processor 30c, the camera control unit 320 transmits imaging information, including imaging-related information such as the drive mode, analog gain, and exposure time of the image sensor 10c, to the image sensor 10c via a communication line in response to imaging. The image sensor 10c receives the imaging information transmitted from the application processor 30c via the interface 161 and passes it to the OPB pattern change determination unit 310. The OPB pattern change determination unit 310 selects and reads a pattern setting signal from a pattern RAM 311 (not shown) based on the imaging information passed from the interface 161, and passes the read pattern setting signal to the OPB division detection unit 120.

[0145] The processing by the OPB division detection unit 120 and the black shift correction unit 300 is the same as the processing described in the first embodiment, and therefore detailed description thereof will be omitted here. The image data in which the periodic black shift has been corrected by the black shift correction unit 300 is subjected to predetermined image processing in the subsequent image signal processing unit 330 and transmitted to the application processor 30c via a communication line. The application processor 30c may input the image data transmitted from the image sensor 10c to an image processing unit such as an ISP (Image Signal Processor) (not shown).

[0146] (6. Application Examples of Each Embodiment of the Present Disclosure) Next, application examples of each embodiment of the present disclosure will be described.

[0147] (Application Example to a Mobile Body) The technology according to the present disclosure (the present technology) can be applied to various products. For example, the present technology may be applied to a digital camera as a consumer product, or a camera mounted on a smartphone or tablet computer. Furthermore, for example, the present technology may be applied to a fixedly installed camera such as a surveillance camera. Furthermore, for example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0148] FIG. 20 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0149] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 20 , the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0150] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0151] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0152] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0153] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0154] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0155] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0156] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0157] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0158] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 20, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0159] FIG. 21 is a diagram showing an example of the installation position of the imaging unit 12031.

[0160] In FIG. 21, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0161] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0162] 21 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0163] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0164] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0165] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0166] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0167] The foregoing describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the imaging unit 12031 of the above-described configuration. Specifically, the imaging devices 1a to 1c according to each embodiment of the present disclosure can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to obtain a captured image that is easier to see, thereby reducing driver fatigue. Furthermore, the technology according to the present disclosure to the imaging unit 12031 can reduce the amount of communication via the communication line between the image sensor and the application processor, thereby reducing battery consumption, for example.

[0168] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0169] Note that the present technology can also be configured as follows. (1) An imaging device including: a dividing unit that divides a detection area provided in a pixel array unit of an imaging unit in order to measure a reference level of pixel data based on signals output from pixels included in the pixel array unit; and a detection unit that detects the signals output from the pixels included in the divided areas into which the detection area is divided by the dividing unit and generates detection data, wherein a division pattern for dividing the detection area into the divided areas is set in a programmable manner in the dividing unit. (2) The imaging device according to (1), wherein the dividing unit sets the divided areas by dividing the detection area into a maximum of N (N is an integer of 2 or greater) areas, and the detection unit is provided for each of the divided areas. (3) The imaging device according to (2), wherein N is determined based on a variation value calculated in advance based on the pixel data of the pixels included in the divided areas. (4) The imaging device according to any one of (1) to (3), further comprising: a determination unit that determines the division pattern to be applied based on at least one of shooting information related to shooting by the imaging unit and an internal temperature of the imaging unit. (5) The imaging device according to (4), wherein the shooting information includes at least one of a drive mode for driving the imaging unit, an analog gain for the signal, and an exposure time for the shooting. (6) The imaging device according to any one of (1) to (5), wherein the division unit generates the division pattern by combining two or more basic patterns that are different from one another. (7) The imaging device according to any one of (1) to (6), wherein the division unit divides the detection area based on the division pattern pre-stored in a non-volatile memory. (8) The imaging device according to any one of (1) to (7), further comprising: a generation unit that generates correction data for correcting the level of the pixel data based on the reference level and the level of the pixel data of the pixels included in the divided area. (9) The imaging device according to any one of (1) to (8), further comprising: the imaging unit including the dividing unit and the detection unit; and an application unit including a control unit that controls the operation of the imaging unit and that communicates with the imaging unit via a communication line.(10) The imaging device according to (9), wherein the application unit further includes: a correction unit that corrects the level of the pixel data based on correction data for correcting the level of the pixel data, the correction data being generated based on the reference level and the level of the pixel data of the pixels included in the divided area. (11) The imaging device according to (10), wherein the imaging unit further includes: a non-volatile memory in which the division pattern is stored in advance. (12) The imaging device according to (9) or (10), wherein the application unit further includes: a determination unit that determines the division pattern to apply based on at least one of shooting information related to shooting by the imaging unit and an internal temperature of the imaging unit. (13) The imaging device according to (9), wherein the imaging unit further includes: a correction unit that corrects the level of the pixel data based on correction data for correcting the level of the pixel data, the correction data being generated based on the reference level and the level of the pixel data of the pixels included in the divided area, and a determination unit that determines the division pattern to apply based on at least one of shooting information related to shooting by the imaging unit and an internal temperature of the imaging unit. (14) A control method for an imaging device, comprising: a division step of dividing a detection area provided in a pixel array section of an imaging section in order to measure a reference level of pixel data based on signals output from pixels included in the pixel array section; and a detection step of detecting the signals output from the pixels included in the divided areas into which the detection area is divided by the division step to generate pixel data, wherein in the division step, a division pattern for dividing the detection area into the divided areas is set in a programmable manner.

[0170] 1, 1a, 1b, 1c, 600 Imaging device 10, 10a, 10b, 10c Image sensor 11 Optical section 12 Image processing section 13 Output processing section 14, 25 Control section 100, 100a, 100b, 100c, 100d Pixel 110 Pixel array section 150, 1501, 1502, 1503, 1504 Pixel block 21 Horizontal scanning / AD conversion section 22, 221, 222, 223, 224 AD converter 401, 402, 403, 404, 405, 406, 407, 408 Correction data 50a, 50b, 50c, 50n Selector 51 Common selector 120 OPB division detection section 130 Image signal processing section 160 Thermometer 161 Interface 170 OTP 300 Black shift correction unit 310 OPB pattern change determination unit 311 Pattern RAM 320 Camera control unit 1100 Detection area 1300 CPU 1301 ROM 1302 RAM 1303 OPB detection circuit 1304 Image signal processing circuit 1200 Division area selection unit 1201 Accumulation unit 1202 Output data 1210, 1210#1, 1210#15 Accumulator

Claims

1. An imaging device comprising: a dividing unit that divides a detection area provided in a pixel array unit of an imaging unit in order to measure a reference level of pixel data based on signals output from pixels included in the pixel array unit; and a detection unit that detects the signals output from the pixels included in the divided areas into which the detection area is divided by the dividing unit and generates detection data, wherein the dividing unit has a division pattern that is programmable to divide the detection area into the divided areas.

2. The imaging device according to claim 1, wherein the division unit divides the detection area into a maximum of N areas (N is an integer of 2 or greater) to set the divided areas, and the detection unit is provided for each of the divided areas.

3. The imaging device according to claim 2, wherein N is determined based on a variation value calculated in advance based on the pixel data of the pixels included in the divided region.

4. The imaging device according to claim 1, further comprising a determination unit that determines the division pattern to be applied based on at least one of shooting information related to shooting by the imaging unit and the internal temperature of the imaging unit.

5. The imaging device according to claim 4, wherein the shooting information includes at least one of a drive mode for driving the imaging unit, an analog gain for the signal, and an exposure time for the shooting.

6. The imaging device according to claim 1, wherein the division section generates the division pattern by combining two or more basic patterns that are different from one another.

7. The imaging device according to claim 1, wherein the dividing section divides the detection area based on the division pattern stored in advance in a non-volatile memory.

8. The imaging device according to claim 1, further comprising a generation unit that generates correction data for correcting the level of the pixel data based on the reference level and the level of the pixel data of the pixels included in the divided area.

9. The imaging device according to claim 1, comprising: an imaging unit including the dividing unit and the detection unit; and an application unit including a control unit that controls the operation of the imaging unit and that communicates with the imaging unit via a communication line.

10. The imaging device of claim 9, wherein the application unit further includes a correction unit that corrects the level of the pixel data based on correction data for correcting the level of the pixel data, the correction data being generated based on the reference level and the level of the pixel data from the pixels included in the divided area.

11. The imaging device according to claim 10, wherein the imaging unit further includes a non-volatile memory in which the division pattern is stored in advance.

12. The imaging device according to claim 9, wherein the application unit further includes a determination unit that determines the division pattern to be applied based on at least one of shooting information related to shooting by the imaging unit and the internal temperature of the imaging unit.

13. The imaging device of claim 9, wherein the imaging unit further includes: a correction unit that corrects the level of the pixel data based on correction data for correcting the level of the pixel data, the correction data being generated based on the reference level and the level of the pixel data of the pixels included in the divided area; and a determination unit that determines the division pattern to be applied based on at least one of shooting information related to shooting by the imaging unit and the internal temperature of the imaging unit.

14. A control method for an imaging device, comprising: a division step of dividing a detection area provided in a pixel array section of an imaging section in order to measure a reference level of pixel data based on signals output from pixels included in the pixel array section; and a detection step of detecting the signals output from the pixels included in the divided areas into which the detection area is divided by the division step to generate pixel data, wherein the division step sets a division pattern for dividing the detection area into the divided areas in a programmable manner.

Images