Image sensor, image processing system, and image sensor control method

Abstract

In this system for processing an image, a light source distribution is estimated in real time.　This image sensor comprises pixels, an analog-to-digital converter, and an interface. The pixels generate analog signals as pixel signals. The analog-to-digital converter performs processing to determine whether the levels of the pixel signals are higher than a prescribed threshold value and generate flag information indicating the determination result, and processing to convert the pixel signals into digital signals and output the result as luminance information. The interface externally outputs the luminance information and the flag information.

Description

Image sensor, image processing system, and image sensor control method

[0001] The present technology relates to an image sensor, and more particularly to an image sensor for performing inverse rendering, an image processing system, and a method for controlling the image sensor.

[0002] Conventionally, inverse rendering technology has been used to automatically generate highly realistic synthetic images. In this inverse rendering technology, the shape, reflection characteristics, and light source distribution of an object are estimated from one or more images. For example, an information processing device has been proposed that acquires meta information including the shooting position and generates virtual object information by inverse rendering technology using a background image and the meta information (see, for example, Patent Document 1).

[0003] International Publication No. 2020 / 045092

[0004] The above-mentioned conventional technology attempts to display a virtual object at an appropriate position by using meta-information such as the image capture position. However, when estimating the light source distribution using the inverse rendering technology, the above-mentioned conventional technology requires a background image captured in advance before compositing. This makes it difficult for the system to estimate the light source distribution from an image every time it captures an image, in other words, to estimate the light source distribution in real time. Therefore, the above-mentioned conventional technology has limitations that make it difficult to use in scenes that require immediacy, such as live video.

[0005] This technology was developed in light of these circumstances, and aims to estimate the light source distribution in real time in image processing systems.

[0006] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof is an image sensor including a pixel that generates an analog signal as a pixel signal, an analog-to-digital converter that performs a process of determining whether or not the level of the pixel signal is higher than a predetermined threshold and generating flag information indicating the determination result, and a process of converting the pixel signal into a digital signal and outputting the digital signal as luminance information, and an interface that outputs the luminance information and the flag information to an external device, and a control method thereof, which provides the effect of estimating a light source distribution in real time.

[0007] In this first aspect, the threshold value may include a first threshold value, and the flag information may include a reset level determination flag indicating whether a reset level, which is the level of the pixel signal when the pixel is initialized, is higher than the first threshold value, thereby providing an effect of determining whether sunlight or the like is incident.

[0008] In this first aspect, the flag information may further include address information of each pixel, thereby enabling a subsequent circuit to acquire the position of the light source.

[0009] In this first aspect, the flag information may further include an adjacent pixel determination flag indicating, for each pixel, whether the reset level of a pixel adjacent to the pixel is higher than the first threshold value, thereby improving the accuracy of estimating the light source distribution.

[0010] In this first aspect, the threshold value may further include a second threshold value, and the flag information may further include a signal level determination flag indicating whether a signal level, which is a level of the pixel signal according to an amount of exposure, is higher than the second threshold value, thereby providing an effect of estimating the amount of light from the light source.

[0011] In this first aspect, the interface may sequentially output the luminance information and the flag information via a predetermined serial transmission line, thereby providing an effect that only one transmission line is required for data transfer.

[0012] In the first aspect, the interface may output the luminance information via a first transmission line and the flag information via a second transmission line, thereby improving the transfer speed.

[0013] In addition, in this first aspect, the analog-to-digital converter may include a comparator that compares the pixel signal with a predetermined reference signal and outputs the comparison result, a column counter that generates and outputs the luminance information based on the comparison result, a determination unit that determines whether the level of the pixel signal is higher than the threshold and generates flag information indicative of the determination result, and a correction unit that corrects the luminance information when the level of the pixel signal is lower than the threshold, thereby providing the effect of preventing sunspots from occurring.

[0014] According to a second aspect of the present technology, there is provided an image processing system including: a pixel that generates an analog signal as a pixel signal; an analog-to-digital converter that performs a process of determining whether a level of the pixel signal is higher than a predetermined threshold and generating flag information indicating the determination result, and a process of converting the pixel signal into a digital signal and outputting the digital signal as luminance information; an image sensor that includes an interface that outputs the luminance information and the flag information to an external device; and a signal processing circuit that performs predetermined image processing on a frame in which the luminance information is arranged using the flag information, thereby providing an effect that a light source distribution is estimated in real time in the signal processing circuit.

[0015] In the second aspect, the image processing may include processing for estimating a light source distribution, thereby providing an effect that the light source distribution is estimated in real time.

[0016] In addition, in this second aspect, the image processing may include a process of adjusting white balance, thereby improving image quality.

[0017] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment of the present technology. FIG. 2 is a block diagram showing an example of a configuration of an image sensor according to the first embodiment of the present technology. FIG. 3 is a circuit diagram showing an example of a configuration of a pixel according to the first embodiment of the present technology. FIG. 4 is a block diagram showing an example of a configuration of a clipping circuit and a column signal processing unit according to the first embodiment of the present technology. FIG. 5 is a circuit diagram showing an example of a configuration of an AD (Analog to Digital) converter according to the first embodiment of the present technology. FIG. 6 is a block diagram showing an example of a configuration of a column counter according to the first embodiment of the present technology. FIG. 7 is a timing chart showing an example of an operation of an image sensor when a reset level is higher than a threshold value according to the first embodiment of the present technology. FIG. 8 is a timing chart showing an example of an operation of an image sensor when a reset level is equal to or lower than a threshold value according to the first embodiment of the present technology. FIG. 9 is a diagram for explaining a data transfer method according to the first embodiment of the present technology. FIG. 10 is a diagram showing an example of an estimation result of a light source distribution according to the first embodiment of the present technology. FIG. 11 is a flowchart showing an example of an operation of an image sensor according to the first embodiment of the present technology. FIG. 11 is a flowchart showing an example of an operation of a DSP (Digital Signal Processing) circuit according to the first embodiment of the present technology. FIG. 12 is a diagram for explaining a data transfer method according to a first modified example of the first embodiment of the present technology. FIG. 1 is a flowchart showing an example of operation of a DSP circuit in a first modified example of the first embodiment of the present technology. FIG. 2 is a diagram showing an example of an estimation result of a light source distribution in a first modified example of the first embodiment of the present technology. FIG. 3 is a diagram for explaining a data transfer method in a second modified example of the first embodiment of the present technology. FIG. 4 is a circuit diagram showing an example configuration of an AD converter in a third modified example of the first embodiment of the present technology. FIG. 5 is a timing chart showing an example operation of an image sensor in a third modified example of the first embodiment of the present technology. FIG. 6 is a block diagram showing an example configuration of a DSP circuit in a second embodiment of the present technology. FIG. 7 is a diagram for explaining processing of an image processing system in a second comparative example. FIG. 8 is a diagram for explaining processing of a DSP circuit in the second embodiment of the present technology. FIG. 9 is a block diagram showing an example of a schematic configuration of a vehicle control system.FIG. 2 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit.

[0018] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order: 1. First embodiment (an example in which an image sensor outputs flag information) 2. Second embodiment (an example in which an image sensor outputs flag information and a DSP circuit performs white balance adjustment using the flag information) 3. Example of application to a moving body

[0019] 1 is a block diagram showing an example of the configuration of an image processing system 100 according to a first embodiment of the present technology. The image processing system 100 is a system that performs various types of image processing, such as synthesis processing, on frames that are image data. Examples of the image processing system 100 include a smartphone, an AR (Augmented Reality) terminal, a VR (Virtual Reality) terminal, and in-vehicle equipment.

[0020] The image processing system 100 includes an image sensor 200, a storage unit 110, a DSP circuit 120, and a display unit 130. These circuits can be arranged in a single device, or can be distributed across multiple devices.

[0021] The image sensor 200 captures frames in synchronization with a vertical synchronization signal VSYNC. The frames include luminance information and flag information for each pixel. Details of the flag information will be described later. The image sensor 200 transmits the captured frames to the DSP circuit 120 via a serial transmission line 209.

[0022] The storage unit 110 stores one or more captured frames. These frames may be captured in advance by the image sensor 200 or may be captured by another device.

[0023] The DSP circuit 120 performs various image processing on frames. The DSP circuit 120 includes, for example, a frame memory 121, an inverse rendering processing unit 122, a rendering processing unit 123, and a compositing processing unit 124. The DSP circuit 120 is an example of a signal processing circuit as defined in the claims.

[0024] The frame memory 121 stores frames from the image sensor 200 .

[0025] The inverse rendering processing unit 122 generates virtual object information, reflection characteristic information, and light source distribution estimation information using inverse rendering technology and supplies them to the rendering processing unit 123. The virtual object information is information that indicates the geometric shape of the virtual object to be synthesized. The reflection characteristic information is information that indicates the reflection characteristics of the surface of the virtual object. The light source distribution information is information that indicates the estimation result of the light source distribution. Of these, the light source distribution information is generated from flag information. For this reason, already-captured frames are not used to estimate the light source distribution. On the other hand, already-captured frames are used to estimate the geometric shape and reflection characteristics of the virtual object. Note that the inverse rendering processing unit 122 can also estimate only one of the virtual object information and the reflection characteristics.

[0026] The rendering processing unit 123 performs rendering. The rendering processing unit 123 places a virtual object model in a three-dimensional space and applies texturing and lighting to it based on the virtual object information, reflection characteristic information, and light source distribution information from the inverse rendering processing unit 122. The rendering processing unit 123 then converts the textured and lit model into a two-dimensional image by rendering, and supplies the two-dimensional image to the composition processing unit 124.

[0027] The composition processing unit 124 composes the two-dimensional image from the rendering processing unit 123 with the frame in the frame memory 121. The composition processing unit 124 supplies the composite frame to the display unit 130 for display.

[0028] The inverse rendering processing unit 122 can further estimate the geometric shape and reflection characteristics of a virtual object in three-dimensional space based on one or more frames. The frames for estimating the shape and reflection characteristics may be captured by the image sensor 200 or by another device.

[0029] 2 is a block diagram showing an example of the configuration of an image sensor 200 according to the first embodiment of the present technology. The image sensor 200 includes a vertical drive unit 211, a system control unit 212, a DAC (Digital to Analog Converter) 213, and a pixel array unit 214. The image sensor 200 further includes a clipping circuit 250, a column signal processing unit 300, a horizontal drive unit 215, a digital calculation unit 216, and an interface 217. In addition, a plurality of pixels 220 are arranged in a two-dimensional lattice pattern in the pixel array unit 214, and vertical signal lines VSL are wired in the vertical direction for each column.

[0030] The vertical drive section 211 sequentially selects rows in the pixel array section 214 and drives the pixels 220 in the selected rows to output analog pixel signals.

[0031] The system control unit 212 controls the operation timing of each of the vertical drive unit 211, the DAC 213, the clip circuit 250, the column signal processing unit 300, and the horizontal drive unit 215 in synchronization with a vertical synchronization signal VSYNC.

[0032] The DAC 213 generates a predetermined reference signal by DA (Digital to Analog) conversion and supplies the reference signal to the column signal processing unit 300. For example, a sawtooth ramp signal is used as the reference signal.

[0033] The pixel 220 generates a pixel signal by photoelectric conversion and outputs the pixel signal to the column signal processing unit 300 via a vertical signal line VSL.

[0034] The clip circuit 250 fixes the potential of the vertical signal line VSL at a predetermined timing.

[0035] The column signal processing unit 300 performs various signal processing such as AD conversion and CDS (Correlated Double Sampling) processing on pixel signals for each column. The column signal processing unit 300 supplies the processed digital signals to the digital calculation unit 216.

[0036] The horizontal drive unit 215 drives the column signal processing unit 300 to output the digital signals for each column to the digital calculation unit 216 in order.

[0037] The digital calculation unit 216 arranges the digital signals from the column signal processing unit 300 to generate frames. The digital calculation unit 216 performs various calculations on the frames as necessary and supplies the frames to an interface 217.

[0038] The interface 217 outputs frames to the DSP circuit 120 via the serial transmission line 209 .

[0039] 3 is a circuit diagram showing an example of the configuration of a pixel 220 according to the first embodiment of the present technology. The pixel 220 includes a photoelectric conversion element 221, a transfer transistor 222, a reset transistor 223, a floating diffusion layer 224, an amplification transistor 225, and a selection transistor 226. The transfer transistor 222, the reset transistor 223, the amplification transistor 225, and the selection transistor 226 are, for example, nMOS (n-channel Metal Oxide Semiconductor) transistors.

[0040] The photoelectric conversion element 221 generates signal charges from incident light by photoelectric conversion.

[0041] The transfer transistor 222 transfers signal charges from the photoelectric conversion element 221 to the floating diffusion layer 224 in accordance with a transfer signal TG from the vertical drive section 211 .

[0042] The reset transistor 223 extracts charges from the floating diffusion layer 224 to initialize it in accordance with a reset signal RST from the vertical drive unit 211 .

[0043] The floating diffusion layer 224 accumulates signal charges and generates a voltage according to the amount of charge.

[0044] The amplifying transistor 225 constitutes a source follower circuit and outputs an analog signal having a voltage corresponding to the voltage of the floating diffusion layer 224 as a pixel signal SIG.

[0045] The selection transistor 226 outputs a pixel signal SIG to a vertical signal line VSL in accordance with a selection signal SEL from the vertical drive section 211. The pixel signal SIG is transmitted to the column signal processing section 300 via the vertical signal line VSL.

[0046] The level of the pixel signal SIG when the floating diffusion layer 224 is initialized is called a reset level or a P-phase level. The level of the pixel signal SIG according to the amount of exposure when the signal charge is transferred is called a signal level or a D-phase level.

[0047] The circuit configuration of the pixel 220 is not limited to the circuit shown in the figure, as long as it can generate a pixel signal. For example, the floating diffusion layer 224 can be shared by multiple pixels. Alternatively, a floating diffusion (FD) gain transistor that changes the charge-to-voltage conversion efficiency can be added.

[0048] 4 is a block diagram showing an example of the configuration of the clipping circuit 250 and the column signal processing unit 300. The clipping circuit 250 includes a p-channel metal oxide semiconductor (pMOS) transistor 251 and an nMOS transistor 252 for each column. These transistors are connected in series between the power supply voltage VDD and the corresponding vertical signal line VSL.

[0049] An inverted signal XPEN from the system control unit 212 is input to the gate of the pMOS transistor 251. The inverted signal XPEN is an inverted signal of an enable signal that indicates whether or not the clip function is enabled, and is set to a low level when the clip function is enabled. A predetermined bias voltage is applied to the gate of the nMOS transistor 252 as a clip voltage VP.

[0050] The column signal processing section 300 includes a Gray code counter 310, a plurality of AD converters 320, a plurality of memories 370, and an output section 380. The AD converters 320 and the memories 370 are arranged for each column.

[0051] The Gray code counter 310 performs counting in synchronization with a clock signal ADCK from the system control unit 212 and outputs a Gray code GC indicating the count value. The Gray code counter 310 supplies the Gray code GC to the AD converter 320 of each column. For example, the Gray code counter 310 is arranged for every M (M is an integer) columns and is shared by the corresponding M AD converters 320.

[0052] The AD converter 320 converts the pixel signal SIG from the vertical signal line VSL of the corresponding column into a digital signal and outputs it as brightness information BIN. A binary code, for example, is used as a code representing the brightness information BIN. The AD converter 320 also generates a P-phase determination flag XPO, which is output to and stored in the memory 370 together with the brightness information BIN. The P-phase determination flag XPO is set when the reset level (in other words, the P-phase level) is equal to or lower than a predetermined threshold value Th. Ｐ The P-phase determination flag is one bit of information indicating whether the threshold Th is higher than the P-phase determination flag. Ｐ is an example of a first threshold value described in the claims.

[0053] The memory 370 holds the luminance information BIN and the P-phase determination flag XPO, and outputs them to the output section 380 under the control of the horizontal drive section 215 .

[0054] The output section 380 outputs the data from the memory 370 to the digital calculation section 216 .

[0055] 5 is a circuit diagram showing an example of the configuration of the AD converter 320 according to the first embodiment of the present technology. The AD converter 320 includes a comparator 321, an inverter 322, a P-phase determination unit 330, a correction unit 340, and a column counter 350.

[0056] The comparator 321 compares the ramp signal RMP from the DAC 213 with the pixel signal SIG from the corresponding column. The comparator 321 supplies the comparison result VCO to the inverter 322.

[0057] The inverter 322 inverts the comparison result VCO and supplies it as an inverted signal XVCO to the P-phase determination section 330 and the correction section 340 .

[0058] The P-phase determination unit 330 determines whether the reset level (P-phase level) is a predetermined threshold value Th Ｐ The P-phase determination unit 330 determines whether the P-phase is higher than the P-phase, and generates a P-phase determination flag XPO indicating the determination result. The P-phase determination unit 330 includes a latch circuit 331.

[0059] The latch circuit 331 holds the inverted signal XVCO. The inverted signal XVCO from the inverter 322 is input to an input terminal D of this latch circuit 331, and the pulse signal PLAT from the system control unit 212 is input to a clock terminal. Furthermore, the inverted value of the inverted signal XPRST from the system control unit 212 is input to a reset terminal of the latch circuit 331. A 1-bit signal indicating the held value is output from an output terminal Q of the latch circuit 331 to the correction unit 340 and the memory 370 as a P-phase determination flag XPO.

[0060] The latch circuit 331 sets the held value to an initial value (such as a logical value "0") when the inverted signal XRST is at a low level. Furthermore, when the inverted signal XRST is at a high level and the pulse signal PLAT is at a high level, the latch circuit 331 updates the held value using the inverted signal XVCO. Furthermore, when the inverted signal XRST is at a high level and the pulse signal PLAT is at a low level, the latch circuit 331 transitions to a hold state and latches the held value.

[0061] The correction unit 340 corrects the P-phase level by the threshold value Th Ｐ If the gray code value is higher than the reference value, the gray code value is corrected to the full scale. The correcting section 340 includes a logic gate 341 and a pulse generating section 342.

[0062] The logic gate 341 generates a NOR of the logical product of the P-phase determination flag XPO and the inverted signal XVCO and the control signal EXVCOL from the system control unit 212, and outputs the result to the pulse generation unit 342 as the internal signal VCOD.

[0063] The pulse generating section 342 generates a pulse signal GCLAT at the timing of the falling edge of the internal signal VCOD, and supplies it to the column counter 350 .

[0064] The column counter 350 receives and latches the gray code GC from the gray code counter 310 at the timing when the pulse signal GCLAT is output, and converts the gray code GC into a binary code. The column counter 350 then supplies the converted binary code to the memory 370 as luminance information BIN.

[0065] 6 is a block diagram showing an example of the configuration of the column counter 350 according to the first embodiment of the present technology. The column counter 350 includes a Gray code latch 351, an ALU (Arithmetic and Logic Unit) 352, a temporary latch 355, and an interface latch 356. The ALU 352 includes a code converter 353 and a full adder 354.

[0066] The Gray code latch 351 latches the Gray code GC at the timing when the pulse signal GCLAT is output. The data size of the Gray code GC is, for example, 11 bits.

[0067] The code converter 353 converts the latched Gray code into a binary code and outputs it to the full adder 354. The full adder 354 adds the binary code held in the temporary latch 355 to the binary code from the code converter 353 and outputs the result to the temporary latch 355 and the interface latch 356.

[0068] When the pixel 220 is initialized, the Gray code latch 351 latches a Gray code of the reset level (P phase). In the initial state, a binary code in which all bits have a logical value of "0" is held in the temporary latch 355. The code converter 353 converts the P phase Gray code into a binary code. The full adder 354 adds the P phase binary code to the binary code in which all bits have a logical value of "0", and holds the result in the temporary latch 355.

[0069] Then, when the signal charge is transferred, the Gray code latch 351 latches the Gray code of the signal level (D phase). The code converter 353 converts the D phase Gray code into a binary code. The full adder 354 adds (effectively subtracts) the D phase binary code to the P phase binary code held in the temporary latch 355, and holds the result in the interface latch 356. This realizes CDS processing to find the difference between the P phase and the D phase.

[0070] The interface latch 356 outputs the held binary code to the memory 370 as luminance information BIN.

[0071] Although the column counter 350 performs conversion to binary code and CDS processing, some or all of these processes can be performed by subsequent circuits.

[0072] [Example of Operation of Image Sensor] FIG. 7 is a diagram showing an example of an image sensor in which the reset level (P-phase level) is equal to or exceeds the threshold Th Ｐ10 is a timing chart showing an example of the operation of the image sensor when the potential of the vertical signal line VSL is higher than that of the vertical signal line VSL, and the broken solid line shows the potential of the ramp signal RMP.

[0073] During the auto-zero period from timing T0, the system control unit 212 sets the inverted signal XPEN to low level, thereby clipping the potential of the vertical signal line VSL to a predetermined potential.

[0074] The ramp signal RMP gradually decreases during the P phase conversion period from timing T1 to T3. During this P phase conversion period, the system control unit 212 supplies a clock signal ADCK. The Gray code is updated in synchronization with this clock signal ADCK.

[0075] In the figure, it is assumed that extremely bright light such as sunlight is not incident on the pixel to be read out. In this case, the potential of the vertical signal line VSL during the P-phase conversion period, i.e., the P-phase level, is below the threshold value Th Ｐ Then, at timing T2 within the P-phase conversion period, the ramp signal RMP becomes equal to or lower than the potential of the vertical signal line VSL (in other words, the P-phase level). At this timing T2, the comparison result VCO is inverted from high level to low level, and becomes high level at timing T3.

[0076] Furthermore, the system control unit 212 changes the control signal EXVCOL from high level to low level at timing T1, and changes the control signal EXVCOL to high level at timing T3.

[0077] The logic gate 341 changes the internal signal VCOD from low level to high level at timing T1 based on the inverted signal XVCO, the control signal EXVCOL, and the P-phase determination flag XPO, and changes the internal signal VCOD to low level at timing T2.

[0078] The pulse generating unit 342 outputs a high-level pulse signal GCLAT for a pulse period from timing T2 when the internal signal VCOD falls, and the P-phase Gray code is latched at this timing T2.

[0079] From timing T4 immediately after timing T3, the system control unit 212 supplies a pulse signal PLAT over a pulse period. As a result, the P-phase determination unit 330 latches a high level and outputs a high-level P-phase determination flag XPO. Here, if the level of the ramp signal RMP at timing T3 is equal to or higher than the aforementioned threshold value Th Ｐ The P-phase determination flag XPO is at a high level when the P-phase level is higher than the threshold value Th Ｐ indicates that it is higher than

[0080] Then, at timing T3, the ramp signal RMP is initialized and gradually decreases during the D-phase conversion period from timing T6 to T8. Over this D-phase conversion period, the system control unit 212 supplies the clock signal ADCK. The Gray code is updated in synchronization with this clock signal ADCK. Then, at timing T7 within the D-phase conversion period, the ramp signal RMP becomes equal to or lower than the potential of the vertical signal line VSL (in other words, the D-phase level). At this timing T7, the comparison result VCO inverts from high to low, and then becomes high at timing T8.

[0081] Furthermore, the system control unit 212 changes the control signal EXVCOL from high level to low level at timing T6, and changes the control signal EXVCOL to high level at timing T8.

[0082] The logic gate 341 changes the internal signal VCOD from low level to high level at timing T6 based on the inverted signal XVCO, the control signal EXVCOL, and the P-phase determination flag XPO, and changes the internal signal VCOD to low level at timing T7.

[0083] The pulse generating unit 342 outputs a high-level pulse signal GCLAT for a pulse period from timing T7 when the internal signal VCOD falls, at which time the D-phase Gray code is latched.

[0084] As shown in the figure, when high-intensity light is not incident on the pixel and the P-phase level is below the threshold value Th ＰIf the P-phase conversion period and the D-phase conversion period are higher than the P-phase conversion period, the gray code is latched when the comparison result VCO is inverted. The AD converter 320 then converts the P-phase and D-phase gray codes into binary codes, calculates the difference between them, and outputs the binary code of the difference as brightness information together with the P-phase determination flag XPO.

[0085] FIG. 8 is a graph showing a state in which the reset level (P phase level) is equal to or exceeds a threshold value Th Ｐ 1 is a timing chart showing an example of the operation of the image sensor in the following cases: In the figure, the dashed dotted line indicates the locus of the potential of the vertical signal line VSL, and the broken solid line indicates the locus of the ramp signal RMP.

[0086] In the figure, it is assumed that extremely bright light such as sunlight is incident on the pixel to be read. In this case, when the pixel is reset, signal charges leak from the photoelectric conversion element 221, causing the P-phase level to exceed the threshold value Th Ｐ It becomes the following.

[0087] During the auto-zero period from timing T0, the system control unit 212 sets the inverted signal XPEN to a low level to clip the potential of the vertical signal line VSL to a predetermined potential. After the clipping is released, the potential of the vertical signal line VSL drops again and reaches the threshold value Th Ｐ Let us assume that the following has been achieved.

[0088] Over the P phase conversion period from timing T1 to T3, the comparison result VCO is not inverted and remains at a high level.

[0089] Furthermore, the system control unit 212 changes the control signal EXVCOL from high level to low level at timing T1, and changes the control signal EXVCOL to high level at timing T3.

[0090] The logic gate 341 changes the internal signal VCOD from low level to high level at timing T1 based on the inverted signal XVCO, the control signal EXVCOL, and the P-phase determination flag XPO, and changes the internal signal VCOD to low level at timing T3.

[0091] The pulse generating unit 342 outputs a high-level pulse signal GCLAT for a pulse period from timing T3 when the internal signal VCOD falls, and the P-phase Gray code is latched at this timing T3.

[0092] From timing T4 immediately after timing T3, the system control unit 212 supplies a pulse signal PLAT over a pulse period. As a result, the P-phase determination unit 330 latches a low level and outputs a low-level P-phase determination flag XPO. This low-level P-phase determination flag XPO is output when the P-phase level is lower than the threshold value Th Ｐ Show that:

[0093] Then, at timing T7' within the D phase change period, the ramp signal RMP becomes equal to or lower than the D phase level. At this timing T7', the comparison result VCO is inverted from high level to low level, and becomes high level at timing T8.

[0094] Furthermore, the system control unit 212 changes the control signal EXVCOL from high level to low level at timing T6, and changes the control signal EXVCOL to high level at timing T8.

[0095] Since the P-phase determination flag XPO is at a low level, the logic gate 341 changes the internal signal VCOD from a high level to a low level at timing T8 at the end of the D-phase conversion period.

[0096] The pulse generating unit 342 outputs a high-level pulse signal GCLAT for the pulse period from timing T8 when the internal signal VCOD falls. At this timing T8, the D-phase Gray code is latched. The Gray code value at this time is full scale. That is, the D-phase Gray code is corrected to full scale based on the P-phase determination flag XPO.

[0097] Now, consider the case where the AD converter 320 does not correct the D-phase Gray code to full scale. In this case, the difference between the P-phase and D-phase becomes a very small value, causing the pixel to sink into black. This phenomenon is called the sunspot phenomenon because sunspot-like spots appear in the image data even though no sunspots actually exist.

[0098] As illustrated in the figure, the AD converter 320 corrects the D-phase Gray code to full scale based on the P-phase determination flag XPO, thereby making the difference between the P-phase and D-phase sufficiently large and suppressing the occurrence of sunspots.

[0099] The AD converter 320 then converts the P-phase and D-phase Gray codes into binary codes, finds the difference between them, and outputs the binary code of the difference as luminance information together with the P-phase determination flag XPO.

[0100] FIG. 9 is a diagram illustrating a data transfer method according to the first embodiment of the present technology. The image sensor 200 sequentially outputs (in other words, transfers) brightness information BIN and flag information x for each pixel to the DSP circuit 120 via, for example, a serial transmission line 209. The flag information x includes, for example, a P-phase determination flag XPO and address information for the pixel. The address information includes V address information indicating a vertical address and H address information indicating a horizontal address, and this information is generated by, for example, the digital calculation unit 216. In the diagram, b0 to b10 indicate bits 0 to 10 in the brightness information BIN. The image sensor 200 may continuously transmit the brightness information BIN and flag information x, or may leave a fixed interval dT between the brightness information BIN and the flag information x.

[0101] It should be noted that the data transfer method is not limited to the example shown in Fig. 1. For example, the image sensor 200 may output the luminance information of all pixels, and then output the flag information x of all pixels together.

[0102] Regarding the address information, the image sensor 200 detects whether the P-phase level is a threshold value Th Ｐ Alternatively, the image sensor 200 may output only the address information of pixels determined to be below the P-phase value (in other words, to have received sunlight or the like). This reduces the amount of output data. Alternatively, the image sensor 200 may output the address information of all pixels regardless of the value of the P-phase determination flag.

[0103] Furthermore, the image sensor 200 can output only the P-phase determination flag XPO without outputting the address information.

[0104] 10 is a diagram illustrating an example of a result of estimating a light source distribution according to the first embodiment of the present technology. A frame 500 captured by the image sensor 200 includes sunlight 511 and a light source 512 equivalent to sunlight. The image sensor 200 estimates the P-phase level of each pixel by a threshold Th Ｐ The gray area in the figure indicates whether the P-phase level is higher than the threshold Th or not, i.e., whether sunlight is incident or not, and outputs a P-phase determination flag XPO indicating the determination result. Ｐ 1 indicates a pixel that is determined to be below the threshold (in other words, that light from sunlight or a light source similar to sunlight is incident thereon).

[0105] Here, a first comparative example is assumed in which the image sensor 200 outputs only luminance information for each pixel. In this first comparative example, even if the luminance indicated by the luminance information is at its maximum, the DSP circuit 120 has difficulty determining whether the light source is sunlight, an artificial light source, or a white subject. This can result in an inaccurate light source distribution for the background image, and the surface reflections and shadows of virtual objects may differ from those in the real world, resulting in an unnatural composite image. To avoid this, the DSP circuit 120 can estimate the light source distribution using inverse rendering technology based on one or more frames captured in advance, but this does not allow for real-time light source estimation. Therefore, this technology is difficult to use in scenes requiring immediacy, such as live video.

[0106] In contrast, in the first embodiment, the image sensor 200 outputs the P-phase determination flag XPO in addition to the luminance information for each pixel, so that the DSP circuit 120 can estimate the light source distribution based on the P-phase determination flag XPO each time a frame is captured. In other words, the light source distribution can be estimated in real time. This makes it possible to output a composite image onto a live video.

[0107] Furthermore, since the image sensor 200 outputs the P-phase determination flag XPO for each pixel, the DSP circuit 120 can easily identify the light source position, which improves the ease of editing the image.

[0108] 11 is a flowchart showing an example of the operation of the image sensor 200 according to the first embodiment of the present technology. This operation is started, for example, when a predetermined application for capturing an image of a frame is executed.

[0109] The image sensor 200 selects a row to be read (step S901), and latches the P-phase Gray code for each pixel in the row when the comparison result is inverted (step S902). Ｐ It is determined whether it is higher (step S903).

[0110] The P phase level is the threshold value Th Ｐ If the P-phase level is higher than the threshold value Th (step S903: Yes), the image sensor 200 latches the D-phase Gray code when the comparison result of that pixel is inverted (step S904). Ｐ In the following cases (step S903: No), the image sensor 200 corrects the D-phase gray code of the pixel to full scale (step S905). Note that the processes of steps S903 to S905 are executed in parallel for each pixel in the row.

[0111] Then, the image sensor 200 performs CDS processing for each pixel in the row (step S906), and outputs luminance information and a P-phase determination flag for each pixel (step S907).

[0112] The image sensor 200 determines whether or not the readout of all rows has been completed (step S908). If the readout of all rows has not been completed (step S908: No), the image sensor 200 repeatedly executes step S901 and subsequent steps. On the other hand, if the readout of all rows has been completed (step S908: Yes), the image sensor 200 ends the operation for image capture.

[0113] When capturing a plurality of frames consecutively, steps S901 to S908 are repeatedly executed in synchronization with a vertical synchronization signal.

[0114] 12 is a flowchart showing an example of the operation of the DSP circuit 120 according to the first embodiment of the present technology. This operation is executed every time a frame is captured.

[0115] The DSP circuit 120 estimates a light source distribution based on the P-phase determination flag in the frame from the image sensor 200. The DSP circuit 120 also estimates reflection characteristics and the like based on the captured frame (step S911). The DSP circuit 120 then lights a model in a three-dimensional space based on the estimated light source distribution, performs texturing, and generates a two-dimensional image by rendering (step S912). The DSP circuit 120 then combines the two-dimensional image with the frame (step S913). After step S913, the DSP circuit 120 ends the combining operation.

[0116] Thus, according to the first embodiment of the present technology, the image sensor 200 outputs brightness information and a P-phase determination flag for each pixel, and the DSP circuit 120 can estimate the light source distribution in real time based on the P-phase determination flag.

[0117] [First Modification] In the first embodiment described above, the image sensor 200 outputs flag information including a P-phase determination flag for each pixel, but this configuration may result in insufficient accuracy when estimating the light source distribution. The image sensor 200 in the first modification of the first embodiment outputs flag information including a P-phase determination flag for each pixel. Ｐ This embodiment differs from the first embodiment in that a flag indicating whether the voltage is higher or not is further output.

[0118] FIG. 13 is a diagram for explaining a data transfer method according to a first modified example of the first embodiment of the present technology.

[0119] The digital calculation unit 216 in the image sensor 200 further generates an adjacent pixel determination flag y for each pixel based on the P-phase determination flag XPO for each pixel, and supplies the flag to the interface 217 .

[0120] The adjacent pixel determination flag y indicates the determination result of the P-phase determination flag XPO of the pixel adjacent to the pixel of interest. For example, y ０ , y １ , y ２ and y ３ The following four flags are generated as adjacent pixel determination flags. ０ indicates the determination result of the P-phase determination flag XPO of the pixel adjacent to the pixel on the upper side of the pixel of interest, and y １ indicates the determination result of the P-phase determination flag XPO of the pixel adjacent to the pixel below the pixel of interest. ２ indicates the determination result of the P-phase determination flag XPO of the adjacent pixel on the left side of the pixel of interest, and y ３ indicates the determination result of the P-phase determination flag XPO of the adjacent pixel on the right side of the pixel of interest.

[0121] The interface 217 outputs brightness information BIN and flag information for each pixel. The flag information includes, in addition to the information x in the first embodiment, an adjacent pixel determination flag y (i.e., y ０ , y １ , y ２ , y ３ ) is included.

[0122] Although the digital calculation unit 216 generates adjacent pixel determination flags for each pixel of interest, with respect to the four pixels above, below, left, and right of the pixel, the configuration is not limited to this. For example, the digital calculation unit 216 can also generate adjacent pixel determination flags for each pixel of interest, with respect to the eight pixels above, below, left, right, upper left, upper right, lower left, and lower right of the pixel of interest.

[0123] 14 is a flowchart showing an example of the operation of the DSP circuit 120 according to the first modification of the first embodiment of the present technology. The DSP circuit 120 focuses on one pixel in a frame and sets it as a pixel of interest (step S921). The DSP circuit 120 refers to the P-phase determination flag of the pixel of interest and determines whether the P-phase level of the pixel is equal to or greater than the threshold value Th Ｐ It is determined whether or not the value is equal to or less than this (step S922).

[0124] The P-phase level of the pixel of interest is the threshold value Th Ｐ If it is higher (step S922: No), the DSD circuit 120 sets the adjacent pixel determination flag y ０ , y １ , y ２ and y ３ Then, the DSP 120 determines whether the P-phase level is equal to or exceeds the threshold value Th based on these flags. Ｐ It is determined whether there are two or more adjacent pixels (step S923).

[0125] The P-phase level of the pixel of interest is the threshold value Th Ｐ If the P-phase level is equal to or greater than the threshold value Th Ｐ If there are two or more adjacent pixels (step S923: Yes), the DSP circuit 120 estimates the pixel of interest as a light source pixel (step S924).Then, the DSP circuit 120 determines whether all pixels have been considered (step S925).

[0126] On the other hand, the P-phase level is the threshold value Th Ｐ If the number of adjacent pixels is one or less (step S923: No), the pixel of interest is not treated as a light source pixel, and step S925 is executed.

[0127] If there are any pixels that have not been focused on (step S925: No), the DSP circuit 120 repeatedly executes steps S921 and onward. If all pixels have been focused on (step S925: Yes), the DSP circuit 120 further estimates the reflection characteristics and the like, and executes steps S912 and S913.

[0128] 15 is a diagram illustrating an example of an estimation result of a light source distribution according to a first modified example of the first embodiment of the present technology. The dark gray part in the figure indicates a region where the P-phase level is equal to or lower than the threshold Th Ｐ The gray and white areas indicate pixels that are determined to have a P-phase level below the threshold Th Ｐ The pixel indicated is determined to have a higher brightness (in other words, no sunlight or the like is incident on it).

[0129] Let's look at the gray pixel at coordinates (0, 5). Of the four pixels adjacent to this pixel, the two pixels to the right and below are pixels that have been determined to have received sunlight or other light. Therefore, this gray pixel is treated as a light source pixel. Similarly, the remaining three gray pixels are also treated as light source pixels, since sunlight or other light has been received by two or more of their adjacent pixels.

[0130] As illustrated in the same figure, by the DSP circuit 120 estimating the light source distribution using the adjacent pixel determination flag in addition to the P-phase determination flag, the estimation accuracy of the light source distribution can be improved compared to when only the P-phase determination flag is used.

[0131] Thus, according to the first modified example of the first embodiment of the present technology, the image sensor 200 further outputs an adjacent pixel determination flag, and therefore the DSP circuit 120 can estimate the light source distribution with high accuracy using the adjacent pixel determination flag in addition to the P-phase determination flag.

[0132] [Second Modification] In the first embodiment described above, the image sensor 200 outputs the luminance information and the flag information via a single serial transmission line 209. However, the present invention is not limited to this transfer method. The image sensor 200 in this second modification of the first embodiment differs from the first embodiment in that the luminance information and the flag information are output via different transmission lines.

[0133] 16 is a diagram for explaining a data transfer method in a second modified example of the first embodiment of the present technology. An image sensor 200 in the second modified example of the first embodiment outputs (transfers) flag information x to a DSP circuit 120 via a transmission line 207. The image sensor 200 also outputs (transfers) luminance information BIN to the DSP circuit 120 via a transmission line 208. The transmission lines 207 and 208 are examples of the first and second transmission lines set forth in the claims.

[0134] As shown in the figure, by transferring flag information x and luminance information BIN in parallel, the transfer speed can be improved compared to the first embodiment.

[0135] The first modification can be applied to the second modification of the first embodiment.

[0136] As described above, according to the second modified example of the first embodiment of the present technology, the image sensor 200 outputs flag information via the transmission line 207 and brightness information via the transmission line 208, thereby improving the transfer speed.

[0137] [Third Modification] In the first embodiment described above, the image sensor 200 detects the P-phase level of each pixel when the P-phase level is equal to or exceeds the threshold value Th Ｐ In the first embodiment, it is determined whether the D-phase level is higher than the threshold value, but it is also possible to further determine whether the D-phase level is higher than the threshold value. The image sensor 200 in this third modification of the first embodiment differs from the first embodiment in that it further determines whether the D-phase level is higher than the threshold value for each pixel.

[0138] 17 is a circuit diagram showing a configuration example of an AD converter 320 according to a third modification of the first embodiment of the present technology. The AD converter 320 according to the third modification of the first embodiment differs from the first embodiment in that it further includes a D-phase determination unit 360.

[0139] The D-phase determination unit 360 determines whether the D-phase level is higher than a threshold value. The D-phase determination unit 360 includes one or more latch circuits such as latch circuits 361 and 362.

[0140] The inverted signal XVCO from the inverter 322 is input to the input terminals D of the latch circuits 361 and 362. The clock terminal of the latch circuit 361 receives the pulse signal D from the system control unit 212. ０ LAT is input, and the pulse signal D from the system control unit 212 is input to the clock terminal of the latch circuit 362. １ LAT is input to the latch circuits 361 and 362. An inverted value of the inverted signal XPRST is input to the reset terminals of the latch circuits 361 and 362. A 1-bit signal indicating the held value is output from the output terminal Q of the latch circuit 361 as a D-phase determination flag XD. ０ The latch circuit 362 outputs a 1-bit signal indicating the held value as the D-phase determination flag XD from the output terminal Q of the latch circuit 362.１ 0 to the memory 370.

[0141] 18 is a timing chart showing an example of the operation of the image sensor 200 according to the third modified example of the first embodiment of the present technology. In the drawing, a dashed dotted line indicates the trajectory of the potential of the vertical signal line VSL, and a broken solid line indicates the trajectory of the ramp signal RMP.

[0142] During the D-phase change period from timing T11 to T16, the ramp signal RMP becomes equal to or lower than the D-phase level at timing T13. At timing T13, the comparison result VCO is inverted.

[0143] The system control unit 212 generates a high-level pulse signal D for a pulse period from timing T12 immediately before timing T13. ０ The level of the ramp signal RMP at this timing T12 is the threshold value Th Ｄ０ Pulse signal D ０ The D-phase determination unit 360 latches the low level by the LAT, and the D-phase determination flag XD ０ O is output. A low-level D-phase determination flag XD ０ O is the D phase level is the threshold Th Ｄ０ Show that:

[0144] Then, the system control unit 212 outputs a high-level pulse signal D for a pulse period from timing T14 immediately after timing T13. １ The level of the ramp signal RMP at this timing T14 is the threshold value Th Ｄ１ Pulse signal D １ The D-phase determination unit 360 latches the high level by the LAT, and the D-phase determination flag XD １ O is output. A high-level D-phase determination flag XD １ O is the D phase level is the threshold Th Ｄ１ Indicates higher.

[0145] Then, the system control unit 212 outputs a high-level pulse signal D for a pulse period from timing T15 after timing T14. ２The level of the ramp signal RMP at this timing T15 is the threshold value Th Ｄ２ Pulse signal D ２ The D-phase determination unit 360 latches the high level by the LAT, and the D-phase determination flag XD ２ O is output. A high-level D-phase determination flag XD ２ O is the D phase level is the threshold Th Ｄ２ Indicates higher.

[0146] As illustrated in the figure, the image sensor 200 detects the D-phase level at a threshold Th Ｄ０ , Th Ｄ１ and Th Ｄ２ The image sensor 200 may output the D-phase determination flag only for pixels for which sunlight is incident based on the P-phase determination flag, or may output the D-phase determination flag for all pixels regardless of the P-phase determination flag.

[0147] Since the image sensor 200 outputs a D-phase determination flag, the DSP circuit 120 at the subsequent stage can estimate the light amount of the light source based on the D-phase determination flag. ０ O is at low level, and the D phase determination flag XD １ Since the light intensity is at a high level from 0 onwards, the light intensity is below the threshold value Th Ｄ０ and threshold Th Ｄ１ It is estimated to be between .

[0148] The threshold value Th Ｄ０ , Th Ｄ１ and Th Ｄ２ is an example of the second threshold value described in the claims.

[0149] In addition, the threshold value for determining the D phase level is set to Th Ｄ０ , Th Ｄ１ and Th Ｄ２ However, the number of thresholds is not limited to three and may be one or more.

[0150] Moreover, the first and second modifications can be applied to the third modification of the first embodiment.

[0151] Thus, according to the third variant of the first embodiment of the present technology, the image sensor 200 outputs a D-phase determination flag in addition to a P-phase determination flag, so that the DSP circuit 120 can further estimate the light intensity of the light source.

[0152] 2. Second Embodiment In the first embodiment described above, the DSP circuit 120 uses flag information to estimate the light source distribution, but the flag information can also be used for other processing. The DSP circuit 120 in this second embodiment differs from the first embodiment in that it performs white balance adjustment using flag information.

[0153] 19 is a block diagram showing an example configuration of a DSP circuit 120 according to the second embodiment of the present technology. The DSP circuit 120 according to the second embodiment differs from the first embodiment in that it further includes a white balance adjustment unit 125.

[0154] The white balance adjustment unit 125 adjusts the white balance of the frame using the flag information. For example, the white balance adjustment unit 125 adjusts the white balance of the frame when the P-phase level is equal to or exceeds a threshold value Th Ｐ If there is a pixel determined to be irradiated with sunlight or less (in other words, sunlight is incident on it), the white balance is adjusted assuming that the light source is sunlight. If there is no pixel determined to be irradiated with sunlight, the white balance adjustment unit 125 analyzes the frame to estimate the type of light source, and adjusts the white balance based on that type of light source. The white balance adjustment unit 125 then supplies the corrected frame to the synthesis processing unit 124.

[0155] As a second comparative example, an image processing system is assumed that includes an IR (Infra-Red) sensor in addition to the image sensor 200 and adjusts the white balance using the IR component from the IR sensor. Generally, artificial light sources have a small IR component, while sunlight has a large IR component. Therefore, by using the IR component, the DSP circuit 120 can determine whether the light source is sunlight.

[0156] 20 is a diagram for explaining the processing of the image processing system in the second comparative example. The dashed dotted line indicates the angle of view of the IR sensor, and the solid diagonal line indicates the angle of view of the image sensor 200.

[0157] As shown in the figure, when an IR sensor is provided separately from the image sensor 200, the angles of view thereof may not match. In the example shown in the figure, the angle of view of the IR sensor is wider than that of the image sensor 200. These differences in the angles of view may result in incorrect white balance adjustment.

[0158] For example, suppose that an artificial light source 610 is located within the angle of view of the IR sensor, but that light source does not exist within the angle of view of the image sensor 200. Also, suppose that sunlight 511 exists within the angle of view of the image sensor 200. In this case, the DSP circuit 120 treats both the artificial light source 610 and sunlight 511 as light sources and adjusts the white balance based on them. As a result, the white balance may be adjusted to an incorrect value due to the influence of the artificial light source 610 outside the angle of view of the image sensor 200, which could result in a deterioration in image quality.

[0159] 21 is a diagram for explaining processing by the DSP circuit 120 according to the second embodiment of the present technology. In the diagram, sunlight 511 is present within the angle of view of the image sensor 200, as in FIG. 20 , but no artificial light source 610 is present. In this case, the DSP circuit 120 estimates that only sunlight 511 is the light source using flag information, and adjusts the white balance.

[0160] As illustrated in the figure, in the second embodiment, there is no need to place an IR sensor in the image processing system 100. Furthermore, since the DSP circuit 120 does not recognize the artificial light source 610 outside the angle of view of the image sensor 200, erroneous white balance adjustment due to the influence of the light source is suppressed, and image quality can be improved compared to the second comparative example.

[0161] It should be noted that the first, second, and third modifications of the first embodiment can be applied to the second embodiment.

[0162] As described above, according to the second embodiment of the present technology, the DSP circuit 120 adjusts the white balance using flag information, so that an IR sensor is not required, and image quality can be improved by appropriate white balance adjustment.

[0163] 3. Application Examples to Mobile Bodies The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0164] FIG. 22 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0165] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 22, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0166] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0167] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0168] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0169] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0170] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0171] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0172] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0173] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0174] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying information to vehicle occupants or the outside of the vehicle. In the example of Fig. 22, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0175] FIG. 23 is a diagram showing an example of the installation position of the imaging unit 12031.

[0176] In FIG. 23, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0177] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0178] 23 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0179] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0180] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0181] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0182] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0183] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 of the above-described configuration. Specifically, the image sensor 200 in FIG. 1 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to estimate the light source distribution in real time, thereby enabling the use of AR technology and VR technology in in-vehicle devices.

[0184] Note that the above-described embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specifying the invention in the claims correspond to each other. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present technology having the same name correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist thereof.

[0185] The effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0186] The present technology may also be configured as follows: (1) An image sensor comprising: a pixel that generates an analog signal as a pixel signal; an analog-to-digital converter that performs a process of determining whether a level of the pixel signal is higher than a predetermined threshold and generating flag information indicating the determination result, and a process of converting the pixel signal into a digital signal and outputting it as luminance information; and an interface that outputs the luminance information and the flag information to an external device. (2) The image sensor according to (1), wherein the threshold includes a first threshold, and the flag information includes a reset level determination flag indicating whether a reset level, which is the level of the pixel signal when the pixel is initialized, is higher than the first threshold. (3) The image sensor according to (2), wherein the flag information further includes address information of the pixel for each pixel. (4) The image sensor according to (2) or (3), wherein the flag information further includes an adjacent pixel determination flag indicating whether the reset level of a pixel adjacent to the pixel for each pixel is higher than the first threshold. (5) The image sensor according to any one of (2) to (4), wherein the threshold value further includes a second threshold value, and the flag information further includes a signal level determination flag indicating whether a signal level, which is a level of the pixel signal according to an amount of exposure, is higher than the second threshold value. (6) The image sensor according to any one of (1) to (5), wherein the interface sequentially outputs the luminance information and the flag information via a predetermined serial transmission line. (7) The image sensor according to any one of (1) to (5), wherein the interface outputs the luminance information via a first transmission line and outputs the flag information via a second transmission line. (8) The image sensor according to any one of (1) to (7), wherein the analog-to-digital converter comprises: a comparator that compares the pixel signal with a predetermined reference signal and outputs the comparison result; a column counter that generates and outputs the luminance information based on the comparison result; a determination unit that determines whether the level of the pixel signal is higher than the threshold value and generates flag information indicating the determination result; and a correction unit that corrects the luminance information if the level of the pixel signal is lower than the threshold value.(9) An image processing system comprising: an image sensor including a pixel that generates an analog signal as a pixel signal; an analog-to-digital converter that determines whether the level of the pixel signal is higher than a predetermined threshold and generates flag information indicative of the determination result, and converts the pixel signal into a digital signal and outputs it as luminance information; and an interface that outputs the luminance information and the flag information to an external device; and a signal processing circuit that performs predetermined image processing on a frame in which the luminance information is arranged using the flag information. (10) The image processing system according to (9), wherein the image processing includes processing to estimate a light source distribution. (11) The information processing system according to (9) or (10), wherein the image processing includes processing to adjust white balance. (12) A control method for an image sensor comprising: a step of generating an analog signal as a pixel signal; a step of determining whether the level of the pixel signal is higher than a predetermined threshold and generating flag information indicative of the determination result, and converting the pixel signal into a digital signal and outputting it as luminance information; and a step of outputting the luminance information and the flag information to an external device.

[0187] 100 Image processing system 110 Memory unit 120 DSP circuit 121 Frame memory 122 Inverse rendering processing unit 123 Rendering processing unit 124 Composition processing unit 125 White balance adjustment unit 130 Display unit 200 Image sensor 207, 208 Transmission line 209 Serial transmission line 211 Vertical driving unit 212 System control unit 213 DAC 214 Pixel array unit 215 Horizontal driving unit 216 Digital calculation unit 217 Interface 220 Pixel 221 Photoelectric conversion element 222 Transfer transistor 223 Reset transistor 224 Floating diffusion layer 225 Amplification transistor 226 Selection transistor 250 Clip circuit 251 pMOS transistor 252 nMOS transistor 300 Column signal processing unit 310 Gray code counter 320 AD converter 321 Comparator 322 Inverter 330 P-phase determination unit 331, 361, 362 Latch circuit 340 Correction unit 341 Logic gate 342 Pulse generation unit 350 Column counter 351 Gray code latch 352 ALU 353 Code converter 354 Full adder 355 Temporary latch 356 Interface latch 360 D-phase determination unit 370 Memory 380 Output unit 12031 Imaging unit

Claims

a pixel that generates an analog signal as a pixel signal; an analog-to-digital converter that performs a process of determining whether or not the level of the pixel signal is higher than a predetermined threshold value and generating flag information indicative of the determination result, and a process of converting the pixel signal into a digital signal and outputting the digital signal as luminance information; an interface for outputting the luminance information and the flag information to the outside; An image sensor comprising: 　 the threshold value includes a first threshold value; The flag information includes a reset level determination flag indicating whether a reset level, which is the level of the pixel signal when the pixel is initialized, is higher than the first threshold value.

2. The image sensor of claim 1. 　 The flag information further includes address information of each pixel.

3. The image sensor according to claim 2. 　 The flag information further includes an adjacent pixel determination flag indicating, for each pixel, whether the reset level of a pixel adjacent to the pixel is higher than the first threshold value.

3. The image sensor according to claim 2. 　 the threshold value further includes a second threshold value; The flag information further includes a signal level determination flag indicating whether a signal level, which is a level of the pixel signal according to an amount of exposure, is higher than the second threshold value.

3. The image sensor according to claim 2. 　 The interface outputs the luminance information and the flag information in sequence via a predetermined serial transmission line.

2. The image sensor of claim 1. 　 The interface outputs the luminance information via a first transmission line and the flag information via a second transmission line.

2. The image sensor of claim 1. 　 The analog-to-digital converter a comparator that compares a predetermined reference signal with the pixel signal and outputs a comparison result; a column counter that generates and outputs the luminance information based on the comparison result; a determination unit that determines whether the level of the pixel signal is higher than the threshold value and generates flag information that indicates the determination result; a correction unit that corrects the luminance information when the level of the pixel signal is less than the threshold value; The image sensor of claim 1 , comprising: 　 an image sensor including: pixels that generate analog signals as pixel signals; an analog-to-digital converter that performs a process of determining whether or not a level of the pixel signal is higher than a predetermined threshold value and generating flag information indicative of the determination result; and a process of converting the pixel signal into a digital signal and outputting the digital signal as luminance information; and an interface that outputs the luminance information and the flag information to an external device; a signal processing circuit that performs predetermined image processing on the frame in which the luminance information is arranged using the flag information; An image processing system comprising: 　 The image processing includes a process of estimating a light source distribution.

10. The image processing system according to claim 9. 　 The image processing includes a process for adjusting white balance.

10. The information processing system according to claim 9. 　 generating an analog signal as a pixel signal; a step of determining whether the level of the pixel signal is higher than a predetermined threshold value and generating flag information indicating the determination result, and a step of converting the pixel signal into a digital signal and outputting the digital signal as luminance information; outputting the luminance information and the flag information to an external device of the image sensor; A method for controlling an image sensor comprising:

Images