Distance measurement system, image sensor, and method for controlling distance measurement system

Abstract

The present invention improves the distance measurement accuracy in a distance measurement system employing an indirect time of flight (iToF) method.　This distance measurement system includes a camera module and a unit. The camera module includes a pixel array unit, an analog-to-digital conversion unit, and a digital gain processing unit. In the pixel array unit, a plurality of pixels including iToF pixels for generating analog signals for calculating a distance by the iToF method are arranged in a two-dimensional lattice shape. The analog-to-digital conversion unit converts the analog signals into digital signals. The digital gain processing unit performs: processing of increasing or decreasing the digital signals by first digital gain and outputting the digital signals; and processing of increasing or decreasing digital signals in a predetermined area of the pixel array unit by second digital gain and outputting the digital signals. The unit processes the signals from the camera module.

Description

Ranging system, image sensor, and method for controlling the ranging system

[0001] The present technology relates to a distance measurement system, and more particularly to a distance measurement system that performs distance measurement, an image sensor, and a method for controlling the distance measurement system.

[0002] Image sensors that measure distances by the indirect time of flight (iToF) method have been used in various devices such as in-vehicle devices and smartphones. For example, a technology using a back-illuminated or moth-eye image sensor has been proposed for a system that measures distances by the iToF method (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-179474

[0004] The above-mentioned conventional technologies use back-illuminated or moth-eye structures to ensure sufficient aperture ratios and reduce reflected light. However, with the above-mentioned conventional technologies, signals from pixels can be saturated by external light, or the signal level can be reduced due to light absorption by the subject. These factors can reduce distance measurement accuracy.

[0005] For example, when applying the image sensor described above to acquire distance measurement images in an in-vehicle system, the analog gain margin narrows when considering EMI (Electro Magnetic Interference), heat issues, middleware adjustments, and other factors. This narrow margin can result in poorer distance measurement accuracy for the rear seat camera compared to the front seat camera. Furthermore, distance measurement accuracy can also deteriorate on the front seat side, potentially due to the following reasons: (1) Signal saturation in some areas of the distance measurement image due to the influence of external light during the day; (2) Signal saturation in some areas of the distance measurement image due to the influence of street lights at night; (3) Light is absorbed by the skin, clothing, and interior equipment of the driver and passengers, preventing it from reaching the sensor; and (4) A combination of factors (1) to (3).

[0006] This technology was developed in light of these circumstances, and aims to improve the distance measurement accuracy in systems that perform distance measurement using the iToF method.

[0007] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof relates to a ranging system including a camera module including a pixel array unit in which a plurality of pixels, including iToF pixels that generate analog signals for calculating distance by an iToF (indirect Time of Flight) method, are arranged in a two-dimensional grid, an analog-to-digital converter unit that converts the analog signals into digital signals, and a digital gain processor that performs processes of increasing or decreasing the digital signals by a first digital gain and outputting the digital signals within a predetermined area of ​​the pixel array unit by a second digital gain, and a unit that processes signals from the camera module, and a control method thereof, thereby improving ranging accuracy.

[0008] In this first aspect, the unit may perform distance measurement calculation processing to calculate a distance by an iToF method based on the output digital signal, thereby producing an effect of generating a distance image.

[0009] In addition, in this first aspect, the pixel array portion may be divided into a predetermined number of pixel blocks, and the unit may further perform a combining process, before the distance measurement calculation process, of calculating an amount of received light for each pixel block and, if the amount of received light is not within a predetermined range, replacing the digital signal corresponding to the first digital gain with the digital signal corresponding to the second digital gain. This brings about the effect of suppressing signal saturation and insufficient signal level.

[0010] In addition, in the first aspect, the camera module may further include a calculation unit that calculates a distance by an iToF method based on the digital signal, thereby reducing the amount of processing by the unit.

[0011] In addition, in this first aspect, the camera module may further include a first serializer that converts first parallel data into first serial data and transmits the first serial data to the unit, thereby reducing the number of signal lines between the camera module and the unit.

[0012] In addition, in this first aspect, the camera module may include a first image sensor, the pixel array unit, the analog-to-digital conversion unit, and the digital gain processing unit are arranged within the first image sensor, and the first serializer may convert the first parallel data from the first image sensor into the first serial data, thereby providing an effect of reducing the number of signal lines between the camera module and the unit.

[0013] In addition, in this first aspect, the camera module may include a first image sensor, and the pixel array unit, the analog-to-digital conversion unit, the digital signal processing unit, and the first serializer may be disposed within the first image sensor, thereby reducing the number of signal lines between the first image sensor and the unit.

[0014] In addition, in the first aspect, the camera module may include a first image sensor in which the pixel array unit, the analog-to-digital conversion unit, and the digital signal processing unit are arranged, and a second image sensor in which visible light pixels that receive visible light are arranged, thereby improving the performance of the ranging system.

[0015] In addition, in the first aspect, the second image sensor may further include an IR (Infra-Red) pixel that receives IR light, thereby improving performance at night, etc.

[0016] In this first aspect, the camera module may further include a serializer that converts parallel data from the first and second image sensors into serial data and transmits the serial data to the unit, thereby reducing the number of signal lines between the camera module and the unit.

[0017] In addition, in this first aspect, the camera module may further include a first serializer that converts the first parallel data into first serial data and transmits the first serial data to the unit, and a second serializer that converts the second parallel data into second serial data and transmits the second serial data to the unit, thereby reducing the number of signal lines between the camera module and the unit.

[0018] In this first aspect, the first serializer may convert the first parallel data from the first image sensor into the first serial data, and the second serializer may convert the second parallel data from the second image sensor into the second serial data, thereby reducing the number of signal lines between the camera module and the unit.

[0019] In this first aspect, the first serializer may be arranged in the first image sensor, and the second serializer may be arranged in the second image sensor, thereby reducing the number of signal lines between the first and second image sensors and the unit.

[0020] In this first aspect, the plurality of pixels may further include a visible light pixel that receives visible light, thereby improving the performance of the ranging system.

[0021] In this first aspect, the unit may set the parameters of the area every time a certain period of time elapses, thereby improving the performance of the ranging system.

[0022] A second aspect of the present technology is an image sensor including a pixel array unit in which a plurality of pixels including iToF pixels that generate analog signals for calculating distance by an iToF method are arranged in a two-dimensional lattice pattern, an analog-to-digital converter unit that converts the analog signals into digital signals, and a digital gain processor that performs a process of increasing or decreasing the digital signals by a first digital gain and outputting the digital signals, and a process of increasing or decreasing the digital signals within a predetermined area of ​​the pixel array unit by a second digital gain, thereby improving distance measurement accuracy.

[0023] In this second aspect, the area may be divided into a predetermined number of pixel blocks, and the digital gain processing unit may further perform a synthesis process of calculating an amount of received light for each pixel block and, if the amount of received light is not within the range, replacing the digital signal corresponding to the first digital gain with the digital signal corresponding to the second digital gain. This brings about an effect of reducing the amount of processing by a unit external to the image sensor.

[0024] In addition, in this second aspect, the image sensor may further include an image processing unit that processes the digital signal from the digital gain processing unit, the area being divided into a predetermined number of pixel blocks, the image processing unit calculating an amount of received light for each pixel block, and outputting the digital signal increased or decreased by the first digital gain if the amount of received light is within a predetermined range, and outputting the digital signal increased or decreased by the second digital gain if the amount of received light is not within the range. This brings about the effect of reducing the amount of processing by a unit external to the image sensor.

[0025] 1 is a block diagram showing an example configuration of a ranging system according to a first embodiment of the present technology. FIG. 2 is a block diagram showing an example configuration of a camera module and an ECU (Electronic Control Unit) according to the first embodiment of the present technology. FIG. 3 is a block diagram showing an example configuration of a camera module and an ECU to which a subsequent IC (Integrated Circuit) is added according to the first embodiment of the present technology. FIG. 4 is a block diagram showing an example configuration of a camera module and an ECU when a serializer is arranged in an image sensor according to the first embodiment of the present technology. FIG. 5 is a block diagram showing an example configuration of an image sensor according to the first embodiment of the present technology. FIG. 6 is a circuit diagram showing an example configuration of an iToF pixel according to the first embodiment of the present technology. FIG. 7 is a diagram for explaining a ranging method according to the first embodiment of the present technology. FIG. 8 is a block diagram showing an example configuration of a digital gain processing unit according to the first embodiment of the present technology. FIG. 9 is a diagram for explaining operation of a digital gain processing unit in a normal mode according to the first embodiment of the present technology. FIG. 10 is a diagram for explaining operation of a digital gain processing unit in an external light influence mode according to the first embodiment of the present technology. FIG. 11 is a diagram for explaining operation of a digital gain processing unit in a night mode according to the first embodiment of the present technology. FIG. 1 is a diagram showing an example of output data of an image sensor in the first embodiment of the present technology. FIG. 2 is a diagram showing an example of image data generated by an ECU in the first embodiment of the present technology. FIG. 3 is a flowchart showing an example of an operation of a camera module in the first embodiment of the present technology. FIG. 4 is a flowchart showing an example of an operation of an ECU in the first embodiment of the present technology. FIG. 5 is a flowchart showing an example of a synthesis process in the first embodiment of the present technology. FIG. 6 is a flowchart showing an example of an operation of a camera module when performing synthesis processing during digital gain processing of the first embodiment of the present technology. FIG. 7 is a flowchart showing an example of an operation of a camera module when performing synthesis processing during image processing of the first embodiment of the present technology.1 is a diagram for explaining an operation of a digital gain processing unit in a normal mode when combining processing is performed on the camera module side according to the first embodiment of the present technology. FIG. 2 is a block diagram showing an example configuration of a distance measuring system according to a first modified example of the first embodiment of the present technology. FIG. 3 is a block diagram showing an example configuration of a distance measuring system according to a second embodiment of the present technology. FIG. 4 is a block diagram showing an example configuration of an image sensor according to the second embodiment of the present technology. FIG. 5 is a block diagram showing an example configuration of a camera module and an ECU according to the second embodiment of the present technology. FIG. 6 is a block diagram showing an example configuration of a camera module and an ECU when a serializer is arranged for each image sensor according to the second embodiment of the present technology. FIG. 7 is a block diagram showing an example configuration of a camera module and an ECU when a serializer is arranged in an image sensor according to the second embodiment of the present technology. FIG. 8 is a block diagram showing an example configuration of a camera module and an ECU to which a subsequent IC is added according to the second embodiment of the present technology. FIG. 9 is a block diagram showing an example configuration of a camera module and an ECU to which a subsequent IC is added according to the second embodiment of the present technology. 1 is a block diagram showing an example configuration of a camera module and an ECU when a subsequent IC is added only to the iToF side and a serializer is arranged for each image sensor according to a second embodiment of the present technology. 2 is a block diagram showing an example configuration of a camera module and an ECU when a subsequent IC is added only to the RGB (Red Green Blue) side according to the second embodiment of the present technology. 3 is a block diagram showing an example configuration of a camera module and an ECU when a subsequent IC is added only to the RGB side and a serializer is arranged for each image sensor according to the second embodiment of the present technology. 4 is a block diagram showing an example configuration of a ranging system according to a third embodiment of the present technology. 5 is a block diagram showing an example configuration of an image sensor according to the third embodiment of the present technology. 6 is a block diagram showing an example configuration of a camera module and an ECU according to the third embodiment of the present technology.10 is a block diagram showing an example configuration of a camera module and an ECU to which a subsequent stage IC is added, according to a third embodiment of the present technology. FIG. 11 is a block diagram showing an example configuration of a camera module and an ECU when a serializer is arranged in an image sensor, according to the third embodiment of the present technology. FIG. 12 is a block diagram showing another example of an image sensor according to the third embodiment of the present technology. FIG. 13 is a diagram summarizing features of each configuration of the first to third embodiments of the present technology. FIG. 14 is a flowchart showing an example operation of an ECU according to a fourth embodiment of the present technology. FIG. 15 is a diagram showing an example setting of an area according to the fourth embodiment of the present technology. FIG. 16 is a diagram showing an example resetting of an area according to the fourth embodiment of the present technology. FIG. 17 is a block diagram showing an example of a schematic configuration of a vehicle control system. FIG. 18 is an explanatory diagram showing an example of installation positions of an outside-vehicle information detection unit and an imaging unit.

[0026] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. First embodiment (an example in which a digital signal is increased or decreased by a digital gain) 2. Second embodiment (an example in which two image sensors are provided and a digital signal is increased or decreased by a digital gain) 3. Third embodiment (an example in which RGB pixels are added and a digital signal is increased or decreased by a digital gain) 4. Fourth embodiment (an example in which an area parameter is variable and a digital signal is increased or decreased by a digital gain) 5. Example of application to a moving body

[0027] 1 is a block diagram showing an example of the configuration of a ranging system according to a first embodiment of the present technology. This ranging system is a system for performing ranging by the iToF method, and includes a camera module 100 and an ECU 300. This ranging system is applied to, for example, a vehicle control system described later.

[0028] The camera module 100 generates RAW data by capturing an image, and includes an LD (Laser Diode) 111 and an LDD (Laser Diode Driver) 112. The camera module 100 further includes a lens group 121, an image sensor 201, an EEPROM (Electrically Erasable Programmable ROM) 131, and a serializer 151.

[0029] The LD 111 emits irradiation light and irradiates the subject with it. Pulsed near-infrared light (in other words, pulsed light) is used as the irradiation light. The LDD 112 drives the LD 111 in synchronization with a rectangular wave light-emission control signal CLKp. The frequency of this light-emission control signal CLKp is, for example, 20 megahertz (MHz). The frequency of the light-emission control signal CLKp is not limited to 20 megahertz (MHz), and may be 5 megahertz (MHz) or 150 megahertz (MHz) or higher.

[0030] The lens group 121 collects reflected light from the irradiated light and guides it to the image sensor 201 .

[0031] The image sensor 201 receives reflected light (in other words, pulsed light) and generates RAW data. The image sensor 201 supplies parallel data including the RAW data to the serializer 151. The image sensor 201 is an example of a first image sensor described in the claims.

[0032] The EEPROM 131 stores various data used by the image sensor 201 .

[0033] The serializer 151 converts parallel data from the image sensor 201 into serial data and transmits the serial data to the ECU 300. The serializer 151 is an example of a first serializer described in the claims.

[0034] The ECU 300 performs various processes such as distance measurement calculations using the RAW data from the image sensor 201. Details of the processes performed by the ECU 300 will be described later. The ECU 300 is an example of a unit described in the claims.

[0035] The distance measuring system of Fig. 1 will be described in detail with reference to Fig. 2. As illustrated in Fig. 2, an I2C (Inter-Integrated Circuit) interface 161 is used within the camera module 100. Data is transmitted and received between the LDD 112, the image sensor 201, the EEPROM 131, and the serializer 151 via the I2C interface 161. Serial data from the serializer 151 is transmitted to the ECU 300 via the serial interface 108.

[0036] The ECU 300 also includes a deserializer 311 , a control circuit 312 , an I2C interface 313 , and a SoC (System-on-a-Chip) 314 .

[0037] The deserializer 311 converts the serial data from the serializer 151 into parallel data and supplies it to the SoC 314 .

[0038] The SoC 314 performs various processes, such as distance measurement calculations, using the RAW data included in the parallel data. For example, the SoC 314 performs distance measurement calculations using the iToF method to generate a distance image, and then uses the distance image to recognize bone structure and gestures. Once the bone structure is recognized, the SoC 314 estimates the vehicle's posture based on the recognition results. Furthermore, the SoC 314 can perform in-vehicle infotainment (IVI) operations based on the recognition and estimation results. In the IVI operations, the SoC 314 can issue a warning to the driver, for example, by voice or buzzer, based on the posture estimation results. In the IVI operations, the SoC 314 can also play music or video based on the gesture recognition results.

[0039] The SoC 314 also transmits various types of control information, such as analog gain values ​​and digital gain values, to the camera module 100. Details of this information will be described later.

[0040] The control circuit 312 controls the deserializer 311 and the SoC 314 via the I2C interface 313 .

[0041] In the figure, the ECU 300 performs the distance measurement calculation, but the invention is not limited to this configuration.

[0042] 3, a subsequent IC 171 may be added to the camera module 100, and distance measurement calculation may be performed by the subsequent IC 171. The subsequent IC 171 is an example of a calculation unit described in the claims.

[0043] 2 and 3, the serializer 151 is arranged outside the image sensor 201, but the present invention is not limited to this configuration.

[0044] 4, the serializer 151 can be disposed inside the image sensor 201. In this case, a sensor block 210 and the serializer 151 are disposed inside the image sensor 201. The configuration of the sensor block 210 will be described later.

[0045] In order to distinguish between the configurations of Figures 2 to 4, an identification number of No. 1 is assigned to Figure 2, No. 2 is assigned to Figure 3, and No. 3 is assigned to Figure 4.

[0046] 5 is a block diagram showing a configuration example of an image sensor 201 according to the first embodiment of the present technology. The image sensor 201 includes a sensor block 210. Arranged within the sensor block 210 are a vertical drive circuit 211, a timing control unit 212, a DAC (Digital to Analog Converter) 213, and a pixel array unit 214. Further arranged within the sensor block 210 are a column ADC (Analog to Digital Converter) 215, a horizontal transfer control unit 216, a digital gain processing unit 250, and an image processing unit 217.

[0047] Furthermore, the sensor block 210 receives control information from the ECU 300 via the I2C interface 161. The sensor block 210 also outputs a light emission control signal CLKp to the LDD 112.

[0048] A plurality of iToF pixels 220 are arranged in a two-dimensional lattice pattern in the pixel array unit 214. Each of the iToF pixels 220 generates an analog signal for calculating a distance by the iToF method, and supplies the analog signal to the column ADC 215.

[0049] The vertical drive circuit 211 sequentially selects rows in the pixel array section 214 and drives the iToF pixels 220 in that row.

[0050] The timing control section 212 controls the operation timing of each of the vertical drive circuit 211 , the DAC 213 , the column ADC 215 , the horizontal transfer control section 216 , the digital gain processing section 250 and the image processing section 217 .

[0051] The DAC 213 generates a sawtooth ramp signal by DA (Digital to Analog) conversion and supplies it to the column ADC 215. The slope of this ramp signal is set in accordance with control information from the ECU 300. The analog gain of the column ADC 215 is controlled in accordance with this slope.

[0052] In the column ADC 215, an ADC is arranged for each column of the pixel array unit 214. The ADC converts an analog signal from the corresponding column into a digital signal and supplies the digital signal to the digital gain processing unit 250 under the control of the horizontal transfer control unit 216. The column ADC 215 is an example of an analog-to-digital conversion unit as defined in the claims.

[0053] The horizontal transfer control unit 216 controls the column ADC 215 to output digital signals in sequence.

[0054] The digital gain processing unit 250 amplifies or attenuates the digital signal by a predetermined digital gain. Here, it is assumed that an area for switching the digital gain is set in advance in the pixel array unit 214. The digital gain processing unit 250 amplifies or attenuates the digital signal of all pixels by a predetermined digital gain g １ The digital signal of the pixels in the area is increased or decreased by g １ Digital gain g, which is different from ２ Yag ３ Then, the digital gain processing unit 250 supplies the digital signal after the increase or decrease to the image processing unit 217. １ The value of each digital gain such as is set in accordance with control information from ECU 300.

[0055] In addition, g １ is an example of a first digital gain as defined in the claims, and g ２ Yag ３ is an example of a second digital gain as defined in the claims.

[0056] The image processing unit 217 performs various types of image processing on the digital signal from the digital gain processing unit 250. The image processing unit 217 supplies data in which the processed signals are arranged to the serializer 151 as RAW data.

[0057] [Configuration Example of iToF Pixel] Figure 6 is a circuit diagram showing a configuration example of an iToF pixel 220 according to the first embodiment of the present technology. The iToF pixel 220 includes a photodiode 221, a charge discharging transistor 222, transfer transistors 223 and 224, reset transistors 225 and 226, and floating diffusion layers 227 and 228. The iToF pixel 220 further includes conversion transistors 229 and 230, additional capacitances 231 and 232, amplification transistors 233 and 234, and selection transistors 235 and 236. For example, nMOS (n-channel metal oxide semiconductor) transistors are used as each transistor in the iToF pixel 220. Note that the conversion transistors 229 and 230 and the additional capacitances 231 and 232 are not essential and may be provided as needed.

[0058] The photodiode 221 photoelectrically converts incident light to generate electric charges. The charge drain transistor 222 drains electric charges from the photodiode 221, the floating diffusion layers 227 and 228, and the additional capacitors 231 and 232 in accordance with a drive signal OFG from the vertical drive circuit 211, for initialization.

[0059] The transfer transistor 223 transfers charges from the photodiode 221 to the floating diffusion layer 227 in accordance with a drive signal TRGa from the vertical drive circuit 211 .

[0060] The transfer transistor 224 transfers charges from the photodiode 221 to the floating diffusion layer 228 in accordance with a drive signal TRGb from the vertical drive circuit 211 .

[0061] The reset transistors 225 and 226 initialize the floating diffusion layers 227 and 228 in accordance with a drive signal RST from the vertical drive circuit 211 .

[0062] The floating diffusion layers 227 and 228 store electric charges and generate a voltage according to the amount of the electric charges.

[0063] The conversion transistor 229 opens and closes the path between the floating diffusion layer 227 and the additional capacitance 231 in accordance with a drive signal FDGa from the vertical drive circuit 211. The conversion transistor 230 opens and closes the path between the floating diffusion layer 228 and the additional capacitance 232 in accordance with a drive signal FDGb from the vertical drive circuit 211. The conversion efficiency of converting electric charge into voltage can be controlled by opening and closing these conversion transistors 229 and 230.

[0064] The amplifier transistor 233 forms a source follower circuit and supplies a voltage corresponding to the voltage of the floating diffusion layer 227 to the selection transistor 235. The amplifier transistor 234 forms a source follower circuit and supplies a voltage corresponding to the voltage of the floating diffusion layer 228 to the selection transistor 236.

[0065] The selection transistor 235 supplies an analog signal of the voltage from the amplification transistor 233 to the column ADC 215 via a vertical signal line VSLa in accordance with a drive signal SELa from the vertical drive circuit 211 .

[0066] The selection transistor 236 supplies an analog signal of the voltage from the amplification transistor 234 to the column ADC 215 via a vertical signal line VSLb in accordance with a drive signal SELb from the vertical drive circuit 211 .

[0067] As shown in the figure, each of the transfer transistor, floating diffusion layer, amplification transistor, and selection transistor is provided in pairs. One of these paired circuits is called the A tap, and the other is called the B tap. The ADC for each column is alternately connected to VSLa and VSLb, and alternately performs AD conversion on the analog signals of the A tap and the B tap.

[0068] 7 is a diagram illustrating a ranging method according to the first embodiment of the present technology. The iToF pixel 220 alternately and repeatedly detects Q1 and Q2 and Q3 and Q4. Hereinafter, the detection period of Q1 and Q2 will be referred to as a "Q1Q2 detection period," and the detection period of Q3 and Q4 will be referred to as a "Q3Q4 detection period." The lengths of the Q1Q2 detection period and the Q3Q4 detection period are the period of the vertical synchronization signal VSYNC (e.g., 1 / 60 seconds).

[0069] Here, Q1 is the charge q1 accumulated over the Q1Q2 detection period from 0 degrees to 180 degrees, with a specific phase (e.g., rising edge) of the light-emission control signal CLKp being 0 degrees. Because the frequency of the light-emission control signal CLKp is as high as 20 megahertz (MHz), the charge q1 per cycle (1 / 20 microsecond) is very small and difficult to detect. Therefore, the iToF pixel 220 accumulates each q1 over a Q1Q2 detection period, such as 1 / 60 second, which is longer than the cycle (1 / 20 microsecond) of the light-emission control signal CLKp, and detects the total amount as Q1. Furthermore, Q2 is the charge q2 accumulated over the Q1Q2 detection period from 180 degrees to 360 degrees of reflected light.

[0070] Furthermore, Q3 is the charge amount q3 accumulated from 90 degrees to 270 degrees over the Q3Q4 detection period, and Q4 is the charge amount q4 accumulated from 270 degrees to 90 degrees over the Q3Q4 detection period.

[0071] The distance d to the object is calculated from Q1, Q2, Q3, and Q4. In this way, Q1 and Q2 detected during the Q1Q2 detection period of 1 / 60 seconds and Q3 and Q4 detected during the Q3Q4 detection period of 1 / 60 seconds are required for distance measurement. Therefore, distance measurement is performed at intervals of, for example, 1 / 30 seconds.

[0072] 6, one of Q1 and Q2 is detected by tap A, and the other is detected by tap B. Also, during the Q3Q4 detection period, one of Q3 and Q4 is detected by tap A, and the other is detected by tap B.

[0073] The iToF pixel 220 transfers charge corresponding to the amount of charge from timing T50 (0 degrees) to timing T52 (180 degrees) within the Q1Q2 detection period to the floating diffusion layer 227 on the A tap side. When reflected light starts to be emitted at timing T51, an amount of charge q1 corresponding to the amount of light received from timing T51 to T52 is transferred.

[0074] Furthermore, the iToF pixel 220 transfers charge corresponding to the amount of light received from timing T52 (180 degrees) to timing T54 (360 degrees) within the Q1Q2 detection period to the floating diffusion layer 228 on the B tap side. When the reflected light finishes emitting light at timing T53, an amount of charge q2 corresponding to the amount of light received from timing T52 to T53 is transferred.

[0075] Then, the iToF pixel 220 transfers an amount of charge corresponding to the amount of light received from timing T55 (90 degrees) to timing T57 (270 degrees) within the Q3Q4 detection period to the floating diffusion layer 227. If reflected light starts to be emitted at timing T56, an amount of charge q3 corresponding to the amount of light received from timing T56 to T57 is transferred.

[0076] Furthermore, the iToF pixel 220 transfers an amount of charge corresponding to the amount of light received from timing T57 (270 degrees) to timing T59 (90 degrees) within the Q3Q4 detection period to the floating diffusion layer 228. When the reflected light finishes emitting light at timing T58, an amount of charge q4 corresponding to the amount of light received from timing T57 to T58 is transferred.

[0077] The iToF pixel 220 detects the cumulative values ​​of q1 and q2 within the Q1Q2 detection period as Q1 and Q2, and sequentially outputs an analog signal indicating Q1 and an analog signal indicating Q2. The iToF pixel 220 also detects the cumulative values ​​of q3 and q4 within the Q3Q4 detection period as Q3 and Q4, and sequentially outputs an analog signal indicating Q3 and an analog signal indicating Q4.

[0078] The digital gain processing unit 250 at the subsequent stage increases or decreases the values ​​Q1 to Q4 after AD conversion using a digital gain. The raw data from the image sensor 201 includes digital signals Q1, Q2, Q3, and Q4 for each pixel.

[0079] The downstream ECU 300 calculates the distance d for each pixel from the increased or decreased Q1, Q2, Q3, and Q4 using the following formula.

[0080] d=(c / 4πf)×tan －１ {(Q3-Q4) / (Q1-Q2)}

[0081] In the above formula, the unit of d is, for example, meters (m). c is the speed of light, and the unit is, for example, meters per second (m / s). －１ is the inverse of the tangent function.

[0082] 8 is a block diagram showing an example of the configuration of the digital gain processing unit 250 according to the first embodiment of the present technology. The digital gain processing unit 250 includes data selection units 251, 252, and 253, multiplication units 254, 255, and 256, and a data output adjustment unit 260.

[0083] The data selection unit 251 sequentially supplies all of the digital signals from the column ADC 215 to the multiplication unit 254 .

[0084] The data selection unit 252 sequentially selects digital signals of pixels within a predetermined area from the digital signals from the column ADC 215 and supplies the selected signals to the multiplication unit 255 .

[0085] The data selection unit 253 sequentially selects digital signals of pixels within a predetermined area from the digital signals from the column ADC 215 and supplies the selected digital signals to the multiplication unit 256 .

[0086] The multiplication unit 254 calculates the digital gain g １ The multiplication unit 255 increases or decreases the digital signal by the digital gain g ２ The multiplication unit 256 increases or decreases the digital signal by the digital gain g ３ The digital signal is increased or decreased by the digital signal output adjustment unit 260 .

[0087] Here, the ECU 300 sets one of the following modes in the image sensor 201: normal mode, external light influence mode, and night mode. The external light influence mode is a mode that is set during the day when there is a concern about the influence of external light. The night mode is a mode that is set at night when there is a concern about the influence of street lights. The normal mode is a mode when neither the external light influence mode nor the night mode is set. The above-mentioned areas correspond to areas where external light or street lights are expected to be reflected by the subject in the external light influence mode or night mode, or areas where light is expected to be reflected or absorbed in the normal mode.

[0088] Depending on these modes, the values ​​of the digital gains of the multiplication units 254, 255, and 256 are controlled by the ECU 300. In the normal mode, the digital gains g ２ and g ３ , the digital gain of all pixels g １ For example, a value higher than or lower than the digital gain g １ , g ２ and g ３ are set to 1.0, 0.5 and 2.0.

[0089] The data output adjusting section 260 adjusts the output timing of the digital signals from the multiplying sections 254, 255, and 256. The data output adjusting section 260 includes, for example, data buffers 261 and 262, and a selector 263.

[0090] The data buffer 261 delays the digital signal from the multiplication unit 255 and supplies it to the selector 263. The data buffer 262 delays the digital signal from the multiplication unit 256 and supplies it to the selector 263.

[0091] The selector 263 sequentially selects the digital signals from the multiplication units 254, 255, and 256. The selector 263 sequentially outputs the digital signal from the multiplication unit 254 to the image processing unit 217 for each pixel. Next, the selector 263 sequentially outputs, for example, the digital signal from the multiplication unit 255 for each pixel. Then, the selector 263 sequentially outputs the digital signal from the multiplication unit 256 for each pixel.

[0092] 9 is a diagram for explaining the operation of the digital gain processing unit 250 in normal mode according to the first embodiment of the present technology. A symbol "a" in the figure shows an example of a digital signal output by the multiplication unit 254. As shown in A in the figure, the multiplication unit 254 sequentially outputs digital signals D0, D1, D11, D12, etc., which are increased or decreased by 1.0. Of these, digital signals D0 and D1 are signals outside the area where the digital gain is switched, and digital signals D11 and D12 are signals within that area.

[0093] In the figure, b shows an example of a digital signal output by the multiplication unit 255. As shown in the example in the figure, the multiplication unit 255 attenuates D11, D12, etc., within the area among the digital signals D0, D1, D11, and D12 by 0.5 and outputs the attenuated digital signal D11. ×０．５ , D12 ×０．５ Let's say.

[0094] In the figure, c shows an example of a digital signal output by the multiplication unit 256. As shown in the example in c, the multiplication unit 256 amplifies D11, D12, etc., within the area among the digital signals D0, D1, D11, and D12 by 2.0 and outputs the amplified digital signal D11. ×２．０ , D12 ×２．０ Let's say.

[0095] 10, d shows an example of a digital signal output by the data output adjustment unit 260. As shown in d, the data output adjustment unit 260 sequentially outputs digital signals D0, D1, D11, D12, etc. of all pixels from the multiplication unit 254. Next, the data output adjustment unit 260 sequentially outputs the attenuated digital signal D11 within the area from the multiplication unit 255. ×０．５ , D12×０．５ Then, the data output adjustment unit 260 outputs the amplified digital signal D11 in the area from the multiplication unit 256. ×２．０ , D12 ×２．０ etc. are output in order.

[0096] 10 is a diagram illustrating an example of the digital gain in the external light influence mode according to the first embodiment of the present technology. In the external light influence mode, the digital gain g １ is set to a value lower than that in normal mode, and the digital gain g ２ and g ３ , and the digital gain g １ For example, the digital gain g １ , g ２ and g ３ are set to 0.5, 1.0 and 2.0.

[0097] 11 is a diagram for explaining the operation of the digital gain processing unit 250 in the external light influence mode according to the first embodiment of the present technology. A in the figure shows an example of a digital signal output by the multiplication unit 254. As shown in A in the figure, the multiplication unit 254 outputs a digital signal D0 attenuated by 0.5. ×０．５ , D1 ×０．５ , D11 ×０．５ , D12 ×０．５ etc. are output in order.

[0098] In the figure, b shows an example of a digital signal output by the multiplication unit 255. As shown in the example in the figure, the multiplication unit 255 increases or decreases D11, D12, etc., within the area among the digital signals D0, D1, D11, and D12 by 1.0 and outputs the increased or decreased signal.

[0099] In the figure, c shows an example of a digital signal output by the multiplication unit 256. As shown in c, the multiplication unit 256 amplifies D11, D12, etc., within the area among the digital signals D0, D1, D11, and D12 by 2.0, and outputs D11 ×２．０ , D12 ×２．０ Output as

[0100] 10, d indicates an example of a digital signal output from the data output adjustment unit 260. As shown in d in the figure, the data output adjustment unit 260 adjusts the digital signal D0 of all pixels from the multiplication unit 254. ×０．５ , D1 ×０．５ , D11 ×０．５ , D12 ×０．５ Next, the data output adjustment unit 260 outputs the digital signals D11, D12, etc. in the area from the multiplication unit 255 in order. Then, the data output adjustment unit 260 outputs the digital signals D11, D12, etc. in the area from the multiplication unit 256 in order. ×２．０ , D12 ×２．０ etc. are output in order.

[0101] 12 is a diagram illustrating an example of digital gain in the night mode according to the first embodiment of the present technology. In the night mode, the digital gain g １ is set to a value higher than that in normal mode, and the digital gain g ２ and g ３ Then, the digital gain g １ For example, the digital gain g １ , g ２ and g ３ are set to 2.0, 0.5 and 1.0.

[0102] 13 is a diagram for explaining the operation of the digital gain processing unit 250 in the night mode according to the first embodiment of the present technology. A in the figure shows an example of a digital signal output by the multiplication unit 254. As shown in A in the figure, the multiplication unit 254 multiplies the digital signal D0 amplified by 2.0 by ×２．０ , D1 ×２．０ , D11 ×２．０ , D12 ×２．０ etc. are output in order.

[0103] 10B shows an example of a digital signal output from the multiplication unit 255. As shown in FIG. 10B, the multiplication unit 255 attenuates D11, D12, etc., within the area among the digital signals D0, D1, D11, and D12 by 0.5, and outputs D11 ×０．５ , D12 ×０．５ Output as

[0104] In the figure, c shows an example of a digital signal output by the multiplication unit 256. As shown in the example in c, the multiplication unit 256 increases or decreases D11, D12, etc., within the area among the digital signals D0, D1, D11, and D12 by 1.0 and outputs the increased or decreased signal.

[0105] 10, d indicates an example of a digital signal output from the data output adjustment unit 260. As shown in d in the figure, the data output adjustment unit 260 adjusts the digital signal D0 of all pixels from the multiplication unit 254. ×２．０ , D1 ×２．０ , D11 ×２．０ , D12 ×２．０ Next, the data output adjustment unit 260 outputs the digital signal D11 in the area from the multiplication unit 255. ×０．５ , D12 ×０．５ Then, the data output adjustment section 260 outputs the digital signals D11, D12, etc. in the area from the multiplication section 256 in order.

[0106] 14 is a diagram showing an example of output data of the image sensor 201 according to the first embodiment of the present technology. As shown in FIG. 14A, the image sensor 201 captures an image of the interior of a vehicle, and the ranging system is used, for example, for monitoring the interior of the vehicle. Such a system is also called an ICM (In-Cabin Monitoring) system.

[0107] The area surrounded by the dotted line a in the figure is an area where digital signal saturation or insufficient signal level is a concern. For example, in normal mode, objects within the area (such as the driver's clothing, skin, or interior vehicle accessories) reflect or absorb light, which could result in digital signal saturation or insufficient signal level. In addition, in the ambient light impact mode, the digital signals of all pixels are attenuated to suppress the effects of ambient light, which could result in insufficient signal level within the area. In addition, in night mode, the digital signals of all pixels are amplified, which could result in digital signal saturation within the area.

[0108] If saturation of the digital signal or an insufficient signal level occurs when applied to an ICM system, this may result in a decrease in distance measurement accuracy, which may have an adverse effect on the ICM system.

[0109] Therefore, the image sensor 201 converts the digital signals in the area into digital gains g １ Digital gain g, which is different from ２ and g ３ It increases or decreases depending on.

[0110] In the figure, "b" shows an example in which the digital signal is attenuated by a digital gain of 0.5 in the normal mode, and "c" shows an example in which the digital signal is amplified by a digital gain of 2.0 in the normal mode.

[0111] The image sensor 201 outputs RAW data including all of the digital signals illustrated as a, b, and c in the figure to the ECU 300.

[0112] As mentioned above, the RAW data is data before distance measurement calculation (in other words, before development) that includes Q1, Q2, Q3, and Q4 for each pixel, and is not a distance image. However, for the sake of convenience, the RAW data is shown in the figure as a developed image.

[0113] FIG. 15 is a diagram showing an example of image data generated by the ECU 300 in the first embodiment of the present technology.

[0114] The ECU 300 performs a synthesis process before the distance measurement calculation. In this synthesis process, the ECU 300 divides the area into a predetermined number of pixel blocks and focuses on each pixel block in turn. For example, the area is divided into pixel blocks of 9 pixels, 3 rows x 3 columns.

[0115] For each pixel in the pixel block of interest, the ECU 300 calculates the g １ The ECU 300 calculates the sum of the increased or decreased digital signals Q1, Q2, Q3, and Q4. This sum indicates the amount of light received by the pixel. The ECU 300 then calculates the average or total of the amount of light received by each pixel in the pixel block as a confidence value. The confidence value is an example of the amount of light received as defined in the claims.

[0116] The ECU 300 determines whether the confidence value is within a predetermined range. If the confidence value is within the range, the ECU 300 １ The digital signals Q1 to Q4 corresponding to the above are used in the distance measurement calculation at the subsequent stage.

[0117] On the other hand, if the confidence value is outside the range, the ECU 300 １ The digital signals Q1, Q2, Q3 and Q4 increased or decreased by g ２ The ECU 300 replaces the confidence value with the digital signals Q1, Q2, Q3, and Q4 that have increased or decreased by g, and calculates the confidence value again. The ECU 300 determines whether the confidence value is within a predetermined range. If the confidence value is within the range, the ECU 300 ２ The digital signals Q1 to Q4 corresponding to the above are used in the distance measurement calculation at the subsequent stage.

[0118] On the other hand, if the confidence value is outside the range, the ECU 300 ２ The digital signals Q1, Q2, Q3 and Q4 increased or decreased by g ３ The ECU 300 replaces the confidence value with the digital signals Q1, Q2, Q3, and Q4 that have increased or decreased by g, and calculates the confidence value again. The ECU 300 determines whether the confidence value is within a predetermined range. If the confidence value is within the range, the ECU 300 ３ The digital signals Q1 to Q4 corresponding to the pixel blocks are used in the distance measurement calculation at a later stage. If the confidence value is out of the range, the ECU 300 performs processing such as interpolating the digital signals of the pixel blocks.

[0119] After the synthesis process, the ECU 300 performs distance measurement calculations for each pixel to generate a distance image.

[0120] In the figure, a shows an example of a distance image 501 in the normal mode when a subject in the area surrounded by a thick line reflects light. In this case, the digital gain g １ Lower g (e.g. 1.0) ２ (e.g., 0.5) will often attenuate the digital signal.

[0121] In the figure, b shows an example of a distance image 502 when a subject in the area absorbs light in normal mode. In this case, the digital gain g １ Higher g (e.g. 1.0) ３ Digital signals are often amplified by a gain of 2.0 or more.

[0122] In the figure, c shows an example of a distance image 503 in which neither reflection nor absorption occurs within the area in normal mode. In this case, the digital gain g １ Digital signals that have been increased or decreased by a value (such as 1.0) are often used as is.

[0123] As shown in the figure, the digital gain g １ Different from g ２ Yag ３ By replacing the digital signal with a digital signal amplified by the , it is possible to prevent saturation of the digital signal and insufficient signal level when a subject in the area reflects or absorbs light, thereby improving distance measurement accuracy and the reliability of the ICM system.

[0124] 16 is a flowchart showing an example of the operation of the camera module 100 according to the first embodiment of the present technology. This operation is started, for example, when a predetermined application for performing distance measurement is executed.

[0125] The camera module 100 sets analog gain and digital gain values ​​under the control of the ECU 300 (step S901) and performs AD conversion on analog signals from the iToF pixels for each column (step S902). The camera module 100 then increases or decreases the digital signals using the digital gain (step S903) and performs various image processing to generate RAW data (step S904). The camera module 100 transmits the RAW data to the ECU 300 (step S905). After step S905, the camera module 100 ends its image capture operation.

[0126] When multiple distance images are generated consecutively, the processes from steps S901 to S905 are repeatedly executed in synchronization with a vertical synchronization signal or the like.

[0127] 17 is a flowchart illustrating an example of an operation of the ECU according to the first embodiment of the present technology. This operation is started, for example, when a predetermined application for performing distance measurement is executed.

[0128] ECU 300 receives the RAW data (step S911) and performs a synthesis process (step S920). Then, ECU 300 performs distance measurement calculations for each pixel to generate a distance image (step S912). ECU 300 performs middleware processing, such as posture estimation and gesture recognition, based on the distance image (step S913), and executes IVI operations based on the estimation and recognition results (step S914). After step S914, ECU 300 ends the distance measurement operation.

[0129] When multiple distance images are generated consecutively, the processes from steps S911 to S914 are repeatedly executed in synchronization with a vertical synchronization signal or the like.

[0130] 18 is a flowchart illustrating an example of a synthesis process according to the first embodiment of the present technology. The ECU 300 focuses on any pixel block in the area (step S921), and calculates the digital gain g １ The confidence value of the pixel block of interest is calculated from the digital signals Q1 to Q4 that have been increased or decreased by (step S922).

[0131] ECU 300 determines whether the calculated confidence value is within a predetermined range. For example, assuming that the upper and lower limits of the range are 50 and 1500, ECU 300 determines whether the confidence value is greater than 50 and less than 1500 (step S923).

[0132] If the confidence value is not within the predetermined range (step S923: No), the ECU 300 outputs a digital signal corresponding to the digital gain g1 as the digital gain g ２and recalculates the confidence value (step S924). Then, ECU 300 determines whether the calculated confidence value is within a predetermined range (step S925).

[0133] If the confidence value is not within the predetermined range (step S925: No), the ECU 300 adjusts the digital gain g ２ The digital signal corresponding to the digital gain g ３ and recalculates the confidence value (step S926). Then, ECU 300 determines whether the calculated confidence value is within a predetermined range (step S927).

[0134] If the confidence value is within the predetermined range in step S923, S925, or S927, ECU 300 determines whether calculations for all blocks in the area have been completed (step S928). If the confidence value is not within the predetermined range (step S927: No), ECU 300 performs interpolation processing or the like and executes step S928.

[0135] If the calculation for all blocks in the area has not been completed (step S928: No), the ECU 300 repeatedly executes step S921 and subsequent steps. If the calculation for all blocks has been completed (step S928: Yes), the ECU 300 ends the synthesis process (step S928).

[0136] In the distance measurement calculation at the later stage, the digital signal g １ In the area, if the digital signals Q1 to Q4 are replaced in step S924 or S926, the digital gain g after replacement is １ or g ２ The distance measurement calculation is performed by the digital signals Q1 to Q4 increased or decreased by the digital signal g １ The distance measurement calculation is performed using the digital signals Q1 to Q4 that have been increased or decreased by the above.

[0137] Although the ECU 300 executes the synthesis process shown in the figure, the camera module 100 may execute this synthesis process instead of the ECU 300 .

[0138] In this case, for example, as illustrated in FIG. 19, the digital gain processing unit 250 increases or decreases the digital signal by the digital gain, and then performs a synthesis process (step S903).

[0139] Alternatively, as shown in FIG. 20, the image processing unit 217 performs a synthesis process during image processing (step S904).

[0140] 19 and 20 , when the synthesis process is performed on the camera module 100 side, the upper and lower limits of the confidence value, in addition to the analog gain and digital gain, are controlled by the ECU 300. Furthermore, the camera module 100 outputs the replaced digital signal within the area.

[0141] For example, assume that digital signals D1, D2, D11, D12, D31, and D32 are increased or decreased by a digital gain of 1.0 in normal mode as illustrated in Fig. 21 a. Of these, digital signals D11 and D12 are signals within the area.

[0142] As shown in b and c in the figure, the multiplication units 255 and 256 increase or decrease D11, D12, etc. in the area by 0.5 or 2.0 and output the increased or decreased values.

[0143] Then, the camera module 100 performs a synthesis process, and synthesizes D11 and D12 in the area based on the confidence value. ×０．５ and D12 ×０．５ shall be replaced with.

[0144] In this case, as illustrated in d in the figure, the camera module 100 detects D11 instead of D11 and D12 in the area. ×０．５ and D12 ×０．５ are output to ECU 300. Outside the area, D0, D1, D31, D32, etc. are output as is to ECU 300. D11 and D12 before replacement are discarded.

[0145] In the first embodiment, the digital gain is set to 0.5, 1.0, 2.0, etc., but the camera module 100 and the ECU 300 can change these values ​​depending on the application. For example, the digital gain can be changed within a range from 0.1 to 64.

[0146] Furthermore, although an LD light source such as the LD 111 is used as the light source, an LED (light emitting diode) light source may be used instead of the LD light source.

[0147] Furthermore, in the synthesis process, the camera module 100 divides an area into pixel blocks of 9 pixels, 3 rows by 3 columns, and performs processes such as calculating confidence values ​​for each pixel block, but the unit of processing is not limited to that size. Depending on the application, the camera module 100 can perform processing for each pixel in a 1 row by 1 column, or can perform processing for each block of more pixels.

[0148] The camera module 100 can also apply a predetermined digital gain to the entire specific area.

[0149] Furthermore, the camera module 100 sets the upper and lower thresholds of the confidence value to 50 and 100, but these values ​​are not limited to these. The camera module 100 can change the upper and lower limit values ​​depending on the application (by filter settings).

[0150] In this way, according to the first embodiment of the present technology, the digital gain processing unit 250 calculates g １ The digital signal is increased or decreased by ２ Yag ３ This increases or decreases the digital signal, improving distance measurement accuracy.

[0151] [Modification] In the first embodiment described above, one LD and one LDD are provided, but this configuration is not limited to this. The camera module 100 in this modification of the first embodiment differs from the first embodiment in that two LDs and two LDDs are provided.

[0152] 22 is a block diagram showing a configuration example of a ranging system according to a first modified example of the first embodiment of the present technology. The ranging system according to the first modified example of the first embodiment differs from the first embodiment in that an LD 113 and an LDD 114 are further provided in the camera module 100.

[0153] The image sensor 201, for example, simultaneously emits light from the LD 111 and the LD 113. This increases the amount of light received compared to when only the LD 111 is used, improving distance measurement accuracy.

[0154] As described above, according to the modification of the first embodiment of the present technology, the LD 113 and the LDD 114 are added, so that the amount of received light can be increased compared to the case where only the LD 111 is used.

[0155] 2. Second Embodiment In the first embodiment described above, only the image sensor 201 having an array of iToF pixels is provided, but an image sensor having an array of RGB pixels can also be added. The ranging system in this second embodiment differs from the first embodiment in that an image sensor having an array of RGB pixels is added.

[0156] 23 is a block diagram showing an example configuration of a ranging system according to a second embodiment of the present technology. The ranging system according to the second embodiment further includes a lens group 122, an image sensor 202, and an EEPROM 132 in a camera module 100. The camera module 100 also includes an ISP (Image Signal Processor) 140 and a DDR SDRAM (Double-Data-Rate SDRAM) 133, and includes a serializer 150 instead of the serializer 151.

[0157] The lens group 122 collects the reflected light of the irradiated light and guides it to the image sensor 202 .

[0158] The image sensor 202 has an array of pixels that receive RGB visible light and pixels that receive IR (Infra-Red) light. The EEPROM 132 stores various data used by the image sensor 202. The image sensor 202 is an example of a second image sensor described in the claims.

[0159] The ISP 140 performs various image processing such as white balance correction on the image data from the image sensor 202, and supplies parallel data including the processed data to the serializer 150. The DDR SDRAM 133 temporarily stores the data processed by the ISP 140.

[0160] The serializer 150 converts the parallel data from the image sensors 201 and 202 into serial data and transmits it to the ECU 300 via the serial interfaces 108 and 109 .

[0161] The ECU 300 can use the image from the image sensor 202, for example, in middleware processing and IVI operations, thereby further improving the performance of the ECU 300.

[0162] 24 is a block diagram showing a configuration example of an image sensor 202 according to the second embodiment of the present technology. In this image sensor 202, R pixels 241, G pixels 242, B pixels 243, and IR pixels 244 are arranged in a pixel array unit 214. In addition, a signal processing unit 218 is arranged in place of a digital gain processing unit 250 and an image processing unit 217.

[0163] The R pixel 241 receives red visible light and supplies an analog signal corresponding to the amount of received light to the column ADC 215. The G pixel 242 receives green visible light and supplies an analog signal corresponding to the amount of received light to the column ADC 215. The B pixel 243 receives blue visible light and supplies an analog signal corresponding to the amount of received light to the column ADC 215. The IR pixel 244 receives IR light and supplies an analog signal corresponding to the amount of received light to the column ADC 215.

[0164] The R pixel 241, the G pixel 242, the B pixel 243, and the IR pixel 244 are arranged in a pattern in which one of a pair of diagonal G pixels in a Bayer array is replaced with an IR pixel, for example. Furthermore, as the circuit for each of the R pixel 241, the G pixel 242, the B pixel 243, and the IR pixel 244, for example, a circuit including a photodiode and only one of the A tap and the B tap shown in FIG. 6 is used.

[0165] The R pixel 241, the G pixel 242, and the B pixel 243 are examples of visible light pixels described in the claims.

[0166] The signal processing unit 218 performs various signal processing on the digital signal. For example, in night mode, the signal processing unit 218 generates an IR image in which only digital signals of IR pixels are arranged, or an image in which digital signals of RGB pixels are combined with digital signals of IR pixels. In normal mode or ambient light influence mode, the signal processing unit 218 generates an RGB image in which only digital signals of RGB pixels are arranged. The signal processing unit 218 then supplies the generated image data to the ISP 140 as RAW data. Note that the ISP 140 may also perform some or all of the processing of the signal processing unit 218.

[0167] The distance measuring system of Fig. 23 will be described in detail with reference to Fig. 25. As illustrated in Fig. 25, an I2C interface 160 is used in the camera module 100. Data is transmitted and received between the image sensor 201, the image sensor 202, etc. via this I2C interface 160.

[0168] In the figure, the I2C interface and serializer are shared between the iToF side and the RGB side, but the present invention is not limited to this configuration.

[0169] For example, as illustrated in Fig. 26, I2C interfaces 161 and 162 and serializers 151 and 152 may be arranged. The serializer 151 converts parallel data from the image sensor 201 into serial data. Meanwhile, the serializer 152 receives parallel data from the image sensor 202 via the ISP 140 and converts it into serial data. In this way, the I2C interface 161 and serializer 151 are used on the iToF side, and the I2C interface 162 and serializer 152 are used on the RGB side.

[0170] The serializer 152 is an example of a second serializer described in the claims.

[0171] In addition, in FIG. 26, the serializers 151 and 152 are arranged outside the image sensors 201 and 202, but the present invention is not limited to this configuration.

[0172] For example, as illustrated in FIG. 27, the serializer 151 may be disposed inside the image sensor 201, and the serializer 152 may be disposed inside the image sensor 202.

[0173] In the figure, part of the processing of the ECU 300 can also be performed on the camera module 100 side.

[0174] 28, subsequent ICs 171 and 172 can be added to the camera module 100. The subsequent IC 171 executes distance measurement calculations, and the subsequent IC 172 executes demosaic processing and part of the middleware processing.

[0175] In the figure, the I2C interface and serializer are shared between the iToF side and the RGB side, but the present invention is not limited to this configuration.

[0176] For example, as illustrated in FIG. 29, subsequent stage ICs 171 and 172 may be added, and I2C interfaces 161 and 162 and serializers 151 and 152 may be arranged.

[0177] As shown in FIG. 30, it is also possible to dispose only the rear IC 171 on the iToF side out of the rear ICs 171 and 172.

[0178] In the figure, the I2C interface and serializer are shared between the iToF side and the RGB side, but the present invention is not limited to this configuration.

[0179] For example, as illustrated in FIG. 31, it is also possible to add only a subsequent stage IC 171 and to arrange I2C interfaces 161 and 162 and serializers 151 and 152.

[0180] As shown in FIG. 32, it is also possible to arrange only the RGB side post-stage IC 172 out of the post-stage ICs 171 and 172.

[0181] In the figure, the I2C interface and serializer are shared between the iToF side and the RGB side, but the present invention is not limited to this configuration.

[0182] For example, as illustrated in FIG. 33, it is also possible to add only a subsequent stage IC 172 and to arrange I2C interfaces 161 and 162 and serializers 151 and 152.

[0183] In order to distinguish between the configurations shown in Fig. 25 to Fig. 33, No. 1 is assigned to Fig. 25 and Fig. 26, No. 2 is assigned to Fig. 27, No. 3 is assigned to Fig. 28 and Fig. 29, No. 4 is assigned to Fig. 30 and Fig. 31, and No. 5 is assigned to Fig. 32 and Fig. 33.

[0184] In the second embodiment, the camera module 100 can also perform the synthesis process.

[0185] Moreover, the first modified example of the first embodiment can be applied to the second embodiment.

[0186] Furthermore, although the second embodiment uses an example of an RGB-IR sensor in which iToF pixels and RGB pixels are arranged, the present invention is not limited to this configuration. Depending on the application, the RGB pixels (RGB sensor) alone or IR pixels (IR sensor) can be used instead.

[0187] Furthermore, although the digital gain is set to 0.5, 1.0, 2.0, etc., the camera module 100 and the ECU 300 can change these values ​​depending on the application. For example, the value can be changed within a range from 0.1 to 64.

[0188] Furthermore, although an LD light source such as the LD 111 is used as the light source, an LED (light emitting diode) light source may be used instead of the LD light source.

[0189] Furthermore, in the synthesis process, the camera module 100 divides an area into pixel blocks of 9 pixels, 3 rows by 3 columns, and performs processes such as calculating confidence values ​​for each pixel block, but the unit of processing is not limited to that size. Depending on the application, the camera module 100 can perform processing for each pixel in a 1 row by 1 column, or can perform processing for each block of more pixels.

[0190] The camera module 100 can also apply a predetermined digital gain to the entire specific area.

[0191] Furthermore, the camera module 100 sets the upper and lower thresholds of the confidence value to 50 and 100, but these values ​​are not limited to these. The camera module 100 can change the upper and lower limit values ​​depending on the application (by filter settings).

[0192] As described above, according to the second embodiment of the present technology, the image sensor 202 having an array of RGB pixels and IR pixels is added, and therefore the ranging system can improve its performance (such as 3D Fusion) by utilizing the image from the image sensor 202.

[0193] 3. Third Embodiment In the first embodiment described above, only iToF pixels are arranged on the image sensor 201, but RGB pixels can also be arranged. The ranging system in this third embodiment differs from the first embodiment in that iToF pixels and RGB pixels are arranged on the image sensor 201.

[0194] 34 is a block diagram showing an example of the configuration of a distance measuring system according to the third embodiment of the present technology. The distance measuring system according to the third embodiment further includes an ISP 140 and a DDR SDRAM 133 in the camera module 100.

[0195] 35 is a block diagram showing a configuration example of an image sensor 201 according to a third embodiment of the present technology. The image sensor 201 according to the third embodiment differs from the first embodiment in that an R pixel 241, a G pixel 242, and a B pixel 243 are further arranged in a pixel array unit 214 in addition to the iToF pixel 220. These pixels are arranged in a pattern in which, for example, one of a pair of G pixels arranged diagonally in a Bayer array is replaced with an iToF pixel.

[0196] For example, the ECU 300 can use the RGB image from the image sensor 202 in addition to the distance image in middleware processing and IVI operations, thereby further improving the performance of the ECU 300.

[0197] The distance measuring system of Fig. 34 will be described in detail with reference to Fig. 36. As illustrated in Fig. 36, an I2C interface 161 is used in the camera module 100.

[0198] In the figure, the ECU 300 performs the distance measurement calculation, but the invention is not limited to this configuration.

[0199] For example, as shown in FIG. 37, a subsequent IC 171 can be added to the camera module 100, and distance measurement calculations can be performed by this subsequent IC 171.

[0200] In addition, in FIGS. 36 and 37, the serializer 151 is arranged outside the image sensor 201, but the present invention is not limited to this configuration.

[0201] For example, as illustrated in FIG. 38, the serializer 151 can be disposed inside the image sensor 201.

[0202] In order to distinguish between the configurations of Fig. 36 to Fig. 38, No. 1 is assigned to Fig. 36, No. 2 is assigned to Fig. 37, and No. 3 is assigned to Fig. 38.

[0203] As shown in FIG. 39, it is also possible to arrange only RGB pixels in a Bayer array or the like without providing iToF pixels in the image sensor 201, and to perform digital gain processing similar to that in the first embodiment.

[0204] In addition, in the same figure, one of a pair of G pixels in the diagonal direction in the image sensor 201 can be replaced with an IR pixel.

[0205] In the third embodiment, the camera module 100 can also perform the synthesis process.

[0206] Moreover, the first modified example of the first embodiment can be applied to the third embodiment.

[0207] As described above, according to the third embodiment of the present technology, in addition to the iToF pixel 220, RGB pixels are further arranged in the image sensor 201, so that the ranging system can improve its performance by utilizing RGB images.

[0208] 40 is a diagram summarizing the features of each configuration of the first to third embodiments of the present technology. In the first embodiment, in No. 1 and No. 2, the serializer is located outside the image sensor, while in No. 3, the serializer is located inside the image sensor. Furthermore, in No. 2, a subsequent-stage IC is added. In all of No. 1 to No. 3, the image sensor can generate RAW data. Furthermore, in No. 1 and No. 3, distance measurement calculations can be performed on the image sensor side, but in No. 2, distance measurement calculations are performed by a subsequent-stage IC, eliminating the need for distance measurement calculations on the image sensor side.

[0209] In the second embodiment, except for No. 2, the serializer is located outside the image sensor, while in No. 2, the serializer is located inside the image sensor. Furthermore, in No. 3, a subsequent IC is added to both the iToF side and the RGB side. In No. 4, a subsequent IC is added only to the iToF side, and in No. 5, a subsequent IC is added only to the RGB side. The image sensors in all of Nos. 1 to 5 are capable of generating RAW data. Furthermore, in Nos. 1, 2, and 5, distance measurement calculations can be performed on the image sensor side, but in Nos. 3 and 4, distance measurement calculations are performed by the subsequent IC on the iToF side, eliminating the need for distance measurement calculations on the image sensor side.

[0210] In the third embodiment, in No. 1 and No. 2, the serializer is external to the image sensor, while in No. 3, the serializer is internal to the image sensor. Furthermore, in No. 2, a subsequent IC is added. The image sensor can generate RAW data in all of No. 1 through No. 3. Furthermore, in No. 1 and No. 3, distance measurement calculations can be performed on the image sensor side, but in No. 2, distance measurement calculations are performed by the subsequent IC, eliminating the need for distance measurement calculations on the image sensor side.

[0211] 4. Fourth Embodiment In the first embodiment described above, the positions, sizes, and numbers of areas where digital gain is switched are fixed, but these can also be made variable. The distance measurement system in this fourth embodiment differs from the first embodiment in that the positions, etc. of the areas are variable.

[0212] 41 is a flowchart showing an example of the operation of the ECU 300 according to the fourth embodiment of the present technology. The operation of the fourth embodiment differs from the first embodiment in that steps S931 and S932 are further executed.

[0213] After the synthesis process (step S920), the ECU 300 determines whether a certain time has elapsed since the previous area setting (step S931). For example, step S931 is executed for each frame or every few frames.

[0214] If a certain time has elapsed since the previous setting (step S931: Yes), the ECU 300 resets the area parameters (position, size, number, etc.) according to the time of day and the in-vehicle situation (step S932). When resetting, the same setting values ​​as those used previously may be used, or different setting values ​​may be used.

[0215] If a certain time has not elapsed since the previous setting (step S931: No), or after step S932, the ECU 300 executes step S912 and subsequent steps.

[0216] 42 is a diagram showing an example of setting an area of ​​a predetermined frame in the fourth embodiment of the present technology. An area is set in a location surrounded by a dotted line in distance image 531. "×0.5" or "×2.0" in the area indicates the digital gain value most frequently used in that area.

[0217] Fig. 43 is a diagram showing an example of resetting areas of a frame different from that shown in Fig. 42 in the fourth embodiment of the present technology. When a certain period of time has elapsed since Fig. 42 , ECU 300 resets the areas. For example, in Fig. 43 , the number of areas, which was seven in Fig. 42 , is reduced to five. ECU 300 can change the number of areas as well as the positions and sizes of the areas to values ​​different from those of the previous time.

[0218] As described above, the ECU 300 sets the area parameters at regular intervals according to the vehicle interior conditions, which allows the ECU 300 to respond more flexibly to changes in the vehicle interior conditions than when the parameters are fixed, thereby improving the performance of the ranging system.

[0219] In the fourth embodiment, the camera module 100 can also perform the synthesis process.

[0220] Moreover, the first modified example of the first embodiment, and the second and third embodiments can be applied to the fourth embodiment.

[0221] Furthermore, although the digital gain is set to 0.5, 1.0, 2.0, etc., the camera module 100 and the ECU 300 can change these values ​​depending on the application. For example, the value can be changed within a range from 0.1 to 64.

[0222] Furthermore, although an LD light source such as the LD 111 is used as the light source, an LED (light emitting diode) light source may be used instead of the LD light source.

[0223] Furthermore, in the synthesis process, the camera module 100 divides an area into pixel blocks of 9 pixels, 3 rows by 3 columns, and performs processes such as calculating confidence values ​​for each pixel block, but the unit of processing is not limited to that size. Depending on the application, the camera module 100 can perform processing for each pixel in a 1 row by 1 column, or can perform processing for each block of more pixels.

[0224] Furthermore, the camera module 100 sets the upper and lower thresholds of the confidence value to 50 and 100, but these values ​​are not limited to these. The camera module 100 can change the upper and lower limit values ​​depending on the application (by filter settings).

[0225] As described above, according to the fourth embodiment of the present technology, the ECU 300 sets the parameters of the area at regular intervals, and therefore, the performance of the distance measuring system can be improved.

[0226] 5. Application Examples to Mobile Bodies The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0227] FIG. 44 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0228] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 44, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0229] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0230] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0231] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0232] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0233] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0234] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0235] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0236] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0237] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying information to vehicle occupants or the outside of the vehicle. In the example of Fig. 44, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0238] FIG. 45 is a diagram showing an example of the installation position of the imaging unit 12031.

[0239] In FIG. 45, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0240] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0241] 45 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0242] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0243] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0244] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0245] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0246] The above describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the in-vehicle information detection unit 12040 and the microcomputer 12051 of the above-described configuration. Specifically, the camera module 100 and the ECU 300 in FIG. 1 can be applied to the in-vehicle information detection unit 12040 and the microcomputer 12051. Applying the technology according to the present disclosure to these can improve distance measurement accuracy and enhance the performance of the vehicle control system.

[0247] Note that the above-described embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specifying the invention in the claims correspond to each other. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present technology having the same name correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist thereof.

[0248] The processing procedures described in the above embodiments may be considered as a method having a series of these procedures, or as a program for causing a computer to execute the series of procedures, or as a recording medium for storing the program. Examples of such a recording medium include a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, and a Blu-ray (registered trademark) Disc.

[0249] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0250] The present technology may also be configured as follows: (1) A ranging system comprising: a camera module including a pixel array unit in which a plurality of pixels, including iToF pixels that generate analog signals for calculating distance by an iToF (indirect Time of Flight) method, are arranged in a two-dimensional lattice pattern, an analog-to-digital converter unit that converts the analog signals into digital signals, and a digital gain processor that performs a process of increasing or decreasing the digital signals by a first digital gain and outputting the digital signals, and a process of increasing or decreasing the digital signals within a predetermined area of ​​the pixel array unit by a second digital gain and outputting the digital signals; and a unit that processes signals from the camera module. (2) The ranging system according to (1), wherein the unit performs a ranging calculation process to calculate distance by the iToF method based on the output digital signals. (3) The ranging system according to (2), wherein the pixel array unit is divided into a predetermined number of pixel blocks, and the unit further performs a synthesis process before the ranging calculation process, in which the amount of received light is calculated for each pixel block, and if the amount of received light is not within a predetermined range, the synthesis process replaces the digital signal corresponding to the first digital gain with the digital signal corresponding to the second digital gain. (4) The ranging system according to any of (1) to (3), wherein the camera module further includes a calculation unit that calculates a distance using an iToF method based on the digital signal. (5) The ranging system according to any of (1) to (4), wherein the camera module further includes a first serializer that converts first parallel data into first serial data and transmits the first parallel data to the unit. (6) The ranging system according to (5), wherein the camera module includes a first image sensor, and the pixel array unit, the analog-to-digital conversion unit, and the digital gain processing unit are disposed within the first image sensor, and the first serializer converts the first parallel data from the first image sensor into the first serial data.(7) The ranging system according to (5), wherein the camera module includes a first image sensor, and the pixel array unit, the analog-to-digital conversion unit, the digital signal processing unit, and the first serializer are arranged within the first image sensor. (8) The ranging system according to (1), wherein the camera module comprises: a first image sensor in which the pixel array unit, the analog-to-digital conversion unit, and the digital signal processing unit are arranged; and a second image sensor in which visible light pixels that receive visible light are arranged. (9) The ranging system according to (8), wherein the second image sensor further comprises IR pixels that receive IR (Infra-Red) light. (10) The ranging system according to (8), wherein the camera module further comprises a serializer that converts parallel data from the first and second image sensors into serial data and transmits the serial data to the unit. (11) The ranging system according to any one of (8) to (10), wherein the camera module further includes: a first serializer that converts first parallel data into first serial data and transmits the first serial data to the unit; and a second serializer that converts second parallel data into second serial data and transmits the second serial data to the unit. (12) The ranging system according to (11), wherein the first serializer converts the first parallel data from the first image sensor into the first serial data, and the second serializer converts the second parallel data from the second image sensor into the second serial data. (13) The ranging system according to (11), wherein the first serializer is disposed within the first image sensor, and the second serializer is disposed within the second image sensor. (14) The ranging system according to any one of (1) to (13), wherein the plurality of pixels further include visible light pixels that receive visible light. (15) The distance measuring system according to any one of (1) to (14), wherein the unit sets the parameters of the area every time a certain period of time elapses.(16) An image sensor comprising: a pixel array unit in which a plurality of pixels including iToF pixels that generate analog signals for calculating distance by an iToF method are arranged in a two-dimensional lattice pattern, an analog-to-digital conversion unit that converts the analog signals into digital signals, and a digital gain processing unit that performs processing to increase or decrease the digital signals by a first digital gain and output the digital signals, and to increase or decrease the digital signals in a predetermined area of ​​the pixel array unit by a second digital gain and output the digital signals. (17) The image sensor according to (16), wherein the area is divided into a predetermined number of pixel blocks, and the digital gain processing unit further performs a synthesis process to obtain an amount of received light for each pixel block, and if the amount of received light is not within the range, replace the digital signal corresponding to the first digital gain with the digital signal corresponding to the second digital gain. (18) The image sensor according to (16), further comprising an image processing unit that processes the digital signal from the digital gain processing unit, wherein the area is divided into a predetermined number of pixel blocks, and the image processing unit calculates an amount of received light for each pixel block, and if the amount of received light is within a predetermined range, outputs the digital signal increased or decreased by the first digital gain, and if the amount of received light is not within the range, outputs the digital signal increased or decreased by the second digital gain. (19) A method for controlling a ranging system, comprising: an analog-to-digital conversion step of converting, into a digital signal, an analog signal from a pixel array unit in which a plurality of pixels including iToF pixels that generate analog signals for calculating distance by an iToF method are arranged in a two-dimensional lattice, and a digital gain processing step of increasing or decreasing the digital signal by a first digital gain and outputting the digital signal, and increasing or decreasing the digital signal in a predetermined area of ​​the pixel array unit by a second digital gain.

[0251] 100 Camera module 108, 109 Serial interface 111, 113 LD 112, 114 LDD 121, 122 Lens group 131, 132 EEPROM 133 DDR SDRAM 140 ISP 150, 151, 152 Serializer 160, 161, 162, 313 I2C interface 171, 172 Subsequent IC 201, 202 Image sensor 210 Sensor block 211 Vertical drive circuit 212 Timing control unit 213 DAC 214 Pixel array unit 215 Column ADC 216 Horizontal transfer control unit 217 Image processing unit 218 Signal processing unit 220 iToF pixel 221 Photodiode 222 Charge discharge transistor 223, 224 Transfer transistor 225, 226 Reset transistor 227, 228 Floating diffusion layer 229, 230 Off-conversion transistor 231, 232 Additional capacitance 233, 234 Amplification transistor 235, 236 Selection transistor 241 R pixel 242 G pixel 243 B pixel 244 IR pixel 250 Digital gain processing unit 251, 252, 253 Data selection unit 254, 255, 256 Multiplication unit 260 Data output adjustment unit 261, 262 Data buffer 263 Selector 300 ECU 311 Deserializer 312 Control circuit 314 SoC 12040 In-vehicle information detection unit 12051 Microcomputer

Claims

1. A ranging system comprising: a camera module having a pixel array section in which a plurality of pixels, including iToF pixels that generate analog signals for calculating distance using the iToF (indirect Time of Flight) method, are arranged in a two-dimensional grid; an analog-to-digital conversion section that converts the analog signals into digital signals; and a digital gain processing section that performs processing to increase or decrease the digital signals using a first digital gain and output the digital signals, and processing to increase or decrease the digital signals within a predetermined area of ​​the pixel array section using a second digital gain and output the digital signals; and a unit that processes signals from the camera module.

2. The distance measuring system according to claim 1, wherein the unit performs distance measurement calculation processing to calculate the distance using the iToF method based on the output digital signal.

3. A ranging system as described in claim 2, wherein the pixel array section is divided into a predetermined number of pixel blocks, and the unit further performs a synthesis process before the ranging calculation process, in which the amount of received light is calculated for each pixel block, and if the amount of received light is not within a predetermined range, the digital signal corresponding to the first digital gain is replaced with the digital signal corresponding to the second digital gain.

4. The distance measuring system according to claim 1, wherein the camera module further comprises a calculation unit that calculates the distance by the iToF method based on the digital signal.

5. The ranging system according to claim 1, wherein the camera module further comprises a first serializer that converts first parallel data into first serial data and transmits the first serial data to the unit.

6. The ranging system according to claim 5, wherein the camera module includes a first image sensor, the pixel array section, the analog-to-digital conversion section, and the digital gain processing section are arranged within the first image sensor, and the first serializer converts the first parallel data from the first image sensor into the first serial data.

7. The ranging system according to claim 5, wherein the camera module includes a first image sensor, and the pixel array section, the analog-to-digital conversion section, the digital signal processing section, and the first serializer are arranged within the first image sensor.

8. The ranging system according to claim 1, wherein the camera module comprises: a first image sensor in which the pixel array section, the analog-to-digital conversion section, and the digital signal processing section are arranged; and a second image sensor in which visible light pixels that receive visible light are arranged.

9. The distance measuring system according to claim 8, wherein the second image sensor further comprises an array of IR (Infra-Red) pixels that receive IR light.

10. The ranging system according to claim 8, wherein the camera module further comprises a serializer that converts parallel data from the first and second image sensors into serial data and transmits the serial data to the unit.

11. The ranging system of claim 8, wherein the camera module further comprises: a first serializer that converts first parallel data into first serial data and transmits the first serial data to the unit; and a second serializer that converts second parallel data into second serial data and transmits the second serial data to the unit.

12. The ranging system according to claim 11, wherein the first serializer converts the first parallel data from the first image sensor into the first serial data, and the second serializer converts the second parallel data from the second image sensor into the second serial data.

13. The ranging system according to claim 11, wherein the first serializer is disposed within the first image sensor, and the second serializer is disposed within the second image sensor.

14. The ranging system according to claim 1, wherein the plurality of pixels further includes a visible light pixel that receives visible light.

15. The distance measuring system according to claim 1, wherein said unit sets the parameters of said area every time a certain period of time elapses.

16. An image sensor comprising: a pixel array section in which a plurality of pixels, including iToF pixels that generate analog signals for calculating distance by the iToF method, are arranged in a two-dimensional lattice; an analog-to-digital conversion section that converts the analog signals into digital signals; and a digital gain processing section that performs processing to increase or decrease the digital signals using a first digital gain and output the digital signals, and processing to increase or decrease the digital signals within a predetermined area of ​​the pixel array section using a second digital gain and output the digital signals.

17. The image sensor according to claim 16, wherein the area is divided into a predetermined number of pixel blocks, and the digital gain processing unit further performs a synthesis process in which the amount of received light is calculated for each pixel block, and if the amount of received light is not within the range, the digital signal corresponding to the first digital gain is replaced with the digital signal corresponding to the second digital gain.

18. An image sensor as described in claim 16, further comprising an image processing unit that processes the digital signal from the digital gain processing unit, wherein the area is divided into a predetermined number of pixel blocks, and the image processing unit calculates the amount of received light for each pixel block, and if the amount of received light is within a predetermined range, outputs the digital signal increased or decreased by the first digital gain, and if the amount of received light is not within the range, outputs the digital signal increased or decreased by the second digital gain.

19. A method for controlling a distance measurement system, comprising: an analog-to-digital conversion procedure for converting analog signals from a pixel array unit, which is composed of a plurality of pixels arranged in a two-dimensional lattice, including iToF pixels that generate analog signals for calculating distance by the iToF method, into digital signals; and a digital gain processing procedure for increasing or decreasing the digital signals using a first digital gain and outputting the digital signals within a predetermined area of ​​the pixel array unit using a second digital gain.

Images