Distance measuring device, distance measuring method, and computer program

Abstract

The system accurately calculates the distance to the subject even when visibility is poor in the direction of the moving object. [Solution] The system comprises a field of view determination unit that determines the field of view in the direction of travel of a moving object, a light emitter that emits pulsed light, a photoelectric conversion element having multiple pixels arranged, each having a first photoelectric conversion unit and a second photoelectric conversion unit that generate a first photoelectric conversion signal and a second photoelectric conversion signal having a predetermined parallax, respectively, a control unit that range gate controls the emission time, exposure time and exposure timing, and a distance measuring unit that calculates the distance to the subject based on the first photoelectric conversion signal and the second photoelectric conversion signal. The control unit switches between executing a process to calculate the distance to the subject based on the first and second photoelectric conversion signals generated without range gate control, or executing a process to calculate the distance to the subject based on the first and second photoelectric conversion signals generated by range gate control, based on the determination result of the field of view determination unit.

Description

Technical Field

[0001] The present invention relates to a distance measuring device, a distance measuring method, a computer program, and the like.

Background Art

[0002] As a device for measuring the distance to a subject, a distance measuring camera system using the triangulation principle is mounted on many automobiles in order to realize the function of an advanced driving assistance system (ADAS). Examples of distance measuring camera systems include a stereo camera system and an imaging plane phase difference distance measuring system.

[0003] The stereo camera system arranges two cameras in parallel at a predetermined interval (baseline length), detects the amount of deviation due to the parallax of the subject imaged in the images taken by each camera, and calculates the distance to the subject based on this.

[0004] The imaging plane phase difference distance measuring system performs imaging and distance measurement with a single camera provided with an image sensor called an imaging plane phase difference image sensor. That is, by photoelectrically converting the light passing through different pupils of the imaging optical system with a plurality of photoelectric conversion units on the imaging plane phase difference image sensor, two images having a phase difference are generated, and the distance to the subject is calculated based on the phase difference.

[0005] These distance measuring camera systems may, for example, erroneously calculate the distance to the subject in bad weather. For example, bad weather refers to a situation where particles such as rain, fog, and snow obstruct the visibility of the subject. [[ID=—]]

[0006] In the above distance measuring camera system, if the camera captures the light diffusely reflected by the particles in front of the subject, an image signal other than the subject may be generated, and the distance to the subject may not be accurately calculated due to the influence of the image signal.

[0007] On the other hand, Patent Document 1 describes a configuration in which a light source and a stereo camera are range-gate controlled to acquire images of only subjects within a certain distance range, with each camera capturing these images clearly. Range-gate control involves emitting pulsed light in front of the camera at a predetermined period, and the image sensor inside the camera performing an exposure operation (accumulation operation) at a predetermined timing corresponding to the target distance. This makes it possible to capture images clearly of only subjects within the target distance.

[0008] Patent Document 1 describes a method in which the amount of parallax-induced displacement of a subject captured in each range-gate image obtained by range-gate control of two cameras is detected, and the distance to the subject is calculated based on this. By using this technology, the distance to the subject can be calculated accurately even in bad weather. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Patent No. 6293134 [Overview of the project] [Problems that the invention aims to solve]

[0010] However, in the technology described in Patent Document 1, in order to measure the distance of a subject with high accuracy under adverse weather conditions, it is necessary to synchronize images acquired by two cameras on the order of nanoseconds. For example, to acquire a range gate image with a target distance range of 3m, assuming the speed of light is 300,000 km / s, the required camera exposure time would be 20ns.

[0011] Therefore, generating highly accurate stereo ranging images requires two range gate images synchronized by nanoseconds. However, synchronizing two cameras by nanoseconds presents challenges in practical application, such as the need for precise wiring length and clock phase accuracy.

[0012] One of the objectives of the present invention is to provide a distance measuring device that can accurately calculate the distance to a subject even when the visibility in the direction of travel of a moving object is poor. [Means for solving the problem]

[0013] The present invention In a rangefinder, A visibility condition determination unit that determines the visibility conditions in the direction of travel of a moving object, A light emitter that emits pulsed light in the direction of travel, A photoelectric conversion element comprising a plurality of pixels, each having a first photoelectric conversion unit and a second photoelectric conversion unit that generate a first photoelectric conversion signal and a second photoelectric conversion signal having a predetermined parallax, A control unit that range-gates the emission time of the light emitter and the exposure time and exposure timing of the photoelectric conversion element, The system includes a distance measuring unit that calculates the distance to the subject based on the first photoelectric conversion signal and the second photoelectric conversion signal, The control unit switches between performing a process to calculate the distance to the subject based on the determination result of the field of view determination unit, using the first and second photoelectric conversion signals generated without range gate control, and performing a process to calculate the distance to the subject based on the first and second photoelectric conversion signals generated by range gate control. It is characterized by the following: [Effects of the Invention]

[0014] According to the present invention, a distance measuring device can be realized that can accurately calculate the distance to a subject even when the visibility in the direction of travel of a moving object is poor. [Brief explanation of the drawing]

[0015] [Figure 1] Figures (A) to (D) show examples of camera 100 mounting positions on the mobile body 200 according to Embodiment 1. [Figure 2] Figures (A) to (C) show examples of mounting positions for the light emitter 300 on the mobile body 200 according to Embodiment 1. [Figure 3] It is a functional block diagram showing a configuration example of the camera 100, the light emitter 300, and the moving body 200 according to Embodiment 1. [Figure 4] (A) and (B) are schematic diagrams showing a configuration example of the photoelectric conversion element 102 according to Embodiment 1. [Figure 5] (A) to (D) are schematic diagrams for explaining the relationship between the subject distance and the incident light in the imaging plane phase difference method. [Figure 6] It is a diagram showing the progress of the light emitted from the light emitter 300 and its reflected light and the exposure timing of the camera 100 in the range gate control according to Embodiment 1. [Figure 7] It is a timing chart showing an example of the light emission and exposure control operation in one frame period in the range gate control according to Embodiment 1. [Figure 8] It is a flowchart showing an example of the image generation process in the distance measurement method according to Embodiment 1. [Figure 9] (A) and (B) are timing charts showing an example of the light emission and exposure control operation in one frame period in the imaging plane phase difference range gate distance measurement mode according to Embodiment 1. [Figure 10] (A) to (D) are diagrams for explaining an example of the composite image generated by the imaging plane phase difference range gate distance measurement mode according to Embodiment 1. [Figure 11] It is a flowchart showing an example of the parallax reliability improvement process in the distance measurement method according to Embodiment 2. [Figure 12] (A) and (B) are diagrams for explaining a method of determining whether the target distance range R information in the range gate image signal according to Embodiment 2 matches the distance measurement result of the subject in the imaging plane phase difference range gate distance measurement mode. [Figure 13] It is a diagram showing a configuration example of the photoelectric conversion element 102 according to Embodiment 3. [Figure 14] It is a schematic diagram showing a configuration example of the photoelectric conversion element 102 including an APD [Figure 15] It is a diagram showing a configuration example of the circuit board 921 [Figure 16] This figure shows an example of an equivalent circuit of the signal processing circuit 1060 according to Embodiment 3. [Figure 17] This diagram schematically illustrates the relationship between the operation of the APD2010 and its output signals. [Modes for carrying out the invention]

[0016] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0017] <Embodiment 1> Embodiment 1 of the present invention will be described below. In Embodiment 1, the camera 100 and light emitter 300 are attached to a mobile body 200 such as a vehicle, but are not limited to this. Details of the camera 100, light emitter 300, and mobile body 200 will be described later.

[0018] Figures 1(A) to 1(D) show examples of camera mounting positions on the mobile body 200 according to Embodiment 1. The camera 100 may be mounted inside the vehicle interior of the mobile body 200, for example, on the inside of the front windshield of the mobile body 200 as shown in Figure 1(A), or on the dashboard as shown in Figure 1(B).

[0019] The camera 100 may also be mounted outside the vehicle interior of the mobile unit 200, for example, around the headlights of the mobile unit 200 as shown in Figure 1(C), or around the side mirrors as shown in Figure 1(D).

[0020] Figures 2(A) to 2(C) show examples of mounting positions for the light emitter 300 on the mobile body 200 according to Embodiment 1. The light emitter 300 may be mounted, for example, around the headlamp of the mobile body 200 as shown in Figure 2(A), on the roof as shown in Figure 2(B), or around the side mirror as shown in Figure 2(C).

[0021] Figure 3 is a functional block diagram showing an example configuration of the camera 100, light emitter 300, and mobile body 200 according to Embodiment 1. Note that some of the functional blocks shown in Figure 3 are realized by having a computer (not shown) included in the camera 100, light emitter 300, and mobile body 200 execute a computer program stored in memory (not shown) as a storage medium.

[0022] However, some or all of these may be implemented in hardware. Hardware, dedicated circuits (ASICs), and processors (reconfigurable processors, DSPs) can be used. Furthermore, each of the components shown in Figure 3 does not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0023] The camera 100 includes an imaging optical system 101, a photoelectric conversion element 102, an image processing unit 103, a distance measuring unit 104, a camera control unit 105, a memory unit 106, a communication unit 107, a bad weather determination unit 108, and the like.

[0024] The imaging optical system 101 forms an image (optical image) of the subject on the photoelectric conversion element 102 and has an exit pupil at a predetermined distance from the photoelectric conversion element 102.

[0025] The photoelectric conversion element 102 is a semiconductor image sensor element, such as a CMOS (Complementary Metal Oxide Semiconductor) sensor. The photoelectric conversion element 102 comprises a pixel region in which multiple pixels having a photoelectric conversion function are arranged in two dimensions, and each pixel region has a microlens and two photoelectric conversion units.

[0026] The subject image formed on the photoelectric conversion element 102 via the microlens is photoelectrically converted by two photoelectric conversion units, generating a first photoelectric conversion signal and a second photoelectric conversion signal, respectively. The two photoelectric conversion units each receive light from different exit pupils of the imaging optical system.

[0027] The image processing unit 103 performs image processing on the image signal output from the photoelectric conversion element 102, such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression, to generate the final image signal. The output of the image processing unit 103 is supplied to the distance measuring unit 104 and the camera control unit 105, as well as to the ECU 201 (Electric Control Unit) of the mobile unit 200.

[0028] The distance measuring unit 104 performs image recognition based on the image signal supplied from the image processing unit 103 to recognize surrounding objects such as people and vehicles. It also calculates the distance to the object based on the phase difference (shift due to parallax) of the first and second photoelectric conversion signals supplied from the image processing unit 103.

[0029] The camera control unit 105 has a built-in CPU and memory that stores computer programs, and the CPU controls each part of the camera 100 by executing the computer programs stored in the memory.

[0030] Furthermore, the camera control unit 105 functions as a control means, transmitting a reference clock signal repeatedly output at predetermined intervals to the photoelectric conversion element 102, etc., thereby controlling the length of the exposure period (charge storage period) for each frame and the timing of various control signals. The camera control unit 105 also transmits the reference clock signal and control signals to the light emitter 300 via the communication unit 107.

[0031] The storage unit 106 includes, for example, a recording medium such as a memory card or a hard disk, and can store and read image signals.

[0032] The communication unit 107 is equipped with wireless and wired interfaces and outputs the generated image signal to the outside of the camera 100 and receives various signals from the outside. The bad weather determination unit 108 determines the visibility conditions according to the weather conditions in the surrounding area in front of the moving body 200 and outputs the determination result to the camera control unit 105. In other words, the bad weather determination unit 108 functions as a visibility condition determination unit that determines the visibility conditions in the direction of travel of the moving body (e.g., fog, rain, snow, etc.).

[0033] The method for acquiring weather conditions may be, for example, based on weather information obtained from the web and location information of the mobile body 200 based on GPS. Alternatively, the technology described in, for example, Japanese Patent Application Publication No. 2008-33872 may be used. That is, the area directly illuminated by the headlights of the mobile body 200 and the area not directly illuminated may be photographed by a camera 100 attached to the mobile body, and the bad weather conditions may be acquired from the difference in brightness between the two.

[0034] The light emitter 300 includes a light-emitting unit 301, a light-emitting control unit 302, and a communication unit 303. The communication unit 303 communicates with the communication unit 107 of the camera 100, receives setting information, control signals, and reference clock signals from the camera control unit 105 to the light-emitting control unit 302, and supplies them to the light-emitting control unit 302.

[0035] The light emission control unit 302 receives a reference clock signal transmitted by the camera control unit 105 of the camera 100 via the communication unit 303, generates a pulse signal at a predetermined timing based on the reference clock signal, and outputs it to the light emission unit 301.

[0036] Here, the light emission control unit 302 can set the period from the reference clock signal to the output of a pulse, the pulse output width, the pulse non-output width, and the repetition period and number of repetitions from one pulse output to the next.

[0037] When the camera control unit 105 transmits a predetermined control signal to the light emission control unit 302 via the communication unit 107 and the communication unit 303, a pulse signal is output to the light emission unit 301 at a predetermined timing based on the reference clock signal, and the light emission period of the light emitter 300 is controlled. In this way, the light emission control unit 302 controls the light emission in synchronization with the reference clock signal input to the camera 100.

[0038] The light-emitting unit 301 is, for example, a near-infrared LED positioned in front of the moving body 200, and consists of a lens and a light-emitting unit. That is, the light emitter 300 can emit pulsed light in the direction of movement of the moving body. In this embodiment, the camera control unit 105 functions as a control unit that range-gates the light emission time of the light emitter and the exposure time and exposure timing of the photoelectric conversion element.

[0039] Furthermore, the frequency band of the light emitted by the light-emitting unit 301 is not limited to this, and may be in the visible light region, for example. That is, the light emitted by the light emitter is either visible light or invisible light. The light-emitting unit 301 may also be a light emitter provided on the mobile body 200, such as a headlight. The light-emitting unit 301 outputs pulsed light for a predetermined emission time in response to a pulse signal output to the light emission control unit 302.

[0040] In this way, the same reference clock signal supplied to the photoelectric conversion element 102 is also transmitted to the light emitter 300, and the light emitter 300 performs light emission control based on the reference clock signal. This synchronizes the exposure (charge accumulation) timing of the photoelectric conversion element 102 with the light emission timing of the light emitter 300.

[0041] The ECU 201 of the mobile unit 200 has a built-in CPU and memory that stores computer programs. The CPU executes the computer programs stored in the memory to control various parts of the mobile unit 200, such as the vehicle control unit 202.

[0042] The output of the distance measuring unit 104 is supplied to the vehicle control unit 202 and the display unit 203 via the ECU 201. The vehicle control unit 202 functions as a movement control means that controls the driving, stopping, and direction of the vehicle as a moving object based on the output of the ECU 201.

[0043] Furthermore, the display unit 203 functions as a display means and includes a display element such as a liquid crystal device or an organic EL, and is mounted on the dashboard of the mobile unit 200, for example.

[0044] In this embodiment, the ECU 201 receives distance measurement result information from the distance measurement unit 104 and can perform vehicle stopping control (such as automatic braking) according to the content of the distance measurement result. The ECU 201 also receives distance measurement processing data from the distance measurement unit 104 and transmits it to the display unit 203.

[0045] Based on the output of the ECU 201, the display unit 203 displays various information to the driver of the mobile unit 200, such as images generated by the photoelectric conversion element 102, distance measurement results from the distance measurement unit 104, and the vehicle's driving status, for example, using a GUI.

[0046] Furthermore, the image processing unit 103, distance measuring unit 104, etc. shown in Figure 3 do not necessarily have to be mounted on the mobile body 200. For example, they may be installed on an external terminal provided separately from the mobile body 200 for remotely controlling the mobile body 200 or for monitoring the movement of the mobile body.

[0047] Next, the distance measurement principle using an imaging plane phase-difference image sensor utilizing a photoelectric conversion element 102 will be explained using Figures 4 and 5.

[0048] Figures 4(A) and 4(B) are schematic diagrams showing an example configuration of the photoelectric conversion element 102 according to Embodiment 1, where Figure 4(A) is a top view of the photoelectric conversion element 102 as seen from the direction of light incidence. The photoelectric conversion element 102 is composed of multiple 2x2 pixel groups 400 arranged on a matrix. Each pixel group 400 has four infrared pixels (IR) that detect IR (infrared) light.

[0049] Furthermore, the arrangement of each pixel in the pixel group 400 is not limited to this and may be changed according to the wavelength band of the light emitted from the light emitter 300. The pixel group 400 may consist, for example, of infrared pixels (IR) for detecting IR light, red pixels for detecting the color red, green pixels for detecting green light, and blue pixels for detecting blue light.

[0050] The pixel group 400 may also have two green pixels for detecting green light, a red pixel for detecting red light, and a blue pixel for detecting blue light. In the above pixel arrangement, the two green pixels are arranged diagonally.

[0051] Figure 4(B) is a cross-sectional view of the pixel group 400 in the I-I' section of Figure 4(A). Each pixel consists of a microlens 413, a light guide layer 414, and a light receiving layer 415. The light guide layer 414 has a microlens 413 for efficiently guiding the light incident on the pixel to the light receiving layer 415, a color filter that transmits light in the wavelength band detected by each pixel, and wiring for pixel readout and pixel driving.

[0052] The light-receiving layer 415 is a photoelectric conversion unit that converts light incident through the light-guide layer 414 into electrical signals and outputs them as electrical signals. The light-receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412. Thus, each pixel has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0053] Figures 5(A) to 5(D) are schematic diagrams illustrating the relationship between the distance to the subject and the incident light in the image plane phase-difference imaging method. Figure 5(A) is a schematic diagram showing the light incident on the exit pupil 500 of the imaging optical system 101 and the infrared pixel IR of the photoelectric conversion element 102. The photoelectric conversion element 102 has multiple pixels, but for simplicity, we will explain using one infrared pixel IR.

[0054] The microlens 413 of the infrared pixel IR is arranged such that it is optically conjugate to the exit pupil 500 and the light-receiving layer 415. As a result, light passing through the first pupil region 510, which is a partial pupil region contained within the exit pupil 500, is incident on the first photoelectric conversion unit 411. Similarly, light passing through the second pupil region 520, which is a partial pupil region, is incident on the second photoelectric conversion unit 412.

[0055] Each pixel's first photoelectric conversion unit 411 converts the received light into photoelectric signals and outputs a first photoelectric conversion signal. A first image signal is generated from the first photoelectric conversion signals output from the multiple first photoelectric conversion units 411 contained in the photoelectric conversion element 102. The first image signal shows the intensity distribution of the image formed on the photoelectric conversion element 102 by light that has mainly passed through the first pupil region 510.

[0056] Each pixel's second photoelectric conversion unit 412 converts the received light into photoelectric signals and outputs a second photoelectric conversion signal. A second image signal is generated from the second photoelectric conversion signals output from the multiple second photoelectric conversion units 412 contained in the photoelectric conversion element 102. The second image signal shows the intensity distribution of the image formed on the photoelectric conversion element 102 by light that has mainly passed through the second pupil region 520.

[0057] As described above, the photoelectric conversion element of this embodiment has a plurality of pixels, each having a first photoelectric conversion unit and a second photoelectric conversion unit that generate a first photoelectric conversion signal and a second photoelectric conversion signal having a predetermined parallax.

[0058] The relative positional shift between the first and second image signals (hereinafter referred to as the disparity amount or phase difference) corresponds to the amount of defocus. The relationship between the disparity amount and the amount of defocus is explained using Figures 5(B) to (D).

[0059] Figures 5(B) to 5(D) are schematic diagrams showing the relationship between the photoelectric conversion element 102, the imaging optical system 101, and the position of the subject. In the figures, 511 represents the first light passing through the first pupil region 510, and 521 represents the second light passing through the second pupil region 520.

[0060] Figure 5(B) shows the state when the image is in focus, with the first light 511 and the second light 521 converging on the imaging optical system 101. At this time, the parallax between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is 0.

[0061] Figure 5(C) shows a state where the image is defocused in the negative direction of the w-axis. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a negative value.

[0062] Figure 5(D) shows the state where the image is defocused in the positive w-axis direction. At this time, the disparity between the first image signal formed by the first light 511 and the second image signal formed by the second light 521 is not zero, but has a positive value.

[0063] A comparison of Figures 5(C) and (D) shows that the direction in which parallax occurs changes depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax is determined by the amount of defocus.

[0064] Therefore, the amount of disparity between the first image signal and the second image signal can be detected using a region-based matching method, such as block matching, and converted into a defocus amount. Block matching is a method in which a region with a high degree of similarity is searched for in the other image for a selected region in one image, and the positional difference between that region and the selected region is used as the disparity.

[0065] Furthermore, by using the imaging formula of the imaging optical system 101, the amount of defocus on the image side can be converted into the distance to the object. The imaging formula of the imaging optical system 101 is given by the following equation (1), where f is the focal length of the photoelectric conversion element 102, Ipp is the distance from the principal point on the image side to the photoelectric conversion element 102, ΔL is the amount of defocus, and D is the distance to the object.

number

[0066] The above is an explanation of the distance measurement principle using the image plane phase difference method. The distance measurement unit 104 of this embodiment calculates the distance to the subject by the above calculation based on the first photoelectric conversion signal and the second photoelectric conversion signal.

[0067] Figure 6 shows the propagation of light emitted from the light emitter 300 and its reflected light, as well as the exposure timing of the camera 100, in a range gate control according to Embodiment 1.

[0068] Using Figure 6, we will explain a method for acquiring an image capturing the target distance (range gate image) by performing range gate control, which synchronizes the light emission timing and exposure timing according to the target distance.

[0069] A camera that acquires images at a target distance using range gate control in this manner is called a range gate camera. In Figure 6, the horizontal axis represents distance and the vertical axis represents time.

[0070] First, let's explain the horizontal axis. Between distance x1 and distance x2, there is fog 610 due to bad weather, and at distance x3, there is a vehicle 620. Also, in Figure 6, range gate control is used, with position D as the starting point, and range gate images are acquired within the target distance range R from there.

[0071] In this case, the target distance range R is the target distance range to be imaged. At this time, the vehicle 620 is located within the target distance range R.

[0072] Next, let's explain the vertical axis. Time 0 is defined as the start time of light emission in the light emitter 300, and time tf is defined as the end time of light emission. In this case, the light emission period is tf.

[0073] Furthermore, when acquiring a range gate image within the target distance range R, starting from position D, the exposure start time is defined as time t1 and the exposure end time as time t2. In this embodiment, "exposure" refers to photoelectric conversion in the photoelectric conversion element 102, "exposure start" refers to the start of photoelectric conversion, and "exposure end" refers to the end of photoelectric conversion.

[0074] Time t1 is the moment when the synchrotron radiation emitted from the light emitter 300 at time 0 returns to the camera 100 as reflected light from a distance D. Also, t2 is the moment when the synchrotron radiation emitted from the light emitter 300 returns to the camera 100 as reflected light from a point that has traveled a distance R from the target distance range from distance D.

[0075] Furthermore, in Figure 6, time t3 is defined as the timing when the first reflected light returns to camera 100, and time t4 is defined as the timing when the last reflected light from fog 610 returns to camera 100.

[0076] In range gate control, exposure is not performed during the period from time t3 to time t4 when the reflected light from the fog 610 reaches the camera 100. Exposure (photoelectric conversion) is performed only during the period from time t1 to time t2 when the reflected light from distance D to the target distance range R reaches the camera. Therefore, the fog 610 can be removed while a clear image of the vehicle 620 can be acquired.

[0077] Here, we will explain the time it takes for reflected light from an object at distance x to return to camera 100. Let time tr be the timing when the synchrotron radiation that started emitting at time 0 strikes an object at distance x and returns to the imaging unit as reflected light. At this time, the relationship between the timing time tr of the reflected light returning and the distance x to the object to be imaged is given by the following equation (2). Time tr=2x / speed of light c (approximately 3×10^8m / s)...Equation (2)

[0078] As shown in Figure 6, when the imaging range is defined as distance D to the target distance range R, the exposure timing time t1 at the start of the range can be obtained by substituting distance D into distance x in equation (1) above, as shown in equation (3) below. Time t1=2D / speed of light c...Equation (3)

[0079] Furthermore, the exposure timing time t2 at the end of the range can be obtained by substituting the distance (D+R) into the time t4 in equation (1) above and adding the time tf, as shown in equation (4) below. t1(A)=tf+2(D+R) / speed of light c...Equation (4)

[0080] In this way, the time tf from the start to the end of light emission, the time t1 from the start of light emission to the start of exposure, and the time t2 until the end of exposure are controlled according to the distance x (target distance range R) to be imaged. This enables range gate control that allows clear imaging of subjects at the target distance even if there is fog or other obstructions between the camera and the target distance.

[0081] Figure 7 is a timing chart showing an example of light emission and exposure control operation during a one-frame period in range gate control according to Embodiment 1.

[0082] In Figure 7, the "vertical synchronization signal" indicates the frame period of the image, with the period between one low pulse and the next low pulse being one frame period. "Emission control" indicates the emission timing of the light emitter 300, with light emission occurring from the light emitter 300 while the level is high. "Photoelectric conversion element exposure control" is an exposure control signal supplied from the camera control circuit, with photoelectric conversion occurring in the photoelectric conversion element 102 while the level is high.

[0083] Specifically, for example, at the rising edge of the exposure control signal, a reset switch (not shown) for resetting the charge in the photoelectric conversion unit is turned off to start charge accumulation. Then, at the falling edge of the exposure control signal, the charge accumulated in the photoelectric conversion unit is transferred to a memory (not shown) within the pixel and held there, and then the reset switch is turned on again.

[0084] In range gate control, the light emission period is controlled in a pulsed manner as shown in Figure 7, and exposure (photoelectric conversion) of the photoelectric conversion element 102 is performed only on reflected light from a specific target distance range R.

[0085] In Figure 7, as described above, tf represents the emission period from the start to the end of emission, t1 represents the time from the start of emission to the start of exposure, and t2 represents the time from the start of emission to the end of exposure. Also, as described above, t1 represents the period from the start of emission until the light reaches a specific target distance range R and the reflected light returns to the camera 100.

[0086] Furthermore, as mentioned above, the time from t1 to t2 is the period during which reflected light within a specific target distance range R is exposed. In order to properly perform range gate control, it is necessary to synchronize the timing of the start of light emission and the start of exposure according to a predetermined target distance range.

[0087] Furthermore, as mentioned above, in this embodiment, the camera control unit 105 synchronizes the photoelectric conversion element 102 and the light emission control unit 302 by transmitting the same reference clock signal. The period from the start of one light emission to the start of the next, as shown in the light emission control on the timing chart, constitutes the range gate operation cycle.

[0088] Then, in one range gate operation cycle, the light exposed to the photoelectric conversion element 102 is converted into an electric charge and stored in the memory within the pixel of the photoelectric conversion element 102.

[0089] In this state, as shown in Figure 7, the next range gate operation cycle within the same 1-frame period is performed, and the light newly exposed to the photoelectric conversion element 102 is converted into electric charge and added up and stored in the memory within the pixel of the photoelectric conversion element 102. The period from one emission to the next is set based on the time until the reflected light has sufficiently attenuated and no longer returns to the camera 100.

[0090] As shown in Figure 7, the range gate operation cycle is performed a predetermined number of times within one frame period, and the charge converted photoelectrically in the last range gate operation cycle within one frame period is further added and stored in the memory within the pixel of the photoelectric conversion element 102.

[0091] Subsequently, the charge in the memory is read out via a vertical signal line, and then the charge in the memory is reset by a predetermined reset switch.

[0092] Thus, in this embodiment, the exposure period is synchronized with the light emission from the light emitter 300, making it possible to obtain a clear image for the target range even in adverse weather conditions such as fog. On the other hand, range gate control can only accurately measure distance within a target distance range R that depends on the exposure time of the camera 100 and the light emission time of the light emitter 300.

[0093] In other words, to improve the distance measurement accuracy, it is necessary to reduce the target distance range R. Therefore, it is conceivable to shorten the exposure time of the camera 100 and the light emission time of the light emitter 300, but this is difficult due to the physical control limitations of the camera 100 and the light emitter 300.

[0094] Furthermore, reducing the target distance range R and attempting to perform range gate imaging over long distances increases the number of exposures for camera 100, resulting in higher power consumption. Additionally, measuring distances over long distances takes considerable time.

[0095] To address this issue, the distance measurement process in the distance measurement method according to this embodiment will be explained with reference to Figure 8. Figure 8 is a flowchart showing an example of the image generation process in the distance measurement method according to Embodiment 1.

[0096] Furthermore, the CPU and other components within the camera control unit 105 execute the computer program stored in memory, thereby sequentially performing the operations of each step in the flowchart shown in Figure 8.

[0097] In step S101, the adverse weather determination unit 108 acquires weather information in front of the moving body 200 in the manner described above and outputs it to the camera control unit 105.

[0098] In step S102, the camera control unit 105 determines whether or not the weather is bad based on the weather information acquired in step S101. If it is determined in step S102 that the weather is not bad, the process proceeds to step S103.

[0099] In step S103, the camera control unit 105 starts the image plane phase difference distance measurement mode, thereby performing the image plane phase difference distance measurement described in Figure 5(A). Here, the image plane phase difference distance measurement mode refers to a mode in which a first photoelectric conversion signal and a second photoelectric conversion signal are generated with an exposure time set according to the brightness of the subject.

[0100] Specifically, for a predetermined exposure period within one frame, the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 of each pixel of the photoelectric conversion element 102 generate a first photoelectric conversion signal and a second photoelectric conversion signal, respectively, based on the light that has passed through the imaging optical system 101.

[0101] The exposure time in step S103 is set, for example, according to the luminance signal level of the image up to the previous frame period. The luminance signal level of the image corresponds to the brightness of the subject.

[0102] Next, in step S104, an image signal is acquired. That is, a first image signal and a second image signal are acquired based on the first and second photoelectric conversion signals of multiple pixels. Then, each image signal is output to the image processing unit 103.

[0103] On the other hand, if bad weather is determined in step S102, the process proceeds to step S105. In step S105, the imaging plane phase-difference range-gate distance measurement mode is started. That is, the camera control unit 105 performs range-gate control on the light emitter 300 and the photoelectric conversion element 102. Through range-gate control, a first photoelectric conversion signal and a second photoelectric conversion signal for the target distance range are generated.

[0104] In this context, the image plane phase-difference range gate distance measurement mode means that a first photoelectric conversion signal and a second photoelectric conversion signal for the target distance range are generated by range gate control.

[0105] In step S106, the camera control unit 105 acquires a first range gate image signal and a second range gate image signal based on the first and second photoelectric conversion signals of multiple pixels for a specific target distance, which are obtained by range gate control. Then, it outputs each range gate image signal to the image processing unit 103.

[0106] Figures 9(A) and (B) are timing charts showing an example of light emission and exposure control operation for one frame period in the imaging plane phase difference range gate distance measurement mode according to Embodiment 1.

[0107] Figure 9(A) shows an example of the flashing timing and exposure timing per frame period when the camera 100 position is set to 0m as the reference point and the target distance range 1 is set to, for example, 0m to 30m. Note that the target distance may be changed according to the movement speed of the moving object 200, for example, so that the distance increases as the movement speed increases.

[0108] For example, let's set the target distance range 1 to 0m to 30m. In that case, the emission time from the start to the end of emission of light from the light emitter 300 is 200ns, the time from the start of emission to the start of exposure in the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 is 0ns, and the time from the start of emission to the end of exposure is 400ns.

[0109] In the timing chart of Figure 9(A), the period from the start of light emission by the light emitter 300 to the start of the next light emission is the range gate operation cycle. In one range gate operation cycle, the light exposed (photoelectrically converted) by the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 is converted into electric charge and stored separately in memory (not shown) within the photoelectric conversion element 102.

[0110] In this state, the next range gate operation cycle is performed, and the light newly exposed by the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 is converted into electric charge, which is then separately added and stored in the memory within the photoelectric conversion element 102.

[0111] In this way, the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 acquire a first range gate image signal (A) and a second range gate image signal (A) in the target distance range of 0 to 30 m, respectively.

[0112] Here, the first range-gate image signal (A) is an image signal acquired in the target distance range 1 (0m to 30m) by range-gate control of the first photoelectric conversion unit 411. The second range-gate image signal (A) is an image signal acquired in the target distance range 1 (0m to 30m) by range-gate control of the second photoelectric conversion unit 412.

[0113] Figure 9(B) shows an example of the light emission timing and exposure timing per frame period when the camera 100 position is set to 0m and the target distance range 2 is set to, for example, 30m to 60m.

[0114] The target distance range 2 is set to 30m to 60m. In this case, the emission time from the start to the end of emission of light emitter 300 is 200ns, the time from the start of emission to the start of exposure in the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 is 200ns, and the time from the start of emission to the end of exposure is 600ns.

[0115] Similar to Figure 9(A), the period from the start of light emission of the light emitter 300 to the start of the next light emission in the timing chart constitutes the range gate operation cycle. Also, similar to Figure 9(A), in one range gate operation cycle, the first range gate image signal (B) and the second range gate image signal (B) are acquired by the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412, respectively, within the target distance range 2 (30 to 60 m).

[0116] The first range-gate image signal (B) is an image signal acquired in the target distance range 2 (30m to 60m) by range-gate control of the first photoelectric conversion unit 411. The second range-gate image signal (B) is an image signal acquired in the target distance range 2 (30m to 60m) by range-gate control of the second photoelectric conversion unit 412.

[0117] In this embodiment, the multiple first photoelectric conversion signals and the multiple second photoelectric conversion signals generated by performing range gate control multiple times are added together.

[0118] In step S107, the image processing unit 103 performs image processing on each image signal acquired in step S104 or step S106. Here, image processing refers to generating a final image signal through processes such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression.

[0119] In step S108, the amount of parallax is calculated and converted into a distance value. An image is also generated based on the distance value. That is, the distance measuring unit 104 calculates the amount of parallax from each image signal acquired in step S107 based on the distance measuring principle of the imaging plane phase difference method shown in Figure 5(A), calculates a distance value, and generates a display image based on the distance value. Furthermore, in step S108, image recognition is performed based on each image signal.

[0120] In addition, in the image plane phase difference distance measurement mode, step S108 performs image recognition based on the first and second image signals acquired in step S104, and calculates the subject distance value based on the parallax amount of the first and second image signals.

[0121] On the other hand, in the imaging plane phase difference range gate distance measurement mode, in step S108, image recognition is performed based on the first range gate image signal (A) and the second range gate image signal (A) acquired in step S106.

[0122] Furthermore, in step S108, a distance value is calculated based on the disparity amount of the first range gate image signal (A) and the second range gate image signal (A), and an image 710 as shown in Figure 10(B) is generated based on the distance value.

[0123] Furthermore, image 710 may be a summation of both the first range gate image signal (A) and the second range gate image signal (A), or it may display only one of them.

[0124] Furthermore, in step S108, similar to the above, a distance value is calculated based on the disparity amount of the first range gate image signal (B) and the second range gate image signal (B) acquired in step S106, and an image 720 as shown in Figure 10(C) is generated based on the distance value.

[0125] Furthermore, image 720 may be a composite of both the first range gate image signal (B) and the second range gate image signal (B), or it may display only one of them.

[0126] Subsequently, by performing an image compositing process on images 710 and 720, a composite image 730 for the range from 0 to 60m is generated, as shown in Figure 10(D).

[0127] Here, image synthesis is the process of obtaining an image for the range 0m to 60m by superimposing image 710, which covers the target distance range R0 to 30m, with image 720, which covers the target distance range R30m to 60m.

[0128] Figures 10(A) to (D) illustrate examples of composite images generated by the imaging plane phase difference range gate distance measurement mode according to Embodiment 1.

[0129] Figure 10(A) shows an example of an image 700 that is not range-gate controlled in this embodiment. That is, it shows an example of an image generated in the imaging plane phase difference distance measurement mode in step S103, for example. In the case of an image generated in the imaging plane phase difference distance measurement mode, the image 700 may be an additive composite of both the first image signal and the second image signal, or it may display only one of them.

[0130] In Figure 10(A), a pedestrian 830, fog 810, and vehicle 820 are photographed. Although vehicle 820 is beyond fog 810, in images without range gate control, vehicle 820 is unclear due to fog 810.

[0131] Therefore, the distance measuring unit 104 cannot accurately measure the distance of the vehicle 820 during the distance measuring process. In this way, the image 700, which is not controlled by range gate, can capture both the vehicle 820 and the pedestrian 830 in one frame, but the image becomes unclear in bad weather conditions such as fog.

[0132] Next, Figure 10(B) shows image 710 in the target distance range 1 (0m to 30m) generated by the imaging plane phase difference range gate distance measurement mode shown in Figure 9(A). Image 710 is subjected to recognition processing by the distance measurement unit 104 and a pedestrian 830 is detected.

[0133] Furthermore, the camera control unit 105 performs range gate control of the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412, which indicates that image 710 falls within the target distance range 1 (0m to 30m), and these values ​​(0m, 30m) are displayed in image 710.

[0134] Furthermore, based on the first range gate image signal (A) and the second range gate image signal (A) measured by the distance measuring unit 104, the distance to the pedestrian 830 is calculated to be 5m and is displayed in image 710.

[0135] In Figure 10(B), pedestrian 830, located within target distance range 1 (0m to 30m), is captured in the image, making it possible to accurately calculate the distance to pedestrian 830. On the other hand, fog 810 and vehicle 820, located within target distance range 2 (30m to 60m), are not captured in the image.

[0136] Next, Figure 10(C) shows image 720 in the target distance range 2 (30-60m) generated by the imaging plane phase difference range gate distance measurement mode shown in Figure 9(B).

[0137] Image 720 shows that the fog 810 has thinned due to range gate control, allowing for a clear image of the vehicle 820, which in turn enables accurate recognition and distance measurement by the range measuring unit 104.

[0138] Image 720 is subjected to recognition processing by the distance measuring unit 104, and the vehicle 820 is detected. Furthermore, range gate control of the first photoelectric conversion unit 411 and the second photoelectric conversion unit 412 by the camera control unit 105 determines that the vehicle 820 falls within target distance range 2 (30m to 60m), and these values ​​(30m, 60m) are displayed in image 720.

[0139] Furthermore, based on the first range gate image signal (B) and the second range gate image signal (B) from the distance measuring unit 104, the distance to the vehicle 820 is calculated to be 40m and is displayed in image 720. On the other hand, pedestrians 830, which are located within the target distance range 1 (0-30m), are not captured in image 720.

[0140] Next, Figure 10(D) shows a composite image 730 obtained by combining images 710 and 720 at a distance of 0 to 60m. The composite image 730 also contains the distance information of images 710 and 720, and displays the combined distance range (0m to 60m) of target distance range 1 and target distance range 2, as well as the distances to the vehicle 820 and pedestrian 830.

[0141] As a result, the driver of, for example, a moving vehicle 200 who views the composite image 730 can understand that vehicle 820 is located beyond the fog 810 and that vehicle 820 is 40m away. Furthermore, if the ECU 201 determines that emergency braking is necessary based on this distance measurement result, it can promptly instruct the vehicle control unit 202 to apply automatic braking.

[0142] Furthermore, in the image plane phase-difference range gate distance measurement mode, the target distance range R in Figure 6 may be divided into, for example, three or more range gates for distance calculation. In that case, the image signal with less noise from each range gate image signal may be combined before calculating the distance. Doing so reduces the number of distance calculation processes and lowers the load on the distance measurement process.

[0143] In step S109, it is determined whether the mobile body 200 is in a moving state. If it is, the process proceeds to step S101. If it is not in a moving state, the process flow shown in Figure 8 is terminated.

[0144] As described above, in this embodiment, based on the determination result of the field of view determination unit, a process is executed to calculate the distance to the subject based on the first and second photoelectric conversion signals generated without range gate control. Alternatively, the process is switched to execute based on the above determination result, using the first and second photoelectric conversion signals generated by range gate control to calculate the distance to the subject.

[0145] As explained using Figure 6, range gate control of only the photoelectric conversion element 102 and the light emitter 300 is insufficient to measure the distance of an object beyond the target distance range R. In contrast, in this embodiment, range gate control is performed on the light emitter 300, the first photoelectric conversion unit 411, and the second photoelectric conversion unit 412, and distance measurement is performed based on the imaging plane phase difference method, making it possible to accurately calculate the distance of an object even in adverse weather conditions.

[0146] <Embodiment 2> Embodiment 2 of the present invention will be described below. In Embodiment 1, a method for accurately measuring the distance of a subject was described by range gate control of the emission time of the light emitter 300 and the exposure time and exposure timing of the photoelectric conversion element 102 for imaging plane phase difference distance measurement.

[0147] Embodiment 2 describes a method for improving the parallax reliability accuracy obtained by range gate control of a photoelectric conversion element 102 for imaging plane phase difference distance measurement based on a target distance range R acquired by range gate control.

[0148] Parallax confidence is an indicator that shows how much error is contained in the calculated parallax value, and each pixel possesses parallax confidence. Furthermore, since distance values ​​are calculated based on parallax, parallax confidence is also an indicator that shows how much error is contained in the distance value.

[0149] For example, the ratio of the standard deviation to the mean value of the signal values ​​included in the matching area can be evaluated as the disparity confidence. When the change in signal values ​​(contrast) within the matching area is large, the standard deviation becomes large.

[0150] Furthermore, a larger amount of light incident on a pixel results in a higher average value and more noise. In other words, the average value of the signal has a positive correlation with the amount of noise. The ratio of the standard deviation to the average value (standard deviation / average value) corresponds to the ratio of the contrast level to the amount of noise.

[0151] If the contrast is sufficiently large relative to the amount of noise, it can be estimated that the error in the calculated parallax value is small. In other words, the higher the parallax confidence, the smaller the error in the calculated parallax value, and the more accurate the parallax value can be said to be.

[0152] Furthermore, the functional block diagram in Embodiment 2 has the same configuration as in Figure 3, and it is preferable that this embodiment be carried out after step S108 in Figure 8. However, it is not limited thereto.

[0153] Figure 11 is a flowchart showing an example of the parallax reliability improvement process in the distance measurement method according to Embodiment 2. The CPU and other components of the camera control unit 105 execute a computer program stored in memory, which sequentially performs the operations of each step in the flowchart of Figure 11.

[0154] In step S201, the camera control unit 105 determines whether the distance measurement process described in Embodiment 1 is in the image plane phase difference range gate distance measurement mode. Specifically, it determines whether the vehicle determined in step S102 in Figure 8 that there was no bad weather in front of it and proceeded to step S103, or whether it determined that there was bad weather and proceeded to step S105.

[0155] If it is determined in step S201 that the imaging plane phase difference range gate distance measurement mode is not in operation, the processing flow shown in Figure 11 is terminated. If it is determined that the imaging plane phase difference range gate distance measurement mode is in operation, the process proceeds to step S202.

[0156] In step S202, it is determined whether the distance measurement result matches the range gate distance information. Note that each pixel in the first range gate image signal (A), the second range gate image signal (A), and the first range gate image signal (B), and the second range gate image signal (B) contains target distance range R information.

[0157] Therefore, the distance measuring unit 104 determines whether the target distance range R information held by each of those pixels matches the distance value held by the pixel corresponding to the subject in the composite image 730 in the imaging plane phase difference range gate distance measuring mode. Here, "match" does not mean a perfect match, but rather a match if it is within a predetermined tolerance range.

[0158] Figures 12(A) and (B) illustrate a method for determining whether the target distance range R information in the range gate image signal according to Embodiment 2 matches the distance measurement result of the subject related to the imaging plane phase difference range gate distance measurement mode.

[0159] Using Figure 12, we will explain how to determine whether the target distance range R information held by each pixel of each range gate image signal matches the distance value held by the pixel corresponding to the subject in the composite image 730 in the imaging plane phase difference range gate distance measurement mode.

[0160] Figure 12(A) shows the case in step S202 when the target distance range R information held by each pixel in each range gate image signal matches the distance value held by the pixel corresponding to the subject in the composite image 730 in the imaging plane phase difference range gate distance measurement mode.

[0161] Figure 12(A) is the composite image 730 shown in Figure 10(D). In Figure 12(A), each pixel in the composite image 730 holds distance information for the target distance range R0m to 30m and 30m to 60m, determined by range gate control, and the distance value of the subject calculated by the image plane phase difference range gate distance measurement mode.

[0162] For example, in Figure 12(A), the distance value of the subject in the image plane phase-difference range gate distance measurement mode is calculated as 5m for pedestrian 830 within the target distance range of 0m to 30m. Similarly, the distance value of vehicle 820 is calculated as 40m within the target distance range of 30m to 60m. Therefore, this matches the R information for each target distance range.

[0163] This suggests that pedestrian 830 is likely to be at 5m and vehicle 820 at 40m, and the parallax confidence representing the distance error for pedestrian 830 and vehicle 820 can be estimated to be high.

[0164] If the target distance range R information held by each pixel in each range gate image signal matches the distance value held by the pixel corresponding to the subject in the composite image 730 in the imaging plane phase difference range gate distance measurement mode, the process proceeds to step S203.

[0165] In step S203, the distance measuring unit 104 increases the parallax confidence in the composite image 730. The method for increasing the parallax confidence value in the composite image 730 is to add a constant X greater than 0 to the parallax confidence, or to multiply it by a variable Y greater than 1, but is not limited to these methods.

[0166] In this embodiment, the parallax reliability is updated based on the distance to the subject calculated from the first and second photoelectric conversion signals generated by range gate control.

[0167] Figure 12(B) shows a case in step S202 where the target distance range R information held by each pixel in each range gate image signal does not match the distance value held by the pixel corresponding to the subject in the composite image 730 in the imaging plane phase difference range gate distance measurement mode.

[0168] Figure 12(B) is a composite image 730 of Figure 10(D), similar to Figure 12(A). In Figure 12(B), each pixel in the composite image 730 holds distance information for the target distance ranges R0m to 30m and 30m to 60m obtained by range gate control, and distance information for the subject calculated by the image plane phase difference range gate distance measurement mode.

[0169] For example, in Figure 12(B), the distance measurement result of the subject using the image plane phase difference range gate distance measurement mode is 40m for pedestrian 830 for the target distance range of 0m to 30m. Also, the distance measurement value of vehicle 820 is calculated to be 20m for the target distance range of 30m to 60m, which does not match the R information for each target distance range.

[0170] Therefore, it is unlikely that pedestrian 830 is at 5m and vehicle 820 is at 40m, and it can be estimated that the parallax confidence representing the distance error for pedestrian 830 and vehicle 820 is low.

[0171] If the target distance range R information held by each pixel in each range gate image signal does not match the distance measurement held by the pixel corresponding to the subject in the composite image 730 in the imaging plane phase difference range gate distance measurement mode, the process proceeds to step S204.

[0172] In step S204, the distance measuring unit 104 reduces the parallax confidence in the composite image 730. The method for reducing the parallax confidence value in the composite image 730 is to subtract a constant X greater than 0 from the parallax confidence, or to multiply it by a variable Y less than 1. However, this is not limited to these methods.

[0173] In this way, it is determined whether the distance measurement value obtained by the range gate control of the photoelectric conversion element 102 for image plane phase difference distance measurement matches the target distance range R obtained by range gate control. Then, by changing the parallax reliability according to the result, the accuracy of the parallax reliability can be improved.

[0174] <Embodiment 3> Embodiment 3 of the present invention will now be described. Embodiment 1 described a configuration in which the photoelectric conversion element 102 is a CMOS sensor. Embodiment 3 describes a configuration in which the photoelectric conversion element 102 includes an avalanche photodiode (hereinafter referred to as APD).

[0175] The photoelectric conversion element 102, including the APD, digitally counts the number of incident photons and outputs the count value as a photoelectrically converted digital signal from the pixel. Unlike a CMOS sensor, the photoelectric conversion element 102, including the APD, can quickly switch the exposure time ON / OFF.

[0176] Furthermore, since the APD has no readout noise, the original signal does not degrade even if it is read multiple times after a single storage operation. The configuration and operation of the photoelectric conversion element 102 equipped with the APD will be explained using Figures 13 to 16.

[0177] Figure 13 shows an example of the configuration of a photoelectric conversion element 102 according to Embodiment 3. Below, an example of a photoelectric conversion element 102 with a so-called stacked structure, in which two substrates, a sensor substrate 911 and a circuit substrate 921, are stacked and electrically connected, will be described.

[0178] However, the photoelectric conversion element may have a so-called non-stacked structure in which the components included in the sensor substrate and the components included in the circuit substrate are arranged on a common semiconductor layer. The sensor substrate 911 includes a pixel region 912. The circuit substrate 921 includes a circuit region 922 for processing the signal detected in the pixel region 912.

[0179] Figure 14 is a schematic diagram showing an example configuration of a photoelectric conversion element 102 including an APD, and is a top view of the photoelectric conversion element 102 as seen from the direction of light incidence. The photoelectric conversion element 102 is composed of multiple 2x2 pixel groups 400 arranged on a matrix. The pixel group 910 has four infrared pixels IR that detect IR (infrared) light.

[0180] Furthermore, the arrangement of each pixel in the pixel group 910 is not limited to this and may be changed depending on the wavelength band of the light emitted from the light emitter 300. That is, the pixel group 910 may consist of, for example, an infrared pixel (IR) for detecting IR light, a red pixel for detecting the color red, a green pixel for detecting green light, and four blue pixels for detecting blue light.

[0181] Alternatively, the pixel group 910 may consist of four pixels: two green pixels for detecting green light, one red pixel for detecting red light, and one blue pixel for detecting blue light. In the above pixel arrangement, the two green pixels are arranged diagonally. Each pixel also includes a photoelectric conversion unit 920.

[0182] Figure 15 shows an example of the configuration of the circuit board 921. The circuit board 921 has a signal processing circuit 1060 for processing the photoelectrically converted charge in each image of Figure 14, a readout circuit 1120, a control pulse generation unit 1150, a horizontal scanning circuit 1100, a vertical signal line 1130, a vertical scanning circuit 1110, and an output circuit 1140.

[0183] The vertical scanning circuit 1110 receives control pulses supplied from the control pulse generation unit 1150 and sequentially supplies control pulses to multiple pixels arranged in the row direction, row by row. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 1110.

[0184] The signals output from the photoelectric conversion unit 920 of each pixel are processed by each signal processing circuit 1060. The signal processing circuit 1060 is equipped with a counter and memory, and digital values ​​are stored in the memory.

[0185] The horizontal scanning circuit 1100 inputs control pulses to the signal processing circuit 1060 to sequentially select each column in order to read the signal from the memory of each pixel in which the digital signal is held.

[0186] Signals are output to the vertical signal line 1130 from the signal processing circuit 1060 for pixels in rows selected by the vertical scanning circuit 1110. The signals output to the vertical signal line 1130 are then output to the outside of the photoelectric conversion element 102 via the readout circuit 1120 and the output circuit 1140. The signal processing circuit 1060 has multiple buffers built in that are connected to the vertical signal line 1130.

[0187] As shown in Figures 13 and 15, in a top view, multiple camera control units 105 are arranged in the area overlapping the pixel area 912. Then, in a top view, the vertical scanning circuit 1110, horizontal scanning circuit 1100, readout circuit 1120, output circuit 1140, and control pulse generation unit 1150 are arranged so as to overlap the peripheral area of ​​the sensor substrate 911 outside the pixel area 912.

[0188] Specifically, the sensor substrate 911 has a pixel region 912 and a non-pixel region arranged around the pixel region 912. A vertical scanning circuit 1110, a horizontal scanning circuit 1100, a readout circuit 1120, an output circuit 1140, and a control pulse generation unit 1150 are arranged in the region that overlaps with the non-pixel region when viewed from above.

[0189] Furthermore, the arrangement of the vertical signal line 1130, the read circuit 1120, and the output circuit 1140 is not limited to the example shown in Figure 15. For example, the vertical signal line 1130 may be arranged extending in the row direction, and the read circuit 1120 may be placed at the end of the vertical signal line 1130.

[0190] Furthermore, the signal processing circuit 1060 does not necessarily need to be provided in one place for each photoelectric conversion unit; a single signal processing unit may be shared among multiple photoelectric conversion units, and sequential signal processing may be performed.

[0191] Figure 16 is a diagram showing an example of the equivalent circuit of the signal processing circuit 1060 according to Embodiment 3, and shows an example of the equivalent circuit of the signal processing circuit 1060 corresponding to each pixel of the pixel group 910 in Figures 13 and 15.

[0192] Although Figure 16 describes one APD, it is assumed that the first photoelectric conversion unit and the second photoelectric conversion unit of Embodiment 1 are each composed of APDs. That is, the configuration shown in Figure 16 is arranged with two APDs for each pixel.

[0193] Specifically, the first photoelectric conversion unit and the second photoelectric conversion unit each include an APD that generates pulses corresponding to photons, a counter that counts the number of pulses, and a memory that stores the count value of the counter.

[0194] The APD2010 included in the photoelectric conversion unit 920 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD2010 is connected to a power line to which the drive voltage VL (first voltage) is supplied.

[0195] Furthermore, the other of the two nodes of the APD2010 is connected to a power line and a quench element 2020, to which a drive voltage VH (second voltage), which is higher than the drive voltage VL, is supplied.

[0196] In Figure 16, one node of the APD2010 is the anode, and the other node of the APD is the cathode. The anode and cathode of the APD2010 are supplied with a reverse bias voltage that causes the APD2010 to perform avalanche multiplication.

[0197] By supplying such a voltage, the charge generated by the incident light undergoes avalanche multiplication, resulting in the generation of an avalanche current.

[0198] Furthermore, when a reverse bias voltage is supplied, there are two modes: Geiger mode, in which the device operates when the voltage difference between the anode and cathode is greater than the breakdown voltage, and linear mode, in which the device operates when the voltage difference between the anode and cathode is near or below the breakdown voltage.

[0199] An APD that operates in Geiger mode is called a SPAD. In the case of a SPAD, for example, the drive voltage VL (first voltage) is -30V and the drive voltage VH (second voltage) is 1V. Note that SPADs are a type of APD.

[0200] The signal processing circuit 1060 includes a quench element 2020, a wave shaping unit 2100, a counter circuit 2110, and a memory circuit 2120. The quench element 2020 is connected to a power line to which a drive voltage VH is supplied and to one of the nodes, either the anode or the cathode, of the APD2010.

[0201] The quench element 2020 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD2010 and thereby suppressing avalanche multiplication (quench operation).

[0202] Furthermore, the quench element 2020 works to restore the voltage supplied to the APD2010 to the drive voltage VH by supplying current to compensate for the voltage drop caused by the quench operation (recharge operation).

[0203] Figure 16 shows an example in which the signal processing circuit 1060 has a wave shaping unit 2100, a counter circuit 2110, and a memory circuit 2120 in addition to the quench element 2020.

[0204] The wave shaping unit 2100 shapes the cathode voltage change of the APD2010 obtained during photon detection and outputs a pulse signal. For example, an inverter circuit is used as the wave shaping unit 2100.

[0205] Figure 16 shows an example where one inverter is used as the wave shaping unit 2100, but a circuit with multiple inverters connected in series may also be used, or other circuits that have a waveform shaping effect may be used.

[0206] The counter circuit 2110 counts the number of pulses output from the wave shaping unit 2100 and stores the count value. When the control pulse RES is supplied via the drive line 2130, the signal held in the counter circuit 2110 is reset. Here, the counter circuit 2110 generates a signal based on the difference between the count values ​​at the start and end of the storage period.

[0207] The memory circuit 2120 is supplied with a control pulse SEL from the vertical scanning circuit 1110 in Figure 15 via the drive line 2140 (not shown in Figure 15) in Figure 16, which switches the electrical connection between the memory circuit 2120 and the vertical signal line 1130.

[0208] The memory circuit 2120 functions as a memory that temporarily stores the counter's count value, and outputs the output signal from the pixel counter circuit 2110 to the vertical signal line 1130 via the memory circuit 2120.

[0209] Furthermore, switches such as transistors may be placed between the quench element 2020 and the APD2010, or between the photoelectric conversion unit 920 and the signal processing circuit 1060, to switch the electrical connections. Similarly, the supply of the drive voltage VH or drive voltage VL to the photoelectric conversion unit 920 may be electrically switched using switches such as transistors.

[0210] Figure 17 schematically shows the relationship between the operation of APD2010 and the output signal. The input side of the wave shaping unit 2100 is node A, and the output side is node B. Between time t0 and time t1, a potential difference of VH-VL is applied to APD2010.

[0211] When a photon is incident on APD2010 at time t1, avalanche multiplication occurs in APD2010, an avalanche multiplication current flows through the quench element 2020, and the voltage at nodeA drops.

[0212] As the voltage drop increases further and the potential difference applied to APD2010 decreases, the avalanche multiplication of APD2010 stops, as shown at time t2, and the voltage level of nodeA no longer drops below a certain value.

[0213] Subsequently, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from the drive voltage VL, and at time t3, nodeA settles back to its original potential level. At this time, any portion of the output waveform at nodeA that falls below a certain threshold is reshaped by the wave shaping unit 2100 and output as a pulse signal at nodeB. This concludes the explanation of the configuration and operation of the APD.

[0214] In this embodiment, by performing range gate control on the photoelectric conversion element 102 including the APD and the light emitter 300, a range gate image signal with less noise can be obtained compared to a CMOS sensor.

[0215] Therefore, in the imaging plane phase difference range gate distance measurement described in Embodiment 1, by measuring the distance of the subject based on a range gate image with less noise, it becomes possible to calculate a distance value with higher accuracy.

[0216] Furthermore, by range-gate control of the photoelectric conversion element 102 including the APD and the light emitter 300, the exposure time width of the photoelectric conversion element 102 can be shortened, thus reducing the target distance range R compared to a CMOS sensor. In other words, by reducing the target distance range R, the distance resolution can be increased and the distance accuracy can be improved.

[0217] In the embodiments described above, the mobile body 200 was explained using an example of a vehicle such as an automobile, but the mobile body can be any mobile object such as an aircraft, train, ship, drone, AGV, or robot.

[0218] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0219] Furthermore, the present invention includes, for example, a system that implements the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0220] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the distance measuring device, etc., via a network or various storage media.

[0221] Furthermore, the computer (or CPU or MPU, etc.) in the distance measuring device may read and execute the program. In that case, the program and the storage medium in which the program is stored constitute the present invention. The present invention includes the following combinations.

[0222] (Configuration 1) A distance measuring device comprising: a field of view determination unit that determines the field of view in the direction of travel of a moving object; a light emitter that emits pulsed light in the direction of travel; a photoelectric conversion element having a plurality of pixels arranged, each having a first photoelectric conversion unit and a second photoelectric conversion unit that generate a first photoelectric conversion signal and a second photoelectric conversion signal having a predetermined parallax; a control unit that range-gates the emission time of the light emitter and the exposure time and exposure timing of the photoelectric conversion element; and a distance measuring unit that calculates the distance to a subject based on the first photoelectric conversion signal and the second photoelectric conversion signal, wherein the control unit switches, based on the determination result of the field of view determination unit, whether to execute a process to calculate the distance to a subject based on the first photoelectric conversion signal and the second photoelectric conversion signal generated without range-gate control, or to execute a process to calculate the distance to a subject based on the first photoelectric conversion signal and the second photoelectric conversion signal generated by range-gate control.

[0223] (Configuration 2) The distance measuring device according to Configuration 1, characterized in that the control unit updates the parallax reliability based on the distance to the subject calculated based on the first photoelectric conversion signal and the second photoelectric conversion signal generated by the range gate control.

[0224] (Configuration 3) The distance measuring device according to Configuration 1 or 2, characterized in that the control unit adds together a plurality of first photoelectric conversion signals and a plurality of second photoelectric conversion signals generated by performing the range gate control a plurality of times.

[0225] (Configuration 4) The distance measuring device according to any one of Configurations 1 to 3, characterized in that the first photoelectric conversion unit and the second photoelectric conversion unit each comprise an APD that generates pulses corresponding to photons, a counter that counts the number of pulses, and a memory that stores the count value of the counter.

[0226] (Configuration 5) The distance measuring device according to any one of Configurations 1 to 4, characterized in that the light emitted by the light emitter is visible light or invisible light.

[0227] (Method) A distance measuring method using a distance measuring device comprising: a field of view determination unit for determining the field of view in the direction of travel of a moving object; a light emitter for emitting pulsed light in the direction of travel; a photoelectric conversion element having a plurality of pixels arranged, each having a first photoelectric conversion unit and a second photoelectric conversion unit for generating a first photoelectric conversion signal and a second photoelectric conversion signal having a predetermined parallax; a control unit for range gate control of the light emission time of the light emitter and the exposure time and exposure timing of the photoelectric conversion element; and a distance measuring unit for calculating the distance to a subject based on the first photoelectric conversion signal and the second photoelectric conversion signal, wherein the control unit switches between executing a process to calculate the distance to a subject based on the first photoelectric conversion signal and the second photoelectric conversion signal generated without range gate control, or executing a process to calculate the distance to a subject based on the first photoelectric conversion signal and the second photoelectric conversion signal generated by range gate control, based on the determination result of the field of view determination unit.

[0228] (Program) A computer program for controlling each part of the distance measuring device described in any one of configurations 1 to 5 by computer. [Explanation of Symbols]

[0229] 100: Camera 200: Mobile 300: Light emitter 101: Imaging Optical System 102: Photoelectric conversion element 103: Image Processing Unit 104: Distance measurement section 108: Severe Weather Judgment Department

Claims

1. A visibility condition determination unit that determines the visibility conditions in the direction of travel of a moving object, A light emitter that emits pulsed light in the direction of travel, A photoelectric conversion element comprising a plurality of pixels, each having a first photoelectric conversion unit that generates a first photoelectric conversion signal and a second photoelectric conversion signal having a predetermined parallax, and a second photoelectric conversion unit. A control unit that range-gates the emission time of the light emitter and the exposure time and exposure timing of the photoelectric conversion element, The system includes a distance measuring unit that calculates the distance to the subject based on the first photoelectric conversion signal and the second photoelectric conversion signal, The control unit switches between performing a process to calculate the distance to the subject based on the determination result of the field of view determination unit, using the first and second photoelectric conversion signals generated without range gate control, and performing a process to calculate the distance to the subject based on the first and second photoelectric conversion signals generated by range gate control. A distance measuring device characterized by the following features.

2. The distance measuring device according to claim 1, characterized in that the control unit updates the parallax reliability based on the distance to the subject calculated based on the first photoelectric conversion signal and the second photoelectric conversion signal generated by the range gate control.

3. The distance measuring device according to claim 1, characterized in that the control unit adds together a plurality of first photoelectric conversion signals and a plurality of second photoelectric conversion signals generated by performing the range gate control a plurality of times.

4. The first photoelectric conversion unit and the second photoelectric conversion unit each include an APD that generates pulses corresponding to photons, A counter for counting the number of pulses, The distance measuring device according to claim 1, further comprising a memory for storing the count value of the counter.

5. The light emitted by the light-emitting device is either visible light or invisible light. The distance measuring device according to feature 1.

6. A visibility condition determination unit that determines the visibility conditions in the direction of travel of a moving object, A light emitter that emits pulsed light in the direction of travel, A photoelectric conversion element comprising a plurality of pixels, each having a first photoelectric conversion unit that generates a first photoelectric conversion signal and a second photoelectric conversion signal having a predetermined parallax, and a second photoelectric conversion unit. A control unit that range-gates the emission time of the light emitter and the exposure time and exposure timing of the photoelectric conversion element, A distance measuring method using a distance measuring device comprising a distance measuring unit that calculates the distance to an object based on the first photoelectric conversion signal and the second photoelectric conversion signal, The control unit switches between performing a process to calculate the distance to the subject based on the determination result of the field of view determination unit, using the first and second photoelectric conversion signals generated without range gate control, and performing a process to calculate the distance to the subject based on the first and second photoelectric conversion signals generated by range gate control. A distance measurement method characterized by the following features.

7. A computer program for controlling each part of the distance measuring device described in any one of claims 1 to 5 by computer.

Images