Sensor, vehicle and surrounding environment sensing method

Abstract

A sensor 100 is provided with a gating camera 20 and an arithmetic processing device 40. The gating camera 20 divides the depth direction into a plurality of ranges, captures images while changing the time difference between light emission and light exposure for each of the ranges, and generates a plurality of slice images corresponding to the respective plurality of ranges. The arithmetic processing device 40 infers the surrounding environment on the basis of pixel values of slice images obtained for given ranges.

Description

[Technical field]

[0001] The present disclosure relates to sensors. [Background technology]

[0002] For autonomous driving and automatic control of headlamp distribution, an object identification system is used to sense the position and type of objects around the vehicle. The object identification system includes a sensor and a processing unit that analyzes the sensor output. The sensor is selected from cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter wave radar, ultrasonic sonar, active sensors, etc., taking into consideration the application, required accuracy, and cost.

[0003] The results of sensing by sensors are affected by the surrounding environment, including weather such as sunny, rainy, and foggy. Therefore, if the surrounding environment can be estimated, the accuracy of sensing can be improved. Information about the surrounding environment is also useful for autonomous driving control and lamp light distribution control. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-257983 A [Patent Document 2] International Publication WO2017 / 110413A1 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventionally, in order to detect the weather, it was necessary to use a dedicated sensor such as a raindrop sensor, which resulted in increased costs.

[0006] The present disclosure has been made in light of such a situation, and one exemplary purpose of an aspect thereof is to provide a sensor or a sensing method capable of estimating an ambient environment at low cost. [Means for solving the problem]

[0007] An aspect of the present disclosure relates to a method / apparatus for sensing an ambient environment. The method / apparatus uses an active sensor including an illumination device and an image sensor, and captures an image by setting the timing of light emission of the illumination device and the timing of exposure of the image sensor so that an object within a predetermined range in the depth direction is captured. The method / apparatus estimates the ambient environment based on pixel values ​​of slice images obtained within the predetermined range. Effect of the Invention

[0008] According to the present disclosure, the surrounding environment can be sensed. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram of a sensor according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating the operation of a gating camera. [Diagram 3] 3(a) and (b) are diagrams illustrating images obtained by the gating camera. [Figure 4] 4(a) to (d) are diagrams showing slice images obtained in clear weather, light rain (precipitation rate of 30 mm / h), heavy rain (precipitation rate of 80 mm / h), and foggy conditions. [Diagram 5] FIG. 5(a) is a diagram showing the relationship between the emission intensity I and the pixel value p, and FIG. 5(b) is a table showing the relationship between the surrounding environment and the transmittance T. [Figure 6] 6(a) and (b) are diagrams showing an automobile equipped with a gating camera. [Figure 7] 1 is a block diagram showing a vehicle lamp equipped with an object detection system; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (Overview of the embodiment) A summary of some exemplary embodiments of the present disclosure will be described. This summary is intended to provide a simplified summary of some concepts of one or more embodiments in order to provide a basic understanding of the embodiments as a prelude to the detailed description that follows, and is not intended to limit the scope of the invention or disclosure. Furthermore, this summary is not intended to be a comprehensive overview of all possible embodiments, nor is it intended to limit essential components of the embodiments. For convenience, the term "one embodiment" may be used to refer to one embodiment (example or variant) or multiple embodiments (examples or variants) disclosed in this specification.

[0011] A sensor according to an embodiment includes a gating camera and a processor. The gating camera is configured to divide the depth direction into a plurality of ranges (also called slices), capture images for each range by changing the time difference between light emission and exposure, and generate a plurality of slice images corresponding to the plurality of ranges. The processor estimates the surrounding environment based on pixel values ​​of the slice images obtained for a predetermined range.

[0012] The amount of light emitted by the gating camera is known, and the distance to a given range is also known. When the reflectance of an object is constant, the pixel value of the object changes depending on the transmittance between the gating camera and the object, in other words, the surrounding environment. Therefore, with this sensor, the light transmittance and therefore the surrounding environment can be estimated based on the pixel values ​​of the slice images.

[0013] In one embodiment, the arithmetic processing device may set the imaging parameters of the gating camera for each range during normal imaging based on the estimated surrounding environment. The imaging parameters (emission intensity of the illumination device, emission time, exposure time of the image sensor, and time difference between them) are set for each range in the gating camera. By dynamically controlling the imaging parameters for each range according to the surrounding environment, the image quality can be improved.

[0014] In one embodiment, the arithmetic processing device may estimate the surrounding environment based on pixel values ​​of specific objects having a predetermined reflectance included in the slice images. The specific objects may include at least one of a sign, a road surface, a white line, a guardrail, and a reflector, each of which has a known reflectance.

[0015] In one embodiment, the processor may estimate the surrounding environment based on slice images obtained for at least two ranges, thereby improving the accuracy of the estimation.

[0016] A sensor according to an embodiment includes an active sensor including an illumination device capable of controlling the timing of light emission and an image sensor capable of controlling the timing of exposure, and a processor that controls the illumination device and the image sensor so that an object included in a predetermined range in the depth direction is captured, and estimates the surrounding environment based on pixel values ​​of a slice image obtained for the predetermined range. The surrounding environment can also be estimated using an active sensor other than a gating camera.

[0017] (Embodiment) Hereinafter, the embodiments will be described with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing will be given the same reference numerals, and duplicated descriptions will be omitted as appropriate. In addition, the embodiments are illustrative and do not limit the disclosure or invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the disclosure or invention.

[0018] 1 is a block diagram of a sensor 100 according to an embodiment of the present invention. The sensor 100 is mounted on a vehicle such as an automobile or motorcycle, and captures an image of the situation ahead of the vehicle.

[0019] The sensor 100 mainly includes a gating camera 20 and a processor 40. The gating camera 20 is an active sensor including a lighting device 22, an image sensor 24, and a controller 26. The gating camera 20 captures images in a plurality of ranges RNG1 to RNG2 (N≧2) in the depth direction. N Adjacent ranges may overlap in the depth direction at their boundaries.

[0020] The lighting device (floodlight) 22 irradiates the probe light L1 ahead of the vehicle in synchronization with a light emission timing signal S1 provided by the controller 26. The probe light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength. The lighting device 22 may be, for example, a laser diode (LD) or an LED.

[0021] The image sensor 24 is capable of performing exposure control in synchronization with an exposure timing signal S2 provided from the controller 26, and is configured to be capable of generating a slice image IMG. The image sensor 24 has sensitivity to the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ.

[0022] The controller 26 changes the light emission timing signal S1 and the exposure timing signal S2 for each range RNG to change the time difference between the light emission by the lighting device 22 and the exposure of the image sensor 24. The light emission timing signal S1 specifies the timing of the start of light emission and the light emission time. The exposure timing signal S2 specifies the timing of the start of exposure (the time difference with light emission) and the exposure time. The controller 26 may also switch the light emission intensity of the lighting device 22 for each range RNG.

[0023] The gating camera 20 has a plurality of ranges RNG1 to RNG N Multiple slice images IMG1 to IMG N The i-th slice image IMG i has a corresponding range RNG iOnly objects contained within will be captured.

[0024] FIG. 2 is a diagram for explaining the operation of the gating camera 20. In FIG. 2, the i-th range RNG i The illumination device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. The range RNG from the gating camera 20 is i The distance to the boundary in front of MINi、 Range RNG i The distance to the inner boundary of MAXi Let us assume that.

[0025] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi =2×d MINi / c where c is the speed of light.

[0026] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, T MAXi =2×d MAXi / c It is.

[0027] Range RNG i When it is desired to photograph only the object OBJ included in the time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.

[0028] i-th range RNG iWhen capturing an image, multiple exposures may be performed. In this case, the controller 26 may repeat the above-described exposure operation multiple times at a predetermined period τ2.

[0029] In this example, the method of determining the start time t2 and end time t3 of the exposure is not limited to that shown in FIG. 2. For example, the center of the light emission time t0 to t1 may be set as the reference timing t REF Then, t2=t REF +T MINi , t3=t REF +T MAXi It may be defined as follows.

[0030] In this embodiment, in order to prevent the exposure (brightness value of the object image in the slice image) from varying for each range, the gating camera 20 optimizes the shutter speed (exposure time), number of exposures, sensitivity, irradiation intensity of the probe light, etc. (imaging parameters) for each range.

[0031] 3(a) and (b) are diagrams for explaining images obtained by the gating camera 20. In the example of FIG. 3(a), an object (pedestrian) OBJ2 exists in range RNG2, and an object (vehicle) OBJ3 exists in range RNG3. FIG. 3(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 3(a). When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, so no object image is captured in the slice image IMG1.

[0032] When the slice image IMG2 is captured, the image sensor is exposed only by the light reflected from the range RNG2, so that only the object image OBJ2 appears in the slice image IMG2. Similarly, when the slice image IMG3 is captured, the image sensor is exposed only by the light reflected from the range RNG3, so that only the object image OBJ3 appears in the slice image IMG3. In this way, the gating camera 20 can capture the object separately for each range.

[0033] Returning to FIG. 1, the calculation processing device 40 calculates the predetermined range RNG obtained by the gating camera 20. X Slice image IMG corresponding to X The surrounding environment is estimated based on the above. The arithmetic processing device 40 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), microcomputer, or GPU (Graphics Processing Unit) and a software program executed by the processor (hardware). The arithmetic processing device 40 may be a combination of multiple processors. Alternatively, the arithmetic processing device 40 may be configured only from hardware. The arithmetic processing device 40 and the controller 26 of the gating camera 20 may be implemented in the same hardware or in separate hardware.

[0034] The calculation processing unit 40 determines a predetermined range RNG X Slice image IMG obtained for X The surrounding environment is estimated based on at least one pixel value included in the pixel value, and environmental information ENV indicating the surrounding environment is generated.

[0035] For example, the arithmetic processing device 40 may receive a slice image IMG X From the above, a predetermined object with a known reflectance may be extracted, and the surrounding environment may be estimated based on the pixel value. In the case of an in-vehicle sensor, the predetermined object may include at least one of a sign, a road surface, a white line, a guardrail, and a reflector (delineator).

[0036] The transmittance T between the object and the gating camera 20 depends on the weather (ambient environment). When the reflectance of the object is R and the emission intensity of the lighting device 22 is I, the pixel value p of the object included in the slice image is expressed as follows: p = A × I × R × T where A is a coefficient. Since A, I, and R can be regarded as constants, the pixel value p is proportional to the transmittance T. Here, the transmittance T depends on the distance between the object and the gating camera 20, and since the object is included in a predetermined range, this distance is known. Therefore, the pixel value p is proportional to the transmittance per unit distance. Here, since the transmittance per unit distance depends on the surrounding environment, the surrounding environment can be estimated based on the pixel value p.

[0037] Figure 4(a)-(d) show slice images obtained under clear skies, light rain (rainfall rate 30 mm / h), heavy rain (rainfall rate 80 mm / h), and foggy conditions. Each image shows a person and a reflector.

[0038] Fig. 5(a) is a diagram showing the relationship between the emission intensity I and the pixel value p, and Fig. 5(b) is a table showing the relationship between the surrounding environment and the transmittance T. Figs. 5(a) and (b) are created based on the pixel value of the reflector.

[0039] According to this embodiment, the surrounding environment can be estimated based on the pixel value p of a predetermined object.

[0040] The estimated surrounding environment can be used to set the shooting parameters of the gating camera 20 itself. The arithmetic processing device 40 may set the shooting parameters of the gating camera 20 for each range RNG during normal shooting based on the estimated surrounding environment. For example, if the depth of the range is widened during rainfall, the image becomes difficult to see due to the raindrops included in the range. Therefore, the shooting parameters may be set so that the depth of each range is shortened during rainfall. Conversely, the depth of each range may be lengthened during fine weather to shoot many objects at once. Also, when fog is estimated, the emission intensity may be increased or the sensitivity of the image sensor may be increased. The arithmetic processing device 40 may have a table that specifies the relationship between the surrounding environment and the shooting parameters. The image quality can be improved by dynamically controlling the shooting parameters of each range according to the surrounding environment.

[0041] The arithmetic processing device 40 has at least two ranges RNG X ,RNG Y Slice image IMG obtained for X ,IMG Y The surrounding environment may be estimated based on the above, thereby improving the estimation accuracy.

[0042] (Application) 6(a) and (b) are diagrams showing an automobile 300 equipped with a gating camera 20. See FIG. 6(a). The automobile 300 is equipped with headlamps (lamp fixtures) 302L, 302R. An image sensor 24 is built into at least one of the left and right headlamps 302L, 302R. The image sensor 24 may be provided on the outside of the headlamps 302L, 302R. The position of the lighting device 22 is not particularly limited, and may be provided on the front bumper (i) or the front grille (ii), or may be attached to the back side of the rearview mirror inside the front window (iii). The position of the controller 26 is also not particularly limited, and may be provided in the engine room, in the passenger compartment, or built into the headlamps.

[0043] Refer to Fig. 6(b). One lighting device 22 includes multiple (for example, two) light sources 22A and 22B. The multiple light sources 22A and 22B emit light at the same timing, and the emitted light forms one probe light. The multiple light sources 22A and 22B are built into the left and right headlamps 302L and 302R.

[0044] At least one of the left and right headlamps 302L, 302R includes an image sensor 24. The image sensor 24 may be provided on the outside of the headlamps 302L, 302R.

[0045] 7 is a block diagram showing a vehicle lamp 200 equipped with an object detection system 210. The vehicle lamp 200 constitutes a lighting system 310 together with a vehicle-side ECU 304. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. The vehicle lamp 200 is further provided with an object detection system 210.

[0046] The arithmetic processing device 40 performs the sensing of the surrounding environment as described above. Furthermore, the arithmetic processing device 40 performs object recognition included in the slice image by image processing. Information on the object OBJ detected by the arithmetic processing device 40 may be used for light distribution control of the vehicle lamp 200. Specifically, the lamp side ECU 208 generates an appropriate light distribution pattern based on the information on the type of object OBJ and its position generated by the arithmetic processing device 40. The lighting circuit 204 and the optical system 206 operate to obtain the light distribution pattern generated by the lamp side ECU 208.

[0047] Furthermore, information regarding the object OBJ detected by the arithmetic processing device 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU may perform automatic driving based on this information.

[0048] The embodiment has been described above. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of this disclosure or the present invention. Such modifications will be described below.

[0049] (Variation 1) In the embodiment, the surrounding environment sensed by the arithmetic processing device 40 is used to control the gating camera 20, but this is not the only possible embodiment. The environmental information may be used for autonomous driving or for controlling the light distribution of headlamps.

[0050] (Variation 2) In the embodiment, the sensing of the surrounding environment using a gating camera has been described, but this is not limited thereto. In other words, the set of the illumination device and the image sensor used for sensing does not necessarily constitute a gating camera, and may be that of a normal active sensor.

[0051] In the second modification, the active sensor includes an illumination device with controllable light emission timing and an image sensor with controllable exposure timing. The arithmetic processing device controls the illumination device and the image sensor so that an object included in a predetermined range in the depth direction is captured when sensing the surrounding environment. The arithmetic processing device can estimate the surrounding environment based on pixel values ​​of the object included in the slice image obtained for the predetermined range.

[0052] Furthermore, the lighting device and image sensor used for sensing the surrounding environment do not even need to constitute an active sensor, and a lighting device and image sensor that are originally provided unrelatedly may be diverted for sensing the surrounding environment. For example, a car may be equipped with an image sensor for driving assistance. In addition, a car may be equipped with LIDAR or infrared lighting. In such cases, they may be combined to sense the surrounding environment.

[0053] (Variation 3) In the embodiment, the pixel values ​​of a specific object having a predetermined reflectance included in the slice image obtained for a predetermined range are used, but this is not limited to the above. When driving for a certain period of time, several slice images are obtained for the same range. The objects captured in these slice images change according to the driving scene, and the types and positions of objects included in the slice images are random, and statistically, the average value of the pixel values ​​converges to a value according to the light transmittance. Therefore, the surrounding environment may be estimated by capturing multiple slice images in different driving scenes for a predetermined range using the gating camera 20 and statistically processing the pixel values.

[0054] (Variation 4) A multi-tap type sensor may be used as the image sensor 24. A multi-tap type sensor has multiple floating diffusions (charge storage regions) FD for each pixel, and the exposure timing can be controlled individually for each floating diffusion FD. When a multi-tap type sensor is used, multiple ranges can be sensed simultaneously by shifting the exposure timing of multiple floating diffusions FD for one light emission.

[0055] The embodiment merely illustrates one aspect of the principles and applications of the present invention, and many modifications and changes in arrangement are permitted to the embodiment without departing from the spirit of the present invention defined in the claims. [Industrial Applicability]

[0056] The present disclosure can be used for sensors. [Explanation of symbols]

[0057] S1 Light emission timing signal S2 Exposure timing signal 100 Sensors 20 Gating Camera 22 Lighting equipment 24 Image Sensor 26 Controller 40 Processing unit

Claims

1. a gating camera that acquires a plurality of slice images in which only objects included in each range are captured by varying the timing of irradiation of the probe light by the illumination device and the timing of exposure by the image sensor for each of a plurality of ranges separated in the depth direction; a processor that outputs environmental information indicating a surrounding environment based on a value corresponding to a transmittance of the probe light per unit distance acquired from at least one of the plurality of slice images; A sensor comprising:

2. The sensor described in Claim 1, characterized in that the shooting parameters of the gating camera are set based on the environmental information.

3. 3. The sensor according to claim 1, wherein the arithmetic processing unit estimates the surrounding environment based on pixel values ​​of an object having a known reflectance that appears in at least one of the plurality of slice images.

4. 4. The sensor according to claim 3, wherein the object includes at least one of a sign, a road surface, a white line, a guardrail, and a reflector.

5. A motor vehicle comprising a sensor according to any one of claims 1 to 4.

6. A step of acquiring a plurality of slice images in which only objects included in each range are captured by varying the timing of irradiation of a probe light by an illumination device and the timing of exposure by an image sensor for each of a plurality of ranges separated in a depth direction; outputting environmental information indicating a surrounding environment based on a value corresponding to a transmittance of the probe light per unit distance acquired from at least one of the plurality of slice images; A method for sensing an ambient environment, comprising:

Images