Gating camera, sensing system, and vehicle lamp

Abstract

A gating camera 20 divides, in the depth direction, a field of view into a plurality of ranges RNG1 to RNGN, and generates a plurality of slice images IMGs1 to IMGsN corresponding to the plurality of ranges RNG1 to RNGN. An illumination device 22 has a light emission timing that is controllable, and emits probe light L1 synchronously with a light emission timing signal S1. An image sensor 24 can be exposed synchronously with an exposure timing signal S2. A camera controller 26 controls, for each of the ranges, the timing at which the illumination device 22 emits light and the timing at which the image sensor 24 captures images, and controls the effective imaging range of the image sensor 24 in accordance with vehicle speed.

Description

Gating cameras, sensing systems, vehicle lighting fixtures

[0001] The present invention relates to a gating camera.

[0002] For autonomous driving and automatic control of headlamp light distribution, an object identification system is used to sense the position and type of objects around the vehicle. The object identification system includes a sensor and a processing unit that analyzes the sensor output. The sensor is selected from among cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter-wave radar, ultrasonic sonar, etc., taking into account the application, required accuracy, and cost.

[0003] A typical monocular camera cannot obtain depth information, making it difficult to separate overlapping objects located at different distances.

[0004] A time-of-flight (TOF) camera is known as a camera that can obtain depth information. A TOF camera projects infrared light using a light-emitting device, measures the time of flight of the reflected light until it returns to an image sensor, and obtains an image by converting the time of flight into distance information.

[0005] Active sensors (hereinafter referred to as gating cameras in this specification) have been proposed as an alternative to TOF cameras (see Patent Documents 1 and 2). A gating camera divides the imaging range into multiple ranges, and captures images multiple times by changing the exposure timing and exposure time for each range. This results in slice images for each target range, and each slice image contains only the objects included in the corresponding range.

[0006] JP 2009-257981 A International Publication WO2017 / 110417A1

[0007] 1. The present disclosure has been made in light of the above circumstances, and one exemplary purpose of an aspect thereof is to provide a gating camera that can reduce the amount of data.

[0008] 2. A ToF camera uses a range image as its output format. A range image is an image in which pixel values ​​are obtained by converting time-of-flight information into distance information. In a range image from a ToF camera, objects at the same distance have the same color regardless of their reflectance. Therefore, when a sign is photographed, for example, information such as the letters and figures written on the sign is lost in the range image from the ToF camera.

[0009] The present disclosure has been made in this context, and one exemplary purpose of an aspect thereof is to provide a sensor capable of generating range image data that contains more information.

[0010] 3. The present disclosure has been made in light of the above circumstances, and one exemplary purpose of an embodiment thereof is to provide a gating camera that can reduce power consumption.

[0011] 1. One aspect of the present disclosure relates to a gating camera that divides a field of view into multiple ranges in the depth direction and generates multiple slice images corresponding to the multiple ranges. The gating camera includes an illumination device that controls the light emission timing and irradiates probe light, an image sensor that controls the exposure timing, and a controller that controls the light emission timing of the illumination device and the image capture timing of the image sensor for each range and controls the effective imaging range of the image sensor in accordance with vehicle speed.

[0012] 2. A gating camera according to an embodiment of the present disclosure includes an illumination device that can control the light emission timing and irradiates probe light, an image sensor that can control the exposure timing, a controller that divides the field of view into N ranges (N≧2) in the depth direction and controls the light emission timing of the illumination device and the imaging timing of the image sensor for each range, and an image processing device that combines N slice images output by the image sensor corresponding to the N ranges to generate a composite image. １ ~C Ｎ Each pixel in the composite image is assigned one of N colors C １ ~C Ｎ are the pixel values ​​v of the corresponding pixels in the N slice images, respectively. １~v Ｎ The pixel values ​​are blended with a coefficient based on the

[0013] Another aspect of the present disclosure relates to an image processing device used with a gating camera that divides a field of view into N ranges in the depth direction and outputs N slice images corresponding to the N ranges. The image processing device is capable of synthesizing the N slice images to generate a synthesized image. １ ~C Ｎ Each pixel in the composite image is assigned one of N colors C １ ~C Ｎ are the pixel values ​​v of the corresponding pixels in the N slice images, respectively. １ ~v Ｎ The pixel values ​​are blended with a coefficient based on the

[0014] An aspect of the present disclosure relates to a gating camera that divides a field of view into multiple ranges in the depth direction and generates multiple slice images corresponding to the multiple ranges. The gating camera includes an illumination device that irradiates probe light, the illumination device including multiple light-emitting elements, and that can control the light distribution of the probe light by selecting whether to use the multiple light-emitting elements in response to a light distribution control signal, an image sensor that can control exposure timing, and a controller that (i) controls the light emission timing of the illumination device and the imaging timing of the image sensor for each range, and (ii) determines an area to be irradiated with the probe light in response to a driving scene, and generates a light distribution control signal.

[0015] Another aspect of the present disclosure is an illumination device that can be used in a gating camera that divides a field of view into multiple ranges in the depth direction and generates multiple slice images corresponding to the multiple ranges. The illumination device includes multiple light-emitting elements arranged in an array, an optical system that projects light emitted from each of the multiple light-emitting elements onto a corresponding one of multiple regions on a virtual vertical screen, and a lighting circuit that can selectively drive the multiple light-emitting elements and control the drive timing of the multiple light-emitting elements in accordance with a light-emission timing signal.

[0016] According to one aspect of the present disclosure, it is possible to reduce the amount of data. According to another aspect, it is possible to generate a distance image that includes more information than a distance image obtained by a ToF sensor. According to yet another aspect of the present disclosure, it is possible to reduce the power consumption of a gating camera.

[0017] 1 is a block diagram of a gating camera according to embodiment 1. It is a diagram explaining the basic operation of the gating camera. 3(a) and (b) are diagrams explaining slice images obtained by the gating camera. 4(a) to (c) are diagrams explaining control of the shooting range (angle of view) according to the vehicle speed. It is a diagram showing a vehicle traveling around a curve. 6(a) and (b) are diagrams showing a vehicle traveling on a straight road. It is a block diagram of a gating camera according to embodiment 2. It is a diagram explaining the basic operation of the gating camera. 9(a) and (b) are diagrams explaining slice images obtained by the gating camera. 10(a) to (d) are diagrams explaining a composite image and a conventional distance image. It is a diagram explaining multiple colors assigned to multiple ranges in the RGB color system. It is a diagram explaining the composition of the composite image IMGc. It is a diagram explaining the composition process when the same object is captured across multiple slice images. It is a diagram showing an example of the correspondence between multiple ranges and colors in embodiment 2.1. It is a diagram explaining multiple colors assigned to multiple ranges in the Yxy color system. 21(a) and (b) are diagrams illustrating slice images obtained by a gating camera. FIGS. 22(a) to (c) are diagrams illustrating an example of control of an illumination area according to a driving scene. FIGS. 23(a) to (c) are diagrams illustrating another example of control of an illumination area according to a driving scene. FIGS. 23(a) to (c) are diagrams illustrating light distribution control by a camera controller. FIGS. 23(a) to (c) are diagrams illustrating changes in the horizontal field of view while driving around a curve. FIGS. 21(a) and (b) are diagrams illustrating an example of a correspondence relationship between a plurality of ranges and colors in Example 2.2. FIGS. 21(a) and (b) are diagrams illustrating an example of a correspondence relationship between a plurality of ranges and colors in Example 2.3. FIGS. 21(a) and (b) are diagrams illustrating an example of a correspondence relationship between a plurality of ranges and colors in Example 2.3. FIGS. 21(a) and (b) are diagrams illustrating an example of a correspondence relationship between a plurality of ranges and colors in Example 2.3.

[0018] (Summary of the Embodiments) A summary of some exemplary embodiments of the present disclosure will be provided. This summary is intended to provide a simplified overview of some concepts of one or more embodiments in order to provide a basic understanding of the embodiments as a prelude to the more detailed description that follows, and is not intended to limit the scope of the invention or disclosure. This summary is not an exhaustive overview of all possible embodiments, and is not intended to identify key elements of all embodiments or to delineate the scope of some or all aspects. For convenience, the term "one embodiment" may be used to refer to one embodiment (example or variant) or multiple embodiments (examples or variants) disclosed herein.

[0019] 1. A gating camera according to one embodiment divides a field of view into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device whose light emission timing is controllable and which irradiates probe light, an image sensor whose exposure timing is controllable, and a controller that controls the light emission timing of the illumination device and the imaging timing of the image sensor for each range and also controls the effective imaging range of the image sensor in accordance with the vehicle speed.

[0020] In scenes where the vehicle is traveling at high speed, the importance of information about objects on the side of the road decreases. Therefore, when the vehicle is traveling at high speed, the effective imaging range of the image sensor can be narrowed to reduce the number of pixels and the data volume of the slice image.

[0021] Furthermore, narrowing the effective imaging range of the image sensor reduces the power consumption of the readout circuit, and shortens the time required to transmit slice images, thereby enabling an increase in frame rate.

[0022] In one embodiment, the controller may acquire a stopping distance according to the vehicle speed and determine the effective imaging range of the image sensor based on the stopping distance and the minimum radius of any curve on the road the vehicle is currently traveling in. This makes it possible to prevent a collision with an object even if braking is initiated after an object is detected by the gating camera.

[0023] In one embodiment, when the stopping distance is L and the minimum radius is R, the effective imaging range of the image sensor may be controlled so that the half angle of view (unit: rad) of the gating camera is included in a range with a lower limit of L / (2R).

[0024] In one embodiment, the controller may acquire a stopping distance according to the vehicle speed. When the stopping distance is L while traveling on a straight road, the effective imaging range of the image sensor may be controlled so that the half angle of view (unit: rad) of the gating camera is included in a range whose lower limit is arcsin(x / L), where x is a variable or constant.

[0025] In one embodiment, the stopping distance may be calculated using road surface conditions as a parameter.

[0026] In one embodiment, the control of the effective imaging range of the image sensor based on the vehicle speed may be effective on expressways. On ordinary roads, the effective imaging range can be widened regardless of the vehicle speed, thereby enabling reliable detection of pedestrians and other obstacles on the side of the road.

[0027] 2. A gating camera according to one embodiment includes an illumination device that can control the light emission timing and irradiates probe light, an image sensor that can control the exposure timing, a controller that divides a field of view into N ranges (N≧2) in the depth direction and controls the light emission timing of the illumination device and the imaging timing of the image sensor for each range, and an image processing device that combines N slice images output by the image sensor corresponding to the N ranges to generate a composite image. １ ~C Ｎ Each pixel in the composite image is assigned one of N colors C １ ~C Ｎ are the pixel values ​​v of the corresponding pixels in the N slice images, respectively. １ ~v Ｎ The pixel values ​​are weighted and added with coefficients based on the above.

[0028] With this configuration, in the composite image, the range (i.e., distance) of an object is represented by a color system, and the reflectance of the object is represented as the brightness of a color of the same system, making it possible to distinguish objects or areas with different reflectances that exist in the same range.

[0029] In one embodiment, in the RGB color system, the i-th color (R ｉ , G ｉ , B ｉ ) and the (i+1)th color (R ｉ＋１ , G ｉ＋１ , B ｉ＋１ ) may have a relationship in which one element of R, G, and B has a different value and the remaining two elements have the same value. The pixel value (R, G, B) of each pixel of the composite image is expressed as follows: (R, G, B) = Σ ｊ＝１：Ｎ [v ｊ ・(R ｊ , G ｊ , B ｊ ) ] may be expressed as Σ ｊ＝１：Ｎ represents the sum when j is changed from 1 to N. This allows the composite image to be calculated with extremely simple calculations.

[0030] In one embodiment, the i-th color is expressed as (x ｉ , y ｉ The pixel value of each pixel of the composite image in the Yxy color system may be defined as follows in the Yxy color system: (Y, x, y)=(Σ ｊ＝１：Ｎ (v ｊ ), Σ ｊ＝１：Ｎ (v ｊ ・x ｊ ) / Σ ｊ＝１：Ｎ v ｊ , Σ ｊ＝１：Ｎ (v ｊ ・y ｊ ) / Σ ｊ＝１：Ｎ v ｊ ) may also be expressed as:

[0031] In one embodiment, the N colors may be defined in the HSV color system so that the hue H is different. The pixel value (H, S, V) of each pixel in the composite image is S ０ As a constant, (H, S, V) = (Σ ｊ＝１：Ｎ (v ｊ ・Hｊ ) / Σ ｊ＝１：Ｎ v ｊ , S ０ , Σ ｊ＝１：Ｎ (v ｊ )) may also be expressed as

[0032] In one embodiment, the image sensor may be a monochrome sensor.

[0033] In one embodiment, the image sensor may be a color sensor, in which case the blending coefficient may be the brightness of the pixel values ​​of the color image.

[0034] A gating camera according to one embodiment divides a field of view into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates probe light, the illumination device including a plurality of light-emitting elements, and that is capable of controlling the light distribution of the probe light by selecting whether to use or not use the plurality of light-emitting elements in accordance with a light distribution control signal; an image sensor that is capable of controlling exposure timing; and a controller that (i) controls the light emission timing of the illumination device and the imaging timing of the image sensor for each range, and (ii) determines an area to be irradiated with the probe light in accordance with a driving scene and generates a light distribution control signal.

[0035] According to this configuration, power consumption can be reduced by stopping the irradiation of the probe light in areas where there are no objects to be detected or where any objects present there are unlikely to affect the vehicle's driving.

[0036] In one embodiment, the controller may exclude areas corresponding to the sky from areas that should be illuminated with the probe light.

[0037] In one embodiment, the controller may exclude areas outside the road from areas that should be illuminated with the probe light.

[0038] In one embodiment, the controller may select the entire region in the reference frame as the region to be irradiated with the probe light, and determine the region to be irradiated with the probe light in m (m≧1) normal frames following the reference frame based on multiple slice images obtained in the reference frame.

[0039] In one embodiment, the controller may determine, as a region corresponding to the sky, a range in which pixel values ​​across all slice images among a plurality of slice images captured in the reference frame are smaller than a predetermined threshold value, thereby enabling detection of a sky portion.

[0040] In one embodiment, the controller may select an area to be irradiated with the probe light based on three-dimensional map information.

[0041] In one embodiment, the controller may dynamically control m, the interval between reference frames, depending on road conditions.

[0042] In one embodiment, the controller may reduce m near slopes, where the proportion of sky in the camera's field of view changes significantly and frequently. In such situations, reducing m and frequently updating the probe light distribution enables appropriate sensing.

[0043] In one embodiment, the controller may control the interval at which reference frames occur while navigating a curve based on the radius of curvature of the curve.

[0044] The present invention will be described below based on preferred embodiments with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing are designated by the same reference numerals, and redundant descriptions will be omitted where appropriate. Furthermore, the embodiments are illustrative and do not limit the invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.

[0045] 1 is a block diagram of a gating camera 20A according to embodiment 1. This gating camera 20A is mounted on a vehicle such as an automobile or a motorcycle, and the output of the gating camera 20A is used to determine the type (also referred to as category or class) of an object OBJ present around the vehicle.

[0046] The gating camera 20A includes an illumination device (projector) 22, an image sensor 24, and a camera controller 26. The image capturing by the gating camera 20A is performed by projecting a field of view into a depth direction using a plurality of N (N≧2) range RNGs. １ ~RNG Ｎ Adjacent ranges may overlap at their boundaries in the depth direction of the field of view.

[0047] The illumination device 22 irradiates a probe light L1 ahead of the vehicle in synchronization with a light emission timing signal S1 provided by the camera controller 26. The probe light L1 is preferably infrared light, but may be visible light or ultraviolet light having a predetermined wavelength. The illumination device 22 may be, for example, a laser diode (LD) or an LED.

[0048] The image sensor 24 includes a plurality of pixels, and is capable of controlling exposure in synchronization with an exposure timing signal S2 provided by the camera controller 26, thereby generating a raw image (RAW image) consisting of a plurality of pixels. The image sensor 24 is sensitive to the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ. The i-th range RNG ｉ The slice image IMG generated by the image sensor 24 ｉ is referred to as a raw image or a primary image as needed, and is a slice image IMGs which is the final output of the gating camera 20A. ｉ Distinguish from.

[0049] The image sensor 24 is also configured so that the effective imaging range can be controlled in response to an imaging range control signal S3 from the camera controller 26. The narrower the effective imaging range, the narrower the effective angle of view.

[0050] The camera controller 26 controls the irradiation timing (light emission timing) of the probe light L1 by the illumination device 22 and the exposure timing by the image sensor 24. The camera controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a microcomputer, and a software program executed by the processor (hardware).

[0051] The image processing device 28 converts the raw image IMG_RAW generated by the image sensor 24 into a ｉ The necessary image processing is performed to obtain slice images IMGs. ｉ The slice images IMGs may be identical to the raw images IMG_RAW, in which case the image processing device 28 may be omitted.

[0052] The camera controller 26 receives vehicle information INFO, which includes at least vehicle speed information, from the vehicle body. The camera controller 26 controls the effective image capture range of the image sensor 24 according to the vehicle speed indicated by the vehicle information INFO. Specifically, the camera controller 26 controls the effective image capture range by outputting an image capture range control signal S3 so that the angle of view becomes wider as the vehicle speed decreases and the angle of view becomes narrower as the vehicle speed increases.

[0053] The gating camera 20A is configured as described above, and its operation will now be described.

[0054] FIG. 2 is a diagram illustrating the basic operation of the gating camera 20A. ｉ The illumination device 22 is synchronized with the light emission timing signal S1 and senses the light at time t ０ ~t １ The light emission period τ １ The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. ｉ The distance to the boundary in front of ＭＩＮｉ、 Range RNG ｉThe distance to the inner boundary of ＭＡＸｉ Let's say.

[0055] Light that leaves the lighting device 22 at a certain time travels a distance d ＭＩＮｉ and the round trip time T ＭＩＮｉ is T ＭＩＮｉ = 2 × d ＭＩＮｉ / c, where c is the speed of light.

[0056] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d ＭＡＸｉ and the round trip time T ＭＡＸｉ is T ＭＡＸｉ = 2 × d ＭＡＸｉ / c.

[0057] Range RNG ｉ When it is desired to photograph only the object OBJ included in the ２ = t ０ +T ＭＩＮｉ Exposure begins at time t ３ = t １ +T ＭＡＸｉ The exposure timing signal S2 is generated so that the exposure is completed at this timing. This is one exposure operation.

[0058] i-th range RNG ｉ When capturing an image, multiple sets of light emission and exposure may be performed to generate a bright image by accumulating charges in the pixels of the image sensor 24. In this case, the camera controller 26 may repeatedly generate the light emission timing signal S1 and the exposure timing signal S2 at a predetermined cycle.

[0059] 3A and 3B are diagrams illustrating slice images obtained by the gating camera 20A. In the example of FIG. 3A, the range RNG ２ Object (pedestrian) OBJ ２ exists and the range RNG ３ Object (vehicle) OBJ ３ FIG. 3B shows a plurality of slice images IMG obtained in the situation of FIG. １ ~IMG ３is shown. １ When taking a picture, the image sensor is in the range RNG １ Since the slice image IMG is exposed only by the reflected light from the １ No object image is captured on the screen.

[0060] Slice image IMG ２ When taking a picture, the image sensor is in the range RNG ２ Since the slice image IMG is exposed only by the reflected light from the ２ In the ２ Similarly, only the slice image IMG ３ When taking a picture, the image sensor is in the range RNG ３ Since the slice image IMG is exposed only by the reflected light from the ３ In the ３ In this way, the gating camera 20A can capture images of objects separately for each range.

[0061] Next, dynamic control of the imaging range in the gating camera 20A will be described.

[0062] 4(a) to 4(c) are diagrams illustrating the control of the imaging range (angle of view) according to the vehicle speed. Fig. 4(a) shows the horizontal angle of view, Fig. 4(b) shows the vertical angle of view, and Fig. 4(c) shows the effective imaging range of the image sensor.

[0063] As shown in the upper parts of Figures 4(a) and (b), when traveling at low speeds, it is necessary to recognize pedestrians and other vehicles at close range, so it is necessary to capture a wide area, including the side of the road. Therefore, both the horizontal half angle of view θ and the vertical half angle of view φ are set wide. Specifically, as shown in the upper part of Figure 4(c), the effective capturing range A of the image sensor 24 is set wide so as to include all pixels of the image sensor. The horizontal half angle of view θ is determined corresponding to the number of horizontal pixels x, and the vertical half angle of view φ is determined corresponding to the number of vertical pixels y.

[0064] As shown in the lower part of Figures 4(a) and (b), when driving on a highway or a motorway (driving at high speed), the importance of nearby pedestrians and other vehicles decreases, while the importance of distant objects increases. Therefore, there is less need to capture a wide area, including the roadside. Therefore, both the horizontal half angle of view θ and the vertical half angle of view φ are set narrow. Specifically, as shown in the lower part of Figure 4(c), the area B around the image sensor 24 is invalidated, and only the central portion is cropped (trimmed), thereby narrowing the effective capture range A, and thereby setting the angle of view narrow.

[0065] Next, a specific example of the relationship between the image sensor 24, the effective imaging range (angle of view), and the vehicle speed will be described.

[0066] The camera controller 26 may acquire the stopping distance L according to the vehicle speed v, and determine the effective imaging range of the image sensor 24 based on the stopping distance L and the minimum radius R of any curve on the road on which the vehicle is currently traveling. This makes it possible to prevent the vehicle from colliding with an object even if braking is initiated after the gating camera 20A detects an object ahead.

[0067] The stopping distance L can be calculated as the sum of the free running distance La and the braking distance Lb using the formula (1): L=La+Lb (1)

[0068] The free running distance La can be calculated using the formula (2): La = t ＲＥＳ ×v ... (2) t ＲＥＳ is the reaction time and v is the vehicle speed.

[0069] The braking distance Lb can be calculated, for example, by the following equation (3): Lb=v' ２ / (254 × μ) ... (3) where v' is the vehicle speed just before braking, and μ is the coefficient of friction. The coefficient of friction μ varies depending on the condition of the road surface. For example, μ = 0.7 on dry asphalt or concrete, 0.5 on wet asphalt or concrete, 0.15 on hardened snow, and 0.07 on ice.

[0070] When calculating the stopping distance L, the road surface condition may be taken into consideration. This allows for an accurate braking distance Lb to be obtained. If the road surface condition is not taken into consideration, the braking distance can be calculated by assuming that the friction coefficient μ is constant.

[0071] FIG. 5 is a diagram showing a vehicle traveling on a curve. Assume that braking is initiated at point P while traveling on a curve with a curvature radius R centered at point O. While the actual curve is a clothoid curve, it is assumed to be a circular arc here for the sake of simplicity. When the stopping distance is L, the angle λ (rad) between lines OP and OQ is given by λ·R=L, so in the figure, angle λ is given by λ=L / R (rad). In this case, the angle θ from point P toward stopping position Q is given by θ=λ / 2=L / (2R) (rad). Therefore, if the half angle of view of gating camera 20A is set to be equal to or greater than angle θ, objects beyond stopping point Q can be detected. In other words, the effective imaging range of image sensor 24 is controlled so that the half angle of view (unit: rad) falls within a range with L / (2R) as its lower limit.

[0072] For example, the half angle of view θ of the gating camera 20A may be determined to be wider than L / (2R) and narrower than 1.2×L / (2R). When the coefficient α is set to 1≦α≦1.2, the half angle of view θ may be determined according to equation (4): θ=α×L / (2R) (4)

[0073] When traveling at 100 km / h on dry asphalt, the stopping distance L is 77 m. In this case, assuming a curve with radius R = 400 m, the half angle of view should be 5.52 degrees or more. R = 400 m is the minimum radius assumed for a road with a speed limit of 100 km / h.

[0074] When traveling at 100 km / h on wet asphalt, the stopping distance L increases to 99 m. In this case, assuming a curve with R = 400 m, the half angle of view should be 7.13 degrees or more.

[0075] When driving on a snow-covered road at 60 km / h, the stopping distance L increases to 107 m. In this case, assuming a curve with R = 400 m, the half angle of view should be 7.66 degrees or more.

[0076] When traveling at 120 km / h on dry asphalt, the stopping distance L is 106 m. In this case, assuming a curve with radius R = 710 m, the half angle of view should be 4.28 degrees or more. R = 710 m is the minimum radius assumed for a road with a speed limit of 120 km / h.

[0077] In the above explanation, the half angle of view is determined based on the curve with the smallest radius expected on the road currently being traveled, but this is not limited to this. The half angle of view θ may be controlled based on the formula (4) using the radius of curvature R of the curve currently being traveled and the braking distance L.

[0078] The half angle of view θ may be determined based on different calculation formulas for curved roads and straight roads. An example of control on a straight road with multiple lanes on one side will be described below. Figures 6(a) and 6(b) are diagrams showing a vehicle traveling on a straight road. Figures 6(a) and 6(b) assume a road with three lanes on each side. The width of one lane is l, and the stopping distance is L. Figure 6(a) shows the vehicle traveling in the left lane, and Figure 6(b) shows the vehicle traveling in the center lane.

[0079] Let x be the horizontal distance from the vehicle to the edge of the road in the lateral direction. In the situation shown in FIG. 6( a), x = 1.5×l, and in the situation shown in FIG. 6( b), x = 2.5×l. The angle θ at which an object present at point Q on the road edge, a stopping distance L away, is viewed is given by θ = arcsin(x / L). Therefore, if the half angle of view of gating camera 20A is set to be equal to or greater than angle θ, objects beyond stopping point Q can be detected. For example, the half angle of view θ of gating camera 20A may be set to be wider than arcsin(x / L) and narrower than 1.2×arcsin(x / L). When the coefficient β is set to be 1≦β≦1.2, the half angle of view θ may be determined according to equation (5): θ = β×arcsin(x / L) (5)

[0080] In FIG. 6A, when the vehicle is traveling at 100 km / h on dry asphalt, the stopping distance L is 77 m. If l = 3.5 m, then x = 3.5 × 1.5. The half angle of view θ is given by θ ≧ sin －１ It is sufficient to determine it so that (3.5×1.5 / 77)=0.06817 (rad)=3.906 (deg).

[0081] In FIG. 6B, when traveling on snow at 60 km / h, the stopping distance L is 107 m. If l=3.5 m, then x=3.5×2.5. The half angle of view θ is given by θ≧sin －１ It is sufficient to determine it as follows: (3.5×2.5 / 107)=0.08188 (rad)=4.691 (deg).

[0082] Note that since the distance x is determined using the lane the vehicle is traveling in as a parameter, it may not be easy for the camera controller 26 to obtain the distance x. In that case, regardless of the actual lane the vehicle is traveling in, the distance x may be obtained by assuming that the vehicle is traveling along the extreme diagonal line as shown in FIG. 6A, and the half angle of view θ may be calculated from equation (5). The number of lanes and the width l of the diagonal lines may be obtained from a car navigation system or from outside the vehicle via wireless communication.

[0083] Alternatively, a representative distance x may be determined as a constant in consideration of various possible roads and stored in the camera controller 26 .

[0084] Assume that the half angle of view is 12.8 degrees when the effective imaging range of the image sensor 24 is maximized. Taking into account various vehicle speeds and road conditions, the half angle of view will vary within a range of 4 degrees to 10 degrees, which is 30% to 80% of the original half angle of view. If cropping is performed at the same rate not only horizontally but also vertically, the number of pixels will be 30% x 30% to 80% to 80% = 9% to 64%, which shows that the amount of data can be significantly reduced.

[0085] A modification related to the first embodiment will be described.

[0086] (Variation 1.1) In the embodiment, the effective imaging range (angle of view) is controlled in accordance with the vehicle speed for two axes, the horizontal and vertical directions. However, this is not limited to this, and the angle of view / effective imaging range may be changed only in the horizontal direction.

[0087] (Variation 1.2) In the embodiment, the effective image capture range is controlled according to the vehicle speed for both curves and straight roads, but this is not limited to this. For example, for curves, the control of the effective image capture range based on the vehicle speed may be disabled and the effective image capture range may be fixed at the maximum, while the effective image capture range may be controlled based on the vehicle speed only for straight roads.

[0088] (Variation 1.3) In the embodiment, the effective imaging range (angle of view) for curves is controlled based on a first relational expression f(v) that uses vehicle speed v as an argument, and the effective imaging range (angle of view) for straight roads is controlled based on a second relational expression g(v) that is different from the first relational expression f(v). Equation (4) is an example of relational expression f(v), and equation (5) is an example of relational expression g(v). L in equations (4) and (5) is a function of v.

[0089] In Modification 1.3, the effective imaging range and therefore the angle of view may be controlled based on a common relational expression h(v) that uses at least the vehicle speed v as an argument. In this case, the relational expression h(v) may be determined to satisfy the following: h(v) ≥ max(f(v), g(v)), where max() is a function indicating the larger one.

[0090] (Variation 1.4) If the half angle of view is determined based on the same calculation formula for both general roads and expressways, an undesirable situation may occur. In such a case, different parameters (e.g., α in equation (4), β and x in equation (5)) may be specified for general roads and expressways.

[0091] Alternatively, on general roads, the control of the effective image capturing range based on the vehicle speed may be disabled and the effective image capturing range may be fixed to the maximum.

[0092] 7 is a block diagram of a gating camera 20B according to embodiment 2. This gating camera 20B is mounted on a vehicle such as an automobile or a motorcycle, and the output of the gating camera 20B is used to determine the type (also referred to as category or class) of an object OBJ present around the vehicle.

[0093] The gating camera 20B includes an illumination device (projector) 22, an image sensor 24, a camera controller 26, and an image processing device 28. The image capturing by the gating camera 20B is performed by capturing an image of a field of view in the depth direction using a plurality of N (N≧2) range RNGs. １ ~RNG Ｎ Adjacent ranges may overlap at their boundaries in the depth direction of the field of view.

[0094] The illumination device 22 irradiates a probe light L1 ahead of the vehicle in synchronization with a light emission timing signal S1 provided by the camera controller 26. The probe light L1 is preferably infrared light, but may be visible light or ultraviolet light having a predetermined wavelength. The illumination device 22 may be, for example, a laser diode (LD) or an LED.

[0095] The image sensor 24 includes a plurality of pixels, and is capable of controlling exposure in synchronization with an exposure timing signal S2 provided by the camera controller 26. The image sensor 24 generates a raw image consisting of the plurality of pixels. The image sensor 24 is sensitive to the same wavelength as the probe light L1, and captures reflected light (return light) L2 reflected by the object OBJ. In this embodiment, the image sensor 24 is a monochrome IR sensor.

[0096] The camera controller 26 controls the irradiation timing (light emission timing) of the probe light L1 by the illumination device 22 and the exposure timing by the image sensor 24. The camera controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a microcomputer, and a software program executed by the processor (hardware).

[0097] The image processing device 28 converts the raw image IMG_RAW generated by the image sensor 24 into a ｉ The necessary image processing is performed to obtain slice images IMGs. ｉ The raw images IMG_RAW generated in each range may be used as slice images IMGs as they are.

[0098] The image processing device 28 also processes a plurality of slice images IMGs １ ~IMGs Ｎ (or IMG_RAW １ ~IMG_RAW Ｎ ) to generate a composite image IMGc.

[0099] The generation of the composite image IMGc by the image processing device 28 will be described below.

[0100] N range RNGs １ ~RNG Ｎ , N different colors C １ ~C Ｎ is assigned.

[0101] Each pixel of the composite image IMGc is assigned to N colors C １ ~C Ｎ are respectively N slice images IMGs １ ~IMGs Ｎ The pixel value v of the corresponding pixel １ ~v Ｎ The pixel values ​​are weighted and added with coefficients based on the above.

[0102] The gating camera 20B is configured as described above, and its operation will now be described.

[0103] FIG. 8 is a diagram illustrating the basic operation of the gating camera 20B. ｉ The illumination device 22 is synchronized with the light emission timing signal S1 and senses the light at time t ０ ~t １ The light emission period τ １The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. ｉ The distance to the boundary in front of ＭＩＮｉ、 Range RNG ｉ The distance to the inner boundary of ＭＡＸｉ Let's say.

[0104] Light that leaves the lighting device 22 at a certain time travels a distance d ＭＩＮｉ and the round trip time T ＭＩＮｉ is T ＭＩＮｉ = 2 × d ＭＩＮｉ / c, where c is the speed of light.

[0105] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d ＭＡＸｉ and the round trip time T ＭＡＸｉ is T ＭＡＸｉ = 2 × d ＭＡＸｉ / c.

[0106] Range RNG ｉ When it is desired to photograph only the object OBJ included in the ２ = t ０ +T ＭＩＮｉ Exposure begins at time t ３ = t １ +T ＭＡＸｉ The exposure timing signal S2 is generated so that the exposure is completed at this timing. This is one exposure operation.

[0107] i-th range RNG ｉ When capturing an image, multiple sets of light emission and exposure may be performed to generate a bright image by accumulating charges in the pixels of the image sensor 24. In this case, the camera controller 26 may repeatedly generate the light emission timing signal S1 and the exposure timing signal S2 at a predetermined cycle.

[0108] 9A and 9B are diagrams illustrating slice images obtained by the gating camera 20B. In the example of FIG. 9A, the range RNG ２ Object (pedestrian) OBJ ２exists and the range RNG ３ Object (vehicle) OBJ ３ FIG. 9B shows a plurality of slice images IMG obtained in the situation of FIG. １ ~IMG ３ is shown. １ When taking a picture, the image sensor is in the range RNG １ Since the slice image IMG is exposed only by the reflected light from the １ No object image is captured on the screen.

[0109] Slice image IMG ２ When taking a picture, the image sensor is in the range RNG ２ Since the slice image IMG is exposed only by the reflected light from the ２ In the ２ Similarly, only the slice image IMG ３ When taking a picture, the image sensor is in the range RNG ３ Since the slice image IMG is exposed only by the reflected light from the ３ In the ３ In this way, the gating camera 20B can capture images of objects separately for each range.

[0110] Next, the generation of the composite image IMGc by the image processing device 28 will be described.

[0111] 10(a) to 10(d) are diagrams illustrating a composite image and a conventional distance image. FIG. 10(a) shows an actual field of view. FIG. 10(b) shows an IR image captured from the field of view of FIG. 10(a). FIG. 10(c) shows a distance image generated by a conventional ToF camera. FIG. 10(d) shows a composite image IMGc generated in this embodiment.

[0112] As shown in Fig. 10(c), in the range image generated by the conventional ToF camera, objects (in this example, signs) at the same distance are displayed in the same color, resulting in the loss of information such as characters and figures contained in the IR image as shown in Fig. 10(b).

[0113] In contrast, the composite image IMGc according to this embodiment includes text and graphics written on the surface of an object present in the same area. Therefore, subsequent image processing can be performed to obtain more detailed information about the object based on the composite image IMGc. Alternatively, by displaying the composite image IMGc on a display, the user can recognize the type of sign or instructions.

[0114] Below, several specific examples of methods for generating the composite image IMGc will be described.

[0115] (Example 2.1) In Example 2.1, the RGB color system is used to perform the synthesis process. １ ~RNG Ｎ corresponding to multiple colors (R １ , G １ , B １ ) ~ (R Ｎ , G Ｎ , B Ｎ ) is assigned.

[0116] 11 is a diagram illustrating a plurality of colors assigned to a plurality of ranges in the RGB color system. For example, the following colors can be used as colors corresponding to the ranges: Red (1,0,0) Yellow (1,1,0) Green (0,1,0) Cyan (0,1,1) Blue (0,0,1) Magenta (1,0,1)

[0117] The i-th (1≦i≦N) color (R ｉ , G ｉ , B ｉ ) and the (i+1)th color (R ｉ＋１ , G ｉ＋１ , B ｉ＋１ ) have a relationship in which the value of one element among R, G, and B is different and the values ​​of the remaining two elements are the same. In the chart of FIG. 11, two adjacent colors satisfy this relationship. In this embodiment, the elements of each color (R, G, and B in this example) are normalized to 1.

[0118] Red and yellow are related in that they have two equal elements, R and B, and one different element, G. Green and cyan are related in that they have two equal elements, R and G, and one different element, B.

[0119] For example, if the number N of ranges is 6 or less, then it is sufficient to use N adjacent colors from the six colors from red to magenta. Note that magenta and red are considered to be adjacent colors.

[0120] In addition to the six colors exemplified above, white (1,1,1) and black (0,0,0) may also be used. White (1,1,1) differs from yellow, cyan, and magenta by one element. Black (0,0,0) differs from red, green, and blue by one element.

[0121] If the number of ranges, N, is greater than 6, intermediate colors can be used. For example, between red (1,0,0) and yellow (1,1,0), the color (1,X,0) can be defined, where 0<X<1, for example, X=0.5. If more m colors are needed, the color (1,α １ , 0), (1, α ２ , 0) ... (1, α ｍ , 0) may be used, where 0<α １ <α ２ <...<α ｍ <1.

[0122] Similarly, colors can be defined between yellow and green, green and cyan, cyan and blue, and blue and magenta.

[0123] The pixel value (R, G, B) of each pixel of the composite image IMGc is expressed by the following equation using the pixel value v of the corresponding pixel in the plurality of slice images: (R, G, B)=Σ ｊ＝１：Ｎ [v ｊ ・(R ｊ , G ｊ , B ｊ ) ]

[0124] 12 is a diagram illustrating the synthesis of the composite image IMGc. As described above, in the theory of the gating camera, any object point (object) is captured only in the slice image corresponding to the range in which it exists. In this case, if a pixel in a slice image has a non-zero pixel value, the pixel value of the corresponding pixel in the other slice images is zero.

[0125] FIG. 12 shows two range RNGs. １ , RNG ２Two slice images IMGs corresponding to １ , IMGs ２ In this example, the first range RNG １ is assigned red (1,0,0), and the second range RNG ２ shall be assigned yellow (1,1,0).

[0126] Slice images IMGs １ , IMGs ２ is an 8-bit monochrome IR image, the pixel value of each pixel is expressed in 256 gradations from 0 to 255. １ , IMGs ２ The numbers in the boxes represent pixel values ​​for each region.

[0127] Slice images IMGs １ is colored red by multiplying the pixel value of each pixel by (1, 0, 0). １ Objects present in the image are represented in the range from black (0,0,0) to red (1,0,0).

[0128] Similarly, slice images IMGs ２ is colored yellow by multiplying the pixel value of each pixel by (1, 1, 0). ２ Objects present in the image are represented in the range of black (0,0,0) to yellow (1,1,0).

[0129] In this way, in the composite image IMGc, the color family represents the range, and the brightness within the same color family represents the reflectance of the object.

[0130] An object point near the boundary of a range may be captured across two adjacent slice images, in which case the same pixel in the two slice images will have a non-zero pixel value. Also, if adjacent ranges are defined to overlap in the depth direction, the same pixel in the two slice images will have a non-zero pixel value.

[0131] 13 is a diagram illustrating a synthesis process when the same object is captured across multiple slice images. In this example, two slice images IMGs １ , IMGs ２ The same sign is shown in the image. If you focus on the sign pole, you can see the slice image IMGs １ The pixel value of the slice image IMGs is 90. ２ The pixel value of the pole in the composite image IMGc is 90×(1,0,0)+25×(1,1,0)=(115,25,0).

[0132] Focusing on the rectangular part in the center of the sign, the slice image IMGs １ The pixel value of the slice image IMGs is 120. ２ The pixel value of the rectangular portion in the center of the sign in the composite image IMGc is 120×(1,0,0)+30×(1,1,0)=(150,30,0).

[0133] In this way, an object that straddles two slice images IMGs can be expressed in multiple tones using a color that is a blend of two colors (1,0,0) and (1,1,0), making it possible to identify the content of the sign.

[0134] FIG. 14 is a diagram showing an example of the correspondence between a plurality of ranges and colors in Example 2.1. In this example, the field of view from 10 m to 110 m in depth is divided into N=21 regions of 5 m each and photographed. Range RNG １ is in the range of 10 to 15 m and is assigned red (1,0,0). The range between red, yellow, green, cyan, and blue is divided into five parts, and intermediate colors are used.

[0135] (Example 2.2) In Example 2.2, a Yxy color system (also called an XYZ color system) is used. FIG. 15 is a diagram illustrating a plurality of colors assigned to a plurality of ranges in the Yxy color system. The color for the i-th range is determined by assigning a chromaticity point (x ｉ , y ｉ ) is defined as:

[0136] As an example, using the chromaticity points R (0.640, 0.330), G (0.300, 0.600), and B (0.150, 0.060) of the international standard sRGB, multiple chromaticity points located on the sides (including the vertices) of a triangle with three vertices may be assigned to multiple ranges.

[0137] For example, a plurality of chromaticity points located on two straight lines starting from R (0.640, 0.330) and going through G (0.300, 0.600) to B (0.150, 0.060) may be used.

[0138] The pixel value (Y, x, y) of each pixel of the composite image IMGc is expressed as follows: (Y, x, y)=(Σ ｊ＝１：Ｎ (v ｊ ), Σ ｊ＝１：Ｎ (v ｊ ・x ｊ ) / Σ ｊ＝１：Ｎ v ｊ , Σ ｊ＝１：Ｎ (v ｊ ・y ｊ ) / Σ ｊ＝１：Ｎ v ｊ ) is expressed as

[0139] The luminance Y of the composite image IMGc is the sum of the pixel values ​​of the slice images.

[0140] The chromaticity point (x, y) of each pixel in the composite image IMGc will be explained. １ ~RNG Ｎ For the chromaticity point (x １ , y １ ) ~ (x Ｎ , y Ｎ ) are assigned, and slice images IMGs １ ~IMGs Ｎ The pixel value of a pixel in １ ~v Ｎ In this case, the chromaticity point (x, y) of the same pixel in the composite image IMGc is expressed as follows: (x, y) = (Σ ｊ＝１：Ｎ (v ｊ ・x ｊ ) / Σ ｊ＝１：Ｎ v ｊ , Σ ｊ＝１：Ｎ (v ｊ ・y ｊ ) / Σ ｊ＝１：Ｎ v ｊ ) = Σｊ＝１：Ｎ {v ｊ ・(x ｊ , y ｊ ) / Σ ｊ＝１：Ｎ v ｊ That is, the chromaticity points (x １ , y １ ) ~ (x Ｎ , y Ｎ ) are weighted and added (i.e., blended) using coefficients based on the pixel values ​​of the corresponding pixels.

[0141] For example, range RNG １ For R, the chromaticity point (x １ , y １ ) = (0.640, 0.330) and assign the next range RNG ２ chromaticity point (x ２ , y ２ ) = (0.606, 0.357).

[0142] Slice images IMGs １ The pixel value of a pixel of interest in the slice image IMGs is 80. ２ Assume that the pixel value of the same pixel of interest in the composite image IMGc is 160. In this case, the Y value of the pixel of interest in the composite image IMGc is 80+160=240.

[0143] Furthermore, the chromaticity point (x, y) of the same pixel in the composite image IMGc is (x, y) = 80 / (80 + 160) x (0.640, 0.330) + 160 / (80 + 160) x (0.606, 0.357) = (0.617, 0.348). This means that the two colors are blended at a ratio of 1:2. In the chromaticity diagram of Fig. 15, the chromaticity point of the composite image IMGc is the point obtained by internally dividing the two chromaticity points assigned to the slice images.

[0144] FIG. 16 is a diagram showing an example of the correspondence between a plurality of ranges and colors in Example 2.2.

[0145] (Example 2.3) Fig. 17 is a diagram showing an example of the correspondence between a plurality of ranges and colors in Example 2.3. N colors assigned to N ranges are determined so that they have different hues H (Hue) in the HSV color system. Hue H can take a value between 0 and 360 degrees. The hue of the color assigned to the i-th range is defined as H ｉ It is written as H ｉ = H ０ +i × ΔH. ０ , ΔH is a constant.

[0146] The pixel value (H, S, V) of each pixel of the composite image IMGc is expressed as follows: (H, S, V)=(Σ ｊ＝１：Ｎ (v ｊ ・H ｊ ) / Σ ｊ＝１：Ｎ v ｊ , S ０ , Σ ｊ＝１：Ｎ (v ｊ )) is expressed as S ０ is a constant, which may be set to, for example, 100%.

[0147] The brightness V (Value) of the composite image IMGc is the sum of the pixel values ​​of the slice images. The saturation S of each pixel of the composite image IMGc is a predetermined constant S ０ is.

[0148] The hue H of each pixel in the composite image IMGc is the hue H assigned to the plurality of slice images. １ ~H Ｎ are weighted by the pixel value v and added together.

[0149] For example, range RNG １ For hue H １ = 0, range RNG ２ For hue H ２ = 12 is assigned.

[0150] Slice images IMGs １ The pixel value of a pixel of interest in the slice image IMGs is 80. ２ Assume that the pixel value of the same pixel of interest in the composite image IMGc is 160. In this case, the lightness V of the pixel of interest in the composite image IMGc is 80+160=240.

[0151] The hue H of the same pixel in the composite image IMGc is H = 80 / (80 + 160) x 0 + 160 / (80 + 160) x 12 = 8. This means that the two hues are blended at a ratio of 1:2. In the color space of FIG. 16 , the hue H of the composite image IMGc is the point obtained by internally dividing the two hues assigned to the slice images.

[0152] 18 is a diagram showing an example of the correspondence between a plurality of ranges and colors in Example 2.3. In this example, hues are defined for 21 ranges at intervals of ΔH=12.

[0153] 19 is a block diagram of a gating camera 20C according to embodiment 3. This gating camera 20C is mounted on a vehicle such as an automobile or a motorcycle, and the output of the gating camera 20C is used to determine the type (also referred to as category or class) of an object OBJ present around the vehicle.

[0154] The gating camera 20C includes an illumination device (projector) 22, an image sensor 24, and a camera controller 26. The imaging by the gating camera 20C is performed by projecting a field of view into a depth direction using a plurality of N (N≧2) range RNGs. １ ~RNG Ｎ Adjacent ranges may overlap at their boundaries in the depth direction of the field of view.

[0155] The illumination device 22 irradiates a probe light L1 ahead of the vehicle in synchronization with a light emission timing signal S1 provided by the camera controller 26. The probe light L1 is preferably infrared light, but may be visible light or ultraviolet light having a predetermined wavelength. The illumination device 22 may be, for example, a laser diode (LD) or an LED.

[0156] In this embodiment, the lighting device 22 is configured to divide the virtual vertical screen 900 into a plurality of regions 902 and switch between irradiating and not irradiating the probe light L1 for each region, thereby enabling light distribution control. A light distribution control signal S4 specifying which region should be irradiated with the probe light is input to the lighting device 22 from the camera controller 26. The lighting device 22 selects one or more regions specified by the light distribution control signal S4 and irradiates the selected region (referred to as an irradiation region) with the probe light L1 to form a light distribution pattern PTN.

[0157] The image sensor 24 includes a plurality of pixels, and is capable of controlling exposure in synchronization with an exposure timing signal S2 provided by the camera controller 26, thereby generating a raw image (RAW image) consisting of a plurality of pixels. The image sensor 24 is sensitive to the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ. The i-th range RNG ｉ The slice image IMG generated by the image sensor 24 ｉ is referred to as a raw image or a primary image as needed, and is a slice image IMGs which is the final output of the gating camera 20C. ｉ Distinguish from.

[0158] The camera controller 26 controls the irradiation timing (light emission timing) of the probe light L1 by the illumination device 22 and the exposure timing by the image sensor 24. The camera controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a microcomputer, and a software program executed by the processor (hardware).

[0159] The image processing device 28 converts the raw image IMG_RAW generated by the image sensor 24 into a ｉ The necessary image processing is performed to obtain slice images IMGs. ｉ The slice images IMGs may be identical to the raw images IMG_RAW, in which case the image processing device 28 may be omitted.

[0160] The camera controller 26 selects an area to be irradiated with the probe light L1 from among multiple areas 902 on the virtual vertical screen 900 according to the driving scene of the vehicle equipped with the gating camera 20C, and generates a light distribution control signal S4.

[0161] The gating camera 20C has the above configuration. Next, its operation will be described.

[0162] FIG. 20 is a diagram illustrating the basic operation of the gating camera 20C. ｉ The illumination device 22 is synchronized with the light emission timing signal S1 and senses the light at time t ０ ~t １ The light emission period τ １ The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. ｉ The distance to the boundary in front of ＭＩＮｉ、 Range RNG ｉ The distance to the inner boundary of ＭＡＸｉ Let's say.

[0163] Light that leaves the lighting device 22 at a certain time travels a distance d ＭＩＮｉ and the round trip time T ＭＩＮｉ is T ＭＩＮｉ = 2 × d ＭＩＮｉ / c, where c is the speed of light.

[0164] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d ＭＡＸｉ and the round trip time T ＭＡＸｉ is T ＭＡＸｉ = 2 × d ＭＡＸｉ / c.

[0165] Range RNG ｉ When it is desired to photograph only the object OBJ included in the ２ = t ０ +T ＭＩＮｉExposure begins at time t ３ = t １ +T ＭＡＸｉ The exposure timing signal S2 is generated so that the exposure is completed at this timing. This is one exposure operation.

[0166] i-th range RNG ｉ When capturing an image, multiple sets of light emission and exposure may be performed to generate a bright image by accumulating charges in the pixels of the image sensor 24. In this case, the camera controller 26 may repeatedly generate the light emission timing signal S1 and the exposure timing signal S2 at a predetermined cycle.

[0167] 21(a) and (b) are diagrams illustrating slice images obtained by the gating camera 20C. In the example of FIG. 21(a), the range RNG ２ Object (pedestrian) OBJ ２ exists and the range RNG ３ Object (vehicle) OBJ ３ FIG. 21(b) shows a plurality of slice images IMG obtained in the situation of FIG. 21(a). １ ~IMG ３ is shown. １ When taking a picture, the image sensor is in the range RNG １ Since the slice image IMG is exposed only by the reflected light from the １ No object image is captured on the screen.

[0168] Slice image IMG ２ When taking a picture, the image sensor is in the range RNG ２ Since the slice image IMG is exposed only by the reflected light from the ２ In the ２ Similarly, only the slice image IMG ３ When taking a picture, the image sensor is in the range RNG ３ Since the slice image IMG is exposed only by the reflected light from the ３ In the ３ In this way, the gating camera 20C can capture images of objects separately for each range.

[0169] Next, the control of the illumination area according to the driving scene will be described.

[0170] Depending on the driving scene, there may be fixed ranges in the field of view where no object to be detected can exist. A typical example of such a range is the sky. Also, depending on the driving scene, it is possible that an object existing in a specific range of the field of view cannot affect the driving of the vehicle. In this embodiment, power consumption can be reduced by stopping the irradiation of probe light in such ranges.

[0171] Furthermore, the image sensor 24 does not need to capture images of areas not irradiated with the probe light. Therefore, the camera controller 26 may change the capture range (readout range) of the image sensor 24 in conjunction with the light distribution of the probe light. This reduces the power consumption of the image sensor 24. Furthermore, the frame rate can be increased because the time required to read and transfer image data can be shortened.

[0172] 22(a) to 22(c) are diagrams illustrating an example of control of the illumination area according to a driving scene. FIG. 22(a) shows the field of view in a certain driving scene. FIG. 22(b) is a diagram showing the field of view in FIG. 22(a) superimposed on a virtual vertical screen 900 divided into multiple areas. In this driving scene, the upper part of the field of view is occupied by the sky, and it is unlikely that other vehicles, pedestrians, signs, or other objects to be detected by the gating camera 20C are present in that range.

[0173] In this example, a range 904 of the virtual vertical screen 900 corresponds to the sky. FIG. 22C shows the light distribution pattern PTN of the probe light L1. The range 904 corresponding to the sky is excluded from the irradiation area 906 of the probe light. As a result, the slice image IMGs captured by the gating camera 20C is １ ~IMGs Ｎ The image will reflect objects that exist outside the sky.

[0174] 23(a) to 23(c) are diagrams illustrating another example of control of the illumination area according to the driving scene. Fig. 23(a) shows the field of view in the driving scene, and Fig. 23(b) is a diagram showing the field of view in Fig. 23(a) superimposed on a virtual vertical screen 900 divided into multiple areas. Fig. 23(c) shows the light distribution pattern PTN of the probe light L1 in the driving scene of Fig. 23(a).

[0175] Depending on the driving scene, it is assumed that objects present within a specific range of the field of view may not affect the driving of the vehicle. Figure 23 (a) is an example of a field of view, and this example shows that only natural objects such as trees exist outside the road. In this case, the level of interest in objects present within the range outside the road can be said to be low. The same applies when only artificial objects such as buildings exist outside the road. In another example, even in a driving scene where both sides of the road are covered with guardrails or side walls and objects outside the guardrails or side walls cannot enter the road, the level of interest in objects present within the range outside the road can be low.

[0176] In this example, a range 904 of the virtual vertical screen 900 corresponds to the sky, and ranges 908A and 908B indicate areas outside the road.

[0177] In this driving scene, as shown in FIG. 23C, an area 904 corresponding to the sky and areas 908A and 908B outside the road are excluded from the irradiation area 906 of the probe light. As a result, the slice image IMGs captured by the gating camera 20C １ ~IMGs Ｎ The image will show objects that exist outside the sky and roads.

[0178] As another control example, assume that the vehicle is traveling on the left side in the driving scene shown in FIG. 22( a). In this case, it may be difficult to imagine that an object outside the road in the oncoming lane (a bicycle in this example) will immediately affect the vehicle's driving. While FIG. 22( a) shows a road with one lane on each side, this tendency is particularly pronounced in the case of a road with two lanes on each side. In such a driving scene, the area outside the road in the oncoming lane may be excluded from the illumination area of ​​the probe light.

[0179] The light distribution control of the probe light described above is required to appropriately recognize the driving scene. １ ~IMGs Ｎ The light distribution of the probe light may be controlled based on the above.

[0180] 24 is a diagram illustrating light distribution control by the camera controller 26. The camera controller 26 alternately repeats a reference frame REF and subsequent m (m≧1) normal frames. Each of the reference frame and normal frame includes sensing over the entire range.

[0181] In the reference frame REF, all regions are irradiated, and the entire field of view is irradiated with the probe light. The camera controller 26 generates slice images IMGs obtained in the reference frame REF. １ ~IMGs Ｎ Based on this, the driving scene is judged and the light distribution of the probe light is determined.

[0182] In principle, there is no object that reflects the probe light L1 in the part corresponding to the sky. Therefore, for the sky part, multiple range RNGs are used. １ ~RNG Ｎ Slice images IMGs obtained for １ ~IMGs Ｎ Therefore, the camera controller 26 selects the upper part of the image and all the slice images IMGs １ ~IMGs Ｎ A portion where the pixel value is smaller than a predetermined threshold value may be determined to be sky.

[0183] Using the light distribution pattern determined based on the slice images of the reference frame REF, sensing is repeated for the following m frames as normal frames.

[0184] The power consumption increases during the reference frames and decreases during the normal frames depending on the driving scene. By repeating this operation, the gating camera 20C can form a light distribution suitable for the driving scene and reduce power consumption.

[0185] The off-road area 908 may be determined based on the slice images IMGs, but this requires advanced image processing. Therefore, the off-road area 908 may be determined by referring to map information. For example, if highly accurate three-dimensional map information is available, the camera controller 26 can accurately reproduce the field of view in each driving scene, making it easy to determine the off-road area 908. Light distribution control not based on slice images (for example, based on three-dimensional map information) may be performed in a reference frame or in a normal frame.

[0186] Here, the power consumption of the gating camera 20C is large in the reference frame, so the lower the frequency of occurrence of the reference frame, in other words, the larger m, the greater the effect of reducing power consumption. On the other hand, if m is set too large, a discrepancy will occur between the current driving scene in the normal frame and the past driving scene in the reference frame, which may result in inappropriate light distribution of the probe light. Therefore, the gating camera 20C may dynamically control the parameter m according to road conditions.

[0187] For example, the camera controller 26 may reduce m near a slope. Around a slope, the proportion of sky included in the camera's field of view changes significantly and frequently. Therefore, in such a situation, appropriate sensing becomes possible by reducing m and frequently updating the light distribution of the probe light. Whether or not the vehicle is near a slope may be determined based on three-dimensional map information. Alternatively, the determination may be based on the output of an inclination sensor (acceleration sensor) mounted on the vehicle.

[0188] On a slope, the vertical field of view changes significantly, while on a curve, the horizontal field of view changes significantly. Therefore, the upper limit of m may be determined taking the curve into consideration. Figure 25 is a diagram illustrating the change in horizontal field of view while traveling around a curve. When traveling at a maximum speed of 120 km / h and a curve radius R of 710 m (the smallest expected curve), the field of view changes by 2.68 degrees, 0.26 degrees, 0.13 degrees, 0.054 degrees, and 0.027 degrees over 1 second, 0.1 seconds, 0.05 seconds, 0.02 seconds, and 0.01 seconds. If the field of view changes by approximately 0.26 degrees, it can be said that the position of the sky hardly changes. Therefore, the upper limit of m may be determined so that a reference frame is generated at least every 0.1 seconds.

[0189] More precisely, the camera controller 26 may dynamically control the intervals at which the reference frames m are generated based on the radius of curvature of the curve while the vehicle is traveling around the curve. Let v be the vehicle speed, R be the radius of curvature, and θ be the allowable change in the field of view between one reference frame and the next. ＭＡＸ (rad). The upper limit of the time interval of the reference frame is t ＭＡＸ Then, the following relation holds: θ ＭＡＸ = v t ＭＡＸ Therefore, the upper limit of the time interval of the reference frame is t ＭＡＸ = 2R θ ＭＡＸ That is, the upper limit of the time interval of the reference frame may be changed so as to be proportional to the radius of curvature R and inversely proportional to the vehicle speed v.

[0190] Next, the configuration of the lighting device 22 will be described. Fig. 26 is a diagram showing an example of the configuration of the lighting device 22. The lighting device 22 includes a plurality of semiconductor light-emitting elements 60, an optical system 62, and a lighting circuit 64. The plurality of semiconductor light-emitting elements 60 are arranged in an array. The array may be one-dimensional or two-dimensional. An LD (laser diode) is suitable as the semiconductor light-emitting element 60, but an LED (light-emitting diode) or an organic EL (electroluminescence) element may also be used.

[0191] The optical system 62 projects the light emitted from each of the semiconductor light emitting elements 60 onto a corresponding one of a plurality of regions on a virtual vertical screen. The configuration of the optical system 62 is not particularly limited, and may be a lens array or a light guide provided with a plurality of steps.

[0192] The lighting circuit 64 selects a plurality of semiconductor light-emitting elements 60 to be driven in response to the light distribution control signal S4. Then, a drive current is supplied to the selected semiconductor light-emitting elements 60 in synchronization with the light-emission timing signal S1. The lighting circuit 64 may include a plurality of constant current drivers 66 corresponding to the plurality of semiconductor light-emitting elements 60. Some of the plurality of constant current drivers 66 become active in response to the light distribution control signal S4, and the active constant current drivers 66 generate a drive current in synchronization with the light-emission timing signal S1.

[0193] A modification related to the third embodiment will be described.

[0194] (Modification 3.1) In the third embodiment, the light distribution of the probe light is divided two-dimensionally and controlled. However, it may be divided only in the vertical direction, and only the control for excluding the sky portion may be performed.

[0195] (Modification 3.2) Alternatively, control to exclude areas outside the road may be performed without control to exclude the sky area.

[0196] Next, the uses of the gating cameras 20A to 20C (hereinafter collectively referred to as the gating camera 20) according to the first to third embodiments will be described.

[0197] 27 is a block diagram of the sensing system 10. The sensing system 10 includes a processing unit 40 in addition to the gating camera 20 described above. The sensing system 10 is an object detection system that is mounted on a vehicle such as an automobile or motorcycle, and determines the type (also referred to as category or class) of an object OBJ that exists around the vehicle.

[0198] The gating camera 20 allows multiple range RNGs １ ~RNG Ｎ A plurality of slice images IMGs corresponding to １ ~IMGsＮ The i-th slice image IMGs is generated. ｉ The corresponding range RNG ｉ Only objects contained within the image will be captured.

[0199] The calculation processing unit 40 calculates a plurality of ranges RNG obtained by the gating camera 20. １ ~RNG Ｎ A plurality of slice images IMGs corresponding to １ ~IMGs Ｎ The computing device 40 is configured to be able to identify the type of object based on the learned model generated by machine learning. The computing device 40 includes a classifier 42 implemented based on a trained model generated by machine learning. The computing device 40 may include multiple classifiers 42 optimized for each range. The algorithm of the classifier 42 is not particularly limited, but may be, for example, You Only Look Once (YOLO), Single Shot Multibox Detector (SSD), R-CNN (Region-based Convolutional Neural Network), Spatial Pyramid Pooling (SPPnet), Faster R-CNN, Deconvolution-SSD (DSSD), or Mask R-CNN, or may be an algorithm developed in the future.

[0200] The arithmetic processing device 40 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware). The arithmetic processing device 40 may be a combination of multiple processors. Alternatively, the arithmetic processing device 40 may be configured solely from hardware. The functions of the arithmetic processing device 40 and the functions of the image processing device 28 may be implemented in the same processor.

[0201] 28(a) and (b) are diagrams showing an automobile 300 equipped with a gating camera 20. Refer to Fig. 28(a). The automobile 300 is equipped with headlamps (lamp fixtures) 302L and 302R.

[0202] As shown in Figure 28(a), the lighting device 22 of the gating camera 20 may be built into at least one of the left and right headlamps 302L, 302R. The image sensor 24 may be attached to a part of the vehicle, for example, behind the rearview mirror. Alternatively, the image sensor 24 may be provided in the front grille or front bumper. The camera controller 26 may be provided in the vehicle cabin, in the engine compartment, or built into the headlamps 302L, 302R.

[0203] As shown in FIG. 28( b ), the image sensor 24 may be built into either the left or right headlamp 302L or 302R together with the lighting device 22 .

[0204] 29 is a block diagram showing a vehicle lamp 200 equipped with a sensing system 10. The vehicle lamp 200, together with a vehicle-side ECU 310, constitutes a lighting system 304. The vehicle lamp 200 is equipped with the lamp-side ECU 210 and a lamp unit 220. The lamp unit 220 is for low beam or high beam, and is equipped with a light source 222, a lighting circuit 224, and an optical system 226. The vehicle lamp 200 is further provided with a sensing system 10.

[0205] Information about the object OBJ detected by the sensing system 10 may be used for light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 210 generates an appropriate light distribution pattern based on information about the type and position of the object OBJ generated by the sensing system 10. The lighting circuit 224 and the optical system 226 operate to obtain the light distribution pattern generated by the lamp-side ECU 210. The arithmetic processing device 40 of the sensing system 10 may be provided outside the vehicle lamp 200, i.e., on the vehicle side.

[0206] Information about the object OBJ detected by the sensing system 10 may be transmitted to the vehicle-side ECU 310. The vehicle-side ECU 310 may use this information for automatic driving or driving assistance.

[0207] The embodiments merely illustrate one aspect of the principles and applications of the present invention, and many modifications and changes in arrangement are permitted to the embodiments as long as they do not deviate from the spirit of the present invention as defined in the claims.

[0208] The present invention relates to a gating camera.

[0209] S1...light emission timing signal, S2...exposure timing signal, S3...shooting range control signal, S4...light distribution control signal, 10...sensing system, 20A...gating camera, 20B...gating camera, 20C...gating camera, 22...lighting device, 24...image sensor, 26...camera controller, 40...arithmetic processing device, 42...classifier, 60...semiconductor light-emitting element, 62...optical system, 64...lighting circuit, 200...vehicle lamp, 210...lamp-side ECU, 220...lamp unit, 222...light source, 224...lighting circuit, 226...optical system, 300...automobile, 302...headlamp, 304...lighting system, 310...vehicle-side ECU.

Claims

1. an illumination device that can control the light emission timing and irradiates the probe light; an image sensor capable of controlling exposure timing; a controller that divides a field of view into N ranges (N≧2) in a depth direction and controls the light emission timing of the illumination device and the image capture timing of the image sensor for each range; an image processing device that synthesizes N slice images output by the image sensor corresponding to the N ranges to generate a composite image; Equipped with N ranges, N different colors C １ ~C Ｎ is assigned, Each pixel of the composite image is １ ~C Ｎ are the pixel values ​​v of the corresponding pixels of the N slice images, respectively. １ ~v Ｎ A gating camera having pixel values ​​weighted and added with coefficients based on the above.

2. In the RGB color system, the i-th color (R ｉ , G ｉ , B ｉ ) and the (i+1)th color (R ｉ＋１ , G ｉ＋１ , B ｉ＋１ ) has a relationship in which the value of one element among R, G, and B is different and the values ​​of the remaining two elements are the same, The pixel values ​​(R, G, B) of each pixel of the composite image are (S,D,D)6Σ ｊ＝１：Ｎ s ｊ ・(S ｊ 19 ｊ 2. The ｊ 6 2. The gating camera according to claim 1, wherein the gating camera is expressed by:

3. The i-th color is expressed as (x ｉ , y ｉ ) is defined as The pixel value (Y, x, y) of each pixel of the composite image in the Yxy color system is (Y, x, y) = (S ｊ＝１：Ｎ (v) ｊ ),S ｊ＝１：Ｎ (v) ｊ ・+ ｊ ) / S ｊ＝１：Ｎ v ｊ ,S ｊ＝１：Ｎ (v) ｊ ・y ｊ ) / S ｊ＝１：Ｎ v ｊ ) 2. The gating camera according to claim 1, wherein the gating camera is expressed by:

4. The N colors are defined so that H is different in the HSV color system, The pixel value (H, S, V) of each pixel of the composite image is S ０ is a constant, (H,S,V)=(Σ ｊ＝１：Ｎ (v ｊ ・H ｊ ) / Σ ｊ＝１：Ｎ v ｊ ,S ０ ,Σ ｊ＝１：Ｎ (v ｊ )) 2. The gating camera according to claim 1, wherein the gating camera is expressed by:

5. 5. The gating camera according to claim 1, wherein the image sensor is a monochrome sensor.

6. A sensing system for a vehicle, comprising the gating camera according to any one of claims 1 to 4.

7. A vehicle lamp comprising the gating camera according to any one of claims 1 to 4.

8. An image processing device used together with a gating camera that divides a field of view into N ranges in a depth direction and outputs N slice images corresponding to the N ranges, The image processing device is capable of synthesizing the N slice images to generate a synthesized image, N ranges, N different colors C １ ~C Ｎ is assigned, Each pixel of the composite image is １ ~C Ｎ are the pixel values ​​v of the corresponding pixels of the N slice images, respectively. １ ~v Ｎ 13. An image processing apparatus comprising: an image processing device having pixel values ​​weighted and added with coefficients based on:

9. In the RGB color system, the i-th color (R ｉ , G ｉ , B ｉ ) and the (i+1)th color (R ｉ＋１ , G ｉ＋１ , B ｉ＋１ ) has a relationship in which the value of one element among R, G, and B is different and the values ​​of the remaining two elements are the same, The pixel values ​​(R, G, B) of each pixel of the composite image are (S,D,D)6Σ ｊ＝１：Ｎ s ｊ ・(S ｊ 19 ｊ 2. The ｊ 6 9. The image processing apparatus according to claim 8, wherein the image processing apparatus is represented by the formula:

10. The i-th color is expressed as (x ｉ , y ｉ ) is defined as The pixel value (Y, x, y) of each pixel of the composite image is (Y, x, y) = (S ｊ＝１：Ｎ (v) ｊ ),S ｊ＝１：Ｎ (v) ｊ ・+ ｊ ) / S ｊ＝１：Ｎ v ｊ ,S ｊ＝１：Ｎ (v) ｊ ・y ｊ ) / S ｊ＝１：Ｎ v ｊ ) 9. The image processing apparatus according to claim 8, wherein the image processing apparatus is represented by the formula:

11. The N colors are defined so that H is different in the HSV color system, The pixel value (H, S, V) of each pixel of the composite image is S ０ With is a constant, (H,S,V)=(Σ ｊ＝１：Ｎ (v ｊ ・H ｊ ) / Σ ｊ＝１：Ｎ v ｊ ,S ０ ,Σ ｊ＝１：Ｎ (v ｊ )) 9. The image processing apparatus according to claim 8, wherein the image processing apparatus is represented by the formula:

12. 1. A method for generating a composite image, comprising: A step of dividing the field of view into N ranges in a depth direction and generating N slice images corresponding to the N ranges; At the boundaries of N ranges, different colors C １ ~C Ｎ and assigning The N colors C １ ~C Ｎ are the pixel values ​​v of the corresponding pixels of the N slice images, respectively. １ ~v Ｎ generating the composite image by weighting and adding the weighted sum with a coefficient based on A generating method comprising: