In-vehicle sensing systems and gating cameras

Abstract

A gating camera (20G) comprises an illumination device (22), an image sensor (24), a camera controller (26G), and an arithmetic processing device (28G). The camera controller (26G) controls the timing of light emission by the illumination device (22) and the timing of light exposure by the image sensor (24). The arithmetic processing device (28G) processes image data (IMGf) generated by the image sensor (24). The arithmetic processing device (28G) obtains information about the weather on the basis of the image data (IMGf) obtained for a predetermined range in a weather detection mode.

Description

[Technical field]

[0001] The present disclosure relates to a sensing system for a vehicle. [Background technology]

[0002] For driving assistance and autonomous driving, object identification systems are used to sense the positions and types of objects around the vehicle. Object identification systems include sensors and a processing unit that analyzes the sensor output. Sensors are selected from cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter wave radar, ultrasonic sonar, active sensors, etc., taking into account the application, required accuracy, and cost.

[0003] As an active sensor to replace the TOF camera, a gating camera is A gated camera has been proposed (Patent Documents 1 and 2). A gating camera divides the shooting range into multiple ranges, and captures images by changing the exposure timing and exposure time for each range. This allows slice images to be obtained for each target range, and each slice image contains only objects that are included in the corresponding range. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-257983 A [Patent Document 2] International Publication WO2017 / 110413A1 Summary of the Invention [Problem to be solved by the invention]

[0005] Rain, snow, fog, etc. while driving are important because they greatly affect visibility and the performance of various sensors. Conventionally, it was necessary to use dedicated sensors such as raindrop sensors and fog sensors to detect weather, which was a factor in increasing costs.

[0006] The present disclosure has been made in this context, and one exemplary purpose of an embodiment thereof is to provide a gating camera capable of acquiring weather-related information. [Means for solving the problem]

[0007] An aspect of the present disclosure relates to a gating camera that divides a field of view into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates a field of view with pulsed illumination light, an image sensor, a camera controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor, and a processing unit that processes image data generated by the image sensor. The camera controller captures images in a predetermined range in a weather detection mode, and the processing unit acquires information about the weather based on image data obtained in a wide range.

[0008] Another aspect of the present disclosure is also a gating camera. This gating camera includes an illumination device that irradiates a field of view with pulsed illumination light, an image sensor, a camera controller that controls the timing of light emission of the illumination device and the timing of exposure of the image sensor, and a processing unit that processes image data generated by the image sensor. In a weather detection mode, the camera controller captures images in a narrow range and a wide range that is an extension of the narrow range in the forward direction, and the processing unit acquires information related to the weather based on first image data obtained for the narrow range and second image data obtained for the wide range. Effect of the Invention

[0009] According to the present disclosure, weather information can be obtained by a gating camera. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a sensing system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating the operation of a gating camera. [Diagram 3] 3(a) and (b) are diagrams illustrating images obtained by the gating camera. [Figure 4] 4(a) to (c) are diagrams illustrating the advantages of the gating camera in bad weather. [Diagram 5] 4 is a time chart illustrating the operation of the sensing system. [Figure 6] FIG. 4 is a diagram for explaining the operation of the gating camera according to the first embodiment. [Figure 7] 10A to 10C are diagrams illustrating the operation of the gating camera according to the second embodiment. [Figure 8] FIG. 11 is a diagram for explaining an imaging mode according to the third embodiment. [Figure 9] FIG. 11 is a block diagram of a sensing system according to a second embodiment. [Figure 10] FIG. 11 is a block diagram of a sensing system according to a third embodiment. [Figure 11] FIG. 11 is a block diagram of a sensing system according to a fourth embodiment. [Figure 12] FIG. 13 is a block diagram of a sensing system according to a fifth embodiment. [Figure 13] FIG. 13 is a block diagram of a sensing system according to a sixth embodiment. [Figure 14] 14(a) and (b) are diagrams showing an automobile equipped with a sensing system according to an embodiment. [Figure 15] 1 is a block diagram showing a vehicle lamp according to an embodiment of the present invention; [Figure 16] FIG. 13 is a block diagram of a sensing system equipped with a gating camera according to a seventh embodiment. [Figure 17]17(a) and (b) are diagrams showing slice images captured by a gating camera. [Figure 18] FIG. 18(a) is a diagram showing a histogram of the slice image in FIG. 17(a), and FIG. 18(b) is a diagram showing a histogram of the slice image in FIG. 17(b). [Figure 19] FIG. 19(a) is image data of the space above the slice image in FIG. 17(b), and FIG. 19(b) is a diagram showing a histogram of the image data in FIG. 19(a). [Figure 20] FIG. 11 is a diagram illustrating a shooting range in a weather detection mode. [Figure 21] 21(a) to (c) are diagrams for explaining generation of a difference image in the third embodiment. [Figure 22] 22(a) and (b) are diagrams for explaining weather estimation based on a difference image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A summary of some exemplary embodiments of the present disclosure will be described. This summary is intended to provide a simplified summary of some concepts of one or more embodiments in order to provide a basic understanding of the embodiments as a prelude to the detailed description that follows, and is not intended to limit the scope of the invention or disclosure. Furthermore, this summary is not intended to be a comprehensive overview of all possible embodiments, nor is it intended to limit essential components of the embodiments. For convenience, the term "one embodiment" may be used to refer to one embodiment (example or variant) or multiple embodiments (examples or variants) disclosed in this specification.

[0012] A gating camera according to an embodiment divides a field of view into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates the field of view with pulsed illumination light, an image sensor, a camera controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor, and a processor that processes image data generated by the image sensor. The processor acquires information about the weather based on image data obtained for a predetermined range in a weather detection mode.

[0013] Fog, rain, and snow exist randomly, in other words uniformly, in the plane of the field of view and in the depth direction. On the other hand, targets other than fog, rain, and snow (vehicles, pedestrians, traffic signs, delineators, etc.) exist locally and non-randomly. In other words, in bad weather, the slice images obtained for a specified range will show light diffusely reflected by the fog, rain, and snow that are uniformly distributed, which shows properties similar to random noise. By analyzing the slice images based on this property, information about the weather can be obtained based on the presence or absence or excess of noise components. The "information about the weather" may be the presence or absence and type of fog, rain, and snow, or the amount (thickness) of them.

[0014] The camera controller may capture a wide range in the weather detection mode, which is deeper than the normal capture mode, as the predetermined range. By capturing an image over a wider range than normal so that an area close to the gating camera is captured, slice images that are significantly affected by fog, rain, and snow can be generated, thereby improving detection accuracy.

[0015] The processor may obtain weather-related information based on a predetermined area included in the image data obtained over a wide range. In bad weather, the detection accuracy can be improved by defining the predetermined area so that only fog, rain, and snow are included.

[0016] The predetermined region may be located above the image data. By selecting a sky portion as the predetermined region, the weather can be estimated without being affected by objects such as vehicles, pedestrians, traffic signs, delineators, etc.

[0017] The arithmetic processing device may calculate a histogram of a predetermined area and obtain weather information based on the histogram. When fog, rain, snow, etc. are photographed, the histogram shows a predetermined tendency based on the type and amount of fog, rain, snow, etc. Therefore, by generating a histogram and analyzing it, the presence and amount of fog, rain, snow, etc. can be estimated.

[0018] The processor may obtain weather-related information based on the shape of the histogram. When fog, rain, snow, etc. are present, the histogram shows a shape close to a normal distribution.

[0019] The calculation processing device may obtain information about the weather based on at least one of the average and the variance of the histogram.

[0020] A gating camera according to one embodiment includes an illumination device that irradiates a field of view with pulsed illumination light, an image sensor, a camera controller that controls the timing of light emission of the illumination device and the timing of exposure of the image sensor, and a processing device that processes image data generated by the image sensor. In a weather detection mode, the camera controller captures images in a narrow range and a wide range that is an extension of the narrow range in the forward direction, and the processing device acquires information related to the weather based on first image data obtained for the narrow range and second image data obtained for the wide range.

[0021] For example, by using image processing to remove the effects of targets that exist in a narrow range, it is possible to extract the effects of fog, rain, or snow that exist in the difference range, thereby obtaining information about the weather.

[0022] The arithmetic processing device may calculate histograms of the first image data and the second image data, and obtain weather information based on the two histograms. For example, consider a case where a target exists in the narrow range. In this case, the histogram of the first image data obtained for the narrow range deviates from a normal distribution. In good weather, the histogram of the second image data obtained for the wide range also deviates from a normal distribution. In bad weather, the second image data obtained for the wide range is superimposed with noise due to fog, rain, and snow, so that the histogram approaches a normal distribution.

[0023] The processor may calculate the mean and variance of each of the two histograms and obtain information about the weather based on the mean and variance. For example, it may determine that the weather is bad when the difference between the means and / or the difference between the variances exceeds a predetermined threshold.

[0024] The calculation processing device may obtain information about the weather based on the shapes of the two histograms.

[0025] The arithmetic processing device may generate a difference image representing the difference between the first image data and the second image data, and obtain information about the weather based on the difference image. The difference image obtained in bad weather will contain predominantly noise components due to fog, rain, and snow. Therefore, information about the weather can be obtained based on the noise components contained in the difference image.

[0026] A difference image may be generated for a range in which objects present in a narrow range are captured.

[0027] The processor may generate a histogram of the difference image and obtain information about the weather based on the histogram.

[0028] (Embodiment) Hereinafter, preferred embodiments will be described with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing will be given the same reference numerals, and duplicated descriptions will be omitted as appropriate. In addition, the embodiments are illustrative and do not limit the disclosure and invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the disclosure and invention.

[0029] (Embodiment 1) 1 is a block diagram of a sensing system 10A according to embodiment 1. This sensing system 10A is mounted on a vehicle such as an automobile or motorcycle for the purpose of driving assistance or automatic driving, and detects an object OBJ present around the vehicle.

[0030] The sensing system 10A includes a main sensor group 50, a main controller 60, and a gating camera 20. The main sensor group 50 may include one or more sensors. For example, the main sensor group 50 includes a camera 52 and a millimeter wave radar 54. Alternatively, the main sensor group 50 may include a stereo camera. Alternatively, the main sensor group 50 may include a LiDAR or the like.

[0031] The main controller 60 detects the positions and types of objects around the vehicle based on the output of the main sensor group 50, and outputs the detection result RESULT. For example, the main controller 60 may be equipped with a discriminator (classifier), and the detection result RESULT may include information on the type (category, class) and position of the target object.

[0032] The gating camera 20 has a field of view divided into a plurality of ranges RNG1 to RNG2 in the depth direction. N Divide into multiple ranges RNG1~RNG N Multiple slice images IMGs1 to IMGs N Adjacent ranges may overlap in depth at their boundaries.

[0033] The gating camera 20 includes an illumination device 22, an image sensor 24, a camera controller 26, and a processing unit 28.

[0034] The lighting device (floodlight) 22 irradiates the field of view ahead of the vehicle with pulsed illumination light L1 in synchronization with a light emission timing signal S1 provided by the camera controller 26. The pulsed illumination light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength. For example, a laser diode (LD) or an LED may be used for the lighting device 22. In a system in which the gating camera 20 is used only at night, the wavelength of the pulsed illumination light L1 may be near infrared around 800 nm. In a system in which the gating camera 20 is used day and night, it is preferable that the pulsed illumination light L1 be in a wavelength range longer than 1 μm.

[0035] The image sensor 24 includes a plurality of pixels, and is capable of controlling exposure in synchronization with an exposure timing signal S2 provided by the camera controller 26, thereby generating a slice image IMGr consisting of a plurality of pixels. The image sensor 24 is sensitive to the same wavelength as the pulsed illumination light L1, and captures the reflected light (return light) L2 reflected by the object OBJ. The i-th range RNG i Regarding the above, the slice image IMGr generated by the image sensor 24 is referred to as a raw image or a primary image as necessary, and is distinguished from the slice image IMGs which is the final output of the gating camera 20. Furthermore, the raw image IMGr and the slice image IMGs are also collectively referred to simply as slice image IMG.

[0036] The camera controller 26 changes the light emission timing signal S1 and the exposure timing signal S2 for each range RNG to change the time difference between the light emission by the lighting device 22 and the exposure of the image sensor 24. The light emission timing signal S1 specifies the timing to start light emission and the light emission time. The exposure timing signal S2 specifies the timing to start exposure (the time difference with light emission) and the exposure time.

[0037] The arithmetic processing device 28 can be implemented by combining a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), microcomputer, or GPU (Graphics Processing Unit) with a software program executed by the processor (hardware). The arithmetic processing device 28 may be configured only with hardware. The arithmetic processing device 28 processes raw image data IMGr generated by the image sensor 24, and outputs the final slice image IMGs. Note that when the output IMGr of the image sensor 24 is used as the slice image IMGs as it is, the arithmetic processing device 28 can be omitted.

[0038] FIG. 2 is a diagram for explaining the operation of the gating camera 20. In FIG. 2, the i-th range RNG i The figure shows how the range RNG is measured from the gating camera 20. The lighting device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. i The distance to the boundary in front of MINi、 Range RNG i The distance to the inner boundary of MAXi Let us assume that.

[0039] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi =2×d MINi / c where c is the speed of light.

[0040] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, T MAXi =2×d MAXi / c It is.

[0041] Range RNG i When it is desired to photograph only the object OBJ included in the time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.

[0042] i-th range RNG i When capturing an image, multiple exposures may be performed. In this case, the camera controller 26 may repeat the above-mentioned set of irradiation and exposure operations multiple times at a predetermined period τ2. At this time, the raw image data IMGr generated by the image sensor 24 is the integrated result of multiple exposures.

[0043] In this embodiment, in order to prevent the exposure (brightness value of the object image in the slice image) from varying for each range, the gating camera 20 optimizes the shutter speed (exposure time), number of exposures, sensitivity, irradiation intensity of the pulsed illumination light, etc. (shooting parameters) for each range.

[0044] 3(a) and (b) are diagrams for explaining images obtained by the gating camera 20. In the example of FIG. 3(a), an object (pedestrian) OBJ2 exists in range RNG2, and an object (vehicle) OBJ3 exists in range RNG3. FIG. 3(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 3(a). When capturing slice image IMG1, the image sensor 24 is exposed only by reflected light from range RNG1, so no object image is captured in slice image IMG1.

[0045] When the slice image IMG2 is captured, the image sensor 24 is exposed only to the light reflected from the range RNG2, so that only the object image OBJ2 appears in the slice image IMG2. Similarly, when the slice image IMG3 is captured, the image sensor 24 is exposed only to the light reflected from the range RNG3, so that only the object image OBJ3 appears in the slice image IMG3. In this way, the gating camera 20 can capture the object separately for each range.

[0046] The gating camera 20 is advantageous in photographing in bad weather. The reason will be explained below. Figures 4(a) to 4(c) are diagrams for explaining the advantages of the gating camera 20 in bad weather. Figure 4(a) shows an example of a driving scene in bad weather. An object (vehicle) OBJ3 exists in the range RNG3. The dots shown in the figure are schematic representations of obstacles such as raindrops, snow, or fog. Figure 4(b) shows a slice image IMG3 of the third range obtained in the situation of Figure 4(a). When photographing the slice image IMG3, the image sensor 24 is exposed only by the reflected light from the range RNG3, so that the obstacles (raindrops, snow, fog) of the ranges RNG1 and RNG2 are not reflected in the slice image IMG3. In other words, the rain, snow, and fog included in the ranges other than the range to be measured can be removed.

[0047] Figure 4(c) shows an image of the same field of view captured by a general camera. When captured by a general camera, the reflected light of all objects in range RNG3 is captured, and many obstacles are captured in the image, blocking object OBJ3.

[0048] A comparison of FIGS. 4(b) and (c) reveals that the slice image IMG generated by the gating camera 20 contains more information than a general camera in bad weather.

[0049] Returning to Fig. 1, in this embodiment, the gating camera 20 is utilized as an auxiliary sensor that assists the main sensor group 50. Therefore, the gating camera 20 does not operate all the time, but adaptively selects an operating state (enabled state EN) or a stopped state (disabled state DIS) according to the driving environment.

[0050] When the gating camera 20 is in an operating state, the camera controller 26 generates a light emission timing signal S1 and an exposure timing signal S2, thereby generating slice images of a plurality of ranges. When the gating camera 20 is in a stopped state, the camera controller 26 does not generate the light emission timing signal S1 and the exposure timing signal S2, and therefore no slice images are generated.

[0051] In bad weather, the slice image IMG generated by the gating camera 20 is supplied to the main controller 60. Then, the output of the gating camera 20 is used for controlling driving assistance or automatic driving.

[0052] The above is the configuration of the sensing system 10A. Next, the operation of the sensing system 10A will be described with reference to FIG. 5, which is a time chart illustrating the operation of the sensing system 10A.

[0053] Under good weather (visibility) conditions, the reliability of the main sensor group 50 is high. In this case, the gating camera 20 is stopped, and the main controller 60 detects targets based on the output of the main sensor group 50.

[0054] In poor weather (visibility) conditions, the reliability of the main sensor group 50 decreases. In this case, the gating camera 20 is in an operating state, and the main controller 60 detects targets based on the detection results of the gating camera 20 instead of or in addition to the output of the main sensor group 50.

[0055] The above is the operation of the sensing system 10 A. According to this sensing system 10 A, it is possible to suppress an increase in power consumption due to the gating camera 20, while suppressing a decrease in performance of the sensing system 10 A in bad weather.

[0056] Returning to Fig. 1, the gating camera 20 is configured to be able to switch between a plurality of shooting modes in an operating state. The plurality of shooting modes include a first shooting mode with relatively high performance and high power consumption, and a second shooting mode with relatively low performance and low power consumption.

[0057] Several examples of the multiple shooting modes will now be described.

[0058] Example 1 Each shooting mode may have a different frame rate. The "frame rate" is the frequency at which all ranges to be measured are measured, and is the reciprocal of the frame period. A shooting mode with a high frame rate enables high-speed sensing, but consumes more power. A shooting mode with a low frame rate reduces the sensing speed, but consumes less power.

[0059] 6 is a diagram for explaining the operation of the gating camera 20 according to the first embodiment. During the first shooting mode MODE1, the first frame period T F1 Then, shooting is performed in a plurality of ranges RNG1 to RNG3. In this example, the number of ranges is three. Note that light emission and exposure may be repeated multiple times during the shooting period of one range.

[0060] During the second shooting mode MODE2, the second frame period T F2 (T F2 >T F1 ) during shooting in a plurality of ranges RNG1 to RNG3. As the frame rate decreases, the power consumption in the second shooting mode MODE2 becomes lower than the power consumption in the first shooting mode MODE1.

[0061] Even in a situation where the gating camera 20 is required, such as in bad weather, the highest performance may not be required depending on the situation. For example, even in bad weather, if the accuracy (reliability) of sensing based on the output of the main sensor group 50 is relatively high, there is no problem with sensing even if the frame rate of the gating camera 20 is reduced. Therefore, in such a case, power consumption can be reduced by selecting the second shooting mode MODE2.

[0062] For example, the shooting mode may be selected according to the amount of rain or snow, the thickness of fog, etc. For example, when there is a relatively large amount of rain or snow or when the fog is relatively thick, the first shooting mode MODE1 may be selected, and when there is a relatively small amount of rain or snow or when the fog is relatively thin, the second shooting mode MODE2 may be selected.

[0063] Alternatively, the mode may be selected depending on the cause of the current bad weather (whether it is rain, snow, or fog). For example, when there is fog or snowfall, the first shooting mode may be selected, and when it is raining, the second shooting mode MODE2 may be selected.

[0064] Alternatively, the first shooting mode MODE1 and the second shooting mode MODE2 may be switched depending on the driving scene. For example, the first shooting mode MODE1 may be selected when the driving speed is relatively high, and the second shooting mode MODE2 may be selected when the driving speed is relatively low.

[0065] Example 2 In the second embodiment, the number of ranges to be measured differs for each shooting mode. For example, in the first shooting mode MODE1, three ranges RNG1 to RNG3 are measured, and in the second shooting mode MODE2, two ranges RNG1 to RNG2 are measured.

[0066] 7 is a diagram for explaining the operation of the gating camera 20 according to the second embodiment. For example, assume that the first range RNG1 is 0 to 25 m, the second range RNG2 is 25 to 50 m, and the third range RNG3 is 50 to 100 m. Depending on the weather, a situation may occur in which sensing beyond 50 m is not possible even with the performance of the gating camera 20. In such a case, power consumption can be reduced by selecting the second shooting mode MODE2.

[0067] In addition, the visibility required for automatic driving control or driving assistance control depends on the vehicle speed. Specifically, the higher the speed, the more distant the target information is required. Therefore, when the vehicle speed is higher than a certain threshold, the first shooting mode MODE1 may be selected, and when the vehicle speed is lower than the threshold, the second shooting mode MODE2 may be selected.

[0068] In FIG. 7, the frame rate of the second shooting mode MODE2 is lower than the frame rate of the first shooting mode MODE1, but the frame rates may be the same.

[0069] The distance to the farthest boundary of the farthest range in the first shooting mode MODE1 (farthest shooting distance) is d MAX3 The farthest object distance in the second shooting mode MODE2 is d MAX 2, which differs between the first shooting mode MODE1 and the second shooting mode MODE2.

[0070] In such a case, the intensity of the pulsed illumination light in the second imaging mode MODE2 may be lower than that in the first imaging mode MODE1, as shown in Fig. 7. Note that the intensity of the pulsed illumination light in Fig. 7 is not intended to indicate continuous emission. In the second imaging mode MODE2, since it is not necessary to project the pulsed illumination light to a long distance, the intensity can be reduced. In addition, the amount of heat generated by the illumination device 22 can be reduced.

[0071] Example 3 In the third embodiment, the depths of the multiple ranges are different for each shooting mode. Fig. 8 is a diagram for explaining the shooting modes according to the third embodiment. For example, in the first shooting mode MODE1, the first range RNG1 is 0 to 25 m, the second range RNG2 is 25 to 50 m, and the third range RNG3 is 50 to 100 m, whereas in the second shooting mode MODE2, the first range RNG1 is 0 to 25 m, the second range RNG2 is 25 to 50 m, and the third range RNG3 is 50 to 75 m. In other words, the longest shooting distance in the first shooting mode MODE1 is 100 m, whereas the longest shooting distance in the second shooting mode MODE2 is shorter at 75 m.

[0072] In the third embodiment, as in the second embodiment, since it is not necessary to project the pulsed illumination light to a long distance in the second imaging mode MODE2, the intensity can be reduced. Also, the amount of heat generated by the illumination device 22 can be reduced.

[0073] The selection of the shooting mode in the third embodiment may be based on the weather and visibility, as in the second embodiment, or may be based on the vehicle speed.

[0074] Example 4 In the fourth embodiment, the gating camera 20 can be switched to a standby mode in addition to a plurality of shooting modes. In the standby mode, the gating camera 20 is ready to shoot at any time, but does not shoot spontaneously and waits. In the standby mode, the camera controller 26 can communicate with the main controller 60, and immediately shoots when it receives a shooting instruction command from the main controller 60. By shooting only when instructed by the main controller 60, unnecessary power consumption can be reduced.

[0075] Next, control of the operating state / stopped state of the gating camera 20 and control of the shooting mode will be described.

[0076] In this embodiment, the operating / stopped state of the gating camera 20 and the shooting mode are controlled autonomously by the gating camera 20 itself.

[0077] (Control method 1) In control method 1, the gating camera 20 switches between an operating state and a stopped state and selects an imaging mode based on a slice image IMG that it generates.

[0078] The slice image IMG captures a subject that is within a specific range in the depth direction of the field of view. By utilizing this property, it is possible to estimate the quality of visibility and the weather, and to use this information to control the gating camera. Weather estimation and visibility estimation based on the slice image IMG will be described later.

[0079] Based on the slice image IMG, the camera controller 26 determines whether to operate or stop the gating camera 20. The camera controller 26 switches the gating camera 20 to an operating state in a situation in which a decrease in the reliability of the main sensor group 50 is estimated.

[0080] Furthermore, the camera controller 26 selects the first imaging mode MODE1 or the second imaging mode MODE2 based on the slice image IMG. The selection conditions may be determined based on the first to fourth embodiments related to the imaging modes.

[0081] In this control method 1, when gating camera 20 is in a stopped state, it intermittently (for example, at intervals of several tens of seconds to several minutes) performs shooting for controlling the shooting mode and operation / stop.

[0082] (Control method 2) In control method 2, the camera controller 26 of the gating camera 20 switches between the operating state and the stopped state based on at least one vehicle signal. Examples of the vehicle signal include the output of the rain sensor, the output of the fog sensor, vehicle speed information, a fail signal indicating a malfunction of the main sensor, a fail signal indicating a malfunction of the main controller or a decrease in recognition accuracy, a wiper control signal, a fog lamp control signal, etc. When the camera controller 26 determines that the weather is bad based on the vehicle signal, it sets the gating camera 20 to the operating state and starts capturing images.

[0083] The camera controller 26 executes the imaging mode based on the slice image IMG, similarly to the control method 1.

[0084] In control method 2, unlike control method 1, image capture for operation / stop control is not required while the camera is stopped.

[0085] (Control method 3) In control method 3, the camera controller 26 performs both the switching between the operating state / stopped state and the control of the shooting mode based on at least one vehicle signal.

[0086] (Control method 4) In the control method 4, the camera controller 26 switches between the moving state and the stopped state based on the slice images IMG, and controls the shooting mode based on at least one vehicle signal.

[0087] The control of the operating state / stopped state and the control of the imaging mode may be performed based on both the vehicle signal and the analysis result of the slice image.

[0088] (Embodiment 2) 9 is a block diagram of a sensing system 10B according to embodiment 2. In embodiment 2, the operating state / stopped state of the gating camera 20 is controlled based on an external enable signal EN, and the imaging mode of the gating camera 20 is autonomously controlled by the gating camera 20 itself.

[0089] 9, for example, the operating state / stopped state of the gating camera 20 is controlled by the main controller 60. The main controller 60 may determine the operating state / stopped state of the gating camera 20 based on at least one piece of vehicle information and generate an enable signal EN, as described in the first embodiment. The main controller 60 may also control the operating state / stopped state of the gating camera 20 based on a slice image IMG generated by the gating camera 20.

[0090] Alternatively, the main controller 60 may control the operating / stopping state of the gating camera 20 based on both the vehicle signal and the slice image.

[0091] On the other hand, in the second embodiment, the camera controller 26 may select the imaging mode of the gating camera 20 based on the slice image IMG. Alternatively, the camera controller 26 may select the imaging mode of the gating camera 20 based on a vehicle signal. Alternatively, the camera controller 26 may select the imaging mode of the gating camera 20 based on both the vehicle signal and the slice image.

[0092] (Embodiment 3) 10 is a block diagram of a sensing system 10C according to embodiment 3. In embodiment 3, a gating camera 20 is mounted on a vehicle lamp 200. The vehicle lamp 200 includes a lamp ECU (Electronic Control Unit) 210 that comprehensively controls a low beam and a high beam (not shown).

[0093] The operating state / stopped state of the gating camera 20 is controlled based on an enable signal EN generated by the lamp ECU 210, and the imaging mode of the gating camera 20 is autonomously controlled by the gating camera 20 itself. The lamp ECU 210 controls the operating state / stopped state of the gating camera 20 based on at least one of a vehicle signal and a slice image.

[0094] (Embodiment 4) 11 is a block diagram of a sensing system 10D according to embodiment 4. In embodiment 4, the gating camera 20 basically operates constantly while the vehicle is running, and only the shooting mode can be switched.

[0095] The gating camera 20 (camera controller 26) selects a shooting mode based on at least one of the slice image IMG and the vehicle signal.

[0096] (Embodiment 5) 12 is a block diagram of a sensing system 10E according to embodiment 5. In embodiment 5, the gating camera 20 does not support switching between a plurality of shooting modes, and is only capable of switching between an operating state (enabled state) and a stopped state (disabled state).

[0097] The gating camera 20 (camera controller 26) switches the gating camera 20 between an operating state and a stopped state based on at least one of the slice image IMG and a vehicle signal.

[0098] (Embodiment 6) 13 is a block diagram of a sensing system 10F according to embodiment 6. In embodiment 6, a part of the main sensor group 50 (the camera 52 in this example) or the whole is omitted.

[0099] As described in the first to fifth embodiments, the gating camera 20 can control both or either of switching between the operating state and the stopped state and switching between the shooting modes. The switching between the operating state and the stopped state and switching between the shooting modes are as described in the first to fifth embodiments.

[0100] 14(a) and (b) are diagrams showing an automobile 300 equipped with the sensing system 10 according to the embodiment. Referring to Fig. 14(a), the automobile 300 is equipped with headlamps (lamp fixtures) 302L and 302R.

[0101] The camera 52 and millimeter wave radar 54 of the main sensor group 50 are disposed in locations suitable for sensing the vehicle. For example, the camera 52 is provided behind the rearview mirror, and the millimeter wave radar 54 is disposed in the front of the vehicle. The main controller 60 is disposed inside the vehicle cabin or in the engine room.

[0102] The lighting device 22 of the gating camera 20 is built into at least one of the left and right headlamps 302L, 302R. The image sensor 24 can be attached to a part of the vehicle, for example, behind the rearview mirror. Alternatively, the image sensor 24 may be provided in the front grille or front bumper. The camera controller 26 may be provided in the vehicle cabin, in the engine compartment, or built into the headlamp.

[0103] Please refer to Fig. 14(b). As shown in Fig. 14(b), the image sensor 24 may be built into either the left or right headlamp 302L, 302R.

[0104] Fig. 15 is a block diagram showing a vehicle lamp 200 according to an embodiment. The vehicle lamp 200 corresponds to the headlamp 302 in Fig. 14(b), and includes a low beam unit 202, a high beam unit 204, a lamp ECU 210, and a gating camera 20.

[0105] The lamp ECU 210 controls the on / off or light distribution of the low beam unit 202 and the high beam unit 204 based on a control command from the vehicle-side ECU 310. In addition, in some embodiments and examples, the lamp ECU 210 may switch the gating camera 20 between an operating state and a stopped state.

[0106] The gating camera 20 is built into the housing of the vehicle lamp 200. At least one of the image sensor 24, the camera controller 26, and the arithmetic processing device 28 may be provided outside the housing of the vehicle lamp 200.

[0107] (Visibility estimation and weather estimation based on slice images) The gating camera can freely set a depth range of interest and capture only objects that are included in the range of interest. Using this characteristic, the gating camera 20 itself can obtain weather information based on slice images IMG that it has captured.

[0108] 16 is a block diagram of a sensing system 10G including a gating camera 20G according to embodiment 7. The gating camera 20G includes an illumination device 22, an image sensor 24, a camera controller 26G, and a processor 28G.

[0109] The camera controller 26G can select a weather detection mode in addition to a normal shooting mode. In the weather detection mode, shooting is performed for a predetermined range determined for weather detection. The calculation processing device 28G acquires information about the weather, INFO_FCST, based on image data (slice image) IMGf obtained for the predetermined range in the weather detection mode.

[0110] Fog, rain, and snow exist randomly, in other words, uniformly, within the plane of the field of view and in the depth direction. On the other hand, targets other than fog, rain, and snow (vehicles, pedestrians, traffic signs, delineators, etc.) exist locally and non-randomly. In other words, in bad weather, the slice image obtained for a specified range RNGf will show light diffusely reflected by the fog, rain, and snow that are uniformly distributed, which shows properties similar to random noise. Based on this property, information about the weather can be obtained by analyzing the image data IMGf.

[0111] Specific examples of weather detection will be described below based on Examples 1 to 3.

[0112] Example 1 In the weather detection mode, the camera controller 26G performs shooting by setting the wide range RNGw, which is longer in the depth direction than the normal shooting mode, as the predetermined range RNGf. The wide range RNGw may be a range in the normal shooting mode expanded toward the front. For example, in the normal shooting mode, shooting is performed for four ranges RNG1 to RNG4, with RNG1 being 0 to 25 m, RNG2 being 25 to 50 m, RNG3 being 50 to 75 m, and RNG4 being 75 to 100 m. The wide range RNGw for weather detection can be a range obtained by expanding the second range RNG2 toward the front, and may be set to, for example, 10 to 50 m. To expand the range, the camera controller 26G lengthens the exposure time of the image sensor 24.

[0113] Figures 17(a) and (b) are diagrams showing slice images captured by a gating camera. Figure 17(a) is a slice image obtained in a certain range (25-100m) in normal shooting mode under bad weather conditions (in fog with visibility of about 50m based on infrared light energy). Figure 17(b) is a slice image IMGf obtained in the same situation for a wide range IMGw (6-100m) that is an expansion of the range of Figure 17(a) to the near side.

[0114] The slice image in FIG. 17(a) shows only the light reflected from the object in the back, whereas the slice image IMGf in FIG. 17(b) shows not only the light reflected from the object in the back, but also the light reflected from the fog (rain, snow) located in the range extended to the front, so that the object in the back is hidden by the fog, rain, and snow in the front. In other words, the slice image IMGf captured widely in the front contains a lot of information about the fog (rain, snow). By capturing such a slice image IMGf in the weather detection mode, it is possible to estimate the weather.

[0115] FIG. 18(a) is a diagram showing a histogram of the slice image of FIG. 17(a), and FIG. 18(b) is a diagram showing a histogram of the slice image of FIG. 17(b). The horizontal axis of each histogram represents the pixel value (tone value), and the vertical axis represents the frequency (occurrence frequency). As shown in FIG. 18(a), the histogram of the reflected light from the target at the back reflects the size and reflectance of the target, and as a result, it has a steep edge or multiple peaks, and deviates from the normal distribution. On the other hand, as shown in FIG. 18(b), in the slice image obtained for the wide range RNGw, the object at the back is hidden by the fog, rain, and snow in the foreground, so that the histogram of the slice image approaches the histogram of the image containing random noise. As a result, the histogram has a single peak, and the shoulder is gentle.

[0116] In this way, in bad weather, the histogram of the slice image IMGf obtained for a given range RNGf tends to approach that shown in Fig. 18(b). Conversely, in good weather with no fog, rain, or snow, the histogram of the slice image IMGf approaches that shown in Fig. 18(a). Therefore, the calculation processing device 28g can determine whether the weather is good or bad by acquiring the histogram of the slice image IMGf.

[0117] In the weather detection mode, it is not essential to increase the depth of the predetermined range RNGf, and the depth may be the same as that in the normal shooting mode.

[0118] The calculation processing device 28g may create a histogram for the entire slice image IMGf, but may also extract only a portion thereof (a predetermined region) to generate a histogram.

[0119] The detection accuracy can be improved by defining the predetermined area so that it contains only fog, rain, and snow in bad weather and does not contain any objects in good weather. Specifically, the predetermined area may be defined as the sky. In this case, the predetermined area should be located in the upper center of the image data.

[0120] Fig. 19(a) is image data of the space above slice image IMGf in Fig. 17(b), and Fig. 19(b) is a diagram showing a histogram of the image data in Fig. 19(a). In this way, when a histogram of the part corresponding to the sky is generated in bad weather, a histogram close to a normal distribution is obtained, whereas in good weather, since there is no reflection, the distribution of the histogram is concentrated in the low gradation region.

[0121] The arithmetic processing device 28g may generate a histogram of a part (or the whole) of the slice image IMGf, and may determine the weather based on the characteristics that appear in the histogram under bad weather conditions. For example, the arithmetic processing device 28g may determine the weather based on the shape of the histogram, and may determine the weather as bad weather when it is determined that the histogram is close to a normal distribution. A known algorithm known as a normality test may be used to determine whether the distribution is close to a normal distribution. Alternatively, in bad weather, as shown in FIG. 18(b), the shoulders of the histogram become gentle, and the variance σ becomes large. Therefore, when the variance σ exceeds a predetermined threshold, the weather may be determined as bad weather (thick fog, heavy rain or snow). Alternatively, the weather may be determined as bad weather (thick fog, heavy rain or snow) as the variance σ increases.

[0122] Example 2 In the first embodiment, the weather was estimated based on one slice image IMGf obtained for one specified range RNGf, whereas in the second embodiment, images are taken for two specified ranges RNGfx and RNGfy, and the weather is estimated based on two slice images IMGfx and IMGfx obtained for each range.

[0123] Specifically, one of the two predetermined ranges (also called narrow range) RNGfx is relatively narrow in the depth direction, and the other (also called wide range) RNGfy is relatively wide in the depth direction. The wide range RNGy may be in a relationship in which the narrow range RNGx is extended toward the front side. FIG. 20 is a diagram for explaining the shooting range in the weather detection mode. The dashed line indicates the field of view of the image sensor 24.

[0124] When narrow-range RNGfx and wide-range RNGy are photographed by a gating camera in bad weather, two slice images (called the first slice image and the second slice image) as shown in FIG. 18(a) and (b) are obtained.

[0125] The calculation processing device 28G can obtain information about the weather based on the first image data IMGfx obtained for the narrow range RNGfx and the second image data IMGfy obtained for the wide range RNGfy.

[0126] Specifically, the calculation processing device 28G calculates histograms for each of the first image data IMGfx and the second image data IMGfy, and obtains weather information based on the two histograms. These two histograms correspond to the two histograms shown in Figures 19(a) and 19(b).

[0127] The histogram obtained for the first image data IMGfx has a shape, variance, and average according to the targets included in the narrow range RNGfx. On the other hand, the histogram obtained for the second image data IMGfy approaches that of the first image data IMGfx in good weather. On the other hand, in bad weather, the histogram of the second image data IMGfy approaches that of an image containing a lot of random noise. Specifically, the histogram in bad weather has the following characteristics: (i) the shape is normal; The distribution exhibits the following characteristics: (i) it is close to a normal distribution; (ii) it has a gentle shoulder; and (iii) it has a large variance.

[0128] Therefore, the calculation processing device 28G may calculate the average of each of the two histograms, and when the difference between the averages exceeds a predetermined threshold value (Condition 1), determine that the weather is bad. Alternatively, the calculation processing device 28G may calculate the variance σ of each of the two histograms, and determine that the weather is bad when the difference between the variances exceeds a predetermined threshold value (condition 2).

[0129] The calculation processing device 28G may determine that the weather is bad when both conditions 1 and 2 are true, or may determine that the weather is bad when either one of them is true.

[0130] The calculation processing device 28G may determine whether or not the shapes of the two histograms are similar, and if they are not similar, determine that the weather is bad.

[0131] Example 3 In the third embodiment, slice images IMGfx and IMGfy are generated for the narrow range RNGfx and the wide range RNGfy, and weather-related information is acquired based on the two slice images IMGfx and IMGfy, similarly to the second embodiment. More specifically, the calculation processing device 28G generates a difference image IMGd representing the difference between the first image data IMGfx and the second image data IMGfy, and acquires weather-related information based on the difference image IMGd.

[0132] 21(a) to (c) are diagrams illustrating generation of a difference image in Example 3. The pixel value IMGd(i) of each pixel of the difference image IMGd is the difference between the pixel values ​​IMGfx(i) and IMGfy(i) of corresponding pixels in the first image data IMGfx and the second image data IMGfy. IMGd(i) = IMGfy(i) - IMGfx(i) IMG#(i) indicates the pixel value of the i-th pixel in image IMG#. In the difference image, most of the reflected light from objects in the background is canceled, and the image contains a lot of noise components (fog, rain, snow) from areas closer to the camera.

[0133] The arithmetic processing device 28G may estimate the weather based on the entire differential image IMGd, or may estimate the weather based on a part of it. As shown in FIG. 20, within the hatched part of the field of view FOV of the camera, no object present in the range RNGfx on the far side is captured. Therefore, the arithmetic processing device 28G may exclude the part of the second image data IMGfy that corresponds to the hatched field of view, and estimate the weather based on the remaining part (effective area). The effective area can be understood as the range in which an object present in the narrow range RMGfx is captured.

[0134] 22(a) and (b) are diagrams for explaining weather estimation based on a difference image IMGd. A rectangular area is shown in the difference image IMGd of FIG. 22(a). This rectangular area is provided in the above-mentioned effective area. FIG. 22(b) shows a histogram within the rectangular area of ​​FIG. 22(a). The calculation processing device 28G obtains information regarding the weather based on this histogram.

[0135] The calculation processing device 28G can estimate the weather based on the histogram in the same manner as described in the embodiment 1. Specifically, it determines whether the histogram is caused by noise or an object based on the shape of the histogram, the number of peaks, the slope of the shoulders, the variance, and the like.

[0136] The weather estimation described in the seventh embodiment can be used in combination with the techniques of the first to sixth embodiments to control the imaging mode of the gating camera 20 and to switch it on / off, but is not limited to this.

[0137] The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing step, and that such modifications are also within the scope of the present invention. Such modifications will be described below.

[0138] (Variation 1) In the embodiment, the slice image IMG is output from the gating camera 20 to the main controller 60, but this is not limiting. For example, a discriminator (classifier) ​​may be implemented in the arithmetic processing device 28 of the gating camera 20, and the discrimination result, i.e., the type (category) and position of the target object, may be output to the main controller 60.

[0139] (Variation 2) In the embodiment, the output of the gating camera 20 is used for driving assistance or automatic driving control, but this is not limited to the above. For example, in bad weather, the gating camera 20 is activated, and the slice image IMG generated by the gating camera 20 is displayed on a head-up display (HUD). The information may be displayed on a display device such as a DisplayPort (DisplayPort), to assist the user's field of vision.

[0140] (Variation 3) In the embodiment, the case where two shooting modes are switched between has been described, but the number of switchable shooting modes may be three or more.

[0141] The present invention has been described using specific terms based on the embodiments, but the embodiments merely show one aspect of the principles and applications of the present invention, and many modifications and changes in arrangement are permitted to the embodiments without departing from the spirit of the present invention as defined in the claims. [Industrial Applicability]

[0142] The present disclosure can be used in a sensing system for a vehicle. [Explanation of symbols]

[0143] S1 Light emission timing signal S2 Exposure timing signal 10. Sensing System 20 Gating Camera 22 Lighting equipment 24 Image Sensor 26 Camera Controller 28 Processing Unit 50 Main sensors 52 Camera 54 Millimeter Wave Radar 60 Main Controller 200 Vehicle lighting fixtures 202 Low beam unit 204 High beam unit 210 Lighting ECU 300 Automobiles 302 Headlamp 310 Vehicle ECU

Claims

1. A gating camera that divides a field of view into a plurality of ranges in a depth direction and generates a plurality of slice images corresponding to the plurality of ranges, an illumination device that irradiates a field of view with pulsed illumination light; An image sensor; a camera controller for controlling a timing of light emission of the illumination device and a timing of exposure of the image sensor; a processor for processing image data generated by the image sensor; Equipped with the arithmetic processing unit acquires information about weather based on the image data obtained for a predetermined range in a weather detection mode; The gating camera is characterized in that in the weather detection mode, the camera controller captures images in a wide range having a greater depth than in a normal shooting mode as the predetermined range.

2. 2. The gating camera according to claim 1, wherein the arithmetic processing unit acquires information relating to weather based on a predetermined area included in the image data.

3. 3. The gating camera according to claim 2, wherein the predetermined area is located above the image data.

4. 4. The gating camera according to claim 2, wherein the arithmetic processing unit calculates a histogram of the predetermined area, and obtains weather information based on the histogram.

5. 5. The gating camera according to claim 4, wherein the arithmetic processing unit acquires information relating to weather based on the shape of the histogram.

6. 5. The gating camera according to claim 4, wherein said arithmetic processing unit acquires information relating to weather based on at least one of the average and variance of said histogram.

7. A gating camera that divides a field of view into a plurality of ranges in a depth direction and generates a plurality of slice images corresponding to the plurality of ranges, an illumination device that irradiates a field of view with pulsed illumination light; An image sensor; a camera controller for controlling a timing of light emission of the illumination device and a timing of exposure of the image sensor; a processor for processing image data generated by the image sensor; Equipped with The camera controller, in a weather detection mode, captures images in a narrow range and a wide range obtained by expanding the narrow range in a forward direction, A gating camera characterized in that the processing device obtains information regarding weather based on first image data obtained for the narrow range and second image data obtained for the wide range.

8. 8. The gating camera according to claim 7, wherein the arithmetic processing device calculates histograms for each of the first image data and the second image data, and obtains the weather information based on the two histograms.

9. 9. The gating camera according to claim 8, wherein said arithmetic processing unit calculates an average and a variance of each of said two histograms, and obtains said weather information based on said average and said variance.

10. 9. The gating camera according to claim 8, wherein the arithmetic processing unit obtains the weather-related information based on the shapes of the two histograms.

11. The gating camera according to claim 7, characterized in that the calculation processing device generates a difference image representing a difference between the first image data and the second image data, and obtains the weather information based on the difference image.

12. 12. The gating camera according to claim 11, wherein the difference image is generated for a range in which an object existing in the narrow range is captured.

13. 13. The gating camera according to claim 11, wherein the arithmetic processing unit generates a histogram of the difference image, and obtains the weather information based on the histogram.

Images