Gating cameras, vehicle sensing systems, vehicle lighting fixtures

Abstract

A gating camera 20 divides the depth direction into a plurality of ranges RNG, and generates a plurality of slice images corresponding to the plurality of ranges. An illumination device 22 irradiates a probe light L1. An image sensor 24 outputs a compressed image IMG_COMP obtained by compressing image information relating to an inactive line not including a target, among a plurality of lines that constitute a raw image IMG_RAW obtained as a result of exposure. A camera controller 26 controls the light emission timing of the illumination device 22, and the exposure timing of the image sensor 24. An image processing device 28 receives the compressed image IMG_COMP from the image sensor 24 and restores a slice image IMG.

Description

[Technical field]

[0001] The present invention relates to a gating camera. [Background technology]

[0002] For autonomous driving and automatic control of headlamp distribution, an object identification system is used to sense the position and type of objects around the vehicle. The object identification system includes a sensor and a processing unit that analyzes the sensor output. The sensor is selected from cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter wave radar, ultrasonic sonar, etc., taking into consideration the application, required accuracy, and cost.

[0003] A typical monocular camera cannot obtain depth information, so it is difficult to separate overlapping objects located at different distances.

[0004] A TOF camera is known as a camera that can obtain depth information. A TOF (Time Of Flight) camera projects infrared light using a light-emitting device, measures the flight time it takes for the reflected light to return to an image sensor, and obtains a TOF image in which the flight time is converted into distance information.

[0005] As an active sensor to replace the TOF camera, a gating camera (also called a gated camera) has been proposed (Patent Documents 1 and 2). A gating camera divides the imaging range into multiple ranges, and captures images by changing the exposure timing and exposure time for each range. This allows a slice image to be obtained for each target range, and each slice image contains only objects included in the corresponding range. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2009-257981 A [Patent Document 2] International Publication WO2017 / 110417A1 Summary of the Invention [Problem to be solved by the invention]

[0007] 1. A gating camera is equipped with an image sensor and an image processing device that processes the output image (sensor image) of the image sensor. In many cases, the image sensor is connected to the image processing device via a serial interface, and the transmission speed of the sensor image between them becomes a bottleneck, which increases the time required to generate one slice image and limits the frame rate of the gating camera.

[0008] An aspect of the present disclosure has been made in light of such a situation, and one of its exemplary purposes is to provide a gating camera capable of reducing the time required to generate a slice image.

[0009] 2. The present inventors have studied a gating camera capable of taking pictures during the day and have come to recognize the following problem. In order to take pictures during the day, it is necessary to use a longer wavelength of infrared light to reduce the influence of sunlight. However, the wavelength that can be actually used is 1 μm to 1.3 μm, and this wavelength band also includes the spectrum of sunlight.

[0010] Therefore, if a gating camera is used during the day, the slice images will contain a large amount of background noise due to the influence of sunlight, and the reflected light from the object, i.e., the signal component, will be buried.

[0011] An aspect of the present disclosure has been made in light of such a situation, and one of its exemplary purposes is to provide a gating camera capable of eliminating the effects of background noise.

[0012] 3. The light incident on the image sensor of the gating camera is the probe light emitted from the gating camera, which reaches the object, and is reflected / scattered by the object and returns, so it makes one round trip between the gating camera and the object. In conditions such as thick fog or heavy rain, the transmittance between the gating camera and the object decreases, so the intensity of the light incident on the image sensor decreases. Therefore, in conditions of poor visibility, it becomes difficult to capture long-range images.

[0013] An aspect of the present disclosure has been made in light of such a situation, and one of its exemplary purposes is to provide a gating camera that can operate appropriately depending on whether the visibility is good or bad. [Means for solving the problem]

[0014] 1. An aspect of the present disclosure relates to a gating camera that divides a field of view into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates a probe light, an image sensor that generates a raw image by exposure and outputs a compressed image obtained by compressing image information related to invalid lines that do not include a target among a plurality of lines that constitute the raw image, a camera controller that controls the timing of light emission of the illumination device and the timing of exposure of the image sensor, and an image processing device that receives the compressed image from the image sensor and restores the slice image.

[0015] 2. A gating camera according to one aspect of the present disclosure includes an illumination device that irradiates probe light, an image sensor, a camera controller that controls the timing of light emission of the illumination device and the timing of exposure of the image sensor and generates multiple slice images corresponding to multiple ranges in the image sensor, and an image processing device that receives the multiple slice images from the image sensor, selects a pixel of interest, sets a slice image that includes an object at the pixel of interest as a valid image, detects a noise level based on the pixel value of the pixel of interest in slice images other than the valid image, and corrects the pixel value of the pixel of interest in each of the multiple slice images based on the noise level.

[0016] A gating camera according to one aspect of the present disclosure includes an illumination device that irradiates probe light, an image sensor, a camera controller that controls the timing of light emission of the illumination device and the timing of exposure of the image sensor and generates the plurality of slice images corresponding to a plurality of ranges on the image sensor, and an image processing device that receives the plurality of slice images from the image sensor, selects a pixel of interest, detects a noise level based on the pixel values ​​of the pixel of interest of all slice images, and corrects the pixel of interest of the plurality of slice images based on the noise level.

[0017] 3. A gating camera according to an embodiment of the present disclosure includes an illumination device that irradiates a probe light, an image sensor, and a camera controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor to generate a plurality of image data corresponding to a plurality of ranges. The camera controller reflects visibility information correlated with the attenuation coefficient of the probe light in the shooting parameters. Effect of the Invention

[0018] According to an aspect of the present disclosure, it is possible to reduce the time required to generate a slice image, it is possible to remove the influence of background noise, and it is possible to perform an operation according to the quality of visibility. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram of a sensing system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a basic operation of a gating camera. [Diagram 3] 3(a) and (b) are diagrams for explaining slice images obtained by the gating camera. [Figure 4] FIG. 1 is a diagram showing a certain driving scene. [Diagram 5] 5(a) to (c) are diagrams showing signal processing in the driving scene of FIG. [Figure 6] 4 is a flowchart showing the operation of the gating camera. [Figure 7] FIG. 7(a) is a diagram showing the compressed image IMG_COMP, and FIG. 7(b) is a diagram showing line data. [Figure 8] FIG. 13 is a diagram for explaining processing of the gating camera according to modification example 1.3. [Figure 9] FIG. 2 is a block diagram of a sensing system according to embodiment 2.1. [Figure 10] FIG. 2 is a diagram illustrating a basic operation of a gating camera. [Figure 11] 11(a) and (b) are diagrams for explaining slice images obtained by the gating camera. [Figure 12] 13 is a flowchart of noise removal according to embodiment 2.1. [Figure 13] FIG. 13 is a diagram illustrating noise removal in embodiment 2.1. [Figure 14] 14(a) and (b) are diagrams illustrating the correction process according to embodiment 2.1. [Figure 15] 15(a) and (b) are diagrams illustrating the correction process according to embodiment 2.2. [Figure 16] FIG. 11 is a block diagram of a sensing system according to a third embodiment. [Figure 17] FIG. 2 is a diagram illustrating the operation of a gating camera. [Figure 18] 18(a) and (b) are diagrams for explaining images obtained by the gating camera. [Figure 19] 17A to 17C are diagrams illustrating a first control method in the gating camera of FIG. 16. [Figure 20] 20(a) and (b) are time charts showing the operation of the first control method of the gating camera of FIG. [Figure 21] 21(a) and (b) are diagrams for explaining the second control method in the gating camera of FIG. [Figure 22]22(a) and (b) are diagrams showing a number of ranges corresponding to the control of FIG. 21(a). [Diagram 23] 17A to 17C are diagrams illustrating the operation of the gating camera of FIG. 16 according to a third control method. [Figure 24] 24(a) to (c) are diagrams showing a number of ranges in the third control method. [Diagram 25] FIG. 1 is a block diagram of a sensing system. [Figure 26] 26(a) and (b) are diagrams showing an automobile equipped with a gating camera. [Figure 27] 1 is a block diagram showing a vehicle lamp equipped with an object detection system; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] A summary of some exemplary embodiments of the present disclosure will be described. This summary is intended to provide a simplified overview of some concepts of one or more embodiments for a basic understanding of the embodiments as a prelude to the detailed description that follows, and is not intended to limit the scope of the invention or disclosure. This summary is not intended to be an exhaustive overview of all possible embodiments, nor is it intended to identify key elements of all embodiments or to delineate the scope of some or all aspects. For convenience, the term "one embodiment" may be used to refer to one embodiment (example or variant) or multiple embodiments (examples or variants) disclosed in this specification.

[0021] 1. A gating camera according to an embodiment divides a field of view into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates a probe light, an image sensor that generates a raw image by exposure and outputs a compressed image obtained by compressing image information related to invalid lines that do not include a target among a plurality of lines that constitute the raw image, a camera controller that controls the timing of light emission of the illumination device and the timing of exposure of the image sensor, and an image processing device that receives the compressed image from the image sensor and restores the slice image.

[0022] For each line of the raw image obtained by the image sensor, it is determined whether or not it contains a target object, and for lines that do not contain a target object, the transmission amount from the image sensor to the image processing device is compressed, thereby shortening the transmission time and the time required to generate a slice image.

[0023] In one embodiment, the image sensor may define a line in which the pixel values ​​of all pixels included therein are smaller than a predetermined threshold as an invalid line.

[0024] In one embodiment, the image sensor may transmit line data for a valid line that is not an invalid line, the line data including a row number, a header whose value is a first value, and pixel values ​​of all pixels included in the valid line, and transmit line data for an invalid line, the line data including a row number and a header whose value is a second value.

[0025] In one embodiment, when the value of the header of the received line data is the second value, the image processing apparatus may set all pixel values ​​of the line to 0.

[0026] 2. A gating camera according to one embodiment is a gating camera that divides a field of view into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges, and includes an illumination device that irradiates probe light, an image sensor, a camera controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor and generates a plurality of slice images corresponding to the plurality of ranges in the image sensor, and an image processing device that receives the plurality of slice images from the image sensor, selects a pixel of interest, sets a slice image that includes an object at the pixel of interest as a valid image, detects a noise level based on the pixel value of the pixel of interest in slice images other than the valid image, and corrects the pixel value of the pixel of interest in each of the plurality of slice images based on the noise level.

[0027] In an ideal gating camera, one object is captured in only one of the slice images. Therefore, when focusing on the same pixel in the slice images, only one of the slice images has a pixel value corresponding to the reflection from the object, and the pixel values ​​of the other slice images can be regarded as noise. According to this embodiment, the noise level can be obtained while performing normal shooting without performing shooting for measuring the noise level.

[0028] In one embodiment, the noise level may be an average value of pixel values ​​of pixels of interest in slice images other than the valid image.

[0029] In one embodiment, the image processing device may determine that, for each pixel, the pixel value of a pixel of interest in a slice image exceeds a predetermined threshold value as a valid image.

[0030] In one embodiment, the image processing device may determine, for each pixel, the slice image with the largest pixel value as the slice image that includes the object.

[0031] A gating camera according to one embodiment includes an illumination device that irradiates probe light, an image sensor, a camera controller that controls the timing of light emission of the illumination device and the timing of exposure of the image sensor, and generates the plurality of slice images corresponding to a plurality of ranges on the image sensor, and an image processing device that receives the plurality of slice images from the image sensor, selects a pixel of interest, detects a noise level based on the pixel values ​​of the pixel of interest of all slice images, and corrects the pixel of interest of the plurality of slice images based on the noise level.

[0032] In imaging with a large number of ranges, in other words, a large number of slice images, the effect of effective images is small, so the noise level detection process can be simplified by treating the pixel values ​​of all slice images as noise.

[0033] In one embodiment, the noise level may be the average value of the pixel values ​​of the pixels of interest in all slice images.

[0034] A gating camera according to an embodiment includes an illumination device that irradiates a probe light, an image sensor, and a camera controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor to generate a plurality of image data corresponding to a plurality of ranges. The camera controller reflects visibility information correlated with the attenuation coefficient of the probe light in the shooting parameters.

[0035] By dynamically and adaptively controlling the imaging parameters according to the attenuation coefficient of the probe light, it becomes possible to operate according to the quality of the visibility.

[0036] In one embodiment, the camera controller may increase the number of exposures for generating one slice image as the attenuation coefficient of the probe light increases. In a situation where the attenuation coefficient of the probe light is large, by increasing the number of exposures, it is possible to accumulate a large amount of reflected / scattered light from an object, and the image quality of the slice image can be improved.

[0037] In one embodiment, the camera controller may hold a plurality of relationships between range numbers and the number of exposures, and select one of them according to the visibility information.

[0038] When the attenuation coefficient is large, the resolution in the depth direction may be increased at the expense of giving up long-distance imaging.

[0039] In one embodiment, the camera controller may reduce the number of ranges to be measured from the far side as the attenuation coefficient of the probe light increases. When the attenuation coefficient is large, long-distance imaging is abandoned and close-distance sensing is concentrated, thereby increasing the frame rate of slice images in close ranges.

[0040] Hereinafter, preferred embodiments will be described with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing will be given the same reference numerals, and duplicated descriptions will be omitted as appropriate. In addition, the embodiments are illustrative and do not limit the disclosure and invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the disclosure and invention.

[0041] (Embodiment 1) 1 is a block diagram of a sensing system 10 according to embodiment 1. The sensing system 10 is mounted on a vehicle such as an automobile or motorcycle, and detects an object OBJ that exists around the vehicle.

[0042] The sensing system 10 mainly includes a gating camera 20. The gating camera 20 includes an illumination device 22, an image sensor 24, a camera controller 26, and an image processing device 28. The imaging by the gating camera 20 is performed by detecting a plurality of N (N≧2) ranges RNG1 to RNG2 in the depth direction of the field of view. N Adjacent ranges may overlap in the depth direction at their boundaries.

[0043] The lighting device 22 irradiates the probe light L1 to the front of the vehicle in synchronization with a light emission timing signal S1 provided from the camera controller 26. The probe light L1 is preferably infrared light, but is not limited thereto and may be visible light or ultraviolet light having a predetermined wavelength.

[0044] The image sensor 24 includes a plurality of pixels, and is capable of controlling exposure in synchronization with an exposure timing signal S2 provided by the camera controller 26, thereby generating a raw image (RAW image) consisting of a plurality of pixels. The image sensor 24 is sensitive to the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ. The i-th range RNG i The slice image IMG generated by the image sensor 24 i The slice images IMGs, which are the final output of the gating camera 20, may be referred to as raw images or primary images as necessary. i Distinguish from.

[0045] The camera controller 26 controls the irradiation timing (light emission timing) of the probe light L1 by the illumination device 22 and the exposure timing by the image sensor 24. The camera controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware).

[0046] The image sensor 24 and the image processing device 28 are connected via a serial interface 30. The raw image IMG_RAW generated in the image sensor 24 is i If all the pixels are transmitted to the image processing device 28 as the primary image IMG, the transmission time will be long, and the sensing time will also be long.

[0047] In this embodiment, the image sensor 24 receives the raw image IMG_RAW i The compressed image IMG_COMP obtained by removing unnecessary lines fromi Primary image IMG i The image processing device 28 transmits the image data to the image processing device 28.

[0048] Specifically, the image sensor 24 outputs a raw image IMG_RAW obtained as a result of exposure. i Among the multiple lines constituting the image, invalid lines that do not include a target, i.e., a subject to be detected, are detected. i Then, compress the image information about the invalid lines to create a compressed image IMG_COMP i Generate.

[0049] The image processor 28 receives the compressed image IMG_COMP from the image sensor 24. i Primary image IMG i Then the compressed image IMG_COMP is received. i From slice image IMGs i Restore.

[0050] The above is the configuration of the gating camera 20. Next, the operation will be described.

[0051] FIG. 2 is a diagram for explaining the basic operation of the gating camera 20. In FIG. i The figure shows how the gating camera 20 senses the range RNG as a range of interest (ROI). The lighting device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. i The distance to the boundary in front of MINi、 Range RNG i The distance to the inner boundary of MAXi Let us assume that.

[0052] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi=2×d MINi / c where c is the speed of light.

[0053] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, T MAXi =2×d MAXi / c It is.

[0054] Range RNG i When it is desired to photograph only the object OBJ included in the time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.

[0055] i-th range RNG i When capturing an image, multiple sets of light emission and exposure may be performed. In this case, the camera controller 26 may repeat the above-described exposure operation multiple times at a predetermined period τ2.

[0056] 3(a) and (b) are diagrams for explaining slice images obtained by the gating camera 20. In the example of FIG. 3(a), an object (pedestrian) OBJ2 exists in range RNG2, and an object (vehicle) OBJ3 exists in range RNG3. FIG. 3(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 3(a). When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, so no object image is captured in the slice image IMG1.

[0057] When the slice image IMG2 is captured, the image sensor is exposed only by the light reflected from the range RNG2, so that only the object image OBJ2 appears in the slice image IMG2. Similarly, when the slice image IMG3 is captured, the image sensor is exposed only by the light reflected from the range RNG3, so that only the object image OBJ3 appears in the slice image IMG3. In this way, the gating camera 20 can capture the object separately for each range.

[0058] Next, image compression and transmission in the gating camera 20 of Fig. 1 will be described with reference to Fig. 4 and Figs. 5(a) to (c). Fig. 4 is a diagram showing a certain driving scene. RNG i is the range of the sensing target (range of interest). FIGS. 5(a) to (c) are diagrams showing signal processing in the driving scene of FIG. 4. FIG. 5(a) shows a raw image IMG_RAW obtained in the driving scene of FIG. i This raw image IMG_RAW is shown. i Range RNG i In this example, the target is in the raw image IMG_RAW. i The middle range of Y M It exists in the upper range Y U is empty and does not contain any targets.

[0059] In this example, the raw image IMG_RAW i Lower range Y L is the previous range RNG i-1 This is the part of the ground that corresponds to this range Y L Also, the target is not included.

[0060] The image sensor 24 has a range Y U and Y L For example, the image sensor 24 determines a line included in the raw image IMG_RAW i The lines are scanned in order, and when all pixels of each line are smaller than a predetermined threshold value, the line is determined to be an invalid line.

[0061] The determination of whether a line is invalid or valid may be performed for each sensing cycle. N This refers to the entire shooting cycle.

[0062] Alternatively, the determination of whether a line is invalid or valid may be performed once every several sensing periods. In this case, a line that has been determined to be invalid once will be treated as an invalid line until the next determination result is obtained.

[0063] Figure 5(b) shows the raw image IMG_RAW of Figure 5(a). i The compressed image IMG_COMP obtained from i Indicates the compressed image IMG_COMP i is the raw image IMG_RAW i Range of Y U and Y L The lines contained in the image are compressed so as not to include image information.

[0064] Fig. 5(c) shows the compressed image IMG_COMP of Fig. 5(b). i The final slice image IMGs obtained by restoring i Indicates the compressed image IMG_COMP i The image processing device 28 receives the image data and replaces the invalid lines that do not contain image information with pixels of a predetermined value (for example, a value of 0) to generate a slice image IMGs. i Play.

[0065] 6 is a flowchart showing the operation of the gating camera 20. First, a variable i is initialized to 1 (S100). The variable i is a variable that indicates an interesting range. For example, i=1 indicates that the first range RNG1 is the measurement target.

[0066] Next, a loop S102 is executed for the variable i. First, the i-th range RNG i The illumination device 22 emits light and the image sensor 24 exposes the light to generate a raw image IMG_RAW. iis generated (S104).

[0067] Next, the raw image IMG_RAW i The validity or invalidity of each line included in the image IMG_RAW is determined. First, a variable j is initialized to 1 (S106). The variable j is i The loop S108 is then executed for the variable j.

[0068] Raw image IMG_RAW i It is determined whether the pixel values ​​of all pixels in the j-th line are smaller than the threshold value (S110). If all pixel values ​​are smaller than the threshold value (Y in S110), the j-th line is determined to be an invalid line (S112). If not (N in S110), the j-th line is determined to be a valid line. Then, the variable j is incremented. The variable j is set to the value of the raw image IMG_RAW i Number of lines Y MAX While the vertical resolution is smaller than the vertical resolution, the loop S108 is repeated. i Number of lines Y MAX Once you pass this point, you will exit Loop S108.

[0069] When loop S108 is exited, variable i is incremented (S118) and the next range becomes the measurement target. While variable i is smaller than the maximum number N of ranges, loop S02 is repeated. When variable i exceeds the maximum number N, all ranges RNG1 to RNG N When the photographing is completed, the process ends.

[0070] Next, the image compression method will be described. Fig. 7(a) is a diagram showing a compressed image IMG_COMP, and Fig. 7(b) is a diagram showing line data. The compressed image IMG_COMP is composed of a plurality of line data LD1 to LD VMAXThe line data LD includes different information depending on whether it is a valid line or an invalid line. In this example, the first and second lines are invalid lines, and the rest are valid lines. Line data 50 of a valid line includes a line number 52, a header 54, and pixel values ​​56 of all pixels in the line. Line data 50' of an invalid line includes the line number 52 and a header 54, but does not include pixel values ​​56. The header 54 includes an identifier indicating whether the line is valid or invalid; for example, the header 54 of a valid line stores a value of 1, and the header 54 of an invalid line stores a value of 0.

[0071] In the compressed image IMG_COMP in Fig. 7, image information is completely removed from invalid lines. Therefore, the time required to transmit an invalid line is the time required to transmit the line number 52 and the header 54, which can be significantly reduced.

[0072] The compressed image IMG_COMP in Fig. 7 can be restored by the image processing device 28 as follows. The image processing device 28 refers to the header 54 of the line data 50. If the value is 1, the following pixel value 56 is set as the pixel value of the corresponding line in the slice image IMG. If the value of the header 54 is 0, the pixel value of the corresponding line in the slice image IMG is filled with 0.

[0073] The present invention has been described above based on an embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. Such modifications will be described below.

[0074] (Variation 1.1) The compressed image IMG_COMP may contain only line data for valid lines. In this case, a valid line includes a line number followed by pixel values ​​for all pixels. The image processor 28 fills in line numbers that are not included in the received line data with zeros.

[0075] (Variation 1.2) In the above examples, pixel information is not transmitted for invalid lines, but image information may be transmitted. For example, pixels may be thinned out and transmitted for invalid lines. If one pixel is transmitted for every M horizontally adjacent pixels, the transmission time can be reduced by 1 / M. The image processing device 28 may restore the slice image IMG by treating the adjacent M pixels as having the same pixel value.

[0076] (Variation 1.3) In the first embodiment, it is determined whether or not a target is included for each line. However, this is not limited to this, and it is also possible to predefine predetermined lines of the raw image IMG_RAW as invalid lines for each range.

[0077] FIG. 8 is a diagram for explaining the processing of the gating camera according to the modified example 1.3. In the figure, the dashed line 80 indicates the highest position of the assumed target. It is assumed that the vertical angle of view θ of the image sensor 24 is constant regardless of the range. When photographing at the close ranges RNG1 and RNG2, only the range below the dashed line 80 is photographed. On the other hand, for the far range RNG3, a range 82 higher than the dashed line is included in the photographing range (the angle of view θ of the image sensor), but it can be said that the possibility of including the target to be sensed in this range 82 is low. Therefore, for the raw image IMG_RAW3, the range 82 above the line corresponding to the dashed line 80 can be determined as an invalid line.

[0078] (Embodiment 2.1) 9 is a block diagram of a sensing system 10 according to embodiment 1. This sensing system 10 is mounted on a vehicle such as an automobile or motorcycle, and detects an object OBJ that exists around the vehicle.

[0079] The sensing system 10 mainly includes a gating camera 20. The gating camera 20 includes an illumination device 22, an image sensor 24, a camera controller 26, and an image processing device 28. The imaging by the gating camera 20 is performed by detecting a plurality of N (N≧2) ranges RNG1 to RNG2 in the depth direction of the field of view.N Adjacent ranges may overlap in the depth direction at their boundaries.

[0080] The lighting device 22 irradiates the probe light L1 ahead of the vehicle in synchronization with a light emission timing signal S1 provided by the camera controller 26. The probe light L1 is preferably infrared light, but is not limited thereto and may be visible light or ultraviolet light having a predetermined wavelength. The gating camera 20 according to this embodiment is capable of sensing not only at night but also during the day, and therefore a wavelength longer than 0.9 μm is selected.

[0081] The image sensor 24 includes a plurality of pixels, and is capable of controlling exposure in synchronization with an exposure timing signal S2 provided by the camera controller 26, thereby generating a slice image IMG consisting of a plurality of pixels. The image sensor 24 is sensitive to the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ. The i-th range RNG i The slice images IMG generated by the image sensor 24 are referred to as raw images or primary images as necessary to be distinguished from the slice images IMGs that are the final output of the gating camera 20.

[0082] The camera controller 26 controls the irradiation timing (light emission timing) of the probe light L1 by the illumination device 22 and the exposure timing by the image sensor 24, and causes the image sensor 24 to display a plurality of ranges RNG1 to RNG2. N Multiple slice images IMG1 to IMG N The camera controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware).

[0083] The image sensor 24 and the image processing device 28 are connected via a serial interface 30, and a plurality of ranges RNG1 to RNG N Multiple slice images (primary images) IMG1 to IMG N is transmitted to the image processing device 28. i is the raw image IMG_RAW i It can be the image itself or the raw image IMG_RAW i The image may be a processed image.

[0084] The above is the basic configuration of the gating camera 20. Next, the operation will be described.

[0085] 10 is a diagram for explaining the basic operation of the gating camera 20. In FIG. i The figure shows how the gating camera 20 senses the range RNG as a range of interest (ROI). The lighting device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. i The distance to the boundary in front of MINi、 Range RNG i The distance to the inner boundary of MAXi Let us assume that.

[0086] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi =2×d MINi / c where c is the speed of light.

[0087] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, TMAXi =2×d MAXi / c It is.

[0088] Range RNG i When it is desired to photograph only the object OBJ included in the time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.

[0089] i-th range RNG i When capturing an image, multiple sets of light emission and exposure may be performed. In this case, the camera controller 26 may repeat the above-described exposure operation multiple times at a predetermined period τ2.

[0090] 11(a) and (b) are diagrams for explaining slice images obtained by the gating camera 20. In the example of FIG. 11(a), an object (pedestrian) OBJ2 exists in range RNG2, and an object (vehicle) OBJ3 exists in range RNG3. FIG. 11(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 11(a). When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, so no object image is captured in the slice image IMG1.

[0091] When the slice image IMG2 is captured, the image sensor is exposed only by the light reflected from the range RNG2, so that only the object image OBJ2 appears in the slice image IMG2. Similarly, when the slice image IMG3 is captured, the image sensor is exposed only by the light reflected from the range RNG3, so that only the object image OBJ3 appears in the slice image IMG3. In this way, the gating camera 20 can capture the object separately for each range.

[0092] Returning to FIG. 9, as described above, the gating camera 20 performs sensing even during the day, and sunlight is incident on the image sensor 24. Since sunlight contains spectral components in the wavelengths used by the gating camera 20, it becomes background noise and reduces the S / N ratio of the image. Therefore, the image processing device 28 processes the multiple slice images IMG1 to IMG N The noise reduction function is described below.

[0093] The image processing device 28 receives a plurality of slice images IMG1 to IMG N The i-th (1≦i≦N) slice image IMG i The j-th pixel of IMG i The image processing device 28 outputs a plurality of slice images IMG1 to IMG N For each pixel, one pixel is selected as a pixel of interest, and the noise level is detected for each pixel of interest.

[0094] The image processing device 28 outputs a plurality of slice images IMG1 to IMG N The j-th pixel of the slice images IMG1 to IMG N For example, the image processing device 28 may select a slice image including an object at the position of the pixel of interest from among the pixels IMG1(j) to IMG N Among (j), the slice image containing the maximum pixel value is determined as the valid image. M When the value of (j) is maximum, the Mth slice image IMG M is regarded as a valid image.

[0095] Then, the image processing device 28 outputs the effective image IMG M Pixels IMG1(j) to IMG M-1 (j),IMG M+1 (j)~IMG N The noise level N(j) is detected based on the value of (j). N Pixels IMG1(j)~IMGN The pixel value of (j) is corrected based on the noise level N(j). The image processing device 28 outputs the corrected slice image as the final slice image IMGs.

[0096] For ease of explanation, all ranges RNG1~RNG N In other words, it is assumed that the exposure time is equal in all ranges. In this case, the noise level N(j) is the noise level of the non-valid image IMG k The noise level N(j) can be expressed by the formula (1) by taking the simple average of the pixel values ​​of the pixels of interest (k=1 to N, where k≠M). N(j)=Σ k k≠M IMG k (j) / (N-1) …(1)

[0097] In addition, the correction is performed on the slice image IMG k The noise level N(j) can be subtracted from (j). k The pixel value of '(j) is expressed by equation (2). IMG k '(j)=IMG k (j)-N(j) …(2) The image processing device 28 repeats the same process for all pixels, except that the pixel value after correction is not made to fall below zero.

[0098] 12 is a flowchart of noise removal, in which i is a variable indicating the number of a slice image, and j is a variable indicating the position of a pixel of interest.

[0099] First, j is initialized (S100), and then a loop S102 is repeated for each pixel.

[0100] Multiple slice images IMG1~IMG N Each jth pixel of interest IMG1(j)~IMG N Refer to (j) and based on those values, the effective image IMG M Then, the effective image IMGM Other images IMG1~IMG M-1 ,IMG M+1 ~IMG N The jth pixel IMG1(j)~IMG M-1 (j),IMG M+1 (j)~IMG N Based on (j), a noise level N(j) is calculated (S106).

[0101] Next, a variable i is initialized (S108), and a loop S110 is executed. i Corresponding pixel image i The noise level N(j) is subtracted from (j) (S112). Then, the variable i is incremented (S114). If i≦N, the process returns to step S112, and if i>N, the process exits from loop S110.

[0102] Next, the variable j is incremented (S116). If j≦X×Y, the process returns to step S102 and repeats the same process for the next pixel. If j>X×Y, the process exits from loop S102 and ends. X×Y is the total number of pixels in the slice image.

[0103] Fig. 13 is a diagram for explaining noise removal. In Fig. 13, N slice images IMG1 to IMG2 obtained in one imaging are shown. N is shown, where each slice image contains X×Y pixels, and the j-th pixel of interest is hatched.

[0104] 14(a) and (b) show N slice images IMG1 to IMG N 1 is a diagram showing pixel values ​​before and after correction of a pixel of interest in FIG. 1, where N=6 and M=4 is a valid image.

[0105] In an ideal gating camera, one object is captured as multiple slice images IMG1 to IMG N Therefore, the slice images IMG1 to IMG NFocusing on the same pixel in only one slice (here, the fourth slice image IMG4, where M=M) has a large pixel value corresponding to reflection from an object, and the pixel values ​​of the other non-valid images IMG1, IMG2, IMG3, IMG5, and IMG6 can be considered as background noise. Therefore, the noise level N(j) can be obtained by processing the pixel values ​​of the target pixels IMG1(j), IMG2(j), IMG3(j), IMG5(j), and IMG6(j) of the non-valid images IMG1, IMG2, IMG3, IMG5, and IMG6.

[0106] Then, the noise level N(j) is subtracted from the pixel value of the pixel of interest in each of the slice images IMG1 to IMG6 to obtain the corrected pixel value.

[0107] The above is the noise elimination in the gating camera 20. The advantages of the gating camera 20 according to this embodiment become clear when compared with comparative techniques.

[0108] In the comparative technique, in addition to capturing N slice images, exposure is performed on the image sensor 24 without emitting light from the illumination device 22, and background noise is captured. In the comparative technique, N+1 captures are required per capture cycle, which lengthens the capture cycle.

[0109] In contrast, according to this embodiment, it is possible to obtain the noise level in parallel with the normal N number of shots without taking a shot to measure the noise level, and therefore it is possible to prevent the shooting cycle from becoming long.

[0110] (Embodiment 2.2) In the embodiment 2.1, a plurality of slice images IMG1 to IMG N In the embodiment 2.2, the effective image is detected from the effective image, and the noise level is detected based on the pixel value of the pixel of interest of the other non-effective images. In the embodiment 2.2, the noise level is detected without detecting the effective image.

[0111] The configuration of the gating camera 20 according to the embodiment 2.2 is similar to that shown in the block diagram of Fig. 9, but the processing of the image processing device 28 differs from that of the embodiment 2.1. The noise removal processing in the embodiment 2.2 will be described below.

[0112] The image processing device 28 processes all slice images IMG1 to IMG N A noise level N(j) is detected based on the pixel value of the pixel of interest in the slice image, and the pixel value of the pixel of interest in each of the slice images is corrected based on the noise level N(j).

[0113] When the exposure times for all ranges are equal, the noise level N(j) may be calculated using equation (3). N(j)=Σ k k=1:N IMG k (j) / N …(3)

[0114] In addition, the correction is performed on the slice image IMG k The noise level N(j) can be subtracted from (j). k The pixel value of '(j) is expressed by equation (4), which is the same as equation (2). IMG k '(j)=IMG k (j)-N(j) …(4)

[0115] 15(a) and (b) are diagrams for explaining the correction process according to embodiment 2.2. N 15(b) is a diagram showing the pixel value of the pixel of interest before correction, and FIG. 15(b) is a diagram showing the pixel value after correction. Here, it is assumed that N=8.

[0116] The above is the correction process according to the embodiment 2.2. According to the correction process of the embodiment 2.2, as in the embodiment 2.1, exposure only for detecting the noise level is not necessary, and therefore the shooting cycle can be shortened.

[0117] Furthermore, in embodiment 2.2, since there is no need to detect a valid image, the calculation load on the image processing device 28 can be reduced compared to embodiment 2.1. On the other hand, in embodiment 2.2, since the reflected light of an object is also treated as a noise component, the detection accuracy of the noise level is inferior to that of embodiment 2.1. However, when the number N of ranges is large, the noise level can be detected with sufficient accuracy.

[0118] Modifications related to embodiments 2.1 and 2.2 will be described.

[0119] (Variation 2.1) Modification 2.1 of embodiment 2.1 will be described. In the above description, the slice image having the maximum value in the pixel of interest is regarded as the valid image, but this is not limited to the above. For example, a predetermined threshold value may be determined, and a slice image in which the pixel value of the pixel of interest exceeds the threshold value may be treated as a valid image. In this case, there may be multiple valid images. The method described in embodiment 2.1 is to select N slice images IMG1 to IMG N The effective image cannot be determined until the imaging of the N slice images IMG1 to IMG2 is completed. N This has the advantage that the noise level calculation process can be performed without waiting for the completion of shooting.

[0120] (Variation 2.2) In the embodiment 2.1, a simple average is used to calculate the noise level of the pixel of interest, but the calculation method is not limited to this. The simple average is effective when the depth length of all ranges is equal, in other words, when the exposure time is equal, but when the exposure time differs for each range, a weighted average weighted by a coefficient according to the exposure time can be calculated.

[0121] The exposure time in the i-th range is T j The noise level in embodiment 2.1 may be calculated based on formula (5). This noise level represents the amount of noise per unit exposure time. N(j) = {Σ k k≠M IMGk (j) / T k} / (N-1) …(5)

[0122] The correction process may also be performed based on equation (6). IMG k '(j)=IMG k (j)-N(j) T k …(6)

[0123] (Variation 2.3) In embodiment 2.2, the exposure time T i If j differs for each range, the noise level N(j) may be calculated based on equation (7). N(j) = {Σ k k=1:N IMG k (j) / T k} / N …(7)

[0124] The correction process may also be performed based on equation (8). IMG k '(j)=IMG k (j)-N(j) T k …(8)

[0125] (Variation 2.4) In the modified examples 2.2 and 2.3, the effective image IMG M First, the slice images IMG1 to IMG2 are detected, and then the correction process is performed by weighting. N All pixels are then processed with exposure time T i Coefficient A according to i (∝T i -1 ) may be used for correction. IMG i "=A i ×IMG i …(9)

[0126] In the case of embodiment 2.1, the corrected slice images IMG1″ to IMG N Regarding ", a valid image may be detected for each pixel of interest, and the noise level may be calculated based on the formula (1'). N(j)=Σ k k≠M IMG k "(j) / (N-1) ...(1') In the case of embodiment 2.2, the noise level may be calculated based on equation (3'). N(j)=Σ k k=1:N IMG k "(j) / N …(3')

[0127] Then, the correction process may be performed based on the equation (10). IMG k '(j)=IMG k (j)”-N(j) …(10)

[0128] (Variation 2.5) Ideally, an object included in a certain range is only captured in the slice image corresponding to it, but in reality, the object may also be captured in slice images in the range adjacent to that range. Therefore, in addition to the slice image having the maximum pixel value at the pixel of interest, one (or two) slice images adjacent to it may be treated as valid images. In this case, the other (N-2) slice images (or N-3 slice images) are processed as non-valid images.

[0129] (Variation 2.6) In the embodiment 2.1, the noise level is calculated based on the pixel values ​​of the effective pixels of the N-1 non-effective images other than the effective image, but this is not limited to the above. For example, the maximum and minimum values ​​of the effective pixels of the N-1 non-effective images may be excluded, and the noise level may be calculated based on the remaining N-3 pixel values.

[0130] The same applies to embodiment 2.2, and the maximum and minimum values ​​of the pixel values ​​of the valid pixels of all slice images may be excluded, and the noise level may be calculated based on the remaining (N-2) pixel values.

[0131] (Variation 2.7) Furthermore, the method of calculating the noise level is not limited to averaging. For example, in the embodiment 2.1, the noise level may be the median value of the pixel values ​​of the pixels of interest in the non-effective image. In the embodiment 2.2, the noise level may be the median value of the pixel values ​​of the pixels of interest in all slice images.

[0132] (Variation 2.8) In the above description, all pixels are sequentially treated as pixels of interest and the noise level is detected, but this is not limited to the above. A slice image may be divided into a number of areas, one pixel of interest may be selected for each area, the noise level may be detected, and pixels within the same area may be corrected using a common noise level.

[0133] (Variation 2.9) The above-described noise removal process is performed only in situations where the influence of sunlight cannot be ignored, and the noise removal process may be omitted at night when there is no influence of sunlight or on cloudy days.

[0134] (Variation 2.10) The above explanation concerns a gating camera 20 intended for use during the day and at night, but the application of the noise removal processing of the present invention is not limited to this and can also be applied to a gating camera 20 that is used only at night.

[0135] (Embodiment 3) 16 is a block diagram of a sensing system 10 according to embodiment 3. This sensing system 10 is mounted on a vehicle such as an automobile or motorcycle, and detects an object OBJ that exists around the vehicle.

[0136] The sensing system 10 mainly includes a gating camera 20. The gating camera 20 includes an illumination device 22, an image sensor 24, a camera controller 26, and an image processing device 28. The imaging by the gating camera 20 is performed by detecting a plurality of N (N≧2) ranges RNG1 to RNG2 in the depth direction of the field of view. N Adjacent ranges may overlap in the depth direction at their boundaries.

[0137] The lighting device (floodlight) 22 irradiates the probe light L1 ahead of the vehicle in synchronization with a light emission timing signal S1 provided by the camera controller 26. The probe light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength. The lighting device 22 may be, for example, a laser diode (LD) or an LED.

[0138] The image sensor 24 is capable of performing exposure control in synchronization with an exposure timing signal S2 provided from the camera controller 26, and is configured to be capable of generating a slice image IMG. The image sensor 24 has sensitivity to the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ.

[0139] The camera controller 26 changes the light emission timing signal S1 and the exposure timing signal S2 for each range RNG to change the time difference between the light emission by the lighting device 22 and the exposure of the image sensor 24. The light emission timing signal S1 specifies the timing to start light emission and the light emission time. The exposure timing signal S2 specifies the timing to start exposure (the time difference with light emission) and the exposure time.

[0140] The gating camera 20 has a plurality of ranges RNG1 to RNG N Multiple slice images IMG1 to IMG N The i-th slice image IMG i has the corresponding range RNG i Only objects contained within will be captured.

[0141] The image processing device 28 can be implemented by combining a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), microcomputer, or GPU (Graphics Processing Unit) with a software program executed by the processor (hardware). The image processing device 28 may be configured only with hardware. The image processing device 28 processes the image generated by the image sensor 24 and outputs a final slice image. Note that when the output of the image sensor 24 is used as the slice image as it is, the image processing device 28 can be omitted.

[0142] 17 is a diagram for explaining the operation of the gating camera 20. In FIG. i The illumination device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. The range RNG from the gating camera 20 is i The distance to the boundary in front of MINi、 Range RNG i The distance to the inner boundary of MAXi Let us assume that.

[0143] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi =2×d MINi / c where c is the speed of light.

[0144] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, T MAXi =2×d MAXi / c It is.

[0145] Range RNG i When it is desired to photograph only the object OBJ included in the time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.

[0146] i-th range RNG i When capturing an image, multiple exposures may be performed. In this case, the camera controller 26 may repeat the above-mentioned set of irradiation and exposure operations multiple times at a predetermined period τ2. The image sensor 24 outputs slice images accumulated by multiple exposures.

[0147] In this embodiment, in order to prevent the exposure (brightness value of the object image in the slice image) from varying for each range, the gating camera 20 optimizes the shutter speed (exposure time), number of exposures, sensitivity, irradiation intensity of the probe light, etc. (imaging parameters) for each range.

[0148] 18(a) and (b) are diagrams for explaining images obtained by the gating camera 20. In the example of FIG. 18(a), an object (pedestrian) OBJ2 exists in range RNG2, and an object (vehicle) OBJ3 exists in range RNG3. FIG. 18(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 18(a). When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, so no object image is captured in the slice image IMG1.

[0149] When the slice image IMG2 is captured, the image sensor is exposed only by the light reflected from the range RNG2, so that only the object image OBJ2 appears in the slice image IMG2. Similarly, when the slice image IMG3 is captured, the image sensor is exposed only by the light reflected from the range RNG3, so that only the object image OBJ3 appears in the slice image IMG3. In this way, the gating camera 20 can capture the object separately for each range.

[0150] Returning to Fig. 16, the process when the gating camera has poor visibility will be explained. Note that since the gating camera uses infrared light or the like as a probe light, the "visibility of the gating camera" does not necessarily match the visibility seen by the human eye. Visibility changes due to weather conditions such as fog and rain, and also due to factors such as dust.

[0151] In this embodiment, the camera controller 26 includes a probe light attenuation coefficient σ[m -1 The visibility information INFO_FOV having a correlation with ] is input. The attenuation coefficient σ, the propagation distance r, and the transmittance T are related to each other as shown in Equation (1). T = exp(-σr) … (1)

[0152] The method of generating the visibility information INFO_FOV is not particularly limited. For example, the attenuation coefficient σ or the transmittance T may be calculated based on image processing in the image processing device 28, and the calculated value may be used as the visibility information INFO_FOV.

[0153] The attenuation coefficient σ and the transmittance may be measured or estimated by a sensor provided separately from the gating camera 20. Alternatively, a user may determine whether the visibility is good or bad and provide visibility information INFO_FOV to the gating camera 20. For visible light, visibility is known as an index of visibility, and information corresponding to this can be used as visibility information INFO_FOV.

[0154] Alternatively, since the attenuation coefficient σ depends on the amount of rainfall or the density of fog, the presence or absence (or density) of rainfall or fog may be detected by a sensor or received from a user, and used as visibility information INFO_FOV.

[0155] The camera controller 26 reflects the field of view information INFO_FOV in the shooting parameters. Hereinafter, some examples of the control of the shooting parameters will be described.

[0156] (First control method) The camera controller 26 changes the number of exposures required to generate one slice image based on the field of view information INFO_FOV. Specifically, the larger the attenuation coefficient σ is, the more the number of exposures is increased.

[0157] Fig. 19 is a diagram for explaining the first control method in gating camera 20 of Fig. 16. The horizontal axis indicates the range position (range number), i.e., distance, and the vertical axis indicates the number of exposures. As the distance to the object increases, the distance r in formula (1) increases, and the transmittance T decreases, so that the intensity of light incident on image sensor 24 decreases. To compensate for this, the shooting parameters are specified so that the number of exposures increases as the range increases.

[0158] In this embodiment, the relationship between the range number and the number of exposures is defined for each attenuation coefficient σ. In this example, the attenuation coefficient σ is expressed in three stages, and the relationship σ1<σ2<σ3 holds. The larger the attenuation coefficient σ, the higher the relationship between the range number and the number of exposures shifts. For example, the relationship in FIG. 19 is non-volatilely stored in the camera controller 26 as a lookup table or an arithmetic expression, and the camera controller 26 determines the number of exposures based on one relationship according to the field of view information INFO_FOV. Here, the relationship between the range number and the number of exposures is shown as a curve, but it may be defined as a straight line or a broken line.

[0159] 20(a) and (b) are time charts showing the operation of the first control method of the gating camera 20 of FIG. 16. FIG. 20(a) shows the operation when the visibility is good, that is, when the attenuation coefficient σ is small. In this example, the first range RNG1 and the second range RNG2 are exposed twice, and the Nth range RNG N The number of exposures is four. The closer the range, the shorter the time difference between light emission and exposure. Here, the time interval between light emission is assumed to be constant regardless of the range or the visibility (attenuation coefficient σ). Note that the time interval between light emission may be different for each range, or may be different depending on the visibility.

[0160] FIG. 20(b) shows the operation when visibility is poor, that is, when the attenuation coefficient σ is large. In this example, the number of exposures for the first range RNG1 and the second range RNG2 is increased from two to three, and the number of exposures for the Nth range RNG N The number of exposures has increased from four to six.

[0161] According to the first control method, in poor visibility conditions, the total amount of light incident on the image sensor 24 can be increased by increasing the number of exposures, thereby obtaining image quality that is comparable to that in good visibility conditions.

[0162] The method of generating one slice image based on multiple exposures is not particularly limited. For example, when using an image sensor 24 capable of multiple exposures, one slice image can be generated by accumulating the total amount of light from multiple exposures.

[0163] If the image sensor 24 does not support multiple exposures, a raw image may be generated for each exposure, and a single slice image may be generated by synthesizing multiple raw images corresponding to the multiple exposures in a downstream image processing device 28.

[0164] (Second control method) In the second control method, the camera controller 26 changes the depth length of the range based on the field of view information INFO_FOV. Specifically, the greater the attenuation coefficient σ, the shorter the range length is made.

[0165] Figures 21(a) and (b) are diagrams for explaining the second control method in the gating camera 20 of Figure 16. The horizontal axis indicates the range position (range number), that is, the distance, and the vertical axis indicates the depth length of each range.

[0166] FIG. 21(a) shows a case where the depth lengths of all ranges are equal. In this example, the depth lengths are controlled in two stages according to the attenuation coefficient σ, and N=4. When the attenuation coefficient σ is small (σ1), the depth of each range is set to 50 m. In other words, the closest range RNG1 is 0 to 50 m, the second range RNG2 is 50 to 100 m, the third range RNG3 is 100 to 150 m, and the fourth range RNG4 is 150 to 200 m.

[0167] When the attenuation coefficient σ is large (σ2), the depth of each range is set to 25 m. That is, the closest range RNG1 is 0 to 25 m, the second range RNG2 is 25 to 50 m, the third range RNG3 is 50 to 75 m, and the fourth range RNG4 is 75 to 100 m.

[0168] Figures 22(a) and (b) are diagrams showing multiple ranges corresponding to the control of Figure 21(a). Figure 22(a) shows multiple ranges under good visibility conditions with a small attenuation coefficient σ. Figure 22(b) shows multiple ranges under poor visibility conditions with fog, a large attenuation coefficient σ.

[0169] As shown in Fig. 21(b), the depth length may be different for each range. For example, the relationship between the range number and the depth length may be determined for each of the attenuation coefficient σ values ​​σ1, σ2, σ3, and σ4, and the relationship according to the current attenuation coefficient may be selected.

[0170] In the second control method, when the attenuation coefficient σ is large, the length of each range is shortened, and the resolution in the depth direction can be increased at the expense of giving up long-distance imaging.

[0171] It should be noted that, among the multiple ranges, the depth lengths of some ranges may be kept constant, and the depth lengths of some ranges may be changed according to the attenuation coefficient σ.

[0172] (Third control method) In the third control method, the camera controller 26 changes the number N of ranges to be measured based on the field of view information INFO_FOV. Specifically, the number of ranges to be measured is decreased in order from the back side as the attenuation coefficient σ increases.

[0173] Fig. 23 is a diagram for explaining the operation according to the third control method of gating camera 20 in Fig. 16. The horizontal axis indicates attenuation coefficient σ, and the vertical axis indicates the number N of ranges to be measured.

[0174] 24(a)-(c) are diagrams showing multiple ranges in the third control method. Here, it is assumed that the depth lengths of all ranges are equal. FIG. 24(a) shows a situation in which visibility is good, and measurements are performed for N=8 ranges. FIG. 24(b) shows a situation in which it is raining and visibility is somewhat poor, and the number of ranges to be measured is reduced to N=6. FIG. 24(c) shows a situation in which fog has occurred, visibility is poor, and the number of ranges to be measured is reduced to N=4.

[0175] In the third control method, when the attenuation coefficient σ is large, long-distance imaging is abandoned and close-distance sensing is concentrated. This allows for a high frame rate for close-range slice images to be achieved by omitting long-distance imaging, which requires a large number of exposures and takes a long time to sense, in poor visibility conditions.

[0176] The first to third control methods may be used alone or in any combination of two or more.

[0177] The use of the above-mentioned gating camera 20 will now be described.

[0178] (Application) 25 is a block diagram of the sensing system 10. The sensing system 10 includes a calculation processing device 40 in addition to the above-mentioned gating camera 20. This sensing system 10 is an object detection system that is mounted on a vehicle such as an automobile or motorcycle, and determines the type (also called category or class) of an object OBJ that exists around the vehicle.

[0179] The gating camera 20 detects multiple ranges RNG1 to RNG N Multiple slice images IMGs1 to IMGs N The i-th slice image IMGs is generated. i has the corresponding range RNG i Only objects contained within the frame are captured.

[0180] The arithmetic processing device 40 detects a plurality of ranges RNG1 to RNG2 obtained by the gating camera 20. N Multiple slice images IMGs1 to IMGs N The computing device 40 is configured to be able to identify the type of object based on the learned model generated by machine learning. The computing device 40 includes a classifier 42 implemented based on a trained model generated by machine learning. The computing device 40 may include a plurality of classifiers 42 optimized for each range. The algorithm of the classifier 42 is not particularly limited, but may be YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution -SSD), Mask R-CNN, or the like, or may be an algorithm developed in the future.

[0181] The arithmetic processing device 40 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware). The arithmetic processing device 40 may be a combination of multiple processors. Alternatively, the arithmetic processing device 40 may be composed of hardware only. The functions of the arithmetic processing device 40 and the functions of the image processing device 28 may be implemented in the same processor.

[0182] 26(a) and (b) are diagrams showing an automobile 300 equipped with a gating camera 20. Referring to Fig. 26(a), the automobile 300 is equipped with headlamps (lamp fixtures) 302L and 302R.

[0183] As shown in Fig. 26(a), the lighting device 22 of the gating camera 20 may be built into at least one of the left and right headlamps 302L, 302R. The image sensor 24 may be attached to a part of the vehicle, for example, the back side of the rearview mirror. Alternatively, the image sensor 24 may be provided in the front grille or the front bumper. The camera controller 26 may be provided in the vehicle cabin, in the engine room, or built into the headlamps 302L, 302R.

[0184] As shown in FIG. 26(b), the image sensor 24 may be built in together with the lighting device 22 in either the left or right headlamp 302L, 302R.

[0185] 27 is a block diagram showing a vehicle lamp 200 equipped with a sensing system 10. The vehicle lamp 200 constitutes a lamp system 304 together with a vehicle-side ECU 310. The vehicle lamp 200 includes the lamp-side ECU 210 and a lamp unit 220. The lamp unit 220 is a low beam or high beam lamp, and includes a light source 222, a lighting circuit 224, and an optical system 226. The vehicle lamp 200 is further provided with a sensing system 10.

[0186] Information on the object OBJ detected by the sensing system 10 may be used for light distribution control of the vehicle lamp 200. Specifically, the lamp side ECU 210 generates an appropriate light distribution pattern based on the information on the type of object OBJ and its position generated by the sensing system 10. The lighting circuit 224 and the optical system 226 operate to obtain the light distribution pattern generated by the lamp side ECU 210. The arithmetic processing device 40 of the sensing system 10 may be provided outside the vehicle lamp 200, i.e., on the vehicle side.

[0187] Furthermore, information regarding the object OBJ detected by the sensing system 10 may be transmitted to the vehicle-side ECU 310. The vehicle-side ECU 310 may use this information for automatic driving or driving assistance.

[0188] The present invention has been described in terms of specific embodiments, but the embodiments merely show one aspect of the principles and applications of the present invention, and many modifications and changes in arrangement are possible within the scope of the invention as defined in the claims. [Industrial Applicability]

[0189] The present disclosure relates to a gating camera. [Explanation of symbols]

[0190] 10. Sensing System 20 Gating Camera 22 Lighting equipment 24 Image Sensor 26 Camera Controller 28 Image Processing Device 40 Processing unit 42 Classifier 200 Vehicle lighting fixtures 210 Lighting side ECU 220 Lamp unit 222 Light source 224 Lighting circuit 226 Optical system 300 Automobiles 302L Headlamp 304 Lighting System 310 Vehicle ECU L1 Probe light S1 Light emission timing signal S2 Exposure timing signal

Claims

1. A gating camera that divides a field of view into a plurality of ranges in a depth direction and generates a plurality of slice images corresponding to the plurality of ranges, an illumination device for irradiating the probe light; an image sensor that generates a raw image by exposure and outputs a compressed image obtained by compressing image information related to invalid lines that do not include a target among a plurality of lines that constitute the raw image; a camera controller for controlling a timing of light emission of the illumination device and a timing of exposure of the image sensor; an image processing device that receives the compressed image from the image sensor and restores the slice image; A gating camera comprising:

2. 2. The gating camera according to claim 1, wherein said image sensor designates a line as said invalid line when the pixel values ​​of all pixels of said line are smaller than a predetermined threshold value.

3. 3. The gating camera according to claim 1, wherein in the slice image, pixel values ​​of the invalid lines are zero.

4. The image sensor includes: For a valid line that is not an invalid line, transmit line data including a row number, a header whose value is a first value, and pixel values ​​of all pixels included in the valid line; 4. The gating camera according to claim 1, wherein for the invalid line, line data including a line number and a header whose value is a second value is transmitted.

5. 5. The gating camera according to claim 4, wherein said image processing device sets all pixel values ​​of said line to 0 when the value of said header of said received line data is said second value.

6. A gating camera that divides a field of view into a plurality of ranges in a depth direction and generates a plurality of slice images corresponding to the plurality of ranges, an illumination device for irradiating the probe light; An image sensor; a camera controller that controls a timing of light emission of the illumination device and a timing of exposure of the image sensor, and causes the image sensor to generate the plurality of slice images corresponding to the plurality of ranges; an image processing device that receives the plurality of slice images from the image sensor, selects a pixel of interest, defines a slice image including an object at the pixel of interest as a valid image, detects a noise level based on a pixel value of the pixel of interest in slice images other than the valid image, and corrects the pixel value of the pixel of interest in each of the plurality of slice images based on the noise level; A gating camera comprising:

7. 7. The gating camera according to claim 6, wherein the noise level is an average value of pixel values ​​of pixels of interest in slice images other than the effective image.

8. 8. The gating camera according to claim 6, wherein the image processing device determines the slice image in which the pixel of interest has the largest pixel value as the effective image.

9. 8. The gating camera according to claim 6, wherein said image processing device determines a slice image as said valid image when a pixel value of a pixel of interest in said slice image exceeds a predetermined threshold value.

10. A gating camera that divides a field of view into a plurality of ranges in a depth direction and generates a plurality of slice images corresponding to the plurality of ranges, an illumination device for irradiating the probe light; An image sensor; a camera controller that controls a timing of light emission of the illumination device and a timing of exposure of the image sensor, and causes the image sensor to generate a plurality of slice images corresponding to the plurality of ranges; an image processing device that receives the plurality of slice images from the image sensor, selects a pixel of interest, detects a noise level based on pixel values ​​of the pixel of interest in all slice images, and corrects the pixel value of the pixel of interest in each of the plurality of slice images based on the noise level; A gating camera comprising:

11. 11. The gating camera according to claim 10, wherein the noise level is an average value of pixel values ​​of pixels of interest in all the slice images.

12. A gating camera that divides a depth direction into a plurality of ranges and generates a plurality of slice images corresponding to the plurality of ranges, an illumination device for irradiating the probe light; An image sensor; a camera controller that controls a timing of light emission of the illumination device and a timing of exposure of the image sensor, and causes the image sensor to generate a plurality of image data corresponding to the plurality of ranges; Equipped with The gating camera is characterized in that the camera controller reflects visibility information correlated with the attenuation coefficient of the probe light in shooting parameters.

13. The gating camera according to claim 12 , wherein the camera controller increases the number of exposures required to generate one slice image as the attenuation coefficient of the probe light increases.

14. 14. The gating camera according to claim 13, wherein the camera controller holds a plurality of relationships between range numbers and exposure times, and selects one of the relationships according to the visibility information.

15. 15. The gating camera according to claim 12, wherein the camera controller increases the resolution in the depth direction when the attenuation coefficient is large, at the expense of giving up long-distance shooting.

16. 16. The gating camera according to claim 12, wherein the camera controller shortens the length of one range as the attenuation coefficient increases.

17. 17. The gating camera according to claim 12, wherein the camera controller reduces the number of ranges to be measured in order from the far side as the attenuation coefficient increases.

18. A method for controlling a gating camera that includes an illumination device that irradiates a probe light and an image sensor, and captures images by dividing a field of view into a plurality of ranges in a depth direction, comprising the steps of: acquiring visibility information correlated with the transmittance of the probe light in a field of view; determining a shooting parameter based on the visibility information; controlling the illumination device and the image sensor in response to the imaging parameters; A control method comprising:

19. A gating camera according to any one of claims 1 to 17; a processor for processing the slice images captured by the gating camera; A sensing system for a vehicle comprising:

20. A vehicle lamp comprising the gating camera according to any one of claims 1 to 17.

Images