Gating camera, automobile, vehicle lighting device, image processing device, and image processing method
The gating camera system effectively generates comprehensive images by controlling illumination and data transmission, addressing depth range and imaging time limitations, and reducing interference from pulsed light sources.
Patent Information
- Application Number
- JP2021548999
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-30
- Filing Date
- 2020-09-24
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2040-09-24
AI Technical Summary
Gating cameras face challenges in generating images that include the entire object of interest while maintaining a reduced depth of field, as expanding the depth range includes unwanted objects, and the transmission speed of images limits imaging time, and active sensors are affected by surrounding pulsed light sources.
The gating camera system employs a controlled illumination device, image sensor, and processing unit to generate slice images within specific ranges, adaptively control data transmission, and reduce the influence of pulsed light sources by varying light emission intervals.
This approach allows for generating images that include the entire object of interest while reducing the depth of field and imaging time, improving visibility and object recognition, and minimizing interference from surrounding light sources.
Smart Images

Figure 0007594539000001 
Figure 0007594539000002 
Figure 0007594539000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a gating camera. [Background technology]
[0002] For autonomous driving and automatic control of headlamp distribution, an object identification system is used to sense the position and type of objects around the vehicle. The object identification system includes a sensor and a processing unit that analyzes the sensor output. The sensor is selected from cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter wave radar, ultrasonic sonar, etc., taking into consideration the application, required accuracy, and cost.
[0003] A typical monocular camera cannot obtain depth information, so it is difficult to separate overlapping objects located at different distances.
[0004] A TOF camera is known as a camera that can obtain depth information. A TOF (Time Of Flight) camera projects infrared light using a light-emitting device, measures the flight time it takes for the reflected light to return to an image sensor, and obtains an image in which the flight time is converted into distance information.
[0005] The present applicant has proposed a gating camera (also called a gated camera) as an active sensor to replace the TOF camera (Patent Documents 1 and 2). A gating camera divides the imaging range into multiple imaging ranges, and captures images multiple times by changing the exposure timing and exposure time for each imaging range. This results in a slice image for each imaging range of the target, and each slice image contains only the object included in the corresponding imaging range. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2009-257983 A [Patent Document 2] International Publication WO2017 / 110413A1 Summary of the Invention [Problem to be solved by the invention]
[0007] 1. As a result of examining gating cameras, the present inventors have come to recognize the following problems. FIGS. 1(a) and 1(b) are diagrams for explaining problems that arise in gating cameras. FIG. 1(a) is an example of a scene photographed by a gating camera. If the depth of the photographing range is narrowed, an object will exist across multiple photographing ranges. FIG. 1(a) shows a state in which a car is photographed from the front with the depth of the photographing range set to 1.5 m. FIG. 1(b) shows multiple slice images obtained in the photographing scene of FIG. 1(a). Slice images IMGs corresponding to a certain photographing range i shows only a part of the car's bumper and hood, and another slice image IMGs i+1 Only part of the car's hood and the windshield are visible in the slice images. Therefore, it is difficult to recognize what the objects are from the individual slice images (IMGs).
[0008] This problem can be solved by extending the depth of the shooting range to several tens of meters. However, if the depth of the shooting range is made too large, the possibility that objects other than the target object will be included in the range increases, and the advantage of the gating camera is lost.
[0009] 1. The present disclosure has been made in consideration of such problems, and one exemplary purpose of an embodiment thereof is to provide a gating camera capable of generating an image that includes an entire object while reducing the depth of the shooting range.
[0010] 2. In the gating camera, the bottleneck is the image transmission speed from the image sensor to the image processor. In other words, although the frame rate of the image sensor can be set very high, the time required for the gating camera to generate slice images of the entire shooting range (hereinafter referred to as the shooting time of the gating camera) is limited by the transmission speed.
[0011] The present disclosure has been made in consideration of such problems, and an exemplary purpose of an embodiment of the present disclosure is to provide a gating camera capable of shortening image capture time.
[0012] 3. Active sensors, including gating cameras, irradiate an object with a probe light and detect the reflected light. Active sensors mainly comprise a projector (lighting) that irradiates the object with light, and an optical sensor that detects the reflected light from the object. Active sensors have the advantage that they can be more resistant to disturbances than passive sensors by matching the wavelength of the probe light with the wavelength range of sensitivity of the sensor.
[0013] If a pulsed light source that has the same wavelength as the illumination device of the active sensor and emits light at an integer multiple or an integer fraction period is present near the active sensor, the detection accuracy of the active sensor will be significantly reduced.
[0014] The present disclosure has been made in view of the above-mentioned problems, and one exemplary purpose of an embodiment of the present disclosure is to provide an active sensor capable of reducing the influence of a pulsed light source in the vicinity. [Means for solving the problem]
[0015] A summary of some exemplary embodiments of the present disclosure will be described. This summary is intended to provide a simplified summary of some concepts of one or more embodiments in order to provide a basic understanding of the embodiments as a prelude to the detailed description that follows, and is not intended to limit the scope of the invention or disclosure. Furthermore, this summary is not intended to be a comprehensive overview of all possible embodiments, nor is it intended to limit essential components of the embodiments. For convenience, the term "one embodiment" may be used to refer to one embodiment (example or variant) or multiple embodiments (examples or variants) disclosed in this specification.
[0016] 1. A gating camera according to one embodiment includes an illumination device that irradiates probe light, an image sensor, a camera controller that divides the depth direction into a plurality of shooting ranges and controls the timing of the illumination device's irradiation of the probe light and the timing of the image sensor's exposure while shifting the shooting ranges so that a slice image output by the image sensor at each shooting range includes only objects included in the corresponding shooting range, and an image processing unit that generates a final image including the entire object of interest (OOI) based on a plurality of slice images corresponding to the plurality of shooting ranges.
[0017] According to this aspect, it is possible to generate a final image that includes the entire target object while reducing the depth of the shooting range.
[0018] The final image may include only the target object. By removing the object other than the target object, the image may be more easily visible on a display screen, or later processing such as object recognition of the target object may be facilitated.
[0019] The image processing unit may synthesize a plurality of slice images to generate a composite image in which each pixel has distance information, and extract a region including the target object from the composite image. By referring to the distance information of the intermediate image, it becomes easy to separate the object.
[0020] A gating camera according to one embodiment includes an illumination device that irradiates probe light, an image sensor, a camera controller that divides the depth direction into a plurality of shooting ranges and controls the timing of the illumination device's irradiation of the probe light and the timing of the image sensor's exposure while shifting the shooting ranges so that a slice image output by the image sensor at each shooting range includes only objects included in the corresponding shooting range, and an image processing unit that combines a plurality of slice images corresponding to the plurality of shooting ranges to generate a composite image in which each pixel has distance information.
[0021] By referring to the distance information of the intermediate images, objects can be easily separated.
[0022] 2. A gating camera according to one embodiment includes an illumination device that irradiates a probe light, an image sensor, a camera controller that divides the depth direction into a plurality of shooting ranges and controls the timing of the illumination device's irradiation of the probe light and the timing of the image sensor's exposure while shifting the shooting ranges so that a slice image output by the image sensor in each shooting range includes only objects included in the corresponding shooting range, and an image processing unit that receives the slice image from the image sensor and processes the slice image. The camera controller is configured to be able to adaptively control the amount of data of the slice image transmitted from the image sensor to the image processing unit.
[0023] According to this aspect, the amount of data of slice images to be transmitted can be reduced depending on the situation, thereby shortening the transmission time and therefore the imaging time.
[0024] The camera controller may reduce the size of the slice image to be transmitted by cropping a part of the slice image using the image sensor or an interface (image transmission circuit) between the image sensor and the image processor. By cropping and transmitting only the necessary part, the amount of data can be reduced.
[0025] Instead of or in addition to cropping, the amount of data may be reduced by lowering the resolution through binning or thinning processing.
[0026] Since a close object moves relatively quickly in a slice image, even if a crop area is determined based on a certain point in time, there is a high possibility that the object will deviate from the crop area after that. On the other hand, a distant object moves relatively slowly in a slice image, so it will stay in the crop area for a long time. In addition, distant objects are likely to appear concentrated near the vanishing point. Therefore, slice image groups corresponding to shooting range groups closer than a predetermined distance are always transmitted without cropping, and slice image groups corresponding to shooting range groups farther than a predetermined distance may be controlled to be cropped or not. This makes it possible to transmit slice images including a target object.
[0027] The gating camera may transmit a plurality of slice images without cropping them, and determine whether or not to crop the subsequent plurality of slice images based on the plurality of uncropped slice images, thereby making it possible to predict and estimate the state of future slice images based on the uncropped slice images and determine whether or not to crop them.
[0028] In the first mode, all slice images corresponding to all imaging ranges are transmitted without cropping, and when only a single target object is included in a far slice image group corresponding to an imaging range group farther than a predetermined distance among the slice images transmitted in the first mode, a transition to a second mode may be made. In the second mode, a target region including the target object may be cropped for the far slice image group and transmitted.
[0029] The illumination device may be configured to change the irradiation range by concentrating and diffusing the probe light. The camera controller may cause the illumination device to concentrate the probe light when capturing a slice image to be cropped and transmitted. When capturing a distant shooting range, the probe light and its reflected light are attenuated to a large extent, so that a slice image with sufficient brightness cannot be generated by a single exposure. In such a case, for a distant shooting range, it is necessary to perform multiple exposures and composite the slice images obtained for each exposure to generate one slice image. By concentrating the probe light, the brightness of the slice image obtained by one exposure can be increased, so that the number of exposures and therefore the number of transmissions of the slice images can be reduced. This allows the shooting time to be shortened.
[0030] A gating camera according to an embodiment includes an illumination device that irradiates a probe light and that is configured to variably irradiate the probe light by concentrating and diffusing the probe light, an image sensor, a camera controller that divides the depth direction into a plurality of shooting ranges and controls the timing of irradiating the probe light by the illumination device and the timing of exposure by the image sensor while shifting the shooting ranges so that a slice image output by the image sensor in each shooting range includes only objects included in the corresponding shooting range, and an image processing unit that receives the slice image from the image sensor and processes the slice image. The camera controller controls the irradiation range of the illumination device based on the detection result of the image processing unit.
[0031] When shooting at a distant shooting range or in thick fog, the attenuation of the probe light and its reflected light increases. In this case, a slice image with sufficient brightness cannot be generated with a single exposure, so multiple exposures must be performed and the slice images obtained with each exposure must be composited to generate a single slice image. By concentrating the probe light, the brightness of the slice image obtained with one exposure can be increased, reducing the number of exposures and therefore the number of transmissions of the slice images. This reduces the shooting time.
[0032] In the first mode, the probe light may be irradiated to the entire irradiation range. When the image processing unit detects an object of interest in a slice image of a predetermined shooting range in the first mode, the image processing unit may transition to a second mode, and when the image processing unit captures the predetermined shooting range in the second mode, the probe light may be irradiated to a region of interest including the object of interest.
[0033] An active sensor according to one embodiment includes a light emitting device that emits pulsed light multiple times at non-uniform time intervals, and an optical sensor that detects reflected light from an object at a timing synchronized with each emission of light by the light emitting device, and accumulates and outputs the detection results from multiple detections.
[0034] Since the optical sensor is synchronized with the light emitting device in the same active sensor, even if the light emission timing changes, the reflected light from the object will be incident on the optical sensor at the same timing. Therefore, the detection value of the reflected light is integrated to obtain a large signal component. Since the light emission timing of the other noise light sources is asynchronous with the exposure timing of the optical sensor, the frequency with which the noise light from the noise light sources is incident on the optical sensor is included in the exposure timing of the optical sensor can be reduced. This reduces the influence of surrounding pulsed light sources.
[0035] A single sensing including a single emission of light from the light emitting device and a single exposure of the optical sensor is repeated in a predetermined cycle, and the emission timing of the light emitting device within the predetermined cycle may be changed for each sensing.
[0036] The active sensor may be a gating camera that is capable of acquiring a plurality of images corresponding to a plurality of ranges by dividing the depth direction into a plurality of ranges and changing the time difference between light emission and image capture for each range.
[0037] The active sensor may be a Time Of Flight (TOF) camera.
[0038] The active sensor may be a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging). Effect of the Invention
[0039] According to one embodiment, it is possible to generate an image including the entire target object while reducing the depth of the shooting range. According to one embodiment, it is possible to shorten the shooting time of the gating camera. According to one embodiment, it is possible to reduce the influence of surrounding active sensors. [Brief description of the drawings]
[0040] [Figure 1] 1(a) and (b) are diagrams illustrating a problem that occurs in a gating camera. [Diagram 2] FIG. 1 is a block diagram of a gating camera according to a first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a shooting operation of the gating camera. [Figure 4] FIG. 2 is a diagram showing an example of a photographed scene. [Diagram 5] 5A to IMGsi+2 out of all slices IMGs1 to IMGsN obtained in the running scene of FIG. 4. FIG. [Figure 6] FIG. 13 is a diagram showing a final image IMGf generated by an image processing unit. [Figure 7] 13 is a diagram illustrating a composite image IMGc generated by an image processing unit. FIG. [Figure 8] 8(a) to (c) are diagrams showing a specific example of the process of generating a composite image IMGc in the image processing unit. [Figure 9] 9(a) to (c) are diagrams illustrating the generation of a final image IMGf based on a composite image IMGc. [Figure 10] FIG. 11 is a block diagram of a gating camera according to a second embodiment. [Figure 11] FIG. 2 is a diagram illustrating a shooting operation of the gating camera. [Figure 12] 12(a) and (b) are diagrams for explaining an example of cropping processing of slice images IMGs. [Figure 13] 13(a) and (b) are diagrams for explaining the capturing and transmission of slice images. [Figure 14] 11 is a flowchart of an example of mode control in a gating camera. [Figure 15] FIG. 11 is a diagram showing an example of a situation in which a transition to a second mode occurs. [Figure 16] FIG. 11 is a block diagram of a gating camera according to a third embodiment. [Figure 17] FIG. 11 is a block diagram of a gating camera according to a fourth embodiment. [Figure 18] FIG. 11 is a block diagram of an active sensor according to a fifth embodiment. [Figure 19] 4 is a timing chart showing the relationship between the light emission timing of the light emitting device and the exposure timing of the optical sensor. [Figure 20] 11 is an operational waveform diagram of an active sensor according to a comparative technique. FIG. [Figure 21] 19 is an operational waveform diagram of the active sensor of FIG. 18. [Figure 22] FIG. 2 is a block diagram of a gating camera according to an embodiment. [Figure 23] FIG. 2 is a diagram illustrating the operation of a gating camera. [Figure 24] 24(a) and (b) are diagrams for explaining images obtained by the gating camera. [Diagram 25] 1 is a diagram showing a vehicle lamp incorporating a gating camera or an active sensor; [Figure 26] 1 is a block diagram showing a vehicle lamp equipped with an object identification system; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0041] The present invention will be described below based on preferred embodiments with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing are given the same reference numerals, and duplicated descriptions are omitted as appropriate. In addition, the embodiments are not intended to limit the invention, but are merely examples, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.
[0042] (Embodiment 1) 2 is a block diagram of the gating camera 20 according to the first embodiment. The gating camera 20 has a plurality of shooting ranges RNG1 to RNG2 (N≧2) in the depth direction. N For example, the depth of one shooting range is about 1 to 2 m (e.g., 1.5 m), and N=100.
[0043] The gating camera 20 includes an illumination device 22, an image sensor 24, a camera controller 26, and an image processing unit .
[0044] The lighting device 22 irradiates the probe light L1 to the front of the vehicle in synchronization with a light emission timing signal S1 provided from the camera controller 26. The probe light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength.
[0045] The image sensor 24 is configured to be capable of performing exposure control in synchronization with an exposure timing signal S2 provided from the camera controller 26, and to be capable of generating slice images IMGs. The image sensor 24 has sensitivity to the same wavelength as the probe light L1, captures reflected light (return light) L2 reflected by the object OBJ, and outputs slice images IMGs.
[0046] The camera controller 26 controls the shooting ranges RNG1 to RNG N The camera controller 26 stores predetermined light emission timing and exposure timing for each shooting range RNG. iWhen imaging the i-th slice image IMGs, a light emission timing signal S1 and an exposure timing signal S2 are generated based on the light emission timing and exposure timing corresponding to the imaging range, and imaging is performed. i The corresponding shooting range RNG i The object contained in the image will be captured.
[0047] In this embodiment, the camera controller 26 controls the shooting range RNG i The camera takes pictures while shifting the camera in the depth direction, and the entire shooting range RNG1~RNG N Regarding slice images IMGs1 to IMGs N Generate.
[0048] 3 is a diagram for explaining the photographing operation of the gating camera 20. In FIG. i The figure shows how the imaging range RNG is measured from the gating camera 20 as a range of interest (ROI). The lighting device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. i The distance to the boundary in front of MINi、 Shooting range RNG i The distance to the inner boundary of MAXi Let us assume that.
[0049] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi =2×d MINi / c where c is the speed of light.
[0050] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, T MAXi =2×d MAXi / c It is.
[0051] Shooting range RNG i When an object OBJ included in the image is to be photographed, the camera controller 26 photographs the object OBJ at time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.
[0052] i-th shooting range RNG i When capturing an image, multiple exposures may be performed (multiple exposures). In this case, the camera controller 26 may repeat the above-described exposure operation multiple times at a predetermined period τ2.
[0053] Returning to FIG. 2, the image processing unit 28 detects a plurality of shooting ranges RNG1 to RNG N Multiple slice images IMGs1 to IMGs N The image processing unit 28 receives a plurality of slice images IMGs1 to IMGs N Based on this, a final image IMGf containing the object of interest (OOI) is generated.
[0054] The above is the configuration of the gating camera 20. Next, the operation will be described.
[0055] 4 is a diagram showing an example of a shooting scene. A vehicle OBJ1 and a pedestrian OBJ2 exist in front of the gating camera 20. The vehicle OBJ1 is photographed at a plurality of shooting ranges RNG. i ~RNG i+2 The pedestrian OBJ2 is located at the shooting range RNG i is included.
[0056] FIG. 5 shows all slices IMGs1 to IMGs2 obtained in the driving scene of FIG. N Three slice images IMGs i ~IMGs i+2FIG. 1 is a diagram showing slice images IMGs i The corresponding shooting range RNG i Therefore, only the objects included in the slice image IMGs i There are pedestrians 2 and a part of vehicle OBJ1. A Contains sliced image IMGs i+1 Part of vehicle OBJ1 B Contains sliced image IMGs i+2 Part of vehicle OBJ1 C Includes:
[0057] For example, the image processing unit 28 processes all slice images IMGs1 to IMGs N In this example, the image processing unit 28 scans the slice images IMGs and sets candidates for the target object contained therein. i If a part of a vehicle is detected, the vehicle is set as the target object. N Scan again and create another part of the same vehicle OBJ1. B ,OBJ1 C Slice images IMGs containing i+1 ,IMGs i+2 Then, these slice images IMGs i ~IMGs i+2 Among them, the part OBJ1 that constitutes the same target object A ~OBJ1 C 6 is a diagram showing a final image IMGf generated by the image processing unit 28. Please note that this final image IMGf does not include a pedestrian OBJ2 that is not a target object.
[0058] The above is the operation of the gating camera 20. With this gating camera 20, it is possible to extract an object of interest while reducing the depth of the shooting range RNG, and generate a final image IMGf including the entire object.
[0059] The gating camera 20 has the advantage of being able to remove objects other than the target object from the final image. This advantage is particularly effective under certain circumstances, such as when it is snowing, in addition to the shooting scene shown in FIG. 4. When it is snowing, many small snowflakes are included in each slice image. The gating camera 20 can remove the snowflakes from the final image IMGf, so an image with excellent visibility can be generated.
[0060] Next, a description will be given of a specific example of the processing in the image processing unit 28. In the process of generating the final image IMGf, the image processing unit 28 may generate a composite image IMGc, and generate the final image IMGf based on the composite image IMGc.
[0061] 7 is a diagram for explaining a composite image IMGc generated by the image processing unit 28. The slice image IMGs and the intermediate image IMGc each have X×Y pixels. ij The intermediate image IMGc is expressed as the sum of all slice images IMGs1 to IMGs N Each pixel P of the intermediate image IMGc is ij is all slice images IMGs1 to IMGs N One of IMGs k (k=1 to N) corresponding pixel P ij Each pixel P ij is the slice image IMGs from which the brightness value is derived. k The identifier k can be regarded as distance information to the object. ij Instead of the identifier k, the distance d to the object from which the pixel originates may be used. ij is expressed in the form [L, k].
[0062] The format of the intermediate image IMGc is not particularly limited. For example, the intermediate image IMGc may include two pieces of image data: X×Y image data representing a luminance value, and X×Y image data representing distance information.
[0063] 8(a) to (c) are diagrams showing a specific example of the generation process of the composite image IMGc in the image processing unit 28. For ease of understanding, it is assumed here that N=6, and the gradation of each pixel is expressed by eight gradations (3 bits) from 0 to 7. In FIG. 8(a), a plurality of slice images IMGs1 to IMGs6 corresponding to the entire imaging range are shown.
[0064] The synthesis method is not particularly limited, but the following method is exemplified.
[0065] FIG. 8B shows an intermediate image IMGc obtained by the first generation method. In the first generation method, each pixel P ij The luminance values of the pixels P ij Among the luminance values, it has the maximum value (or a value obtained by calculating the maximum value).
[0066] FIG. 8(c) shows an intermediate image IMGc obtained by the second generation method. In the second generation method, each pixel P ij The luminance values of the pixels P ij Among the brightness values of the slice images having valid values, the brightness value of the slice image closest to the front (i.e., the closest distance) (or a value obtained by calculating the brightness value) is included. A valid value may be non-zero or may be a value greater than a predetermined threshold value.
[0067] The above is a description of the composite image IMGc. Figures 9(a) to 9(c) are diagrams for explaining the generation of the final image IMGf based on the composite image IMGc.
[0068] The composite image IMGc contains an identifier k that indicates the distance to the object in addition to the luminance value L, which makes it easier to separate the target object OOI, because an algorithm for the object extraction process can be created under the premise that pixels separated by identifier k are likely to belong to different objects, and pixels with close identifier k are likely to belong to the same object.
[0069] When a plurality of target objects OOI1, OOI2 are included in the composite image IMGc, final images IMGf1, IMGf2 may be generated for each target object, as shown in FIG. 9(b).
[0070] Alternatively, when a plurality of target objects OOI1, OOI2 are included in the composite image IMGc, a single final image IMGf including the plurality of target objects may be generated as shown in FIG. 9(c).
[0071] Once the composite image IMGc is generated, the original slice images IMGc1 to IMGs are N This has the advantage of reducing memory capacity.
[0072] (Embodiment 2) 10 is a block diagram of the gating camera 20 according to the second embodiment. The gating camera 20 has a plurality of shooting ranges RNG1 to RNG2 (N≧2) in the depth direction. N For example, the depth d of one shooting range is about 1 to 2 m (for example, 1.5 m), and N=100.
[0073] The gating camera 20 includes an illumination device 22, an image sensor 24, a camera controller 32, and an image processing unit .
[0074] The lighting device 22 irradiates the probe light L1 to the front of the vehicle in synchronization with a light emission timing signal S1 provided from the camera controller 32. The probe light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength.
[0075] The image sensor 24 is capable of performing exposure control in synchronization with an exposure timing signal S2 provided from the camera controller 32, and is configured to be capable of generating slice images IMGs. The image sensor 24 has sensitivity to the same wavelength as the probe light L1, captures reflected light (return light) L2 reflected by the object OBJ, and outputs slice images IMGs.
[0076] The camera controller 32 controls the shooting ranges RNG1 to RNG N The camera controller 32 stores predetermined light emission timing and exposure timing for each shooting range RNG. i When imaging the i-th slice image IMGs, a light emission timing signal S1 and an exposure timing signal S2 are generated based on the light emission timing and exposure timing corresponding to the imaging range, and imaging is performed. i The corresponding shooting range RNG i The object contained in the image will be captured.
[0077] In this embodiment, the camera controller 32 controls the shooting range RNG i The camera takes pictures while shifting the camera in the depth direction, and the entire shooting range RNG1~RNG N Regarding slice images IMGs1 to IMGs N Generate.
[0078] 11 is a diagram for explaining the shooting operation of the gating camera 20. In FIG. i The illumination device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. The imaging range RNG from the gating camera 20 is i The distance to the boundary in front of MINi、 Shooting range RNG i The distance to the inner boundary of MAXi Let us assume that.
[0079] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi =2×d MINi / c where c is the speed of light.
[0080] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, T MAXi =2×d MAXi / c It is.
[0081] Shooting range RNG i When an object OBJ included in the image is to be photographed, the camera controller 32 photographs the object OBJ at time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.
[0082] i-th shooting range RNG i When capturing an image, multiple exposures may be performed (multiple exposures). In this case, the camera controller 32 may repeat the above-described exposure operation multiple times at a predetermined period τ2.
[0083] Returning to FIG. 10, the image processing unit 34 receives the slice images IMGs from the image sensor 24. i (i=1,2,…N) and receive the slice images IMGs i Process.
[0084] The camera controller 32 and the image processor 34 may be implemented as separate pieces of hardware or as a single piece of hardware. For example, the camera controller 32 and the image processor 34 may be implemented as a combination of a processor 30 such as a microcomputer and a software program executed thereby.
[0085] The image sensor 24 and the arithmetic processing device 30 are connected via an interface 26. The interface 26 transmits slice images IMGs from the image sensor 24 to the image processing unit 34. The type of the interface 26 is not particularly limited, but for example, a Mobile Industry Processor Interface (MIPI) or a Universal Serial Bus (USB) can be adopted.
[0086] In this embodiment, the camera controller 32 controls at least one of the image sensor 24 and the interface 26, and is configured to be able to adaptively control the amount of data of the slice images IMGs transmitted from the image sensor 24 to the image processing unit .
[0087] In this embodiment, the camera controller 32 reduces the size (number of pixels) of the slice images IMGs to be transmitted by cropping a part of the slice images IMGs in the image sensor 24 or the interface 26, thereby reducing the amount of data. The camera controller 32 generates a control signal S3 that specifies whether or not to crop. This control signal S3 may include information on the size and position of the area to be cropped.
[0088] The cropping method is not particularly limited, and a part of the slice images IMGs may be cropped in the image sensor 24. That is, the image itself read by the image sensor 24 may be cropped. Alternatively, a part of the slice images IMGs may be cropped in the interface 26. That is, all pixels may be read from the image sensor 24, and a part of the area of the read full-size image may be cropped.
[0089] 12(a) and (b) are diagrams for explaining an example of cropping processing of slice images IMGs. As shown in FIG. 12(a), when an object OBJ is likely to exist only in a part of a slice image IMGs obtained by the image sensor 24, a range (region of interest ROI) including the object OBJ is cropped, and the cropped slice image (cropped image) IMGs' is transmitted.
[0090] As shown in FIG. 12(b), when there is a high possibility that one or more objects OBJ exist over a wide range of slice images IMGs obtained by the image sensor 24, the original slice images IMGs are transmitted without being cropped.
[0091] Since nearby objects move relatively quickly within a slice image, even if a crop area is determined based on a certain point in time, there is a high possibility that the object will deviate from the crop area after that. Therefore, in this embodiment, nearby slice images are always transmitted without being cropped. This makes it possible to prevent losing sight of the object.
[0092] On the other hand, distant objects move relatively slowly through the slice images, so they tend to stay in the crop region for a long time. Distant objects are also more likely to appear concentrated near the vanishing point.
[0093] Therefore, in this embodiment, slice image groups IMGs1 to IMGs2 corresponding to imaging range groups (10 to 100 m) closer than a predetermined distance (for example, 100 m) are Y The slice image group IMGs corresponding to the shooting range group (100-150m) farther than the specified distance (100m) is always transmitted without cropping. Y+1 ~IMGs N For each of the above, the presence or absence of cropping may be controlled.
[0094] It should be noted that the arithmetic processing device 30 cannot see the slice images IMGs "before transmission" obtained by the image sensor 24. Therefore, in this embodiment, the camera controller 32 needs to estimate the state of the current slice images IMGs based on the non-cropped slice images IMGs previously received by the image processing unit 34, and determine whether or not to crop.
[0095] Therefore, the gating camera 20 captures all slice images IMGs1 to IMGs N The uncropped slice images IMGs1 to IMG N Based on this, the subsequent slice image groups IMGs1 to IMGs N In this way, it is possible to predict and estimate the state of future slice images based on the uncropped slice images, and to determine whether or not to crop the slice images. At this time, in addition to determining whether or not to crop, the size and position of the crop area may also be determined.
[0096] The above is the basic configuration of the gating camera 20. Next, the operation will be described.
[0097] 13(a) and (b) are diagrams for explaining the capturing and transmission of slice images. In Fig. 13(a), for all imaging ranges, the entire slice images captured by the image sensor 24 are transmitted as they are.
[0098] 13B, in at least one shooting range RNG, a part of the slice image IMGs captured by the image sensor 24 is cropped, and the cropped slice image IMGs' is transmitted. Since the data amount of the cropped slice image IMGs' is smaller than the data amount of the uncropped slice image IMGs, the transmission time is shortened.
[0099] The above is the operation of the gating camera 20. Next, its advantages will be explained. The shooting time of the gating camera 20 can be understood as the sum of the exposure time and the transmission time of the slice images in all shooting ranges. According to the gating camera 20, by reducing the amount of data of the slice images IMGs transmitted from the image sensor 24 to the image processing unit 34 depending on the situation, the transmission time can be shortened, and the shooting time of the gating camera 20 can be shortened.
[0100] In particular, the size of the slice image is reduced by cropping a part of the slice image. By cropping and transmitting only the necessary part, the amount of data can be reduced. Here, instead of cropping, the amount of data may be reduced by binning or pixel thinning. In this case, however, the resolution of the slice image input to the image processing unit 34 is reduced, and the accuracy of the object recognition process in the subsequent stage may be reduced. In contrast, if cropping is used, the resolution is not reduced, so there is no need to worry about a decrease in the accuracy of object recognition.
[0101] Next, a specific example of control over whether or not to crop will be described.
[0102] The gating camera 20 can be switched between a first mode and a second mode. In the first mode, all of the shooting ranges RNG1 to RNG N All slice images IMGs1 to IMGs N In the second mode, the short-distance shooting range RNG1~RNG Y Slice image groups IMGs1 to IMGs Y is transmitted without cropping, and the long-distance shooting range RNG Y+1 ~RNG N Slice image groups IMGs corresponding to Y+1 ~IMGs N is cropped and transmitted.
[0103] The transition to the second mode is determined based on the slice images IMGs1 to IMGs2 transmitted in the first mode. NAmong them, the shooting range group RNG that is farther than a specified distance Y+1 ~RNG N The corresponding far slice image groups IMGs Y+1 ~IMGs N Specifically, the far slice image group IMGs Y+1 ~IMGs N If the image contains only a single object of interest, the system transitions to the second mode. In the second mode, the part containing the object of interest is set as the region of interest, and only the part of interest is cropped and transmitted.
[0104] FIG. 14 is a flowchart of an example of mode control in the gating camera 20. First, the first mode is set (S100). As described above, in the first mode, all slice images IMGs1 to IMGs N is input to the image processing unit 34 (S102).
[0105] The image processor 34 outputs the far slice image group IMGs Y+1 ~IMGs Y+N If only a single object of interest (OOI) is included in the image (Y in S104), the gating camera 20 is switched to the second mode (S106). If not (N in S104), the gating camera 20 remains in the first mode and returns to process S102.
[0106] For example, in step S104, when the image processing unit 34 detects an object of interest in the long-distance slice image group IMGs, it calculates the position of the object of interest and sets the long-distance shooting range RNG Y+1 ~RNG N If it is determined that no object other than the target object is present, the mode may transition to the second mode.
[0107] 15 is a diagram showing an example of a situation in which the gating camera 20 transitions to the second mode. The gating camera 20 is mounted on a vehicle 800. An oncoming vehicle 802 is present ahead of the vehicle 800. The oncoming vehicle 802 is in the far range RNG Y+1 ~RNG N Object of interest that exists in the far range RNG Y+1~RNG N There is no target object other than the oncoming vehicle 802. When the gating camera 20 detects such a situation, it transitions to the second mode.
[0108] Returning to FIG. 14, when moving to the second mode (S106), the size and position of the ROI including the target object detected in the process S104 are determined. During the second mode, the far slice image group IMGs Y+1 ~IMGs N The ROI is cropped and the slice images (IMGs) containing the ROI are Y+1 ~IMGs N is transmitted.
[0109] As described above, in the second mode, the close-distance slice images IMGs1 to IMGs Y is transmitted in its original uncropped size.
[0110] The second mode is maintained while the condition for returning to the first mode is not satisfied (N in S108). When the condition for returning to the first mode is satisfied (Y in S108), the mode returns to the first mode S100.
[0111] The condition for returning to the first mode may be the passage of a predetermined time or the long-distance shooting range RNG Y+1 ~RNG N This may be that a plurality of target objects are present in the long distance shooting range RNG, or that the original target object is no longer present in the long distance shooting range RNG.
[0112] (Embodiment 3) 16 is a block diagram of a gating camera 20A according to embodiment 3. In the gating camera 20A, the illumination device 22 is configured to be able to vary the irradiation range by collecting and diffusing the probe light L1. The camera controller 32 is able to adaptively control the irradiation range of the illumination device 22 by a control signal S4.
[0113] When capturing slice images IMGs to be cropped and transmitted, the camera controller 32 causes the illumination device 22 to focus the probe light L1 to narrow the irradiation range.
[0114] For example, in the second mode, the camera controller 32 may Y+1 ~IMGs N When capturing an image, the probe light L1 is focused and irradiated intensively onto the area of interest. This increases the illuminance of the irradiated area, making it possible to obtain a clear image.
[0115] The above is the configuration of the gating camera 20A. Next, its advantages will be described.
[0116] When imaging a distant imaging range, the attenuation of the probe light L1 and its reflected light L2 becomes large. Therefore, there are cases where a slice image IMGs with sufficient brightness cannot be generated with one exposure. In such a case, for a distant imaging range, it is necessary to perform multiple exposures and composite the slice images obtained for each exposure to generate one slice image. In this embodiment, by concentrating the probe light L1, the brightness of the slice image IMGs obtained with one exposure can be increased, so that the number of exposures and therefore the number of transmissions of the slice images can be reduced. This allows the imaging time to be shortened.
[0117] (Embodiment 4) 17 is a block diagram of a gating camera 20B according to the fourth embodiment. In the gating camera 20B, only the irradiation range of the lighting device 22 is controlled, and no cropping is performed during transmission of the slice image. The control of the lighting device 22 is the same as in the third embodiment. This gating camera 20B can also reduce the number of exposures when shooting a distant shooting range, and can obtain a clear slice image.
[0118] 18 is a block diagram of an active sensor 70 according to embodiment 5. The active sensor 70 is a gating camera, a ToF camera, a LIDAR, or the like, and includes a light emitting device 72, an optical sensor 74, and a controller 76.
[0119] The light emitting device 72 emits pulsed light multiple times during one sensing operation. In this embodiment, the time interval T INT The time interval T INT may be determined randomly for each period, or may vary according to a predetermined pattern. For example, the time interval T INT may monotonically increase or decrease for each operation period Tc.
[0120] Light L1 emitted from the light emitting device 72 is reflected by an object OBJ and enters the optical sensor 74. The reflected light L2 is delayed by τ with respect to the emitted light L1. τ depends on the distance z to the object OBJ and is expressed by the formula (1). τ is called the round trip time of light. τ = 2 × z / c … (1) c represents the speed of light.
[0121] The exposure timing and exposure time of the optical sensor 74 are controlled so that the optical sensor 74 can detect each pulse contained in the reflected light L1 in synchronization with each emission of the light emitting device 72. The emission timing of the light emitting device 72 and the exposure timing of the optical sensor 74 are controlled by the controller 74.
[0122] Reflected light L2 from an object is incident multiple times on the light emitting sensor 74 in response to multiple emission of light from the light emitting device 72. The light sensor 74 integrates the reflected light received multiple times and outputs a signal according to the integrated value.
[0123] 19 is a time chart showing the relationship between the light emission timing of the light emitting device 72 and the exposure timing of the optical sensor 74. For example, the start time of the exposure of the optical sensor 74 may be delayed by a certain time Td from the start time of the light emission of the light emitting device 72. The method of determining the time difference Td depends on the type and method of the active sensor 70, and may be determined based on, for example, the round trip time τ to the object to be photographed. The exposure time Te may be the same as the light emission time (pulse width) Tp of the light emitting device 72, or it may be longer than that.
[0124] One sensing including one emission of the light emitting device 72 and one exposure of the optical sensor may be repeated in a predetermined cycle Tc. In this case, the emission timing of the light emitting device 72 in the predetermined cycle Tc (the time Tx from the beginning of the cycle Tc to the emission of light) may be changed for each sensing. The time interval T between the i-th and (i+1)-th emission timings INTi is expressed by equation (2). T INTi =Tc+Tx i+1 -Tx i …(2)
[0125] The above is the configuration of the active sensor 70. Next, the operation of the active sensor 70 will be described.
[0126] In order to clarify the advantages of the active sensor 70, the comparative technique will be described first. In the comparative technique, the light emitting device emits light at regular time intervals, and the light sensor is also exposed at regular time intervals. FIG. 20 is an operational waveform diagram of the active sensor according to the comparative technique. Assume that there is another light source near the active sensor that emits pulsed light at a period Tc' that is an integer multiple of the operating period Tc or an integer multiple of the operating period Tc. In FIG. 20, the light of this other light source is shown as disturbance light L3, and in this example, the light emission period Tc' of the disturbance light is equal to the operating period Tc of the active sensor.
[0127] When each pulse of the disturbance light L3 is included in the exposure period Te, it affects the sensing of the active sensor. INTis constant (Tc), and therefore the exposure timing interval is also constant (Tc).
[0128] Therefore, each pulse of the disturbance light L3 is always included in the exposure time Te, and the influence thereof is accumulated and appears as a large error. In Fig. 20, the error component is indicated by hatching.
[0129] Now, the following describes the operation of the active sensor 70 according to embodiment 5. FIG 21 is an operational waveform diagram of the active sensor 70 in FIG.
[0130] As described above, when each pulse of the disturbance light L3 is included in the exposure period Te, it affects the sensing of the active sensor 70. INT is not uniform, and therefore the exposure start time differs for each operation cycle Tc. Therefore, each pulse of the disturbance light L3 is sometimes included in the exposure time Te and sometimes not. In the example of FIG. 20, the first pulse of the disturbance light L3 is erroneously detected, but the subsequent pulses are not erroneously detected because they are outside the exposure period. Therefore, the influence of the disturbance light L3 can be reduced, and highly accurate detection is possible.
[0131] Next, a description will be given of the use of the active sensor 70. One embodiment of the active sensor 70 is a gating camera.
[0132] 22 is a block diagram of the gating camera 20 according to an embodiment. The gating camera 20 has a plurality of ranges RNG1 to RNG2 (N≧2) in the depth direction. N The imaging is performed in separate sections.
[0133] The gating camera 20 includes an illumination device 22, an image sensor 24, a camera controller 26, and an image processing unit 28. The illumination device 22 corresponds to the light emitting device 72 in FIG. 18, the image sensor 24 corresponds to the light sensor 74 in FIG. 18, and the camera controller 26 corresponds to the controller 76 in FIG.
[0134] The lighting device 22 irradiates the vehicle front with a probe light L1 including a plurality of pulses in synchronization with a light emission timing signal S1 provided by the camera controller 26. The probe light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength. As described above, the time intervals of the pulses are non-uniform.
[0135] The image sensor 24 is capable of performing exposure control in synchronization with an exposure timing signal S2 provided from the camera controller 26, and is configured to be capable of generating a slice image IMG. The image sensor 24 has sensitivity to the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ.
[0136] The camera controller 26 holds a light emission timing and an exposure timing that are predetermined for each range RNG. i When shooting a range, the gating camera 20 generates a light emission timing signal S1 and an exposure timing signal S2 based on the light emission timing and exposure timing corresponding to the range, and shoots the image. N Multiple slice images IMG1 to IMG N The i-th slice image IMG i has the corresponding range RNG i The object contained in the image will be captured.
[0137] 23 is a diagram for explaining the operation of the gating camera 20. In FIG. 23, the i-th range RNG i The illumination device 22 emits light for a light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. The top row shows a diagram of light rays with time on the horizontal axis and distance on the vertical axis. The range RNG from the gating camera 20 is i The distance to the boundary in front of MINi、 Range RNG i The distance to the inner boundary of MAXi Let us assume that.
[0138] Light that leaves the lighting device 22 at a certain time travels a distance d MINi The round trip time T MINi teeth, T MINi =2×d MINi / c where c is the speed of light.
[0139] Similarly, light that leaves the lighting device 22 at a certain time travels a distance d MAXi The round trip time T MAXi teeth, T MAXi =2×d MAXi / c It is.
[0140] Range RNG i When an object OBJ included in the image is to be photographed, the camera controller 26 photographs the object OBJ at time t2=t0+T MINi Exposure begins at time t3=t1+T MAXi The exposure timing signal S2 is generated so as to end the exposure at the specified timing. This is one exposure operation.
[0141] i-th range RNG i When capturing an image, light emission and exposure are repeated multiple times, and the image sensor 24 accumulates the measurement results.
[0142] 24(a) and (b) are diagrams for explaining images obtained by the gating camera 20. In the example of FIG. 24(a), an object (pedestrian) OBJ1 exists in range RNG1, and an object (vehicle) OBJ3 exists in range RNG3. FIG. 24(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 24(a). When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, so that the object image OBJ1 of the pedestrian OBJ1 appears in the slice image IMG1.
[0143] When the slice image IMG2 is captured, the image sensor is exposed to the reflected light from the range RNG2, and therefore, no object image is captured in the slice image IMG2.
[0144] Similarly, when the slice image IMG3 is captured, the image sensor is exposed to the reflected light from the range RNG3, so that only the object image OBJ3 appears in the slice image IMG3. In this way, the gating camera 20 can capture the object separately for each range.
[0145] The above is the operation of the gating camera 20. In this gating camera, by making the time intervals of light emission from the lighting device 22 non-uniform, it is possible to reduce the influence of surrounding pulsed light sources and obtain clear images with fewer noise components.
[0146] A modification related to the third embodiment will be described.
[0147] (Variation 1) The time interval T of the pulsed light emission of the light-emitting device 72 INT The method of changing is not limited to that shown in Fig. 19. For example, Tx may be kept constant and the operation period Tc may be changed for each cycle.
[0148] (Variation 2) The active sensor 70 is not limited to a gating camera, and may be a Time Of Flight (TOF) camera or a Light Detection and Ranging (LIDAR, Laser Imaging Detection and Ranging).
[0149] 25 is a diagram showing a vehicle lamp 200 incorporating a gating camera 20 or an active sensor 70 (hereinafter, described as a gating camera). The vehicle lamp 200 includes a housing 210, an outer lens 220, high beam and low beam lamp units 230H / 230L, and a gating camera 20. The lamp units 230H / 230L and the gating camera 20 are housed in the housing 210.
[0150] It should be noted that a part of the gating camera 20, for example the image sensor 24, may be installed outside the vehicle lamp 200, for example on the back side of the rearview mirror.
[0151] 26 is a block diagram showing a vehicle lamp 200 equipped with an object identification system 10. The vehicle lamp 200 constitutes a lighting system 310 together with a vehicle-side ECU 304. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. The vehicle lamp 200 is further provided with an object identification system 10. The object identification system 10 includes a gating camera 20 (or an active sensor 70) and a processing device 40.
[0152] The arithmetic processing device 40 detects a plurality of shooting ranges RNG1 to RNG2 obtained by the gating camera 20. N Multiple slice images IMG1 to IMG N The computing device 40 is configured to be able to identify the type of object based on the prediction model generated by machine learning. The arithmetic processing device 40 includes a classifier implemented based on a prediction model generated by machine learning. The algorithm of the classifier is not particularly limited, but may be YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution -SSD), Mask R-CNN, or the like, or may be an algorithm to be developed in the future.
[0153] The arithmetic processing device 40 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware). The arithmetic processing device 40 may be a combination of multiple processors. Alternatively, the arithmetic processing device 40 may be configured only with hardware.
[0154] Information about the object OBJ detected by the object identification system 10 may be used for light distribution control of the vehicle lamp 200. Specifically, the lamp side ECU 208 generates an appropriate light distribution pattern based on the information about the type and position of the object OBJ generated by the object identification system 10. The lighting circuit 204 and the optical system 206 operate to obtain the light distribution pattern generated by the lamp side ECU 208.
[0155] Furthermore, information regarding the object OBJ detected by the arithmetic processing device 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU 304 may use this information for automatic driving or driving assistance.
[0156] The present invention has been described in terms of specific embodiments, but the embodiments merely show one aspect of the principles and applications of the present invention, and many modifications and changes in arrangement are possible within the scope of the invention as defined in the claims. [Industrial Applicability]
[0157] The present invention relates to a gating camera. [Explanation of symbols]
[0158] 10 Object Identification System 20 Gating Camera 22 Lighting equipment 24 Image Sensor 26 Camera Controller 28 Image Processing Section S1 Light emission timing signal S2 Exposure timing signal 40 Processing unit 70 Active Sensor 72 Light emitting device 74 Optical Sensor 76 Controller 200 Vehicle lighting fixtures 210 Cabinet 220 Outer Lens 230 Lighting unit 230 310 Lighting System 304 Vehicle ECU 200 Vehicle lighting fixtures 202 Light source 204 Lighting circuit 206 Optical system 310 Lighting System 304 Vehicle ECU
Claims
1. an illumination device for irradiating the probe light; An image sensor; a camera controller for acquiring a plurality of slice images each including only an object included in each of a plurality of imaging ranges by varying a timing of irradiation of the probe light by the illumination device and a timing of exposure by the image sensor for each of a plurality of imaging ranges separated in a depth direction; an image processing unit that judges whether a predetermined target object is included across a plurality of slice images, and when it is judged that the target object is included, synthesizes the plurality of slice images to generate a final image that includes the entire target object and removes objects that are included in only one of the plurality of slice images; A gating camera comprising:
2. The gating camera of claim 1, characterized in that the specified object of interest is a vehicle, and the object included in only one of the plurality of slice images is a person.
3. an illumination device for irradiating the probe light; An image sensor; a camera controller for acquiring a plurality of slice images each including only an object included in each of a plurality of imaging ranges by varying a timing of irradiation of the probe light by the illumination device and a timing of exposure by the image sensor for each of a plurality of imaging ranges separated in a depth direction; an image processing unit that synthesizes the plurality of slice images to generate a composite image and generates a final image based on the composite image; Equipped with Each of the plurality of pixels forming the composite image includes luminance value information indicating one luminance value selected from a plurality of luminance values of corresponding pixels in the plurality of slice images so as to satisfy a predetermined condition, and distance information indicating a distance corresponding to one of the plurality of slice images including a pixel having the one luminance value, A gating camera, wherein the final image includes at least one object of interest extracted based on the distance information.
4. A gating camera according to any one of claims 1 to 3; a processor for processing an output of the gating camera; A vehicle comprising:
5. A vehicle lamp comprising the gating camera according to any one of claims 1 to 3.
6. An image processing device for a gating camera, comprising: The gating camera includes: an illumination device for irradiating the probe light; An image sensor; a camera controller for acquiring a plurality of slice images each including only an object included in each of a plurality of imaging ranges by varying a timing of irradiation of the probe light by the illumination device and a timing of exposure by the image sensor for each of a plurality of imaging ranges separated in a depth direction; Equipped with The image processing device includes: An image processing device which determines whether a specified object of interest is contained across multiple slice images, and if it is determined that the object of interest is contained across multiple slice images, synthesizes the multiple slice images to generate a final image that includes the entire object of interest and removes any object that is contained in only one of the multiple slice images.
7. An image processing method for a gating camera, comprising: The gating camera includes: an illumination device for irradiating the probe light; An image sensor; a camera controller for acquiring a plurality of slice images each including only an object included in each of a plurality of imaging ranges by varying a timing of irradiation of the probe light by the illumination device and a timing of exposure by the image sensor for each of a plurality of imaging ranges separated in a depth direction; Equipped with The image processing method includes: A step of determining whether a predetermined object of interest is included across a plurality of slice images; determining whether a predetermined object of interest is included across a plurality of slice images, and if it is determined that the object of interest is included across a plurality of slice images, synthesizing the plurality of slice images to generate a final image that includes the entire object of interest and removes any object that is included in only one of the plurality of slice images; An image processing method comprising:
8. An image processing device for a gating camera, comprising: The gating camera includes: an illumination device for irradiating the probe light; An image sensor; a camera controller for acquiring a plurality of slice images each including only an object included in each of a plurality of imaging ranges by varying a timing of irradiation of the probe light by the illumination device and a timing of exposure by the image sensor for each of a plurality of imaging ranges separated in a depth direction; Equipped with The image processing device includes: generating a composite image by synthesizing the plurality of slice images, and generating a final image based on the composite image; Each of the plurality of pixels forming the composite image includes luminance value information indicating one luminance value selected from a plurality of luminance values of corresponding pixels in the plurality of slice images so as to satisfy a predetermined condition, and distance information indicating a distance corresponding to one of the plurality of slice images including a pixel having the one luminance value, 4. An image processing apparatus according to claim 1, wherein the final image includes at least one target object extracted based on the distance information.
9. An image processing method for a gating camera, comprising: The gating camera includes: an illumination device for irradiating the probe light; An image sensor; a camera controller for acquiring a plurality of slice images each including only an object included in each of a plurality of imaging ranges by varying a timing of irradiation of the probe light by the illumination device and a timing of exposure by the image sensor for each of a plurality of imaging ranges separated in a depth direction; Equipped with The image processing method includes: synthesizing the plurality of slice images to generate a composite image; generating a final image based on the composite image; Equipped with Each of the plurality of pixels forming the composite image includes luminance value information indicating one luminance value selected from a plurality of luminance values of corresponding pixels in the plurality of slice images so as to satisfy a predetermined condition, and distance information indicating a distance corresponding to one of the plurality of slice images including a pixel having the one luminance value, The image processing method according to claim 1, wherein the final image includes at least one target object extracted based on the distance information.
Citation Information
Patent Citations
Device and method for generating distance image data for vehicle
JP2009257983A
Image processing apparatus, image capturing apparatus, image processing method, and program
JP2014197824A
Gated imaging using applicable depth of field
JP2015527761A
High Dynamic Range Imaging of Environment with a High Intensity Reflecting / Transmitting Source
US20170234976A1
Gated imaging apparatus, system and method
US20190056498A1
