Gating camera, vehicle sensing system, vehicle lighting device

Abstract

This gating camera divides a depth direction into a plurality of ranges and generates a plurality of slice images corresponding to the plurality of ranges. An illumination apparatus performs irradiation of probe light. A controller controls a light emitting timing of the illumination apparatus and a timing of exposure of an image sensor. An image processing apparatus generates a slice image IMG on the basis of a sensor image SI transmitted from the image sensor. The image processing apparatus selects M (M≥2) images in an ascending order of pixel values from each line of the sensor image SI to calculate an average value AVE of the pixel values, and subtracts the average value from each pixel value in the same line.

Description

Technical Field

[0001] The present disclosure relates to a gating camera.

Background Art

[0002] For automatic driving and automatic control of the light distribution of headlights, an object recognition system for sensing the position and type of objects existing around a vehicle is used. The object recognition system includes a sensor and an arithmetic processing unit that analyzes the output of the sensor. The sensor is selected from among cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter-wave radar, ultrasonic sonar, etc., in consideration of the application, required accuracy, and cost.

[0003] Ordinary monocular cameras cannot obtain depth information. Therefore, when a plurality of objects located at different distances overlap, it is difficult to separate them.

[0004] As a camera that can obtain depth information, a TOF camera is known. A TOF (Time Of Flight) camera projects infrared light by a light-emitting device, measures the flight time until the reflected light returns to the image sensor, and obtains a TOF image in which the flight time is converted into distance information.

[0005] An active sensor that replaces a TOF camera (hereinafter referred to as a gating camera or a gated camera in this specification) has been proposed (Patent Documents 1 and 2). The gating camera divides the imaging range into a plurality of ranges, changes the exposure timing and exposure time for each range, and captures an image. As a result, sliced images are obtained for each target range, and each sliced image includes only the objects included in the corresponding range.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] 1. The inventor recognized that when operating a gating camera, horizontal stripe noise randomly occurs in the output image (sensor image) of the image sensor, which may deteriorate the image quality of the slice image.

[0008] One aspect of the present disclosure has been made in such a situation, and one of its exemplary purposes is to provide a gating camera capable of suppressing deterioration of a slice image.

[0009] 2. A gating camera includes an image sensor and an image processing device that processes the output image (sensor image) of the image sensor. Here, the connection between the image sensor and the image processing device is often through a serial interface. The transmission speed of the sensor image during this period becomes a bottleneck, the time required to generate one slice image becomes long, and the frame rate of the gating camera is limited.

[0010] One aspect of the present disclosure has been made in such a situation, and one of its exemplary purposes is to provide a gating camera capable of shortening the generation time of a slice image.

[0011] 3. A conventional gating camera performs one exposure for one emission. Therefore, for each emission, only a slice image of one range is generated. Thus, in order to sense all N ranges, N sets of emission and exposure are required, and the sensing time becomes long. When repeatedly integrating emission and exposure multiple times when photographing a distant range, the sensing time becomes even longer.

[0012] Certain aspects of the present disclosure have been made in such circumstances, and one of the exemplary purposes is to provide a gating camera capable of shortening the sensing time.

Means for Solving the Problems

[0013] 1. The gating camera according to an aspect of the present disclosure divides into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates probe light, an image sensor, a controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor, and an image processing device that generates a slice image based on the sensor image transmitted from the image sensor. The image processing device selects M (M≧2) in ascending order of pixel values for each line of the sensor image and calculates their average value, and subtracts the average value from each pixel value in the same line.

[0014] 2. Certain aspects of the present disclosure relate to a gating camera that divides into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates probe light, an image sensor including a plurality of pixels, a controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor, and an image processing device that generates a slice image based on the sensor image transmitted from the image sensor. The resolution of the sensor image transmitted from the image sensor to the image processing device is lower as the range is closer.

[0015] 3. Certain aspects of the present disclosure relate to a gating camera that divides into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates probe light, an image sensor including a plurality of pixels, and a controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor. The plurality of pixels of the image sensor are classified into a plurality of pixel groups, and the controller exposes the plurality of pixel groups at different timings for one light emission of the illumination device.

Advantages of the Invention

[0016] According to Aspect 1 of the present disclosure, the image quality of the slice image can be improved. According to Aspect 2 of the present disclosure, the generation time of the slice image can be shortened. According to Aspect 3 of the present disclosure, the sensing time can be shortened.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

MODE FOR CARRYING OUT THE INVENTION

[0018] (Outline of Embodiment) An overview of some exemplary embodiments of the present disclosure will be described. This overview simplifies and describes some concepts of one or more embodiments for the purpose of providing a basic understanding of the embodiments as a prelude to the detailed description below, and does not limit the scope of the invention or disclosure. Also, this overview is not an exhaustive overview of all possible embodiments and does not limit essential components of the embodiments. For convenience, the term "one embodiment" may be used herein to refer to one embodiment (example or variation) or a plurality of embodiments (examples or variations) disclosed herein.

[0019] This overview is not an extensive overview of all possible embodiments, nor is it intended to identify essential elements of all embodiments or to delineate the scope of some or all aspects. Its sole purpose is to present, in a simplified form, some concepts of one or more embodiments as a prelude to the more detailed description presented later.

[0020] 1. A gating camera according to one embodiment divides a depth direction into a plurality of ranges and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates probe light, an image sensor, a controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor, and an image processing device that generates a slice image based on a sensor image transmitted from the image sensor. The image processing device selects M (M≥2) in ascending order of pixel values for each line of the sensor image, calculates their average value, and subtracts the average value from each pixel value of the same line.

[0021] According to this configuration, horizontal stripe noise can be reduced and image quality can be improved.

[0022] In one embodiment, M may be 2 to 8% of the number of pixels in a line.

[0023] 2. The gating camera according to one embodiment divides a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates probe light, an image sensor that includes a plurality of pixels, a controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor, and an image processing device that generates a slice image based on the sensor image transmitted from the image sensor. The resolution of the sensor image transmitted from the image sensor to the image processing device is lower as the range is closer.

[0024] When photographing the same subject with the gating camera, when the subject is in a far range, the subject appears small, that is, it is imaged with low resolution, and when the subject is in a near range, the subject appears large, that is, it is imaged with high resolution. Therefore, the closer the distance to the subject, in other words, the closer the range, the lower the resolution of the sensor image transmitted from the image sensor to the image processing device, so that while maintaining the resolution required in the subsequent processing, the transmission time of the sensor image can be shortened, and the generation time of the slice image can be shortened.

[0025] In one embodiment, for a plurality of pixels, the image sensor can specify whether to transmit line by line, and the number of skipped lines may be larger for closer ranges.

[0026] In one embodiment, for a plurality of pixels, the image sensor can specify whether to transmit column by column, and the number of skipped columns may be larger for closer ranges.

[0027] In one embodiment, for a plurality of pixels, the image sensor can specify whether to transmit pixel by pixel, and the number of skipped pixels may be larger for closer ranges.

[0028] In one embodiment, all pixels may be exposed in all ranges of the image sensor.

[0029] In one embodiment, in each range, only the pixels to be transmitted may be exposed.

[0030] In one embodiment, a plurality of pixels of the image sensor may be classified into a plurality of groups. The controller may expose the plurality of groups at different timings for one emission of the illumination device. Thereby, a plurality of ranges can be photographed in parallel.

[0031] In one embodiment, the image processing apparatus may scale the sensor image transmitted from the image sensor so that slice images having the same aspect ratio are obtained for each range. The scaling may use interpolation processing or decimation processing.

[0032] A gating camera according to one embodiment divides into a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges. The gating camera includes an illumination device that irradiates probe light, an image sensor that includes a plurality of pixels, and a controller that controls the emission timing of the illumination device and the exposure timing of the image sensor. The plurality of pixels of the image sensor are classified into a plurality of pixel groups, and the controller exposes the plurality of pixel groups at different timings for one emission of the illumination device.

[0033] According to one embodiment, images of a plurality of ranges can be generated for one emission. Therefore, the sensing time required to generate slice images of all ranges can be shortened.

[0034] In one embodiment, when the number of the plurality of pixel groups is n (n≥2), the i-th pixel group may include the (i + n×j)-th line (j is an integer).

[0035] In one embodiment, when the number of the plurality of pixel groups is n (n≥2), the i-th pixel group may include the (i + n×j)-th column (j is an integer).

[0036] In one embodiment, the gating camera may further include an image processing device that generates a slice image based on a sensor image transmitted from an image sensor. The image sensor may transmit the sensor image generated for each pixel group as a unit to the image processing device.

[0037] (Embodiment) Hereinafter, embodiments will be described with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing are denoted by the same reference numerals, and redundant descriptions are omitted as appropriate. Also, the embodiments are illustrative and not restrictive of the invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.

[0038] (Embodiment 1) FIG. 1 is a block diagram of a sensing system 10 according to Embodiment 1. This sensing system 10 is mounted on a vehicle such as an automobile or a motorcycle, and detects an object OBJ existing around the vehicle.

[0039] The sensing system 10 mainly includes a gating camera 20. The gating camera 20 includes an illumination device 22, an image sensor 24, a controller 26, and an image processing device 28. Imaging by the gating camera 20 is performed by dividing the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG N and performing it. Adjacent ranges may overlap in the depth direction at their boundaries.

[0040] The illumination device 22 irradiates the front of the vehicle with probe light L1 in synchronization with a light emission timing signal S1 given from the controller 26. The probe light L1 is preferably infrared light, but is not limited thereto, and may be visible light or ultraviolet light having a predetermined wavelength.

[0041] The image sensor 24 includes a plurality of pixels, is capable of performing exposure control synchronized with the exposure timing signal S2 given from the controller 26, and generates a sensor image SI. The image sensor 24 has sensitivity at the same wavelength as the probe light L1, and captures the reflected light (return light) L2 reflected by the object OBJ. The sensor image obtained for the i-th range RNG i is denoted as SI i .

[0042] The controller 26 controls the irradiation timing (light emission timing) of the probe light L1 by the illumination device 22 and the exposure timing by the image sensor 24. The controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a microcomputer, and a software program executed by the processor (hardware).

[0043] The image sensor 24 and the image processing device 28 are connected via a serial interface, and the sensor image SI captured by the image sensor 24 i is transmitted to the image processing device 28. The image processing device 28 generates a slice image IMG i based on the sensor image SI i transmitted from the image sensor 24.

[0044] FIG. 2 is a diagram for explaining the basic operation of the gating camera 20. FIG. 2 shows the state when sensing the i-th range RNG i . The illumination device 22 emits light during the light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. In the uppermost row, a diagram of a light ray with time on the horizontal axis and distance on the vertical axis is shown. From the gating camera 20, the distance to the boundary in front of the range RNG i is d MINi、 The distance to the boundary on the back side of the range RNG i is d MAXi .

[0045] The round-trip time T from when the light emitted by the lighting device 22 at a certain time reaches the distance d MINi and the reflected light returns to the image sensor 24 MINi is T MINi = 2×d MINi / c where c is the speed of light

[0046] Similarly, the round-trip time T from when the light emitted by the lighting device 22 at a certain time reaches the distance d MAXi and the reflected light returns to the image sensor 24 MAXi is T MAXi = 2×d MAXi / c where c is the speed of light

[0047] When it is desired to photograph only the object OBJ included in the range RNG i the controller 26 generates an exposure timing signal S2 so as to start the exposure at time t2 = t0 + T MINi and end the exposure at time t3 = t1 + T MAXi This is one exposure operation

[0048] When photographing the i-th range RNG i a plurality of sets of light emission and exposure may be performed. In this case, the controller 26 may repeat the above-described exposure operation a plurality of times at a predetermined period τ2

[0049] FIGS. 3(a) and (b) are diagrams for explaining the slice images obtained by the gating camera 20. In the example of FIG. 3(a), an object (pedestrian) OBJ2 exists in the range RNG2, and an object (vehicle) OBJ3 exists in the range RNG3. FIG. 3(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 3(a). When photographing the slice image IMG1, since the image sensor is exposed only by the reflected light from the range RNG1, no object image appears in the slice image IMG1

[0050] When taking the slice image IMG2, since the image sensor is exposed only by the reflected light from the range RNG2, only the object image OBJ2 appears in the slice image IMG2. Similarly, when taking the slice image IMG3, since the image sensor is exposed only by the reflected light from the range RNG3, only the object image OBJ3 appears in the slice image IMG3. Thus, according to the gating camera 20, objects can be separated and photographed for each range.

[0051] The image processing in the image processing apparatus 28 will be described. FIG. 4 is a diagram for explaining the noise of the sensor image SI. The sensor image SI includes noise in the form of horizontal stripes along the lines. The lines where the noise occurs are random, and the noise levels are also random.

[0052] The image processing apparatus 28 calculates the average value of M (M≧2) pixels with small pixel values for each line of the sensor image SI, and subtracts the average value from the pixel values of the pixels included in the same line.

[0053] M is preferably 2% to 8% of the number of pixels in a line, and can be, for example, about 5%.

[0054] FIG. 5 is a diagram for explaining the noise cancellation by the image processing apparatus 28. FIG. 5 shows the processing of one line. Refer to the line data which is a set of pixel values of a certain line, and select M pixel values in ascending order of pixel values. In this example, M = 8, and the pixel values 10, 10, 11, 11, 12, 13, 14, 15 are selected. The image processing apparatus 28 calculates the average value AVE (12 in this example) of the selected M pixel values. Then, the average value AVE is subtracted from the corresponding line data of the original sensor image SI.

[0055] FIG. 6 is a diagram showing sensor images before and after noise subtraction processing. FIG. 7 is a diagram showing sensor images before and after noise subtraction processing. In FIG. 6, an image that does not include an object (subject) is targeted, and in FIG. 7, an image that includes an object is targeted. In this example, the average value of the number of pixels M corresponding to 5% of each line is calculated. It can be seen that only background noise can be suitably removed whether the object is included or the image includes the object.

[0056] (Embodiment 2) FIG. 8 is a block diagram of the sensing system 10 according to Embodiment 2. This sensing system 10 is mounted on a vehicle such as an automobile or a motorcycle, and detects an object OBJ existing around the vehicle.

[0057] The sensing system 10 mainly includes a gating camera 20. The gating camera 20 includes an illumination device 22, an image sensor 24, a controller 26, and an image processing device 28. Imaging by the gating camera 20 is performed by dividing the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG N and may overlap in the depth direction at the boundaries between adjacent ranges.

[0058] The illumination device 22 irradiates the vehicle front with the probe light L1 in synchronization with the light emission timing signal S1 given from the controller 26. The probe light L1 is preferably infrared light, but is not limited thereto, and may be visible light or ultraviolet light having a predetermined wavelength.

[0059] The image sensor 24 includes a plurality of pixels, can perform exposure control in synchronization with the exposure timing signal S2 given from the controller 26, and generates a sensor image SI. The image sensor 24 has sensitivity to the same wavelength as the probe light L1, and photographs the reflected light (return light) L2 reflected by the object OBJ. The sensor image obtained for the i-th range RNG i is denoted as SI i .

[0060] The controller 26 controls the irradiation timing (light emission timing) of the probe light L1 by the lighting device 22 and the exposure timing by the image sensor 24.

[0061] The image sensor 24 and the image processing device 28 are connected via a serial interface, and the sensor image SI captured by the image sensor 24 i is transmitted to the image processing device 28. The image processing device 28 generates a slice image IMG i based on the sensor image SI i transmitted from the image sensor 24.

[0062] FIG. 9 is a diagram for explaining the basic operation of the gating camera 20. FIG. 9 shows a state when sensing the i-th range RNG i . The lighting device 22 emits light during the light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. At the uppermost stage, a diagram of a light ray with time on the horizontal axis and distance on the vertical axis is shown. From the gating camera 20, the distance to the boundary in front of the range RNG i is d MINi、 range RNG i and the distance to the boundary on the back side of the range RNG MAXi is d

[0063] The round-trip time T from when the light emitted from the lighting device 22 at a certain time reaches the distance d MINi and the reflected light returns to the image sensor 24 is MINi T T MINi = 2×d MINi / c where c is the speed of light.

[0064] Similarly, the round-trip time T from when the light emitted from the lighting device 22 at a certain time reaches the distance d MAXi and the reflected light returns to the image sensor 24 is MAXi T T MAXi = 2×d MAXi / c is.

[0065] Range RNG i When only the object OBJ included in the range RNG is to be photographed, the controller 26 starts exposure at time t2 = t0 + T MINi and ends the exposure at time t3 = t1 + T MAXi to generate an exposure timing signal S2. This is one exposure operation.

[0066] the i-th range RNG i When photographing, a plurality of sets of light emission and exposure may be performed. In this case, the controller 26 may repeat the above-described exposure operation a plurality of times at a predetermined period τ2.

[0067] FIGS. 10(a) and (b) are diagrams for explaining the slice images obtained by the gating camera 20. In the example of FIG. 10(a), an object (pedestrian) OBJ2 exists in the range RNG2, and an object (vehicle) OBJ3 exists in the range RNG3. FIG. 10(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of FIG. 10(a). When photographing the slice image IMG1, since the image sensor is exposed only by the reflected light from the range RNG1, no object image is shown in the slice image IMG1.

[0068] When photographing the slice image IMG2, since the image sensor is exposed only by the reflected light from the range RNG2, only the object image OBJ2 is shown in the slice image IMG2. Similarly, when photographing the slice image IMG3, since the image sensor is exposed only by the reflected light from the range RNG3, only the object image OBJ3 is shown in the slice image IMG3. Thus, according to the gating camera 20, objects can be separated and photographed for each range.

[0069] Return to FIG. 1. In this embodiment, the resolution (i.e., the number of pixels) of the sensor image SI transmitted from the image sensor 24 to the image processing device 28 changes according to the range. Specifically, the closer the range, the lower the resolution of the sensor image SI (fewer pixels), and the farther the range, the higher the resolution of the sensor image SI (more pixels). In this embodiment, in all ranges, exposure is performed on all pixels of the image sensor 24, only the necessary pixels are read out from all pixels, and the unnecessary pixels are decimated to generate the sensor image SI. The controller 26 generates a control signal S3 that instructs the pixels to be transmitted for each range and supplies it to the image sensor 24.

[0070] The controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware).

[0071] FIG. 11 is a diagram for explaining an example of controlling the resolution of the sensor image SI depending on the range. The hatched pixels indicate the effective pixels (effective lines) constituting the sensor image SI, and the white pixels indicate the invalid pixels (invalid lines) that are not transmitted. In this example, the vertical resolution of the sensor image SI to be transmitted, that is, the number of lines, is controlled according to the range. For example, in the farthest range, all lines of the image sensor 24 are made effective, and all pixels are transmitted as effective pixels. As the range approaches, the ratio (number) of decimated lines increases, and the number of effective pixels decreases.

[0072] FIG. 12(a) is a diagram showing a certain driving scene, and FIG. 12(b) is a sensor image SI x , SI y obtained in the driving scene of FIG. 12(a).

[0073] In the driving scene of Fig. 12(a), a preceding vehicle OBJ1 exists at a position close to the host vehicle on the driving lane, and a oncoming vehicle OBJ2 exists at a position far from the host vehicle on the oncoming lane. The preceding vehicle OBJ1 is included in the x-th range RNG x and the oncoming vehicle OBJ2 is assumed to be included in the y-th range RNG y .

[0074] In the sensor image SI x of Fig. 12(b), the preceding vehicle OBJ1 in the range RNG x is shown, and in the sensor image SI y , the oncoming vehicle OBJ2 in the range RNG y is shown. The horizontal line in Fig. 12(b) indicates the effective lines constituting the sensor image.

[0075] The resolution of the sensor image SI in each range may be determined so that when the same object (a vehicle in this example) is imaged, it crosses the same number of effective lines.

[0076] Fig. 13(a) is a time chart showing the operation of the gating camera 20. Fig. 13(b) is a time chart showing the operation of the gating camera according to the comparative technique.

[0077] First, referring to Fig. 13(b), the comparative technique will be described. In the comparative technique, sensor images SI1 to SI3 with the same resolution are transmitted for all ranges. In this case, all the sensor images SI1, SI2, and SI3 have the same transmission time, and the time required for sensing the three ranges RNG1 to RNG3 included in one cycle becomes long.

[0078] Next, referring to Fig. 13(a), the operation of the gating camera 20 according to Embodiment 2 will be described. In this example, the number of ranges is 3, and range RNG1 covers 0 to 25 m, range RNG2 covers 25 to 50 m, and range RNG3 covers 50 to 100 m. For the farthest third range RNG3, all lines are made valid. Since the distance of the second range RNG2 is approximately half of the distance of the third range RNG3, the resolution of the sensor image SI is set to 1 / 2. Since the distance of the first range RNG1 is 1 / 4 of the distance of the third range RNG3, the resolution of the sensor image SI is set to 1 / 4. In this case, specifically, for the sensor image SI3, all lines of the image sensor are valid lines. For the sensor image SI2, valid lines are selected at a ratio of 1 in 2 lines, and for the sensor image SI1, valid lines are selected at a ratio of 1 in 4 lines.

[0079] Since the number of pixels of the sensor images SI1, SI2, and SI3 is proportional to the number of lines, a relationship of approximately 1:2:4 holds. Therefore, the transmission time of the sensor image SI1 is shortened to 1 / 4 of the transmission time of the sensor image SI3, and the transmission time of the sensor image SI2 is shortened to 1 / 2 of the transmission time of the sensor image SI3. Therefore, the time required for sensing the three ranges RNG1 to RNG3 included in one cycle can be shortened.

[0080] Note that the sensor images SI1 and SI2 generated by line skipping as described above are compressed in the vertical direction compared to the sensor image SI3, and have different aspect ratios. Therefore, the image processing device 28 may generate the slice images IMG1 to IMG3 by equalizing the aspect ratios of the sensor images SI1 to SI3 through image processing.

[0081] For example, the image processing device 28 may j complement the invalid lines thinned out during transmission for the sensor image SI by image processing. In this case, all the slice images IMG1 to IMG3 can have the same resolution.

[0082] Alternatively, the image processing device 28 may, during transmission, for the sensor image SI jWhen the vertical resolution of [the object] is multiplied by X (X < 1), the horizontal resolution may be multiplied by X. That is, the image processing apparatus 28 may generate a slice image IMGi by thinning out a plurality of columns of the received sensor image SI j from the received sensor image SI.

[0083] In the above description, the vertical resolution of the sensor image is variable, but this is not the only case. FIG. 14 is a diagram for explaining another example of controlling the resolution of the sensor image SI depending on the range. In this example, the horizontal resolution of the sensor image SI to be transmitted, that is, the number of columns, is controlled according to the range. For example, in the farthest range, all columns of the image sensor 24 are made effective, and all pixels are transmitted as effective pixels. As the range approaches, the ratio (number) of thinned-out columns increases, and the number of effective pixels decreases.

[0084] FIG. 15 is a diagram for explaining yet another example of controlling the resolution of the sensor image SI depending on the range. In this example, the horizontal and vertical resolutions of the sensor image SI to be transmitted are controlled according to the range.

[0085] Subsequently, a modified example of the gating camera 20 will be described.

[0086] (Modified Example 1) In the above description, regardless of the range, all pixels of the image sensor 24 are exposed, and the resolution of the sensor image SI is controlled by selecting the lines and columns to be read out, but this is not the only case. Depending on the specifications of the image sensor 24, there are those that can control the presence or absence of exposure for each line, each column, or each pixel. When an image sensor 24 with such specifications is adopted, only the effective pixels to be transmitted may be exposed in each range.

[0087] (Modified Example 2) When using the image sensor 24 capable of exposure control for each pixel as in the first modification example, in a range where the utilization rate of pixels (lines / columns) is low, the pixels (lines / columns) can be divided into a plurality of groups and used for imaging in different ranges. FIG. 16 is a block diagram of the gating camera 20 according to the second modification example. For one emission of the illumination device 22, the first pixel group (line group or column group) of the image sensor 24 is exposed at the first timing, and the second pixel group (line group or column group) of the image sensor 24 is exposed at the second timing. The sensor image formed by the first pixel group and the sensor image SI i ,SI j are images of different ranges RNG i ,RNG j taken.

[0088] The controller 26 generates a light emission timing signal S1 for the illumination device 22, and generates an exposure timing signal S2A for the first pixel group so that an object in a certain range RNG i is exposed, and generates an exposure timing signal S2B for the second pixel group so that an object in another range RNG j is exposed.

[0089] FIG. 17 is a diagram for explaining the exposure of the gating camera 20 according to the second modification example. In this example, the utilization rate of pixels (lines) during imaging of one range is 50%. All the pixels constituting the image sensor 24 are classified into a first pixel group of odd lines and a second pixel group of even lines for each row, and exposure control is performed at different timings.

[0090] FIG. 18 is a time chart for explaining the operation of the gating camera 20 according to the second modification example. The first pixel group is assigned to the relatively close i-th range RNG i , and the second pixel group is assigned to the j-th range RNG jIt is assigned to (j>i). The lighting device 22 emits light during the light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. At the uppermost stage, a diagram of light rays with time on the horizontal axis and distance d on the vertical axis is shown. From the gating camera 20 to the range RNG i The distance to the boundary in front is d MINi、 Range RNG i The distance to the boundary on the back side of the range RNG is d MAXi Similarly, from the gating camera 20 to the range RNG j The distance to the boundary in front is d MINj、 Range RNG i The distance to the boundary on the back side of the range RNG is d MAXj Let it be.

[0091] Range RNG i To photograph the object OBJ included in i At time t2 = t0 + T MINi The exposure of the first pixel group starts, and at time t3 = t1 + T MAXi An exposure timing signal S2A is generated so that the exposure ends.

[0092] Also, using the same probe light, to photograph the object OBJ included in the range RNG j At time t4 = t0 + T j The exposure of the second pixel group starts, and at time t5 = t1 + T MINj An exposure timing signal S2B is generated so that the exposure ends. MAXj T MINj = 2×d MINj / c T MAXj = 2×d MAXj / c

[0093] Note that the two ranges RNG i ,RNG j that are photographed simultaneously may be adjacent (j = i + 1).

[0094] According to this modification, when photographing a short range, the transmission time can be shortened by reducing the resolution of the sensor image SI.

[0095] In addition, since it becomes possible to acquire images of two ranges with one emission of the lighting device 22, the utilization efficiency of hardware resources is increased.

[0096] Generally, in a situation where the utilization efficiency of pixels is 1 / N, it is also possible to classify the pixels of the image sensor 24 into N pixel groups and perform shooting of N ranges. The classification of pixel groups is not limited to line units, and may be in column units or pixel units.

[0097] (Embodiment 3)

[0098] FIG. 19 is a block diagram of a sensing system 10 according to an embodiment. This sensing system 10 is mounted on a vehicle such as an automobile or a motorcycle, and detects an object OBJ existing around the vehicle.

[0099] The sensing system 10 mainly includes a gating camera 20. The gating camera 20 includes a lighting device 22, an image sensor 24, a controller 26, and an image processing device 28. Imaging by the gating camera 20 is performed by dividing it into a plurality of N (N≧2) ranges RNG1 to RNG N in the depth direction, and a slice image is generated for each range. Adjacent ranges may overlap in the depth direction at their boundaries.

[0100] The image sensor 24 includes a plurality of pixels, and the exposure timing is configured to be controllable individually for each pixel, each line, or each column. The plurality of pixels are classified into a plurality of pixel groups. In the following description, the number of pixel groups is assumed to be 2, and they are referred to as the first pixel group and the second pixel group.

[0101] The gating camera 20 exposes the first pixel group (line group or column group) of the image sensor 24 at a first timing and exposes the second pixel group (line group or column group) of the image sensor 24 at a second timing for one emission of the lighting device 22. The sensor image formed by the first pixel group and the sensor image SI i ,SI j are images taken of different ranges RNG i ,RNG j .

[0102] Specifically, the controller 26 exposes the first pixel group and the second pixel group at different timings for one emission of the lighting device 22.

[0103] The controller 26 can be implemented as a combination of a processor (hardware) such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or microcomputer, and a software program executed by the processor (hardware).

[0104] The image sensor 24 and the image processing device 28 are connected via a serial interface, and the sensor image SI taken by the first pixel group of the image sensor 24 i and the sensor image SI taken by the second pixel group of the image sensor 24 j may be transmitted as separate images. The image processing device 28 generates a slice image IMG i based on the sensor image SI transmitted from the image sensor 24. i The image processing in the image processing device 28 is not particularly limited, and for example, interpolation processing or decimation processing for changing the aspect ratio may be performed.

[0105] the i-th range RNG i and the j-th range RNG jWhen taking a picture, multiple sets of light emission and exposure may be performed. In this case, the controller 26 may repeat the light emission and exposure operations multiple times. In this case, the image processing device 28 may synthesize a plurality of sensor images SI i obtained for the same range RNG i to generate a single slice image IMG i .

[0106] The above is the configuration of the gating camera 20. Next, its operation will be described.

[0107] FIG. 20 is a time chart for explaining the operation of the gating camera 20 of FIG. 19. The first pixel group is assigned to the relatively close i-th range RNG i , and the second pixel group is assigned to the j-th range RNG j (j>i). The lighting device 22 emits light during the light emission period τ1 between times t0 and t1 in synchronization with the light emission timing signal S1. A diagram of a light ray with time on the horizontal axis and distance d on the vertical axis is shown in the uppermost row. From the gating camera 20, the distance to the boundary in front of the range RNG i is d MINi、 range RNG i The distance to the boundary on the back side of is d MAXi . Similarly, from the gating camera 20, the distance to the boundary in front of the range RNG j is d MINj、 range RNG i The distance to the boundary on the back side of is d MAXj .

[0108] The round-trip time T MINi from the time when the light emitted from the lighting device 22 reaches the distance d MINi and the reflected light returns to the image sensor 24 is T MINi =2×d MINi / c . c is the speed of light.

[0109] Similarly, the round-trip time from the time when the light emitted from the lighting device 22 reaches the distance d MAXiThe round-trip time T until it reaches there and the reflected light returns to the image sensor 24 MAXi is T MAXi = 2×d MAXi / c where.

[0110] When only the object OBJ included in the range RNG i is to be photographed, the controller 26 starts the exposure at time t2 = t0 + T MINi and ends the exposure at time t3 = t1 + T MAXi to generate an exposure timing signal S2. This is one exposure operation.

[0111] Therefore, in order to photograph the object OBJ i included in the range RNG i the exposure of the first pixel group starts at time t2 = t0 + T MINi and the exposure timing signal S2A is generated so that the exposure ends at time t3 = t1 + T MAXi

[0112] Also, using the same probe light, in order to photograph the object OBJ j included in the range RNG j the exposure of the second pixel group starts at time t4 = t0 + T MINj and the exposure timing signal S2B is generated so that the exposure ends at time t5 = t1 + T MAXj T MINj = 2×d MINj / c T MAXj = 2×d MAXj / c

[0113] Note that the two ranges RNG i , RNG j to be photographed simultaneously may be adjacent (j = i + 1).

[0114] ​​Figs. 21(a) and (b) are diagrams for explaining the slice images obtained by the gating camera 20 of Fig. 19. In the example of Fig. 21(a), an object (pedestrian) OBJ2 exists in range RNG2, and an object (vehicle) OBJ3 exists in range RNG3. Fig. 21(b) shows a plurality of slice images IMG1 to IMG3 obtained in the situation of Fig. 21(a). When taking the slice image IMG1, since the image sensor is exposed only by the reflected light from range RNG1, no object image is shown in the slice image IMG1.

[0115] Suppose range RNG2 is assigned to the first pixel group and range RNG3 is assigned to the second pixel group. At this time, since the first pixel group is exposed only by the reflected light from range RNG2, only the object image OBJ2 is shown in the sensor image SI2. Similarly, since the second pixel group is exposed only by the reflected light from range RNG3, only the object image OBJ3 is shown in the sensor image SI3. Thus, according to the gating camera 20, objects can be separated and photographed for each range.

[0116] With one emission of the lighting device 22, images of two ranges RNG i ,RNG j can be acquired, so that the sensing time required to photograph all ranges can be shortened. Also, in a system that photographs only one range with one emission, the reflected light from objects other than the range to be photographed is wasted. In this embodiment, however, since the reflected light from objects in another range is also detected, the energy utilization efficiency is increased.

[0117] Subsequently, the division of the pixel group will be described. Figs. 22(a) to (d) are diagrams showing examples of the pixel group. In Fig. 22(a), it is divided into the first pixel group and the second pixel group skipping one line. When the number of a plurality of pixel groups is n (n ≥ 2), the i-th pixel group includes the (i + n × j)-th line (j is an integer). This example is suitable for an image sensor capable of controlling the exposure timing for each line.

[0118] In FIG. 22(b), they are assigned to the first pixel group and the second pixel group skipping one column. When the number of a plurality of pixel groups is n (n ≥ 2), the i-th pixel group includes the (i + n×j)-th column (j is an integer). This example is suitable for an image sensor capable of controlling the exposure timing for each column.

[0119] In FIG. 22(c), the left half of all the pixels is assigned to the first pixel group and the right half is assigned to the second pixel group. As a modification, the upper half of all the pixels may be assigned to the first pixel group and the lower half may be assigned to the second pixel group.

[0120] In FIG. 22(d), the number of pixel groups is 4. All the pixels are divided into blocks each including four adjacent pixels, and by selecting one pixel from each block, the first pixel group to the fourth pixel group are formed.

[0121] The techniques described in Embodiments 1 to 3 can be implemented in any combination.

[0122] (Application) FIG. 23 is a block diagram of a sensing system 10. The sensing system 10 includes an arithmetic processing unit 40 in addition to any one of the gating cameras 20 described in Embodiments 1 to 3. This sensing system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (also referred to as category or class) of an object OBJ existing around the vehicle.

[0123] The gating camera 20 generates a plurality of slice images IMG1 to IMG corresponding to a plurality of ranges RNG1 to RNG. N The i-th slice image IMG N shows only the objects included in the corresponding range RNG. i The corresponding range RNG i

[0124] The arithmetic processing unit 40 processes a plurality of slice images IMG1 to IMG corresponding to a plurality of ranges RNG1 to RNG obtained by the gating camera 20. N NBased on this, it is configured to be able to identify the type of the object. The arithmetic processing unit 40 includes a classifier 42 implemented based on a learned model generated by machine learning. The arithmetic processing unit 40 may include a plurality of classifiers 42 optimized for each range. The algorithm of the classifier 42 is not particularly limited, and YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution -SSD), Mask R-CNN, etc. can be adopted, or an algorithm developed in the future can be adopted.

[0125] The arithmetic processing unit 40 can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a microcomputer, and a software program executed by the processor (hardware). The arithmetic processing unit 40 may be a combination of a plurality of processors. Alternatively, the arithmetic processing unit 40 may be composed of only hardware. The functions of the arithmetic processing unit 40 and the image processing device 28 may be implemented by the same processor.

[0126] Figs. 24(a) and (b) are diagrams showing an automobile 300 equipped with a gating camera 20. Referring to Fig. 24(a), the automobile 300 is provided with headlamps (lamps) 302L and 302R. In the automobile 300 of Fig. 24(a), one lighting device 22 is provided at the center of the vehicle, and an image sensor 24 is built into one or both of the left and right headlamps 302L and 302R. The position of the lighting device 22 is not particularly limited. For example, it may be provided on the front bumper (i) or the front grille (ii), or may be attached to the back side of the rearview mirror inside the front window (iii). Also, the position of the controller 26 is not particularly limited, and it may be provided in the engine room, in the passenger compartment, or built into the headlamp.

[0127] Referring to Fig. 24(b), the lighting device 22 includes a plurality (for example, two) of light sources 22A and 22B. The plurality of light sources 22A and 22B emit light at the same timing, and their emitted lights form one probe light. The plurality of light sources 22A and 22B are built into the left and right headlamps 302L and 302R.

[0128] The image sensor 24 is built into one or both of the headlamps 302L and 302R. Alternatively, the image sensor 24 may be provided outside the headlamps 302L and 302R, for example, in the vicinity of the lighting device 22.

[0129] Fig. 25 is a block diagram showing a vehicle lamp 200 equipped with an object detection system 210. The vehicle lamp 200 constitutes a lamp system 310 together with a vehicle-side ECU 304. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. Further, an object detection system 210 is provided in the vehicle lamp 200. The object detection system 210 corresponds to the above-described sensing system 10 and includes a gating camera 20 and an arithmetic processing unit 40.

[0130] The information about the object OBJ detected by the arithmetic processing unit 40 may be used for the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information about the type and position of the object OBJ generated by the arithmetic processing unit 40. The lighting circuit 204 and the optical system 206 operate so as to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0131] Also, the information about the object OBJ detected by the arithmetic processing unit 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU may perform automatic driving based on this information.

[0132] Based on the embodiments, the present invention has been described using specific terms. However, the embodiments merely show one aspect of the principle and application of the present invention, and many modifications and changes in arrangement are allowed within the scope that does not deviate from the idea of the present invention defined in the claims.

Industrial Applicability

[0133] The present invention relates to a gating camera.

Explanation of Signs

[0134] S1 Light emission timing signal S2 Exposure timing signal 10 Sensing system 20 Gating camera 22 Lighting device 24 Image sensor 26 Controller 28 Image processing device 40 Arithmetic processing unit 42 Classifier 200 Vehicle lamp 202 Light source 204 Lighting circuit 206 Optical system 300 Automobile 302 Headlamp 304 Vehicle-side ECU 310 Lamp system

Claims

1. A gating camera that divides a plurality of ranges in the depth direction and generates a plurality of slice images corresponding to the plurality of ranges, An illumination device that irradiates probe light, An image sensor, A controller that controls the light emission timing of the illumination device and the exposure timing of the image sensor, An image processing device that generates the slice image based on the sensor image transmitted from the image sensor, comprising, The image processing device selects M (M≥2) in ascending order of pixel values for each line of the sensor image, calculates their average value, and subtracts the average value from each pixel value of the same line. The gating camera is characterized by this.

2. The gating camera according to claim 1, wherein M is 2 to 8% of the number of pixels in the line.

3. The gating camera according to claim 1 or 2, characterized in that it is mounted on a vehicle.

4. The gating camera according to claim 1 or 2, An arithmetic processing device that processes the plurality of slice images photographed by the gating camera, A vehicle sensing system characterized by comprising.

5. A vehicle lamp characterized by comprising the gating camera according to claim 1 or 2.

Images