Object detection device

Abstract

The present invention provides an object detection device. The object detection device (100) uses a lidar (1) and a camera (2) to detect a target object. The object detection device (100) includes a calibration processing unit (31) that calibrates the lidar (1) and the camera (2). The lidar (1) includes a light receiving element (12) that can receive the reflected light of the irradiated laser and the reflected light of ambient light, and detect the intensity of the reflected light of the received ambient light, which is light other than the laser. The calibration processing unit (31) performs calibration based on the light receiving result of the reflected light of the received ambient light and the camera image captured by the camera (2).

Description

Technical Field

[0001] The present disclosure relates to an object detection device. Background Art

[0002] For example, in order to detect surrounding target objects (object targets), a light detection and ranging (LIDAR) that detects a target object based on reflected light of irradiated laser is mounted on an autonomous vehicle. In addition, for example, as described in Non-Patent Document 1, a LIDAR has been developed that detects a target object using the received light result of reflected light of ambient light other than the irradiated laser in addition to the reflected light of the irradiated laser.

[0003] Prior Art Documents

[0004] Non-Patent Documents

[0005] Non-Patent Document 1: Seigo Ito, Ryota Tsukazaki, Mitsuhiko Ohta, Hiroyuki Matsubara, Masaru Ogawa, "Position and Orientation Estimation Using a Small Imaging LIDAR and DCNN", Proceedings of the 79th National Conference of the Information Processing Society of Japan Summary of the Invention

[0006] Technical Problem to be Solved by the Invention

[0007] Sometimes, in order to detect surrounding target objects, in an autonomous vehicle, in addition to a LIDAR, a camera is also mounted. In this case, calibration of the LIDAR and the camera is performed, and target object detection is performed based on the detection results of both. Here, as described above, there is a LIDAR that detects a target object using the received light result of reflected light of ambient light other than the irradiated laser. Even in the case where such a LIDAR is mounted in addition to a camera, it is required to perform calibration of the two with good accuracy.

[0008] Therefore, the present disclosure describes an object detection device that can accurately calibrate a LIDAR and a camera, and the LIDAR can also detect the reflected light of ambient light in addition to the reflected light of the irradiated laser.

[0009] Technical Solution for Solving the Problem

[0010] One aspect of the present disclosure is an object detection device that uses a LIDAR and a camera to detect a target object, and includes a calibration processing unit that calibrates the LIDAR and the camera. The LIDAR includes a light receiving unit that can receive the reflected light of the irradiated laser and the reflected light of ambient light, and detect the intensity of the received reflected light of ambient light. The ambient light is light other than the laser. The calibration processing unit performs calibration based on the received light result of the reflected light of ambient light received by the light receiving unit and the camera image captured by the camera.

[0011] In the object detection device, the calibration processing unit performs calibration based on the light reception result of the reflected light of the ambient light received by the lidar and the camera image of the camera. That is, the object detection device performs calibration of the lidar and the camera based on the light reception result of the reflected light of the ambient light and the same type of information as the camera image. Thus, the object detection device can perform calibration of the lidar and the camera with good accuracy, and the lidar can detect the reflected light of the ambient light in addition to the reflected light of the irradiated laser.

[0012] In the object detection device, it can also be that the lidar further includes a corresponding information generation unit that generates corresponding information obtained by making the position of the reflection point of the reflected light of the received laser correspond to the intensity of the reflected light of the received ambient light, and the calibration processing unit performs calibration based on the corresponding information generated by the corresponding information generation unit and the camera image. In this case, calibration can be performed with good accuracy based on the camera image and the generated corresponding information.

[0013] In the object detection device, it can also be that the corresponding information includes the light reception time of the reflected light of the ambient light used to generate the corresponding information, the camera image includes the shooting time of the camera image, and the calibration processing unit uses the corresponding information and the camera image in which the difference between the light reception time and the shooting time is within a predetermined time to perform calibration. For example, when the object detection device is mounted on a vehicle, as the vehicle moves, the object detection device also moves. Thus, when the light reception time of the reflected light of the ambient light and the shooting time of the camera image are significantly different, even for the same target object, the appearance is very different, and calibration cannot be performed with good accuracy using the ambient light information and the camera image. Therefore, the object detection device can perform calibration with better accuracy by using the corresponding information and the camera image in which the difference between the light reception time and the shooting time is within a predetermined time.

[0014] It can also be that the object detection device further includes a projection unit that projects a reference image into the projection area in front of the lidar and the camera, the lidar irradiates a laser into the irradiation area including the projection area, receives the reflected light of the irradiated laser and the reflected light of the ambient light, the camera shoots the shooting area including the projection area, and the calibration processing unit performs calibration based on the light reception result of the reflected light of the ambient light reflected in the irradiation area received by the light reception unit and the camera image in the shooting area captured by the camera. In this case, the object detection device can perform calibration of the lidar and the camera with better accuracy based on the reflected light of the ambient light including the reference image and the camera image including the reference image, and use the reference images included in both.

[0015] Effect of the Invention

[0016] According to one aspect of the present disclosure, calibration of a lidar and a camera can be performed with good accuracy, and the lidar can detect reflected light of ambient light in addition to reflected light of the irradiated laser. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram showing an example of an object detection device according to the first embodiment.

[0018] Figure 2 (a) is a diagram showing a lidar ambient light image. Figure 2 (b) is a diagram showing an example of an image after image processing of the lidar ambient light image. Figure 2 (c) is a diagram showing an example of an image after image processing of the lidar ambient light image.

[0019] Figure 3 (a) is a diagram showing an example of an image after image processing of the lidar ambient light image. Figure 3 (b) is a diagram showing an example of an image after image processing of the lidar ambient light image.

[0020] Figure 4 is a block diagram showing an example of an object detection device according to the second embodiment.

[0021] REFERENCE SIGNS LIST

[0022] 1 Lidar; 2 Camera; 4 Projection unit; 12 Light receiving element (light receiving unit); 14 Light separation unit (light receiving unit); 15 Laser processing unit (light receiving unit); 16 Ambient light processing unit (light receiving unit, corresponding information generation unit); 31, 31A Calibration processing unit; 100, 100A Object detection device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Hereinafter, the illustrated embodiments will be described with reference to the drawings. In addition, in each figure, the same or corresponding elements are given the same reference numerals, and repeated descriptions are omitted.

[0024] (First Embodiment)

[0025] First, a first embodiment of the object detection device will be described. Figure 1The object detection device 100 according to the first embodiment shown is mounted on a vehicle (own vehicle) and detects objects around the own vehicle. The objects detected by the object detection device 100 can be used, for example, for various controls such as the autonomous driving of the own vehicle. The object detection device 100 includes a lidar (Light Detection and Ranging), a camera 2, and a calibration ECU (Electronic Control Unit). The object detection device 100 uses the lidar 1 and the camera 2 to detect objects around the vehicle. In addition, the object detection device 100 has a function of calibrating the lidar 1 and the camera 2.

[0026] The lidar 1 irradiates laser light around the own vehicle and receives the reflected light (reflected light of the laser) of the irradiated laser light reflected by the object. In addition, the lidar 1 detects the intensity of the reflected light of the laser. The lidar 1 in the present embodiment can receive, in addition to the reflected light of the irradiated laser light, the reflected light (reflected light of the ambient light) of the ambient light reflected by the object, and the ambient light is light other than the irradiated laser light. In addition, the lidar 1 can detect the intensity of the reflected light of the received ambient light. The ambient light is, for example, the light around the own vehicle such as sunlight and lighting.

[0027] More specifically, the lidar 1 includes a laser irradiation unit 11, a light receiving element 12, and a light processing ECU 13. The laser irradiation unit 11 irradiates laser light to each position within a predetermined irradiation area around the own vehicle on which the object detection device 100 is mounted.

[0028] The light receiving element 12 can receive the reflected light of the laser light irradiated from the laser irradiation unit 11 and output a signal corresponding to the intensity of the received reflected light of the laser light. In addition, the light receiving element 12 can receive the reflected light of the ambient light other than the laser light irradiated from the laser irradiation unit 11 and output a signal corresponding to the intensity of the received reflected light of the ambient light.

[0029] The light processing ECU 13 is an electronic control unit having a CPU, a ROM, a RAM, etc. The light processing ECU 13, for example, loads the program recorded in the ROM into the RAM and executes the program loaded into the RAM by the CPU, thereby realizing various functions. The light processing ECU 13 may be composed of a plurality of electronic units.

[0030] The light processing ECU 13 detects the intensity of the reflected light of the laser received by the light receiving element 12 and the intensity of the reflected light of the ambient light based on the output signal of the light receiving element 12. The light processing ECU 13 functionally includes a light separation unit 14, a laser processing unit 15, and an ambient light processing unit (corresponding information generation unit) 16. In this way, the light receiving element 12, the light separation unit 14, the laser processing unit 15, and the ambient light processing unit 16 function as a light receiving unit that can receive the reflected light of the laser and the reflected light of the ambient light and detect the intensity of each received reflected light.

[0031] The light separation unit 14 separates the light received by the light receiving element 12 into the reflected light of the laser and the reflected light of the ambient light. The light separation unit 14 can, for example, determine the light with a specific on-off (flashing) pattern as the reflected light of the laser and the light other than that as the reflected light of the ambient light. Additionally, for example, the light separation unit 14 can determine the light received within a predetermined time after the laser is irradiated from the laser irradiation unit 11 as the reflected light of the laser and the light received at other timings as the reflected light of the ambient light. For this predetermined time, it is set in advance based on the time from when the laser is irradiated by the laser irradiation unit 11 until the irradiated laser is reflected by an object around the own vehicle and the reflected light of the laser reaches the light receiving element 12. Furthermore, as described above, the reflected light of the ambient light does not include the reflected light of the laser irradiated from the lidar 1. However, in the case where the ambient light includes light having the same wavelength as the laser, the reflected light of the ambient light will include the reflected light of the light having the same wavelength as the laser.

[0032] The laser processing unit 15 generates laser information (point cloud of the laser) based on the light receiving result of the reflected light of the laser received by the light receiving element 12. The laser information is generated based on the light receiving results of multiple lasers irradiated to each position within a predetermined irradiation area (the light receiving results of multiple reflected lights). Additionally, after the lidar 1 has completed irradiating the laser to all positions within the irradiation area, the laser is irradiated again to each position within the irradiation area. In this way, after the lidar 1 has completed the irradiation process of irradiating the laser to all positions within the irradiation area, the next irradiation process is performed again. Laser information is generated every time the lidar 1 performs the irradiation process.

[0033] More specifically, regarding the multiple lasers irradiated into the irradiation area, the laser processing unit 15 generates laser point information by corresponding the three-dimensional position of the reflection point of the irradiated laser and the intensity of the laser. The laser processing unit 15 generates laser information based on the multiple pieces of generated laser point information. Additionally, the laser processing unit 15 can measure the three-dimensional position of the reflection point of the laser based on the irradiation angle of the laser irradiated from the laser irradiation unit 11 and the arrival time from when the laser is irradiated until the reflected light of the laser reaches the light receiving element 12.

[0034] The ambient light processing unit 16 generates ambient light information (point cloud of ambient light) based on the received light result of the reflected light of the ambient light received by the light receiving element 12. Regarding the ambient light information, similarly to the laser information, this ambient light information is generated each time the lidar 1 irradiates multiple lasers into the irradiation area.

[0035] More specifically, the ambient light processing unit 16 first obtains the three-dimensional position of the reflection point of the laser from the laser processing unit 15. Here, in a state where the states of each part of the lidar 1 such as the irradiation angle of the laser do not change, the position of the reflection point of the laser received by the light receiving element 12 and the position of the reflection point of the ambient light become the same as each other. Therefore, the lidar 1 can detect the intensity of the reflected light of the ambient light reflected at the same position as the reflection point of the laser by detecting the intensity of the reflected light of the ambient light in the state when the reflected light of the laser is received. Therefore, the ambient light processing unit 16 generates ambient light point information by making the three-dimensional position of the reflection point of the laser obtained from the laser processing unit 15 correspond to the intensity of the reflected light of the ambient light received by the light receiving element 12. This ambient light point information is generated for each of the multiple lasers irradiated into the irradiation area.

[0036] The ambient light processing unit 16 generates ambient light information based on the generated multiple ambient light point information. That is, the ambient light processing unit 16 generates ambient light information (corresponding information) in which the position of the reflection point of the reflected light of the received laser (ambient light) and the intensity of the reflected light of the received ambient light are made to correspond according to the position of the reflection point.

[0037] In this way, the lidar 1 can generate laser information and ambient light information based on the received light result of the light receiving element 12. That is, since the lidar 1 can generate laser information and ambient light information based on the received light result of one light receiving element 12, calibration of the laser information and the ambient light information is not required.

[0038] The object detection device 100 detects a target object based on the laser information generated by the lidar 1. In addition, the object detection device 100 may also use the ambient light information generated by the lidar 1 in addition to the laser information to detect the target object.

[0039] The camera 2 photographs a predetermined photographing area around its own vehicle and generates a camera image as a photographing result. The camera 2 includes a photographing element 21 and an image processing ECU 22. The photographing element 21 can receive the reflected light of the ambient light reflected in the photographing area and outputs a signal corresponding to the received reflected light of the ambient light.

[0040] The image processing ECU 22 is an electronic control unit having the same structure as the optical processing ECU 13. The image processing ECU 22 functionally includes an image processing unit 23. The image processing unit 23 generates a camera image by a well-known method based on the output signal of the imaging element 21.

[0041] The calibration ECU 3 is an electronic control unit having the same structure as the optical processing ECU 13. The calibration ECU 3 may also be integrated with the optical processing ECU 13 or the image processing ECU 22. The calibration ECU 3 functionally includes a calibration processing unit 31.

[0042] The calibration processing unit 31 performs calibration of the lidar 1 and the camera 2. More specifically, the calibration processing unit 31 performs calibration based on the light reception result (ambient light information) of the reflected light of the ambient light received by the lidar 1 and the camera image captured by the camera 2. The calibration here refers to aligning the positions of the lidar 1 and the camera 2 and calculating the corresponding relationship at their positions.

[0043] Here, as the calibration of the lidar 1 and the camera 2, the calibration processing unit 31 calculates, for example, the external parameters of the lidar 1 and the camera 2. The external parameters represent the corresponding relationship at the positions of the lidar 1 and the camera 2. The calibration processing unit 31 generates external parameters based on one piece of ambient light information and one camera image.

[0044] Specifically, the calibration processing unit 31 uses an initial value of the external parameters generated randomly or based on predetermined conditions to perform a transformation process of transforming the data of either or both of the ambient light information and the camera image. Then, the calibration processing unit 31 calculates the similarity between the ambient light information and the camera image based on the ambient light information and the camera image after the transformation process. And the calibration processing unit 31 updates the external parameters to increase the similarity. The calibration processing unit 31 repeats the update of the external parameters until a predetermined number of times or the similarity exceeds a predetermined similarity threshold. When the update of the external parameters is completed, the calibration processing unit 31 outputs the updated external parameters as the calibration result. The object detection device 100 can identify the corresponding relationship at the positions of the lidar 1 and the camera 2 based on the calculated external parameters.

[0045] Here, the ambient light information and the camera image used for calibrating the calibration processing unit 31 are described. In addition, the ambient light information includes the light reception time of the reflected light of the ambient light used to generate the ambient light information, and the camera image includes the shooting time of the camera image. When the calibration processing unit 31 extracts the ambient light information and the camera image for calibration, it extracts the ambient light information and the camera image in which the difference between the light reception time included in the ambient light information and the shooting time included in the camera image is within a predetermined time. The calibration processing unit 31 can perform calibration as described above using the extracted ambient light information and camera image.

[0046] Next, an example of various methods for calculating the external parameters by the calibration processing unit 31 is described.

[0047] (First method)

[0048] First, the first method for calculating the external parameters is described. In the first method, the calibration processing unit 31 uses the histogram of the image to calculate the external parameters.

[0049] (1) The calibration processing unit 31 transforms the ambient light information into "an image with the intensity of the reflected light of the ambient light as the pixel value" based on, for example, the following projection matrix and transformation formula. Hereinafter, the "image with the intensity of the reflected light of the ambient light as the pixel value" generated by transforming the ambient light information is referred to as "the lidar ambient light image". And the calibration processing unit 31 cuts the lidar ambient light image to the same size (width × height) as the camera image.

[0050]

[0051] Two-dimensional transformation:

[0052] (2) Next, the calibration processing unit 31 transforms the camera image into grayscale.

[0053] (3) The calibration processing unit 31 transforms the lidar ambient light image and the camera image transformed into grayscale into histograms respectively. The calibration processing unit 31 uses the similarity of the histograms as a score, and while transforming the external parameters (parameters of position and orientation), performs an optimization calculation to make the similarity higher. This score can be calculated, for example, based on the mutual information amount of the two histograms, etc.

[0054] (Second method)

[0055] A second method for calculating the external parameters will be described. In the second method, the calibration processing unit 31 processes the image through image processing, calculates the similarity at the pixel level, and thereby calculates the external parameters.

[0056] (1) Similar to the processing in (1) of the above-described first method, the calibration processing unit 31 transforms the ambient light information into a lidar ambient light image.

[0057] (2) Next, the calibration processing unit 31 performs image processing on the lidar ambient light image. Here, various examples of the image processing method will be described. For example, as shown in (a) of Figure 2 , there is a lidar ambient light image representing the situation in front of the own vehicle. As a first image processing method, the calibration processing unit 31 can also generate a binary image as shown in (b) of Figure 2 by performing binary processing on the lidar ambient light image shown in (a) of Figure 2 . In addition, the calibration processing unit 31 can also perform multi-valued processing other than binary processing.

[0058] In addition, as a second image processing method, the calibration processing unit 31 can also generate an image with the edge part emphasized as shown in (c) of Figure 2 by performing edge detection processing on the lidar ambient light image shown in (a) of Figure 2 . As a third image processing method, the calibration processing unit 31 can also generate an image partitioned by attribute as shown in (a) of Figure 2 by performing semantic segmentation processing on the lidar ambient light image shown in (a) of Figure 3 . As a fourth image processing method, the calibration processing unit 31 can also perform segmentation processing by extracting only the shadow area part from the lidar ambient light image shown in (a) of Figure 2 and generate an image with only the shadow area part partitioned as shown in (b) of Figure 3 . By partitioning only the shadow part in this way, there are more parts where the disconnection of the area is linear, and the image matching processing becomes easier.

[0059] (3) The calibration processing unit 31 also performs the same image processing on the camera image as in (2).

[0060] (4) The calibration processing unit 31 calculates the difference between the laser radar ambient light image after image processing and the camera image after image processing. The calibration processing unit 31 transforms the external parameters (parameters of position and direction) while performing optimization calculations to reduce the calculated difference. For example, the calibration processing unit 31 can use the following formula to calculate the difference between the laser radar ambient light image and the camera image, and perform optimization calculations to reduce the loss.

[0061] loss=Σ w Σ h (Camera image [h] [w] - LiDAR ambient light image [h] [w]) 2

[0062] Here, "w" is the position in the width direction of the image, and "h" is the position in the height direction of the image. Here, the calibration processing unit 31 calculates the similarity of the two images according to the method used for image processing and performs optimization calculation. For example, when each pixel has a value after binarization, the calibration processing unit 31 can calculate the similarity based on the difference in the numerical value of each pixel, and when each pixel is given an attribute, it can calculate the similarity based on the consistency or inconsistency of the attribute.

[0063] In addition, the calibration processing unit 31 may also use a mapping table of the ambient light information and the pixel level in the camera image as an external parameter. The mapping table is, for example, a map indicating which point of the ambient light information (information of a certain reflection point) corresponds to which pixel in the camera image. The calibration processing unit 31 may, for example, create a mapping table as follows.

[0064] (1) First, the calibration processing unit 31 corrects the reflection intensity value of the ambient light information. For example, the calibration processing unit 31 may perform γ correction on the reflection intensity value.

[0065] (2) Next, the calibration processing unit 31 converts the camera image into grayscale.

[0066] (3) The calibration processing unit 31 picks up pixels of the camera image having a value close to the reflection intensity of each point of the ambient light information.

[0067] (4) The calibration processing unit 31 searches for a pair of points of the ambient light information and pixels of the camera image that does not destroy the positional relationship of each point in the ambient light information based on the pixels of the captured camera image. For example, when point A of the ambient light information exists on the left side of point B, but the corresponding pixel A of the camera image exists on the right side of pixel B, the calibration processing unit 31 makes the pair invalid.

[0068] (5) When the calibration processing unit 31 has completed the search for pairs above a predetermined pair threshold in (4), it creates a mapping table based on the positional relationship of the pairs. When no pairs above the pair threshold are found, the calibration processing unit 31 relaxes the picking conditions in (3) and searches for pairs again. As an example of relaxing the picking conditions, the calibration processing unit 31 relaxes the "predetermined range" in which "if the pixel value is within a predetermined range, it is determined to be the same (similar) and picked up".

[0069] In addition, the calibration processing unit 31 may also generate a projection matrix for transforming the coordinate system of the lidar 1 into the coordinate system of the camera 2 based on the mapping table created by the above method.

[0070] As described above, in the object detection device 100, the calibration processing unit 31 calibrates the lidar 1 and the camera 2 based on the light reception result of the reflected light of the ambient light received by the lidar 1 and the camera image of the camera 2. That is, the object detection device 100 calibrates the lidar 1 and the camera 2 based on the same type of information, namely, the light reception result of the reflected light of the ambient light and the camera image. Thus, the object detection device 100 can calibrate the lidar 1 and the camera 2 with good accuracy, and the lidar 1 can detect the reflected light of the ambient light in addition to the reflected light of the irradiated laser.

[0071] The ambient light processing unit 16 of the lidar 1 generates ambient light information (corresponding information) in which the position of the reflection point of the reflected light of the received laser corresponds to the intensity of the reflected light of the ambient light. The calibration processing unit 31 performs calibration based on the generated ambient light information and the camera image. In this case, the object detection device 100 can perform calibration with good accuracy based on the ambient light information in which the intensity of the reflected light is corresponding to the position of the reflection point of the reflected light and the camera image.

[0072] The calibration processing unit 31 performs calibration using the ambient light information in which the difference between the light reception time of the reflected light of the ambient light included in the ambient light information and the shooting time when the camera image is taken is within a predetermined time. For example, when the object detection device 100 is mounted on a vehicle, as the vehicle moves, the object detection device 100 also moves. Thus, when the light reception time of the reflected light of the ambient light is significantly different from the shooting time of the camera image, even for the same target object, the appearance is very different, and calibration cannot be performed with good accuracy using the ambient light information and the camera image. Therefore, the object detection device 100 can perform calibration with better accuracy by using the ambient light information and the camera image in which the difference between the light reception time and the shooting time is within a predetermined time.

[0073] (Second Embodiment)

[0074] Next, a second embodiment of the object detection device will be described. Hereinafter, the description will focus on the differences from the object detection device 100 according to the first embodiment, and the corresponding elements will be given the same reference numerals and detailed descriptions will be omitted. As Figure 4 shown, the object detection device 100A according to the second embodiment uses a projected reference image to calibrate the lidar 1 and the camera 2. The object detection device 100A includes a lidar 1, a camera 2, a calibration ECU 3A, and a projection unit 4.

[0075] The projection unit 4 projects a reference image into the projection area in front of the lidar 1 and the camera 2. The reference image can be, for example, a predetermined number, character, graphic, etc. The lidar 1 irradiates laser light into the irradiation area including the projection area where the reference image is projected, and receives the reflected light of the irradiated laser light and the reflected light of the ambient light. The ambient light here also includes the reflected light of the reference image irradiated from the projection unit 4. In addition, the camera 2 captures an image of the imaging area including the projection area where the reference image is projected.

[0076] The calibration ECU 3A includes a calibration processing unit 31A. The calibration processing unit 31A calibrates the lidar 1 and the camera 2 using the same method as in the first embodiment based on the light reception result (ambient light information) of the reflected light of the ambient light reflected in the irradiation area received by the lidar 1 and the camera image in the imaging area captured by the camera 2. That is, the calibration processing unit 31A performs calibration based on the ambient light information including the influence of the reference image and the camera image including the reference image. Here, since the reference image is included in both the ambient light information and the camera image, the calibration processing unit 31A can calculate the similarity between the two based on the reference image and perform calibration. In addition, the lidar 1 in this embodiment can detect the reflected light of the ambient light in addition to the reflected light of the laser. Therefore, the calibration processing unit 31A can perform calibration using the ambient light information including the reference image irradiated from the projection unit 4.

[0077] As described above, the object detection device 100A in this embodiment includes a projection unit 4 that projects a reference image. In this case, the object detection device 100A can perform calibration of the lidar and the camera with higher accuracy based on the ambient light information including the reference image and the camera image including the reference image, using the reference image included in both.

[0078] In addition, the projection unit 4 may also periodically change the projected reference image. In this case, without moving the vehicle equipped with the object detection device 100A and without moving components such as the camera 2, it is possible to obtain ambient light information and camera images including a reference image different from the previous one. Additionally, the projection unit 4 may project a reference image suitable for calibration by the calibration processing unit 31A. In this case, the calibration processing unit 31A can perform calibration with better accuracy.

[0079] Furthermore, the projection unit 4 may project a reference image embedded with time information. For example, the projection unit 4 may project a reference image embedded with numbers and QR codes (registered trademarks), etc. as time information. Additionally, the projection unit 4 switches the reference image embedded with time information at a cycle equal to or less than the irradiation cycle (cycle of the irradiation process) of the lidar 1 or the shooting cycle of the camera 2.

[0080] Here, in the lidar 1, there is a data transfer delay between the light receiving element 12 and the light processing ECU 13. Additionally, in the camera 2, there is also a data transfer delay between the imaging element 21 and the image processing ECU 22. Therefore, when the data transfer delay in the lidar 1 is T1 and the data transfer delay in the camera 2 is T2, without considering the time correction of |T1 - T2|, when comparing the ambient light information and the camera image, the position of the target object will not match. Thus, the calibration processing unit 31A can consider the data transfer delays of the lidar 1 and the camera 2 respectively based on the embedded time information by using the ambient light information and the camera image including the reference image embedded with time information, and compare the ambient light information and the camera image. Thereby, the calibration processing unit 31A can perform calibration with better accuracy.

[0081] As described above, embodiments of the present disclosure have been described, but the present disclosure is not limited to the above embodiments. The present disclosure can be variously modified without departing from the gist of the present disclosure.

Claims

1. An object detection device, which uses a lidar and a camera to detect a target object and is mounted on a vehicle, comprises a calibration processing unit for calibrating the lidar and the camera, wherein the lidar comprises: a light receiving unit that can receive the reflected light of the irradiated laser and the reflected light of ambient light, and detect the intensity of the received reflected light of the ambient light, the ambient light being light other than the laser; and a corresponding information generation unit that generates corresponding information obtained by making the position of the reflection point of the reflected light of the laser received through the light receiving unit correspond to the intensity of the reflected light of the ambient light received through the light receiving unit, the corresponding information includes the light receiving time of the reflected light of the ambient light used to generate the corresponding information, the camera image captured by the camera includes the capture time of the camera image, the calibration processing unit performs the calibration using the corresponding information and the camera image in which the difference between the light receiving time and the capture time is equal to or less than a predetermined time when the vehicle is moving, and the calibration processing unit uses the time corrected by considering the transmission delay of the data of the reflected light of the ambient light in the lidar and the transmission delay of the data of the camera image in the camera as the light receiving time and the capture time when the vehicle is moving.

2. The object detection device according to claim 1, further comprises a projection unit that projects a reference image into a projection area in front of the lidar and the camera, the lidar irradiates the laser into an irradiation area including the projection area and receives the reflected light of the irradiated laser and the reflected light of the ambient light, the camera captures an image of a capture area including the projection area, and the calibration processing unit performs the calibration based on the light receiving result of the reflected light of the ambient light reflected in the irradiation area received by the light receiving unit and the camera image in the capture area captured by the camera.

Images