Object detection device

Abstract

The present disclosure relates to an object detection device. The object detection device (100) includes: a laser irradiation unit (11); a light receiving element (12) capable of detecting reflected light of the laser and reflected light of ambient light which is light other than the laser; a camera (2); an information correction unit (31) that corrects ambient light information based on a camera image to generate corrected ambient light information, where the ambient light information is information on reflected light of ambient light detected by the light receiving element (12); and an object detection unit (32) that detects a target based on the corrected ambient light information and laser information received by the light receiving element (12).

Description

Technical Field

[0001] The present disclosure relates to an object detection device. Background Art

[0002] For example, in order to detect surrounding landmarks, a light detection and ranging (LIDAR) that detects a landmark based on reflected light of irradiated laser is mounted on an autonomous vehicle. In addition, for example, as described in Non-Patent Document 1, a LIDAR has been developed that detects a landmark using the detection result of reflected light of ambient light other than the irradiated laser in addition to the reflected light of the irradiated laser.

[0003] In this LIDAR, the reflected light of the laser and the reflected light of the ambient light are received by the same light receiving element. Therefore, a positional correspondence relationship is obtained between the detection results of the two. Therefore, by using such a LIDAR, it is possible to detect a landmark based on the detection results of the two lights having a positional correspondence relationship.

[0004] Prior Art Documents

[0005] Non-Patent Document 1: Seigo Ito, Seira Tsukazaki, Mitsuhiko Ohta, Hiroyuki Matsubara, Masaru Ogawa, "Position and Orientation Estimation Using a Small Imaging LIDAR and DCNN", Proceedings of the 79th National Conference of the Information Processing Society of Japan, 2017 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] Generally, the number of light receiving elements in the light receiving unit of a camera is overwhelmingly larger than the number of light receiving elements in the light receiving unit of a LIDAR. Therefore, the amount of environmental light information obtained by the LIDAR is smaller than the amount of information in a camera image captured by the camera, and is insufficient for accurately detecting a landmark. Therefore, when using the detection result of the reflected light of the ambient light in addition to the reflected light of the laser, there is room for improvement in more accurately detecting a landmark.

[0008] Thus, the present disclosure describes an object detection device that can accurately detect a landmark based on the detection results of the reflected light of the laser and the reflected light of the ambient light.

[0009] Technical Means for Solving the Problems

[0010] In one aspect of the present disclosure, an object detection device includes: a laser irradiation unit that irradiates laser light; a light receiving unit that can detect the reflected light of the laser light and the reflected light of ambient light which is light other than the laser light; a camera; an information correction unit that corrects ambient light information based on a camera image captured by the camera to generate corrected ambient light information, where the ambient light information is information on the reflected light of the ambient light detected by the light receiving unit; and an object detection unit that detects a target based on the corrected ambient light information and laser information, where the laser information is information on the reflected light of the laser light received by the light receiving unit.

[0011] The object detection device corrects the ambient light information based on the camera image to generate corrected ambient light information. Further, the object detection device detects a target based on the generated corrected ambient light information and laser information. In this way, by using the corrected ambient light information obtained by correcting the ambient light information based on the camera image, the object detection device can accurately detect a target based on the detection results of the reflected light of the laser light and the reflected light of the ambient light (laser information and corrected ambient light information).

[0012] In the object detection device, the information correction unit may also generate corrected ambient light information by adding the color information of the camera image to the ambient light information. In this case, the information correction unit can increase the number of color channels of the ambient light information. As a result, the object detection device can more accurately detect a target by using the corrected ambient light information with the added color information of the camera image and the laser information.

[0013] In the object detection device, the information correction unit may also generate corrected ambient light information by adding the information of the camera image between pixels of the ambient light information. In this case, the information correction unit can improve the resolution of the ambient light information. As a result, the object detection device can more accurately detect a target by using the corrected ambient light information with improved resolution and the laser information.

[0014] In the object detection device, the information correction unit may also correct the ambient light information based on segmentation information obtained from the camera image to generate corrected ambient light information. In this case, the object detection device can more accurately detect a target by using the corrected ambient light information generated based on the segmentation information and the laser information.

[0015] In the object detection device, the information correction unit may also calculate the distance to the reflection point of the obtained ambient light based on the parallax between the camera image and the ambient light information, and add the calculated distance information to the ambient light information, thereby generating corrected ambient light information. In this case, the object detection device can more accurately detect a target by using the corrected ambient light information with the added distance information to the reflection point of the ambient light and the laser information.

[0016] Advantages of the Invention

[0017] According to one aspect of the present disclosure, a target can be accurately detected based on the detection results of the reflected light of the laser and the reflected light of the ambient light. Description of the Drawings

[0018] Figure 1 It is a block diagram showing an example of an object detection device according to one embodiment.

[0019] Figure 2 It is a flowchart showing the flow of the target detection process performed in the object detection ECU.

[0020] Reference Signs Description

[0021] 2 Camera, 11 Laser Irradiation Unit, 12 Light Receiving Element (Light Receiving Unit), 14 Light Separation Unit (Light Receiving Unit), 15 Laser Processing Unit (Light Receiving Unit), 16 Ambient Light Processing Unit (Light Receiving Unit), 31 Information Correction Unit, 32 Object Detection Unit, 100 Object Detection Device. Detailed Embodiments

[0022] Hereinafter, exemplary embodiments will be described with reference to the drawings. In addition, in each figure, the same or corresponding elements are given the same reference signs, and repeated descriptions are omitted.

[0023] As Figure 1 shown, the object detection device 100 is mounted on a vehicle (own vehicle) and detects targets around the own vehicle. The targets detected by the object detection device 100 can be used, for example, for various controls such as the autonomous driving of the own vehicle. The object detection device 100 includes a lidar [LIDAR: Light Detection and Ranging] 1, a camera 2, and an object detection ECU [Electronic Control Unit] 3.

[0024] The lidar 1 irradiates laser light around the own vehicle and receives the reflected light (reflected light of the laser) of the irradiated laser light reflected by the target. In addition, the lidar 1 detects the intensity of the reflected light of the laser. The lidar 1 in the present embodiment can receive, in addition to the reflected light of the irradiated laser, the reflected light (reflected light of the ambient light) of the ambient light, which is light other than the irradiated laser, reflected by the target. In addition, the lidar 1 can detect the intensity of the reflected light of the received ambient light. This ambient light refers to, for example, the light around the own vehicle such as sunlight and illumination.

[0025] More specifically, the lidar 1 includes a laser irradiation unit 11, a light receiving element 12, and a light processing ECU 13. The laser irradiation unit 11 irradiates laser light to each position within a predetermined irradiation area around the host vehicle on which the object detection device 100 is mounted.

[0026] The light receiving element 12 can receive the reflected light of the laser light irradiated by the laser irradiation unit 11 and output a signal corresponding to the intensity of the received reflected light of the laser light. In addition, the light receiving element 12 can receive the reflected light of ambient light other than the laser light irradiated by the laser irradiation unit 11 and output a signal corresponding to the intensity of the received reflected light of the ambient light.

[0027] The light processing ECU 13 is an electronic control unit having a CPU, a ROM, a RAM, etc. The light processing ECU 13 realizes various functions, for example, by loading the program recorded in the ROM into the RAM and executing the program loaded into the RAM by the CPU. The light processing ECU 13 may be composed of multiple electronic units.

[0028] Based on the output signal of the light receiving element 12, the light processing ECU 13 detects the intensity of the reflected light of the laser light and the intensity of the reflected light of the ambient light received by the light receiving element 12, respectively. The light processing ECU 13 functionally includes a light separation unit 14, a laser processing unit 15, and an ambient light processing unit 16. In this way, the light receiving element 12, the light separation unit 14, the laser processing unit 15, and the ambient light processing unit 16 function as a light receiving unit capable of detecting the reflected light of the laser light and the reflected light of the ambient light.

[0029] The light separation unit 14 separates the light received by the light receiving element 12 into the reflected light of the laser light and the reflected light of the ambient light. The light separation unit 14 can, for example, determine the light of a specific bright-dark pattern as the reflected light of the laser light and the other light as the reflected light of the ambient light. In addition, for example, the light separation unit 14 can determine the light received within a predetermined time after the laser light is irradiated by the laser irradiation unit 11 as the reflected light of the laser light and the light received at other timings as the reflected light of the ambient light. This predetermined time is set in advance based on the time from when the laser light is irradiated by the laser irradiation unit 11 until the irradiated laser light is reflected by a target around the host vehicle and the reflected light of the laser light reaches the light receiving element 12. In addition, as described above, the reflected light of the ambient light does not include the reflected light of the laser light irradiated by the lidar 1. However, when the ambient light includes light of the same wavelength as the laser light, the reflected light of the ambient light will include the reflected light of the light of the same wavelength as the laser light.

[0030] The laser processing unit 15 generates laser information based on the light reception result of the reflected light of the laser received by the light receiving element 12. The laser information is generated based on the light reception results of a plurality of lasers irradiated to each position within a predetermined irradiation area (the light reception results of a plurality of reflected lights). In addition, after the lidar 1 has completed irradiating the laser to all positions within the irradiation area, the laser is irradiated again to each position within the irradiation area. In this way, after the lidar 1 has completed the irradiation process of irradiating the laser to all positions within the irradiation area, the next irradiation process is performed again. Each time the lidar 1 performs the irradiation process, laser information is generated.

[0031] More specifically, the laser processing unit 15 associates the three-dimensional position of the reflection point of the irradiated laser and the intensity of the laser for each of the plurality of lasers irradiated into the irradiation area to generate laser point information. The laser processing unit 15 generates laser information based on the generated plurality of laser point information. In addition, the laser processing unit 15 can measure the three-dimensional position of the reflection point of the laser based on the irradiation angle of the laser irradiated from the laser irradiation unit 11 and the arrival time from the irradiation of the laser to the arrival of the reflected light of the laser at the light receiving element 12.

[0032] The ambient light processing unit 16 generates ambient light information as information on the reflected light of the ambient light based on the light reception result of the reflected light of the ambient light received by the light receiving element 12. Regarding the ambient light information, the ambient light information is generated each time the lidar 1 performs the irradiation process of irradiating a plurality of lasers into the irradiation area, in the same manner as the laser information.

[0033] More specifically, the ambient light processing unit 16 first obtains the three-dimensional position of the reflection point of the laser from the laser processing unit 15. Here, in a state where the irradiation angle of the laser and the like and the states of the respective parts of the lidar 1 do not change, the positions of the reflection points of the laser received by the light receiving element 12 and the reflection points of the ambient light are the same as each other. Therefore, the lidar 1 can detect the intensity of the reflected light of the ambient light reflected at the same position as the reflection point of the laser by detecting the intensity of the reflected light of the ambient light in the state when the reflected light of the laser is received. Therefore, the ambient light processing unit 16 generates ambient light point information by associating the three-dimensional position of the reflection point of the laser obtained from the laser processing unit 15 and the intensity of the reflected light of the ambient light received by the light receiving element 12. The ambient light point information is generated for each of the plurality of lasers irradiated into the irradiation area.

[0034] The ambient light processing unit 16 generates ambient light information based on the generated multiple ambient light point information. That is, the ambient light processing unit 16 generates ambient light information that associates the position of the reflected light of the received laser (ambient light) and the intensity of the reflected light of the received ambient light according to the position of each reflection point.

[0035] In this way, the lidar 1 can generate laser information and ambient light information based on the light reception result of the light receiving element 12. That is, the lidar 1 can generate laser information and ambient light information based on the light reception result of one light receiving element 12. Therefore, calibration of the laser information and the ambient light information is no longer required.

[0036] The camera 2 captures an image within a predetermined imaging area around its own vehicle and generates a camera image as the imaging result. In addition, at least a part of the imaging range of the camera 2 overlaps with the irradiation range of the laser of the lidar 1. The camera 2 includes an imaging element 21 and an image processing ECU 22. The imaging element 21 can receive the reflected light of the ambient light reflected within the imaging area and output a signal corresponding to the received reflected light of the ambient light.

[0037] The image processing ECU 22 is an electronic control unit having the same structure as the light processing ECU 13. The image processing ECU 22 functionally includes an image processing unit 23. The image processing unit 23 generates a camera image by a well-known method based on the output signal of the imaging element 21.

[0038] The object detection ECU 3 detects objects around its own vehicle based on the detection result of the lidar 1. The object detection ECU 3 is an electronic control unit having the same structure as the light processing ECU 13. The object detection ECU 3 may also be integrated with the light processing ECU 13 or the image processing ECU 22. The object detection ECU 3 functionally includes an information correction unit 31 and an object detection unit 32.

[0039] The information correction unit 31 corrects the ambient light information generated by the ambient light processing unit 16 of the lidar 1 based on the camera image generated by the camera 2 to generate corrected ambient light information. The information correction unit 31 can generate corrected ambient light information based on the camera image by various methods. In the present embodiment, the information correction unit 31 can generate corrected ambient light information based on any one of the following first method to fourth method, or a combination of two or more of the following first method to fourth method. In addition, the information correction unit 31 can obtain the correspondence relationship (positional correspondence relationship) between the camera image and the ambient light information by performing calibration in advance or matching feature points when generating the corrected ambient light information.

[0040] (First method)

[0041] First, the first method will be described. In the first method, the information correction unit 31 generates corrected ambient light information by adding the color information (such as RGB information) of the camera image to the ambient light information. Thus, for example, color information is added to grayscale ambient light information, increasing the number of color channels of the ambient light information.

[0042] (Second method)

[0043] Next, the second method will be described. In the second method, the information correction unit 31 generates corrected ambient light information by adding the information of the camera image between the pixels of the ambient light information. Here, the information correction unit 31 can use the color information or luminance information of the camera image, etc., as the information of the camera image added between the pixels of the ambient light information. In this way, the information correction unit 31 can generate corrected ambient light information with improved resolution of the ambient light information by adding the information of the camera image between the pixels of the ambient light information.

[0044] (Third method)

[0045] Next, the third method will be described. In the third method, the information correction unit 31 corrects the ambient light information based on the segmentation information obtained from the camera image to generate corrected ambient light information. Here, the information correction unit 31 can use various well-known methods such as semantic segmentation or instance segmentation, etc., to obtain the segmentation information from the camera image. For example, the information correction unit 31 can generate corrected ambient light information with category information of regions for the ambient light information by adding the segmentation information to the ambient light information. Or, for example, the information correction unit 31 can generate corrected ambient light information by correcting the pixel values of the ambient light information based on the category information of the regions of the segmentation information (such as making the boundaries of the regions clear, etc.).

[0046] (Fourth method)

[0047] Next, the fourth method will be described. In the fourth method, the information correction unit 31 calculates the distance to the reflection point of the obtained ambient light based on the disparity between the camera image and the ambient light information. Here, the information correction unit 31 can calculate the distance to the reflection point of the ambient light based on the principle of triangulation, similar to a stereo camera, etc. And the information correction unit 31 generates corrected ambient light information by adding the calculated distance information to the ambient light information. Thus, the information correction unit 31 can generate corrected ambient light information including the information of the distance to the reflection point of the ambient light.

[0048] The object detection unit 32 detects a target based on the corrected ambient light information generated by the information correction unit 31 and the laser information generated by the laser processing unit 15. Here, the object detection unit 32 can fuse the two pieces of information, namely the corrected ambient light information and the laser information, and perform target detection processing and recognition processing by various well-known methods.

[0049] Next, use Figure 2 the flowchart of Figure 2 to illustrate an example of the process of target detection processing performed in the object detection ECU 3 of the object detection device 100. In addition, Figure 2 the detection processing shown is repeatedly executed at each predetermined time after the start of the target detection processing. In addition, the order of the processes of S101 to S104 is not limited to

[0050] As Figure 2 shown, the object detection unit 32 acquires the laser information generated by the laser processing unit 15 of the lidar 1 (S101). The information correction unit 31 acquires the ambient light information generated by the ambient light processing unit 16 of the lidar 1 (S102). In addition, the information correction unit 31 acquires the camera image generated by the camera 2 (S103).

[0051] The information correction unit 31 corrects the ambient light information based on the acquired camera image to generate corrected ambient light information (S104). The object detection unit 32 performs target detection (S105) based on the acquired laser information and the corrected ambient light information generated by the information correction unit 31.

[0052] As described above, the object detection device 100 corrects the ambient light information based on the camera image to generate corrected ambient light information. And, the object detection device 100 detects the target based on the generated corrected ambient light information and the laser information. In this way, the object detection device 100 can accurately detect the target based on the detection results of the reflected light of the laser and the reflected light of the ambient light (laser information and corrected ambient light information) by using the corrected ambient light information obtained by correcting the ambient light information based on the camera image.

[0053] In the first method of generating the corrected ambient light information, the information correction unit 31 generates the corrected ambient light information by adding the color information of the camera image to the ambient light information. In this case, the information correction unit 31 can increase the number of color channels of the ambient light information. Thereby, the object detection device 100 can more accurately detect the target by using the corrected ambient light information with the added color information of the camera image and the laser information.

[0054] In the second method of generating the corrected ambient light information, the information correction unit 31 generates the corrected ambient light information by adding the information of the camera image between the pixels of the ambient light information. In this case, the information correction unit 31 can improve the resolution of the ambient light information. Thereby, the object detection device 100 can use the corrected ambient light information with improved resolution and the laser information to detect the target more accurately.

[0055] In the third method of generating the corrected ambient light information, the information correction unit 31 corrects the ambient light information based on the segmentation information obtained from the camera image to generate the corrected ambient light information. In this case, the object detection device 100 can detect the target more accurately by using the corrected ambient light information generated based on the segmentation information and the laser information.

[0056] In the fourth method of generating the corrected ambient light information, the information correction unit 31 generates the corrected ambient light information by calculating the distance to the reflection point of the ambient light and adding the information of the calculated distance to the ambient light information. In this case, the object detection device 100 can use the corrected ambient light information with the information of the distance to the reflection point of the ambient light added and the laser information to detect the target more accurately.

[0057] As described above, the embodiments of the present disclosure have been described, but the present disclosure is not limited to the above embodiments. The present disclosure can be variously modified without departing from the spirit of the present disclosure.

Claims

1. An object detection device, comprising: a laser irradiation unit that irradiates laser light; a light receiving unit that can detect the reflected light of the laser light and the reflected light of ambient light that is light other than the laser light; a camera; an information correction unit that corrects ambient light information based on a camera image captured by the camera to generate corrected ambient light information, where the ambient light information is information on the reflected light of the ambient light detected by the light receiving unit; and an object detection unit that detects a target based on the corrected ambient light information and laser information, where the laser information is information on the reflected light of the laser light received by the light receiving unit, the information correction unit generates the corrected ambient light information by adding color information of the camera image to the grayscale ambient light information, the information correction unit generates the corrected ambient light information by adding information of the camera image between pixels of the ambient light information using luminance information of the camera image, the information correction unit corrects the ambient light information based on segmentation information obtained from the camera image to generate the corrected ambient light information that gives the ambient light information regional class information, the information correction unit calculates the distance to the reflection point of the obtained ambient light based on the disparity between the camera image and the ambient light information, and adds the calculated distance information to the ambient light information, thereby generating the corrected ambient light information.

Images