DEVICE FOR DETERMINING THE GROUND CLEARANCE OF A STRUCTURE

The integration of radar and camera data with a convolutional neural network effectively addresses the challenge of determining ground clearance, enhancing collision avoidance by accurately identifying stationary structures and providing timely clearance information.

DE102018118427B4Active Publication Date: 2025-09-25GM GLOBAL TECHNOLOGY OPERATIONS LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
DE102018118427
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2017-07-31
Filing Date
2018-07-31
Publication Date
2025-09-25
Estimated Expiration
2038-07-31

AI Technical Summary

Technical Problem

Existing systems struggle to accurately determine the ground clearance of horizontal structures above a driving surface, particularly when towing trailers that exceed the vehicle height, leading to potential collisions.

Method used

A method and apparatus using a combination of radar and camera information, employing a convolutional neural network to process radar reflection points and image data, to identify stationary horizontal structures and calculate the ground distance, filtering noise and distinguishing between moving and stationary objects.

Benefits of technology

Accurately determines the ground clearance, enabling vehicles to avoid collisions by providing real-time alerts and ensuring sufficient clearance for trailers and vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Apparatus for determining the ground clearance of a structure (100), the apparatus comprising: a long-range radar configured to emit radio waves in an area and generate first reflection point information of the area based on the emitted radio waves; a camera configured to capture the image of the area; at least one memory (103) comprising computer-executable instructions; and at least one processor configured to read and execute the computer-executable instructions, wherein the computer-executable instructions cause the at least one processor to: Obtaining the first reflection point information from the radar; Obtaining the image from the camera; Removing (S210) reflection points caused by noise from the first reflection point information based on temporal persistence and generating second reflection point information; Extracting (S220) visual features from an image of the camera based on a convolutional neural network; Projecting (S230) the second reflection point information onto the image; Generating (S240) region suggestions based on the projected second reflection point information and the image, the region suggestions indicating potential horizontal structures above a roadway; Detecting (S250) stationary horizontal structures above the roadway based on the generated region proposals; and Determining (S260) the distance between the ground and the detected stationary horizontal structure based on projected reflection point information and the image.
Need to check novelty before this filing date? Find Prior Art

Description

INTRODUCTION

[0001] Devices and methods in accordance with exemplary embodiments relate to detecting the ground clearance of horizontal structures above a driving surface. More specifically, devices and methods in accordance with exemplary embodiments relate to detecting the ground clearance of horizontal structures above a driving surface and determining whether an object could pass through the detected ground clearance.

[0002] DE 10 2011 113 077 A1 discloses a system with a 3D camera for determining the traversability of an object for a vehicle, wherein object dimensions are recorded and compared with the vehicle dimensions. DE 10 2011 119 767 A1 shows a system for tracking objects using a camera and a distance sensor, wherein data from the camera and the distance sensor are combined. SUMMARY

[0003] One or more exemplary embodiments provide a method and apparatus that determine the ground clearance of horizontal structures above a driving surface by using image information and radar information corresponding to the roadway. More specifically, one or more exemplary embodiments provide a method and apparatus that can determine whether a vehicle can pass beneath a horizontal structure by correlating reflection point information from a long-range radar and image information from a camera.

[0004] According to one aspect of an exemplary embodiment, a method for determining the ground clearance of a structure is provided.The method includes removing reflection points caused by the noise from the first reflection point information based on temporal persistence and generating second reflection point information that extracts visual features from an image of a camera based on a convolutional neural network, wherein the second reflection point information is projected onto the image, wherein region proposals are generated based on projected second reflection point information and the image, wherein the region proposals indicate potential horizontal structures above a roadway, wherein stationary horizontal structures above a roadway are detected based on generated region proposals, and determining the distance between the ground and the detected stationary horizontal structure based on projected reflection point information and the image.

[0005] The method may further include obtaining the first reflection point information from a radar configured to emit radio waves in an area and generate the first reflection point information of the area based on the emitted radio waves, and obtaining the image from a camera configured to capture an image of the area.

[0006] Removing the reflection points may include determining whether a noisy reflection point reaching a ceiling multiple times is present in reflection point information acquired multiple times, and removing the noisy reflection point from the first reflection point information when the noisy reflection is not present in the reflection point information of the multiple ceiling-reaching acquisition.

[0007] Detecting the stationary horizontal structure above the roadway based on the generated region proposals may include discarding the region proposals corresponding to moving objects from the generated region proposals.

[0008] Detecting the stationary horizontal structure above the roadway based on the generated region proposals may further include detecting the horizontal structure based on dimensions of the remaining generated region proposals that do not include the discarded region proposals.

[0009] Projecting the second reflection point information onto the camera image may include associating a location of the second reflection point information with an image position.

[0010] Extracting the visual features from the camera image according to the convolutional neural network may involve applying a bank of kernels to the input image and determining a threshold for a convolutional response output.

[0011] The method may further include obtaining the camera image from a camera facing an area in front of the vehicle.

[0012] The method may include obtaining the first reflection point information from a long-range radar.

[0013] According to one aspect, an apparatus for determining the ground clearance of a structure comprises a long-range radar and a camera, wherein the long-range radar is configured to emit radio waves in an area and generate first reflection point information of the area based on the emitted radio waves, and wherein the camera is configured to capture an image of the area, and the computer-executable instructions can cause the at least one processor to obtain the first reflection point information from the radar and obtain the image from the camera. The apparatus further comprises at least one memory containing computer-executable instructions and at least one processor configured to read and execute the computer-executable instructions.The computer-executable instructions cause the at least one processor to remove reflection points caused by the noise from the first reflection point information based on temporal persistence and generate second reflection point information that extracts visual features from an image of the camera based on a convolutional neural network, wherein the second reflection point information is projected onto the image, wherein region proposals are generated based on projected second reflection point information to the image, wherein the region proposals detect stationary horizontal structures above a roadway based on generated region proposals, and determining the distance between the ground and the detected stationary horizontal structure based on projected reflection point information and the image.

[0014] The computer-executable instructions may cause the at least one processor to remove the reflection points caused by the noise by determining whether a noisy reflection point is present a threshold number of times in the multiple-acquired reflection point information and by removing the noisy reflection point from the first reflection point information if the noisy reflection point is not present the threshold number of times in the reflection point information.

[0015] The computer-executable instructions may cause the at least one processor to detect the stationary horizontal structure above the roadway based on the generated region proposals by discarding the region proposals corresponding to moving objects from the generated region proposals.

[0016] The executable instructions may further cause the at least one processor to detect the stationary horizontal structure above the roadway based on the generated region proposals by detecting the horizontal structure based on dimensions of the remaining generated region proposals that do not include the discarded region proposals.

[0017] The computer-executable instructions may further cause the at least one processor to project the second reflection point information onto the camera image by associating a location of the second reflection point information with a position on the image.

[0018] The computer-executable instructions may further cause the at least one processor to extract visual features from the image of the camera according to the convolutional neural network by applying a bank of kernels to the input image and determining a threshold for convolutional activation.

[0019] The computer-executable instructions may further cause the at least one processor to obtain the camera image from a camera facing an area in front of a vehicle.

[0020] The computer-executable instructions may further cause the at least one processor to obtain the first reflection point information from a long-range radar sensor.

[0021] The computer-executable instructions may further cause the at least one processor to extract visual features from the image of the camera according to the convolutional neural network by applying a bank of kernels to the input image, determining a threshold for convolutional activation, and scaling a feature map size.

[0022] Further purposes, advantages and novel features of the embodiments will become apparent from the following detailed description of the embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a block diagram of an apparatus that determines the ground clearance of a structure according to an exemplary embodiment; Fig. 2 shows a flowchart of a method for determining a ground clearance of a structure according to an exemplary embodiment; Fig. 3 shows an illustration of region proposals according to an aspect of an exemplary embodiment; and Fig. 4 shows a flowchart of ground clearance estimation according to an exemplary embodiment. DETAILED DESCRIPTION OF EMBODIMENTS

[0023] An apparatus and method for determining the ground clearance of a structure will now be described in detail with reference to the Fig. 1 - 4 of the accompanying drawings, in which like reference numbers refer to like elements.

[0024] The following disclosure will enable those skilled in the art to practice inventive concepts. However, the exemplary embodiments disclosed herein are merely exemplary and do not limit the scope of the embodiments described herein. Furthermore, descriptions of the features or aspects of each exemplary embodiment should normally be considered available to aspects of other embodiments.

[0025] It is also to be understood that where it is stated herein that a first element is "connected to", "formed on", or "applied" to a second element, the first element may be directly connected to, directly formed on, or directly disposed on the second element, and that intermediate elements may be present between the first element and the second element unless it is stated that a first element is "directly" connected to, attached to, formed on, or disposed on the second element.Furthermore, when a first element is configured to "send" or "receive" information from a second element, the first element may send or receive the information directly to or from the second element, send or receive the information over a bus, send or receive the information over a network, or send or receive the information through intermediate elements, unless the first element is indicated to send or receive information "directly" to or from the second element.

[0026] Throughout the disclosure, one or more of the disclosed elements may be combined into a single device or combined into one or more devices. Additionally, individual elements may be provided on separate devices.

[0027] Vehicles such as trucks can sometimes pull trailers that can be towed by the vehicle. Some trailers that can be towed by a vehicle include a fifth wheel, a gooseneck hitch, or a low-profile trailer hitch. The vehicle can pull these trailers beneath horizontal structures while passing under a bridge or highway overpass. Often, the height of the trailer can exceed the height of the vehicle. In addition, ground clearance, or the distance between the ground and a horizontal structure located above the vehicle's path, may not be easily determined.

[0028] To solve the above problems, vehicle operators can view information provided by one or more sensors. The sensors may include ultrasonic sensors, radar sensors, and image sensors. The information from the sensors may indicate both the ground clearance of a horizontal structure and the height of the vehicle and / or trailer. This information can be used to determine if there is sufficient ground clearance below the horizontal structure and to warn a vehicle operator in situations where there is a possibility of a collision between a horizontal structure and the vehicle or trailer.

[0029] Fig. 1 shows a block diagram of an apparatus for determining the ground clearance of a structure 100 according to an exemplary embodiment. As in Fig. 1, the apparatus for determining the ground clearance of a structure 100 according to an exemplary embodiment includes a controller 101, a power supply 102, a memory 103, an output 104, a radar sensor 105, a user input 106, an image sensor 107, and a communication device 108. However, the apparatus for determining the ground clearance of a structure 100 is not limited to the aforementioned configuration and may be configured to include additional elements and / or omit one or more of the aforementioned elements. The apparatus for determining the ground clearance of a structure 100 may be implemented as part of a vehicle, as a standalone component, as a hybrid between a vehicle and a non-vehicle device, or another computing device.

[0030] The controller 101 controls the overall operation and function of the device for determining the ground clearance of a structure 100. The controller 101 may control one or more of a memory 103, an output 104, a radar sensor 105, a user input 106, an image sensor 107, and a communication device 108 of the device for determining the ground clearance of a structure 100. The controller 101 may include one or more of a processor, a microprocessor, a central processing unit (CPU), a graphics processor, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, circuitry, and a combination of hardware, software, and firmware components.

[0031] The controller 101 is configured to send and / or receive information from one or more of the memory 103, the output 104, the radar sensor 105, the user input 106, the image sensor 107, and the communication device 108 of the device for determining the ground clearance of a structure 100. The information can be sent and received via a bus or network, or can be read or written directly from one or more of the memory 103, the output 104, the radar sensor 105, the user input 106, the image sensor 107, and the communication device 108 of the device for determining the ground clearance of a structure 100. Examples of suitable network connections include a Controller Area Network (CAN), a Media Oriented System Transfer (MOST), a Local Area Network (LIN), a Local Area Network (LAN), wireless networks such as Bluetooth and 802.11, and other suitable connections, such asEthernet.

[0032] The power supply 102 supplies power to one or more of the controller 101, memory 103, output 104, radar sensor 105, user input 106, image sensor 107, and communication device 108 of the device for determining the ground clearance of a structure 100. The power supply 102 may include one or more of a battery, an outlet, a capacitor, a solar energy cell, a generator, a wind energy device, an alternator, etc.

[0033] The memory 103 is configured to store and retrieve information used by the device for determining the ground clearance of a structure 100. The memory 103 can be controlled by the controller 101 to store and retrieve information received from the radar sensor 105 or the image sensor 107. The stored information can include image information acquired by the image sensor 107, including information of visual features, objects, structures, object movements, etc. In addition, the stored information can include time point information corresponding to the time captured by the radar sensor 105, including reflection point information. In addition, the stored information can also include convolutional neural networks used to identify objects, structures, visual features, etc.The memory 103 may also contain the computer-executable instructions configured to be executed by a processor to perform the functions of the apparatus for determining the ground clearance of a structure 100.

[0034] The memory 103 may include one or more of floppy disks, optical disks, CD-ROMs (Compact Disc-Read Only Memories), magneto-optical disks, ROMs (Read Only Memories), RAMs (Random Access Memories), EPROMs (Erasable Programmable Read Only Memories), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, cache memory, and other types of media / machine-readable media suitable for storing machine-executable instructions.

[0035] Output 104 outputs information in one or more forms, including visual, audible, and / or haptic. Output 104 may be controlled via controller 101 to provide outputs to the user of the structure ground clearance device 100. Output 104 may include one or more of a speaker, an audio device, a display, a centrally located display, a head-up display, a windshield display, a haptic feedback device, a vibration device, a tactile feedback device, a tap feedback device, a holographic display, an instrument light, an indicator light, etc.

[0036] Output 104 may output notifications that include one or more of an audible notification, a light notification, and a display notification, etc. The notification may indicate that there is sufficient ground clearance for the vehicle or trailer, or may indicate that there is insufficient clearance and warn of a potential collision between the horizontal structure and the vehicle or trailer. Additionally, output 104 may output an image showing an area ahead of the vehicle, including one or more visual features, the horizontal structure, ground clearance, or moving objects.

[0037] The radar sensor 105 is configured to emit radio waves and provide reflection point information and corresponding time point information. According to a first example, the radar sensor 105 may be a long-range radar sensor. The radar sensor 105 may include a transmitter or transmitting antenna configured to emit one or more of radio waves, electromagnetic waves, and microwaves. The radar sensor 105 may include a receiver or receiving antenna configured to receive one or more of radio waves, electromagnetic waves, and microwaves that reflect an object or structure on a roadway ahead of the radar sensor. In one example, the reflection point information may indicate the position, speed, or distance to an object or structure.

[0038] The user input 106 is configured to provide information and instructions to the device for detecting the ground clearance of a structure 100. The user input 106 can be used to provide user input, etc., to the controller 101. The user input 106 can include one or more of a touchscreen, a keyboard, a soft keyboard, a button, a motion detector, a voice input detector, a microphone, a camera, a trackpad, a mouse, a touchpad, etc. The user input 106 can be configured to receive user input to confirm or dismiss the notification through the output 104. The user input 106 can also be configured to receive user input to indicate a vehicle height, a trailer height, or a required ground clearance.In addition, the user input 106 may also be configured to receive input to enable or disable the device for determining the ground clearance of a structure 100.

[0039] The image sensor 107 may include one or more of a variety of sensors, including an image sensor, a camera, an infrared camera, a video camera, and a LIDARS. The image sensor 107 may provide one or more images from one or more forward-facing cameras or image sensors, which may be analyzed to identify visual features and to determine which of the visual features correspond to a horizontal structure above a roadway in front of the image sensor 107.

[0040] Additionally, other types of information, such as infrared images, a distance to, a speed from, an rpm from, an acceleration from, a direction of travel from, a dimension of objects, structures, or vehicles may be provided by the radar sensor 105 or the image sensor 107, or other vehicle sensors. In one example, the focal length of a camera of the image sensor 107, the detection of an edge and a visual feature, and / or a pixel coordinate and distance information may be used to analyze an image provided by the image sensor 107 to determine dimensions and locations of objects, structures, etc. The dimensions and locations of the objects and structures in multiple images at multiple different times may be analyzed by the controller 101 to determine the other types of information disclosed above.

[0041] The communication device 108 can be used by the structure ground clearance determination device 100 to communicate with various types of external devices according to various communication methods. The communication device 108 can be used to send / receive information including reflection point information, timing information from the radar sensor 105, information from the image sensor 107, such as image information, and other types of information.

[0042] The communication device 108 may include various communication modules, such as one or more broadcast receiving modules, a short-range communication (NFC) module, a GPS module, and a wireless communication module. The broadcast receiving module may include a terrestrial broadcast receiving module, which includes an antenna for receiving a terrestrial broadcast signal, a demodulator, and an equalizer, etc. The NFC module is a module that communicates with an external device located at a nearby distance according to an NFC method. The GPS receiver is a module that receives a GPS signal from a GPS satellite and detects a current location. The wired communication module may be a module that receives information via a wired network, such as a local area network, a controller area network (CAN), or an external network.The wireless communication module is a module that connects to and communicates with an external network using a wireless communication protocol, such as IEEE 802.11, WiMAX, Wi-Fi, or IEEE communication protocol. The wireless communication module may further include a mobile communication module that accesses a mobile communication network and performs communication according to various mobile communication standards, such as 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), Bluetooth, EVDO, CDMA, GPRS, EDGE, or Zigbee.

[0043] According to an exemplary embodiment, the controller 101 of the device for determining the ground distance of a structure 100 can be configured to remove the reflection points caused by the noise from the first reflection point information based on temporal persistence and to generate second reflection point information, extract visual features from an image of a camera based on a convolutional neural network, wherein the second reflection point information is projected onto the image, wherein region proposals are generated based on projected second reflection point information from the image, wherein the region proposals detect stationary horizontal structures above a roadway based on generated region proposals, and determining the distance between the ground and the detected stationary horizontal structure based on projected reflection point information and the image.The controller 101 of the apparatus for determining the ground clearance of a structure 100 may be configured to obtain the first reflection point information from the radar and obtain the image from the camera.

[0044] The convolution operation can apply a bank of small (e.g., 3x3) kernels to an input image. A rectified linear unit (ReLU) can then be applied to a threshold with respect to the convolution response output. A max-pooling operation can select the maximum value from a 2x2 template while downscaling the feature map size by a factor of 1 / 2. The convolutional neural network can generate a stack of feature maps at different scales; the lower layer can have a small receptive field for detecting small objects, and a higher layer can have a larger receptive field for detecting large objects. A range map, e.g., a map projecting radar reflectance points onto the image plane, is concatenated with the visual feature maps.

[0045] The controller 101 of the apparatus for determining the ground clearance of a structure 100 may be configured to remove the reflection points by determining whether a noisy reflection point reaching an upper limit multiple times is present in multiple acquired reflection point information, and to remove the noisy reflection point from the first reflection point information if the noisy reflection is not present in the reflection point information reaching an upper limit multiple times and / or for a predetermined period of time.

[0046] The controller 101 of the structure ground clearance determination device 100 may be configured to detect the stationary horizontal structure above the roadway based on the generated region proposals by discarding the region proposals corresponding to moving objects from the generated region proposals.

[0047] The controller 101 of the structure ground clearance determination device 100 may be configured to detect the stationary horizontal structure above the roadway based on the generated region proposals by detecting the horizontal structure based on the dimensions of the remaining generated region proposals to which the discarded region does not belong.

[0048] The controller 101 may be configured to project the second reflection point information onto the image of the camera by associating a location of the second reflection point information with a position on the image.

[0049] The controller 101 may also be configured to obtain the camera image from a camera facing an area in front of a vehicle and to obtain the first reflection point information from a long-range radar sensor.

[0050] Fig. 2 shows a flowchart for a method for the device for detecting the trailer according to an exemplary embodiment. The method of Fig. 2 may be performed by the apparatus for determining the ground clearance of a structure 100 or may be encoded into a computer-readable medium as instructions executable by a computer to perform the method.

[0051] With reference to Fig. 2, reflection points caused by noise are removed from the first reflection point information based on temporal persistence, and the second reflection point information is generated in operation S210. In one example, the noisy reflection points are removed when the noisy reflection points are present in the reflection point information for a period of time less than a predetermined threshold time, or when the noisy reflection points are present in a number of scans that is less than the predetermined threshold number of scans of the radar.

[0052] In operation S220, the visual features are extracted from an image captured by a camera. The visual features are extracted by using a convolutional neural network to identify the visual features to be extracted. The second reflection point information is projected onto the image in operation S230. In one example, the second reflection point information is projected onto a position in the image that corresponds to a real-world position of the reflection point information.

[0053] Based on the projected second reflection point information and the image, area suggestions indicating potential horizontal structures are generated in operation S240. In operation S250, a stationary horizontal structure above a roadway located in front of a camera is detected. Then, the distance between the ground and the detected stationary horizontal structure is determined in operation S260 based on the projected reflection point information and the image.

[0054] Fig. 3 shows an illustration of the range suggestions according to one aspect of an exemplary embodiment. With reference to Fig. 3, the area suggestions are illustrated by rectangles or outlines that correspond to visual features in the image.

[0055] Rectangles 301 correspond to the detected horizontal stationary structure above the roadway in front of the camera. Rectangles 302 correspond to moving objects that can be removed from the region suggestions because they do not correspond to a horizontal stationary structure above the roadway in front of the camera. Furthermore, rectangles 303 correspond to all visual features that can be detected in an image for use with a convolutional neural network.

[0056] Fig. 4 shows a flowchart of ground clearance estimation according to one aspect of an exemplary embodiment.

[0057] With reference to Fig.4, visual feature maps 401 are generated from an image of a roadway in front of the camera. A range map 402, determined from information provided by the radar or reflection point information, is analyzed. Region proposals 403 are then generated based on the range map 402. The region proposals can be captured by retaining cells within the feature map that have the strongest local response and that correlate with the presence of radar reflection points. Range proposal pooling 404 is performed using region proposals 403 and the visual feature map 401.

[0058] Classification 405 of region proposals is performed using a convolutional neural network to detect a stationary horizontal structure. Ground distance estimation 406 is then performed to detect a distance between the ground and the stationary horizontal structure.

[0059] The processes, methods, or algorithms disclosed herein may be provided / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control device or a dedicated electronic control device. Likewise, the processes, methods, or algorithms may be stored as data or executable instructions by a controller or computer in a variety of ways, including, without limitation, permanent storage on non-writable storage media, such as ROM, and as alterable information on writable storage media, such as floppy disks, magnetic tapes, CDs, RAM, and other magnetic and optical media. The processes, methods, or algorithms may also be implemented in a software-executable object.Alternatively, the processes, methods, or algorithms may be embodied in whole or in part with suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0060] One or more embodiments have been described above with reference to the drawings. The embodiments described above should be considered only in a descriptive sense and not as limiting. Furthermore, the embodiments may be modified without departing from the spirit and scope of the inventive concept, which is defined in the following claims.

Claims

[1] A device for determining the ground clearance of a structure (100), the device comprising: a long-range radar configured to emit radio waves in an area and generate first reflection point information of the area based on the emitted radio waves; a camera configured to capture the image of the area; at least one memory (103) comprising computer-executable instructions; and at least one processor configured to read and execute the computer-executable instructions, wherein the computer-executable instructions cause the at least one processor to: Obtaining the first reflection point information from the radar; Obtaining the image from the camera; Removing (S210) reflection points caused by noise from the first reflection point information based on temporal persistence and generating second reflection point information; Extracting (S220) visual features from an image of the camera based on a convolutional neural network; Projecting (S230) the second reflection point information onto the image; Generating (S240) region suggestions based on the projected second reflection point information and the image, the region suggestions indicating potential horizontal structures above a roadway; Detecting (S250) stationary horizontal structures above the roadway based on the generated region proposals; and Determining (S260) the distance between the ground and the detected stationary horizontal structure based on projected reflection point information and the image. [2] The apparatus of claim 1, wherein the computer-executable instructions cause the at least one processor to remove the reflection points caused by noise by: Determining whether a noisy reflection point exists a threshold number of times in the multiple-acquired reflection point information, and Remove the noisy reflection point from the first reflection point information if the noisy reflection point does not exist the threshold number of times in the reflection point information. [3] The apparatus of claim 1, wherein the computer-executable instructions cause the at least one processor to detect the stationary horizontal structure above the roadway based on the generated region proposals by discarding the region proposals corresponding to moving objects from the generated region proposals. [4] The apparatus of claim 3, wherein executable instructions further cause the at least one processor to detect the stationary horizontal structure above the roadway based on the generated region proposals by detecting the horizontal structure based on dimensions of the remaining generated region proposals not including the discarded region proposals. [5] The apparatus of claim 1, wherein computer-executable instructions further cause the at least one processor to project the second reflection point information onto the camera image by associating a location of the second reflection point information with a position on the image. [6] The apparatus of claim 1, wherein the computer-executable instructions cause the at least one processor to extract visual features from the image of the camera according to the convolutional neural network by applying a bank of kernels to an input image and determining a threshold for convolutional activation. [7] The apparatus of claim 1, wherein the computer-executable instructions cause at least one processor to obtain the camera image from the camera facing an area in front of the vehicle. [8] The apparatus of claim 1, wherein the computer-executable instructions cause the at least one processor to obtain the first reflection point information from a long-range radar sensor. [9] The apparatus of claim 1, wherein the computer-executable instructions cause the at least one processor to extract visual features from the image of the camera according to the convolutional neural network by applying a bank of kernels to an input image, determining a threshold for convolutional activation, and scaling a feature map size.

Citation Information

Patent Citations

  • Method for determining the traversability of an object for a vehicle using a 3D camera

    DE102011113077A1

  • APPEARANCE-BASED UNION OF CAMERA AND DISTANCE DATA FOR MULTI-OBJECTS

    DE102011119767A1