LANE DETECTION WITH REAR VIEW CAMERA

Abstract

Method comprising: Receiving image data of a roadway from a camera of a vehicle; Receiving perception data measuring the reflectance of the road surface from a source comprising a Light Detection and Ranging (LIDAR) system; Determine whether there are lane markings on the roadway; in response to determining that there are no lane markings, estimating a number of lanes on the roadway based on a position of shoulders of the roadway as determined only from the perception data; in response to determining that there are lane markings, identifying which lane on the roadway the vehicle is in by processing merged data using a neural network, the merged data comprising the image data and the reflectance of the roadway as determined by the perception data; and Providing an indication of the number of lanes to an automated driving system or driver assistance system, wherein the automated driving system or driver assistance system actuates one or more vehicle controls based on the indication of the number of lanes.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to assisted or automated driving systems, methods, and apparatus, and more particularly to lane detection using a rearview camera. GENERAL STATE OF THE ART

[0002] Automobiles provide a significant portion of transportation for commercial, government, and private entities. Autonomous vehicles and driver-assistance systems are currently being developed and deployed to provide safety features, reduce the amount of required user input, or even eliminate user involvement entirely. For example, some driver-assistance systems, such as crash avoidance systems, can monitor the driving, locations, and speed of the vehicle and other objects while a human is driving. If the system detects that a crash or impact is imminent, the crash avoidance system can intervene and apply braking, steer the vehicle, or perform other avoidance or safety maneuvers. As another example, autonomous vehicles can drive and navigate a vehicle with little or no user input.However, due to the hazards associated with driving and the cost of vehicles, it is very important that autonomous vehicles and driver assistance systems operate safely and are able to accurately navigate roads in a variety of different roadway and driving environments.

[0003] US Pat. No. 8,457,827 B1 discloses changing the behavior of an autonomous vehicle based on the predicted behavior of other vehicles. For this purpose, the vehicle is equipped with cameras and lidar.

[0004] US 2012 / 0 062 747 A1 discloses a lane fusion system with forward and rearward cameras. The camera data is fused to determine lane information and obtain lane position.

[0005] The document US 2015 / 0 025 789 A1 discloses a technique for lane assignment in a vehicle. For this purpose, trajectory data is fused with lane geometry data. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Non-limiting and non-exhaustive implementations of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the several views unless otherwise indicated. Advantages of the present disclosure may be better understood by reference to the following description and accompanying drawings, in which: Fig. 1 is a schematic block diagram illustrating an implementation of a vehicle control system including an automated driving / assistance system; Fig. 2 illustrates a perspective view of an exemplary road environment; Fig. 3 illustrates a perspective view of another exemplary road environment; Fig. 4 illustrates a perspective view of yet another exemplary road environment; Fig. 5 is a schematic block diagram illustrating data flow for determining lane information according to one implementation; Fig. 6 is a schematic diagram illustrating an example configuration of a deep neural network according to one implementation; Fig. 7 is a schematic block diagram illustrating example components of a lane component according to one implementation; Fig. 8 is a schematic block diagram illustrating a method for determining lane information according to one implementation; and Fig. 9 is a schematic block diagram illustrating a computing system according to one implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] An automated driving system or driver assistance system may use data from a variety of sources during decision-making, navigation, or driving to determine optimal paths or maneuvers. For example, an automated driving / assistance system may include sensors to sense a driving environment in real time and / or access local or remote data storage to obtain specific details about a current position or positions along a planned travel path. In one embodiment, details about positions visited by a host vehicle may be stored in a driving history database for later access. For example, when a vehicle returns to a position for which driving history data exists, the automated driving / assistance system may use data from the driving history to obtain details about a position that may not (yet) be apparent to the vehicle's sensors.

[0008] Since driving history data is extremely useful for an automated driving / assistance system, the applicant has recognized the advantages of utilizing available sensors and computing power in a vehicle to acquire driving history data. In the present application, the applicant discloses systems, methods, and apparatus for acquiring lane line information using a rearview or rearview camera of a vehicle, which can be stored in a driving history. According to an exemplary embodiment, a system includes a perception data component, a lane component, and a reporting component. The perception data component is configured to receive perception data from at least two sensors, wherein the at least two sensors include a rear-facing camera of a vehicle.The lane component is configured to determine, using a neural network, a number of lanes on a roadway within a field of view captured by the perception data. The reporting component is configured to provide an indication of the number of lanes to an automated driving system or driver assistance system.

[0009] The rear-facing cameras on vehicles are typically used for support and are inactive or unused while a vehicle is moving forward. Embodiments disclosed herein use rearview cameras to examine a roadway to determine a number of lanes. For example, in some embodiments, the rearview cameras can be caused to capture video / images whenever the vehicle is started or is moving (forward or backward). Information about the number of lanes can be used to inform or update a driving history database. In one embodiment, the data from the rearview camera can also be used to detect which lane the vehicle is currently in (i.e., from the time an image or perception data was collected).In one embodiment, detecting a number of lanes or a current lane may be performed using a machine learning algorithm, such as a deep neural network (DNN).

[0010] In one embodiment, a driving system can determine how many lanes a road is wide while reversing or while the camera is inactive while the vehicle is moving forward. The system can also use the rearview camera to determine which lane the vehicle is currently in. Additional sensors can be used in conjunction with the rearview camera to generate aggregated sensor data. For example, a LIDAR (light detection and ranging) system can be used to collect data such as measuring surface reflectivity to improve the robustness and localization accuracy of lane markings. Radar can be used to detect curbs or the edges of the road. The fusion of LIDAR, radar, and / or the rearview camera can enable robust detection of lane markings under a variety of different conditions.

[0011] In one embodiment, when no lane markings are present, a system may switch to a pure LIDAR mode or a pure radar mode (or a pure LIDAR and radar mode), since a LIDAR system or radar system may be able to detect the shoulders of the lane. To determine the number of lanes, the distance between the shoulders may be measured and divided by the standard lane width. Any fractional remainder may be apportioned and allocated between the lanes or road shoulders. In one embodiment, LIDAR and / or radar may be used to subdivide the road plane and thus determine the grade of the road in both the horizontal and vertical directions. Applicant contemplates the use and / or merging of other types of sensor data for determining the lane number as being within the scope of the present disclosure.

[0012] The number of lanes and / or the current lane may be provided to an automated driving system or driver assistance system. For example, the system may use the lane information to determine how to drive the vehicle, how to avoid obstacles, or the like. The number of lanes and the current lane may be sent to all controllers of an automated driving system or driver assistance system for use in decision-making, navigation, or the like. In one embodiment, the current number of lanes and the currently occupied lane may be used to calculate safety avoidance options and the autonomous trajectory. As another example, a user may be provided with a notification regarding the number of lanes and / or the current lane of the vehicle (e.g., on a front display or a navigation screen).

[0013] In one embodiment, the number of lanes at a particular location or along a particular roadway may be stored in a driving history database for later retrieval. In one embodiment, the number of lanes may be uploaded to a remote server or to the cloud. The remote storage location may include a data store accessible via a mobile communications network and / or the Internet. The remote storage location may also receive and store data from a large number of other vehicles to generate a cloud-based driving history shared by the various vehicles. The data may be aggregated or compiled to allow the vehicles to access the shared data when they have access to communications networks.On subsequent trips, the current lane count can be retrieved from either a local trip history database or the cloud trip history database. In one embodiment, the lane count is sent to all controllers on a vehicle control bus of the vehicle to determine travel paths or the like.

[0014] Further embodiments and examples are explained below with reference to the figures.

[0015] Referring to the figures, Fig. 1 illustrates an exemplary vehicle control system 100. The vehicle control system 100 includes an automated driving / assistance system 102. The automated driving / assistance system 102 may be used, for example, to automate or control the operation of a vehicle or to assist a human driver. The automated driving / assistance system 102 may control one or more of braking, steering, acceleration, lights, alarms, driver notifications, radio, and / or any other driving or auxiliary systems of the vehicle. In another example, the automated driving / assistance system 102 may not be capable of providing any driving control (e.g., steering, acceleration, or braking), but may provide notifications and alarms to assist a human driver in driving safely.For example, the automated driving / assistance system 102 may include one or more controllers (such as those discussed herein) that provide or receive data via a control bus and use the data to determine actions to be performed and / or provide instructions or signals to initiate those actions. The automated driving / assistance system 102 may include a lane component 104 configured to determine a number of lanes and / or a current lane based on images or video from a rear-facing camera.

[0016] The vehicle control system 100 also includes one or more sensor systems / devices for detecting the presence of nearby objects, lane markings, and / or determining a position of a host vehicle (e.g., a vehicle including the vehicle control system 100). The vehicle control system 100 may include, for example, radar systems 106, one or more LIDAR systems 108, one or more camera systems 110, a global positioning system (GPS) 112, and / or ultrasonic systems 114. The vehicle control system 100 may include a data store 116 for storing relevant or useful data for navigation and safety, such as map data, trip history (i.e., driving history), or other data.The vehicle control system 100 may also include a transceiver 118 for wirelessly communicating with a mobile or wireless network, other vehicles, infrastructure, the cloud, or remote computing or storage resources, or any other communication system.

[0017] The vehicle control system 100 may include vehicle control actuators 120 to control various aspects of driving the vehicle, such as electric motors, switches, or other actuators to control braking, acceleration, steering, or the like. The vehicle control system 100 may also include one or more displays 122, speakers 124, or other devices so that notifications can be provided to a human driver or occupants. A display 122 may include a front-panel display, a dashboard display or indicator, a screen, or any other visual display device visible to a driver or occupants of a vehicle. The speakers 124 may include one or more speakers of a vehicle's sound system or may include a speaker dedicated for driver notification.The vehicle control actuators 120, the displays 122, the speakers 124, or other parts of the vehicle control system 100 may be controlled by one or more controllers of the automated driving / assistance system 102.

[0018] In one embodiment, the automated driving / assistance system 102 is configured to control the driving or navigation of a host vehicle. For example, the automated driving / assistance system 102 may control the vehicle control actuators 120 to travel along a path within travel lanes on a road, parking lot, driveway, or other location. For example, the automated driving / assistance system 102 may determine a path based on information or perception data provided by any of the components 106-118. The sensor systems / devices 106-110 and 114 may be used to obtain real-time sensor data so that the automated driving / assistance system 102 can assist a driver or drive a vehicle in real time.In one embodiment, the automated driving / assistance system 102 also uses information stored in a driving history (locally or remotely) to determine conditions in a current environment. The automated driving / assistance system 102 may implement one or more algorithms, applications, programs, or functionality that drive or assist the vehicle in driving.

[0019] In one embodiment, camera systems 110 may include a rearward-facing camera, such as a rearview camera. For example, camera systems 110 may include cameras facing in different directions to provide different views and different fields of view for areas near or around the vehicle. For example, some cameras may face forward, sideways, backward, angled, or in any other direction. In one embodiment, images from a rearview camera may be used to determine a number of lanes on a roadway and / or a current lane for the vehicle.

[0020] It must be acknowledged that the design of the Fig. 1 is merely an example. Other embodiments may include fewer or additional components without departing from the scope of the disclosure. Furthermore, illustrated components may be combined or incorporated into other components without limitation.

[0021] Fig. 2 illustrates a perspective view of a roadway 200. The view illustrates what may be captured in an image from a rear-facing camera, such as a rearview camera, of a vehicle. The roadway 200 includes a variety of markings, including lane division markings 204, roadway separation markings 202, and rumble strips 206. The roadway 200 may include a roadway for a highway or freeway where all lanes of the roadway 200 travel in the same direction. The markings may include mechanical and non-mechanical markings. Mechanical markings may include reflectors, rumble strips (e.g., 206), or the like. Non-mechanical markings may include colored lines or markings (white, yellow, etc.) created with paint, plastics, tape, epoxy, or the like.

[0022] In one embodiment, a lane component 104 is configured to detect and determine a number of lanes on the roadway 200. For example, the lane component 104 may identify the road boundary markers 202 and determine that three lanes exist based on two rows of lane division markers 204. It is important to note that images captured using a rear-facing camera acquire information that is reversed compared to that from a forward-facing camera or from a human driver's perspective. For example, if a centerline is generally to the left of a vehicle in a particular driving position, a rear-facing camera may capture images showing the centerline on a right side of the image.Thus, any lane count, lane positioning, and lane detection algorithms that use data from rear-facing cameras must reverse the sequences or detection rules to reflect a common format as seen from other sensors or from a driver's perspective.

[0023] The lane component 104 may use the presence of rumble strips 206, as well as a marking color (white, yellow, etc.) or pattern (broken or solid line), to determine the boundaries of a roadway (or the outermost lane boundary of a roadway). For example, the roadway boundary markings 202 include a solid line pattern, while the lane divider markings 204 include a broken line pattern. The lane component 104 may identify solid lines as boundary markings and broken lines as lane divider markings. Other types of lane markings (such as reflectors) may be identified as roadway boundary markings, centerlines, lane dividers, or the like based on color, frequency, or the like.In light of the present disclosure, one skilled in the art will understand that boundary marking type detection can be applied to any type of lane marking, whether mechanical or non-mechanical.

[0024] As in Fig. 1, the automated driving / assistance system 102 may also include a position component 126 and a driving history component 128. The position component 126 may determine a current position of the vehicle in which the system 100 is located. For example, the position component 126 may receive position information from the GPS 112 and / or the transceiver 118 indicating a position of the vehicle. The driving history component 128 is configured to retrieve data from a driving history (i.e., trip history) and provide it to other controllers or portions of the system 100. For example, data in a driving history (stored locally in the data store 116 or stored remotely at a location accessible over a network using the transceiver 118) may be retrieved for a current or future position to inform the automated driving / assistance system 102 about road or driving conditions.In one embodiment, the driving history component 128 is configured to retrieve driving history data from a remote storage location. The driving history may indicate a number of lanes or a gradient for a current position of a vehicle. In one embodiment, the driving history component 128 is configured to send the number of lanes or the gradient for the current position to one or more vehicle controllers of an automated driving system or driver assistance system. For example, the controllers may use the data from the driving history to determine how to steer the vehicle to travel a portion of the road. For example, the controllers may determine a steering angle or change in steering angle, a speed or change in speed, acceleration, braking, or any other control input / output values ​​to control a vehicle.

[0025] Fig. 3 illustrates a perspective view of a roadway 300 in a residential area according to one embodiment. The view illustrates what may be captured in an image from a rear-facing camera, such as a rearview camera, of a vehicle traveling through a residential area. Because the roadway 300 does not include conventional mechanical or non-mechanical lane markings, such as lines or reflectors, the number of lanes must be determined based on other factors, such as a width 302 of the roadway 300. The lane component 104 may determine the width 302 of the roadway based on the road edges. For example, curbs 304 on each side of the roadway 300 may be detected using images from a rearview camera and / or data from another sensing system, such as LIDAR data or radar data.LIDAR data and radar data can be particularly useful for detecting curbs or other three-dimensional road or environmental features.

[0026] In one embodiment, aggregated data based on images and other sensor data may be generated to determine a position of curbs, a shoulder, or the like near the roadway. For example, the position of a roadway edge may be determined based on an image processing technique, such as edge detection or boundary detection, or based on LIDAR data. The position or road width determined based on the image data may be averaged (e.g., using a weighted average) with the position or road determined based on the LIDAR data. The calculated / determined width may be divided by a stored lane width to determine how many lanes are present on the roadway 300. For example, the stored lane width may include an average or legal lane width known for a geographic location.In one embodiment, the number of lanes may also be determined based on a road environment. For example, if the vehicle is in the vicinity of a highway (such as the one shown in . Fig. 2), lane widths may be larger than those in residential or industrial areas. The environment may be determined based on the presence of curbs 304, parking lanes 306, driveways 308 or connecting roadways, sidewalks 310, houses, buildings, trees, or the like. For example, the presence of houses and / or sidewalks within a certain distance from the road may trigger the use of a smaller average lane width. In one embodiment, the absence of lane or other road markings may indicate that a smaller lane width should be used.

[0027] Fig. 4 illustrates a perspective view of a roadway 400 in a commercial setting according to one embodiment. The view illustrates what may be captured in an image from a rear-facing camera, such as a rearview camera, of a vehicle. Similarly, a LIDAR or radar system may capture information about one or more features of the roadway 400 or in the vicinity of the roadway 400. The roadway 400 is a two-way roadway with a variety of markings, including centerline markers 402, lane split markers 404, and roadway separation markers 406. A curb 408 is proximate the roadway 400, and a cross street 410 intersects the roadway 400. In one embodiment, a lane component 104 is configured to detect and determine a number of lanes on the roadway 400.For example, the lane component 104 may identify the road boundary markings 406 and determine that four lanes exist on the roadway based on the two rows of lane split markings 404 and the centerline markings 402.

[0028] The lane component 104 may also determine a direction of travel for each lane based on the presence of the centerline markings 402. The lane component 104 may identify the centerline markings 402 based on the line pattern (e.g., double solid or double broken lines) or color (e.g., yellow versus white). The lane component 104 may determine that the lanes on one side of the centerline markings 402 of the roadway 400 are for travel in a first direction and that lanes on the other side of the centerline markings 402 of the roadway 400 are for travel in a second, opposite direction.

[0029] The number of lanes, directions of travel, and other details may be stored in a local or remote driving history for later access. In one embodiment, the number of lanes, their corresponding directions, or other lane information may be provided to a control system or automated driving / assistance system for determining a travel path or collision avoidance maneuvers. For example, an automated driving / assistance system may determine whether a maneuver is available to avoid a collision with an object, pedestrian, or other vehicle remaining in the roadway 400 but avoiding entering an oncoming lane. In some cases, entering an oncoming lane or exiting the roadway 400 may be preferred over alternative paths that may result in an impending collision.

[0030] Fig. 5 is a schematic block diagram illustrating a data flow for a method 500 for determining a number of lanes and / or a current lane for a vehicle. A variety of perception data types, including camera data, radar data, LIDAR data, and / or ultrasonic data, may be received and combined for sensor fusion 502. The camera data may include data from a rear-facing camera, such as a rearview camera. Sensor fusion 502 may generate information about lane marking position, curb position, a shoulder, or the position of any other object or feature in the environment based on the combined perception data.

[0031] For example, if only camera data and LIDAR data are received, then a position for a lane marker may be determined based on an average or other combination of camera and LIDAR data. Sensor merging 502 may use averages or weighted averages for different data types to determine merged or combined data. If only one type of data is received, sensor merging may run through that raw data or modify the raw data to conform to a format expected by neural networks 504. Neural networks 504 may receive the raw or merged data and process it to generate a number of lanes for a roadway and / or a current lane for a vehicle on the roadway.

[0032] The neural networks 504 may include one or more networks that calculate one or more outputs, including the number of lanes and current lanes. Since data from a rear-facing camera may be used, the number of lanes may be based on a portion of the roadway over which the vehicle has already traveled. However, this data may still be useful for automated driving or assisted driving, as it can help inform assumptions about the number of lanes ahead of a vehicle. Additionally, storing the number of lanes in a driving history can be retrieved at a later time as the user approaches or reapproaches a road, and the previously stored lane data can assist a vehicle in determining the characteristics of a roadway the vehicle will soon travel.In one embodiment, neural networks 504 include one or more deep neural networks that have been qualified to detect a number of lanes and / or a current lane of a vehicle within the detected lanes.

[0033] Fig. Figure 6 is a schematic diagram illustrating a configuration of a deep neural network 600. Deep neural networks have gained attention in recent years because they have surpassed conventional machine learning concepts in challenging tasks such as image classification and speech recognition. Deep neural networks are connected algorithmic graphs with input nodes (such as input node 602), one or more hidden layers (such as hidden layers 604, 606, and 608), and output nodes (such as output node 610). To classify content or information about an image, pixel values ​​in the input image are assigned to input nodes and then passed through the hidden layers 604, 606, and 608 of the network, undergoing a number of nonlinear transformations. At the end of the computation, the output nodes 610 yield values ​​corresponding to the class inferred by the neural network.The number of input nodes 602, hidden layers 604-608, and output nodes 610 is illustrative only. For example, larger images may include an input node 602 for each pixel and may thus have hundreds, thousands, or other numbers of input nodes.

[0034] According to one embodiment, a deep neural network 600 of the Fig. 6 can be used to classify the content(s) of an image into four different classes: a first class, a second class, a third class, and a fourth class. According to the present disclosure, a similarly or differently sized neural network may be capable of outputting a value indicative of a number of lanes in an image. For example, the first class may correspond to one lane, the second class may correspond to two lanes, the third class may correspond to three lanes, and the fourth class may correspond to four lanes. This is only illustrative, as a neural network for classifying the number of lanes based on an image may include hundreds or thousands of pixels and may need to detect a larger number of lanes (e.g., ten or more in some embodiments).Thus, a neural network to classify the number of lanes may require hundreds or thousands of nodes at an input layer and / or more than four output nodes.

[0035] For example, feeding a raw image of the roadway 200 of the Fig. 2 into the network yields the highest probability for the third class, or three lanes. Thus, the neural network can enable a computing system to automatically infer the number of lanes within an image. Similar techniques or principles can be used to derive information about or detect pedestrians, traffic signs, a current lane, and / or the like.

[0036] In order for the deep neural network to be able to distinguish between all desired classes, the neural network must be qualified based on examples. For example, if a deep neural network is to be created that can classify a number of lanes in an image, then a large number of example images (hundreds to thousands for roads with different numbers of lanes) with a label assigned to each image corresponding to the number of lanes within each specific image may be required. Labeled data can be a significant challenge for qualifying deep neural networks because humans are required to assign the labels to the training images (which often number in the millions). Thus, the time and equipment required to process the image, as well as manually labeling them, can be expensive.Once the images with the labels (training data) have been developed, the network can be qualified. An example qualification algorithm involves the backpropagation algorithm, which uses the images, including the large number of images with labels. The backpropagation algorithm can take several hours, days, or weeks to complete.

[0037] Referring to Fig. Figure 7 shows a schematic block diagram illustrating components of a lane component 104 according to one embodiment. The lane component 104 includes a perception data component 702, a merge component 704, a lane count component 706, a current lane component 708, a slope component 710, a maneuver component 712, and a report component 714. Components 702-714 are for illustrative purposes only, and not all may be included in all embodiments. Indeed, some embodiments may include only one or any combination of two or more of the components 702-714. Some of the components 702-714 may reside external to the lane component 104, such as within the automated driving / assistance system 102 or elsewhere.

[0038] The perception data component 702 is configured to obtain or receive perception data from one or more sensors or sensing systems of a vehicle or a vehicle control system. In one embodiment, the perception data component 702 receives perception data that includes information about an environment of a vehicle or a vehicle control system. Example perception data includes data from LIDAR, radar, camera, ultrasonic, infrared, or other systems. In one embodiment, the perception data component 702 is configured to receive perception data from at least two sensors or sensing systems. In one embodiment, the at least one of the sensors or sensing systems includes a rear-facing camera of a vehicle. Data from other sensors may also be received, such as data from a LIDAR, radar, ultrasonic, infrared, or other system.

[0039] The merge component 704 is configured to perform data merging with perception data obtained from the perception data component 702. For example, the merge component 704 may populate data from the perception data into fields or entries expected by one or more of the other components 702, 706-714. For example, if an image is expected from the lane count component 706 (but may not be needed), the merge component 704 may provide an image in a table or matrix to be provided to the lane component. Similarly, if lidar data could be used by a component, the merge component 704 may provide the lidar data in a different field or area of ​​the table or matrix.The merging component 704 may compile perception data from various data sources for use by a lane count component 706, a current lane component 708, and / or a slope component 710 for processing using a neural network or other machine learning algorithm or model.

[0040] In one embodiment, the merging component 704 is configured to generate merged sensor data based on the sensing data from at least two sensors. For example, the merged sensor data may include a position of a lane marking, a number of lanes, or the like. For example, the merging component 704 may determine the presence or position of one or more lane lines based on data from many sensors. For example, data from one sensor may indicate the presence of a lane marking with high confidence, while data from another sensor may indicate the presence of a lane marking with low or medium confidence. Based on the combined data, the merging component 704 may determine that a lane marking is present.Similarly, data from one sensor may indicate a first position of a lane marking, while data from another sensor may indicate that the lane marking has a slightly different position. The merging component 704 may determine a new or modified position calculated based on the combined data.

[0041] The lane count component 706 is configured to determine a number of lanes on a roadway based on the perception data. In one embodiment, the lane count component 706 uses perception data that has been merged or processed by the merging component 704. In another embodiment, the lane count component 706 uses raw data or data that has not been merged or processed by the merging component 704. In one embodiment, the lane count component 706 includes one or more neural networks that have been qualified to detect a number of lanes within an image, a LIDAR frame, a radar frame, or another frame or unit of sensor data. Other embodiments may include other types of machine learning algorithms or models for determining the number of lanes.The lane count component 706 may receive perception data as input and provide an output indicating a number of inferred lanes. The output of the neural network or other machine learning algorithm may indicate to other components or systems how many lanes were present within a field of view of the perception sensors when the perception data was captured / acquired.

[0042] The current lane component 708 is configured to determine a current lane of the vehicle. For example, the current lane component 708 may determine a current lane of a vehicle within the number of lanes detected by the lane count component based on the perception data. For example, the current lane component 708 may determine a current lane of the vehicle based on angles formed by lane lines or consecutive lane markings, a position within an image or a LIDAR frame, or the like. The current lane may include a number indicating which of the detected lanes (e.g., left-to-right or right-to-left with respect to the vehicle's direction of travel) the vehicle is in.For example, if the lane count component 706 detects six lanes, the current lane component 708 may output a "1" to indicate that the vehicle is in the rightmost lane, a "2" to indicate that the vehicle is in the center lane, or a "3" to indicate that the vehicle is in the leftmost lane. This configuration is provided only as an example, and other types of outputs may be provided in other embodiments within the scope of the present disclosure.

[0043] According to one embodiment, the current lane component 708 uses a deep neural network that has been qualified to determine the current lane. For example, the neural network of the current lane component 708 may receive an image, a LIDAR frame, and / or other perception data along with the number of lanes output by the lane count component 706. Based on this input, the neural network may output a number or other indication regarding the lane the vehicle is likely in. The current lane indication may indicate a location of the vehicle in the lane at a time the perception data was acquired.

[0044] The slope component 710 is configured to determine a slope of a roadway. In one embodiment, the slope component 710 may perform a subdivision of the road plane based on data from a LIDAR or radar system. Based on the subdivision of the road plane, the slope component 710 may determine a slope of the roadway in a direction of travel (i.e., a vertical direction) or in a direction perpendicular to the direction of travel (i.e., a horizontal direction). For example, the slope of the roadway along the direction of travel may be useful in determining an amount of braking or acceleration that may be necessary to perform a particular maneuver or achieve a particular speed. The three-dimensional data acquired from a LIDAR or radar system may enable an accurate calculation of the slope of the roadway.

[0045] The maneuver component 712 is configured to determine a travel path or maneuver to be performed by a vehicle or driving system. For example, the maneuver component 712 may determine a driving maneuver to follow a path or avoid a collision. The maneuver component 712 may determine a maneuver or path based on a number of lanes, a current lane, and / or a road grade. For example, the maneuver may include maneuvering to stay in a roadway, stay within lanes of a roadway, or stay within lanes that travel in a same direction as a current lane of the vehicle. The maneuver component 712 may determine maneuvers that include a trajectory or collision avoidance option based on the number of lanes.

[0046] The reporting component 714 is configured to report lane or roadway information to an automated driving system or driver assistance system. For example, the reporting component 714 may provide an indication of a number of lanes, a current lane, and / or a road gradient. The reporting component 714 may provide any data acquired or determined by the perception data component 702, the merging component 704, the lane count component 706, the current lane component 708, the gradient component 710, and / or the maneuver component 712. In one embodiment, the reporting component 714 provides the automated driving / assistance system 102 with Fig. 1 Information about a current lane, number of lanes, gradient, and / or specific maneuvers in real time so that driving decisions and maneuvers can be performed in real time. For example, the reporting component 714 can provide an indication of the number of lanes to an automated driving system or driver assistance system. Additionally, the reporting component 714 can provide reports or data to a driving history component 128 or for storage in a local or remote driving history. For example, the reporting component 714 or the driving history component 128 can upload an indication of the number of lanes for a specific location to a remote storage location.

[0047] Fig. 8 is a schematic flow diagram illustrating a method 800 for determining lane information. The method 800 may be performed by a lane component, an automated driving / assistance system, or vehicle control system, such as the lane component 104, the automated driving / assistance system 102, or the vehicle control system 100 of the Fig. 1, be carried out.

[0048] The method 800 begins, and a perception data component 702 receives, at 802, perception data from at least two sensors. The at least two sensors include a rear-facing camera of a vehicle. For example, the perception data may include information from a rear-facing camera with data from one or more of a radar system, a LIDAR system, an ultrasonic sensing system, an infrared sensing system, or the like. The lane count component 706 determines, at 804, based on the perception data, a number of lanes on a roadway within a field of view captured by the perception data. The lane count component 706 may include a deep neural network that receives perception data and provides an indication of a number of lanes visible / shown in the perception data.In one embodiment, lane count component 706 may determine the number of lanes based on aggregated data from a plurality of sensors or sensor systems. A reporting component 714 provides an indication of the number of lanes to an automated driving system or driver assistance system at 806. For example, the number of lanes may be collected to determine a path or maneuver, or for storage in a remote or local driving history.

[0049] Referring to Fig. 9 illustrates a block diagram of an example computing device 900. Computing device 900 may be used to perform various methods, such as those discussed herein. Computing device 900 may serve as lane component 104, automated driving / assistance system 102, a server, or any other computing device. Computing device 900 may perform various monitoring functions described herein and may execute one or more application programs, such as the application programs or functionality described herein. Computing device 900 may be any of a variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a handheld computer, tablet computers, and the like.

[0050] Computing device 900 includes one or more processors 902, one or more memory devices 904, one or more interfaces 906, one or more mass storage devices 908, one or more input / output (I / O) devices 910, and a display device 930, all coupled to a bus 912. Processor(s) 902 include one or more processors or controllers that execute instructions stored on memory devices 904 and / or mass storage devices 908. Processor(s) 902 may also include various types of computer-readable media, such as cache memory.

[0051] The storage device(s) 904 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 914) and / or non-volatile memory (e.g., read-only memory (ROM) 916). The storage device(s) 904 may also include rewritable ROM, such as flash memory.

[0052] The mass storage device(s) 908 include / include various computer-readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state storage (e.g., flash memory), and so forth. As in Fig. As shown in Figure 9, a particular mass storage device is a hard disk drive 924. Various drives may be included in the mass storage device(s) 908 to enable reading and / or writing to the various computer-readable media. The mass storage device(s) 908 may include removable media 926 and / or non-removable media.

[0053] The I / O device(s) 910 include various devices that allow data and / or other information to be input to or retrieved from the computing device 900. The example I / O device(s) 910 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, and the like.

[0054] Display device 930 includes any type of device capable of displaying information to one or more users of computing device 900. Examples of display device 930 include a monitor, display terminal, video projection device, and the like.

[0055] The interface(s) 906 include(s) various interfaces that enable the computing device 900 to interact with other systems, devices, or computing environments. The example interface(s) 906 include(s) any number of different network interfaces 920, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interface(s) include(s) a user interface 918 and a peripheral device interface 922. The interface(s) 906 may also include one or more user interface elements 918.The interface(s) 906 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mice, touchpad, or any suitable user interface known or later discovered by those skilled in the art), keyboards, and the like.

[0056] Bus 912 enables processor(s) 902, memory device(s) 904, interface(s) 906, mass storage device(s) 908, and I / O device(s) 910 to communicate with each other and with other devices or components coupled to bus 912. Bus 912 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE bus, USB bus, and so on.

[0057] For illustrative purposes, programs and other executable program components are depicted herein as discrete blocks, although it is understood that such programs and components may reside in different memory components of computing device 900 at various times and may be executed by processor(s) 902. Alternatively, the systems and methods described herein may be implemented in hardware or a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) may be programmed to execute one or more of the systems and methods described herein. Examples

[0058] The following examples relate to further embodiments.

[0059] Example 1 is a method including receiving perception data from at least two sensors, wherein the at least two sensors include a rear-facing camera of a vehicle. The method includes determining a number of lanes on a roadway within a field of view captured by the perception data based on the perception data using a neural network. The method further includes providing an indication of the number of lanes to an automated driving system or driver assistance system.

[0060] In Example 2, Example 1 further includes determining a current lane in which the vehicle is located using a neural network, wherein the current lane corresponds to a location of the vehicle in the lane at a time when the perception data was acquired.

[0061] In Example 3, the at least two sensors in each of Examples 1-2 include the rear-facing camera and one or more of a LIDAR system, a radar system, an ultrasonic sensing system, or an infrared camera system.

[0062] In Example 4, the method in each of Examples 1-3 further includes determining a road gradient in one or more of a vertical direction or a horizontal direction based on the perception data.

[0063] In Example 5, the method in each of Examples 1-4 further includes determining one or more of a trajectory or collision avoidance options based on the number of lanes.

[0064] In Example 6, the method in each of Examples 1-5 further includes storing the indication of the number of lanes in a driving history for later access.

[0065] In Example 7, the method in each of Examples 1-6 further includes uploading the indication of the number of lanes to a remote storage location.

[0066] In Example 8, the method in Example 7 further includes: determining a current position of the vehicle; retrieving driving history data from the remote storage location, the driving history indicating a number of lanes for the current position; and sending the number of lanes for the current position to one or more vehicle controllers of an automated driving system or driver assistance system.

[0067] Example 9 is a system including a perception data component, a lane component, and a reporting component. The perception data component is configured to receive perception data from at least two sensors, wherein the at least two sensors include a rear-facing camera of a vehicle. The lane component is configured to determine, using a neural network, a number of lanes on a roadway within a field of view captured by the perception data. The reporting component is configured to provide an indication of the number of lanes to an automated driving system or driver assistance system.

[0068] In Example 10, the system of Example 9 further includes a current lane component configured to determine a current lane in which the vehicle is located using a neural network, wherein the current lane corresponds to a location of the vehicle in the lane at a time when the perception data was acquired.

[0069] In Example 11, the system in each of Examples 9-10 further includes a merging component configured to generate merged sensor data based on the perception data from the at least two sensors, wherein determining the number of lanes comprises determining based on the merged sensor data.

[0070] In Example 12, the at least two sensors in each of Examples 9-11 include the rear-facing camera and one or more of a LIDAR system, a radar system, an ultrasonic sensing system, or an infrared camera system.

[0071] In Example 13, the system in each of Examples 9-12 further includes a slope component configured to determine a road slope in one or more of a vertical direction or a horizontal direction based on the perception data.

[0072] In Example 14, the system in each of Examples 9-13 further includes a maneuver component configured to determine one or more of a trajectory or collision avoidance options based on the number of lanes.

[0073] In Example 15, the report component in each of Examples 9-14 is further configured to upload the lane count information to a remote storage location.

[0074] In Example 16, the system of Example 15 further includes a position component and a driving history component. The position component is configured to determine a current position of the vehicle. The driving history component is configured to: retrieve driving history data from the remote storage location, wherein the driving history indicates one or more of a number of lanes or a gradient for the current position; and send the number of lanes or the gradient for the current position to one or more vehicle controllers of an automated driving system or driver assistance system.

[0075] Example 17 is a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to receive perception data from at least two sensors, wherein the at least two sensors include a rear-facing camera of a vehicle. The instructions cause the processors to determine a number of lanes on a roadway within a field of view of the perception data based on the perception data using a neural network. The instructions cause the processors to provide an indication of the number of lanes to an automated driving system or driver assistance system.

[0076] In Example 18, the computer-readable storage medium of Example 17 further stores instructions that cause the processors to determine a current lane in which the vehicle is located using a neural network, wherein the current lane corresponds to a location of the vehicle in the lane at a time when the perception data was acquired.

[0077] In Example 19, the computer-readable storage medium of each of Examples 17-18 further stores instructions that cause the processors to determine one or more of a trajectory or collision avoidance options based on the number of lanes.

[0078] In Example 20, the computer-readable storage medium of any of Examples 17-19 further stores instructions that cause the processors to: determine a current position of the vehicle; retrieve driving history data from the remote storage location, the driving history indicating a number of lanes for the current position; and send the number of lanes for the current position to one or more vehicle controllers of an automated driving system or driver assistance system.

[0079] Example 21 is a system or apparatus that includes means for implementing a method, system, or apparatus in any of Examples 1-20.

[0080] In the foregoing disclosure, reference has been made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific implementations in which the disclosure may be practiced. It should be understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure. References in the specification to "one embodiment," "an embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes that particular feature, structure, or characteristic. Furthermore, such terms do not necessarily refer to the same embodiment.When a feature, structure, or characteristic is described in connection with one embodiment, it is believed that it is within the knowledge of one skilled in the art to implement the feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0081] Implementations of the systems, apparatus, and methods disclosed herein may include or utilize a special-purpose or general-purpose computer including computer hardware, such as one or more processors and system memory, as discussed herein. Implementations within the scope of the present disclosure may also include disks and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or special-purpose computer system. Computer-readable media storing computer-executable instructions is computer storage media (device). Computer-readable media carrying computer-executable instructions is transmission media.Therefore, by way of example and not limitation, implementations of the disclosure may include at least two distinctly different types of computer-readable media: computer storage media (devices) and transmission media.

[0082] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives (“SSDs”) (e.g., based on RAM), flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code means in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0083] An implementation of the devices, systems, and methods disclosed herein may communicate over a computer network. A "network" is defined as one or more data connections that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. Accordingly, when information is transmitted or provided to a computer over a network or other communications connection (either wired, wireless, or a combination of wired and wireless), the computer considers the connection to be a transmission medium. Transmission media may include a network and / or data connections that may be used to transmit desired program code means in the form of computer-executable instructions or data structures, and that may be accessed by a general-purpose or special-purpose computer.Combinations of the above also fall within the scope of computer-readable media.

[0084] Computer-executable instructions include, for example, instructions and data that, when executed on a processor, cause a general-purpose computer, special-purpose computer, or special-purpose processing apparatus to perform a particular function or set of functions. The computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the features or acts described above. Rather, the described features and acts are disclosed as exemplary forms of implementing the claims.

[0085] Those skilled in the art will understand that the disclosure may be practiced in network computing environments having many types of computer system configurations, including dashboard vehicle computers, PCs, desktop computers, laptops, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cellular phones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. This disclosure may also be practiced in distributed systems environments in which both local computer systems and remote computer systems connected by a network (either by wired data links, wireless data links, or a combination of wired and wireless data links) perform tasks.In a distributed system environment, program modules can be located on both local and remote storage devices.

[0086] Additionally, where applicable, the functions described herein may be implemented in one or more of the following: hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and methods described herein. Certain terms are used in the specification and claims with reference to specific system components. As one skilled in the art will understand, components may be referred to using various names. This document is not intended to distinguish between components that differ in name but not in function.

[0087] It should be noted that the sensor embodiments discussed above may include computer hardware, software, firmware, or any combination thereof to perform at least some of their functions. For example, a sensor may include computer code configured to execute on one or more processors and may include hardware logic / electrical circuitry controlled by the computer code. These example devices are provided herein for illustrative purposes and are not intended to be limiting. Embodiments of the present disclosure may be implemented in other types of devices, as would be known to one of ordinary skill in the art(s).

[0088] At least some embodiments of the disclosure relate to computer program products comprising such logic (e.g., in the form of software) stored on any computer-usable medium. Such software, when executed in one or more computing devices, causes a device to operate as described herein.

[0089] While various embodiments of the present disclosure have been described above, it is to be understood that these have been presented only by way of example and not by way of limitation. It will be apparent to those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the disclosure. Therefore, the breadth and scope of the present disclosure should not be limited by any of the foregoing embodiments, but should be defined only in accordance with the following claims and their equivalents. The foregoing description has been presented for purposes of illustration and description. It is not intended to be exhaustive and is not intended to limit the disclosure to the specific form disclosed. Many modifications and variations are possible in light of the above teachings.It is further noted that any or all of the foregoing alternative implementations may be used in any desired combination to form additional hybrid implementations of the disclosure.

[0090] Furthermore, although specific implementations of the disclosure have been described and illustrated, the disclosure is not intended to be limited to the specific forms or arrangements of parts thus described and illustrated. The scope of the disclosure is to be defined by the claims appended hereto, any future claims sought herein, and in various applications and their equivalents.

Claims

[1] Method comprising: Receiving image data of a roadway from a camera of a vehicle; Receiving perception data measuring the reflectance of the road surface from a source comprising a Light Detection and Ranging (LIDAR) system; Determine whether there are lane markings on the roadway; in response to determining that there are no lane markings, estimating a number of lanes on the roadway based on a position of shoulders of the roadway as determined only from the perception data; in response to determining that there are lane markings, identifying which lane on the roadway the vehicle is in by processing merged data using a neural network, the merged data comprising the image data and the reflectance of the roadway as determined by the perception data; and Providing an indication of the number of lanes to an automated driving system or driver assistance system, wherein the automated driving system or driver assistance system actuates one or more vehicle controls based on the indication of the number of lanes. [2] The method of claim 1, further comprising one or more of the following: Determining a road gradient in one or more of a vertical direction or a horizontal direction based on the perception data; Determining one or more trajectory or collision avoidance options based on the number of lanes; or Saving the number of lanes in a driving history for later access. [3] The method of claim 1, wherein further perception data from a radar system, an ultrasonic detection system or an infrared camera system is received. [4] The method of claim 1, further comprising uploading the indication of the number of lanes to a remote storage location. [5] The method of claim 4, further comprising: Determining a current position of the vehicle; Retrieving the driving history data from the remote storage location, wherein the driving history specifies a number of lanes for the current position; and Sending the number of lanes for the current position to one or more vehicle controllers of an automated driving system or driver assistance system. [6] System comprising: a rear-facing camera of a vehicle; a source of perception data comprising a Light Detection and Ranging (LIDAR) system of the vehicle; and a processor configurable to execute instructions stored in non-transitory computer-readable storage media, the instructions comprising: Receiving image data of a roadway from the vehicle's rear-facing camera as the vehicle moves forward on the roadway; Receiving perception data measuring the reflectance of the road surface from the LIDAR system; Determine whether there are lane markings on the roadway; in response to determining that there are no lane markings, estimating a number of lanes on the roadway based on a position of shoulders of the roadway as determined only from the perception data; in response to determining that there are lane markings, identifying which lane on the roadway the vehicle is in by processing merged data using a neural network, the merged data comprising the image data and the reflectance of the roadway as determined by the perception data; and Providing an indication of the number of lanes to an automated driving system or driver assistance system, wherein the automated driving system or driver assistance system actuates one or more vehicle controls based on the indication of the number of lanes. [7] The system of claim 6, further comprising one or more of the following: a current lane component configured to determine a current lane in which the vehicle is located using a neural network, the current lane corresponding to a location of the vehicle in the lane at a time when the perception data was acquired; a merging component configured to generate merged sensor data based on the perception data from the at least two sensors, wherein determining the number of lanes comprises determining based on the merged sensor data; or a slope component configured to determine a road slope in one or more of a vertical direction or a horizontal direction based on the perception data. [8] The system of claim 6, further comprising a radar system, an ultrasonic detection system, or an infrared camera system. [9] The system of claim 6, further comprising a maneuver component configured to determine one or more of a trajectory or collision avoidance options based on the number of lanes. [10] The system of claim 6, wherein the reporting component is further configured to upload the indication of the number of lanes to a remote storage location. [11] The system of claim 10, further comprising: a position component configured to determine a current position of the vehicle; and a journey component configured to: retrieve driving history data from the remote storage location, the driving history specifying one or more of a number of lanes or a gradient for the current position; and to send the number of lanes or the gradient for the current position to one or more vehicle controllers of an automated driving system or driver assistance system. [12] Computer-readable storage medium that stores instructions that, when executed by one or more processors, cause the processors to: To receive image data of a roadway from a rear-facing camera of a vehicle while the vehicle is moving forward on the roadway; to receive perception data measuring the reflectance of the road surface from a source comprising a Light Detection and Ranging (LIDAR) system; to determine whether there are lane markings on the roadway; in response to determining that there are no lane markings, estimate a number of lanes on the roadway based on a position of shoulders of the roadway determined only from the perception data; in response to determining that there are lane markings, identify which lane on the roadway the vehicle is in by processing merged data using a neural network, the merged data comprising the image data and the reflectance of the roadway determined by the perception data; and to provide an indication of the number of lanes to an automated driving system or driver assistance system, wherein the automated driving system or driver assistance system operates one or more vehicle controls based on the indication of the number of lanes. [13] The computer-readable storage medium of claim 12, further storing instructions that cause the processors to determine one or more of a trajectory or collision avoidance options based on the number of lanes. [14] The computer-readable storage medium of claim 12, further storing instructions that cause the processors to: to determine the current position of the vehicle; retrieve driving history data from the remote storage location, the driving history specifying a number of lanes for the current position; and to send the number of lanes for the current position to one or more vehicle controllers of an automated driving system or driver assistance system.

Images