Virtual lane method and system

Abstract

A system and method for controlling a vehicle are provided. In one embodiment, a system includes: a sensor system configured to generate sensor data sensed from an environment of the vehicle; and a control module configured to predict, by a processor, a parked vehicle within the environment scene, identify an outer edge associated with the parked vehicle, generate a virtual lane line based on the outer edge associated with the parked vehicle, and generate signal data based on the virtual lane line to at least one of: display the virtual lane line within the scene and control the vehicle.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to perception systems for vehicles, and more specifically, to methods and systems for providing virtual lane information in images of a vehicle's environment. BACKGROUND

[0002] An autonomous vehicle is a vehicle that is capable of sensing its environment and navigating with little or no user input. This is achieved through the use of sensing devices such as radio radar, laser radar, image sensors, etc. The autonomous vehicle further uses information from global positioning system (GPS) technology, navigation systems, vehicle-to-vehicle communication, vehicle-to-infrastructure technology, and / or drive-by-wire systems to navigate the vehicle.

[0003] While autonomous vehicles offer many potential advantages over traditional vehicles, there are situations in which the operation of autonomous vehicles can need to be improved. For example, challenges are encountered when navigating residential roads or other areas with street parking, construction zones, and parking lots. Generally, these areas do not have marked lane markings, and the autonomous vehicle determines the lane based on the environment.

[0004] Accordingly, it is desirable to provide methods and systems for providing virtual lane lines. Other desirable features and characteristics of the embodiments described herein will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background. SUMMARY

[0005] In various embodiments, systems and methods for controlling a vehicle are provided. In one embodiment, a system includes a sensor system configured to generate sensor data sensed from an environment of a vehicle, and a control module configured to predict, by a processor, a parked vehicle within the environment scene, identify an outer edge associated with the parked vehicle, generate a virtual lane line based on the outer edge associated with the parked vehicle, and generate signal data based on the virtual lane line to at least one of display the virtual lane line within the scene and control the vehicle.

[0006] In various embodiments, the sensor system includes one or more cameras of the vehicle.

[0007] In various embodiments, the sensor system includes a satellite system.

[0008] In various embodiments, the control module is configured to identify the parked vehicle with a bounding box, and wherein the control module identifies the outer edge based on the bounding box.

[0009] In various embodiments, the control module is configured to generate the virtual lane line by fitting a polynomial to the outer edges of the parked vehicle. In various embodiments, the control module is configured to generate the virtual lane line by using data points from a virtual lane centerline of the scene or from a left lane line identified in the scene.

[0010] In various embodiments, the control module is configured to identify one or more available parking spots based on a comparison of the virtual lane line to map data. In various embodiments, the control module modifies the display signal to modify the display of the virtual lane line based on the one or more available parking spots.

[0011] In various embodiments, the control module is configured to adjust a prediction of things within the scene based on the virtual lane line. In various embodiments, the control module is configured to modify the map data to include the virtual lane line as a feature of the map.

[0012] A method for controlling a vehicle, comprising: receiving, by a processor, sensor data sensed from an environment of the vehicle; processing, by the processor, the sensor data to predict a parked vehicle within an environmental scene; identifying, by the processor, outer edges associated with the parked vehicle; generating, by the processor, a virtual lane line based on the outer edges associated with the parked vehicle; and generating, by the processor, signal data based on the virtual lane line to at least one of: display the virtual lane line within the scene and control the vehicle.

[0013] In various embodiments, the sensor data is received from one or more cameras of the vehicle.

[0014] In various embodiments, the sensor data is received from a satellite system.

[0015] In various embodiments, the method further comprises identifying the parked vehicle with a bounding box, and wherein identifying the outer edges is based on the bounding box.

[0016] In various embodiments, the method further comprises generating the virtual lane line by fitting a polynomial to the outer edges of the parked vehicle.

[0017] In various embodiments, the method further comprises generating the virtual lane line by using data points from a virtual lane centerline of the scene or from a left lane line identified in the scene.

[0018] In various embodiments, the method further comprises identifying one or more available parking spots based on a comparison of the virtual lane line to map data.

[0019] In various embodiments, the method further comprises modifying the display of the virtual lane line based on the one or more available parking spots modifying the display signal.

[0020] In various embodiments, the method further includes adjusting predictions of things within the scene based on the virtual lane lines.

[0021] In various embodiments, the method further includes modifying map data to include the virtual lane lines as features of the map. BRIEF DESCRIPTION OF DRAWINGS

[0022] Exemplary embodiments will be described below with reference to the following drawings, in which like elements are referred to with like reference numerals, and in which:

[0023] Figure 1 is an illustration of a top perspective view schematic of a vehicle with a virtual lane system in accordance with various embodiments;

[0024] Figure 2 is a functional block diagram illustrating a virtual lane system in accordance with various embodiments;

[0025] Figures 3A-3E is an exemplary interface of a virtual lane system in accordance with various embodiments;

[0026] Figure 4 is a data flow diagram illustrating a control module of a virtual lane system in accordance with various embodiments; and

[0027] Figure 5 is a flowchart illustrating a method of displaying virtual lane information in accordance with various embodiments. DETAILED DESCRIPTION

[0028] The following detailed description is merely exemplary in nature and is not intended to limit the application and use. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description. It should be appreciated that the detailed description set forth below in connection with the appended drawings is intended as a description of various embodiments and is not intended to limit the scope of the application. Rather, the intent is to cover all alternatives, modifications, equivalents, and variations that fall within the scope of the application. As used herein, the term system or module can refer to any combination of mechanical and electronic hardware, software, firmware, electronic control components, processing logic, and / or processor devices, individually or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or group), memory (shared, dedicated, or group) that stores one or more executable software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0029] For the sake of brevity, conventional components and techniques and other functional aspects of the systems (and the individual operating components of the systems) can not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and / or physical couplings between the various elements. It should be noted that many alternatives or equivalents exist that are capable of being implemented other than as shown.

[0030] Figure 1 is a top view of a vehicle equipped with a virtual lane system, generally shown at 12, in accordance with various embodiments. As will be discussed in greater detail below, the virtual lane system 12 generally uses data from a sensor system 14 of the vehicle 10 and / or satellite / aerial images to predict virtual lane lines on a roadway, such as but not limited to residential and urban roadways, parking areas, and construction zones. In such examples, a display screen 16( Figure 2 ) can be placed anywhere in the vehicle 10 and can display images and / or video that display virtual lane lines on an image or video of the environment of the vehicle 10, for example, as if the lane lines were part of the roadway.

[0031] While the background discussed herein is directed to the vehicle 10 being a passenger car, it should be understood that the teachings herein are compatible with all types of vehicles, including but not limited to sedans, coupes, sport utility vehicles, pick-up trucks, mini-vans, full-size vans, trucks, and buses, as well as any type of tow vehicle, such as a trailer.

[0032] As shown by way of example in Figure 1 , the vehicle 10 generally includes a body 13, front wheels 18, rear wheels 20, a suspension system 21, a steering system 22, and a propulsion system 24. The wheels 18-20 are each rotatably connected to the vehicle 10 near a respective corner of the body 13. The wheels 18-20 are connected to the body 13 by the suspension system 21. The wheels 18 and / or 20 are driven by the propulsion system 24. The wheels 18 are steerable by the steering system 22.

[0033] The body 13 is disposed on or integrated with a chassis (not shown) and substantially encloses components of the vehicle 10. The body 13 is structured to separate a powertrain compartment 28, which includes at least the propulsion system 24, from a passenger compartment 30, which includes seating (not shown) for one or more passengers of the vehicle 10, as well as other features. As used herein, components “underneath” the vehicle 10 are components disposed under the body 13, such as but not limited to the wheels 18 and 20 (including their respective tires) and the suspension system 21.

[0034] The vehicle 10 also includes a sensor system 14 and an operator selection device 15. The sensor system 14 includes one or more sensing devices that sense observable states of components of the vehicle 10 and / or that sense observable states of the external environment of the vehicle 10. The sensing devices can include, but are not limited to, radio radar, laser radar, global positioning system, optical camera, thermal camera, ultrasonic sensor, altitude sensor, pressure sensor, steering angle sensor, and / or other sensors. The operator selection device 15 includes one or more user-operable devices that a user can operate to provide input. The input can relate to, for example, activation of a display of virtual reality content and a desired perspective of the content to be displayed. The operator selection device 15 can include knobs, switches, touchscreens, voice recognition modules, etc.

[0035] As Figure 2 shown in greater detail in Figure 1 , the virtual lane system 12 includes a display screen 32 communicatively coupled to a control module 34. The control module 34 is communicatively coupled to the sensor system 14 and the operator selection device 15.

[0036] The display screen 32 can be disposed within the passenger cabin 30 in a position that can be viewed by a passenger of the vehicle 10. For example, the display screen 32 can be integrated with an infotainment system (not shown) or instrument panel (not shown) of the vehicle 10. The display screen 32 displays content such that a viewer experiences a partial virtual reality. For example, as shown in FIG. 3, in various embodiments, the content 42 includes a depiction of a scene 48 in which the vehicle 10 is traveling, including the ground, curb, road markings, buildings, other vehicles, pedestrians, etc., as well as virtual lane lines determined by the virtual lane system and method described herein.

[0037] For example, as Figures 3A-3E shown, sensor data obtained from one or more sensors of the vehicle 10 produces an environmental scene. Thereafter, virtual lane lines are determined and overlaid on the scene to illustrate virtual lanes. In various embodiments, as Figure 3A shown, the virtual lane lines identify parking lanes of a parking lot.

[0038] In various embodiments, as Figure 3B shown, the virtual lane lines identify lanes through an intersection. In various embodiments, as Figure 3C shown, the virtual lane lines identify lanes around a parking area. In various embodiments, as Figure 3D shown, the virtual lane lines identify available parking along a parking lane. In various embodiments, as Figure 3E shown, the virtual lane lines identify lanes around a parallel parked car.

[0039] In various embodiments, virtual lane lines can be displayed as solid or dashed lines with specific colors, highlights, or patterns that identify lane types or characteristics.

[0040] Referring again to Figure 2 , control module 34 can be dedicated to display screen 32, can control display screen 32 and other features of vehicle 10 (e.g., a body control module, an instrument control module, or other feature control modules), and / or can be implemented as a combination of control modules that control display screen 32 and other features of vehicle 10. For illustrative purposes, control module 34 will be discussed and illustrated as a single control module associated with display screen 32. Control module 34 can directly control display screen 32 and / or communicate data to display screen 32, whereby scenes and virtual lane content can be displayed.

[0041] Control module 34 includes at least memory 36 and processor 38. As will be discussed in greater detail below, control module 34 includes instructions that, when processed by processor 38, control content to be displayed on display screen 32 based on sensor data received from sensor system 14 and user input received from operator selection device 15.

[0042] Referring now to Figure 4 and with continued reference to Figure 1 -3, a data flow diagram illustrates various embodiments of control module 34 in greater detail. Various embodiments of control module 34 according to the present disclosure can include any number of sub-modules. It can be appreciated that Figure 4 the sub-modules illustrated in FIG. 3 can be combined and / or further partitioned to similarly generate virtual reality content for viewing by an operator. Inputs to control module 34 can be received from sensor system 14, from operator selection device 15, from other control modules (not shown) of vehicle 10, and / or determined by other sub-modules (not shown) of control module 34. In various embodiments, control module 34 includes parked vehicle determination module 50, feature extraction module 52, map feature determination module 54, and display / control module 56.

[0043] In various embodiments, parked vehicle determination module 50 receives image data 58. Image data 58 can be generated by one or more sensors of, for example, sensor system 14 and / or by, for example, a satellite system associated with the vehicle. Parked vehicle determination module 50 processes image data 58 to determine parked vehicles in the scene.

[0044] For example, a trained deep neural network can be used to predict that a vehicle is parked within a scene. It can be appreciated that various methods can be used to identify vehicles and determine the state of the vehicle to be parked. These methods can depend on whether the image data 58 includes a ground view or an aerial view. Embodiments of the present disclosure are not limited to any one example. In various embodiments, the parked vehicle determination module 50 labels each vehicle identified as parked with a bounding box around the identified vehicle within the scene. The parked vehicle determination module 50 provides parked vehicle data 60 based on the bounding boxes.

[0045] In various embodiments, the feature extraction module 52 receives parked vehicle data 60 identifying parked vehicles in a scene. The feature extraction module processes the parked vehicle data 60 to define an external line of the identified parked vehicles. For example, the feature extraction module 52 identifies the outer edge of each bounding box and fits a smooth polynomial to the outer edge to generate a virtual line representing the outer edge of the parking lane.

[0046] In various embodiments, when drawing the polynomial fit to ensure smoothness with respect to the road geometry, the feature extraction module 52 further smooths the virtual lane line by using lane lines from the scene or a virtual centerline in the scene as additional data points. It can be appreciated that other methods of smoothing the line can be implemented in various embodiments. The feature extraction module 52 generates virtual lane line data 62 based on the smoothed virtual line.

[0047] In various embodiments, the map feature determination module 54 receives parked vehicle data 60 and virtual lane line data 62. The map feature determination module 54 compares the virtual lane line data 62 and the parked vehicle data 60 to map data 64 stored in a map data database 65. The map data 64 identifies a navigable map of the environment.

[0048] In various embodiments, based on the comparison, the map feature determination module 54 updates the map data store 65 by providing map feature data 66 to include the virtual lane line as a map feature. In various embodiments, based on the comparison, the map feature determination module 54 identifies a parking spot adjacent to the virtual lane line that is available for parking. The map feature determination module 54 modifies the virtual lane line data 62 (e.g., color, shading, highlighting, etc.) to identify the location along the virtual lane line that has available parking space and provides map feature data 66 based thereon.

[0049] In various embodiments, the display / control module 56 receives virtual lane data 62 and map feature data 66 as inputs. The display / control module 56 generates signal data 68 to display virtual lane lines and / or map features including available parking spots related to the scene provided by the image data 58 to, for example, a passenger of the vehicle 10 via the display screen 32. The display / control module 56 produces signal data 68 to control one or more features of the vehicle 10. For example, the virtual lane line data 62 can be used to append lane polynomials to other lanes predicted by the perception model. In another example, the virtual lane line data 62 can be used to improve detection of the actions of vehicles / pedestrians around the virtual parking lane, and / or to control the path or route of the vehicle 10. It will be appreciated that the virtual lane line data 62 and / or the map feature data 66 can be used to control the vehicle 10 in various ways, and are not limited to the present example.

[0050] Referring now to Figure 5 and with continued reference to Figures 1-4 a flowchart illustrates a method 200 that can be performed by the virtual lane system 12 in accordance with various embodiments. It will be appreciated in accordance with the present disclosure that the order of operations within the method 200 is not limited to the order of execution as shown in Figure 5 but can be performed in one or more different orders in accordance with the present disclosure. It will also be appreciated that the method 200 can be scheduled to run at predetermined time intervals during operation of the vehicle 10 and / or can be scheduled to run based on predetermined events.

[0051] In one example, the method can begin at 205. Image data is received at 210, and parked vehicles within the scene are identified and labeled with bounding boxes at 220. For example, as discussed above at 230, the bounding boxes within the scene are evaluated to extract exterior lines. Then, at 240, the exterior lines are fitted with a smooth polynomial to generate lane lines. Optionally, another identified lane line (e.g., a virtual center lane line, a left lane line, etc.) can be used as an additional data point to augment the lane line when fitting the polynomial at 250. Further, other lane lines can be used with the virtual lane lines to construct a nominal path for the vehicle that ensures the vehicle does not encroach into other lanes when adjusting its path to accommodate adjacent parked cars or other obstacles separated by the virtual lane lines. Thereafter, at 260, the virtual lane lines can be evaluated with map data to determine available parking spots. The virtual lane lines and available parking spots are displayed to a user at 270, and used by other systems to update map information, predict the actions of other things within the scene, and / or control the operation of the vehicle at 280. Thereafter, the method can end at 290.

[0052] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or embodiments are only examples, and are not intended to limit the scope, applicability or configuration of the disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing an exemplary embodiment or embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope of the disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A system for controlling a vehicle, comprising: a sensor system configured to generate sensor data sensed from an environment of the vehicle; and a control module configured to predict, by a processor, a plurality of parked vehicles within an environmental scene, identify each of the plurality of parked vehicles within a bounding box, fit a polynomial to an outer edge of each of the plurality of parked vehicles, generate virtual lane lines based on the fitted polynomials from the plurality of parked vehicles, and generate signal data based on the virtual lane lines to at least one of: display the virtual lane lines within the scene and control the vehicle; wherein a vehicle nominal path is constructed using other lane lines together with the virtual lane lines, the vehicle nominal path ensuring that the vehicle does not encroach into other lanes when adjusting its path to accommodate adjacent parked cars or other obstacles separated by the virtual lane lines.

2. The system of claim 1, wherein the sensor system comprises one or more cameras of the vehicle.

3. The system of claim 1, wherein the sensor system comprises a satellite system.

4. The system of claim 1, wherein, the control module is configured to generate the virtual lane lines by using data points from a virtual lane centerline of the scene or from a left lane line identified in the scene.

5. The system of claim 1, wherein the control module is configured to identify one or more available parking spots based on a comparison of the plurality of parked vehicles and the virtual lane lines to map data.

6. The system of claim 5, wherein, the control module modifies a display signal at a location of the one or more available parking spots to modify a display of the virtual lane lines.

7. The system of claim 1, wherein, the control module is configured to at least one of: adapt a prediction of things within the scene based on the virtual lane lines and modify map data to include the virtual lane lines as a feature of the map.

8. A method for controlling a vehicle, comprising: receiving, by a processor, sensor data sensed from an environment of the vehicle; processing, by the processor, the sensor data to predict a plurality of parked vehicles within an environmental scene; identifying, by the processor, each of the plurality of parked vehicles within a bounding box; fitting, by the processor, a polynomial to an outer edge of each of the plurality of parked vehicles generating, by a processor, virtual lane lines based on the fitted polynomials from the plurality of parked vehicles; and generating, by a processor, signal data based on the virtual lane lines to at least one of: display the virtual lane lines within the scene and control the vehicle; wherein a vehicle nominal path is constructed using other lane lines together with the virtual lane lines, the vehicle nominal path ensuring that the vehicle does not encroach into other lanes when adjusting its path to accommodate adjacent parked cars or other obstacles separated by the virtual lane lines.

Images