Method, server, autonomous vehicle and medium for autonomous vehicle navigation
Through the sparse map system and sparse road model, combined with camera and sensor data, the problem of large amount of map data in autonomous vehicle navigation is solved, and efficient navigation and navigation accuracy under low data density is achieved.
Patent Information
- Application Number
- CN202010056873.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2016-01-11
- Filing Date
- 2016-02-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2036-02-10
AI Technical Summary
In autonomous vehicle navigation, traditional map data is large and difficult to update, resulting in inefficient navigation and processing massive information has caused computing challenges.
The sparse map system is adopted to analyze image data through cameras and combine GPS and sensor data to build a sparse road model, provide navigation response, and use polynomial representation and predetermined landmarks to realize a navigation map with low data density and support autonomous vehicle navigation.
It realizes providing sufficient navigation information under low data density, reduces the difficulty of storing and updating maps, and improves the efficiency and accuracy of autonomous vehicle navigation.
Smart Images

Figure CN111380545B_ABST
Abstract
Description
[0001] This application is a divisional application of an invention patent application with an application date of February 10, 2016, application number 201680021179.6 (international application number PCT / US2016 / 017411), and invention name “Sparse Map for Autonomous Vehicle Navigation”.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 114,091, filed February 10, 2015; U.S. Provisional Patent Application No. 62 / 164,055, filed May 20, 2015; U.S. Provisional Patent Application No. 62 / 170,728, filed June 4, 2015; U.S. Provisional Patent Application No. 62 / 181,784, filed June 19, 2015; U.S. Provisional Patent Application No. 62 / 192,576, filed July 15, 2015; U.S. Provisional Patent Application No. 62 / 215,764, filed September 9, 2015; U.S. Provisional Patent Application No. 62 / 219,733, filed September 17, 2015; U.S. Provisional Patent Application No. 62 / 261,578, filed December 1, 2015; U.S. Provisional Patent Application No. 62 / 261598, filed December 1, 2015; U.S. Provisional Patent Application No. 62 / 267643, filed December 15, 2015; U.S. Provisional Patent Application No. 62 / 269818, filed December 28, 2015; U.S. Provisional Patent Application No. 62 / 270408, filed December 21, 2015; U.S. Provisional Patent Application No. 62 / 270418, filed December 21, 2015; U.S. Provisional Patent Application No. 62 / 270431, filed December 21, 2015; U.S. Provisional Patent Application No. 62 / 271,103, filed on January 22, 2016; U.S. Provisional Patent Application No. 62 / 274,883, filed on January 5, 2016; U.S. Provisional Patent Application No. 62 / 274,968, filed on January 5, 2016; U.S. Provisional Patent Application No. 62 / 275,007, filed on January 5, 2016; U.S. Provisional Patent Application No. 62 / 275,046, filed on January 5, 2016; and U.S. Provisional Patent Application No. 62 / 277,068, filed on January 11, 2016. All of the foregoing applications are hereby incorporated by reference in their entireties. Technical Field
[0004] The present disclosure generally relates to autonomous vehicle navigation and sparse maps for autonomous vehicle navigation. In addition, the present disclosure relates to systems and methods for constructing, using, and updating sparse maps for autonomous vehicle navigation. Background Art
[0005] As technology continues to advance, the goal of fully autonomous vehicles capable of navigating roads is imminent. Autonomous vehicles may need to consider a wide variety of factors and make appropriate decisions based on those factors to safely and accurately reach their desired destination. For example, autonomous vehicles may need to process and interpret visual information (e.g., information captured from cameras) and may also use information obtained from other sources (e.g., from GPS devices, speed sensors, accelerometers, suspension sensors, etc.). Furthermore, to navigate to their destination, autonomous vehicles may also need to identify their position within a specific road surface (e.g., a specific lane within a multi-lane road), navigate alongside other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, and switch from one road to another at appropriate intersections or junctions. Utilizing and interpreting the vast amount of information collected by an autonomous vehicle as it travels to its destination poses numerous design challenges. The vast amount of data (e.g., captured image data, map data, GPS data, sensor data, etc.) that autonomous vehicles may need to analyze, access, and / or store creates challenges that can effectively limit or even adversely impact autonomous navigation. Moreover, if autonomous vehicles rely on traditional mapping technologies to navigate, the sheer volume of data required to store and update maps poses a daunting challenge. Summary of the Invention
[0006] Embodiments consistent with the present disclosure provide systems and methods for autonomous vehicle navigation. The disclosed embodiments may use cameras to provide autonomous vehicle navigation features. For example, consistent with the disclosed embodiments, the disclosed system may include one, two, or more cameras that monitor the vehicle's environment. The disclosed system may provide a navigation response based on, for example, analysis of images captured by one or more of the cameras. The navigation response may also take into account other data, including, for example, global positioning system (GPS) data, sensor data (from accelerometers, rate sensors, suspension sensors, etc.), and / or other map data.
[0007] In some embodiments, the disclosed systems and methods can use sparse maps for autonomous vehicle navigation. For example, sparse maps can provide sufficient information for navigation without requiring excessive data storage.
[0008] In other embodiments, the disclosed systems and methods can construct road models for autonomous vehicle navigation. For example, the disclosed systems and methods can utilize crowdsourced data, including recommended trajectories, for autonomous vehicle navigation. As another example, the disclosed systems and methods can identify landmarks in the vehicle's environment and refine landmark positioning.
[0009] In yet other embodiments, the disclosed systems and methods can utilize sparse road models for autonomous vehicle navigation. For example, the disclosed systems and methods can provide navigation based on identified landmarks, align the rear of a vehicle for navigation, allow a vehicle to navigate road junctions, allow a vehicle to navigate using partially overlapping maps, allow a vehicle to navigate using sparse maps, navigate based on expected landmark locations, autonomously navigate roads based on road signatures, provide forward navigation based on a rear-facing camera, navigate based on free space determination, navigate in snow, provide autonomous vehicle velocity calibration, determine lane assignments based on identified landmark locations, and utilize super landmarks as navigation aids.
[0010] In still other embodiments, the disclosed systems and methods can provide adaptive autonomous navigation. For example, the disclosed systems and methods can provide adaptive navigation based on user intervention, provide self-aware adaptive navigation, provide an adaptive road model manager, and manage road models based on selective feedback.
[0011] In some embodiments, a non-transitory computer-readable medium may include a sparse map for autonomous vehicle navigation along a road segment. The sparse map may include a polynomial representation of a target trajectory for the autonomous vehicle along the road segment; and a plurality of predetermined landmarks associated with the road segment, wherein the plurality of predetermined landmarks may be spaced at least 50 meters apart, and wherein the sparse map may have a data density of no more than 1 megabyte per kilometer.
[0012] In some embodiments of the non-transitory computer-readable medium, the polynomial representation may be a three-dimensional polynomial representation. The polynomial representation of the target trajectory may be determined based on two or more reconstructed trajectories of a vehicle previously traversed along a road segment. The plurality of predetermined landmarks may include traffic signs represented in the sparse map by no more than 50 bytes of data. The plurality of predetermined landmarks may include directional signs represented in the sparse map by no more than 50 bytes of data. The plurality of predetermined landmarks may include universal signs represented in the sparse map by no more than 100 bytes of data. The plurality of predetermined landmarks may include generally rectangular objects represented in the sparse map by no more than 100 bytes of data. The representation of the generally rectangular objects in the sparse map may include a condensed image signature associated with the generally rectangular objects. The plurality of predetermined landmarks may be represented in the sparse map by parameters including landmark size, distance from a previous landmark, landmark type, and landmark location. The plurality of predetermined landmarks included in the sparse map may be separated by at least 2 kilometers. The plurality of predetermined landmarks included in the sparse map may be separated by at least 1 kilometer. The plurality of predetermined landmarks included in the sparse map may be separated by at least 100 meters. The sparse map may have a data density of no more than 100 kilobytes per kilometer. The sparse map may have a data density of no more than 10 kilobytes per kilometer. The plurality of predetermined landmarks may appear in the sparse map at a frequency sufficient to maintain a longitudinal position determination accuracy within 1 meter.
[0013] In some embodiments, an autonomous vehicle may include a body; and a non-transitory computer-readable medium that may include a sparse map for navigation of the autonomous vehicle along a road segment. The sparse map may include a polynomial representation of a target trajectory for the autonomous vehicle along the road segment; and a plurality of predetermined landmarks associated with the road segment, wherein the plurality of predetermined landmarks are separated by at least 50 meters, and wherein the sparse map may have a data density of no more than 1 megabyte per kilometer. The autonomous vehicle may include a processor configured to execute data included in the sparse map to provide autonomous vehicle navigation along the road segment.
[0014] In some embodiments of the autonomous vehicle, the polynomial representation may be a three-dimensional polynomial representation.The polynomial representation of the target trajectory may be determined based on two or more reconstructed trajectories of prior traversals of the vehicle along the road segment.
[0015] In some embodiments, an autonomous vehicle may include a body; and a processor configured to receive data included in a sparse map and execute the data for navigation of the autonomous vehicle along a road segment. The sparse map may include a polynomial representation of a target trajectory for the autonomous vehicle along the road segment; and a plurality of predetermined landmarks associated with the road segment, wherein the plurality of predetermined landmarks are spaced at least 50 meters apart, and wherein the sparse map may have a data density of no more than 1 megabyte per kilometer.
[0016] In some embodiments, a method for processing vehicle navigation information for autonomous vehicle navigation may include receiving, by a server, navigation information from a plurality of vehicles. The navigation information from the plurality of vehicles may be associated with a common road segment. The method may include storing, by the server, the navigation information associated with the common road segment. The method may include: generating, by the server, at least a portion of an autonomous vehicle road navigation model for the common road segment based on the navigation information from the plurality of vehicles; and distributing, by the server, the autonomous vehicle road navigation model to one or more autonomous vehicles for use in autonomously navigating the one or more autonomous vehicles along the common road segment.
[0017] In some embodiments of the method, navigation information may include a trajectory from each of the plurality of vehicles as each vehicle travels on a common road segment. The trajectory may be determined based on sensed motion of a camera, the motion including three-dimensional translation and three-dimensional rotation. The navigation information may include lane assignments. Generating at least a portion of an autonomous vehicle road navigation model may include clustering vehicle trajectories along the common road segment and determining a target trajectory along the common road segment based on the clustered vehicle trajectories. The autonomous vehicle road navigation model may include a three-dimensional spline corresponding to the target trajectory along the common road segment. The target trajectory may be associated with a single lane of the common road segment. The autonomous vehicle road navigation model may include multiple target trajectories, each associated with a separate lane of the common road segment. Determining the target trajectory along the common road segment based on the clustered vehicle trajectories may include finding a mean or average trajectory based on the clustered vehicle trajectories. The target trajectory may be represented by a three-dimensional spline. The spline may be defined using less than 10 kilobytes per kilometer. The autonomous vehicle road navigation model may include an identification of at least one landmark, the identification including a location of the at least one landmark. The location of the at least one landmark may be determined based on position measurements performed using sensor systems associated with the plurality of vehicles. The position measurements may be averaged to obtain a position of the at least one landmark. The at least one landmark may include at least one of a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost.
[0018] In some embodiments, a navigation system for a vehicle may include the following: at least one processor programmed to receive at least one environmental image associated with the vehicle from a camera; analyze the at least one environmental image to determine navigation information related to the vehicle; and transmit the navigation information from the vehicle to a server. The at least one processor may be programmed to receive an autonomous vehicle road navigation model from the server. The autonomous vehicle road navigation model may include at least one update based on the transmitted navigation information. The at least one processor may be programmed to cause the vehicle to perform at least one navigation maneuver based on the autonomous vehicle road navigation model.
[0019] In some embodiments of the navigation system, the navigation information may include a trajectory from each of the plurality of vehicles as each vehicle travels on a common road segment.
[0020] In some embodiments, a server for processing vehicle navigation information for use in autonomous vehicle navigation may include: a communication unit configured to communicate with a plurality of vehicles; and at least one processor programmed to receive navigation information from the vehicles via the communication unit. The at least one processor may be programmed to: generate at least a portion of an autonomous vehicle road navigation model based on the navigation information; and transmit at least a portion of the autonomous vehicle road navigation model to at least one of the vehicles to cause a navigation maneuver by the at least one vehicle based on the portion of the autonomous vehicle road navigation model.
[0021] In some embodiments of the server, the navigation information may include a trajectory from each of the plurality of vehicles as each vehicle travels on the common road segment.The portion of the autonomous vehicle road navigation model may include an update to the autonomous vehicle road navigation model.
[0022] In some embodiments, a navigation system for a vehicle may include: at least one processor programmed to receive output indicative of movement of the vehicle from one or more sensors; determine an actual trajectory of the vehicle based on the output from the one or more sensors; receive at least one environmental image associated with the vehicle from a camera; analyze the at least one environmental image to determine information associated with at least one navigation constraint; determine a target trajectory comprising the actual trajectory of the vehicle and one or more modifications to the actual trajectory based on the determined information associated with the at least one navigation constraint; and send the target trajectory from the vehicle to a server.
[0023] In some embodiments of the system, one or more sensors may include a rate sensor. One or more sensors may include an accelerometer. One or more sensors may include a camera. At least one navigation constraint may include at least one of: an obstacle, an object, a lane marking, a sign, or another vehicle. The camera may be included in the vehicle.
[0024] In some embodiments, a method for uploading a target trajectory to a server may include: receiving output indicating movement of a vehicle from one or more sensors; determining an actual trajectory of the vehicle based on the output from the one or more sensors; receiving at least one environmental image associated with the vehicle from a camera; analyzing the at least one environmental image to determine information associated with at least one navigation constraint; determining a target trajectory, the target trajectory comprising the actual trajectory of the vehicle and one or more modifications to the actual trajectory based on the determined information associated with the at least one navigation constraint; and sending the target trajectory from the vehicle to the server.
[0025] In some embodiments of the method, the one or more sensors may include a rate sensor. The one or more sensors may include an accelerometer. The one or more sensors may include a camera. The at least one navigation constraint may include at least one of: an obstacle, an object, a lane marking, a sign, or another vehicle. The camera may be included in the vehicle.
[0026] In some embodiments, a system for identifying landmarks for autonomous vehicle navigation may include at least one processor programmed to: receive at least one identifier associated with a landmark; associate the landmark with a corresponding road segment; update an autonomous vehicle road navigation model relative to the corresponding road segment to include the at least one identifier associated with the landmark; and distribute the updated autonomous vehicle road navigation model to a plurality of autonomous vehicles. The identifier may be determined based on: acquiring at least one image representing an environment of the host vehicle from a camera associated with the host vehicle; analyzing the at least one image to identify the landmark in the environment of the host vehicle; and analyzing the at least one image to determine the at least one identifier associated with the landmark.
[0027] In some embodiments of the system, at least one identifier may include the location of a landmark. At least one identifier may include the shape of the landmark. At least one identifier may include the dimensions of the landmark. At least one identifier may include the distance of the landmark relative to another landmark. At least one identifier may be determined based on the landmark being identified as one of a plurality of landmark types. The landmark type may include a traffic sign. The landmark type may include a post. The landmark type may include a directional indicator. The landmark type may include a rectangular sign. At least one identifier may also include a condensed signature representation. The condensed signature representation of the landmark may be determined based on mapping an image of the landmark into a sequence of numbers of a predetermined data size. The condensed signature representation may indicate the presence of the landmark. The condensed signature representation may indicate at least one of a color pattern or a brightness pattern of the image of the landmark. The landmark may include at least one of a directional sign, a traffic sign, a lamppost, a road marking, and a commercial sign.
[0028] In some embodiments, a method for identifying landmarks for use in autonomous vehicle navigation may include: receiving at least one identifier associated with the landmark; associating the landmark with a corresponding road segment; updating an autonomous vehicle road navigation model relative to the corresponding road segment to include at least one identifier associated with the landmark; and distributing the updated autonomous vehicle road navigation model to multiple autonomous vehicles.
[0029] In some embodiments, the method may include determining at least one identifier. Determining the at least one identifier may include: acquiring at least one image representing an environment of the host vehicle from a camera associated with the host vehicle; analyzing the at least one image to identify a landmark in the environment of the host vehicle; and analyzing the at least one image to determine at least one identifier associated with the landmark. The at least one identifier may include a distance of the landmark relative to another landmark, and wherein determining the at least one identifier includes determining the distance of the landmark relative to the other landmark. The at least one identifier may also include a condensed signature representation, and wherein determining the at least one identifier includes determining the condensed signature representation from the at least one image.
[0030] In some embodiments, a system for determining a location of a landmark for use in navigation of an autonomous vehicle may include at least one processor programmed to: receive a measured location of a landmark; and determine a refined location of the landmark based on the measured location of the landmark and at least one previously acquired location for the landmark. The measured location and the at least one previously acquired location may be determined based on: acquiring at least one image of an environment associated with a host vehicle from a camera associated with the host vehicle, analyzing the at least one image of the environment to identify a landmark in the host vehicle's environment, receiving Global Positioning System (GPS) data representing a location of the host vehicle, analyzing the at least one image of the environment to determine a relative location of the identified landmark relative to the host vehicle, and determining a globally localized location of the landmark based at least on the GPS data and the determined relative location.
[0031] In some embodiments of the system, the landmark may include at least one of: a traffic sign, an arrow, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost. Analyzing at least one image to determine the relative position of the identified landmark relative to the vehicle may include calculating a distance based on a scale associated with the at least one image. Analyzing the at least one image to determine the relative position of the identified landmark relative to the vehicle may include calculating a distance based on an optical flow associated with the at least one image. GPS data may be received from a GPS device included in the host vehicle. A camera may be included in the host vehicle. Determining the refined position of the landmark may include averaging the measured position of the landmark with at least one previously acquired position.
[0032] In some embodiments, a method for determining a location of a landmark for use in navigation of an autonomous vehicle may include: receiving a measured location of the landmark; and determining a refined location of the landmark based on the measured location of the landmark and at least one previously acquired location for the landmark. The measured location and the at least one previously acquired location may be determined based on: acquiring at least one image of an environment associated with a host vehicle from a camera associated with the host vehicle, analyzing the at least one image of the environment to identify a landmark in the host vehicle's environment, receiving Global Positioning System (GPS) data representing a location of the host vehicle, analyzing the at least one image of the environment to determine a relative location of the identified landmark relative to the host vehicle, and determining a globally localized location of the landmark based on at least the GPS data and the determined relative location.
[0033] In some embodiments of the method, the landmark may include at least one of: a traffic sign, an arrow, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost. Analyzing at least one image to determine the relative position of the identified landmark relative to the vehicle may include calculating a distance based on a scale associated with the at least one image. Analyzing at least one image to determine the relative position of the identified landmark relative to the vehicle may include calculating a distance based on an optical flow associated with the at least one image. GPS data may be received from a GPS device included in the host vehicle. A camera may be included in the host vehicle. Determining the refined position of the landmark may include averaging the measured position of the landmark with at least one previously acquired position.
[0034] In some embodiments, an autonomous vehicle may include a body and at least one processor programmed to: receive a measured position of a landmark; and determine a refined position of the landmark based on the measured position of the landmark and at least one previously acquired position for the landmark. The at least one processor may also be programmed to determine the measured position and the at least one previously acquired position based on: acquiring at least one image of an environment associated with the vehicle from a camera associated with the vehicle, analyzing the at least one image of the environment to identify a landmark in the vehicle's environment, receiving Global Positioning System (GPS) data representing a position of the vehicle, analyzing the at least one image of the environment to determine a relative position of the identified landmark with respect to the vehicle, and determining a globally localized position of the landmark based on at least the GPS data and the determined relative position.
[0035] In some embodiments of the vehicle, the landmark may include at least one of: a traffic sign, an arrow, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost. Analyzing at least one image to determine a relative position of the identified landmark relative to the vehicle may include calculating a distance based on a scale associated with the at least one image. Analyzing at least one image to determine a relative position of the identified landmark relative to the vehicle may include calculating a distance based on an optical flow associated with the at least one image. GPS data may be received from a GPS device included in the host vehicle. Determining a refined position of the landmark may include averaging a measured position of the landmark with at least one previously acquired position.
[0036] In some embodiments, a system for autonomously navigating a vehicle along a road segment may include at least one processor programmed to: receive at least one image representing the vehicle's environment from an image capture device; analyze the at least one image to identify at least one identified landmark; determine a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based at least in part on a predetermined position of the identified landmark; and determine an autonomous steering action for the vehicle based on a direction of the predetermined road model trajectory at the vehicle's determined current position relative to the predetermined road model trajectory.
[0037] In some embodiments of the system, the identified landmarks may include at least one of the following: a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The identified landmark may include a change in the spacing of lines on the road segment. The identified landmark may include a commercial sign. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. Navigation between the identified landmarks may include integrating the vehicle speed to determine the position of the vehicle along the predetermined road model trajectory. The processor may also be programmed to adjust the vehicle's steering system based on the autonomous steering action to navigate the vehicle. The processor may also be programmed to: determine a distance of the vehicle from at least one identified landmark; and determine whether the vehicle is located on a predetermined road model trajectory associated with the road segment based on the distance. The processor may also be programmed to: when the vehicle is not located on the predetermined road model trajectory, adjust the vehicle's steering system to move the vehicle from the vehicle's current position to a position on the predetermined road model trajectory.
[0038] In some embodiments, a vehicle may include: a body; at least one image capture device configured to acquire at least one image representing an environment of the vehicle; and at least one processor programmed to: receive at least one image from the at least one image capture device, analyze the at least one image to identify at least one identified landmark, determine a current position of the vehicle relative to a predetermined road model trajectory associated with a road segment based at least in part on a predetermined position of the identified landmark, and determine an autonomous steering action for the vehicle based on a direction of the predetermined road model trajectory at the determined current position of the vehicle relative to the predetermined road model trajectory.
[0039] In some embodiments of the vehicle, the identified landmarks may include at least one of the following: a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, a lamppost, a change in the spacing of lines on the road, or a commercial sign. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. Navigation between the identified landmarks may include integrating the vehicle's speed to determine the vehicle's position along the predetermined road model trajectory. The processor may also be programmed to adjust the vehicle's steering system based on the autonomous steering action to navigate the vehicle. The processor may also be programmed to determine a distance from the vehicle to at least one identified landmark; and based on the distance, determine whether the vehicle is located on the predetermined road model trajectory associated with the road segment. The processor may also be programmed to adjust the vehicle's steering system to move the vehicle from the vehicle's current position to a position on the predetermined road model trajectory when the vehicle is not located on the predetermined road model trajectory.
[0040] In some embodiments, a method of navigating a vehicle may include: receiving at least one image representing the vehicle's environment from an image capture device associated with the vehicle; using a processor associated with the vehicle to analyze the at least one image to identify at least one identified landmark; determining a current position of the vehicle relative to a predetermined road model trajectory associated with a road segment based at least in part on a predetermined position of the identified landmark; determining an autonomous steering action for the vehicle based on a direction of the predetermined road model trajectory at the vehicle's determined current position relative to the predetermined road model trajectory; and adjusting a steering system of the vehicle based on the autonomous steering action to navigate the vehicle.
[0041] In some embodiments, a method may include determining a position of a vehicle along a predetermined road model trajectory by integrating vehicle speed. The method may include determining, using a processor, a distance between the vehicle and at least one identified landmark; and determining, based on the distance, whether the vehicle is located on the predetermined road model trajectory associated with the road segment. The method may include determining a transformation required to move the vehicle from the vehicle's current position to a position on the predetermined road model trajectory; and adjusting a steering system of the vehicle based on the transformation.
[0042] In some embodiments, a system for autonomously navigating an autonomous vehicle along a road segment may include at least one processor programmed to: receive a plurality of images representing an environment of the autonomous vehicle from an image capture device; determine a traveled trajectory of the autonomous vehicle along the road segment based at least in part on an analysis of one or more of the plurality of images; determine a current position of the autonomous vehicle along a predetermined road model trajectory based on the analysis of one or more of the plurality of images; determine a heading direction for the autonomous vehicle based on the determined traveled trajectory; and determine a steering direction for the autonomous vehicle relative to the heading direction by comparing the traveled trajectory to the predetermined road model trajectory at the current position of the autonomous vehicle.
[0043] In some embodiments of the system, the comparison between the traveled trajectory and the predetermined road model trajectory may include determining a transformation that reduces the error between the traveled trajectory and the predetermined road model trajectory. The processor may also be programmed to adjust the steering system of the autonomous vehicle based on the transformation. The predetermined road model trajectory may include a three-dimensional polynomial representation of the target trajectory along the road segment. The predetermined road model trajectory may be retrieved from a database stored in a memory included in the autonomous vehicle. The predetermined road model trajectory may be retrieved from a database accessible to the autonomous vehicle via a wireless communication interface. An image capture device may be included in the autonomous vehicle. The determination of the steering direction may also be based on one or more additional cues, including one or more of the following: a left lane marking polynomial model, a right lane marking polynomial model, an overall path prediction, the motion of the vehicle ahead, a determined free space ahead of the autonomous vehicle, and a virtual lane or virtual lane constraint determined based on the position of a vehicle ahead of the autonomous vehicle. The determination of the steering direction may be based on weights applied to the one or more additional cues.
[0044] In some embodiments, an autonomous vehicle may include a body; at least one image capture device configured to acquire at least one image representing an environment of the autonomous vehicle; and at least one processor configured to: receive multiple images representing the environment of the autonomous vehicle from the image capture device; determine a driving trajectory of the autonomous vehicle along a road segment based at least in part on an analysis of one or more of the multiple images; determine a current position of the autonomous vehicle along a predetermined road model trajectory based on the analysis of one or more of the multiple images; determine a heading direction for the autonomous vehicle based on the determined driving trajectory; and determine a steering direction for the autonomous vehicle relative to the heading direction by comparing the driving trajectory and the predetermined road model trajectory at the current position of the autonomous vehicle.
[0045] In some embodiments of the autonomous vehicle, the comparison between the driving trajectory and the predetermined road model trajectory can include determining a transformation that reduces the error between the driving trajectory and the predetermined road model trajectory. The predetermined road model trajectory can include a three-dimensional polynomial representation of the target trajectory along the road segment. The predetermined road model trajectory can be retrieved from a database stored in a memory included in the autonomous vehicle and one of a database accessible to the autonomous vehicle via a wireless communication interface. The determination of the steering direction can also be based on one or more additional clues, including one or more of the following: a left lane marking polynomial model, a right lane marking polynomial model, an overall path prediction, the motion of the vehicle ahead, a determined free space in front of the autonomous vehicle, and a virtual lane or virtual lane constraint determined based on the position of a vehicle ahead of the autonomous vehicle. The determination of the steering direction can be based on weights applied to the one or more additional clues.
[0046] In some embodiments, a method for navigating an autonomous vehicle may include: receiving a plurality of images representing an environment of the autonomous vehicle from an image capture device; determining a driving trajectory of the autonomous vehicle along a road segment based at least in part on an analysis of one or more of the plurality of images; determining a current position of the autonomous vehicle along a predetermined road model trajectory based on the analysis of one or more of the plurality of images; determining a heading direction for the autonomous vehicle based on the determined driving trajectory; and determining a steering direction for the autonomous vehicle relative to the heading direction by comparing the driving trajectory to the predetermined road model trajectory at the current position of the autonomous vehicle.
[0047] In some embodiments of the method, comparing the driving trajectory to the predetermined road model trajectory may include determining a transformation that reduces an error between the driving trajectory and the predetermined road model trajectory. Determining the steering direction may be based on one or more additional cues, the additional cues including one or more of: a left lane marking polynomial model, a right lane marking polynomial model, an overall path prediction, motion of a preceding vehicle, a determined free space ahead of the ego vehicle, and a virtual lane or virtual lane constraint determined based on a position of a vehicle ahead of the ego vehicle. Determining the steering direction may include applying weights to the one or more additional cues.
[0048] In some embodiments, a system for autonomously navigating a vehicle through a road junction may include at least one processor programmed to: receive at least one image representing the vehicle's environment from an image capture device; analyze the at least one image to identify two or more landmarks located in the vehicle's environment; determine a directional indicator relative to the vehicle for each of the two or more landmarks; determine a current position of the vehicle relative to the road junction based on an intersection of the directional indicators for the two or more landmarks; determine a heading for the vehicle based on the directional indicators for the two or more landmarks; and determine a steering angle for the vehicle by comparing the vehicle heading to a predetermined road model trajectory at the vehicle's current position.
[0049] In some embodiments of the system, the predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. The two or more landmarks may include three or more landmarks. The at least one processor may also be programmed to send a control signal specifying a steering angle to a steering system of the vehicle. The processor may be configured to retrieve the predetermined road model trajectory from a database stored in a memory included in the vehicle. The processor may be configured to retrieve the predetermined road model trajectory from a database accessible to the vehicle via a wireless communication interface. A camera may be included in the vehicle. The processor may also be programmed to determine a heading for the vehicle by: determining a previous position of the vehicle relative to a road hub based on an intersection of direction indicators for two or more landmarks; and determining a heading based on the previous position and the current position.
[0050] In some embodiments, an autonomous vehicle may include a body; at least one image capture device configured to acquire at least one image representing an environment of the vehicle; and at least one processor programmed to: receive at least one image representing the environment of the vehicle from a camera; analyze at least one image to identify two or more landmarks located in the environment of the vehicle; determine a direction indicator relative to the vehicle for each of the two or more landmarks; determine a current position of the vehicle relative to a road junction based on an intersection of the direction indicators for the two or more landmarks; determine a heading for the vehicle based on the direction indicators for the two or more landmarks; and determine a steering angle for the vehicle by comparing the vehicle heading to a predetermined road model trajectory at the vehicle's current position.
[0051] In some embodiments of the vehicle, the predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. The two or more landmarks may include three or more landmarks. The at least one processor may also be programmed to send a control signal specifying a steering angle to a steering system of the vehicle. The predetermined road model trajectory may be retrieved from one of a database stored in a memory included in the vehicle and a database accessible to the vehicle via a wireless communication interface. The processor may also be programmed to determine a heading for the vehicle by: determining a previous position of the vehicle relative to the road hub based on an intersection of direction indicators for the two or more landmarks; and determining a heading based on the previous position and the current position.
[0052] In some embodiments, a method for navigating an autonomous vehicle may include: receiving at least one image representing an environment of the vehicle from an image capture device; analyzing the at least one image using at least one processor to identify two or more landmarks located in the environment of the vehicle; determining a direction indicator relative to the vehicle for each of the two or more landmarks; determining a current position of the vehicle relative to a road hub based on an intersection of the direction indicators for the two or more landmarks; determining a heading for the vehicle based on the direction indicators for the two or more landmarks; and determining a steering angle for the vehicle by comparing the vehicle heading to a predetermined road model trajectory at the vehicle's current position.
[0053] In some embodiments of the method, the predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. The method may include retrieving the predetermined road model trajectory from one of a database stored in a memory included in the vehicle and a database accessible to the vehicle via a wireless communication interface. The method may include sending a control signal specifying a steering angle to a steering system of the vehicle. Determining a heading for the vehicle may include: determining a previous position of the vehicle relative to the road junction based on an intersection of direction indicators for two or more landmarks; and determining a heading based on the previous position and the current position.
[0054] In some embodiments, a system for autonomously navigating a vehicle based on multiple overlapping navigation maps may include at least one processor programmed to: receive a first navigation map for use in autonomously controlling the vehicle, wherein the first navigation map is associated with a first road segment; determine at least a first autonomous navigation response for the vehicle along the first road segment based on an analysis of the first navigation map; receive a second navigation map for use in autonomously controlling the vehicle, wherein the second navigation map is associated with a second road segment, wherein the first road segment is different from the second road segment, and wherein the first road segment and the second road segment overlap at an overlapping segment; determine at least a second autonomous navigation response for the vehicle along the second road segment based on an analysis of the second navigation map; and determine at least a third autonomous navigation response for the vehicle in the overlapping segment based on at least one of the first navigation map and the second navigation map.
[0055] In some embodiments of the system, each of the multiple overlapping navigation maps may have its own coordinate system. Each of the multiple overlapping navigation maps may include a polynomial representation of a target trajectory along a road segment. Each of the overlapping navigation maps may be a sparse map with a data density of no more than 10 kilobytes per kilometer. The overlapping segment may have a length of at least 50 meters. The overlapping segment may have a length of at least 100 meters. At least one processor may be programmed to determine a third autonomous navigation response based on both the first navigation map and the second navigation map. The third autonomous navigation response may be a combination of the first autonomous navigation response and the second autonomous navigation response. The third autonomous navigation response may be an average of the first autonomous navigation response and the second autonomous navigation response. The processor may also be programmed to: determine an error between the first autonomous navigation response and the second autonomous navigation response; and when the error is less than a threshold error, determine the third autonomous navigation response based on the second autonomous navigation response.
[0056] In some embodiments, an autonomous vehicle may include a body; at least one image capture device configured to acquire at least one image representing an environment of the vehicle; and at least one processor programmed to: determine a current location of the vehicle based on the at least one image; receive a first navigation map associated with a first road segment; determine at least a first autonomous navigation response for the vehicle based on an analysis of the first navigation map when the current location of the vehicle is on the first navigation map; receive a second navigation map associated with a second road segment different from the first road segment, the first road segment and the second road segment overlapping each other at an overlapping segment; determine at least a second autonomous navigation response for the vehicle based on an analysis of the second navigation map when the current location of the vehicle is on the second navigation map; and determine at least a third autonomous navigation response for the vehicle based on at least one of the first navigation map and the second navigation map when the current location of the vehicle is in the overlapping segment.
[0057] In some embodiments of the autonomous vehicle, each of the first navigation map and the second navigation map can have its own coordinate system. Each of the first navigation map and the second navigation map can include a polynomial representation of a target trajectory along the road segment. At least one processor can be programmed to determine a third autonomous navigation response based on both the first navigation map and the second navigation map. The third autonomous navigation response can be a combination of the first autonomous navigation response and the second autonomous navigation response. The processor can also be programmed to: determine an error between the first autonomous navigation response and the second autonomous navigation response; and when the error is less than a threshold error, determine the third autonomous navigation response based on the second autonomous navigation response.
[0058] In some embodiments, a method for navigating an autonomous vehicle may include: receiving at least one image representing an environment of the vehicle from an image capture device; determining, using a processor associated with the vehicle, a current position of the vehicle based on the at least one image; receiving a first navigation map associated with a first road segment; determining at least a first autonomous navigation response for the vehicle based on an analysis of the first navigation map when the current position of the vehicle is on the first navigation map; receiving a second navigation map associated with a second road segment different from the first road segment, the first road segment and the second road segment overlapping each other at an overlapping segment; determining at least a second autonomous navigation response for the vehicle based on an analysis of the second navigation map when the current position of the vehicle is on the second navigation map; and determining at least a third autonomous navigation response for the vehicle based on at least one of the first navigation map and the second navigation map when the current position of the vehicle is in the overlapping segment.
[0059] In some embodiments of the method, each of the plurality of overlapping navigation maps may have its own coordinate system, and each of the plurality of overlapping navigation maps may include a polynomial representation of a target trajectory along the road segment. Determining the third autonomous navigation response may include determining a combination of the first autonomous navigation response and the second autonomous navigation response. The method may include determining an error between the first autonomous navigation response and the second autonomous navigation response; and when the error is less than a threshold error, determining the third autonomous navigation response based on the second autonomous navigation response.
[0060] In some embodiments, a system for sparse map autonomous navigation of a vehicle along a road segment may include at least one processor programmed to: receive a sparse map of the road segment, wherein the sparse map has a data density of no more than 1 megabyte per kilometer; receive at least one image representing the vehicle's environment from a camera; analyze the sparse map and the at least one image received from the camera; and determine an autonomous navigation response for the vehicle based solely on the analysis of the sparse map and the at least one image received from the camera.
[0061] In some embodiments of the system, the sparse map may include a polynomial representation of a target trajectory along a road segment. The sparse map may include one or more identified landmarks. The identified landmarks may be spaced in the sparse map at a frequency of no more than 0.5 per kilometer. The identified landmarks may be spaced in the sparse map at a frequency of no more than 1 per kilometer. The identified landmarks may be spaced in the sparse map at a frequency of no more than 1 per 100 meters. The sparse map may have a data density of no more than 100 kilobytes per kilometer. The sparse map may have a data density of no more than 10 kilobytes per kilometer.
[0062] In some embodiments, a method for sparse map autonomous navigation of a vehicle along a road segment may include: receiving a sparse map of the road segment, wherein the sparse map has a data density of no more than 1 megabyte per kilometer; receiving at least one image representing the vehicle's environment from a camera; analyzing the sparse map and the at least one image received from the camera; and determining an autonomous navigation response for the vehicle based solely on the analysis of the sparse map and the at least one image received from the camera.
[0063] In some embodiments of the method, the sparse map may include a polynomial representation of a target trajectory along a road segment. The sparse map may include one or more identified landmarks. The identified landmarks may be spaced in the sparse map at a frequency of no more than 0.5 per kilometer. The identified landmarks may be spaced in the sparse map at a frequency of no more than 1 per kilometer. The identified landmarks may be spaced in the sparse map at a frequency of no more than 1 per 100 meters. The sparse map may have a data density of no more than 100 kilobytes per kilometer. The sparse map may have a data density of no more than 10 kilobytes per kilometer.
[0064] In some embodiments, a non-transitory computer-readable medium may store instructions that cause at least one processor to perform sparse map autonomous navigation of a vehicle along a road segment, which may include receiving a sparse map of the road segment. The instructions may cause the processor to perform the following steps: receiving a sparse map of the road segment, wherein the sparse map has a data density of no more than 1 megabyte per kilometer; receiving at least one image representing the vehicle's environment from a camera; analyzing the sparse map and the at least one image received from the camera; and determining an autonomous navigation response for the vehicle based solely on the analysis of the sparse map and the at least one image received from the camera.
[0065] In some embodiments of the non-transitory computer-readable medium, the sparse map may include a polynomial representation of the target trajectory along the road segment. The sparse map may include one or more identified landmarks. The identified landmarks may be spaced in the sparse map at a frequency of no more than 0.5 per kilometer.
[0066] In some embodiments, a system for autonomously navigating a vehicle along a road segment based on predetermined landmark locations may include at least one processor programmed to: receive at least one image representing the vehicle's environment from a camera; determine a position of the vehicle along a predetermined road model trajectory associated with the road segment based at least in part on information associated with the at least one image; identify an identified landmark in front of the vehicle based on the determined position, wherein the identified landmark is beyond the field of view of the camera; determine a current distance between the vehicle and the identified landmark by comparing the determined position of the vehicle with a predetermined position of the identified landmark; and determine an autonomous navigation response for the vehicle based on the determined current distance.
[0067] In some embodiments of the system, the predetermined position of the identified landmark can be determined as an average of a plurality of acquired position measurements associated with the identified landmark, wherein the plurality of acquired position measurements are determined based on: acquisition of at least one image of the environment, analysis of the at least one image of the environment to identify the identified landmark in the environment, reception of global positioning system (GPS) data, analysis of the at least one image of the environment to determine a relative position of the identified landmark relative to the vehicle, and determination of a globally localized position of the identified landmark based on at least the GPS data and the determined relative position. The autonomous navigation response can include application of brakes associated with the vehicle. The autonomous navigation response can include modifying a steering angle of the vehicle. The identified landmark can include a stop line, a traffic light, a stop sign, or a curve along a road segment. A camera can be included in the vehicle.
[0068] In some embodiments, a method for autonomously navigating a vehicle along a road segment based on predetermined landmark locations may include receiving at least one image representing the vehicle's environment from a camera; determining a position of the vehicle along a predetermined road model trajectory associated with the road segment based at least in part on information associated with the at least one image; identifying an identified landmark in front of the vehicle based on the determined position, wherein the identified landmark is beyond the field of view of the camera; determining a current distance between the vehicle and the identified landmark by comparing the determined position of the vehicle with a predetermined position of the identified landmark; and determining an autonomous navigation response for the vehicle based on the determined current distance.
[0069] In some embodiments of the method, the predetermined position of the identified landmark can be determined as an average of a plurality of acquired position measurements associated with the identified landmark, wherein the plurality of acquired position measurements can be determined based on: acquisition of at least one image of the environment, analysis of the at least one image of the environment to identify the identified landmark in the environment, reception of global positioning system (GPS) data, analysis of the at least one image of the environment to determine a relative position of the identified landmark relative to the vehicle, and determination of a globally localized position of the identified landmark based on at least the GPS data and the determined relative position. The autonomous navigation response can include application of brakes associated with the vehicle. The autonomous navigation response can include modifying a steering angle of the vehicle. The identified landmark can include a stop line, a traffic light, a stop sign, or a curve along a road segment. The camera can be included in the vehicle.
[0070] In some embodiments, a non-transitory computer-readable medium may store instructions that cause at least one processor to perform autonomous navigation of a vehicle along a road segment. The instructions may cause the processor to perform the following steps: receiving at least one image representing an environment of the vehicle from a camera; determining a position of the vehicle along a predetermined road model trajectory associated with the road segment based at least in part on information associated with the at least one image; identifying a recognized landmark in front of the vehicle based on the determined position, wherein the recognized landmark is beyond a field of view of the camera; determining a current distance between the vehicle and the recognized landmark by comparing the determined position of the vehicle to a predetermined position of the recognized landmark; and determining an autonomous navigation response for the vehicle based on the determined current distance.
[0071] In some embodiments of the non-transitory computer-readable medium, the autonomous navigation response may include applying brakes associated with the vehicle. The autonomous navigation response may include modifying a steering angle of the vehicle. The identified landmark may include a stop line, a traffic light, a stop sign, or a curve along the road segment.
[0072] In some embodiments, a system for autonomously navigating a vehicle along a road segment may include at least one processor programmed to: receive information relating to one or more aspects of the road segment from at least one sensor; determine local characteristics of the road segment based on the received information; compare the local characteristics with predetermined signature characteristics for the road segment; determine a current position of the vehicle along a predetermined road model trajectory associated with the road segment based on the comparison of the local characteristics and the predetermined signature characteristics; and determine an autonomous steering action for the vehicle based on the direction of the predetermined road model trajectory at the determined position.
[0073] In some embodiments of the system, the at least one processor may be further programmed to determine a heading direction of the vehicle at a current location and determine an autonomous steering action by comparing the direction of a predetermined road model trajectory with the heading direction. The heading direction may be determined based on the vehicle's travel trajectory. The at least one sensor may include an image capture device configured to acquire at least one image representing the vehicle's environment. The signature feature may include a road width profile along at least a portion of the road segment. The signature feature may include a lane width profile along at least a portion of the road segment. The signature feature may include a dashed line spacing profile along at least a portion of the road segment. The signature feature may include a predetermined number of road markings along at least a portion of the road segment. The signature feature may include a road surface profile along at least a portion of the road segment. The signature feature may include a predetermined curvature associated with the road segment. Determining the vehicle's current location may include comparing a first parameter value indicating the curvature of the predetermined road model trajectory with a second parameter value indicating the curvature of a measured trajectory for the vehicle. The at least one sensor may include a suspension assembly monitor.
[0074] In some embodiments, a vehicle may include a body; at least one sensor configured to acquire information relating to one or more aspects of a road segment; and at least one processor programmed to: determine local features of the road segment based on information received from the at least one sensor; compare the local features with predetermined signature features for the road segment; determine a current position of the vehicle along a predetermined road model trajectory associated with the road segment based on the comparison of the local features and the predetermined signature features; and determine an autonomous steering action for the vehicle based on the direction of the predetermined road model trajectory at the current position.
[0075] In some embodiments of the vehicle, the signature feature may include at least one of: a road width profile over at least a portion of the road segment, a lane width profile over at least a portion of the road segment, a dashed spaced profile over at least a portion of the road segment, a predetermined number of road markings along at least a portion of the road segment, a road surface profile over at least a portion of the road segment, and a predetermined curvature associated with the road segment. The vehicle may include a suspension component monitor, wherein the processor is further programmed to determine the local feature based on a signal from the suspension component monitor. The processor may also be programmed to: determine a heading direction of the vehicle; determine a direction of a predetermined road model trajectory at a current location; and determine an autonomous steering action by comparing the direction to the heading direction.
[0076] In some embodiments, a method of navigating a vehicle may include: receiving information related to one or more aspects of a road segment from at least one sensor; determining, using at least one processor, local characteristics of the road segment based on the information received from the at least one sensor; comparing the received information with a predetermined signature characteristic for the road segment; determining a current position of the vehicle along a predetermined road model trajectory associated with the road segment based on the comparison of the received information and the predetermined signature characteristic; and determining an autonomous steering action for the vehicle based on the direction of the predetermined road model trajectory at the current position.
[0077] In some embodiments, the method may include determining a heading direction of the vehicle at a current location; determining a direction of a predetermined road model trajectory at the current location; and determining an autonomous steering action by comparing the direction of the predetermined road model trajectory with the heading direction. The local features may include at least one of: a road width profile on at least a portion of the road segment, a lane width profile on at least a portion of the road segment, a dashed line spacing profile on at least a portion of the road segment, a predetermined number of road markings along at least a portion of the road segment, a road surface profile on at least a portion of the road segment, and a predetermined curvature associated with the road segment. The method may include: determining a road surface profile using a suspension assembly monitor; comparing the road surface profile to a predetermined road surface profile; and determining the current location based on the comparison of the road surface profile to the predetermined road surface profile.
[0078] In some embodiments, a system for autonomously navigating a vehicle may include at least one processor programmed to: receive at least one image representing an area behind the vehicle from a rearward-facing camera; analyze the at least one rearward-facing image to locate a representation of at least one landmark in the image; determine at least one indicator of the position of the landmark relative to the vehicle; determine a forward trajectory for the vehicle based at least in part on the indicator of the position of the landmark relative to the vehicle; and cause the vehicle to navigate along the determined forward trajectory.
[0079] In some embodiments of the system, the indicator of position may include a distance between the vehicle and a landmark. The indicator of position may include a relative angle between the vehicle and the landmark. The landmark may include a road edge, a lane marking, a reflector, a pole, a change in a line pattern on the road, or a road sign. The landmark may include the back of a road sign. The at least one processor may also be programmed to determine a lane offset of the vehicle within a current lane of travel based on the indicator of the position of the landmark, and wherein the determination of the forward trajectory is further based on the determined lane offset. The at least one processor may also be programmed to receive at least one image representing another area of the vehicle from another camera, and wherein the determination of the forward trajectory is further based on the at least one image received from the other camera.
[0080] In some embodiments, a method of autonomously navigating a vehicle may include receiving at least one image from a rearward-facing camera representing an area behind the vehicle; analyzing the at least one rearward-facing image to locate a representation of at least one landmark in the image; determining at least one indicator of the position of the landmark relative to the vehicle; determining a forward trajectory for the vehicle based at least in part on the indicator of the position of the landmark relative to the vehicle; and causing the vehicle to navigate along the determined forward trajectory.
[0081] In some embodiments of the method, the indicator of position may include a distance between the vehicle and a landmark. The indicator of position may include a relative angle between the vehicle and the landmark. The landmark may include a road edge, a lane marking, a reflector, a pole, a change in line pattern on the road, or a road sign. The landmark may include the back of a road sign. The method may include determining a lane offset of the vehicle within a current lane of travel based on the indicator of the position of the landmark, and wherein determining the forward trajectory may be based on the determined lane offset.
[0082] In some embodiments, a vehicle may include a body; a rearward-facing camera; and at least one processor programmed to: receive at least one image representing an area behind the vehicle via a rearward-facing camera interface connected to the rearward-facing camera; analyze the at least one rearward-facing image to locate a representation of at least one landmark in the image; determine at least one indicator of the position of the landmark relative to the vehicle; determine a forward trajectory for the vehicle based at least in part on the indicator of the position of the landmark relative to the vehicle; and cause the vehicle to navigate along the determined forward trajectory.
[0083] In some embodiments of the vehicle, the rear-facing camera can be mounted on an object connected to the vehicle. The object can be a trailer, a bike rack, a ski / snowboard rack, a mount, or a luggage rack. The rear-facing camera interface can include a detachable interface. The rear-facing camera interface can include a wireless interface.
[0084] In some embodiments, a system for navigating a vehicle by determining a free space zone in which the vehicle can travel may include at least one processor programmed to: receive a plurality of images associated with the vehicle's environment from an image capture device; analyze at least one of the plurality of images to identify a first free space boundary on a driver's side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger side of the vehicle and extending forward of the vehicle, and a front free space boundary in front of the vehicle and extending between the first free space boundary and the second free space boundary; wherein the first free space boundary, the second free space boundary, and the front free space boundary define a free space zone in front of the vehicle; determine a navigation path for the vehicle through the free space zone; and cause the vehicle to travel on at least a portion of the determined navigation path within the free space zone in front of the vehicle.
[0085] In some embodiments of the system, the first free space boundary may correspond to at least one of the following: a road edge, a curb, an obstacle, a lane-dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The second free space boundary may correspond to at least one of the following: a road edge, a curb, an obstacle, a lane-dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The front free space may correspond to a horizontal line of the road. The at least one processor may be further programmed to identify an obstacle in front of the vehicle based on an analysis of at least one of the plurality of images, and exclude the identified obstacle from the free space zone in front of the vehicle. The obstacle may include a pedestrian. The obstacle may include another vehicle. The obstacle may include a rock. The at least one processor may be further programmed to identify an obstacle in front of the vehicle based on an analysis of at least one of the plurality of images, and exclude a zone surrounding the identified obstacle from the free space zone in front of the vehicle. The at least one processor may be further programmed to determine the zone surrounding the identified obstacle based on one or more of the following: the speed of the vehicle, the type of obstacle, the image capture rate of the image capture device, and the movement rate of the obstacle.
[0086] In some embodiments, a vehicle may include: a body comprising a driver side and a passenger side; an image capture device; and at least one processor programmed to: receive a plurality of images associated with an environment of the vehicle from the image capture device; analyze at least one of the plurality of images to identify a first free space boundary on the driver side of the body and extending forward of the body, a second free space boundary on the passenger side of the body and extending forward of the body, and a front free space boundary in front of the body and extending between the first free space boundary and the second free space boundary; wherein the first free space boundary, the second free space boundary, and the front free space boundary define a free space region in front of the body; determine a navigation path for the vehicle through the free space region; and cause the vehicle to travel on at least a portion of the determined navigation path within the free space region in front of the vehicle.
[0087] In some embodiments, a method for navigating a vehicle by determining a free space zone in which the vehicle can travel may include: receiving a plurality of images associated with an environment of the vehicle from an image capture device; analyzing at least one of the plurality of images to identify a first free space boundary on a driver's side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger side of the vehicle and extending forward of the vehicle, and a front free space boundary in front of the vehicle and extending between the first free space boundary and the second free space boundary; wherein the first free space boundary, the second free space boundary, and the front free space boundary define a free space zone in front of the vehicle; determining a navigation path for the vehicle through the free space zone; and causing the vehicle to travel on at least a portion of the determined navigation path within the free space zone in front of the vehicle.
[0088] In some embodiments of the method, the first free space boundary may correspond to at least one of the following: a road edge, a curb, an obstacle, a lane-dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The second free space boundary may correspond to at least one of the following: a road edge, a curb, an obstacle, a lane-dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The front free space may correspond to a road level. The method may include identifying an obstacle in front of the vehicle based on an analysis of at least one of the plurality of images, and excluding the identified obstacle from the free space region in front of the vehicle. The obstacle may include a pedestrian. The obstacle may include another vehicle. The obstacle may include a rock. The method may include identifying an obstacle in front of the vehicle based on an analysis of at least one of the plurality of images, and excluding a region surrounding the identified obstacle from the free space region in front of the vehicle. The method may include determining the region surrounding the identified obstacle based on one or more of the following: a velocity of the vehicle, a type of the obstacle, an image capture rate of an image capture device, and a rate of movement of the obstacle.
[0089] In some embodiments, a system for navigating a vehicle on a road where snow covers at least some lane markings and an edge of the road may include at least one processor programmed to: receive at least one image of the environment in front of the vehicle from an image capture device, including an area where snow covers at least some lane markings and an edge of the road; identify at least a portion of the road covered with snow and possible locations of an edge of the road for defining at least a portion of the road covered with snow based on an analysis of the at least one image; and cause the vehicle to navigate a navigation path that includes the identified portion of the road and falls within the determined possible locations of the edge of the road.
[0090] In some embodiments of the system, analyzing the at least one image may include identifying at least one tire impression in the snow. Analyzing the at least one image may include identifying changes in light across the surface of the snow. Analyzing the at least one image may include identifying a plurality of trees along the edge of a road. Analyzing the at least one image may include identifying changes in curvature of the surface of the snow. The identified changes in curvature may be determined to correspond to a possible location of an edge of the road. Analyzing the at least one image may include pixel analysis of the at least one image, wherein at least a first pixel and at least a second pixel are compared to determine a feature associated with the surface of snow covering at least some lane markings and the edge of the road. The feature may correspond to an edge of the tire impression. The feature may correspond to an edge of the road. The at least one processor may also be programmed to cause the vehicle to navigate between the determined edges of the road. The at least one processor may also be programmed to cause the vehicle to navigate by at least partially following the tire impressions in the snow.
[0091] In some embodiments, a method for navigating a vehicle on a road where snow covers at least some lane markings and an edge of the road may include: receiving at least one image of the environment in front of the vehicle from an image capture device, including an area where snow covers at least some lane markings and an edge of the road; identifying at least a portion of the road covered with snow and possible locations of an edge of the road for defining at least a portion of the road covered with snow based on an analysis of the at least one image; and causing the vehicle to navigate a navigation path that includes the identified portion of the road and falls within the determined possible locations of the edge of the road.
[0092] In some embodiments of the method, analyzing the at least one image may include identifying at least one tire impression in the snow. Analyzing the at least one image may include identifying changes in light across the surface of the snow. Analyzing the at least one image may include identifying a plurality of trees along an edge of a road. Analyzing the at least one image may include identifying changes in curvature of the surface of the snow. The identified changes in curvature may be determined to correspond to a possible location of an edge of the road. Analyzing the at least one image may include pixel analysis of the at least one image, wherein at least a first pixel and at least a second pixel are compared to determine a feature associated with the surface of snow covering at least some lane markings and the edge of the road. The feature may correspond to an edge of the tire impression. The feature may correspond to an edge of the road. The method may include causing the vehicle to navigate between the determined edges of the road. The method may include causing the vehicle to navigate by at least partially following the tire impressions in the snow.
[0093] In some embodiments, a system for navigating a vehicle on a road at least partially covered with snow may include at least one processor programmed to: receive from an image capture device a plurality of images capturing an environment in front of the vehicle, including an area of the road on which the vehicle is traveling where snow covers snow; analyze at least one of the plurality of images to identify a first free space boundary on a driver's side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger side of the vehicle and extending forward of the vehicle, and a front free space boundary in front of the vehicle and extending between the first free space boundary and the second free space boundary; wherein the first free space boundary, the second free space boundary, and the front free space boundary define a free space region in front of the vehicle; determine a first suggested navigation path for the vehicle through the free space region; provide at least one of the plurality of images to a neural network, and based on the neural network's analysis of at least one of the plurality of images, receive from the neural network a second suggested navigation path for the vehicle; determine whether the first suggested navigation path is consistent with the second suggested navigation path; and if it is determined that the first suggested navigation path is consistent with the second suggested navigation path, cause the vehicle to travel on at least a portion of the first suggested navigation path.
[0094] In some embodiments, a system for calibrating an indicator of a velocity of an autonomous vehicle may include at least one processor programmed to: receive a plurality of images representing the vehicle's environment from a camera; analyze the plurality of images to identify at least two identified landmarks; determine a value indicating a distance between the at least two identified landmarks based on known positions of the two identified landmarks; determine a measured distance between the at least two landmarks based on an output of at least one sensor associated with the autonomous vehicle; and determine a correction factor for at least one sensor based on a comparison of the value indicating the distance between the at least two identified landmarks and the measured distance between the at least two landmarks.
[0095] In some embodiments of the system, a correction factor can be determined so that the operation of the correction factor on the determined distance along the road segment matches the distance value received via the wireless transceiver. The two identified landmarks can include one or more of the following: a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. At least one sensor can include a speedometer associated with the vehicle. The known positions of the two identified landmarks can be received from a server-based system remotely located relative to the vehicle. Each of the known positions can constitute a refined position determined based on multiple GPS-based measurements.
[0096] In some embodiments, a system for calibrating an indicator of a velocity of an autonomous vehicle may include at least one processor programmed to: determine a distance along a road segment based on an output of at least one sensor associated with the autonomous vehicle; receive a distance value associated with the road segment via a wireless transceiver; and determine a correction factor for the at least one sensor based on the determined distance along the road segment and the distance value received via the wireless transceiver.
[0097] In some embodiments of the system, a distance value associated with a road segment and received via a wireless transceiver can be determined based on prior measurements performed by a plurality of measurement vehicles. The plurality of measurement vehicles can include at least 100 measurement vehicles. The plurality of measurement vehicles can include at least 1,000 measurement vehicles. A correction factor can be determined such that the operation of the correction factor on the determined distance along the road segment matches the distance value received via the wireless transceiver. The at least one processor can be programmed to determine a composite correction factor based on the plurality of determined correction factors. The composite correction factor can be determined by averaging the plurality of determined correction factors. The composite correction factor can be determined by finding the mean of the plurality of determined correction factors.
[0098] In some embodiments, a vehicle may include a body; a camera; and at least one processor programmed to: receive a plurality of images representing the vehicle's environment from the camera; analyze the plurality of images to identify at least two identified landmarks; determine a value indicating a distance between the at least two identified landmarks based on the known positions of the two identified landmarks; determine a measured distance between the at least two landmarks based on an output of at least one sensor associated with the autonomous vehicle; and determine a correction factor for at least one sensor based on a comparison of the value indicating the distance between the at least two identified landmarks and the measured distance between the at least two landmarks.
[0099] In some embodiments of the vehicle, the at least one sensor may include a speedometer associated with the vehicle. The two identified landmarks may include one or more of the following: a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The known positions of the two identified landmarks may be received from a server-based system remotely located relative to the vehicle.
[0100] In some embodiments, a system for determining a lane assignment for an autonomous vehicle along a road segment may include at least one processor programmed to: receive at least one image representing the vehicle's environment from a camera; analyze the at least one image to identify at least one identified landmark; determine an indicator of a lateral offset distance between the vehicle and the at least one identified landmark; and determine a lane assignment for the vehicle along the road segment based on the indicator of the lateral offset distance between the vehicle and the at least one identified landmark.
[0101] In some embodiments of the system, the vehicle's environment may include a road segment, the number of lanes, and at least one identified landmark. The at least one identified landmark may include at least one of the following: a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one identified landmark may include a commercial sign. The lateral offset distance between the vehicle and the at least one identified landmark may be the sum of a first distance between the vehicle and a first side of the road segment and a second distance between the first side of the road and the at least one identified landmark. The determination of the indicator of the lateral offset distance between the vehicle and the at least one identified landmark may be based on a predetermined location of the at least one identified landmark. The determination of the indicator of the lateral offset distance between the vehicle and the at least one identified landmark may be based on a scale associated with at least one image. The determination of the lane assignment may also be based on at least one of the following: the width of the road segment, the number of lanes in the road segment, and the lane width. The determination of the lane assignment may also be based on a predetermined road model trajectory associated with the road segment. The at least one identified landmark may include a first identified landmark on a first side of the vehicle and a second identified landmark on a second side of the vehicle, and wherein determination of a lane assignment for the vehicle along the road segment is based on a first indicator of a lateral offset distance between the vehicle and the first identified landmark, and a second indicator of a lateral offset distance between the vehicle and the second identified landmark.
[0102] In some embodiments, a computer-implemented method for determining a lane assignment for an autonomous vehicle along a road segment may include the following operations performed by one or more processors: receiving at least one image representing the vehicle's environment from a camera; analyzing the at least one image to identify at least one identified landmark; determining an indicator of a lateral offset distance between the vehicle and the at least one identified landmark; and determining a lane assignment for the vehicle along the road segment based on the indicator of the lateral offset distance between the vehicle and the at least one identified landmark.
[0103] In some embodiments of the method, the at least one identified landmark may include at least one of a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one identified landmark may include a commercial sign. The determination of lane assignment may also be based on a predetermined road model trajectory associated with the road segment. The at least one identified landmark may include a first identified landmark on a first side of the vehicle and a second identified landmark on a second side of the vehicle, and wherein the determination of lane assignment for the vehicle along the road segment is based on a first indicator of a lateral offset distance between the vehicle and the first identified landmark, and a second indicator of a lateral offset distance between the vehicle and the second identified landmark.
[0104] In some embodiments, a computer-readable storage medium may include a set of instructions executable by at least one processor to cause the at least one processor to perform a method for determining a lane assignment for an autonomous vehicle along a road segment. The method may include receiving at least one image representing an environment of a vehicle from a camera; analyzing the at least one image to identify at least one recognized landmark; determining an indicator of a lateral offset distance between the vehicle and the at least one recognized landmark; and determining a lane assignment for the vehicle along the road segment based on the indicator of the lateral offset distance between the vehicle and the at least one recognized landmark.
[0105] In some embodiments of the computer-readable storage medium, at least one identified landmark may include at least one of the following: a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. At least one identified landmark may include a commercial sign. The determination of lane assignment may also be based on a predetermined road model trajectory associated with the road segment. At least one identified landmark may include a first identified landmark on a first side of the vehicle and a second identified landmark on a second side of the vehicle, and wherein the determination of lane assignment for the vehicle along the road segment is based on a first indicator of a lateral offset distance between the vehicle and the first identified landmark, and a second indicator of a lateral offset distance between the vehicle and the second identified landmark.
[0106] In some embodiments, a system for autonomously navigating a vehicle along a road segment may include at least one processor programmed to: receive at least one image representing the vehicle's environment from a camera; analyze the at least one image to identify at least one identified landmark, wherein the at least one identified landmark is part of a group of identified landmarks, and the identification of the at least one identified landmark is based at least in part on one or more landmark group characteristics associated with the group of identified landmarks; determine a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based at least in part on a predetermined position of the identified landmark; and determine an autonomous steering action for the vehicle based on the direction of the predetermined road model trajectory at the vehicle's determined current position relative to the predetermined road model trajectory.
[0107] In some embodiments of the system, at least one identified landmark may include at least one of the following: a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one identified landmark may include a commercial sign. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. The at least one processor may also be programmed to determine a current position of the vehicle along the predetermined road model trajectory based on the vehicle speed. The one or more landmark group characteristics may include relative distances between members of the group of identified landmarks. The one or more landmark group characteristics may include a sorted sequence of members of the group of identified landmarks. The one or more landmark group characteristics may include the number of landmarks included in the group of identified landmarks. The identification of the at least one identified landmark may be based at least in part on a super landmark signature associated with the group of identified landmarks. The at least one processor may be programmed to determine an autonomous steering action for the vehicle by comparing the vehicle's heading direction to the predetermined road model trajectory at the vehicle's determined current position.
[0108] In some embodiments, a computer-implemented method for autonomously navigating a vehicle along a road segment may include the following operations performed by one or more processors: receiving at least one image representing the vehicle's environment from a camera; analyzing the at least one image to identify at least one identified landmark, wherein the at least one identified landmark is part of a group of identified landmarks and the identification of the at least one identified landmark is based at least in part on one or more landmark group characteristics associated with the group of identified landmarks; determining, relative to the vehicle, a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based at least in part on predetermined positions of the identified landmarks; and determining an autonomous steering action for the vehicle based on the direction of the predetermined road model trajectory at the vehicle's determined current position relative to the predetermined road model trajectory.
[0109] In some embodiments of the method, the at least one identified landmark may include at least one of a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one identified landmark may include a commercial sign. The one or more landmark group characteristics may include relative distances between members of the group of identified landmarks. The one or more landmark group characteristics may include a sorted sequence of the members of the group of identified landmarks.
[0110] In some embodiments, a computer-readable storage medium may include a set of instructions executable by at least one processor to cause the at least one processor to perform a method for autonomously navigating a vehicle along a road segment. The method may include receiving at least one image representing an environment of the vehicle from a camera; analyzing the at least one image to identify at least one recognized landmark, wherein the at least one recognized landmark is part of a group of recognized landmarks and the identification of the at least one recognized landmark is based at least in part on one or more landmark group characteristics associated with the group of recognized landmarks; determining a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based at least in part on a predetermined position of the recognized landmark relative to the vehicle; and determining an autonomous steering action for the vehicle based on a direction of the predetermined road model trajectory at the determined current position of the vehicle relative to the predetermined road model trajectory.
[0111] In some embodiments of the computer-readable storage medium, the at least one identified landmark may include at least one of a traffic sign, an arrow marking, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one identified landmark may include a commercial sign. The one or more landmark group characteristics may include relative distances between members of the group of identified landmarks. The one or more landmark group characteristics may include a sorted sequence of the members of the group of identified landmarks.
[0112] In some embodiments, a navigation system for a vehicle may include at least one processor programmed to: receive at least one environmental image associated with the vehicle from a camera; determine a navigation maneuver for the vehicle based on an analysis of the at least one environmental image; cause the vehicle to initiate a navigation maneuver; receive user input associated with a navigation response from a user different from the user who initiated the navigation maneuver; determine navigation situation information related to the vehicle based on the received user input; and store the navigation situation information in conjunction with information related to the user input.
[0113] In some embodiments of the system, navigation maneuvers may be based on identified landmarks identified in at least one image of the environment. Information related to user input may include information specifying at least one of a degree of vehicle turning, an amount of vehicle acceleration, and an amount of vehicle braking. The control system may include at least one of steering control, acceleration control, and braking control. Navigation situation information may include one or more images captured by a camera on the vehicle. User input may include at least one of braking, steering, or acceleration. Navigation situation information may include the location of the vehicle. Navigation situation information may include at least one output of a sensor on the vehicle. The sensor may be a ratemeter. The sensor may be an accelerometer. The sensor may be an IR sensor. The navigation situation information may include the time of day. The navigation situation information may include an indication of the presence of a vision inhibitor. A vision inhibitor may be caused by glare. The navigation situation information may be determined based on at least one image of the environment. The system may include a transmitter for transmitting the navigation situation information to a server remote from the vehicle.
[0114] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor to cause the at least one processor to perform a method. The method may include receiving at least one environmental image associated with a vehicle from a camera; determining a navigation maneuver for the vehicle based on an analysis of the at least one environmental image; causing the vehicle to initiate the navigation maneuver; receiving user input associated with a navigation response from a user different from the initiated navigation maneuver; determining navigation situation information related to the vehicle based on the received user input; and storing the navigation situation information in conjunction with information related to the user input.
[0115] In some embodiments, a navigation system for a vehicle may include at least one processor programmed to: determine a navigation maneuver for the vehicle based at least in part on a comparison of the vehicle's movement relative to a predetermined model representing a road segment; receive at least one image representing the vehicle's environment from a camera; determine the presence of a navigation adjustment condition in the vehicle's environment based on an analysis of the at least one image; cause the vehicle to adjust the navigation maneuver based on the presence of the navigation adjustment condition; and store information related to the navigation adjustment condition.
[0116] In some embodiments of the system, the navigation adjustment condition may include a parked car. The navigation adjustment condition may include a lane shift. The navigation adjustment condition may include at least one of a recently encountered traffic sign or a recently encountered traffic light. The navigation adjustment condition may include a construction site area. The processor may also be programmed to cause stored information related to the navigation adjustment condition to be sent to a road model management system for determining whether the navigation adjustment condition warrants an update to a predetermined model representing the road segment. The information stored with respect to the navigation adjustment condition may include at least one of: an indicator of the location where the navigation adjustment condition was encountered, an indicator of adjustments made to the navigation maneuver, and at least one image. The predetermined model representing the road segment may include a three-dimensional spline representing a predetermined path for travel along the road segment.
[0117] In some embodiments, a method for navigating a vehicle may include: determining a navigation maneuver for the vehicle based at least in part on a comparison of the vehicle's movement relative to a predetermined model representing a road segment; receiving at least one image representing the vehicle's environment from a camera; determining the presence of a navigation adjustment condition in the vehicle's environment based on an analysis of the at least one image; causing the vehicle to adjust the navigation maneuver based on the presence of the navigation adjustment condition; and storing information related to the navigation adjustment condition.
[0118] In some embodiments of the method, the navigation adjustment condition may include a parked car. The navigation adjustment condition may include a lane change. The navigation adjustment condition may include at least one of a recently encountered traffic sign or a recently encountered traffic light. The navigation adjustment condition may include a construction site area. The method may include causing stored information related to the navigation adjustment condition to be sent to a road model management system for determining whether the navigation adjustment condition warrants an update to a predetermined model representing the road segment. The information stored with respect to the navigation adjustment condition may include at least one of: an indicator of the location where the navigation adjustment condition was encountered, an indicator of the adjustment made to the navigation maneuver, and at least one image. The predetermined model representing the road segment may include a three-dimensional spline representing a predetermined path for travel along the road segment.
[0119] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor to cause the at least one processor to perform a method. The method may include: determining a navigation maneuver for a vehicle based at least in part on a comparison of the vehicle's motion relative to a predetermined model representing a road segment; receiving at least one image representing an environment of the vehicle from a camera; determining the presence of a navigation adjustment condition in the environment of the vehicle based on an analysis of the at least one image; causing the vehicle to adjust the navigation maneuver based on the presence of the navigation adjustment condition; and storing information related to the navigation adjustment condition.
[0120] In some embodiments of the computer-readable medium, a method may include causing stored information related to a navigation adjustment condition to be sent to a road model management system for use in determining whether the navigation adjustment condition warrants updating a predetermined model representing the road segment. The information stored regarding the navigation adjustment condition may include at least one of: an indicator of a location where the navigation adjustment condition was encountered, an indicator of an adjustment made to the navigation maneuver, and at least one image. The predetermined model representing the road segment may include a three-dimensional spline representing a predetermined path for travel along the road segment.
[0121] In some embodiments, a system for interacting with multiple autonomous vehicles may include: a memory including a predetermined model representing at least one road segment; and at least one processor programmed to: receive navigation situation information associated with the occurrence of an adjustment to a determined navigation maneuver from each of the multiple autonomous vehicles; analyze the navigation situation information; determine whether the adjustment to the determined navigation maneuver is due to a transient condition based on the analysis of the navigation situation information; and update the predetermined model representing at least one road segment if the adjustment to the determined navigation maneuver is not due to a transient condition.
[0122] In some embodiments of the system, the predetermined model representing at least one road segment may include a three-dimensional spline representing a predetermined path for travel along the at least one road segment. Updating the predetermined model may include updating the three-dimensional spline representing the predetermined path for travel along the at least one road segment. Adjustments to a determined navigation maneuver may be caused by user intervention. Adjustments to a determined navigation maneuver may be caused by automatic determination based on image analysis of the presence of navigation adjustment conditions in the vehicle environment. Navigation situation information may include at least one image representing the environment of the autonomous vehicle. Navigation situation information may include a video representing the environment of the autonomous vehicle. Transient conditions may be associated with parked cars, intervening cars, pedestrians, low light conditions, glare conditions, temporary obstacles, or temporary road construction.
[0123] In some embodiments, a method for interacting with multiple autonomous vehicles may include receiving navigation situation information associated with the occurrence of an adjustment to a determined navigation maneuver from each of the multiple autonomous vehicles; analyzing the navigation situation information; determining whether the adjustment to the determined navigation maneuver is due to a transient condition based on the analysis of the navigation situation information; and updating a predetermined model representing at least one road segment if the adjustment to the determined navigation maneuver is not due to a transient condition.
[0124] In some embodiments of the method, the predetermined model representing at least one road segment may include a three-dimensional spline representing a predetermined path for travel along the at least one road segment. Updating the predetermined model may include updating the three-dimensional spline representing the predetermined path for travel along the at least one road segment. Adjustment of the determined navigation maneuver may be caused by user intervention. Adjustment of the determined navigation maneuver may be caused by automatic determination based on image analysis of the presence of navigation adjustment conditions in the vehicle environment. The navigation situation information may include at least one image representing the environment of the autonomous vehicle. The navigation situation information may include a video representing the environment of the autonomous vehicle. The transient condition may be associated with parked cars, intervening cars, pedestrians, low light conditions, glare conditions, temporary obstacles, or temporary road construction.
[0125] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor to cause the at least one processor to perform a method. The method may include: receiving navigation situation information associated with an occurrence of an adjustment to a determined navigation maneuver from each of a plurality of autonomous vehicles; analyzing the navigation situation information; determining whether the adjustment to the determined navigation maneuver is due to a transient condition based on the analysis of the navigation situation information; and updating a predetermined model representing at least one road segment if the adjustment to the determined navigation maneuver is not due to the transient condition.
[0126] In some embodiments of the computer-readable medium, the predetermined model representing the at least one road segment may include a three-dimensional spline representing a predetermined path for travel along the at least one road segment. Updating the predetermined model may include updating the three-dimensional spline representing the predetermined path for travel along the at least one road segment. The transient condition may be associated with parked vehicles, intervening vehicles, pedestrians, low light conditions, glare conditions, temporary obstructions, or temporary road construction.
[0127] In some embodiments, a system for interacting with multiple autonomous vehicles may include a memory including a predetermined road model representing at least one road segment; and at least one processor programmed to: selectively receive road environment information from the multiple autonomous vehicles based on navigation by the multiple autonomous vehicles through their respective road environments; determine whether one or more updates to the predetermined road model are required based on the road environment information; and update the predetermined road model to include the one or more updates.
[0128] In some embodiments of the system, the road model may include a three-dimensional spline representing a predetermined path for travel along at least one road segment. Selectively receiving road environment information may include limiting the frequency of information transmissions received from specific vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from groups of vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles traveling within a specific geographic area. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles based on a determined model confidence level associated with the specific geographic area. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles to only those transmissions that include potential contradictions with respect to at least one aspect of the predetermined road model.
[0129] In some embodiments, a method for interacting with multiple autonomous vehicles may include: selectively receiving road environment information from the multiple autonomous vehicles based on navigation of the multiple autonomous vehicles through their respective road environments; determining whether one or more updates to a predetermined road model are required based on the road environment information; and updating the predetermined road model to include the one or more updates.
[0130] In some embodiments of the method, the road model may include a three-dimensional spline representing a predetermined path for travel along at least one road segment. Selectively receiving road environment information may include limiting the frequency of information transmissions received from specific vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from groups of vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles traveling within a specific geographic area. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles based on a determined model confidence level associated with the specific geographic area. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles to only those transmissions that include potential contradictions with respect to at least one aspect of the predetermined road model.
[0131] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor to cause the at least one processor to perform a method. The method may include: selectively receiving road environment information from a plurality of autonomous vehicles based on navigation of the plurality of autonomous vehicles through their respective road environments; determining whether one or more updates to a predetermined road model are required based on the road environment information; and updating the predetermined road model to include the one or more updates.
[0132] In some embodiments of the computer-readable medium, the road model may include a three-dimensional spline representing a predetermined path for travel along at least one road segment. Selectively receiving road environment information may include limiting the frequency of information transmissions received from specific vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from groups of vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles traveling within a specific geographic area. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles to only those transmissions that include potential contradictions with respect to at least one aspect of the predetermined road model.
[0133] In some embodiments, a method for processing collected navigation information for autonomous vehicle navigation may include: processing, by a mapping server, navigation information collected from multiple vehicles and obtained by sensors coupled to the multiple vehicles, wherein the navigation information describes lanes of a road segment; collecting data about landmarks identified near the road segment, wherein the landmarks include traffic signs; generating, by the mapping server, an autonomous vehicle map of the road segment, wherein the autonomous vehicle map includes splines corresponding to lanes in the road segment and landmarks identified near the road segment; and distributing, by the mapping server, the autonomous vehicle map to autonomous vehicles for autonomous navigation on the road segment.
[0134] In some embodiments, at least one machine-readable medium may include instructions for processing collected navigation information for use in autonomous vehicle navigation, the instructions, when executed by a mapping server, causing the mapping server to perform operations, the operations comprising: processing navigation information collected from multiple vehicles and obtained by sensors coupled to the multiple vehicles, wherein the navigation information describes lanes of a road segment; collecting data about landmarks identified near the road segment, the landmarks including traffic signs; generating an autonomous vehicle map of the road segment, wherein the autonomous vehicle map includes splines corresponding to lanes in the road segment and landmarks identified near the road segment; and distributing the autonomous vehicle map to autonomous vehicles for autonomous navigation on the road segment.
[0135] In some embodiments, a mapping server is used to process collected navigation information for autonomous vehicle navigation, and the mapping server may include: a processor; and a memory device containing instructions, which, when executed by the processor, cause the processor to: process navigation information collected from multiple vehicles and obtained by sensors coupled to the multiple vehicles, wherein the navigation information describes lanes of a road segment; collect data about landmarks identified near the road segment, wherein the landmarks include traffic signs; generate an autonomous vehicle map of the road segment, wherein the autonomous vehicle map includes splines corresponding to lanes in the road segment and landmarks identified near the road segment; and distribute the autonomous vehicle map to autonomous vehicles for autonomous navigation on the road segment.
[0136] In some embodiments, a vehicle system for using collected navigation information to assist in autonomous vehicle navigation may include: a processor; and a memory device containing instructions that, when executed by the processor, cause the processor to: access navigation information at the vehicle system, the navigation information being derived from sensor data obtained by multiple sensors coupled to the vehicle system; provide the navigation information to a mapping server, the navigation information being processed by the mapping server as crowdsourced navigation information from multiple vehicles obtained by sensors coupled to the multiple vehicles, wherein the navigation information describes lanes of a road segment; and receive an autonomous vehicle map, the autonomous vehicle map including splines corresponding to lanes in the road segment and landmarks identified near the road segment for autonomous navigation on the road segment.
[0137] In some embodiments, at least one machine-readable device may include instructions for using collected navigation information to assist in autonomous vehicle navigation, the instructions, when executed by a vehicle system, causing the vehicle system to perform operations, the operations may include: accessing navigation information at the vehicle system, the navigation information being derived from sensor data obtained by multiple sensors coupled to the vehicle system; providing the navigation information to a mapping server, the navigation information being processed by the mapping server into collected navigation information from multiple vehicles obtained by sensors coupled to the multiple vehicles, wherein the navigation information describes lanes of a road segment; and receiving an autonomous vehicle map, the autonomous vehicle map including splines corresponding to lanes in the road segment and landmarks identified near the road segment for autonomous navigation on the road segment.
[0138] In some embodiments, a vehicle system for assisting autonomous vehicle navigation is installed in a vehicle and may include: a processor; and a memory containing instructions that, when executed by the processor, cause the processor to: access a sparse map generated from collected data, the sparse map generated from the collected data including a spline representation of a lane marking and metadata describing the lane represented by the lane marking, and the sparse map generated from the collected data is created using data obtained from multiple vehicles; obtain image data corresponding to an image from a camera, the image corresponding to an environment outside the vehicle; and analyze the sparse map and image data generated from the collected data to determine an autonomous navigation response of the vehicle.
[0139] In some embodiments, at least one machine-readable medium may include instructions for assisting autonomous vehicle navigation, which, when executed by a machine, causes the machine to perform operations including: accessing a sparse map generated from collected data, the sparse map generated from the collected data including a spline representation of a lane marking and metadata describing the lane represented by the lane marking, and the sparse map generated from the collected data is created using data obtained from multiple vehicles; obtaining image data corresponding to an image from a camera, the image corresponding to an environment outside the vehicle; and analyzing the sparse map and image data generated from the collected data to determine an autonomous navigation response of the vehicle.
[0140] In some embodiments, an autonomous vehicle may include: a body; a camera mounted on the body; and a processing unit: accessing a sparse map generated by collected data, the sparse map generated by the collected data including a spline representation of a lane marking and metadata describing the lane represented by the lane marking, and the sparse map generated by the collected data is created by data obtained from multiple vehicles; obtaining image data corresponding to an image from the camera, the image corresponding to an environment outside the autonomous vehicle; and analyzing the sparse map and image data generated by the collected data to determine an autonomous navigation response of the autonomous vehicle.
[0141] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executed by at least one processing device and perform any of the methods described herein.
[0142] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0143] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:
[0144] Figure 1 is a pictorial representation of an exemplary system consistent with embodiments of the present disclosure.
[0145] Figure 2A is a diagrammatic side view representation of an exemplary vehicle including a system consistent with embodiments of the present disclosure.
[0146] Figure 2B is consistent with the embodiment of the present disclosure Figure 2A Diagrammatic top view representation of the vehicle and systems shown in .
[0147] Figure 2C is a diagrammatic top view representation of another embodiment of a vehicle including a system consistent with embodiments of the present disclosure.
[0148] Figure 2D is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with embodiments of the present disclosure.
[0149] Figure 2E is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with embodiments of the present disclosure.
[0150] Figure 2F is a pictorial representation of an exemplary vehicle control system consistent with an embodiment of the present disclosure.
[0151] Figure 3A is a pictorial representation of the interior of a vehicle including a rearview mirror and a user interface for a vehicle imaging system, consistent with an embodiment of the present disclosure.
[0152] Figure 3B is an illustration of an example of a camera installation configured to be placed behind a rearview mirror and against a vehicle windshield, consistent with embodiments of the present disclosure.
[0153] Figure 3C It is consistent with the embodiment of the present disclosure and is viewed from different perspectives. Figure 3B An illustration of a camera installation is shown in .
[0154] Figure 3D is an illustration of an example of a camera installation configured to be placed behind a rearview mirror and against a vehicle windshield, consistent with embodiments of the present disclosure.
[0155] Figure 4 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations consistent with embodiments of the present disclosure.
[0156] Figure 5A is a flow chart illustrating an exemplary process for eliciting one or more navigation responses based on monocular image analysis, consistent with embodiments of the present disclosure.
[0157] Figure 5B is a flow chart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a collection of images, consistent with embodiments of the present disclosure.
[0158] Figure 5C is a flow chart illustrating an exemplary process for detecting road markings and / or lane geometry information in a collection of images, consistent with embodiments of the present disclosure.
[0159] Figure 5D is a flow chart illustrating an exemplary process for detecting traffic lights in a collection of images, consistent with embodiments of the present disclosure.
[0160] Figure 5E is a flow chart illustrating an exemplary process for eliciting one or more navigation responses based on a vehicle path, consistent with an embodiment of the present disclosure.
[0161] Figure 5F is a flow chart illustrating an exemplary process for detecting whether a leading vehicle changes lanes, consistent with an embodiment of the present disclosure.
[0162] Figure 6 is a flow chart illustrating an exemplary process for eliciting one or more navigation responses based on stereo image analysis, consistent with embodiments of the present disclosure.
[0163] Figure 7 is a flow chart illustrating an exemplary process for eliciting one or more navigational responses based on analysis of three sets of images, consistent with embodiments of the present disclosure.
[0164] Figure 8 A sparse map for providing autonomous vehicle navigation consistent with an embodiment of the present disclosure is shown.
[0165] Figure 9A A polynomial representation of a partial road segment consistent with an embodiment of the present disclosure is shown.
[0166] Figure 9B A curve in three-dimensional space representing a target trajectory of a vehicle for a specific road segment included in a sparse map is shown, consistent with an embodiment of the present disclosure.
[0167] Figure 10 Illustrative landmarks that may be included in a sparse map consistent with embodiments of the present disclosure are shown.
[0168] Figure 11A A polynomial representation of a trajectory consistent with an embodiment of the present disclosure is shown.
[0169] Figure 11B and Figure 11CA target trajectory along a multi-lane road consistent with an embodiment of the present disclosure is shown.
[0170] Figure 11D An exemplary road signature profile consistent with an embodiment of the present disclosure is shown.
[0171] Figure 12 is a schematic diagram of a system for receiving crowdsourced data from multiple vehicles for autonomous vehicle navigation, consistent with an embodiment of the present disclosure.
[0172] Figure 13 An exemplary autonomous vehicle road navigation model represented by a plurality of three-dimensional splines is shown, consistent with an embodiment of the present disclosure.
[0173] Figure 14 A block diagram illustrating a server consistent with an embodiment of the present disclosure is shown.
[0174] Figure 15 A block diagram illustrating a memory consistent with embodiments of the present disclosure is shown.
[0175] Figure 16 A process of clustering vehicle trajectories associated with a vehicle consistent with an embodiment of the present disclosure is shown.
[0176] Figure 17 A navigation system that can be used for autonomous navigation consistent with an embodiment of the present disclosure is shown.
[0177] Figure 18 is a flow chart illustrating an exemplary process for processing vehicle navigation information for use in autonomous vehicle navigation, consistent with an embodiment of the present disclosure.
[0178] Figure 19 is a flow chart illustrating an exemplary process performed by a navigation system of a vehicle consistent with an embodiment of the present disclosure.
[0179] Figure 20 An exemplary diagram illustrating a memory consistent with embodiments of the present disclosure.
[0180] Figure 21 A flow chart of uploading recommended trajectories to a server consistent with an embodiment of the present disclosure is shown.
[0181] Figure 22 An exemplary environment including a system for identifying landmarks for use in an autonomous vehicle, consistent with embodiments of the present disclosure, is shown.
[0182] Figure 23 An exemplary environment including a system for identifying landmarks for use in an autonomous vehicle, consistent with embodiments of the present disclosure, is shown.
[0183] Figure 24A method of determining a condensed signature representation of a landmark consistent with embodiments of the present disclosure is shown.
[0184] Figure 25 Another method of determining a condensed signature representation of a landmark, consistent with embodiments of the present disclosure, is shown.
[0185] Figure 26 An exemplary block diagram of a memory consistent with embodiments of the present disclosure is shown.
[0186] Figure 27 is a flow chart illustrating an exemplary process for determining an identifier of a landmark consistent with embodiments of the present disclosure.
[0187] Figure 28 is a flow chart illustrating an exemplary process for updating and distributing a vehicle road navigation model consistent with an embodiment of the present disclosure.
[0188] Figure 29 An exemplary block diagram of a system for determining the positions of landmarks for use in navigation of an autonomous vehicle, consistent with embodiments of the present disclosure, is shown.
[0189] Figure 30 An exemplary block diagram of a memory consistent with embodiments of the present disclosure is shown.
[0190] Figure 31 An exemplary scaling method for determining a distance from a vehicle to a landmark is shown, consistent with an embodiment of the present disclosure.
[0191] Figure 32 An exemplary optical flow for determining the distance from a vehicle to a landmark is shown, consistent with an embodiment of the present disclosure.
[0192] Figure 33A A flow chart illustrating an exemplary process for determining the location of a landmark for use in navigation of an autonomous vehicle, consistent with an embodiment of the present disclosure.
[0193] Figure 33B A flow chart illustrating an exemplary process for measuring the position of a landmark for use in navigation of an autonomous vehicle, consistent with an embodiment of the present disclosure.
[0194] Figure 34 is a diagrammatic top view representation of an exemplary vehicle including a system in which the vehicle navigates using landmarks consistent with an embodiment of the present disclosure.
[0195] Figure 35 is another illustrative top view representation of an exemplary vehicle including a system in which the vehicle navigates using landmarks consistent with embodiments of the present disclosure.
[0196] Figure 36is a flow chart illustrating a flowchart of an exemplary process for navigating an exemplary vehicle using landmarks.
[0197] Figure 37 is a diagrammatic top view representation of an exemplary autonomous vehicle including a system in which the autonomous vehicle navigates using tail alignment, consistent with embodiments of the present disclosure.
[0198] Figure 38 is another illustrative top view representation of an exemplary autonomous vehicle including a system in which the autonomous vehicle navigates using tail alignment, consistent with embodiments of the present disclosure.
[0199] Figure 39 is a flow chart illustrating an exemplary process for navigating an exemplary autonomous vehicle using tail alignment.
[0200] Figure 40 is a pictorial top view representation of an exemplary vehicle including a system in which the vehicle navigates a road junction using two or more landmarks, consistent with embodiments of the present disclosure.
[0201] Figure 41 is a flow chart illustrating an exemplary process for navigating an exemplary vehicle at a road junction using two or more landmarks.
[0202] Figure 42 is a pictorial top view representation of an exemplary vehicle including a system in which the vehicle navigates using an overlaid map, consistent with embodiments of the present disclosure.
[0203] Figure 43A 、 Figure 43B ,and Figure 43C A flow chart illustrating an exemplary process for navigating an exemplary vehicle using overlaid maps is shown.
[0204] Figure 44 An exemplary remote server in a communication vehicle is shown, consistent with embodiments of the present disclosure.
[0205] Figure 45 A vehicle is shown navigating along a multi-lane road consistent with an embodiment of the present disclosure.
[0206] Figure 46 A vehicle is shown navigating using a target trajectory along a multi-lane road, consistent with an embodiment of the present disclosure.
[0207] Figure 47 An example of a road signature profile consistent with an embodiment of the present disclosure is shown.
[0208] Figure 48 An exemplary environment consistent with embodiments of the present disclosure is shown.
[0209] Figure 49 is a flow chart illustrating an exemplary process for sparse map autonomous vehicle navigation consistent with an embodiment of the present disclosure.
[0210] Figure 50 An exemplary environment for autonomous navigation based on desired landmark locations, consistent with embodiments of the present disclosure, is shown.
[0211] Figure 51 A configuration for autonomous navigation consistent with an embodiment of the present disclosure is shown.
[0212] Figure 52 Another exemplary environment for autonomous navigation based on desired landmark locations, consistent with embodiments of the present disclosure, is shown.
[0213] Figure 53 Another exemplary environment for autonomous navigation based on desired landmark locations, consistent with embodiments of the present disclosure, is shown.
[0214] Figure 54 is a flow chart illustrating an exemplary process for autonomous navigation based on desired landmark locations, consistent with embodiments of the present disclosure.
[0215] Figure 55 is a pictorial representation of an exemplary vehicle control system consistent with an embodiment of the present disclosure.
[0216] Figure 56 is a pictorial top view representation of an exemplary vehicle including a system in which the vehicle navigates using lane width profiles and road width profiles consistent with an embodiment of the present disclosure.
[0217] Figure 57 is a diagram illustrating an exemplary profile that may be used by a vehicle control system consistent with embodiments of the present disclosure.
[0218] Figure 58 is a diagrammatic top view representation of an exemplary vehicle including a system in which the vehicle navigates using the length or spacing of road markings, consistent with embodiments of the present disclosure.
[0219] Figure 59 is a pictorial top view representation of an exemplary vehicle including a system in which the vehicle navigates using information regarding the curvature of a road segment, consistent with embodiments of the present disclosure.
[0220] Figure 60 is a flow chart illustrating an exemplary process for navigating an exemplary vehicle using road signatures.
[0221] Figure 61A is a diagrammatic side view representation of an exemplary vehicle consistent with embodiments of the present disclosure.
[0222] Figure 61B is a diagrammatic side view representation of an exemplary vehicle consistent with embodiments of the present disclosure.
[0223] Figure 62 is a pictorial top-view representation of an exemplary vehicle autonomously navigating a road, consistent with an embodiment of the present disclosure.
[0224] Figure 63 is a flow chart illustrating an exemplary process for autonomous navigation consistent with embodiments of the present disclosure.
[0225] Figure 64 is an illustrative perspective view of an environment captured by a forward-facing image capture device on an exemplary vehicle consistent with embodiments of the present disclosure.
[0226] Figure 65 is an exemplary image received from a forward-facing image capture device of a vehicle consistent with an embodiment of the present disclosure.
[0227] Figure 66 is a flow chart illustrating an exemplary process for navigating a vehicle by determining free space zones in which the vehicle can travel, consistent with the disclosed embodiments.
[0228] Figure 67 is a pictorial top view representation of an exemplary vehicle navigating a road having snow covering at least some lane markings and road edges, consistent with an embodiment of the present disclosure.
[0229] Figure 68 is a flow chart of an exemplary process for navigating a vehicle on a road where snow covers at least some lane markings and road edges, consistent with an embodiment of the present disclosure.
[0230] Figure 69 is a pictorial top view representation of an exemplary vehicle including a method for calibrating the vehicle's velocity, consistent with an embodiment of the present disclosure.
[0231] Figure 70 is a flow chart illustrating an exemplary process for calibrating a vehicle's velocity, consistent with an embodiment of the present disclosure.
[0232] Figure 71 is another illustrative top view representation of an exemplary vehicle including a method for calibrating vehicle velocity, consistent with an embodiment of the present disclosure.
[0233] Figure 72 is a flow chart illustrating another exemplary process for calibrating a vehicle's velocity, consistent with embodiments of the present disclosure.
[0234] Figure 73 is an illustration of a street view of an exemplary road segment, consistent with the disclosed embodiments.
[0235] Figure 74 is a bird's-eye view of an exemplary road segment consistent with an embodiment of the present disclosure.
[0236] Figure 75 is a flow chart illustrating an exemplary process for determining a lane assignment for a vehicle, consistent with an embodiment of the present disclosure.
[0237] Figure 76 is an illustration of a street view of an exemplary road segment, consistent with the disclosed embodiments.
[0238] Figure 77A is an illustration of a bird's-eye view of an exemplary road segment, consistent with the disclosed embodiments.
[0239] Figure 77B is a street view of an exemplary road segment consistent with an embodiment of the present disclosure.
[0240] Figure 78 is a flow chart illustrating an exemplary process for autonomously navigating a vehicle along a road segment, consistent with an embodiment of the present disclosure.
[0241] Figure 79A A plan view of a vehicle traveling on a road approaching wintry and icy road conditions at a particular location is shown, consistent with the disclosed embodiments.
[0242] Figure 79B A plan view of a vehicle traveling on a road approaching pedestrians is shown, consistent with an embodiment of the present disclosure.
[0243] Figure 79C A plan view of a vehicle traveling on a road in close proximity to another vehicle is shown, consistent with an embodiment of the present disclosure.
[0244] Figure 79D Plan view showing a vehicle traveling on a road in an ending lane consistent with the disclosed embodiments
[0245] Figure 80 A diagrammatic side view representation of an exemplary vehicle including a system consistent with embodiments of the present disclosure is shown.
[0246] Figure 81 An exemplary flow chart representing a method for adaptive navigation of a vehicle based on user intervention, consistent with an embodiment of the present disclosure, is shown.
[0247] Figure 82A A plan view of a vehicle traveling on a road with parked cars is shown, consistent with an embodiment of the present disclosure.
[0248] Figure 82BA plan view of a vehicle traveling on a road in an ending lane is shown, consistent with an embodiment of the present disclosure.
[0249] Figure 82C A plan view of a vehicle traveling on a road approaching pedestrians is shown, consistent with an embodiment of the present disclosure.
[0250] Figure 82D A plan view of a vehicle traveling on a road in an area approaching a construction site is shown, consistent with an embodiment of the present disclosure.
[0251] Figure 83 An exemplary flow chart representing a method for self-aware navigation of a vehicle, consistent with an embodiment of the present disclosure, is shown.
[0252] Figure 84A A plan view of a vehicle traveling on a road with multiple parked cars is shown, consistent with an embodiment of the present disclosure.
[0253] Figure 84B A plan view of a vehicle traveling on a road with a car cutting directly in front of the vehicle is shown, consistent with an embodiment of the present disclosure.
[0254] Figure 84C A plan view of a vehicle traveling on a road with a temporary obstacle directly in front of the vehicle is shown, consistent with an embodiment of the present disclosure.
[0255] Figure 84D A plan view of a vehicle traveling on a road with a temporary construction site directly in front of the vehicle is shown, consistent with an embodiment of the present disclosure.
[0256] Figure 85A A plan view of a vehicle traveling on a road surface having a pothole directly in front of the vehicle is shown, consistent with an embodiment of the present disclosure.
[0257] Figure 85B A plan view of a vehicle traveling on a road with an animal and a pedestrian straddling in front of the vehicle is shown, consistent with an embodiment of the present disclosure.
[0258] Figure 86 An exemplary flow chart representing a method for an adaptive road model manager consistent with an embodiment of the present disclosure is shown.
[0259] Figure 87A A plan view of a single vehicle traveling on a road surface is shown, consistent with an embodiment of the present disclosure.
[0260] Figure 87B A plan view of a group of vehicles traveling on a city road is shown, consistent with an embodiment of the present disclosure.
[0261] Figure 87C A plan view of a vehicle traveling on a road within a particular rural geographic area is shown, consistent with an embodiment of the present disclosure.
[0262] Figure 87D A plan view of a vehicle traveling on a road surface with a lane change is shown, consistent with an embodiment of the present disclosure.
[0263] Figure 88 An exemplary flow chart illustrating a method for selective feedback-based road model management consistent with an embodiment of the present disclosure is shown. Specific embodiments
[0264] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numerals used in the drawings and the following description refer to the same or similar parts. Although several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, the components shown in the drawings may be replaced, added to, or modified, and the illustrative methods described herein may be modified by replacing, reordering, removing, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the embodiments and examples of the present disclosure. Instead, the appropriate scope is defined by the appended claims.
[0265] Autonomous Vehicle Overview
[0266] As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle that is capable of implementing at least one navigation change without driver input. A "navigation change" refers to one or more of the steering, braking, or acceleration of the vehicle. By autonomous, it is meant that the vehicle need not be fully automatic (e.g., fully operating without a driver or without driver input). Rather, autonomous vehicles include those vehicles that are capable of operating under driver control during certain time periods and without driver control during other time periods. Autonomous vehicles may also include vehicles that control only some aspects of vehicle navigation, such as steering (e.g., maintaining the vehicle's route between vehicle lane limits), but may leave other aspects to the driver (e.g., braking). In some cases, an autonomous vehicle may handle some or all aspects of the vehicle's braking, rate control, and / or steering.
[0267] Because human drivers typically rely on visual cues and observation sequences to control vehicles, transportation infrastructure has been built with lane markings, traffic signs, and traffic lights, all designed to provide visual information to the driver. Given these design characteristics of the transportation infrastructure, autonomous vehicles can include cameras and processing units that analyze visual information captured from the vehicle's environment. Visual information can include, for example, components of the transportation infrastructure observed by the driver (e.g., lane markings, traffic signs, traffic lights, etc.) and other obstacles (e.g., other vehicles, pedestrians, rocks, etc.). Autonomous vehicles can also use stored information, such as information that provides a model of the vehicle's environment when navigating. For example, a vehicle can use GPS data, sensor data (e.g., from accelerometers, velocity sensors, suspension sensors, etc.), and / or other map data to provide information about its environment while the vehicle is traveling, and the vehicle (and other vehicles) can use this information to localize itself on the model.
[0268] In some embodiments of the present disclosure, an autonomous vehicle may use information obtained while navigating (e.g., from a camera, GPS device, accelerometer, rate sensor, suspension sensor, etc.). In other embodiments, an autonomous vehicle may use information obtained by the vehicle (or by other vehicles) from past navigations while navigating. In yet another embodiment, an autonomous vehicle may use a combination of information obtained while navigating and information obtained from past navigations. The following section provides an overview of a system consistent with an embodiment of the present disclosure, followed by an overview of a forward imaging system and methods consistent with the system. The following section discloses systems and methods for constructing, using, and updating a sparse map for autonomous vehicle navigation.
[0269] System Overview
[0270] Figure 1is a block diagram representing a system 100 consistent with an exemplary disclosed embodiment. System 100 may include various components, as required by a particular implementation. In some embodiments, system 100 may include a processing unit 110, an image acquisition unit 120, a location sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. Processing unit 110 may include one or more processing devices. In some embodiments, processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, image acquisition unit 120 may include any number of image acquisition devices and components, as required by a particular application. In some embodiments, image acquisition unit 120 may include one or more image capture devices (e.g., cameras), such as image capture device 122, image capture device 124, and image capture device 126. System 100 may also include a data interface 128 that communicatively connects processing device 110 to image acquisition device 120. For example, the data interface 128 may include any wired and / or wireless link or links for transmitting image data acquired by the image acquisition device 120 to the processing unit 110 .
[0271] The wireless transceiver 172 may include one or more devices configured to exchange transmissions over an air interface to one or more networks (e.g., cellular, Internet, etc.) using radio frequencies, infrared frequencies, magnetic fields, or electric fields. The wireless transceiver 172 may transmit and / or receive data using any well-known standard (e.g., Wi-Fi, Bluetooth Smart, 802.15.4, ZigBee, etc.)
[0272] Both the application processor 180 and the image processor 190 may include various types of processing devices. For example, either or both of the application processor 180 and the image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, memory, or any other type of device suitable for running applications and for image processing and analysis. In some embodiments, the application processor 180 and / or the image processor 190 may include any type of single-core or multi-core processor, a mobile device microcontroller, a central processing unit, and the like. Various processing devices may be used, including, for example, processors that can be provided to the image processor such as and the various processing devices may include various architectures (e.g., x86 processors, etc).
[0273] In some embodiments, the application processor 180 and / or the image processor 190 may include a processor that can be Any of the EyeQ series processor chips available. These processors are designed so that each processor includes multiple processing units with local memory and instruction sets. Such a processor may include video input for receiving image data from multiple image sensors and may also include video output capabilities. In one example, Using 90nm-micron technology operating at 332Mhz. The architecture consists of two floating-point, hyperthreaded 32-bit RISC CPUs ( Multi-core), five Vision Computing Engines (VCE), three Vector Microcode Processors ( ), Denali 64-bit mobile DDR controller, 128-bit internal supersonic interconnect, dual 16-bit video input and 18-bit video output controllers, 16-channel DMA and several peripherals. The MIPS34K CPU manages five VCEs, three VMPs TM and DMA, a second MIPS34K CPU and multi-channel DMA and other peripherals. Five VCEs, three VMPs TM and MIPS34K CPUs are capable of performing the intensive visual computations required by multi-function bundled applications. In another example, as a third generation processor and is Six times more powerful can be used in the embodiments of the present disclosure.
[0274] Any of the processing devices disclosed herein can be configured to perform certain functions. Configuring a processing device, such as any of the described EyeQ processors or other controllers or microprocessors, to perform certain functions can include programming computer-executable instructions and making these instructions available to the processing device for execution during operation of the processing device. In some embodiments, configuring the processing device can include programming the processing device directly with architecture instructions. In other embodiments, configuring the processing device can include storing executable instructions in a memory accessible to the processing device during operation. For example, the processing device can access the memory during operation to obtain and execute the stored instructions.
[0275] Although Figure 1While two separate processing devices are depicted as being included in processing unit 110, more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to perform the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Additionally, in some embodiments, system 100 may include one or more of processing units 110 without including other components, such as image acquisition unit 120.
[0276] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices such as a controller, an image preprocessor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, memory, or any other type of device for image processing and analysis. The image preprocessor may include a video processor for capturing, digitizing, and processing images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The support circuits may be any number of circuits commonly known in the art, including cache circuits, power supply circuits, clock circuits, and input / output circuits. The memory may store software that controls the operation of the system when executed by the processor. The memory may include a database and image processing software. The memory may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage, tape storage, removable storage, and other types of storage. In one embodiment, the memory may be separate from the processing unit 110. In another embodiment, the memory may be integrated into the processing unit 110.
[0277] Each memory 140, 150 may include software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may include various databases and image processing software. The memory units may include random access memory, read-only memory, flash memory, a disk drive, optical storage, tape storage, removable storage, and / or any other type of storage. In some embodiments, the memory units 140, 150 may be separate from the application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into the application processor 180 and / or image processor 190.
[0278] Position sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, position sensor 130 may include a GPS receiver. Such a receiver is capable of determining user location and velocity by processing signals broadcast by global positioning system satellites. Position information from position sensor 130 may be provided to application processor 180 and / or image processor 190.
[0279] In some embodiments, system 100 may include components such as a rate sensor (eg, a tachometer) for measuring the rate of vehicle 200 , and / or an accelerometer for measuring the acceleration of vehicle 200 .
[0280] The user interface 170 may include any device suitable for providing information to or receiving input from one or more users of the system 100. In some embodiments, the user interface 170 may include user input devices including, for example, a touch screen, a microphone, a keyboard, a pointing device, a track wheel, a camera, knobs, buttons, etc. Using such input devices, a user may be able to provide information input or commands to the system 100 by typing instructions or information, providing voice commands, selecting menu options on a screen using buttons, a pointer, or eye tracking capabilities, or by any other suitable technique for communicating information to the system 100.
[0281] The user interface 170 may be equipped with one or more processing devices configured to provide information to and receive information from a user and process the information for use by, for example, the application processor 180. In some embodiments, such processing devices may execute instructions for recognizing and tracking eye movements, receiving and interpreting voice commands, recognizing and interpreting touches and gestures made to a touch screen, responding to keyboard entries or menu selections, etc. In some embodiments, the user interface 170 may include a display, a speaker, a haptic device, and / or any other device for providing output information to a user.
[0282] Map database 160 may comprise any type of database for storing map data useful to system 100. In some embodiments, map database 160 may include data relating to the locations of various items in a reference coordinate system, including roads, hydrographic features, geographic features, businesses, points of interest, restaurants, gas stations, and the like. Map database 160 may store not only the locations of such items, but also descriptors associated with those items, including, for example, the names associated with any stored features. In some embodiments, map database 160 may be physically located with the other components of system 100. Alternatively or additionally, map database 160 or a portion thereof may be remotely located relative to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 may be downloaded via a wired or wireless data connection to a network (e.g., via a cellular network and / or the Internet, etc.).
[0283] Image capture devices 122, 124, and 126 may each comprise any type of device suitable for capturing at least one image from an environment. Furthermore, any number of image capture devices may be used to acquire images for input to an image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four or more image capture devices. Image capture devices 122, 124, and 126 will also be referred to below. Figure 2B-2E To describe.
[0284] The system 100, or its various components, may be incorporated into a variety of different platforms. In some embodiments, the system 100 may be included on a vehicle 200, such as Figure 2A For example, as shown above with respect to Figure 1 As described above, the vehicle 200 may be equipped with the processing unit 110 and any of the other components of the system 100. Although in some embodiments, the vehicle 200 may be equipped with only a single image capture device (e.g., a camera), in other embodiments, such as a combination of Figure 2B-2E For those embodiments discussed, multiple image capture devices may be used. For example, Figure 2A As shown in FIG, one of the image capture device 122 and the image capture device 124 of the vehicle 200 may be part of an ADAS (Advanced Driver Assistance System) imaging setup.
[0285] The image capture device included on the vehicle 200 as part of the image acquisition unit 120 may be placed in any suitable location. Figure 2A-2E ,as well as Figure 3A-3C As shown in , the image capture device 122 can be located near the rearview mirror. This location can provide a line of sight similar to that of the driver of the vehicle 200, which can assist in determining what is visible and what is not visible to the driver. The image capture device 122 can be placed anywhere near the rearview mirror, but placing the image capture device 122 on the driver's side of the mirror can also help obtain an image representative of the driver's field of view and / or line of sight.
[0286] Other positions of the image capture device for image acquisition unit 120 can also be used. For example, image capture device 124 can be located on or in the bumper of vehicle 200. Such a position can be particularly suitable for image capture devices with a wide field of view. The line of sight of the image capture device located at the bumper can be different from the line of sight of the driver, so the bumper image capture device and the driver may not always see the same object. Image capture devices (e.g., image capture devices 122, 124, and 126) can also be located at other positions. For example, the image capture device can be located on or in a side mirror or both side mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on the side of vehicle 200, can be installed on any window of vehicle 200, be placed behind any window of vehicle 200, or be placed in front of any window of vehicle 200, and can be installed in or near the light pattern on the front and / or rear of vehicle 200.
[0287] In addition to the image capture device, the vehicle 200 may include various other components of the system 100. For example, the processing unit 110 may be included on the vehicle 200, either integrated with or separate from the vehicle's engine control unit (ECU). The vehicle 200 may also be equipped with a location sensor 130, such as a GPS receiver, and may also include a map database 160 and memory units 140 and 150.
[0288] As discussed earlier, wireless transceiver 172 can transmit and / or receive data over one or more networks (e.g., a cellular network, the Internet, etc.). For example, wireless transceiver 172 can upload data collected by system 100 to one or more servers and download data from one or more servers. System 100 can receive, via wireless transceiver 172, for example, periodic or on-demand updates to data stored in map database 160, memory 140, and / or storage 150. Similarly, wireless transceiver 172 can upload any data received by system 100 (e.g., images captured by image acquisition unit 120, data received by position sensor 130 or other sensors, vehicle control systems, etc.) and / or any data processed by processing unit 110 to one or more servers.
[0289] The system 100 can upload data to a server (e.g., to the cloud) based on a privacy level setting. For example, the system 100 can implement a privacy level setting to specify or limit the type of data (including metadata) sent to the server that can uniquely identify the vehicle and / or the driver or owner of the vehicle. Such settings can be set by a user via, for example, the wireless transceiver 172, can be initialized by factory default settings, or by data received by the wireless transceiver 172.
[0290] In some embodiments, the system 100 may upload data according to a "high" privacy level, and under this setting, the system 100 may send data without any details about a specific vehicle and / or driver / owner (e.g., location information related to the route, captured images, etc.). For example, when uploading data according to the "high" privacy setting, the system 100 may not include the vehicle identification number (VIN) or the name of the vehicle driver or owner, and may instead send data such as captured images and / or location information related to the route.
[0291] Other privacy levels may be considered. For example, the system 100 may send data to the server according to a "medium" privacy level and may include additional information that is not included at a "high" privacy level, such as the model and / or make of the vehicle, and / or the type of vehicle (e.g., passenger vehicle, sport utility vehicle, truck, etc.). In some embodiments, the system 100 may upload data according to a "low" privacy level. At the "low" privacy level setting, the system 100 uploads data and includes information sufficient to uniquely identify a particular vehicle, owner / driver, and / or a portion or the entire route traveled by the vehicle. Such "low" privacy level data may include at least one of the following: for example, the VIN, the driver / owner name, the vehicle's point of origin prior to departure, the vehicle's desired destination, the vehicle's make and / or model, the vehicle's type, and the like.
[0292] Figure 2A is a diagrammatic side view representation of an exemplary vehicle imaging system consistent with an embodiment of the present disclosure. Figure 2B Yes Figure 2A A diagrammatic top view illustration of the embodiment shown in FIG. Figure 2B As shown in , the disclosed embodiments may include a vehicle 200 including a system 100 in a body of the vehicle 200, the system 100 having a first image capture device 122 positioned near a rearview mirror and / or near a driver of the vehicle 200, a second image capture device 124 positioned on or in a bumper area (e.g., one of the bumper areas 210) of the vehicle 200, and a processing unit 110.
[0293] like Figure 2C As shown in FIG, image capture device 122 and image capture device 124 may both be placed near a rearview mirror of vehicle 200 and / or near the driver. Additionally, although Figure 2B and Figure 2C Two image capture devices are shown, image capture device 122 and image capture device 124, but it should be understood that other embodiments may include more than two image capture devices. Figure 2D and Figure 2E In the embodiment shown in , first, second, and third image capture devices 122 , 124 , and 126 are included in the system 100 of a vehicle 200 .
[0294] like Figure 2D As shown in , image capture device 122 may be placed near a rearview mirror of vehicle 200 and / or near the driver, and image capture device 124 and image capture device 126 may be placed on or in a bumper area (e.g., one of bumper areas 210) of the vehicle. Figure 2EAs shown in FIG, image capture devices 122, 124, and 126 may be placed near the rearview mirror and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, and the image capture devices may be placed in any suitable location within and / or on vehicle 200.
[0295] It will be understood that the disclosed embodiments are not limited to vehicles and can be applied in other contexts. It will also be understood that the disclosed embodiments are not limited to a specific type of vehicle 200 and are applicable to all types of vehicles including cars, trucks, trailers, and other types of vehicles.
[0296] The first image capture device 122 may include any suitable type of image capture device. The image capture device 122 may include an optical axis. In one instance, the image capture device 122 may include an AptinaM9V024 WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280×960 pixels and may include a rolling shutter. The image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included, for example, to provide a desired focal length and field of view for the image capture device. In some embodiments, the image capture device 122 may be associated with a 6 mm lens or a 12 mm lens. In some embodiments, the image capture device 122 may be configured to capture an image having a desired field-of-view (FOV) 202, such as Figure 2DAs shown. For example, the image capture device 122 can be configured to have a conventional FOV, such as within the range of 40 degrees to 56 degrees, including a 46-degree FOV, a 50-degree FOV, a 52-degree FOV, or a larger FOV. Alternatively, the image capture device 122 can be configured to have a narrow FOV within the range of 23 degrees to 40 degrees, such as a 28-degree FOV, or a 36-degree FOV. In addition, the image capture device 122 can be configured to have a wide FOV within the range of 100 degrees to 180 degrees. In some embodiments, the image capture device 122 may include a wide-angle bumper camera, or a camera with a FOV of up to 180 degrees. In some embodiments, the image capture device 122 may be a 7.2M pixel image capture device with an aspect ratio of approximately 2:1 (e.g., H×V=3800×1900 pixels) and a horizontal FOV of approximately 100 degrees. Such an image capture device can be used in a three-image capture device configuration. Due to severe lens distortion, in embodiments where the image capture device uses a radially symmetric lens, the vertical FOV of such an image capture device may be significantly less than 50 degrees. For example, such a lens may not be radially symmetric, which may allow a vertical FOV greater than 50 degrees with a 100 degree horizontal FOV.
[0297] The first image capture device 122 may acquire a plurality of first images relative to the scene associated with 200. Each of the plurality of first images may be acquired as a series of image scan lines, which may be captured using a rolling shutter. Each scan line may include a plurality of pixels.
[0298] The first image capture device 122 may have a scanning frequency associated with the acquisition of each of the first series of image scan lines. The scanning frequency may refer to a frequency at which the image sensor can acquire image data associated with each pixel included in a particular scan line.
[0299] Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be employed in conjunction with a rolling shutter, thereby reading each pixel in a row at a time, and scanning the rows on a row-by-row basis until the entire image frame has been captured. In some embodiments, the rows may be captured sequentially from top to bottom relative to the frame.
[0300] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute a high-resolution imager and may have a resolution greater than 5 M pixels, 7 M pixels, 10 M pixels, or greater.
[0301] The use of a rolling shutter can cause pixels in different rows to be exposed and captured at different times, which can cause skew and other image artifacts in the captured image frame. On the other hand, when image capture device 122 is configured to operate with a global shutter or synchronized shutter, all pixels can be exposed for the same amount of time and during a common exposure period. As a result, the image data in a frame collected from a system employing a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a specific time. In contrast, in a rolling shutter application, each row in the frame is exposed at a different time, and the data is captured at different times. As a result, moving objects can appear distorted in an image capture device with a rolling shutter. This phenomenon is described in more detail below.
[0302] Second image capture device 124 and third image capture device 126 can be any type of image capture device. Similar to first image capture device 122, each of image capture device 124 and image capture device 126 can include an optical axis. In one embodiment, each of image capture device 124 and image capture device 126 can include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of image capture device 124 and image capture device 126 can include a rolling shutter. Similar to first image capture device 122, image capture device 124 and image capture device 126 can be configured to include various lenses and optical elements. In some embodiments, the lenses associated with image capture device 124 and image capture device 126 can provide a FOV (such as FOV 204 and FOV 206) that is the same as or narrower than the FOV associated with image capture device 122 (such as FOV 202). For example, image capture device 124 and image capture device 126 may have a FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.
[0303] Image capture device 124 and image capture device 126 may capture a plurality of second images and a plurality of third images relative to a scene associated with vehicle 200. Each of the plurality of second images and the plurality of third images may be captured as a second series of image scan lines and a third series of image scan lines, which may be captured using a rolling shutter. Each scan line or row may have a plurality of pixels. Image capture device 124 and image capture device 126 may have a second scanning frequency and a third scanning frequency associated with capturing each of the image scan lines included in the second series and the third series.
[0304] Image capture devices 122, 124, and 126 can each be placed at any suitable location and orientation relative to vehicle 200. The relative locations of image capture devices 122, 124, and 126 can be selected to facilitate fusing information acquired from the image capture devices. For example, in some embodiments, the FOV associated with image capture device 124 (such as FOV 204) can partially or completely overlap with the FOV associated with image capture device 122 (such as FOV 202) and the FOV associated with image capture device 126 (such as FOV 206).
[0305] Image capture devices 122, 124, and 126 may be located at any suitable relative height above vehicle 200. In one example, there may be height differences between image capture devices 122, 124, and 126 that provide sufficient parallax information to enable stereo analysis. Figure 2A As shown in FIG, two image capture devices, image capture device 122 and image capture device 124, are at different heights. There may also be lateral displacement differences between image capture devices 122, 124, and 126, providing additional parallax information for, for example, stereo analysis by processing unit 110. The difference in lateral displacement may be represented by dx, as Figure 2C and Figure 2D In some embodiments, there can be a fore-and-aft displacement (e.g., a range displacement) between image capture devices 122, 124, and 126. For example, image capture device 122 can be positioned 0.5 meters to 2 meters or more behind image capture device 124 and / or image capture device 126. This type of displacement can enable one image capture device to cover a potential blind spot of the other image capture device(s).
[0306] Image capture device 122 may have any suitable resolution capability (e.g., the number of pixels associated with the image sensor), and the resolution of the image sensor(s) associated with image capture device 122 may be higher, lower, or the same as the resolution of the image sensor(s) associated with image capture device 124 and image capture device 126. In some embodiments, the image sensor(s) associated with image capture device 122 and / or image capture device 124 and image capture device 126 may have a resolution of 640×480, 1024×768, 1280×960, or other suitable resolution.
[0307] The frame rate (e.g., the frequency at which an image capture device acquires a set of pixel data for one image frame before continuing to capture pixel data associated with the next image frame) can be controllable. The frame rate associated with image capture device 122 can be higher, lower, or the same as the frame rate associated with image capture device 124 and image capture device 126. The frame rate associated with image capture devices 122, 124, and 126 can depend on various factors that can affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 can include a selectable pixel delay period that is applied before or after the acquisition of image data associated with one or more pixels of the image sensor in image capture devices 122, 124, and / or 126. Generally, image data corresponding to each pixel can be acquired according to a clock frequency for the device (e.g., one pixel per clock cycle). Additionally, in embodiments including a rolling shutter, one or more of image capture devices 122, 124, and 126 may include a selectable horizontal blanking period applied before or after the acquisition of image data associated with a row of pixels of an image sensor in image capture devices 122, 124, and / or 126. Additionally, one or more of image capture devices 122, 124, and / or 126 may include a vertical blanking period applied before or after the acquisition of image data associated with an image frame in image capture devices 122, 124, and 126.
[0308] These timing controls may enable synchronization of the frame rates associated with image capture devices 122, 124, and 126, even when the line scan frequency of each image capture device is different. Additionally, as will be discussed in greater detail below, these selectable timing controls, along with other factors (e.g., image sensor resolution, maximum line scan frequency, etc.), may enable synchronization of image capture from areas where the FOV of image capture device 122 overlaps with one or more of the FOVs of image capture device 124 and image capture device 126, even when the field of view of image capture device 122 is different from the FOVs of image capture device 124 and image capture device 126.
[0309] The frame rate timing in image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensors. For example, assuming the line scan frequencies of the two devices are similar, if one device includes an image sensor with a resolution of 640×480 and the other device includes an image sensor with a resolution of 1280×960, then obtaining a frame of image data from the sensor with the higher resolution will require more time.
[0310] Another factor that can influence the timing of image data acquisition in image capture devices 122, 124, and 126 is the maximum line scan frequency. For example, the acquisition of a line of image data from the image sensors included in image capture devices 122, 124, and 126 will require a certain minimum amount of time. Assuming no pixel latency is added, this minimum amount of time for the acquisition of a line of image data will be related to the maximum line scan frequency of the particular device. Devices offering higher maximum line scan frequencies have the potential to provide higher frame rates than devices with lower maximum line scan frequencies. In some embodiments, one or more of image capture device 124 and image capture device 126 may have a maximum line scan frequency that is higher than the maximum line scan frequency associated with image capture device 122. In some embodiments, the maximum line scan frequency of image capture device 124 and / or image capture device 126 may be 1.25 times, 1.5 times, 1.75 times, 2 times, or more times the maximum line scan frequency of image capture device 122.
[0311] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan frequency, but image capture device 122 may operate at a scan frequency that is less than or equal to its maximum scan frequency. The system may be configured such that one or more of image capture device 124 and image capture device 126 operates at a line scan frequency that is equal to the line scan frequency of image capture device 122. In other examples, the system may be configured such that the line scan frequency of image capture device 124 and / or image capture device 126 may be 1.25 times, 1.5 times, 1.75 times, or 2 times, or more, greater than that of image capture device 122.
[0312] In some embodiments, image capture devices 122, 124, and 126 may be asymmetric. That is, they may include cameras with different fields of view (FOVs) and focal lengths. The fields of view of image capture devices 122, 124, and 126 may include, for example, any desired area related to the environment of vehicle 200. In some embodiments, one or more of image capture devices 122, 124, and 126 may be configured to capture image data from the front of vehicle 200, the rear of vehicle 200, the sides of vehicle 200, or a combination thereof.
[0313] In addition, the focal length associated with each image capture device 122, 124, and / or 126 can be selectable (e.g., in accordance with the inclusion of an appropriate lens, etc.) so that each device captures images of objects at a desired range of distances relative to the vehicle 200. For example, in some embodiments, the image capture devices 122, 124, and 126 can capture images of close-up objects within a few meters from the vehicle. The image capture devices 122, 124, and 126 can also be configured to capture images of objects at greater distances from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more). Additionally, the focal lengths of image capture devices 122, 124, and 126 may be selected such that one image capture device (e.g., image capture device 122) is capable of capturing images of objects that are relatively close to the vehicle (e.g., within 10 m or 20 m), while other image capture devices (e.g., image capture device 124 and image capture device 126) are capable of capturing images of objects that are farther away from vehicle 200 (e.g., greater than 20 m, 50 m, 100 m, 150 m, etc.).
[0314] According to some embodiments, the FOV of one or more image capture devices 122, 124, and 126 may have a wide angle. For example, having a FOV of 140 degrees may be beneficial, particularly for image capture devices 122, 124, and 126 that may be used to capture images of an area near vehicle 200. For example, an image capture device may be used to capture images of an area to the right or left of vehicle 200, and in such embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).
[0315] The field of view associated with each of image capture devices 122, 124, and 126 may depend on the respective focal length. For example, as the focal length increases, the corresponding field of view decreases.
[0316] Image capture devices 122, 124, and 126 can be configured to have any suitable field of view. In one specific example, image capture device 122 can have a horizontal FOV of 46 degrees, image capture device 124 can have a horizontal FOV of 23 degrees, and image capture device 126 can have a horizontal FOV between 23 and 46 degrees. In another example, image capture device 122 can have a horizontal FOV of 52 degrees, image capture device 124 can have a horizontal FOV of 26 degrees, and image capture device 126 can have a horizontal FOV between 26 and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 can vary between 1.5 and 2.0. In other embodiments, the ratio can vary between 1.25 and 2.25.
[0317] System 100 can be configured such that the field of view of image capture device 122 at least partially or completely overlaps the field of view of image capture device 124 and / or image capture device 126. In some embodiments, system 100 can be configured such that the fields of view of image capture device 124 and image capture device 126, for example, fall within the field of view of image capture device 122 (e.g., are narrower than the field of view of image capture device 122) and share a common center with the field of view of image capture device 122. In other embodiments, image capture devices 122, 124, and 126 can capture adjacent FOVs or can have partial overlap in their FOVs. In some embodiments, the fields of view of image capture devices 122, 124, and 126 can be aligned such that the center of image capture device 124 and / or image capture device 126, with a narrower FOV, is located in the lower half of the field of view of device 122, with a wider FOV.
[0318] Figure 2Fis a pictorial representation of an exemplary vehicle control system consistent with an embodiment of the present disclosure. As indicated in FIG. 2F , vehicle 200 may include a throttle system 220, a braking system 230, and a steering system 240. System 100 may provide input (e.g., control signals) to one or more of throttle system 220, braking system 230, and steering system 240 via one or more data links (e.g., any wired and / or wireless links, or links for transmitting data). For example, based on analysis of images acquired by image capture devices 122, 124, and 126, system 100 may provide control signals to one or more of throttle system 220, braking system 230, and steering system 240 to navigate vehicle 200 (e.g., by causing acceleration, turning, lane change, etc.). In addition, system 100 may receive input indicating the operating condition of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or turning, etc.) from one or more of throttle system 220, braking system 230, and steering system 240. The following will be combined with Figure 4-Figure 7 Provide further details.
[0319] like Figure 3A As shown in FIG, vehicle 200 may also include a user interface 170 for interacting with the driver or passengers of vehicle 200. For example, in a vehicle application, user interface 170 may include a touch screen 320, a knob 330, a button 340, and a microphone 350. The driver or passenger of vehicle 200 may also use a handle (e.g., located on or near the steering column of vehicle 200, including, for example, a turn signal handle), a button (e.g., located on the steering wheel of vehicle 200), etc. to interact with system 100. In some embodiments, microphone 350 may be positioned adjacent to rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, system 100 may provide various notifications (e.g., warnings) via speakers 360.
[0320] Figure 3B-3D is an illustration of an exemplary camera mount 370 configured to be placed behind a rearview mirror and against a vehicle windshield, consistent with the disclosed embodiments. Figure 3BAs shown in FIG, camera mount 370 may include image capture devices 122, 124, and 126. Image capture device 124 and image capture device 126 may be placed behind a sun visor 380, which may be flush with the vehicle windshield and include a combination of film and / or anti-reflective material. For example, sun visor 380 may be placed so that it is aligned with the vehicle windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 may be placed behind sun visor 380, for example, as Figure 3D The disclosed embodiments are not limited to any particular configuration of image capture devices 122 , 124 , and 126 , camera mount 370 , and light shield 380 . Figure 3C From the perspective of the past, Figure 3B An illustration of the camera mount 370 shown in .
[0321] As will be appreciated by those skilled in the art having the benefit of this disclosure, many variations and / or modifications may be made to the foregoing disclosed embodiments. For example, not all components are necessary for the operation of the system 100. Additionally, any component may be located in any appropriate portion of the system 100, and the components may be rearranged into various configurations while providing the functionality of the disclosed embodiments. Thus, the foregoing configurations are examples, and regardless of the configurations discussed above, the system 100 is capable of providing a wide range of functionality to analyze the surroundings of the vehicle 200 and navigate the vehicle 200 in response to the analysis.
[0322] As discussed in more detail below and consistent with the various disclosed embodiments, system 100 can provide various features related to autonomous driving and / or driver assistance technologies. For example, system 100 can analyze image data, location data (e.g., GPS location information), map data, velocity data, and / or data from sensors included in vehicle 200. System 100 can collect data for analysis from, for example, image acquisition unit 120, location sensor 130, and other sensors. Additionally, system 100 can analyze the collected data to determine whether vehicle 200 should take a certain action and then automatically take the determined action without human intervention. For example, while vehicle 200 is navigating without human intervention, system 100 can automatically control braking, acceleration, and / or steering of vehicle 200 (e.g., by transmitting control signals to one or more of throttle system 220, braking system 230, and steering system 240). Additionally, system 100 can analyze the collected data and, based on the analysis of the collected data, issue warnings and / or alerts to vehicle occupants. The following provides additional details regarding various embodiments provided by system 100 .
[0323] Forward multi-imaging system
[0324] As discussed above, system 100 can provide driving assistance functionality using a multi-camera system. The multi-camera system can use one or more cameras facing the vehicle's forward direction. In other embodiments, the multi-camera system can include one or more cameras facing the side or rear of the vehicle. In one embodiment, for example, system 100 can use a two-camera imaging system, wherein a first camera and a second camera (e.g., image capture device 122 and image capture device 124) can be placed in front of and / or on the side of a vehicle (e.g., vehicle 200). The first camera can have a field of view that is larger than, smaller than, or partially overlaps with the field of view of the second camera. In addition, the first camera can be connected to a first image processor to perform monocular image analysis on the image provided by the first camera, and the second camera can be connected to a second image processor to perform monocular image analysis on the image provided by the second camera. The outputs (e.g., processed information) of the first and second image processors can be combined. In some embodiments, the second image processor can receive images from both the first and second cameras to perform stereo analysis. In another embodiment, system 100 can use a three-camera imaging system, wherein each camera has a different field of view. Thus, such a system can make decisions based on information derived from objects located at different distances both in front of and to the sides of the vehicle. References to monocular image analysis may refer to instances in which image analysis is performed based on images captured from a single viewpoint (e.g., from a single camera). Stereoscopic image analysis may refer to instances in which image analysis is performed based on two or more images captured using one or more variations of image capture parameters. For example, captured images suitable for performing stereoscopic image analysis may include: images captured from two or more different positions, images captured from different fields of view, images captured using different focal lengths, images captured with disparity information, and the like.
[0325] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122-126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees, or another value selected from a range of approximately 20 to 45 degrees), image capture device 124 may provide a wide field of view (e.g., 150 degrees, or another value selected from a range of approximately 100 to approximately 180 degrees), and image capture device 126 may provide an intermediate field of view (e.g., 46 degrees, or another value selected from a range of approximately 35 to approximately 60 degrees). In some embodiments, image capture device 126 may serve as the primary or primary camera. Image capture devices 122-126 may be positioned behind rearview mirror 310 and positioned substantially side-by-side (e.g., 6 cm apart). Additionally, in some embodiments, as discussed above, one or more of image capture devices 122-126 may be mounted behind sun visor 380 flush with the windshield of vehicle 200. Such shielding is done to minimize the effect of any reflections from the interior of the car on the image capture devices 122-126.
[0326] In another embodiment, as above combined Figure 3B and Figure 3C As discussed, a wide field of view camera (e.g., image capture device 124 in the above example) can be mounted lower than the narrow field of view camera and the main field of view camera (e.g., camera device 122 and camera device 126 in the above example). This configuration can provide a clear line of sight from the wide field of view camera. To reduce reflections, the camera can be mounted close to the windshield of vehicle 200, and a polarizing filter can be included on the camera to suppress reflected light.
[0327] A three-camera system can provide certain performance characteristics. For example, some embodiments may include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the three-camera configuration discussed above, the processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processing chips, as discussed above), each processing device dedicated to processing images captured by one or more of the image capture devices 122-126.
[0328] In a three-camera system, a first processing device can receive images from both the main camera and the narrow field of view camera and perform vision processing on the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the first processing device can calculate the disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the vehicle 200's environment. The first processing device can then combine the 3D reconstruction with 3D map data, or with 3D information calculated based on information from another camera.
[0329] The second processing device can receive images from the primary camera and perform visual processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the second processing device can calculate camera displacement and, based on this displacement, calculate the disparity of pixels between successive images and create a 3D reconstruction of the scene (e.g., structure from motion). The second processing device can transmit the structure from motion-based 3D reconstruction to the first processing device for combination with the stereoscopic 3D image.
[0330] The third processing device can receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device can also execute additional processing instructions to analyze the images and identify objects moving in the images, such as vehicles changing lanes, pedestrians, etc.
[0331] In some embodiments, having independently captured and processed streams of image-based information can provide an opportunity to provide redundancy in the system. Such redundancy can include, for example, using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.
[0332] In some embodiments, system 100 may utilize two image capture devices (e.g., image capture device 122 and image capture device 124) to provide navigation assistance to vehicle 200, and utilize a third image capture device (e.g., image capture device 126) to provide redundancy and validate the analysis of data received from the other two image capture devices. For example, in such a configuration, image capture device 122 and image capture device 124 may provide images for stereo analysis by system 100 to navigate vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and validate information derived from images captured from image capture device 122 and / or image capture device 124. In other words, image capture device 126 (and the corresponding processing device) may be considered to provide a redundant subsystem for providing a check on the analysis derived from image capture device 122 and image capture device 124 (e.g., to provide an automatic emergency braking (AEB) system).
[0333] Those skilled in the art will recognize that the above camera configurations, camera placements, number of cameras, camera positions, etc. are merely examples. These components and other components described with respect to the overall system can be assembled and used in a variety of different configurations without departing from the scope of the embodiments of the present disclosure. Further details regarding the use of a multi-camera system for providing driver assistance and / or autonomous vehicle functionality are described below.
[0334] Figure 4 1 is an exemplary functional block diagram of memory 140 and / or storage 150 that may be stored / programmed with instructions for performing one or more operations consistent with embodiments of the present disclosure. Although reference is made to memory 140 below, those skilled in the art will recognize that instructions may be stored in memory 140 and / or storage 150.
[0335] like Figure 4 As shown in FIG, memory 140 may store a monocular image analysis module 402, a stereo image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. In addition, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 402-408 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer individually or collectively to application processor 180 and image processor 190. Therefore, any step in the following process may be performed by one or more processing devices.
[0336] In one embodiment, monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform monocular image analysis on a set of images acquired by one of image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the set of images with additional sensory information (e.g., information from radar) to perform monocular image analysis. Figures 5A-5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on this analysis, the system 100 (e.g., via the processing unit 110) may cause one or more navigation responses in the vehicle 200, such as a turn, a lane change, a change in acceleration, etc., as discussed below in conjunction with the navigation response module 408.
[0337] In one embodiment, the stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform stereo image analysis on a first set and a second set of images acquired by a combination of any selected image capture devices from the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the first set and the second set of images with additional sensory information (e.g., information from radar) to perform the stereo image analysis. For example, the stereo image analysis module 404 may include instructions for performing stereo image analysis based on a first set of images acquired by the first capture device 124 and a second set of images acquired by the image capture device 126. As shown below in conjunction with Figure 6 As described, stereo image analysis module 404 may include instructions for detecting a set of features within the first set and the second set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, etc. Based on this analysis, processing unit 110 may cause one or more navigation responses in vehicle 200, such as a turn, a lane change, a change in acceleration, etc., as discussed below in conjunction with navigation response module 408.
[0338] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computing and electromechanical devices in the vehicle 200, the one or more computing and electromechanical devices being configured to cause changes in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target velocity for the vehicle 200 based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include, for example, target position, velocity, and / or acceleration, the position and / or velocity of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects, position information of the vehicle 200 relative to lane markings on the road, and the like. Furthermore, the processing unit 110 may calculate the target velocity for the vehicle 200 based on sensory input (e.g., information from radar) and input from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200. Based on the calculated target rate, the processing unit 110 can send electronic signals to the throttle system 220, braking system 230, and / or steering system 240 of the vehicle 200 to trigger a change in the speed and / or acceleration of the vehicle 200 by, for example, physically depressing the brake or releasing the accelerator.
[0339] In one embodiment, navigation response module 408 may store software executable by processing unit 110 to determine a desired navigation response based on data derived from the execution of monocular image analysis module 402 and / or stereo image analysis module 404. Such data may include position and velocity information associated with nearby vehicles, pedestrians, and road objects, target position information for vehicle 200, and the like. Additionally, in some embodiments, the navigation response may be based (in part or in whole) on map data, a predetermined position of vehicle 200, and / or a relative velocity or acceleration between vehicle 200 and one or more objects detected from the execution of monocular image analysis module 402 and / or stereo image analysis module 404. Navigation response module 408 may also determine the desired navigation response based on sensory input (e.g., information from radar) and input from other systems of vehicle 200, such as vehicle 200's throttle system 220, braking system 230, and / or steering system 240. Based on the desired navigation response, processing unit 110 may send electronic signals to throttle system 220, brake system 230, and steering system 240 of vehicle 200 to trigger the desired navigation response by, for example, turning the steering wheel of vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, processing unit 110 may use the output of navigation response module 408 (e.g., the desired navigation response) as input to execution of velocity and acceleration module 406 for calculating the change in velocity of vehicle 200.
[0340] Figure 5A is a flow chart illustrating an exemplary process 500A for causing one or more navigation responses based on monocular image analysis consistent with an embodiment of the present disclosure. At step 510, the processing unit 110 may receive a plurality of images via the data interface 128 between the processing unit 110 and the image acquisition unit 120. For example, a camera included in the image acquisition unit 120 (such as the image capture device 122 having the field of view 202) may capture a plurality of images of the area in front of the vehicle 200 (or, for example, the areas to the sides and rear of the vehicle) and send them to the processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). The processing unit 110 may execute the monocular image analysis module 402 to analyze the plurality of images at step 520, as described below in conjunction with Figure 5B-5D By performing this analysis, processing unit 110 may detect a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and the like.
[0341] Processing unit 110 may also run monocular image analysis module 402 in step 520 to detect various road hazards, such as, for example, parts of truck tires, fallen road signs, loose cargo, small animals, and the like. Road hazards can vary in structure, shape, size, and color, which can make detecting such hazards more challenging. In some embodiments, processing unit 110 may run monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera motion between consecutive image frames and calculate disparity in pixels between frames to construct a 3D map of the road. Processing unit 110 may then use the 3D map to detect the road surface and hazards present on the road surface.
[0342] At step 530, the processing unit 110 may execute the navigation response module 408 to perform the navigation response based on the analysis performed at step 520 and the above combination. Figure 4 The described techniques induce one or more navigation responses in vehicle 200. Navigation responses may include, for example, a turn, a lane change, a change in acceleration, and the like. In some embodiments, processing unit 110 may use data derived from the operation of velocity and acceleration module 406 to induce one or more navigation responses. Furthermore, multiple navigation responses may occur simultaneously, sequentially, or any combination thereof. For example, processing unit 110 may cause vehicle 200 to shift lanes and then accelerate by, for example, sequentially sending control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 may cause vehicle 200 to brake while shifting lanes by, for example, simultaneously sending control signals to braking system 230 and steering system 240 of vehicle 200.
[0343] Figure 5Bis a flow chart illustrating an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a collection of images consistent with an embodiment of the present disclosure. Processing unit 110 may run monocular image analysis module 402 to implement process 500B. At step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify possible locations within each image that may contain objects of interest (e.g., vehicles, pedestrians, or parts thereof). The predetermined patterns may be designed in such a way as to achieve a high "false hit" rate and a low "miss" rate. For example, processing unit 110 may apply a low threshold value for similarity to the predetermined patterns to identify candidate objects as possible vehicles or pedestrians. Doing so may allow processing unit 110 to reduce the probability of missing (e.g., failing to identify) candidate objects representing vehicles or pedestrians.
[0344] At step 542, processing unit 110 may filter the set of candidate objects based on classification criteria to exclude certain candidates (e.g., irrelevant or less relevant objects). Such criteria may be derived from various attributes associated with object types stored in a database (e.g., a database stored in memory 140). Attributes may include object shape, size, texture, location (e.g., relative to vehicle 200), etc. Thus, processing unit 110 may use one or more sets of criteria to reject incorrect candidates from the set of candidate objects.
[0345] At step 544, processing unit 110 may analyze multiple frames of the image to determine whether an object in the set of candidate objects represents a vehicle and / or a pedestrian. For example, processing unit 110 may track the detected candidate object across consecutive frames and accumulate data associated with the detected object (e.g., size, position relative to vehicle 200, etc.) on a frame-by-frame basis. In addition, processing unit 110 may estimate parameters of the detected object and compare the frame-by-frame position data of the object with a predetermined position.
[0346] At step 546, the processing unit 110 may construct a set of measurements for the detected objects. Such measurements may include, for example, position, velocity, and acceleration values associated with the detected objects (relative to the vehicle 200). In some embodiments, the processing unit 110 may construct the measurements based on a series of time-based observation estimation techniques such as a Kalman filter or linear quadratic estimation (LQE), and / or based on modeling data available for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on a measurement of the scale of the object, where the scale measurement is proportional to the time to collision (e.g., the amount of time it takes for the vehicle 200 to reach the object). Thus, by executing steps 540-546, the processing unit 110 may identify vehicles and pedestrians that appear within the set of captured images and derive information associated with the vehicles and pedestrians (e.g., position, velocity, size). Based on the identification and the derived information, the processing unit 110 may cause one or more navigation responses in the vehicle 200, as described above in conjunction with Figure 5A described.
[0347] At step 548, processing unit 110 may perform optical flow analysis on one or more images to reduce the probability of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to, for example, analyzing motion patterns relative to vehicle 200 in one or more images associated with other vehicles and pedestrians, and the motion patterns are distinct from road surface motion. Processing unit 110 may calculate the motion of candidate objects by observing different positions of the objects across multiple image frames captured at different times. Processing unit 110 may use the position and time values as input to a mathematical model for calculating the motion of candidate objects. Thus, optical flow analysis may provide another method for detecting vehicles and pedestrians near vehicle 200. Processing unit 110 may perform optical flow analysis in conjunction with steps 540-546 to provide redundancy for detecting vehicles and pedestrians and increase the reliability of system 100.
[0348] Figure 5Cis a flow chart illustrating an exemplary process 500C for detecting road markings and / or lane geometry in a collection of images, consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a collection of objects by scanning one or more images. To detect lane markings, lane geometry, and other relevant road marking segments, processing unit 110 may filter the collection of objects to exclude those determined to be irrelevant (e.g., small potholes, small rocks, etc.). At step 552, processing unit 110 may group the segments detected in step 550 that belong to the same road marking or lane marking. Based on the grouping, processing unit 110 may develop a model, such as a mathematical model, representing the detected segments.
[0349] At step 554, processing unit 110 may construct a set of measurements associated with the detected segments. In some embodiments, processing unit 110 may create a projection of the detected segments from the image plane onto a real-world plane. The projection may be characterized using a cubic polynomial with coefficients corresponding to physical properties, such as the position, slope, curvature, and curvature derivatives of the detected road. When generating the projection, processing unit 110 may account for changes in the road surface, as well as the pitch and roll rates associated with vehicle 200. In addition, processing unit 110 may model road elevation by analyzing position and motion cues presented on the road surface. Additionally, processing unit 110 may estimate the pitch and roll rate associated with vehicle 200 by tracking a set of feature points in one or more images.
[0350] At step 556, processing unit 110 may perform multi-frame analysis by, for example, tracking the detected segments across consecutive image frames and accumulating data associated with the detected segments on a frame-by-frame basis. As processing unit 110 performs multi-frame analysis, the set of measurements constructed at step 554 may become more reliable and associated with increasingly higher confidence levels. Thus, by executing steps 550-556, processing unit 110 may identify road markings that appear within the set of captured images and derive lane geometry information. Based on this identification and the derived information, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in conjunction with Figure 5A described.
[0351] At step 558, processing unit 110 may consider information from additional sources to further develop a safety model for vehicle 200 within the environment surrounding vehicle 200. Processing unit 110 may use this safety model to define a context within which system 100 may safely perform autonomous control of vehicle 200. To develop the safety model, in some embodiments, processing unit 110 may consider the positions and motions of other vehicles, detected road edges and obstacles, and / or a general road shape description extracted from map data (such as data from map database 160). By considering information from additional sources, processing unit 110 may provide redundancy for detecting road markings and lane geometry and increase the reliability of system 100.
[0352] Figure 5D is a flow chart illustrating an exemplary process 500D for detecting traffic lights in a collection of images, consistent with disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500D. At step 560, processing unit 110 may scan the collection of images and identify objects appearing at locations in the images that are likely to contain traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects that excludes those objects that are unlikely to correspond to traffic lights. Filtering may be based on various attributes associated with traffic lights, such as shape, size, texture, position (e.g., relative to vehicle 200), and so on. Such attributes may be based on multiple examples of traffic lights and traffic control signals and may be stored in a database. In some embodiments, processing unit 110 may perform multi-frame analysis on the set of candidate objects that reflect possible traffic lights. For example, processing unit 110 may track candidate objects across consecutive image frames, estimate the real-world positions of the candidate objects, and filter out objects that are in motion (which are unlikely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate objects and identify relative locations of detected colors that appear within possible traffic lights.
[0353] At step 562, processing unit 110 may analyze the geometry of the hub. This analysis may be based on any combination of: (i) the number of lanes detected on either side of vehicle 200; (ii) detected markings on the road (such as arrows); and (iii) descriptions of the hub extracted from map data (such as data from map database 160). Processing unit 110 may use information derived from the operation of monocular analysis module 402 to perform this analysis. Furthermore, processing unit 110 may determine a correspondence between the traffic lights detected at step 560 and the lanes present near vehicle 200.
[0354] As the vehicle 200 approaches the hub, at step 564, the processing unit 110 may update the confidence level associated with the analyzed hub geometry and detected traffic lights. For example, comparing the number of traffic lights estimated to be present at the hub with the number of traffic lights actually present at the hub may affect the confidence level. Thus, based on the confidence level, the processing unit 110 may delegate control to the driver of the vehicle 200 in order to improve safety conditions. By executing steps 560-564, the processing unit 110 may identify traffic lights that appear within the set of captured images and analyze hub geometry information. Based on this identification and analysis, the processing unit 110 may cause one or more navigation responses in the vehicle 200, as described above in conjunction with Figure 5A described.
[0355] Figure 5E is a flow chart illustrating an exemplary process 500E for inducing one or more navigation responses in vehicle 200 based on a vehicle path, consistent with an embodiment of the present disclosure. At step 570, processing unit 110 may construct an initial path associated with vehicle 200. The vehicle path may be represented by a set of points expressed as coordinates (x, z), and the distance di between two points in the set of points may fall within a range of 1 to 5 meters. In one embodiment, processing unit 110 may construct the initial vehicle path using two polynomials, such as a left road polynomial and a right road polynomial. Processing unit 110 may calculate the geometric midpoint between the two polynomials and, if applicable, offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset) (an offset of zero may correspond to driving in the middle of a lane). The offset may be in a direction perpendicular to a line segment between any two points in the lane path. In another embodiment, processing unit 110 may use a polynomial and an estimated lane width to offset each point of the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).
[0356] At step 572, processing unit 110 may update the vehicle path constructed at step 570. Processing unit 110 may reconstruct the vehicle path constructed at step 570 using a higher resolution, such that the distance dk between two points in the set of points representing the vehicle path is less than the aforementioned distance di. For example, distance dk may fall within the range of 0.1 to 0.3 meters. Processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may yield a cumulative distance vector S corresponding to the total length of the vehicle path (e.g., based on the set of points representing the vehicle path).
[0357] At step 574, processing unit 110 may determine a look-ahead point (expressed as coordinates (xl, zl)) based on the updated vehicle path reconstructed at step 572. Processing unit 110 may extract the look-ahead point from the accumulated distance vector S, and the look-ahead point may be associated with a look-ahead distance and a look-ahead time. The look-ahead distance, which may have a lower limit ranging from 10 to 20 meters, may be calculated as the product of the velocity of vehicle 200 and the look-ahead time. For example, as the velocity of vehicle 200 decreases, the look-ahead distance may also decrease (e.g., until it reaches the lower limit). The look-ahead time, which may range from 0.5 to 1.5 seconds, may be inversely proportional to the gain of one or more control loops associated with inducing a navigation response in vehicle 200, such as a heading error tracking control loop. For example, the gain of the heading error tracking control loop may depend on the yaw rate loop, the steering actuator loop, the vehicle's lateral dynamics, and so on. Therefore, the higher the gain of the heading error tracking control loop, the smaller the look-ahead time.
[0358] At step 576, the processing unit 110 may determine a heading error and a yaw rate command based on the look-ahead point determined at step 574. The processing unit 110 may determine the heading error by calculating the inverse tangent of the look-ahead point (e.g., arctan(xl / zl)). The processing unit 110 may determine the yaw rate command as the product of the heading error and a high-level control gain. If the look-ahead distance is not at the lower limit, the high-level control gain may be equal to: (2 / look-ahead time). Otherwise, the high-level control gain may be equal to: (2*velocity of vehicle 200 / look-ahead distance).
[0359] Figure 5F FIG. 5 is a flow chart illustrating an exemplary process 500F for determining whether a leading vehicle is changing lanes, consistent with an embodiment of the present disclosure. In step 580, the processing unit 110 may determine navigation information associated with a leading vehicle (e.g., a vehicle traveling in front of the vehicle 200). For example, the processing unit 110 may use the above combined Figure 5A and 5B The described techniques may be used to determine the position, velocity (eg, direction and speed), and / or acceleration of the vehicle ahead. The processing unit 110 may also use the above combined Figure 5E The described techniques determine one or more road polynomials, look-ahead points (associated with vehicle 200 ), and / or a snail trail (eg, a collection of points describing a path taken by a leading vehicle).
[0360] At step 582, processing unit 110 may analyze the navigation information determined at step 580. In one embodiment, processing unit 110 may calculate the distance between the snail trail and the road polynomial (e.g., along the trail). If the variance of this distance along the trail exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters on a straight road, 0.3 to 0.4 meters on a moderately curving road, and 0.5 to 0.6 meters on a road with a sharp turn), processing unit 110 may determine that the leading vehicle is likely changing lanes. In the event that multiple vehicles are detected traveling ahead of vehicle 200, processing unit 110 may compare the snail trails associated with each vehicle. Based on this comparison, processing unit 110 may determine that a vehicle whose snail trail does not match the snail trails of the other vehicles is likely changing lanes. Processing unit 110 may additionally compare the curvature of the snail trail (associated with the leading vehicle) with the expected curvature of the road segment in which the leading vehicle is traveling. The expected curvature may be extracted from map data (e.g., data from map database 160), from a road polynomial, from snail trails of other vehicles, from prior knowledge about the road, etc. If the difference between the curvature of the snail trail and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the leading vehicle is likely changing lanes.
[0361] In another embodiment, processing unit 110 may compare the instantaneous position of the preceding vehicle to a look-ahead point (associated with vehicle 200) over a specific time period (e.g., 0.5 to 1.5 seconds). If the distance between the instantaneous position of the preceding vehicle and the look-ahead point changes over the specific time period, and the cumulative sum of the changes exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a moderately curving road, and 1.3 to 1.7 meters on a road with a sharp curve), processing unit 110 may determine that the preceding vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the snail trail by comparing the lateral distance traveled along the trail to the expected curvature of the snail trail. The expected radius of curvature may be determined according to the following calculation: (δz² + δx²) / 2 / (δx), where δx represents the lateral distance traveled and δz represents the longitudinal distance traveled. If the difference between the lateral distance traveled and the expected curvature exceeds a predetermined threshold (e.g., 500 to 700 meters), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the position of the leading vehicle. If the position of the leading vehicle obscures the road polynomial (e.g., the leading vehicle is overlaid on top of the road polynomial), processing unit 110 may determine that the leading vehicle is likely changing lanes. If the position of the leading vehicle is such that another vehicle is detected ahead of the leading vehicle, and the snail trails of the two vehicles are not parallel, processing unit 110 may determine that the (closer) leading vehicle is likely changing lanes.
[0362] At step 584, processing unit 110 may determine whether the leading vehicle 200 is changing lanes based on the analysis performed at step 582. For example, processing unit 110 may make a determination based on a weighted average of the individual analyses performed at step 582. Under such an approach, for example, a determination by processing unit 110 that the leading vehicle is likely changing lanes based on a particular type of analysis may be assigned a value of "1" (and a "0" indicating a determination that the leading vehicle is unlikely to be changing lanes). Different analyses performed at step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analyses and weights.
[0363] Figure 66 is a flow chart illustrating an exemplary process 600 for inducing one or more navigation responses based on stereo image analysis, consistent with the disclosed embodiments. At step 610, processing unit 110 may receive a first plurality of images and a second plurality of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture device 122 having field of view 202 and image capture device 124 having field of view 204) may capture a first plurality of images and a second plurality of images of the area in front of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first plurality of images and the second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.
[0364] At step 620, the processing unit 110 runs the stereo image analysis module 404 to perform stereo image analysis on the first plurality of images and the second plurality of images, thereby creating a 3D map of the road in front of the vehicle and detecting features such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. within the image. Figures 5A-5D Stereo image analysis can be performed in a manner similar to the steps described above. For example, processing unit 110 can execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road signs, traffic lights, road hazards, etc.) within the first and second pluralities of images, filter out a subset of candidate objects based on various criteria, perform multi-frame analysis, construct measurements, and determine confidence levels for the remaining candidate objects. When performing the above steps, processing unit 110 can consider information from both the first and second pluralities of images, rather than information from a single set of images. For example, processing unit 110 can analyze differences in pixel-level data (or other subsets of data from the two streams of captured images) for candidate objects appearing in both the first and second pluralities of images. As another example, processing unit 110 can estimate the position and / or velocity (e.g., relative to vehicle 200) of a candidate object by observing whether the object appears in one of the plurality of images but not in the other, or by observing other differences (such as differences that may exist between objects appearing in the two image streams). For example, the position, velocity, and / or acceleration relative to vehicle 200 may be determined based on features of the trajectory, position, movement characteristics, etc. associated with objects appearing in one or both of the image streams.
[0365] At step 630, the processing unit 110 may execute the navigation response module 408 to perform the navigation response based on the analysis performed at step 620 and the above combination. Figure 4 The described techniques cause one or more navigation responses in the vehicle 200. The navigation responses may include, for example, a turn, a lane change, a change in acceleration, a change in speed, braking, etc. In some embodiments, the processing unit 110 may use data derived from the operation of the speed and acceleration module 406 to cause the one or more navigation responses. In addition, multiple navigation responses may occur simultaneously, sequentially, or any combination thereof.
[0366] Figure 7 FIG7 is a flow chart illustrating an exemplary process 700 for inducing one or more navigation responses based on analysis of three sets of images, consistent with the disclosed embodiments. At step 710, processing unit 110 may receive a first plurality of images, a second plurality of images, and a third plurality of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture device 122 having field of view 202, image capture device 124 having field of view 204, and image capture device 126 having field of view 206) may capture a first plurality of images, a second plurality of images, and a third plurality of images of the front and / or side areas of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first plurality of images, the second plurality of images, and the third plurality of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface for communicating data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.
[0367] At step 720, the processing unit 110 may analyze the first plurality of images, the second plurality of images, and the third plurality of images to detect features within the images such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. Figure 5A - Figure 5D and Figure 6 For example, the processing unit 110 may (e.g., via the operation of the monocular image analysis module 402 and based on the above combination) perform the analysis in a similar manner to the steps described above. Figures 5A-5D Alternatively, the processing unit 110 may (e.g., via the operation of the stereo image analysis module 404 and based on the above combination) perform monocular image analysis on each of the first plurality of images, the second plurality of images, and the third plurality of images. Figure 6The processing unit 110 may perform stereoscopic image analysis on the first and second pluralities of images, the second and third pluralities of images, and / or the first and third pluralities of images (as described in the steps described). Processed information corresponding to the analysis of the first, second, and / or third pluralities of images may be combined. In some embodiments, the processing unit 110 may perform a combination of monocular image analysis and stereoscopic image analysis. For example, the processing unit 110 may perform monocular image analysis on the first pluralities of images (e.g., via execution of the monocular image analysis module 402) and perform stereoscopic image analysis on the second and third pluralities of images (e.g., via execution of the stereoscopic image analysis module 404). The configuration of the image capture devices 122, 124, and 126—including their respective positions and fields of view 202, 204, and 206—may affect the type of analysis performed on the first, second, and third pluralities of images. The disclosed embodiments are not limited to a particular configuration of the image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third pluralities of images.
[0368] In some embodiments, processing unit 110 may perform testing on system 100 based on the images acquired and analyzed at steps 710 and 720. Such testing may provide an indication of the overall performance of system 100 for certain configurations of image capture devices 122, 124, and 126. For example, processing unit 110 may determine the proportion of "false hits" (e.g., instances where system 100 incorrectly determined the presence of a vehicle or pedestrian) and "misses."
[0369] At step 730, the processing unit 110 may cause one or more navigation responses in the vehicle 200 based on information derived from two of the first plurality of images, the second plurality of images, and the third plurality of images. The selection of two of the first plurality of images, the second plurality of images, and the third plurality of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the plurality of images. The processing unit 110 may also make the selection based on image quality and resolution, the effective field of view reflected in the image, the number of captured frames, the degree to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the objects appear, the proportion of the objects appearing in each such frame), and the like.
[0370] In some embodiments, processing unit 110 may select information derived from two of the first, second, and third pluralities of images by determining how consistent the information derived from one image source is with information derived from other images. For example, processing unit 110 may combine processed information derived from each of image capture devices 122, 124, and 126 (whether through monocular analysis, stereo analysis, or a combination of both) and determine visual indicators that are consistent across images captured from each of image capture devices 122, 124, and 126 (e.g., lane markings, detected vehicles and their locations and / or paths, detected traffic lights, etc.). Processing unit 110 may also exclude information that is inconsistent across captured images (e.g., vehicles changing lanes, vehicle models indicating vehicles that are too close to vehicle 200, etc.). Thus, processing unit 110 may select information derived from two of the first, second, and third pluralities of images based on the determination of consistent and inconsistent information.
[0371] The navigation response may include, for example, a turn, a lane change, a change in acceleration, etc. The processing unit 110 may perform the following operations based on the analysis performed at step 720 and the above combination. Figure 4 The processing unit 110 may also use data derived from the operation of the velocity and acceleration module 406 to cause one or more navigation responses. In some embodiments, the processing unit 110 may cause one or more navigation responses based on the relative position, relative velocity, and / or relative acceleration between the vehicle 200 and an object detected in any of the first plurality of images, the second plurality of images, and the third plurality of images. The multiple navigation responses may occur simultaneously, sequentially, or any combination thereof.
[0372] Sparse road models for autonomous vehicle navigation
[0373] In some embodiments, the disclosed systems and methods may utilize sparse maps for autonomous vehicle navigation. For example, a sparse map may provide sufficient information for navigating an autonomous vehicle without storing and / or updating large amounts of data. As discussed in greater detail below, an autonomous vehicle may utilize a sparse map to navigate one or more roads based on one or more trajectories.
[0374] Sparse maps for autonomous vehicle navigation
[0375] In some embodiments, the disclosed systems and methods may utilize sparse maps for autonomous vehicle navigation. For example, sparse maps can provide sufficient information for navigation without requiring excessive data storage or data transfer rates. As discussed in more detail below, a vehicle (which may be an autonomous vehicle) can use a sparse map to navigate one or more roads. For example, in some embodiments, a sparse map can include data related to roads and potential landmarks along the roads that may be sufficient for vehicle navigation while also exhibiting a small data footprint. For example, compared to digital maps that include detailed map information, such as image data collected along the roads, the sparse data maps described in detail below can require significantly less storage space and data transfer bandwidth. For example, instead of storing detailed representations of road segments, a sparse map can store a three-dimensional polynomial representation of a preferred vehicle path along the road. These paths can require very little data storage space. Additionally, in the described sparse data maps, landmarks can be identified and included in the sparse map road model to assist in navigation. These landmarks can be located at any spacing suitable to enable vehicle navigation, but in some cases, where the density and spacing are high, such landmarks do not need to be identified and included in the model. Instead, in some cases, navigation may be possible based on landmarks that are spaced at least 50 meters, at least 100 meters, at least 500 meters, at least 1 kilometer, or at least 2 kilometers apart. As will be discussed in more detail elsewhere, a sparse map may be generated based on data collected and measured by vehicles equipped with various sensors and devices, such as image capture devices, global positioning system sensors, motion sensors, and the like, as the vehicles travel along a road surface. In some cases, a sparse map may be generated based on data collected during multiple drives of one or more vehicles along a particular road surface.
[0376] Consistent with embodiments of the present disclosure, autonomous vehicle systems may utilize sparse maps for navigation. At the core of the sparse map, one or more three-dimensional contours may represent predetermined trajectories that autonomous vehicles may traverse as they move along associated road segments. The sparse map may also include data representing one or more road features. Such road features may include recognized landmarks, road signature contours, and other road-related features useful in navigating the vehicle. Sparse maps may enable autonomous vehicle navigation based on the relatively small amount of data included in the sparse map. For example, embodiments of the disclosed sparse maps may require a relatively small amount of storage space, rather than a detailed representation of a road that includes data such as road edges, road curvature, images associated with a road segment, or data detailing other physical features associated with the road segment, yet still adequately provide for autonomous vehicle navigation. The small data footprint of the disclosed sparse maps, discussed in more detail below, may be achieved in some embodiments by storing road-related elements that require a small amount of data but still enable autonomous navigation. For example, rather than storing a detailed representation of every aspect of the road, the disclosed sparse maps may store polynomial representations of one or more trajectories that a vehicle may follow along the road. Thus, rather than storing (or having to communicate) details about the physical properties of a road to enable navigation along the road, using the disclosed sparse map, a vehicle can be navigated along a particular road segment without, in some cases, having to interpret the physical aspects of the road, but rather by aligning its travel path with a trajectory (e.g., a polynomial spline) along the particular road segment. In this way, the vehicle can be navigated primarily based on the stored trajectory (e.g., the polynomial spline), which can require less storage space than solutions involving the storage of road surface images, road parameters, and road layouts.
[0377] In addition to the stored polynomial representations of trajectories along a road segment, the disclosed sparse map may also include small data objects that can represent road features. In some embodiments, the small data objects may include digital signatures derived from digital images (or digital signals) acquired by sensors (e.g., cameras, or other sensors such as suspension sensors) on a vehicle traveling along the road. The digital signatures may have a reduced size relative to the signals acquired by the sensors. In some embodiments, the digital signatures may be created to be compatible with classification functions configured to detect and identify road features from signals acquired by the sensors, for example, during a subsequent drive. In some embodiments, the digital signatures may be created to have as small a footprint as possible while maintaining the ability to associate or match road features with stored signatures based on images of the road features captured by a camera on a vehicle traveling along the same road segment at a subsequent time (or, if the stored signature is not based on images and / or includes other data, digital signals generated by the sensors). In some embodiments, the size of the data objects may also be associated with the uniqueness of the road features. For example, for a road feature that is detectable by a camera on a vehicle, and where the camera system on the vehicle is coupled to a classifier that is capable of distinguishing image data corresponding to the road feature as being associated with a particular type of road feature (e.g., a road sign), and where such a road sign is locally unique in that area (e.g., no identical road signs or road signs of the same type are present nearby), it is sufficient to store data indicating the type of the road feature and its location.
[0378] As will be discussed in more detail below, road features (e.g., landmarks along a road segment) can be stored as small data objects that can represent road features with relatively few bytes while providing sufficient information for identifying and using such features for navigation. In just one example, a road sign can be identified as a recognized landmark, and the vehicle's navigation can be based on the road sign. A representation of a road sign can be stored in a sparse map to include, for example, a few bytes of data indicating the landmark type (e.g., a stop sign) and a few bytes of data indicating the landmark's location. Navigation based on such a lightweight data representation of a landmark (e.g., using a landmark-based representation sufficient for positioning, identification, and navigation) can provide the desired level of navigation functionality associated with a sparse map without significantly increasing the data overhead associated with the sparse map. This lean representation of landmarks (and other road features) can utilize sensors and processors included on such a vehicle that are configured to detect, identify, and / or classify certain road features. When, for example, a sign or even a specific type of sign is unique in a given area (e.g., when there are no other signs or other signs of the same type), the sparse map can use data indicating the type of landmark (the sign or the specific type of sign), and during navigation (e.g., autonomous navigation), when a camera on an autonomous vehicle captures an image of an area that includes the sign (or an image of the specific type of sign), the processor can process the image, detect the sign (if indeed present in the image), classify it as a sign (or as a specific type of sign), and associate its location with the locations of the signs stored in the sparse map.
[0379] In some embodiments, an autonomous vehicle may include a vehicle body; and a processor configured to receive data included in a sparse map and generate navigation instructions for navigating the vehicle along a road segment based on the data in the sparse map.
[0380] Figure 8 A sparse map 800 is shown that can be accessed by vehicle 200 (which can be an autonomous vehicle) for providing autonomous vehicle navigation. The sparse map 800 can be stored in a memory such as memory 140 or memory 150. Such a memory device can include any type of non-transitory storage device or computer-readable medium. For example, in some embodiments, memory 140 or memory 150 can include a hard drive, an optical disk, a flash memory, a magnetic-based storage device, an optical-based storage device, and the like. In some embodiments, the sparse map 800 can be stored in a database (e.g., a map database 160), which can be stored in memory 140 or memory 150, or other types of storage devices.
[0381] In some embodiments, sparse map 800 may be stored on a storage device or non-transitory computer-readable medium provided on vehicle 200 (e.g., a storage device included in a navigation system on vehicle 200). A processor provided on vehicle 200 (e.g., processing unit 110) may access sparse map 800 stored on the storage device or computer-readable medium provided on vehicle 200 to generate navigation instructions for guiding autonomous vehicle 200 as it traverses a road segment.
[0382] However, for the vehicle, the sparse map 800 does not need to be stored locally. In some embodiments, the sparse map 800 can be stored on a storage device or computer-readable medium provided on a remote server communicating with the vehicle 200 or on a device associated with the vehicle 200. A processor (e.g., processing unit 110) provided on the vehicle 200 can receive the data included in the sparse map 800 from the remote server and can execute the data to guide the autonomous driving of the vehicle 200. In such an embodiment, the sparse map 800 can be made accessible to multiple vehicles (e.g., tens, hundreds, thousands, or millions of vehicles, etc.) crossing various road sections. It should also be noted that the sparse map 800 can include multiple sub-maps. For example, in some embodiments, the sparse map 800 can include hundreds, thousands, millions, or more subsets that can be used to navigate the vehicle. Such sub-maps can be referred to as local maps, and a vehicle traveling along the road can access any number of local maps related to the location where the vehicle is traveling. The local map portion of the sparse map 800 may be stored using a Global Navigation Satellite System (GNSS) key as an index into the database of the sparse map 800. Thus, while calculations of steering angles for navigating the host vehicle may be performed without reliance on the host vehicle's GNSS position, road features, or landmarks in current systems, such GNSS information may be used for retrieval of relevant local maps.
[0383] The collection of data and the generation of sparse map 800 are described in detail in other sections. Generally, however, sparse map 800 can be generated based on data collected from one or more vehicles as they travel along the road. For example, using sensors (e.g., cameras, speedometers, GPS, accelerometers, etc.) on one or more vehicles, the trajectories of one or more vehicles traveling along a road section can be recorded, and a polynomial representation of the preferred trajectory of the vehicle that subsequently travels along the road section can be determined based on the collected trajectories traveled by the one or more vehicles. Similarly, the data collected by one or more vehicles can assist in identifying potential landmarks along a particular road section. The data collected from the vehicles being crossed can also be used to identify road profile information such as road width profile, road roughness profile, traffic line spacing profile, etc. Using the collected information, sparse map 800 can be generated and distributed (e.g., for local storage or via instant data transmission) for use in navigating one or more autonomous vehicles. However, after the map is initially generated, map generation may not be terminated. As will be discussed in greater detail elsewhere, the sparse map 800 may be continuously or periodically updated based on data collected from vehicles as those vehicles continue to traverse the roads included in the sparse map 800 .
[0384] The data recorded in sparse map 800 may include location information based on Global Positioning System (GPS) data. For example, location information for various map elements, including, for example, landmark locations, road delineation locations, and the like, may be included in sparse map 800. GPS data collected from vehicles traversing the roadway may be used to obtain the locations of map elements included in sparse map 800. For example, a vehicle passing by an identified landmark may determine the location of the identified landmark using GPS location information associated with the vehicle and a determination of the location of the identified landmark relative to the vehicle (e.g., based on image analysis of data collected from one or more cameras on the vehicle). Such location determinations of the identified landmark may be repeated as additional vehicles pass by the location of the identified landmark (or any other feature included in sparse map 800). Some or all of these additional location determinations may be used to refine the location information stored in sparse map 800 relative to the identified landmark. For example, in some embodiments, multiple location measurements relative to a particular feature stored in sparse map 800 may be averaged together. However, any other mathematical operations may also be used to refine the stored locations of map elements based on the multiple determined locations for the map elements.
[0385] The sparse map of the disclosed embodiment can make it possible to carry out autonomous navigation of the vehicle using a relatively small amount of stored data. In certain embodiments, sparse map 800 can have the following data densities (comprising the data representing target track, landmarks, and the road features of any other storage): every kilometer of road is less than the data density of 2MB, every kilometer of road is less than the data density of 1MB, every kilometer of road is less than the data density of 500kB, or every kilometer of road is less than the data density of 100kB. In certain embodiments, the data density of sparse map 800 can be: every kilometer of road is less than 10kB or even every kilometer of road is less than 2kB (e.g., every kilometer 1.6kB), or every kilometer of road is not more than 10kB, or every kilometer of road is not more than 20kB. In certain embodiments, data with a total of 4GB or less can be used to autonomously navigate most (if not all) road surfaces in the U.S. These data density values may represent an average over the entire sparse map 800 , an average over a local map within the sparse map 800 , and / or an average over a specific road segment within the sparse map 800 .
[0386] As already mentioned, the sparse map 800 may include representations of multiple target trajectories 810 for guiding autonomous driving or navigation along a road segment. Such target trajectories may be stored as three-dimensional splines. The target trajectories stored in the sparse map 800 may be determined based on two or more reconstructed trajectories of a vehicle previously traversing a particular road segment. A road segment may be associated with a single target trajectory or multiple target trajectories. For example, for a two-lane road, a first target trajectory may be stored to represent a desired path for traveling along the road in a first direction, and a second target trajectory may be stored to represent a desired path for traveling along the road in another direction (e.g., opposite to the first direction). Additional target trajectories may be stored for a particular road segment. For example, with respect to a multi-lane road, one or more target trajectories may be stored, which represent the desired driving path for a vehicle in one or more lanes associated with the multi-lane road. In some embodiments, each lane in the multi-lane road may be associated with its own target trajectory. In other embodiments, there may be fewer stored target trajectories than the number of lanes present on the multi-lane road. In such a case, a vehicle navigating a multi-lane road may use any of the stored target trajectories to guide its navigation by taking into account the amount of lane offset from the lane for which the target trajectory is stored (e.g., if a vehicle is traveling in the leftmost lane of a three-lane highway, and target trajectories are stored only for the middle lane of the highway, the vehicle may navigate using the target trajectory for the middle lane by taking into account the amount of lane offset between the middle lane and the leftmost lane when generating navigation instructions).
[0387] In some embodiments, target track can represent the ideal path that vehicle should take when vehicle is traveling.Target track can be located at, for example, the approximate center of the lane of travel.In other cases, target track can be located at other places relative to road section.For example, target track can be approximately consistent with the center of road, the edge of road or the edge of lane etc.In such case, navigation based on target track can comprise determining the offset that will be maintained relative to the position of target track.And, in some embodiments, based on the type of vehicle, the offset that will be maintained determined relative to the position of target track can be different (for example, along at least a portion of target track, a passenger vehicle comprising two axles can have the offset different from the truck comprising more than two axles).
[0388] The sparse map 800 may also include data related to a plurality of predetermined landmarks 820, which are associated with specific road segments, local maps, and the like. As discussed in detail elsewhere, these landmarks can be used for navigation of the autonomous vehicle. For example, in some embodiments, the landmarks can be used to determine the vehicle's current position relative to a stored target track. Using this position information, the autonomous vehicle can adjust its heading to match the direction of the target track at the determined location.
[0389] A plurality of landmarks 820 may be identified and stored in the sparse map 800 at any suitable interval. In some embodiments, landmarks may be stored at a relatively high density (e.g., every few meters or more). However, in some embodiments, significantly larger landmark intervals may be employed. For example, in the sparse map 800, identified (or recognized) landmarks may be spaced 10 meters, 20 meters, 50 meters, 100 meters, 1 kilometer, or 2 kilometers apart. In some cases, identified landmarks may be located at intervals even greater than 2 kilometers apart. A vehicle may navigate between landmarks based on dead reckoning, and therefore, between determinations of the vehicle's position relative to a target trajectory, in which the vehicle uses sensors to determine its ego motion and estimate its position relative to the target trajectory. Because errors may accumulate during navigation by dead reckoning, the position determination relative to the target trajectory may become increasingly inaccurate over time. The vehicle may use the landmarks (and their known positions) that appear in the sparse map 800 to remove errors caused by dead reckoning in its position determination. In this way, the marked landmarks included in the sparse map 800 can serve as navigation anchors, according to which the accurate position of the vehicle relative to the target track can be determined. Because a certain amount of error is acceptable in the positioning position, marked landmarks do not always need to be available to autonomous vehicles. On the contrary, even based on 10 meters, 20 meters, 50 meters, 100 meters, 500 meters, 1 kilometer, 2 kilometers or more landmark intervals as mentioned above, suitable navigation may be possible. In certain embodiments, the density of 1 marked landmark per 1km of road can be enough to maintain longitudinal position accuracy within 1m. Therefore, not every potential landmark that occurs along the road section needs to be stored in the sparse map 800.
[0390] In addition to target tracks and identified landmarks, the sparse map 800 may include information related to various other roads. For example, Figure 9A A representation of a curve along a particular road segment is shown that may be stored in the sparse map 800. In some embodiments, a single lane of a road may be modeled by a three-dimensional polynomial description of the left and right sides of the road. Figure 9A Such polynomials representing the left and right sides of a single lane are shown in . Regardless of how many lanes a road may have, it is possible to express it in a manner similar to Figure 9A For example, the left and right sides of a multi-lane road can be represented by polynomials similar to Figure 9A , and intermediate lane markings included on multi-lane roads (e.g., dashed markings indicating lane boundaries, solid yellow lines indicating boundaries between lanes traveling in different directions, etc.) may also be represented using polynomials such as those shown in Figure 9A. The polynomials are represented by those shown in .
[0391] like Figure 9A As shown in , lane 900 can be represented using a polynomial (e.g., a first-order, second-order, third-order, or any suitable order polynomial). For illustration, lane 900 is shown as a two-dimensional lane, and the polynomial is shown as a two-dimensional polynomial. Lane 900 includes a left side 910 and a right side 920. In some embodiments, more than one polynomial can be used to represent the location of each side of the road or lane boundary. For example, each of left side 910 and right side 920 can be represented by multiple polynomials of any suitable length. In some cases, the polynomials can have a length of approximately 100 meters, although lengths greater or less than 100 meters can also be used. In addition, the polynomials can overlap with each other to facilitate seamless transitions in navigation based on subsequently encountered polynomials as the host vehicle travels along the road. For example, each of left side 910 and right side 920 can be represented by multiple third-order polynomials divided into segments of approximately 100 meters in length (an example of a first predetermined range), with the segments overlapping by approximately 50 meters. The polynomials representing left side 910 and right side 920 may or may not have the same order. For example, in some embodiments, some of the polynomials may be second-order polynomials, some may be third-order polynomials, and some may be fourth-order polynomials.
[0392] exist Figure 9A In the example shown in , the left side 910 of lane 900 is represented by two sets of third-order polynomials. The first set includes polynomial segment 911, polynomial segment 912, and polynomial segment 913. The second set includes polynomial segment 914, polynomial segment 915, and polynomial segment 916. The two sets, although substantially parallel to each other, follow the positions of their respective side roads. Polynomial segments 911-916 have a length of approximately 100 meters and overlap with adjacent segments in the series by approximately 50 meters. However, as previously mentioned, polynomials of different lengths and different amounts of overlap can also be used. For example, the polynomials can have a length of 500m, 1km, or more, and the amount of overlap can vary from 0 to 50m, 50m to 100m, or more than 100m. In addition, although Figure 9A Shown as representing polynomials extending in 2D space (eg, on the surface of paper), it will be understood that these polynomials may represent curves extending in three dimensions (eg, including a height component) to represent elevation changes in a road segment in addition to XY curvature.
[0393] Returning to the target trajectory of the sparse map 800, Figure 9BA three-dimensional polynomial representing a target trajectory for a vehicle traveling along a particular road segment is shown. The target trajectory represents not only the XY path that the host vehicle should travel along the particular road segment, but also the elevation changes that the host vehicle will experience while traveling along the road segment. Figure 9B Each target track in the sparse map 800 can be represented by one or more three-dimensional polynomials, similar to the three-dimensional polynomial 950 shown in FIG. 8 . The sparse map 800 may include multiple tracks (e.g., millions, billions, or more to represent the tracks of vehicles along various road segments throughout the world). In some embodiments, each target track can correspond to a spline connecting the three-dimensional polynomial segments.
[0394] Regarding the data footprint of the polynomial curves stored in the sparse map 800, in some embodiments, each cubic polynomial can be represented by four parameters, each requiring four bytes of data. A suitable representation can be achieved using a cubic polynomial requiring approximately 192 bytes of data per 100 meters. This translates to a data usage / transfer requirement of approximately 200 kB per hour for a host vehicle traveling approximately 100 km / hr.
[0395] Sparse map 800 can use a combination of geometric descriptors and metadata to describe the lane network. The geometry can be described by polynomials and splines as described above. The metadata can describe the number of lanes, special features (such as carpool lanes), and possibly other sparse markings. The total occupancy of such indicators can be negligible.
[0396] As previously mentioned, sparse map 800 can include a plurality of predetermined landmarks associated with road segments. Instead of storing actual images of the landmarks and, for example, relying on image recognition analysis based on captured and stored images, each landmark in sparse map 800 can be represented and identified using less data than would be required if actual images were stored. The data representing the landmarks can include sufficient information to describe or identify the landmarks along the road. Storing data describing the characteristics of the landmarks, rather than actual images of the landmarks, can reduce the size of sparse map 800.
[0397] Figure 10Examples of the types of landmarks that may be represented in the sparse map 800 are shown. Landmarks may include any object that is visible and identifiable along a road segment. Landmarks may be selected so that they are fixed and do not change frequently with respect to their location and / or content. Landmarks included in the sparse map 800 may be useful in determining the position of the vehicle 200 relative to a target trajectory as the vehicle traverses a particular road segment. Examples of landmarks may include: traffic signs, directional signs, general signs (e.g., rectangular signs), fixed attachments to curbs (e.g., lamp posts, reflectors, etc.), and any other suitable category. In some embodiments, lane markings on the road may also be included in the sparse map 800 as landmarks.
[0398] Figure 10 Examples of landmarks shown in the diagram include traffic signs, directional signs, fixed attachments, and general signs. Traffic signs may include, for example, speed limit signs (e.g., speed limit sign 1000), yield signs (e.g., yield sign 1005), route numbers (e.g., route number sign 1010), traffic light signs (e.g., traffic light sign 1015), and stop signs (e.g., stop sign 1020). Directional signs may include signs that include one or more arrows indicating one or more directions to different locations. For example, directional signs may include highway signs 1025 with arrows for directing vehicles to different roads or locations, exit signs 1030 with arrows directing vehicles to exit a road, and the like.
[0399] General signs may not be related to transportation. For example, general signs may include billboards used for advertising, or welcome signs near the border between two countries, states, counties, cities, or towns. Figure 10 A generic sign 1040 ("Joe's Restaurant") is shown. Although the generic sign 1040 may have a rectangular shape, such as Figure 10 , but the universal logo 1040 can have other shapes such as square, circle, triangle, etc.
[0400] Landmarks may also include curb fixtures. Curb fixtures may be objects that are not signs and may not be objects related to traffic or directions. For example, curb fixtures may include lamp posts (e.g., lamp post 1035), power line posts, traffic light posts, and the like.
[0401] Landmarks may also include beacons that may be specifically designed for use in autonomous vehicle navigation systems. For example, such beacons may include independent structures placed at predetermined intervals to assist in navigating the host vehicle. Such beacons may also include visual / graphic information (e.g., icons, emblems, barcodes, etc.) that can be added to existing road signs to identify or recognize vehicles traveling along the road segment. Such beacons may also include electronic components. In such embodiments, electronic beacons (e.g., RFID tags, etc.) may be used to transmit non-visual information to the host vehicle. Such information may include, for example, landmark identification and / or landmark location information that the host vehicle may use when determining its position along the target trajectory.
[0402] In some embodiments, the landmarks included in the sparse map 800 can be represented by data objects of predetermined sizes. The data representing the landmarks can include any suitable parameters for identifying a specific landmark. For example, in some embodiments, the landmarks stored in the sparse map 800 can include parameters such as: the physical size of the landmark (for example, to support the estimation of the distance to the landmark based on known size / scale), the distance to the previous landmark, the lateral offset, the altitude, the type code (for example, the landmark type - what type of direction sign, traffic sign, etc.), the GPS coordinates (for example, to support global localization), and any other suitable parameters. Each parameter can be associated with a data size. For example, 8 bytes of data can be used to store the landmark size. 12 bytes of data can be used to specify the distance to the previous landmark, the lateral offset, and the altitude. The type code associated with a landmark such as a direction sign or a traffic sign can require about 2 bytes of data. For universal landmarks, the image signature of the universal landmark allows identification to be stored using a 50-byte data store. The GPS location of a landmark can be associated with a 16-byte data store. These data sizes for each parameter are merely examples, and other data sizes can also be used.
[0403] Representing landmarks in the sparse map 800 in this way can provide a lean solution for efficiently representing landmarks in a database. In some embodiments, signs can be referred to as semantic signs and non-semantic signs. Semantic signs can include any type of sign for which there is a standard meaning (e.g., speed limit signs, warning signs, direction signs, etc.). Non-semantic signs can include any sign that is not associated with a standard meaning (e.g., general advertising signs, signs identifying business premises, etc.). For example, each semantic sign can be represented using 38 bytes of data (e.g., 8 bytes for representing the size; 12 bytes for representing the distance, lateral offset, and altitude to the previous landmark; 2 bytes for representing the type code, and 16 bytes for representing GPS coordinates). The sparse map 800 can use a tag system to represent the landmark type. In some cases, each traffic sign or direction sign can be associated with its own tag, which can be stored in the database as part of the landmark identification. For example, the database can include approximately 1,000 different tags to represent various traffic signs, and approximately 10,000 different tags to represent direction signs. Of course, any suitable number of tags can be used, and additional tags can be created when necessary. In some embodiments, a universal landmark can be represented using less than about 100 bytes (e.g., about 86 bytes, including: 8 bytes for size; 12 bytes for distance to previous landmark, lateral offset, and altitude; 50 bytes for image signature; and 16 bytes for GPS coordinates).
[0404] Thus, for semantic road signs that do not require image signatures, even at a relatively high landmark density of approximately 1 landmark per 50 m, the data density impact on the sparse map 800 can be on the order of approximately 760 bytes per kilometer (e.g., 20 landmarks per km x 38 bytes per landmark = 760 bytes). Even for generic signs that include an image signature component, the data density impact is approximately 1.72 kB per km (e.g., 20 landmarks per km x 86 bytes per landmark = 1720 bytes). For semantic road signs, this equates to approximately 76 kB of data usage per hour for a vehicle traveling at 100 km / hr. For generic signs, this equates to approximately 170 kB of data usage per hour for a vehicle traveling at 100 km / hr.
[0405] In some embodiments, a generic rectangular object, such as a rectangular landmark, can be represented in the sparse map 800 using no more than 100 bytes of data. The representation of the generic rectangular object (e.g., generic landmark 1040) in the sparse map 800 can include a condensed image signature (e.g., condensed image signature 1045) associated with the generic rectangular object. This condensed image signature can be used, for example, to assist in the identification of the generic landmark (e.g., a recognized landmark). Such a condensed image signature (e.g., image information derived from actual image data representing the object) can avoid the need to store the actual image of the object or the need to perform comparative image analysis on the actual image to identify the landmark.
[0406] refer to Figure 10 , the sparse map 800 may include or store a condensed image signature 1045 associated with the universal sign 1040, rather than an actual image of the universal sign 1040. For example, after an image capture device (e.g., image capture device 122, 124, or 126) captures an image of the universal sign 1040, a processor (e.g., image processor 190 and any other processor capable of processing images located on or remotely from the host vehicle) may perform image analysis to extract / create a condensed image signature 1045 that includes a unique signature or pattern associated with the universal sign 1040. In one embodiment, the condensed image signature 1045 may include a shape, a color pattern, a brightness pattern, or any other feature that can be extracted from an image of the universal sign 1040 to describe the universal sign 1040. For example, in Figure 10 , the circles, triangles, and stars shown in the condensed image signature 1045 can represent areas of different colors. The patterns represented by the circles, triangles, and stars can be stored in the sparse map 800, for example, within the 50 bytes designated for comprising the image signature. It is worth noting that the circles, triangles, and stars are not necessarily intended to indicate that such shapes are stored as part of the image signature. Instead, these shapes are intended to conceptually represent recognizable areas with discernible color differences, text areas, image shapes, or variations in other characteristics that can be associated with a universal sign. Such condensed image signatures can be used to identify landmarks in the form of universal signs. For example, the condensed image signatures can be used to perform a same-not-same analysis based on comparing the stored condensed image signatures with, for example, image data captured using a camera on an autonomous vehicle.
[0407] Returning to the target trajectory that the host vehicle can use to navigate a specific road segment, Figure 11A800. A polynomial representation of a target trajectory captured during the process of building or maintaining a sparse map 800 is shown. The polynomial representation of a target trajectory included in the sparse map 800 can be determined based on two or more reconstructed trajectories of a vehicle previously traversed along the same road segment. In some embodiments, the polynomial representation of a target trajectory included in the sparse map 800 can be an aggregation of two or more reconstructed trajectories of a vehicle previously traversed along the same road segment. In some embodiments, the polynomial representation of a target trajectory included in the sparse map 800 can be an average of two or more reconstructed trajectories of a vehicle previously traversed along the same road segment. Other mathematical operations can also be used to construct a target trajectory along a road path based on reconstructed trajectories collected from vehicles traveling along a road segment.
[0408] like Figure 11A As shown in , road section 1100 can be driven by several vehicles 200 at different times. Each vehicle 200 can collect data related to the path it takes along the road section. The path driven by a specific vehicle can be determined based on camera data, accelerometer information, rate sensor information, and / or GPS information, and other potential sources. Such data can be used to reconstruct the trajectory of the vehicle traveling along the road section, and based on the trajectory of these reconstructions, a target trajectory (or multiple target trajectories) for a specific road section can be determined. Such a target trajectory can represent its preferred path when the host vehicle (e.g., guided by an autonomous navigation system) travels along the road section.
[0409] In such Figure 11A In the example shown, a first reconstructed trajectory 1101 can be determined based on data received from a first vehicle traversing road segment 1100 during a first time period (e.g., day 1), a second reconstructed trajectory 1102 can be determined based on data received from a second vehicle traversing road segment 1100 during a second time period (e.g., day 2), and a third reconstructed trajectory 1103 can be determined based on data received from a third vehicle traversing road segment 1100 during a third time period (e.g., day 3). Each of trajectories 1101, 1102, and 1103 can be represented by a polynomial, such as a three-dimensional polynomial. It should be noted that in some embodiments, any of the reconstructed trajectories can be assembled on the vehicle traversing road segment 1100.
[0410] Additionally or alternatively, such reconstructed trajectories may be determined on the server side based on information received from vehicles traversing the road segment 1100. For example, in some embodiments, the vehicles 200 may send data to one or more servers regarding their movement along the road segment 1100 (e.g., steering angle, heading, time, location, velocity, sensed road geometry, and / or sensed landmarks, etc.). The server may reconstruct a trajectory for the vehicle 200 based on the received data. The server may also generate a target trajectory for guiding navigation of an autonomous vehicle based on the first trajectory 1101, the second trajectory 1102, and the third trajectory 1103, which autonomous vehicle will later travel along the same road segment 1100. Although a target trajectory may be associated with a single prior traversal of the road segment, in some embodiments, each target trajectory included in the sparse map 800 may be determined based on two or more reconstructed trajectories of vehicles traversing the same road segment. Figure 11A , the target trajectory is represented by 1110. In some embodiments, the target trajectory 1110 may be generated based on an average of the first trajectory 1101, the second trajectory 1102, and the third trajectory 1103. In some embodiments, the target trajectory 1110 included in the sparse map 800 may be an aggregation (e.g., a weighted combination) of two or more reconstructed trajectories.
[0411] Figure 11B and Figure 11C Also shown is the concept of target trajectories associated with road segments presented within the geographic area 1111. Figure 11B As shown in FIG, a first road segment 1120 within geographic area 1111 may include a multi-lane road including two lanes 1122 designated for vehicle travel in a first direction and two additional lanes 1124 designated for vehicle travel in a second direction opposite the first direction. Lanes 1122 and 1124 may be separated by a double yellow line 1123. Geographic area 1111 may also include a branch road segment 1130 intersecting road segment 1120. Road segment 1130 may include a two-lane road, each lane designated for a different direction of travel. Geographic area 1111 may also include other road features such as a stop line 1132, a stop sign 1134, a speed limit sign 1136, and a hazard sign 1138.
[0412] like Figure 11CAs shown in FIG, sparse map 800 may include local map 1140, which includes a road model for assisting autonomous navigation of a vehicle within geographic area 1111. For example, local map 1140 may include target tracks for one or more lanes associated with road segment 1120 and / or road segment 1130 within geographic area 1111. For example, local map 1140 may include target track 1141 and / or target track 1142, which may be accessed or relied upon when the autonomous vehicle crosses lane 1122. Similarly, local map 1140 may include target track 1143 and / or target track 1144, which may be accessed or relied upon when the autonomous vehicle crosses lane 1124. Additionally, local map 1140 may include target track 1145 and / or target track 1146, which may be accessed or relied upon when the autonomous vehicle crosses lane 1130. Target trajectory 1147 represents the preferred path that the autonomous vehicle should follow when transitioning from lane 1120 (and specifically, relative to target trajectory 1140 associated with the rightmost lane of lane 1120) to road segment 1130 (and specifically, relative to target trajectory 1145 associated with the first side of road segment 1130). Similarly, target trajectory 1148 represents the preferred path that the autonomous vehicle should follow when transitioning from road segment 1130 (and specifically, relative to target trajectory 1146) to a portion of road segment 1124 (and specifically, as shown, relative to target trajectory 1143 associated with the left lane of lane 1124).
[0413] Sparse map 800 may also include representations of other road-related features associated with geographic area 1111. For example, sparse map 800 may also include representations of one or more landmarks identified in geographic area 1111. Such landmarks may include a first landmark 1150 associated with stop line 1132, a second landmark 1152 associated with stop sign 1134, a third landmark associated with speed limit sign 1154, and a fourth landmark 1156 associated with hazard sign 1138. Such landmarks may be used, for example, to assist the autonomous vehicle in determining its current position relative to any target track shown, so that the vehicle can adjust its heading to match the direction of the target track at the determined location.
[0414] In some embodiments, the sparse map 800 may also include road signature profiles. Such road signature profiles may be associated with any discernible / measurable change in at least one parameter associated with a road. For example, in some cases, such profiles may be associated with a change in surface roughness of a particular road segment, a change in road width along a particular road segment, a change in the distance between dashed lines drawn along a particular road segment, a change in road curvature along a particular road segment, and the like. Figure 11DAn example of a road signature profile 1160 is shown. While profile 1160 can represent any of the aforementioned parameters, or others, in one example, profile 1160 can represent a measurement of road surface roughness, such as obtained by monitoring one or more sensors that provide an indication of suspension displacement as a vehicle travels along a particular road segment. Alternatively, profile 1160 can represent variations in road width, as determined based on image data obtained by a camera onboard the vehicle while traversing the particular road segment. Such profiles are useful, for example, in determining a particular position of the host vehicle relative to a particular target trajectory. That is, as the ego vehicle traverses the road segment, it can measure a profile associated with one or more parameters associated with the road segment. If the measured profile can be correlated / matched with a predetermined profile that plots variations in the parameters relative to position along the road segment, the measured and predetermined profiles can be used (e.g., by overlaying corresponding portions of the measured and predetermined profiles) to determine the current position along the road segment, and therefore relative to the target trajectory for the road segment.
[0415] In some embodiments, the sparse map 800 may include different tracks based on different characteristics associated with the user of the autonomous vehicle, environmental conditions, and / or other parameters related to driving. For example, in some embodiments, different tracks may be generated based on different user preferences and / or profiles. Sparse maps 800 including such different tracks may be provided to different autonomous vehicles of different users. For example, some users prefer to avoid toll roads, while other users may prefer to take the shortest or fastest route regardless of whether there are toll roads on the route. The disclosed system may generate different sparse maps with different tracks based on such different user preferences or profiles. As another example, some users may prefer to drive in a fast-moving lane, while other users may prefer to always maintain a position in the center lane.
[0416] Based on different environmental conditions such as day and night, snow, rain, fog, etc., different trajectories can be generated and included in the sparse map 800. The sparse map 800 generated based on such different environmental conditions can be provided to an autonomous vehicle driving under different environmental conditions. In some embodiments, a camera provided on the autonomous vehicle can detect the environmental conditions and such information can be provided back to a server that generates and provides the sparse map. For example, the server can generate or update an already generated sparse map 800 to include trajectories that may be more suitable or safer for autonomous driving under the detected environmental conditions. The update of the sparse map 800 based on the environmental conditions can be performed dynamically while the autonomous vehicle is driving along the road.
[0417] Other different parameters related to driving can also be used as the basis for generating and providing different sparse maps to different autonomous vehicles. For example, when the autonomous vehicle is traveling at high speed, the turns may be tighter. Tracks associated with specific lanes rather than roads can be included in the sparse map 800 so that when the autonomous vehicle follows a specific track, it can maintain within the specific lane. When images captured by a camera on the autonomous vehicle indicate that the vehicle has drifted outside the lane (e.g., crossed a lane marking), actions can be triggered within the vehicle based on the specific track to bring the vehicle back to the designated lane.
[0418] Building road models for autonomous vehicle navigation
[0419] In some embodiments, the disclosed systems and methods can construct a road model for autonomous vehicle navigation. For example, the road model can include crowdsourced data. The disclosed systems and methods can refine the crowdsourced data based on observed local conditions. Additionally, the disclosed systems and methods can determine a refined trajectory for the autonomous vehicle based on sensor information. Furthermore, the disclosed systems and methods can identify landmarks for use in the road model and refine the locations of landmarks in the road model. The following sections disclose these systems and methods in more detail.
[0420] Crowdsourced data for autonomous vehicle navigation
[0421] In some embodiments, the disclosed systems and methods can construct road models for autonomous vehicle navigation. For example, the disclosed systems and methods can use crowdsourced data to generate autonomous vehicle road models that one or more autonomous vehicles can use to systematically navigate along a road. By crowdsourced, it is meant that data is received from various vehicles (e.g., autonomous vehicles) traveling on a road segment at different times, and such data can be used to generate and / or update the road model. In turn, the model can be sent to vehicles or other vehicles traveling later along the road segment to assist in autonomous vehicle navigation. The road model can include multiple target trajectories representing preferred trajectories that the autonomous vehicles should follow when traversing the road segment. The target trajectory can be the same as the reconstructed actual trajectory collected from vehicles traversing the road segment, which can be sent from the vehicle to a server. In some embodiments, the target trajectory can be different from the actual trajectory previously adopted by one or more vehicles when traversing the road segment. The target trajectory can be generated based on the actual trajectory (e.g., by averaging or other suitable operations).
[0422] The vehicle trajectory data that a vehicle may upload to a server may correspond to the vehicle's actual reconstructed trajectory, or it may correspond to a recommended trajectory that may be based on or related to the vehicle's actual reconstructed trajectory, but may differ from the actual reconstructed trajectory. For example, a vehicle may modify its actual reconstructed trajectory and submit (e.g., recommend) the modified actual trajectory to the server. The road model may use the recommended, modified trajectory as a target trajectory for autonomous navigation of other vehicles.
[0423] In addition to trajectory information, other information potentially used in constructing the sparse data map 800 may include information related to potential landmark candidates. For example, by crowdsourcing information, the disclosed systems and methods can identify potential landmarks in the environment and refine landmark locations. Landmarks can be used by the autonomous vehicle's navigation system to determine and / or adjust the vehicle's position along a target trajectory.
[0424] The reconstructed trajectory that a vehicle can generate as it travels along a road can be obtained by any suitable method. In some embodiments, the reconstructed trajectory can be developed by integrating segments of the vehicle's motion using, for example, ego-motion estimates (e.g., the three-dimensional translation and three-dimensional rotation of the camera, and the three-dimensional translation and three-dimensional rotation of the vehicle body estimated therefrom). The rotation and translation estimates can be determined based on analysis of images captured by one or more image capture devices together with information from other sensors or devices such as inertial sensors and rate sensors. For example, the inertial sensors can include accelerometers, or other suitable sensors configured to measure changes in translation and / or rotation of the vehicle body. The vehicle can include a rate sensor that measures the vehicle's rate.
[0425] In some embodiments, the camera's ego-motion (and thus the vehicle's ego-motion) can be estimated based on optical flow analysis of the captured images. Optical flow analysis of a sequence of images identifies the movement of pixels from the sequence of images, and the vehicle's motion is determined based on the identified movement. The ego-motion can be integrated over time and along a road segment to reconstruct a target trajectory associated with the road segment that the vehicle has followed.
[0426] The data collected by multiple vehicles at different times along a road segment (e.g., reconstructed trajectories) can be used to reconstruct a road model (e.g., including target trajectories, etc.) included in the sparse map 800. The data collected by multiple vehicles at different times along a road segment can also be averaged to increase the accuracy of the model. In some embodiments, data about road geometry and / or landmarks can be received from multiple vehicles traveling through a common road segment at different times. Such data received from different vehicles can be combined to generate a road model and / or to update the road model.
[0427] The disclosed systems and methods can enable autonomous vehicle navigation (e.g., steering control) using low-occupancy models that can be collected by the autonomous vehicles themselves without the assistance of expensive survey equipment. To support autonomous navigation (e.g., steering applications), a road model can include the geometry of the road, its lane structure, and landmarks that can be used to determine the position or location of the vehicle along a trajectory included in the model. Generation of the road model can be performed by a remote server that communicates with and receives data from vehicles traveling on the road. The data can include sensed data, a trajectory reconstructed based on the sensed data, and / or a recommended trajectory that can represent a modified reconstructed trajectory. The server can send the model back to the vehicle or other vehicles traveling on the road later to assist in autonomous navigation.
[0428] The geometry of the reconstructed trajectory (and target trajectory) along the road segment can be represented by a curve in three-dimensional space, which can be a spline connecting three-dimensional polynomials. The reconstructed trajectory curve can be determined from an analysis of a video stream or multiple images captured by a camera mounted on the vehicle. In some embodiments, a position a few meters ahead of the vehicle's current position is identified in each frame or image. This position is the location to which the vehicle is expected to travel within a predetermined time period. This operation can be repeated frame by frame, and at the same time, the vehicle can calculate the camera's ego motion (rotation and translation). In each frame or each image, a short-range model of the desired path is generated by the vehicle in a reference frame attached to the camera. The short-range models can be integrated together to obtain a three-dimensional model of the road in the same coordinate system, which can be an arbitrary or predetermined coordinate system. The three-dimensional model of the road can then be fitted by a spline, which can include or connect one or more polynomials of a suitable order.
[0429] To infer the short-range road model at each frame, one or more detection modules can be used. For example, a bottom-up lane detection module can be used. The bottom-up lane detection module can be useful when lane markings are painted on the road. This module can find edges in the image and assemble them together to form lane markings. A second module can be used together with the bottom-up lane detection module. The second module is an end-to-end deep neural network that can be trained to predict the correct short-range path from the input image. In both the first module and the second module, the road model can be detected in the image coordinate system and transformed into a three-dimensional space that can be virtually attached to the camera.
[0430] Although the reconstructed trajectory modeling approach may introduce an accumulation of errors due to the integration of ego motion over long time periods, which may include a noise component, such errors may be negligible when the generated model can provide for navigation at a local scale. In addition, it is possible to eliminate the integrated errors by using external information sources such as satellite imagery or geodetic measurements. For example, the disclosed systems and methods may use a GNSS receiver to eliminate accumulated errors. However, GNSS position signals may not always be available and accurate. The disclosed systems and methods may enable steering applications that are weakly dependent on the availability and accuracy of GNSS position. In such a system, the use of GNSS signals may be limited. For example, in some embodiments, the disclosed system may use GNSS signals only for database indexing purposes.
[0431] In some embodiments, a range scale (e.g., a local scale) that may be relevant for autonomous vehicle navigation and steering applications may be approximately 50 meters, 100 meters, 200 meters, 300 meters, etc. Such distances may be used when the geometric road model is primarily used for two purposes: planning a forward trajectory and localizing the vehicle on the road model. In some embodiments, when a control algorithm steers the vehicle based on a target point located 1.3 seconds ahead (or any other time, such as 1.5 seconds, 1.7 seconds, 2 seconds, etc.), the planning task may use the model at a typical range of 40 meters ahead (or any other suitable forward distance, such as 20 meters, 30 meters, 50 meters). According to a method called "tail alignment" described in more detail in another section, the localization task may use the road model at a typical range of 60 meters behind the car (or any other suitable distance, such as 50 meters, 100 meters, 150 meters, etc.). The disclosed systems and methods can generate a geometric model with sufficient accuracy over a specific range, such as 100 meters, so that the planned trajectory will not deviate from the center of the lane by, for example, 30 cm.
[0432] As explained above, a three-dimensional road model can be constructed by detecting short-range segments and integrating them. This integration can be accomplished by calculating a six-degree ego-motion model using video and / or images captured by a camera, data from inertial sensors reflecting the vehicle's motion, and the host vehicle's speed signal. The accumulated error can be sufficiently small on some local mileage scale, such as approximately 100 meters. All of this can be accomplished in a single drive along a specific road segment.
[0433] In some embodiments, multiple driving can be used to average the model obtained to further increase its accuracy. The same car can travel the same route multiple times, or multiple cars can transmit their collected model data to a central server. In any case, a matching program can be performed to identify overlapping models and enable averaging to generate a target trajectory. Once the convergence criterion is met, the constructed model (e.g., including the target trajectory) can be used for steering. Subsequent driving can be used for further model improvement and to adapt to changes in infrastructure.
[0434] If multiple cars are connected to a central server, sharing driving experiences (such as sensor data) between them becomes feasible. Each vehicle client can store a partial copy of the common road model, which may be relevant to the current locat...
Claims
1. A method for processing collected navigation information for use in autonomous vehicle navigation, the method comprising: processing, by a mapping server, navigation information collected from a plurality of vehicles and obtained by sensors coupled to the plurality of vehicles, wherein the navigation information relates to road segments; collecting, by the plurality of vehicles, data regarding landmark locations of landmarks identified near the road segment, the landmarks comprising traffic signs; generating, by the mapping server, an autonomous vehicle map of the road segment by clustering data from the collected navigation information and from the landmark locations of the plurality of vehicles, wherein the autonomous vehicle map includes splines corresponding to lanes in the road segment, wherein the autonomous vehicle map includes landmarks identified near the road segment, and wherein clustering the data accounts for variance in data occurring with the landmark locations of the same landmark identified by a plurality of the plurality of vehicles; and The autonomous vehicle map is distributed by the mapping server to the autonomous vehicle for autonomous navigation on the road segment, wherein the autonomous navigation aligns a trajectory of the autonomous vehicle through the road segment with a spline of the autonomous vehicle map based at least on the positioning of the autonomous vehicle relative to the landmark.
2. The method according to claim 1, wherein the navigation information includes road geometry data.
3. The method according to claim 1, wherein: The plurality of vehicles includes an autonomous vehicle.
4. The method according to claim 1, wherein: The sensor data obtained by the sensor includes road geometry data.
5. At least one machine-readable medium comprising instructions for processing collected navigation information for autonomous vehicle navigation, the instructions, when executed by a mapping server, causing the mapping server to perform operations comprising: processing navigation information collected from a plurality of vehicles and obtained by sensors coupled to the plurality of vehicles, wherein the navigation information relates to road segments; collecting, by the plurality of vehicles, data regarding landmark locations of landmarks identified near the road segment, the landmarks comprising traffic signs; generating an autonomous vehicle map of the road segment by clustering data from the collected navigation information and landmark locations from the plurality of vehicles, wherein the autonomous vehicle map includes splines corresponding to lanes in the road segment, wherein the autonomous vehicle map includes landmarks identified near the road segment, and wherein clustering the data accounts for variance in data occurring with the landmark locations of the same landmark identified by a plurality of the plurality of vehicles; and The autonomous vehicle map is distributed to an autonomous vehicle for autonomous navigation on the road segment, wherein the autonomous navigation aligns a trajectory of the autonomous vehicle through the road segment with a spline of the autonomous vehicle map based at least on the positioning of the autonomous vehicle relative to the landmark.
6. The at least one machine-readable medium of claim 5, the navigation information comprising road geometry data.
7. The at least one machine-readable medium of claim 5, wherein: The plurality of vehicles includes an autonomous vehicle.
8. The at least one machine-readable medium of claim 5, wherein: The sensor data obtained by the sensor includes road geometry data.
9. A mapping server for processing collected navigation information for autonomous vehicle navigation, the mapping server comprising: processor; as well as a memory device containing instructions that, when executed by the processor, cause the processor to: processing navigation information collected from a plurality of vehicles and obtained by sensors coupled to the plurality of vehicles, wherein the navigation information relates to road segments; collecting, by the plurality of vehicles, data regarding landmark locations of landmarks identified near the road segment, the landmarks comprising traffic signs; generating an autonomous vehicle map of the road segment by clustering data from the collected navigation information and landmark locations from the plurality of vehicles, wherein the autonomous vehicle map includes splines corresponding to lanes in the road segment, wherein the autonomous vehicle map includes landmarks identified near the road segment, and wherein clustering the data accounts for variance in data occurring with the landmark locations of the same landmark identified by a plurality of the plurality of vehicles; and The autonomous vehicle map is distributed to an autonomous vehicle for autonomous navigation on the road segment, wherein the autonomous navigation aligns a trajectory of the autonomous vehicle through the road segment with a spline of the autonomous vehicle map based at least on the positioning of the autonomous vehicle relative to the landmark.
10. The mapping server of claim 9, wherein the navigation information comprises road geometry data.
11. The mapping server according to claim 9, wherein: The plurality of vehicles includes an autonomous vehicle.
12. The mapping server according to claim 9, wherein: The sensor data obtained by the sensor includes road geometry data.
13. A vehicle system for using collected navigation information to assist in autonomous vehicle navigation, comprising: processor; as well as a memory device containing instructions that, when executed by the processor, cause the processor to: accessing navigation information at the vehicle system, the navigation information being derived from sensor data obtained via a plurality of sensors coupled to the vehicle system; providing the navigation information to a mapping server, the navigation information being processed by the mapping server as crowdsourced navigation information from a plurality of vehicles obtained by sensors coupled to the plurality of vehicles, wherein the navigation information relates to a road segment and landmark locations of landmarks identified near the road segment; and An autonomous vehicle map is received, the autonomous vehicle map being created by clustering landmark locations from the plurality of vehicles, the autonomous vehicle map including splines corresponding to lanes in the road segment, and the autonomous vehicle map including landmarks identified near the road segment for autonomous navigation on the road segment using the autonomous vehicle map, wherein clustering the data resolves variance in data occurring with the landmark locations of the same landmark identified by a plurality of the plurality of vehicles, and wherein the autonomous navigation aligns a trajectory of the autonomous vehicle through the road segment with the splines of the autonomous vehicle map based at least on a positioning of the autonomous vehicle relative to the landmarks.
14. The vehicle system according to claim 13, wherein: Navigation information includes road geometry data.
15. The vehicle system according to claim 13, wherein: The plurality of vehicles includes an autonomous vehicle.
16. The vehicle system according to claim 13, wherein: The sensor data obtained by the sensor includes road geometry data.
17. At least one machine-readable device comprising instructions for using collected navigation information to assist in autonomous vehicle navigation, the instructions, when executed by a vehicle system, causing the vehicle system to perform operations comprising: accessing navigation information at the vehicle system, the navigation information being derived from sensor data obtained via a plurality of sensors coupled to the vehicle system; providing the navigation information to a mapping server, the navigation information being processed by the mapping server as collected navigation information from a plurality of vehicles obtained by sensors coupled to the plurality of vehicles, wherein the navigation information relates to road segments and landmark locations of landmarks identified near the road segments; and receiving an autonomous vehicle map created by clustering locations of landmarks from the plurality of vehicles, the autonomous vehicle map comprising splines corresponding to lanes in the road segment, and the autonomous vehicle map comprising landmarks identified near the road segment for autonomous navigation on the road segment using the autonomous vehicle map, wherein clustering the data accounts for variance in data occurring together with the landmark locations of the same landmark identified by a plurality of the plurality of vehicles, and Wherein, the autonomous navigation aligns the trajectory of the autonomous vehicle through the road section with the spline of the autonomous vehicle map based at least on the positioning of the autonomous vehicle relative to the landmark.
18. At least one machine-readable device according to claim 17, wherein: Navigation information includes road geometry data.
19. The at least one machine-readable device of claim 17, wherein: The plurality of vehicles includes an autonomous vehicle.
20. The at least one machine-readable device of claim 17, wherein: The sensor data obtained by the sensor includes road geometry data.
21. A vehicle system for assisting autonomous vehicle navigation, the vehicle system being installed in a vehicle and comprising: processor; as well as a memory comprising instructions that, when executed by the processor, cause the processor to: accessing a sparse map generated from collected data, the sparse map generated from the collected data comprising a spline representation of a lane marking and metadata describing a lane represented by the lane marking, and the sparse map generated from the collected data being created by clustering data obtained from a plurality of vehicles regarding locations of landmarks relative to a road segment, wherein clustering the data accounts for variance in data occurring together with the locations of the landmarks for the same landmark identified by a plurality of the plurality of vehicles; obtaining image data corresponding to an image from a camera, the image corresponding to an environment external to the vehicle; and The sparse map and image data generated by the collected data are analyzed to determine an autonomous navigation response of the vehicle, wherein the autonomous navigation response aligns a trajectory of the autonomous vehicle through the road segment with a spline of the sparse map generated by the collected data based at least on the positioning of the autonomous vehicle relative to the landmark.
22. The system of claim 21, wherein: The plurality of vehicles includes an autonomous vehicle.
23. At least one machine-readable medium comprising instructions for assisting autonomous vehicle navigation of a vehicle, the instructions, when executed by a machine, causing the machine to perform operations comprising: accessing a sparse map generated from collected data, the sparse map generated from the collected data comprising a spline representation of a lane marking and metadata describing a lane represented by the lane marking, and the sparse map generated from the collected data being created by clustering data obtained from a plurality of vehicles regarding locations of landmarks relative to a road segment, wherein clustering the data accounts for variance in data occurring together with the locations of the landmarks for the same landmark identified by a plurality of the plurality of vehicles; obtaining image data corresponding to an image from a camera, the image corresponding to an environment external to the vehicle; and The sparse map and image data generated by the collected data are analyzed to determine an autonomous navigation response of the vehicle, wherein the autonomous navigation response aligns a trajectory of the autonomous vehicle through the road segment with a spline of the sparse map generated by the collected data based at least on the positioning of the autonomous vehicle relative to the landmark.
24. The at least one machine-readable medium of claim 23, wherein: The plurality of vehicles includes an autonomous vehicle.
25. An autonomous vehicle comprising: ontology; A camera mounted on the body; as well as Processing Unit: accessing a sparse map generated from collected data, the sparse map generated from the collected data comprising a spline representation of a lane marking and metadata describing a lane represented by the lane marking, and the sparse map generated from the collected data being created by clustering data obtained from a plurality of vehicles regarding locations of landmarks relative to a road segment, wherein clustering the data accounts for variance in data occurring together with the locations of the landmarks for the same landmark identified by a plurality of the plurality of vehicles; obtaining image data corresponding to an image from a camera, the image corresponding to an environment external to the autonomous vehicle; and The sparse map and image data generated by the collected data are analyzed to determine an autonomous navigation response of the autonomous vehicle, wherein the autonomous navigation response aligns a trajectory of the autonomous vehicle through the road segment with a spline of the sparse map generated by the collected data based at least on the positioning of the autonomous vehicle relative to the landmark.
26. The autonomous vehicle of claim 25, further comprising a throttling system.
27. The autonomous vehicle of claim 25, further comprising a braking system.
28. The autonomous vehicle of claim 25, further comprising a steering system.
29. The autonomous vehicle of claim 25, wherein: The plurality of vehicles includes an autonomous vehicle.
