Vehicle apparatus for controlling the operation of a vehicle, a method performed by a vehicle navigation control system, a vehicle, a program, and at least one machine-readable storage medium

Abstract

To provide a system and a method for constructing, using, and updating a sparse map for autonomous vehicle navigation.SOLUTION: A device of a vehicle for controlling an operation of the vehicle, comprises: a processor; and a memory including instructions. When the device is executed by the processor, it causes the processor to: access routing information for navigation of a route by the vehicle, the routing information including data related to a speed along the route; calculate the speed of the vehicle along the route on the basis of the routing information; drive the vehicle at the speed calculated along the route; acquire dynamic information on the route, the dynamic information being based on the data collected from one or more other vehicles on the route, and the dynamic information indicating a current state on the route which affects the speed of the vehicle along the route; and drive the vehicle at the speed updated along the route on the basis of the dynamic information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 62 / 114,091, filed February 10, 2015, U.S. Provisional Patent Application No. 62 / 164,055, filed May 20, 2015, U.S. Provisional Patent Application No. 62 / 170,728, filed June 4, 2015, U.S. Provisional Patent Application No. 62 / 181,784, filed June 19, 2015, U.S. Provisional Patent Application No. 62 / 192,576, filed July 15, 2015, and U.S. Provisional Patent Application No. 62 / 192,576, filed July 15, 2015. U.S. Provisional Patent Application No. 62 / 215,764 filed September 9, 2015, U.S. Provisional Patent Application No. 62 / 219,733 filed September 17, 2015, U.S. Provisional Patent Application No. 62 / 261,578 filed December 1, 2015, U.S. Provisional Patent Application No. 62 / 261,598 filed December 1, 2015, U.S. Provisional Patent Application No. 62 / 267,643 filed December 15, 2015 U.S. Provisional Patent Application No. 62 / 269,818 filed on December 18, 2015; U.S. Provisional Patent Application No. 62 / 270,408 filed on December 21, 2015; U.S. Provisional Patent Application No. 62 / 270,418 filed on December 21, 2015; U.S. Provisional Patent Application No. 62 / 270,431 filed on December 21, 2015; U.S. Provisional Patent Application No. 62 / 271,103 filed on January 5, 2016 This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 274,883 filed on Jan. 5, 2016, U.S. Provisional Patent Application No. 62 / 274,968 filed on Jan. 5, 2016, U.S. Provisional Patent Application No. 62 / 275,007 filed on Jan. 5, 2016, U.S. Provisional Patent Application No. 62 / 275,046 filed on Jan. 5, 2016, and U.S. Provisional Patent Application No. 62 / 277,068 filed on Jan. 11, 2016, all of which are incorporated herein by reference in their entireties.

[0002] Background Technology Field The present disclosure relates generally to autonomous vehicle navigation and sparse maps for autonomous vehicle navigation. Additionally, the present disclosure relates to systems and methods for building, using, and updating sparse maps for autonomous vehicle navigation. [Background technology]

[0003] Background information As technology continues to evolve, the goal of fully autonomous vehicles capable of navigating roads becomes more realistic. An autonomous vehicle may need to consider various factors and, based on those factors, make appropriate decisions to safely and accurately reach its intended destination. For example, an autonomous vehicle may need to process and interpret visual information (e.g., information captured from a camera) and may also use information obtained from other sources (e.g., from a GPS device, speed sensors, accelerometers, suspension sensors, etc.). At the same time, to navigate to its destination, an autonomous vehicle may also need to identify its position within a particular road (e.g., a particular lane within a multi-lane road), navigate alongside other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, and navigate from one road to another at appropriate intersections or interchanges. Utilizing and interpreting the vast amount of information collected by an autonomous vehicle as it travels to its destination poses many design challenges. The vast amount of data that an autonomous vehicle may need to analyze, access, and / or store (e.g., captured image data, map data, GPS data, sensor data, etc.) poses challenges that may actually limit or even adversely affect autonomous navigation. Furthermore, if an autonomous vehicle relies on traditional mapping technology to navigate, the vast amount of data required to store and update maps poses difficult challenges. Summary of the Invention [Means for solving the problem]

[0004] overview Embodiments according to the present disclosure provide systems and methods for autonomous vehicle navigation. The disclosed embodiments may use cameras to provide autonomous vehicle navigation features. For example, according to embodiments of the present disclosure, the disclosed system may include one, two, or more cameras that monitor the vehicle's environment. The disclosed system may provide a navigation response based on, for example, analysis of images captured by one or more of the cameras. The navigation response may also consider other data, including, for example, global positioning (GPS) data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data.

[0005] In some embodiments, the disclosed systems and methods may use sparse maps for autonomous vehicle navigation, e.g., sparse maps may provide sufficient information for navigation without requiring excessive data storage.

[0006] In other embodiments, the disclosed systems and methods may build road models for autonomous vehicle navigation. For example, the disclosed systems and methods may use crowd-sourced data for autonomous vehicle navigation, including recommended trajectories. As another example, the disclosed systems and methods may identify landmarks in the vehicle's environment and refine landmark locations.

[0007] In still other embodiments, the disclosed systems and methods may use sparse road models for autonomous vehicle navigation. For example, the disclosed systems and methods may provide navigation based on recognized landmarks, align a vehicle's tail for navigation, enable a vehicle to navigate at intersections, enable a vehicle to navigate using overlapping local maps, enable a vehicle to navigate using sparse maps, navigate based on expected landmark locations, autonomously navigate roads based on road signatures, provide forward navigation based on a rear-facing camera, navigate based on free space identification, navigate through snow, provide autonomous vehicle speed calibration, identify lane assignments based on recognized landmark locations, and use superlandmarks as a navigation aid.

[0008] In still other embodiments, the disclosed systems and methods may provide adaptive autonomous navigation, for example, the disclosed systems and methods may provide adaptive navigation based on user intervention, provide self-aware adaptive navigation, provide an adaptive road model manager, and manage road models based on selective feedback.

[0009] In some embodiments, a non-transitory computer-readable medium may include a sparse map for autonomous vehicle navigation along a road segment. The sparse map may include a polynomial representation of a desired trajectory of the autonomous vehicle along the road segment and a plurality of predetermined landmarks associated with the road segment, the plurality of predetermined landmarks may be spaced apart by at least 50 meters, and the sparse map may have a data density of 1 megabyte per kilometer or less.

[0010] In some embodiments of the non-transitory computer-readable medium, the polynomial representation may be a three-dimensional polynomial representation. The polynomial representation of the target trajectory may be determined based on two or more reconstructed trajectories of past travels of the vehicle along the road segment. The plurality of predetermined landmarks may include traffic signs represented in the sparse map with 50 bytes of data or less. The plurality of predetermined landmarks may include directional signs represented in the sparse map with 50 bytes of data or less. The plurality of predetermined landmarks may include generic signs represented in the sparse map with 100 bytes of data or less. The plurality of predetermined landmarks may include substantially rectangular objects represented in the sparse map with 100 bytes of data or less. The representation of the substantially rectangular objects in the sparse map may include a simplified image signature associated with the substantially rectangular objects. The plurality of predetermined landmarks may be represented in the sparse map by parameters including landmark size, distance to previous landmark, landmark type, and landmark location. The plurality of predetermined landmarks included in the sparse map may be separated by at least 2 kilometers. The plurality of predetermined landmarks included in the sparse map may be spaced apart by at least 1 kilometer. The plurality of predetermined landmarks included in the sparse map may be spaced apart by at least 100 meters. The sparse map may have a data density of 100 kilobytes per kilometer or less. The sparse map may have a data density of 10 kilobytes per kilometer or less. The plurality of predetermined landmarks may appear in the sparse map at a rate greater than sufficient to maintain longitudinal localization accuracy within 1 meter.

[0011] In some embodiments, an autonomous vehicle may include a body and a non-transitory computer-readable medium that may include a sparse map for autonomous vehicle navigation along a road segment. The sparse map may include a polynomial representation of a desired trajectory of the autonomous vehicle along the road segment and a plurality of predetermined landmarks associated with the road segment, the plurality of predetermined landmarks being spaced apart by at least 50 meters, and the sparse map having a data density of 1 megabyte per kilometer or less. The autonomous vehicle may include a processor configured to execute data included in the sparse map to provide autonomous vehicle navigation along the road segment.

[0012] In some embodiments of the autonomous vehicle, the polynomial representation may be a three-dimensional polynomial representation. The polynomial representation of the target trajectory may be determined based on two or more reconstructed trajectories of past travels of the vehicle along the road segment.

[0013] In some embodiments, an autonomous vehicle may include a body and a processor configured to receive data included in a sparse map and execute the data for autonomous vehicle navigation along a road segment, the sparse map may include a polynomial representation of a desired trajectory of the autonomous vehicle along the road segment and a plurality of predetermined landmarks associated with the road segment, the plurality of predetermined landmarks being spaced apart by at least 50 meters, and the sparse map having a data density of 1 megabyte per kilometer or less.

[0014] In some embodiments, a method for processing vehicle navigation information for use in autonomous vehicle navigation may include receiving, by a server, navigation information from a plurality of vehicles. The navigation information from the plurality of vehicles may be associated with a common road segment. The method may include storing, by the server, the navigation information associated with the common road segment. The method may include generating, by the server, at least a portion of an autonomous vehicle road navigation model of the common road segment based on the navigation information from the plurality of vehicles, and distributing, by the server, the autonomous vehicle road navigation model to one or more autonomous vehicles for use in autonomously navigating the one or more autonomous vehicles along the common road segment.

[0015] In some embodiments of the method, the navigation information may include trajectories from each of the multiple vehicles as each vehicle travels across the common road segment. The trajectories may be determined based on detected camera movements, including three-dimensional translational and three-dimensional rotational movements. The navigation information may include lane assignments. Generating at least a portion of the autonomous vehicle road navigation model may include clustering vehicle trajectories along the common road segment and determining a target trajectory along the common road segment based on the clustered vehicle trajectories. The autonomous vehicle road navigation model may include a cubic spline corresponding to the target trajectory along the common road segment. The target trajectory may be associated with one lane of the common road segment. The autonomous vehicle road navigation model may include multiple target trajectories, each target trajectory associated with a distinct lane of the common road segment. Determining the target trajectory along the common road segment based on the clustered vehicle trajectories may include finding a mean value or average trajectory based on the clustered vehicle trajectories. The target trajectory may be represented by a cubic spline. The spline may be defined at less than 10 kilobytes per kilometer. The autonomous vehicle road navigation model may include at least one landmark identification information, including a location of the at least one landmark. The location of the at least one landmark may be determined based on position measurements performed using sensor systems associated with multiple vehicles. The position measurements may be averaged to obtain the location of the at least one landmark. The at least one landmark may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a landmark beacon, or a light pole.

[0016] In some embodiments, a navigation system for a vehicle may include at least one processor, the at least one processor being programmed to receive at least one environmental image associated with the vehicle from a camera, analyze the at least one environmental image to identify navigation information relevant to the vehicle, and transmit the navigation information from the vehicle to a server. The at least one processor may be programmed to receive an autonomous vehicle road navigation model from the server. The autonomous vehicle road navigation model may include at least one update based on the transmitted navigation information. The at least one processor may be programmed to cause at least one navigation maneuver by the vehicle based on the autonomous vehicle road navigation model.

[0017] In some embodiments of the navigation system, the navigation information may include trajectories from each of multiple vehicles as each vehicle travels across a common road segment.

[0018] In some embodiments, a server that processes vehicle navigation information for use in autonomous vehicle navigation may include a communication unit configured to communicate with a plurality of vehicles and at least one processor programmed to receive navigation information from the vehicles via the communication unit. The at least one processor may be programmed to generate at least a portion of an autonomous vehicle road navigation model based on the navigation information, and to transmit at least a portion of the autonomous vehicle road navigation model to at least one of the vehicles to cause a navigation operation by the at least one of the vehicles based on the portion of the autonomous vehicle road navigation model.

[0019] In some embodiments of the server, the navigation information may include trajectories from each of the multiple vehicles as each vehicle travels across a common road segment. Part of the autonomous vehicle road navigation model may include updates to the autonomous vehicle road navigation model.

[0020] In some embodiments, a navigation system for a vehicle may include at least one processor that is programmed to receive output from one or more sensors indicative of vehicle operation, determine an actual trajectory of the vehicle based on the output from the one or more sensors, receive at least one environmental image associated with the vehicle from a camera, analyze the at least one environmental image to determine information associated with at least one navigation constraint, determine a target trajectory that includes the actual trajectory of the vehicle and one or more corrections to the actual trajectory based on the determined information associated with the at least one navigation constraint, and transmit the target trajectory from the vehicle to a server.

[0021] In some embodiments of the system, the one or more sensors may include a speed sensor. The one or more sensors may include an accelerometer. The one or more sensors may include a camera. The at least one navigation constraint may include at least one of a barrier, an object, a lane marking, a sign, or another vehicle. The camera may be included within the vehicle.

[0022] In some embodiments, a method of uploading a target trajectory to a server may include receiving output from one or more sensors indicative of vehicle operation; determining an actual trajectory of the vehicle based on the output from the one or more sensors; receiving at least one environmental image associated with the vehicle from a camera; analyzing the at least one environmental image to determine information associated with at least one navigation constraint; determining a target trajectory including the actual trajectory of the vehicle and one or more corrections to the actual trajectory based on the determined information associated with the at least one navigation constraint; and transmitting the target trajectory from the vehicle to the server.

[0023] In some embodiments of the method, the one or more sensors may include a speed sensor. The one or more sensors may include an accelerometer. The one or more sensors may include a camera. The at least one navigation constraint may include at least one of a barrier, an object, a lane marking, a sign, or another vehicle. The camera may be included within the vehicle.

[0024] In some embodiments, a system for identifying landmarks used in autonomous vehicle navigation may include at least one processor programmed to receive at least one identifier associated with the landmark, associate the landmark with a corresponding road segment, update an autonomous vehicle road navigation model for the corresponding road segment to include the at least one identifier associated with the landmark, and distribute the updated autonomous vehicle road navigation model to a plurality of autonomous vehicles. The at least one identifier may be identified based on acquiring at least one image representing an environment of a host vehicle from a camera associated with the host vehicle, analyzing the at least one image to identify landmarks within the environment of the host vehicle, and analyzing the at least one image to determine the at least one identifier associated with the landmark.

[0025] In some embodiments of the system, the at least one identifier may include a location of the landmark. The at least one identifier may include a shape of the landmark. The at least one identifier may include a size of the landmark. The at least one identifier may include a distance of the landmark relative to another landmark. The at least one identifier may be determined based on identifying the landmark as one of a plurality of landmark types. The landmark type may include a traffic sign. The landmark type may include a pole. The landmark type may include a directional indicator. The landmark type may include a rectangular sign. The at least one identifier may further include a simplified signature representation. The simplified signature representation of the landmark may be determined based on mapping an image of the landmark to a series of numbers of a predetermined data size. The simplified signature representation may indicate an appearance of the landmark. The simplified signature representation may indicate at least one of a color pattern of the image of the landmark or a brightness pattern of the image. The landmark may include at least one of a directional sign, a traffic sign, a lamppost, a road marking, and a business sign.

[0026] In some embodiments, a method for identifying landmarks for use in autonomous vehicle navigation may include receiving at least one identifier associated with the landmark, associating the landmark with a corresponding road segment, updating an autonomous vehicle road navigation model for the corresponding road segment to include the at least one identifier associated with the landmark, and distributing the updated autonomous vehicle road navigation model to a plurality of autonomous vehicles.

[0027] In some embodiments, the method may include identifying at least one identifier. Identifying the at least one identifier may include acquiring, from a camera associated with the host vehicle, at least one image representing an environment of the host vehicle, analyzing the at least one image to identify landmarks within the environment of the host vehicle, and analyzing the at least one image to identify at least one identifier associated with the landmark. The at least one identifier may include a distance of the landmark relative to another landmark, and identifying the at least one identifier may include determining the distance of the landmark relative to the other landmark. The at least one identifier may further include a reduced-weighted signature representation, and identifying the at least one identifier includes determining the reduced-weighted signature representation from the at least one image.

[0028] In some embodiments, a system for determining the location of a landmark used in navigation of an autonomous vehicle may include at least one processor programmed to receive a measured position of the landmark and determine a refined position of the landmark based on the measured position of the landmark and at least one previously acquired position of the landmark. The measured position and the at least one previously acquired position may be determined based on acquiring at least one image of an environment associated with the host vehicle from a camera associated with the host vehicle, analyzing the at least one image of the environment to identify landmarks within the host vehicle's environment, receiving Global Positioning System (GPS) data representative of a position of the host vehicle, analyzing the at least one image of the environment to determine a relative position of the identified landmark with respect to the host vehicle, and determining a globally localized position of the landmark based on at least the GPS data and the determined relative position.

[0029] In some embodiments of the system, the landmark may include at least one of a traffic sign, an arrow, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a landmark beacon, or a light pole. Analyzing the at least one image to determine the relative position of the identified landmark with respect to the vehicle may include calculating a distance based on a scale associated with the at least one image. Analyzing the at least one image to determine the relative position of the identified landmark with respect to the vehicle may include calculating a distance based on an optical flow associated with the at least one image. The GPS data may be received from a GPS device included in the host vehicle. A camera may be included in the host vehicle. Identifying the improved position of the landmark may include averaging the measured position of the landmark with at least one previously obtained position.

[0030] In some embodiments, a method for determining a location of a landmark used in navigation of an autonomous vehicle may include receiving a measured location of the landmark and determining a refined location of the landmark based on the measured location of the landmark and at least one previously acquired location of the landmark. The measured location and the at least one previously acquired location may be determined based on acquiring at least one image of an environment associated with the host vehicle from a camera associated with the host vehicle, analyzing the at least one image of the environment to identify landmarks within the environment of the host vehicle, receiving Global Positioning System (GPS) data representative of a location of the host vehicle, analyzing the at least one image of the environment to determine relative locations of the identified landmarks with respect to the host vehicle, and determining a global location of the landmark based on at least the GPS data and the determined relative location.

[0031] In some embodiments of the method, the landmark may include at least one of a traffic sign, an arrow, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost. Analyzing the at least one image to determine a relative position of the identified landmark with respect to the vehicle may include calculating a distance based on a scale associated with the at least one image. Analyzing the at least one image to determine a relative position of the identified landmark with respect to the vehicle may include calculating a distance based on an optical flow associated with the at least one image. The GPS data may be received from a GPS device included in the host vehicle. The camera may be included in the host vehicle. Identifying the improved position of the landmark may include averaging the measured position of the landmark with at least one previously obtained position.

[0032] In some embodiments, the autonomous vehicle may include a body and at least one processor, where the at least one processor is programmed to receive a measured position of a landmark and determine a refined position of the landmark based on the measured position of the landmark and at least one previously acquired position of the landmark. The at least one processor may be further programmed to determine the measured position and the at least one previously acquired position based on acquiring at least one environmental image associated with the vehicle from a camera associated with the vehicle, analyzing the at least one environmental image to identify landmarks within the vehicle's environment, receiving Global Positioning System (GPS) data representative of a position of the vehicle, analyzing the at least one environmental image to determine a relative position of the identified landmark with respect to the vehicle, and determining a global position of the landmark based at least on the GPS data and the determined relative position.

[0033] In some embodiments of the vehicle, the landmark may include at least one of a traffic sign, an arrow, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost. Analyzing the at least one image to determine the relative position of the identified landmark with respect to the vehicle may include calculating distance based on a scale associated with the at least one image. Analyzing the at least one image to determine the relative position of the identified landmark with respect to the vehicle may include calculating distance based on an optical flow associated with the at least one image. GPS data may be received from a GPS device included in the host vehicle. Determining the refined position of the landmark may include averaging the measured position of the landmark with at least one previously obtained position.

[0034] In some embodiments, a system for autonomously navigating a vehicle along a road segment may include at least one processor programmed to receive, from an image capture device, at least one image representing an environment of the vehicle; analyze the at least one image to identify at least one recognized landmark; determine a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based at least in part on predetermined positions of the recognized landmarks; and determine an autonomous steering action for the vehicle based on an orientation of the predetermined road model trajectory at the determined current position of the vehicle relative to the predetermined road model trajectory.

[0035] In some embodiments of the system, the recognized landmarks may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The recognized landmarks may include line spacing changes on a road segment. The recognized landmarks may include business signs. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. Navigating between recognized landmarks may include integrating vehicle speed to determine a position of the vehicle along the predetermined road model trajectory. The processor may be further programmed to adjust a steering system of the vehicle based on an autonomous steering operation to navigate the vehicle. The processor may be further programmed to determine a distance of the vehicle from at least one recognized landmark and, based on the distance, determine whether the vehicle is located on a predetermined road model trajectory associated with the road segment. The processor may be further programmed to adjust a steering system of the vehicle to move the vehicle from its current position to a position on the predetermined road model trajectory if the vehicle is not located on the predetermined road model trajectory.

[0036] In some embodiments, the vehicle may include a body, at least one image capture device configured to acquire at least one image representing an environment of the vehicle, and at least one processor, the at least one processor being programmed to: receive at least one image from the at least one image capture device; analyze the at least one image to identify at least one recognized landmark; determine a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based at least in part on predetermined positions of the recognized landmarks; and determine an autonomous steering operation of the vehicle based on an orientation of the predetermined road model trajectory at the identified current position of the vehicle relative to the predetermined road model trajectory.

[0037] In some embodiments of the vehicle, the recognized landmarks may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, a lamppost, a change in the spacing of lines on a road, or a business sign. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. Navigating between the recognized landmarks may include integrating vehicle speed to determine a position of the vehicle along the predetermined road model trajectory. The processor may be further programmed to adjust a steering system of the vehicle based on the autonomous steering operation to navigate the vehicle. The processor may be further programmed to determine a distance of the vehicle from at least one recognized landmark and, based on the distance, determine whether the vehicle is located on a predetermined road model trajectory associated with the road segment. The processor may be further programmed to adjust a steering system of the vehicle to move the vehicle from its current position to a position on the predetermined road model trajectory if the vehicle is not located on the predetermined road model trajectory.

[0038] In some embodiments, a method of navigating a vehicle may include receiving at least one image representing an environment of the vehicle from an image capture device associated with the vehicle; analyzing the at least one image using a processor associated with the vehicle to identify at least one recognized landmark; determining a current position of the vehicle relative to a predetermined road model trajectory associated with a road segment based at least in part on predetermined positions of the recognized landmarks; determining an autonomous steering action of the vehicle based on an orientation of the predetermined road model trajectory at the determined current position of the vehicle relative to the predetermined road model trajectory; and adjusting a steering system of the vehicle based on the autonomous steering action to navigate the vehicle.

[0039] In some embodiments, the method may include determining a position of the vehicle along a predetermined road model trajectory by integrating vehicle speed. The method may include using a processor to determine a distance of the vehicle from at least one recognized landmark and, based on the distance, determining whether the vehicle is located on a predetermined road model trajectory associated with a road segment. The method may include determining a transformation required to move the vehicle from the vehicle's current position to a position on the predetermined road model trajectory and adjusting a steering system of the vehicle based on the transformation.

[0040] In some embodiments, a system for autonomously navigating an autonomous vehicle along a road segment may include at least one processor that is programmed to: receive, from an image capture device, a plurality of images representing an environment of the autonomous vehicle; determine a driving trajectory of the autonomous vehicle along the road segment based at least in part on an analysis of one or more of the plurality of images; determine a current position of the autonomous vehicle along a predetermined road model trajectory based on the analysis of the one or more of the plurality of images; determine a heading for the autonomous vehicle based on the determined driving trajectory; and determine a steering direction for the autonomous vehicle relative to the heading direction by comparing the driving trajectory with the predetermined road model trajectory at the current position of the autonomous vehicle.

[0041] In some embodiments of the system, comparing the driving trajectory with the predetermined road model trajectory may include determining a transformation that reduces the error between the driving trajectory and the predetermined road model trajectory. The processor may be further programmed to adjust a steering system of the autonomous vehicle based on the transformation. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along a road segment. The predetermined road model trajectory may be retrieved from a database stored in a memory included in the autonomous vehicle. The predetermined road model trajectory may be retrieved from a database accessible to the autonomous vehicle via a wireless communication interface. The image capture device may be included in the autonomous vehicle. The steering direction determination may be further based on one or more additional cues, including one or more of a left lane marking polynomial model, a right lane marking polynomial model, a comprehensive path prediction, forward vehicle behavior, an identified free space in front of the autonomous vehicle, and a virtual lane or virtual lane constraint identified based on the position of a vehicle in front of the autonomous vehicle. The steering direction determination may be based on weights applied to the one or more additional cues.

[0042] In some embodiments, the autonomous vehicle may include a body, at least one image capture device configured to acquire at least one image representing an environment of the autonomous vehicle, and at least one processor, wherein the at least one processor is programmed to: receive a plurality of images representing an environment of the autonomous vehicle from the image capture device; determine a driving trajectory of the autonomous vehicle along a road segment based at least in part on an analysis of one or more of the plurality of images; determine a current position of the autonomous vehicle along a predetermined road model trajectory based on the analysis of the one or more of the plurality of images; determine a heading for the autonomous vehicle based on the determined driving trajectory; and determine a steering direction for the autonomous vehicle relative to the heading for the autonomous vehicle by comparing the driving trajectory with the predetermined road model trajectory at the current position of the autonomous vehicle.

[0043] In some embodiments of the autonomous vehicle, comparing the driving trajectory with a predetermined road model trajectory may include determining a transformation that reduces the error between the driving trajectory and the predetermined road model trajectory. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along a road segment. The predetermined road model trajectory may be retrieved from one of a database stored in a memory included in the autonomous vehicle and a database accessible to the autonomous vehicle via a wireless communication interface. The steering direction determination may be further based on one or more additional cues, including one or more of a left lane marking polynomial model, a right lane marking polynomial model, a comprehensive path prediction, forward vehicle behavior, an identified free space in front of the autonomous vehicle, and a virtual lane or virtual lane constraints identified based on the position of a vehicle in front of the autonomous vehicle. The steering direction determination may be based on weights applied to the one or more additional cues.

[0044] In some embodiments, a method for navigating an autonomous vehicle may include receiving, from an image capture device, a plurality of images representing an environment of the autonomous vehicle; determining a driving trajectory of the autonomous vehicle along a road segment based at least in part on an analysis of one or more of the plurality of images; determining a current position of the autonomous vehicle along a predetermined road model trajectory based on an analysis of the one or more of the plurality of images; determining a heading for the autonomous vehicle based on the determined driving trajectory; and determining a steering direction for the autonomous vehicle relative to the heading by comparing the driving trajectory with the predetermined road model trajectory at the current position of the autonomous vehicle.

[0045] In some embodiments of the method, comparing the driving trajectory to the predetermined road model trajectory may include determining a transformation that reduces the error between the driving trajectory and the predetermined road model trajectory. The steering direction determination may be further based on one or more additional cues, including one or more of a left lane marking polynomial model, a right lane marking polynomial model, a global path prediction, forward vehicle behavior, an identified free space in front of the autonomous vehicle, and a virtual lane or virtual lane constraints identified based on the position of a vehicle in front of the autonomous vehicle. The steering direction determination may be based on weights applied to the one or more additional cues.

[0046] In some embodiments, a system for autonomously navigating a vehicle through an intersection may include at least one processor that is programmed to: receive at least one image representing an environment of the vehicle from an image capture device; analyze the at least one image to identify two or more landmarks within the environment of the vehicle; determine a directional indicator for the vehicle for each of the two or more landmarks; determine a current position of the vehicle relative to the intersection based on an intersection of the directional indicators of the two or more landmarks; determine a heading of the vehicle based on the directional indicators of the two or more landmarks; and determine a steering angle for the vehicle by comparing the heading of the vehicle to a predetermined road model trajectory at the current position of the vehicle.

[0047] In some embodiments of the system, the predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. The two or more landmarks may include three or more landmarks. The at least one processor may be further programmed to send a control signal specifying a steering angle to a steering system of the vehicle. The processor may be configured to retrieve the predetermined road model trajectory from a database stored in a memory included in the vehicle. The processor may be configured to retrieve the predetermined road model trajectory from a database accessible to the vehicle via a wireless communication interface. The camera may be included in the vehicle. The processor may be further programmed to determine a heading for the vehicle by determining a previous position of the vehicle relative to the intersection based on an intersection of direction indicators of the two or more landmarks, and determining a heading based on the previous position and the current position.

[0048] In some embodiments, the autonomous vehicle may include a body, at least one image capture device configured to acquire at least one image representing the vehicle's environment, and at least one processor, wherein the at least one processor is programmed to: receive at least one image representing the vehicle's environment from a camera; analyze the at least one image to identify two or more landmarks within the vehicle's environment; determine a directional indicator for the vehicle for each of the two or more landmarks; determine a current position of the vehicle relative to an intersection based on an intersection of the directional indicators of the two or more landmarks; determine a heading for the vehicle based on the directional indicators of the two or more landmarks; and determine a steering angle for the vehicle by comparing the vehicle heading to a predetermined road model trajectory at the vehicle's current position.

[0049] In some embodiments of the vehicle, the predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. The two or more landmarks may include three or more landmarks. The at least one processor may be further programmed to send a control signal specifying a steering angle to a steering system of the vehicle. The predetermined road model trajectory may be retrieved from one of a database stored in a memory included in the vehicle and a database accessible to the vehicle via a wireless communication interface. The processor may be further programmed to determine a heading for the vehicle by determining a previous position of the vehicle relative to the intersection based on an intersection of direction indicators of the two or more landmarks, and determining a heading based on the previous position and the current position.

[0050] In some embodiments, a method for navigating an autonomous vehicle may include receiving at least one image representing an environment of the vehicle from an image capture device; analyzing the at least one image using at least one processor to identify two or more landmarks within the environment of the vehicle; determining a directional indicator for the vehicle for each of the two or more landmarks; determining a current position of the vehicle relative to an intersection based on an intersection of the directional indicators of the two or more landmarks; determining a heading for the vehicle based on the directional indicators of the two or more landmarks; and determining a steering angle for the vehicle by comparing the vehicle heading to a predetermined road model trajectory at the current position of the vehicle.

[0051] In some embodiments of the method, the predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along the road segment. The method may include retrieving the predetermined road model trajectory from one of a database stored in a memory included in the vehicle and a database accessible to the vehicle via a wireless communication interface. The method may include transmitting a control signal specifying a steering angle to a steering system of the vehicle. Determining a heading of the vehicle may include determining a previous position of the vehicle relative to the intersection based on an intersection of direction indicators of two or more landmarks, and determining a heading based on the previous position and the current position.

[0052] In some embodiments, a system for autonomously navigating a vehicle based on multiple overlapping navigation maps may include at least one processor, the at least one processor being programmed to: receive a first navigation map for use in autonomously controlling the vehicle, the first navigation map associated with a first road segment; determine a first autonomous navigation response of the vehicle along the first road segment based on an analysis of the first navigation map; receive a second navigation map for use in autonomously controlling the vehicle, the second navigation map associated with the second road segment, the first road segment being different from the second road segment, and the first road segment and the second road segment overlap each other at the overlapping segment; determine at least a second autonomous navigation response of the vehicle along the second road segment based on an analysis of the second navigation map; and determine at least a third autonomous navigation response of the vehicle within the overlapping segment based on at least one of the first navigation map and the second navigation map.

[0053] In some embodiments of the system, each of the multiple overlapping navigation maps may have its own coordinate system. Each of the multiple overlapping navigation maps may include a polynomial representation of a target trajectory along a road segment. Each overlapping navigation map may be a sparse map having a data density of 10 kilobytes per kilometer or less. The overlapping segment may have a length of at least 50 meters. The overlapping segment may have a length of at least 100 meters. The at least one processor may be programmed to determine a third autonomous navigation response based on both the first navigation map and the second navigation map. The third autonomous navigation response may be a combination of the first autonomous navigation response and the second autonomous navigation response. The third autonomous navigation response may be an average of the first autonomous navigation response and the second autonomous navigation response. The processor may be further programmed to identify an error between the first autonomous navigation response and the second autonomous navigation response, and, if the error is less than a threshold error, determine the third autonomous navigation response based on the second autonomous navigation response.

[0054] In some embodiments, an autonomous vehicle may include a body, at least one image capture device configured to acquire at least one image representing an environment of the vehicle, and at least one processor, wherein the at least one processor is programmed to: determine a current position of the vehicle based on the at least one image; receive a first navigation map associated with a first road segment; and, if the current position of the vehicle is on the first navigation map, determine at least a first autonomous navigation response of the vehicle based on an analysis of the first navigation map; receive a second navigation map associated with a second road segment different from the second road segment, wherein the first road segment and the second road segment overlap each other at an overlap segment; if the current position of the vehicle is on the second navigation map, determine at least a second autonomous navigation response of the vehicle based on an analysis of the second navigation map; and, if the current position of the vehicle is on the overlap segment, determine at least a third autonomous navigation response of the vehicle based on at least one of the first navigation map and the second navigation map.

[0055] In some embodiments of the autonomous vehicle, the first navigation map and the second navigation map may each have their own coordinate system. The first navigation map and the second navigation map may each include a polynomial representation of a target trajectory along a road segment. The at least one processor may be programmed to determine a third autonomous navigation response based on both the first navigation map and the second navigation map. The third autonomous navigation response may be a combination of the first autonomous navigation response and the second autonomous navigation response. The processor may be further programmed to identify an error between the first autonomous navigation response and the second autonomous navigation response, and, if the error is less than a threshold error, determine the third autonomous navigation response based on the second autonomous navigation response.

[0056] In some embodiments, a method for navigating an autonomous vehicle may include receiving, from an image capture device, at least one image representing an environment of the vehicle; using a processor associated with the vehicle, identifying a current position of the vehicle based on the at least one image; receiving a first navigation map associated with a first road segment; if the current position of the vehicle is in the first navigation map, determining at least a first autonomous navigation response of the vehicle based on analysis of the first navigation map; receiving a second navigation map associated with a second road segment different from the second road segment, wherein the first road segment and the second road segment overlap each other at an overlap segment; if the current position of the vehicle is in the second navigation map, determining at least a second autonomous navigation response of the vehicle based on analysis of the second navigation map; and if the current position of the vehicle is in the overlap segment, determining at least a third autonomous navigation response of the vehicle based on at least one of the first navigation map and the second navigation map.

[0057] In some embodiments of the method, each of the multiple overlapping navigation maps may have its own coordinate system, and each of the multiple overlapping navigation maps may include a polynomial representation of a target trajectory along a road segment. Determining the third autonomous navigation response may include identifying a combination of the first autonomous navigation response and the second autonomous navigation response. The method may include identifying an error between the first autonomous navigation response and the second autonomous navigation response, and if the error is less than a threshold error, determining the third autonomous navigation response based on the second autonomous navigation response.

[0058] In some embodiments, a system for autonomous navigation using a sparse map of a vehicle along a road segment may include at least one processor, which may be programmed to: receive a sparse map of the road segment, the sparse map having a data density of 1 megabyte per kilometer or less; receive at least one image from a camera representing an environment of the vehicle; analyze the sparse map and the at least one image received from the camera; and determine an autonomous navigation response of the vehicle based solely on an analysis of the sparse map and the at least one image received from the camera.

[0059] In some embodiments of the system, the sparse map may include a polynomial representation of a target trajectory along a road segment. The sparse map may include one or more recognized landmarks. The recognized landmarks may be spaced apart in the sparse map at a rate of 0.5 per kilometer or less. The recognized landmarks may be spaced apart in the sparse map at a rate of 1 per kilometer or less. The recognized landmarks may be spaced apart in the sparse map at a rate of 1 per 100 meters or less. The sparse map may have a data density of 100 kilobytes per kilometer or less. The sparse map may have a data density of 10 kilobytes per kilometer or less.

[0060] In some embodiments, a method for autonomous navigation with a sparse map of a vehicle along a road segment may include receiving a sparse map of the road segment, the sparse map having a data density of 1 megabyte per kilometer or less; receiving at least one image representing an environment of the vehicle from a camera; analyzing the sparse map and the at least one image received from the camera; and determining an autonomous navigation response of the vehicle based solely on an analysis of the sparse map and the at least one image received from the camera.

[0061] In some embodiments of the method, the sparse map may include a polynomial representation of a target trajectory along a road segment. The sparse map may include one or more recognized landmarks. The recognized landmarks may be spaced apart in the sparse map at a rate of 0.5 per kilometer or less. The recognized landmarks may be spaced apart in the sparse map at a rate of 1 per kilometer or less. The recognized landmarks may be spaced apart in the sparse map at a rate of 1 per 100 meters or less. The sparse map may have a data density of 100 kilobytes per kilometer or less. The sparse map may have a data density of 10 kilobytes per kilometer or less.

[0062] In some embodiments, a non-transitory computer-readable medium may store instructions that cause at least one processor to perform sparse-map autonomous navigation of a vehicle along a road segment, which may include receiving a sparse map of the road segment. The instructions may cause the processor to perform the steps of receiving the sparse map of the road segment, where the sparse map has a data density of 1 megabyte per kilometer or less, receiving at least one image from a camera representing an environment of the vehicle, analyzing the sparse map and the at least one image from the camera, and determining an autonomous navigation response of the vehicle based solely on an analysis of the sparse map and the at least one image received from the camera.

[0063] In some embodiments of the non-transitory computer-readable medium, the sparse map may include a polynomial representation of a target trajectory along a road segment. The sparse map may include one or more recognized landmarks. The recognized landmarks may be spaced apart in the sparse map at a rate of 0.5 per kilometer or less.

[0064] In some embodiments, a system for autonomously navigating a vehicle along a road segment based on predetermined landmark locations may include at least one processor programmed to: receive, from a camera, at least one image representing an environment of the vehicle; determine a position of the vehicle along a predetermined road model trajectory associated with the road segment based at least in part on information associated with the at least one image; identify a recognized landmark ahead of the vehicle based on the determined position, the recognized landmark being beyond the sight range of the camera; determine a current distance between the vehicle and the recognized landmark by comparing the determined position of the vehicle with predetermined positions of the recognized landmark; and determine an autonomous navigation response of the vehicle based on the determined current distance.

[0065] In some embodiments of the system, the predetermined position of the recognized landmark may be determined as an average of multiple obtained position measurements associated with the recognized landmark, and the multiple obtained position measurements may be determined based on acquiring at least one environmental image, analyzing the at least one environmental image to identify the recognized landmark within the environment, receiving Global Positioning System (GPS) data, analyzing the at least one environmental image to determine a relative position of the recognized landmark with respect to the vehicle, and determining a global position of the recognized landmark based at least on the GPS data and the determined relative position. The autonomous navigation response may include applying brakes associated with the vehicle. The autonomous navigation response may include changing a steering angle of the vehicle. The recognized landmark may include a stop line, a traffic light, a stop sign, or a curve along a road segment. The camera may be included in the vehicle.

[0066] In some embodiments, a method for autonomously navigating a vehicle along a road segment based on predetermined landmark locations may include receiving, from a camera, at least one image representing an environment of the vehicle; determining a position of the vehicle along a predetermined road model trajectory associated with the road segment based at least in part on information associated with the at least one image; identifying a recognized landmark ahead of the vehicle based on the determined position, the recognized landmark being beyond the field of view of the camera; determining a current distance between the vehicle and the recognized landmark by comparing the determined position of the vehicle with a predetermined position of the recognized landmark; and determining an autonomous navigation response of the vehicle based on the determined current distance.

[0067] In some embodiments of the method, the predetermined position of the recognized landmark may be determined as an average of multiple obtained position measurements associated with the recognized landmark, and the multiple obtained position measurements may be determined based on acquiring at least one environmental image, analyzing the at least one environmental image to identify the recognized landmark within the environment, receiving Global Positioning System (GPS) data, analyzing the at least one environmental image to determine a relative position of the recognized landmark with respect to the vehicle, and determining a global position of the recognized landmark based at least on the GPS data and the determined relative position. The autonomous navigation response may include applying brakes associated with the vehicle. The autonomous navigation response may include changing a steering angle of the vehicle. The recognized landmark may include a stop line, a traffic light, a stop sign, or a curve along a road segment. The camera may be included in the vehicle.

[0068] In some embodiments, a non-transitory computer-readable medium may store instructions that cause at least one processor to perform autonomous navigation of a vehicle along a road segment. The instructions may cause the processor to receive, from a camera, at least one image representing an environment of the vehicle, determine a position of the vehicle along a predetermined road model trajectory associated with the road segment based at least in part on information associated with the at least one image, identify a recognized landmark ahead of the vehicle based on the determined position, where the recognized landmark is beyond the field of view of the camera, determine a current distance between the vehicle and the recognized landmark by comparing the determined position of the vehicle with a predetermined position of the recognized landmark, and determine an autonomous navigation response of the vehicle based on the determined current distance.

[0069] In some embodiments of the non-transitory computer-readable medium, the autonomous navigation response may include applying brakes associated with the vehicle. The autonomous navigation response may include changing a steering angle of the vehicle. The recognized landmark may include a stop line, a traffic light, a stop sign, or a curve along the road segment.

[0070] In some embodiments, a system for autonomously navigating a vehicle along a road segment may include at least one processor that is programmed to receive information related to one or more aspects of the road segment from at least one sensor, identify local features of the road segment based on the received information, compare the local features to predetermined signature features of the road segment, identify a current position of the vehicle along a predetermined road model trajectory associated with the road segment based on the comparison of the local features to the predetermined signature feature, and determine an autonomous steering action of the vehicle based on a direction of the predetermined road model trajectory at the identified position.

[0071] In some embodiments of the system, the at least one processor may be further programmed to determine a heading of the vehicle at a current location and to determine an autonomous steering operation by comparing a direction of a predetermined road model trajectory with the heading. The heading may be determined based on a driving trajectory of the vehicle. The at least one sensor may include an image capture device configured to acquire at least one image representing the vehicle's environment. The signature feature may include a road width profile across at least a portion of the road segment. The signature feature may include a lane width profile across at least a portion of the road segment. The signature feature may include a dashed line spacing profile across at least a portion of the road segment. The signature feature may include a predetermined number of road marks along at least a portion of the road segment. The signature feature may include a road surface profile across at least a portion of the road segment. The signature feature may include a predetermined curvature associated with the road segment. Identifying the current location of the vehicle may include comparing a first parameter value indicative of the curvature of the predetermined road model trajectory with a second parameter value indicative of the curvature of a measured trajectory of the vehicle. The at least one sensor may include a suspension component monitor.

[0072] In some embodiments, the vehicle may include a body, at least one sensor configured to acquire information related to one or more aspects of the road segment, and at least one processor, wherein the at least one processor is programmed to: identify local features of the road segment based on information received from the at least one sensor; compare the local features to predetermined signature features of the road segment; identify a current position of the vehicle along a predetermined road model trajectory associated with the road segment based on the comparison of the local features to the predetermined signature feature; and determine an autonomous steering operation of the vehicle based on a direction of the predetermined road model trajectory at the current position.

[0073] In some embodiments of the vehicle, the signature features may include at least one of a road width profile across at least a portion of the road segment, a lane width profile across at least a portion of the road segment, a dashed line spacing profile across at least a portion of the road segment, a predetermined number of road marks along at least a portion of the road segment, a road surface profile across at least a portion of the road segment, and a predetermined curvature associated with the road segment. The vehicle may include a suspension component monitor, in which case the processor is further programmed to identify local features based on signals from the suspension component monitor. The processor may be further programmed to identify a heading of the vehicle, identify a direction of a predetermined road model trajectory at the current location, and determine an autonomous steering operation by comparing the direction with the heading.

[0074] In some embodiments, a method for navigating a vehicle may include receiving information related to one or more aspects of a road segment from at least one sensor; identifying, using at least one processor, local characteristics of the road segment based on the information received from the at least one sensor; comparing the received information to predetermined signature characteristics of the road segment; identifying a current position of the vehicle along a predetermined road model trajectory associated with the road segment based on the comparison of the received information to the predetermined signature characteristics; and determining an autonomous steering operation of the vehicle based on a direction of the predetermined road model trajectory at the current position.

[0075] In some embodiments, the method may include determining a heading of the vehicle at a current location, determining a direction of a predetermined road model trajectory at the current location, and determining an autonomous steering operation by comparing the direction of the predetermined road model trajectory to the heading. The local features may include at least one of a road width profile over at least a portion of the road segment, a lane width profile over at least a portion of the road segment, a dashed line spacing profile over at least a portion of the road segment, a predetermined number of road marks along at least a portion of the road segment, a road surface profile over at least a portion of the road segment, and a predetermined curvature associated with the road segment. The method may include determining the road surface profile using a suspension component monitor, comparing the road surface profile to the predetermined road surface profile, and determining the current location based on the comparison of the road surface profile to the predetermined road surface profile.

[0076] In some embodiments, a system for autonomously navigating a vehicle may include at least one processor that is programmed to receive, from a rear-facing camera, at least one image representing an area behind the vehicle; analyze the at least one rear-facing image to find a representation of at least one landmark within the image; identify at least one indicator of the position of the landmark relative to the vehicle; determine a forward trajectory for the vehicle based at least in part on the indicator of the position of the landmark relative to the vehicle; and navigate the vehicle along the determined forward trajectory.

[0077] In some embodiments of the system, the position indicator may include a distance between the vehicle and a landmark. The position indicator may include a relative angle between the vehicle and the landmark. The landmark may include a road edge, a lane marking, a reflector, a pole, a change in a line pattern on the road, or a road sign. The landmark may include the backside of a road sign. The at least one processor may be further programmed to determine a lane offset amount of the vehicle within a current lane of travel based on the landmark position indicator, in which case the determination of the forward trajectory is further based on the determined lane offset amount. The at least one processor may be further programmed to receive at least one image representing another area of ​​the vehicle from another camera, in which case the determination of the forward trajectory is further based on the at least one image received from the other camera.

[0078] In some embodiments, a method of autonomously navigating a vehicle may include receiving at least one image representing an area behind the vehicle from a rear-facing camera; analyzing the at least one rear-facing image to find a representation of at least one landmark within the image; identifying at least one indicator of the position of the landmark relative to the vehicle; determining a forward trajectory for the vehicle based at least in part on the indicator of the position of the landmark relative to the vehicle; and navigating the vehicle along the determined forward trajectory.

[0079] In some embodiments of the method, the position indicator may include a distance between the vehicle and a landmark. The position indicator may include a relative angle between the vehicle and the landmark. The landmark may include a road edge, a lane marking, a reflector, a pole, a change in a line pattern on a road, or a road sign. The landmark may include the backside of a road sign. The method may include determining a lane offset amount for the vehicle within the current travel lane based on the landmark position indicator, in which case the determination of the forward trajectory may be further based on the determined lane offset amount.

[0080] In some embodiments, the vehicle may include a body, a rear-facing camera, and at least one processor, the at least one processor being programmed to receive at least one image representing an area behind the vehicle via a rear camera interface connecting the rear-facing camera, analyze the at least one rear-facing image to find a representation of at least one landmark within the image, identify at least one indicator of the position of the landmark relative to the vehicle, determine a forward trajectory for the vehicle based at least in part on the indicator of the position of the landmark relative to the vehicle, and navigate the vehicle along the determined forward trajectory.

[0081] In some embodiments of the vehicle, the rear-facing camera may be mounted on an object connected to the vehicle. The object may be a trailer, a bike carrier, a ski / snowboard carrier, a mounting base, or a luggage carrier. The rear camera interface may include a detachable interface. The rear camera interface may include a wireless interface.

[0082] In some embodiments, a system for navigating a vehicle by identifying a free space region in which the vehicle may travel may include at least one processor, the at least one processor being programmed to: receive a plurality of images associated with the vehicle's environment from an image capture device; analyze at least one of the plurality of images to identify a first free space boundary on a driver's side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger's side of the vehicle and extending forward of the vehicle, and a forward free space boundary on a front side of the vehicle and extending between the first and second free space boundaries, wherein the first free space boundary, the second free space boundary, and the forward free space boundary define a free space region forward of the vehicle; determine a navigation path for the vehicle through the free space region; and drive the vehicle along at least a portion of the determined navigation path within the free space region forward of the vehicle.

[0083] In some embodiments of the system, the first free space boundary may correspond to at least one of a road edge, a curve, a barrier, a lane dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The second free space boundary may correspond to at least one of a road edge, a curve, a barrier, a lane dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The forward free space boundary may correspond to a road horizon. The at least one processor may be further programmed to identify an obstacle ahead of the vehicle based on analysis of at least one of the plurality of images and to exclude the identified obstacle from the free space region ahead of the vehicle. The obstacle may include a pedestrian. The obstacle may include another vehicle. The obstacle may include debris. The at least one processor may be further programmed to identify an obstacle ahead of the vehicle based on analysis of at least one of the plurality of images and to exclude an area surrounding the identified obstacle from the free space region ahead of the vehicle. The at least one processor may be further programmed to identify an area surrounding the identified obstacle based on one or more of the following: the speed of the vehicle, the type of obstacle, the image capture rate of the image capture device, and the speed of movement of the obstacle.

[0084] In some embodiments, the vehicle may include a body including a driver side and a passenger side, an image capture device, and at least one processor, the at least one processor being programmed to: receive from the image capture device a plurality of images associated with the vehicle's environment; analyze at least one of the plurality of images to identify a first free space boundary on the driver side of the body and extending forward of the body, a second free space boundary on the passenger side of the body and extending forward of the body, and a forward free space boundary on the body and extending between the first and second free space boundaries, wherein the first free space boundary, the second free space boundary, and the forward free space boundary define a free space region forward of the body; determine a navigation path for the vehicle through the free space region; and drive the vehicle along at least a portion of the determined navigation path within the free space region forward of the vehicle.

[0085] In some embodiments, a method of navigating a vehicle by identifying a free space region in which the vehicle can travel may include receiving, from an image capture device, a plurality of images associated with the vehicle's environment; analyzing at least one of the plurality of images to identify a first free space boundary on a driver's side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger's side of the vehicle and extending forward of the vehicle, and a forward free space boundary on a front side of the vehicle and extending between the first free space boundary and the second free space boundary, wherein the first free space boundary, the second free space boundary, and the forward free space boundary define a free space region forward of the vehicle; determining a navigation path for the vehicle through the free space region; and driving the vehicle along at least a portion of the determined navigation path within the free space region forward of the vehicle.

[0086] In some embodiments of the method, the first free space boundary may correspond to at least one of a road edge, a curve, a barrier, a lane dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The second free space boundary may correspond to at least one of a road edge, a curve, a barrier, a lane dividing structure, a parked vehicle, a tunnel wall, or a bridge structure. The forward free space boundary may correspond to a road horizon. The method may include identifying an obstacle ahead of the vehicle based on analysis of at least one of the plurality of images and excluding the identified obstacle from a free space region ahead of the vehicle. The obstacle may include a pedestrian. The obstacle may include another vehicle. The obstacle may include debris. The method may include identifying an obstacle ahead of the vehicle based on analysis of at least one of the plurality of images and excluding a region surrounding the identified obstacle from the free space region ahead of the vehicle. The method may include determining an area surrounding the identified obstacle based on one or more of the following: a speed of the vehicle, a type of obstacle, an image capture rate of the image capture device, and a speed of movement of the obstacle.

[0087] In some embodiments, a system for navigating a vehicle on a road with snow covering at least some lane markings and road edges may include at least one processor that is programmed to receive, from an image capture device, at least one image of an environment ahead of the vehicle including an area where snow covers at least some lane markings and road edges, identify, based on analysis of the at least one image, at least a portion of the snow-covered road and an estimated location of the road edge that bounds at least a portion of the snow-covered road, and navigate the vehicle on a navigation route that includes the identified portion of the road and that is within the determined estimated location of the road edge.

[0088] In some embodiments of the system, analyzing the at least one image may include identifying at least one tire track in the snow. Analyzing the at least one image may include identifying light variations across the snow surface. Analyzing the at least one image may include identifying trees along the edge of the road. Analyzing the at least one image may include recognizing curvature changes in the snow surface. The recognized curvature changes may be identified to correspond to an estimated location of the road edge. Analyzing the at least one image may include pixel analysis of the at least one image, where at least first pixels are compared with at least second pixels, to identify features associated with at least some lane markings and the surface of snow covering the road edge. The features may correspond to edges of the tire track. The features may correspond to edges of the road. The at least one processor may be further programmed to navigate the vehicle between the identified edges of the road. The at least one processor may be further programmed to navigate the vehicle by at least partially following the tire track in the snow.

[0089] In some embodiments, a method for navigating a vehicle on a road with snow covering at least some lane markings and road edges may include receiving, from an image capture device, at least one image of an environment ahead of the vehicle including an area where snow covers at least some lane markings and road edges; identifying, based on analysis of the at least one image, at least a portion of the snow-covered road and an estimated location of the road edge that bounds at least a portion of the snow-covered road; and navigating the vehicle on a navigation route that includes the identified portion of the road and that is within the determined estimated location of the road edge.

[0090] In some embodiments of the method, analyzing the at least one image may include identifying at least one tire track in the snow. Analyzing the at least one image may include identifying light variations across the snow surface. Analyzing the at least one image may include identifying a plurality of trees along the edge of the road. Analyzing the at least one image may include recognizing curvature changes in the snow surface. The recognized curvature changes may be identified as corresponding to an estimated location of the road edge. Analyzing the at least one image may include pixel analysis of the at least one image, where at least first pixels are compared with at least second pixels, to identify features associated with at least some lane markings and the surface of snow covering the road edge. The features may correspond to edges of the tire track. The features may correspond to edges of the road. The method may include navigating the vehicle between the identified edges of the road. The method may include navigating the vehicle by at least partially following the tire track in the snow.

[0091] In some embodiments, a system for navigating a vehicle on an at least partially snow-covered road may include at least one processor that receives, from an image capture device, a plurality of captured images of an environment forward of the vehicle including an area where snow has covered a road on which the vehicle is traveling; and analyzes at least one of the plurality of images to identify a first free space boundary on a driver's side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger's side of the vehicle and extending forward of the vehicle, and a forward free space boundary on a passenger's side of the vehicle and extending between the first and second free space boundaries, is programmed to define and identify a free space region ahead of the vehicle; determine a first navigation route proposal for the vehicle through the free space region; provide at least one of the plurality of images to a neural network; receive from the neural network a second navigation route proposal for the vehicle based on analysis of the at least one of the plurality of images by the neural network; determine whether the first navigation route proposal matches the second navigation route proposal; and, if the first navigation route proposal is determined to match the second navigation route proposal, cause the vehicle to travel at least a portion of the first navigation route proposal.

[0092] In some embodiments, a system for calibrating an indicator of speed of an autonomous vehicle may include at least one processor that is programmed to receive, from a camera, a plurality of images representing an environment of the vehicle; analyze the plurality of images to identify at least two recognized landmarks; determine a value indicative of a distance between the at least two recognized landmarks based on known positions of the two recognized landmarks; determine a measured distance between the at least two landmarks based on output of at least one sensor associated with the autonomous vehicle; and determine a correction factor for the at least one sensor based on a comparison of the value indicative of the distance between the at least two recognized landmarks and the measured distance between the at least two landmarks.

[0093] In some embodiments of the system, a correction factor may be determined such that the calculation of the determined distance along the road segment by the correction factor matches the distance value received via the wireless transceiver. The two recognized landmarks may include one or more of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one sensor may include a speedometer associated with the vehicle. The known positions of the two recognized landmarks may be received from a server-based system located remotely with respect to the vehicle. The known positions may each constitute a refined position determined based on multiple GPS-based measurements.

[0094] In some embodiments, a system for calibrating an indicator of speed of an autonomous vehicle may include at least one processor programmed to: determine a distance along a road segment based on an output of at least one sensor associated with the autonomous vehicle; receive, via a wireless transceiver, a distance value associated with the road segment; and determine a calibration factor for the at least one sensor based on the determined distance along the road segment and the distance value received via the wireless transceiver.

[0095] In some embodiments of the system, distance values ​​associated with a road segment received via a wireless transceiver may be determined based on previous measurements made by a plurality of measurement vehicles. The plurality of measurement vehicles may include at least 100 measurement vehicles. The plurality of measurement vehicles may include at least 1000 measurement vehicles. Correction factors may be determined such that an application of the correction factors to the determined distance along the road segment matches the distance value received via the wireless transceiver. The at least one processor may be programmed to determine a composite correction factor based on the plurality of determined correction factors. The composite correction factor may be determined by averaging the plurality of determined correction factors. The composite correction factor may be determined by finding the average value of the plurality of determined correction factors.

[0096] In some embodiments, the vehicle may include a body, a camera, and at least one processor, the at least one processor being programmed to: receive a plurality of images from the camera representing an environment of the vehicle; analyze the plurality of images to identify at least two recognized landmarks; determine a value indicative of a distance between the at least two recognized landmarks based on known positions of the two recognized landmarks; determine a measured distance between the at least two landmarks based on output of at least one sensor associated with the autonomous vehicle; and determine a correction factor for the at least one sensor based on a comparison of the value indicative of the distance between the at least two recognized landmarks and the measured distance between the at least two landmarks.

[0097] In some embodiments of the vehicle, the at least one sensor may include a speedometer associated with the vehicle. The two recognized landmarks may include one or more of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a light pole. The known locations of the two recognized landmarks may be received from a server-based system located remotely from the vehicle.

[0098] In some embodiments, a system for determining a lane assignment for an autonomous vehicle along a road segment may include at least one processor that is programmed to receive, from a camera, at least one image representing an environment for the vehicle; analyze the at least one image to identify at least one recognized landmark; determine an indicator of a lateral offset distance between the vehicle and the at least one recognized landmark; and determine a lane assignment for the vehicle along the road segment based on the indicator of the lateral offset distance between the vehicle and the at least one recognized landmark.

[0099] In some embodiments of the system, the vehicle's environment may include a road segment, a number of lanes, and at least one recognized landmark. The at least one recognized landmark may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one recognized landmark may include a business sign. The lateral offset distance between the vehicle and the at least one recognized landmark may be the sum of a first distance between the vehicle and a first side of the road segment and a second distance between the first side of the road and the at least one recognized landmark. Identifying the indicator of the lateral offset distance between the vehicle and the at least one recognized landmark may be based on a predetermined position of the at least one recognized landmark. Identifying the indicator of the lateral offset distance between the vehicle and the at least one recognized landmark may be based on a scale associated with the at least one image. Identifying the lane assignment may further be based on at least one of a width of the road segment, a number of lanes of the road segment, and a lane width. The determination of the lane assignment may be further based on a predetermined road model trajectory associated with the road segment. The at least one recognized landmark may include a first recognized landmark on a first side of the vehicle and a second recognized landmark on a second side of the vehicle, and the determination of the lane assignment of the vehicle along the road segment is based on a first indicator of a lateral offset distance between the vehicle and the first recognized landmark and a second indicator of a lateral offset distance between the vehicle and the second recognized landmark.

[0100] In some embodiments, a computer-implemented method for determining a lane assignment for an autonomous vehicle along a road segment may include the following operations performed by one or more processors: receiving at least one image representing an environment of the vehicle from a camera; analyzing the at least one image to identify at least one recognized landmark; determining an indicator of a lateral offset distance between the vehicle and the at least one recognized landmark; and determining a lane assignment for the vehicle along the road segment based on the indicator of the lateral offset distance between the vehicle and the at least one recognized landmark.

[0101] In some embodiments of the method, the at least one recognized landmark may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one recognized landmark may include a business sign. The determination of the lane assignment may be further based on a predetermined road model trajectory associated with the road segment. The at least one recognized landmark may include a first recognized landmark on a first side of the vehicle and a second recognized landmark on a second side of the vehicle, and the determination of the vehicle's lane assignment along the road segment is based on a first indicator of a lateral offset distance between the vehicle and the first recognized landmark and a second indicator of a lateral offset distance between the vehicle and the second recognized landmark.

[0102] In some embodiments, a computer-readable storage medium may include a set of instructions executable by at least one processor and causing the at least one processor to perform a method for determining a lane assignment for an autonomous vehicle along a road segment. The method may include receiving at least one image from a camera representing an environment for the vehicle, analyzing the at least one image to identify at least one recognized landmark, determining an indicator of a lateral offset distance between the vehicle and the at least one recognized landmark, and determining a lane assignment for the vehicle along the road segment based on the indicator of the lateral offset distance between the vehicle and the at least one recognized landmark.

[0103] In some embodiments of the computer-readable storage medium, the at least one recognized landmark may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one recognized landmark may include a business sign. The determination of the lane assignment may be further based on a predetermined road model trajectory associated with the road segment. The at least one recognized landmark may include a first recognized landmark on a first side of the vehicle and a second recognized landmark on a second side of the vehicle, and the determination of the vehicle's lane assignment along the road segment is based on a first indicator of a lateral offset distance between the vehicle and the first recognized landmark and a second indicator of a lateral offset distance between the vehicle and the second recognized landmark.

[0104] In some embodiments, a system for autonomously navigating a vehicle along a road segment may include at least one processor programmed to: receive, from a camera, at least one image representing an environment of the vehicle; analyze the at least one image to identify at least one recognized landmark, where the at least one recognized landmark is part of a group of recognized landmarks, and the identification of the at least one recognized landmark is based, at least in part, on one or more landmark group features associated with the group of recognized landmarks; determine, based at least in part on a predetermined position of the recognized landmark, a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment; and determine an autonomous steering operation of the vehicle based, at least in part, on an orientation of the predetermined road model trajectory at the identified current position of the vehicle relative to the predetermined road model trajectory.

[0105] In some embodiments of the system, the at least one recognized landmark may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one recognized landmark may include a business sign. The predetermined road model trajectory may include a three-dimensional polynomial representation of a target trajectory along a road segment. The at least one processor may be further programmed to determine a current position of the vehicle along the predetermined road model trajectory based on the vehicle speed. The one or more landmark group features may include relative distances between members of the group of recognized landmarks. The one or more landmark group features may include an ordered sequence of members of the group of recognized landmarks. The one or more landmark group features may include a number of landmarks included in the group of recognized landmarks. Identification of the at least one recognized landmark may be based, at least in part, on a super landmark signature associated with the group of recognized landmarks. The at least one processor may be programmed to determine an autonomous steering operation for the vehicle by comparing a heading of the vehicle with a predetermined road model trajectory at the identified current position of the vehicle.

[0106] In some embodiments, a computer-implemented method for autonomously navigating a vehicle along a road segment may include the following operations performed by one or more processors: receiving at least one image from a camera representing an environment of the vehicle; analyzing the at least one image to identify at least one recognized landmark, where the at least one recognized landmark is part of a group of recognized landmarks, and the identification of the at least one recognized landmark is based, at least in part, on one or more landmark group features associated with the group of recognized landmarks; determining, relative to the vehicle, a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based, at least in part, on predetermined positions of the recognized landmarks; and determining an autonomous steering operation of the vehicle based on an orientation of the predetermined road model trajectory at the determined current position of the vehicle relative to the predetermined road model trajectory.

[0107] In some embodiments of the method, the at least one recognized landmark may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one recognized landmark may include a business sign. The one or more landmark group features may include a relative distance between members of the group of recognized landmarks. The one or more landmark group features may include an ordered sequence of members of the group of recognized landmarks.

[0108] In some embodiments, a computer-readable storage medium may include a set of instructions executable by at least one processor and causing the at least one processor to perform a method for autonomously navigating a vehicle along a road segment. The method may include receiving, from a camera, at least one image representing an environment of the vehicle; analyzing the at least one image to identify at least one recognized landmark, where the at least one recognized landmark is part of a group of recognized landmarks, and the identification of the at least one recognized landmark is based, at least in part, on one or more landmark group features associated with the group of recognized landmarks; determining, relative to the vehicle, a current position of the vehicle relative to a predetermined road model trajectory associated with the road segment based, at least in part, on a predetermined position of the recognized landmark; and determining an autonomous steering operation of the vehicle based on an orientation of the predetermined road model trajectory at the determined current position of the vehicle relative to the predetermined road model trajectory.

[0109] In some embodiments of the computer-readable storage medium, the at least one landmark may include at least one of a traffic sign, an arrow mark, a lane mark, a dashed lane mark, a traffic light, a stop line, a directional sign, a reflector, a landmark beacon, or a lamppost. The at least one recognized landmark may include a business sign. The one or more landmark group features may include a relative distance between members of the group of recognized landmarks. The one or more landmark group features may include an ordered sequence of members of the group of recognized landmarks.

[0110] In some embodiments, the vehicle navigation system may include at least one processor programmed to receive, from a camera, at least one environmental image associated with the vehicle; determine a navigation operation for the vehicle based on an analysis of the at least one environmental image; cause the vehicle to initiate the navigation operation; receive user input associated with a user's navigation response that differs from the initiated navigation operation; determine navigation context information associated with the vehicle based on the received user input; and store the navigation context information in association with information related to the user input.

[0111] In some embodiments of the system, the navigation operation may be based on recognized landmarks identified in at least one environmental image. The information related to the user input may include information specifying at least one of a turn rate of the vehicle, an amount of acceleration of the vehicle, and an amount of braking of the vehicle. The control system may include at least one of steering control, acceleration control, and braking control. The navigation context information may include one or more images captured by an onboard camera. The user input may include at least one of braking, steering, or acceleration. The navigation context information may include a position of the vehicle. The navigation context information may include an output of at least one onboard sensor. The sensor may be a speedometer. The sensor may be an accelerometer. The sensor may be an IR sensor. The navigation context information may include a time of day. The navigation context information may include an indication of the presence of a visual obstruction. The visual obstruction may be caused by glare. The navigation context information may be identified based on the at least one environmental image. The system may include a transmitter that transmits the navigation context information from the vehicle to a remote server.

[0112] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor and causing the at least one processor to perform a method. The method may include receiving at least one environmental image associated with a vehicle from a camera, determining a navigation operation for the vehicle based on an analysis of the at least one environmental image, causing the vehicle to initiate the navigation operation, receiving user input associated with a user navigation response that differs from the initiated navigation operation, determining navigation context information associated with the vehicle based on the received user input, and storing the navigation context information in association with information related to the user input.

[0113] In some embodiments, a navigation system for a vehicle may include at least one processor that is programmed to: determine a navigation operation for the vehicle based at least in part on a comparison of vehicle behavior to a predetermined model representing a road segment; receive at least one image representing an environment of the vehicle from a camera; identify the presence of a navigation adjustment situation in the environment of the vehicle based on analysis of the at least one image; cause the vehicle to adjust its navigation operation based on the presence of the navigation adjustment situation; and store information related to the navigation adjustment situation.

[0114] In some embodiments of the system, the navigation adjustment situation may include a parked vehicle. The navigation adjustment situation may include a lane shift. The navigation adjustment situation may include at least one of a newly encountered traffic sign or a newly encountered traffic light. The navigation adjustment situation may include a construction area. The processor may be further programmed to cause stored information related to the navigation adjustment situation to be transmitted to the road model management system to identify whether the navigation adjustment situation requires an update to the predetermined model representing the road segment. The stored information related to the navigation adjustment situation may include at least one of an indicator of a location where the navigation adjustment situation was encountered, an indication of an adjustment made to the navigation operation, and at least one image. The predetermined model representing the road segment may include a cubic spline representing a predetermined driving path along the road segment.

[0115] In some embodiments, a method of navigating a vehicle may include determining a navigation operation of the vehicle based at least in part on a comparison of vehicle behavior to a predetermined model representing a road segment; receiving at least one image representing an environment of the vehicle from a camera; identifying the presence of a navigation adjustment situation in the environment of the vehicle based on analysis of the at least one image; causing the vehicle to adjust its navigation operation based on the presence of the navigation adjustment situation; and storing information related to the navigation adjustment situation.

[0116] In some embodiments of the method, the navigation adjustment situation may include a parked vehicle. The navigation adjustment situation may include a lane shift. The navigation adjustment situation may include at least one of a newly encountered traffic sign or a newly encountered traffic light. The navigation adjustment situation may include a construction area. The method may include causing stored information related to the navigation adjustment situation to be transmitted to a road model management system to identify whether the navigation adjustment situation requires an update to a predetermined model representing the road segment. The stored information related to the navigation adjustment situation may include at least one of an indicator of a location where the navigation adjustment situation was encountered, an indication of an adjustment made to the navigation operation, and at least one image. The predetermined model representing the road segment may include a cubic spline representing a predetermined driving path along the road segment.

[0117] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor and causing the at least one processor to perform a method. The method may include determining a navigation operation for a vehicle based at least in part on a comparison of vehicle behavior to a predetermined model representing a road segment, receiving at least one image representing an environment of the vehicle from a camera, identifying the presence of a navigation adjustment situation in the environment of the vehicle based on analysis of the at least one image, causing the vehicle to adjust its navigation operation based on the presence of the navigation adjustment situation, and storing information related to the navigation adjustment situation.

[0118] In some embodiments of the computer-readable medium, the method may include causing stored information related to a navigation adjustment situation to be transmitted to a road model management system to identify whether the navigation adjustment situation requires an update to a predetermined model representing the road segment. The stored information related to the navigation adjustment situation may include at least one of an indicator of a location where the navigation adjustment situation was encountered, an indication of an adjustment made to the navigation operation, and at least one image. The predetermined model representing the road segment may include a three-dimensional spline representing a predetermined driving path along the road segment.

[0119] In some embodiments, a system for interacting with a plurality of autonomous vehicles may include a memory including a predetermined model representing at least one road segment, and at least one processor, the at least one processor being programmed to: receive, from each of the plurality of autonomous vehicles, navigation context information associated with an occurrence of an adjustment to a determined navigation operation; analyze the navigation context information; determine, based on the analysis of the navigation context information, whether the adjustment to the determined navigation operation was due to a temporary condition; and update the predetermined model representing the at least one road segment if the adjustment to the determined navigation operation was not due to a temporary condition.

[0120] In some embodiments of the system, the predetermined model representing at least one road segment may include a cubic spline representing a predetermined travel path along the at least one road segment. The update to the predetermined model may include an update to the cubic spline representing the predetermined travel path along the at least one road segment. The adjustment to the determined navigation maneuver may result from user intervention. The adjustment to the determined navigation maneuver may result from automatic identification of the presence of a navigation adjustment situation in the vehicle environment based on image analysis. The navigation situation information may include at least one image representing the environment of the autonomous vehicle. The navigation situation information may include a video representing the environment of the autonomous vehicle. The temporary situation may relate to a parked vehicle, an intervening vehicle, a pedestrian, a low light condition, a glare condition, a temporary barrier, or temporary road construction.

[0121] In some embodiments, a method of interacting with a plurality of autonomous vehicles may include receiving, from each of the plurality of autonomous vehicles, navigation context information associated with an occurrence of an adjustment to a determined navigation operation; analyzing the navigation context information; determining whether the adjustment to the determined navigation operation was due to a temporary condition based on the analysis of the navigation context information; and updating a predetermined model representing at least one road segment if the adjustment to the determined navigation operation was not due to a temporary condition.

[0122] In some embodiments of the method, the predetermined model representing at least one road segment may include a cubic spline representing a predetermined travel path along the at least one road segment. The update to the predetermined model may include an update to the cubic spline representing the predetermined travel path along the at least one road segment. The adjustment to the determined navigation maneuver may result from user intervention. The adjustment to the determined navigation maneuver may result from automatic identification of the presence of a navigation adjustment situation in the vehicle environment based on image analysis. The navigation situation information may include at least one image representing the environment of the autonomous vehicle. The navigation situation information may include a video representing the environment of the autonomous vehicle. The temporary situation may relate to a parked vehicle, an intervening vehicle, a pedestrian, a low light condition, a glare condition, a temporary barrier, or temporary road construction.

[0123] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor and causing the at least one processor to perform a method. The method may include receiving, from each of a plurality of autonomous vehicles, navigation context information associated with an occurrence of an adjustment to a determined navigation operation, analyzing the navigation context information, determining whether the adjustment to the determined navigation operation was due to a temporary condition based on the analysis of the navigation context information, and updating a pre-defined model representing at least one road segment if the adjustment to the determined navigation operation was not due to a temporary condition.

[0124] In some embodiments of the computer-readable medium, the predetermined model representing the at least one road segment may include a cubic spline representing a predetermined travel path along the at least one road segment. The update of the predetermined model may include an update to the cubic spline representing the predetermined travel path along the at least one road segment. The temporary condition may involve a parked vehicle, an intervening vehicle, a pedestrian, a low light condition, a glare condition, a temporary barrier, or temporary road construction.

[0125] In some embodiments, a system for interacting with a plurality of autonomous vehicles may include a memory including a predetermined road model representing at least one road segment, and at least one processor, the at least one processor being programmed to selectively receive, from the plurality of autonomous vehicles, road environment information based on navigation by the plurality of autonomous vehicles through respective road environments of the autonomous vehicles, identify whether one or more updates to the predetermined road model are needed based on the road environment information, and update the predetermined road model to include the one or more updates.

[0126] In some embodiments of the system, the road model may include a cubic spline representing a predetermined driving path along at least one road segment. Selectively receiving road environment information may include limiting the frequency of information transmissions received from a particular vehicle. Selectively receiving road environment information may include limiting the frequency of information transmissions received from a group of vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles traveling within a particular geographic area. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles based on a determined model confidence associated with the particular geographic area. Selectively receiving road environment information may include limiting information transmissions received from vehicles to only transmissions that may be inconsistent with at least one aspect of the predetermined road model.

[0127] In some embodiments, a method of interacting with a plurality of autonomous vehicles may include selectively receiving, from the plurality of autonomous vehicles, road environment information based on navigation by the plurality of autonomous vehicles through respective road environments of the autonomous vehicles; identifying, based on the road environment information, whether one or more updates to a predetermined road model are required; and updating the predetermined road model to include the one or more updates.

[0128] In some embodiments of the method, the road model may include a cubic spline representing a predetermined driving path along at least one road segment. Selectively receiving road environment information may include limiting the frequency of information transmissions received from a particular vehicle. Selectively receiving road environment information may include limiting the frequency of information transmissions received from a group of vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles traveling within a particular geographic area. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles based on a determined model confidence associated with the particular geographic area. Selectively receiving road environment information may include limiting information transmissions received from vehicles to only transmissions that may be inconsistent with respect to at least one aspect of the predetermined road model.

[0129] In some embodiments, a non-transitory computer-readable medium may include instructions executable by at least one processor and causing the at least one processor to perform a method. The method may include selectively receiving, from a plurality of autonomous vehicles, road environment information based on navigation by the plurality of autonomous vehicles through respective road environments of the autonomous vehicles, identifying, based on the road environment information, whether one or more updates to a predetermined road model are required, and updating the predetermined road model to include the one or more updates.

[0130] In some embodiments of the computer-readable medium, the road model may include a cubic spline representing a predetermined driving path along at least one road segment. Selectively receiving road environment information may include limiting the frequency of information transmissions received from a particular vehicle. Selectively receiving road environment information may include limiting the frequency of information transmissions received from a group of vehicles. Selectively receiving road environment information may include limiting the frequency of information transmissions received from vehicles traveling within a particular geographic area. Selectively receiving road environment information may include limiting information transmissions received from vehicles to only transmissions that may be inconsistent with respect to at least one aspect of the predetermined road model.

[0131] According to other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executed by at least one processing device and that perform any of the methods described herein.

[0132] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the scope of the claims.

[0133] BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. [Brief explanation of the drawings]

[0134] [Figure 1] 1 is a diagrammatic representation of an exemplary system according to the disclosed embodiments. [Figure 2A] 1 is a side view representation of an exemplary vehicle including a system according to disclosed embodiments. [Figure 2B] 2B is a top view representation of the vehicle and system shown in FIG. 2A according to a disclosed embodiment. [Figure 2C] 1 is a top view representation of another embodiment of a vehicle including a system according to the disclosed embodiments. [Figure 2D] 1 is a top view representation of yet another embodiment of a vehicle including a system according to the disclosed embodiments. [Figure 2E] 1 is a top view representation of yet another embodiment of a vehicle including a system according to the disclosed embodiments. [Figure 2F] 1 is a diagrammatic representation of an exemplary vehicle control system according to the disclosed embodiments. [Figure 3A] 1 is a diagrammatic representation of the interior of a vehicle including a rearview mirror and a user interface of a vehicle imaging system according to disclosed embodiments. [Figure 3B] 1 is a diagram of an example of a camera mount configured to be positioned behind a rearview mirror, facing a vehicle windshield, according to a disclosed embodiment. [Figure 3C] 3C is a diagram of the camera mount shown in FIG. 3B from a different perspective, according to a disclosed embodiment. [Figure 3D] 1 is a diagram of an example of a camera mount configured to be positioned behind a rearview mirror, facing a vehicle windshield, according to a disclosed embodiment. [Figure 4] FIG. 1 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 5A] 1 is a flowchart illustrating an exemplary process for generating one or more navigational responses based on monocular image analysis, according to disclosed embodiments. [Figure 5B] 1 is a flowchart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images, according to disclosed embodiments. [Figure 5C] 1 is a flowchart illustrating an exemplary process for detecting road markings and / or lane geometry information in a set of images, according to disclosed embodiments. [Figure 5D] 1 is a flowchart illustrating an exemplary process for detecting traffic lights in a set of images, according to disclosed embodiments. [Figure 5E] 1 is a flowchart of an exemplary process for generating one or more navigational responses based on a vehicle path, according to disclosed embodiments. [Figure 5F] 1 is a flowchart illustrating an example process for determining whether a leading vehicle is changing lanes, according to disclosed embodiments. [Figure 6] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on stereo image analysis, according to disclosed embodiments. [Figure 7] 1 is a flowchart illustrating an exemplary process for generating one or more navigational responses based on an analysis of three sets of images, according to disclosed embodiments. [Figure 8]1 illustrates a sparse map for providing autonomous vehicle navigation according to disclosed embodiments. [Figure 9A] 1 illustrates a polynomial representation of a portion of a road segment according to a disclosed embodiment. [Figure 9B] 1 illustrates a curve in three-dimensional space representing a target trajectory of a vehicle for a particular road segment contained in a sparse map, according to a disclosed embodiment. [Figure 10] 1 illustrates example landmarks that may be included in a sparse map according to the disclosed embodiments. [Figure 11A] 1 illustrates a polynomial representation of a trajectory according to a disclosed embodiment. [Figure 11B] 1 illustrates a target trajectory along a multi-lane road according to a disclosed embodiment. [Figure 11C] 1 illustrates a target trajectory along a multi-lane road according to a disclosed embodiment. [Figure 11D] 1 illustrates an example road signature profile according to the disclosed embodiments. [Figure 12] FIG. 1 is a schematic diagram of a system that uses crowd-sourced data received from multiple vehicles for autonomous vehicle navigation, according to disclosed embodiments. [Figure 13] 1 illustrates an example autonomous vehicle road navigation model represented by multiple cubic splines, according to the disclosed embodiments. [Figure 14] FIG. 1 illustrates a block diagram of a server according to the disclosed embodiments. [Figure 15] FIG. 1 illustrates a block diagram of a memory according to disclosed embodiments. [Figure 16] 1 illustrates a clustering process for vehicle trajectories associated with a vehicle according to a disclosed embodiment. [Figure 17] 1 illustrates a navigation system for a vehicle that may be used for autonomous navigation, according to disclosed embodiments. [Figure 18] 1 is a flowchart illustrating an example process for processing vehicle navigation information for use in autonomous vehicle navigation, according to disclosed embodiments. [Figure 19]4 is a flowchart illustrating an example process performed by a navigation system of a vehicle, according to disclosed embodiments. [Figure 20] 1 illustrates an example diagram of a memory according to disclosed embodiments; [Figure 21] 10 is a flowchart illustrating an example process for uploading a recommended trajectory to a server, according to the disclosed embodiments. [Figure 22] 1 illustrates an example environment including a system for identifying landmarks for use in autonomous vehicle navigation according to disclosed embodiments. [Figure 23] 1 illustrates an example environment including a system for identifying landmarks for use in autonomous vehicle navigation according to disclosed embodiments. [Figure 24] 1 illustrates a method for identifying a simplified signature representation of a landmark according to a disclosed embodiment. [Figure 25] 1 illustrates another method for identifying a simplified signature representation of a landmark, according to disclosed embodiments. [Figure 26] FIG. 1 illustrates an example block diagram of a memory according to disclosed embodiments. [Figure 27] 1 is a flowchart illustrating an example process for identifying a landmark identifier, according to the disclosed embodiments. [Figure 28] 1 is a flowchart illustrating an example process for updating and distributing a vehicle road navigation model based on an identifier, according to the disclosed embodiments. [Figure 29] FIG. 1 illustrates an example block diagram of a system for locating landmarks for use in autonomous vehicle navigation, according to disclosed embodiments. [Figure 30] FIG. 1 illustrates an example block diagram of a memory according to disclosed embodiments. [Figure 31] 1 illustrates an example scaling method for determining distance from a vehicle to a landmark, according to disclosed embodiments. [Figure 32] 1 illustrates an example optical flow method for determining distance from a vehicle to a landmark, according to disclosed embodiments. [Figure 33A]1 is a flowchart illustrating an example process for locating landmarks for use in autonomous vehicle navigation, according to disclosed embodiments. [Figure 33B] 1 is a flowchart illustrating an example process for determining the position of a landmark for use in navigation of an autonomous vehicle, according to disclosed embodiments. [Figure 34] 1 is a top view representation of an exemplary vehicle including a system according to disclosed embodiments, where the vehicle is navigated using landmarks. [Figure 35] 1 is another top view representation of an exemplary vehicle including a system according to disclosed embodiments, where the vehicle is navigated using landmarks. [Figure 36] 1 is a flowchart illustrating an example process for navigating an example vehicle using landmarks. [Figure 37] 1 is a top view representation of an exemplary autonomous vehicle including a system according to disclosed embodiments, in which the autonomous vehicle is navigated using tail alignment. [Figure 38] 1 is another top view representation of an example autonomous vehicle including a system according to disclosed embodiments, where the autonomous vehicle is navigated using tail alignment. [Figure 39] 1 is a flowchart illustrating an example process for navigating an example autonomous vehicle using tail alignment. [Figure 40] 1 is a top view representation of an exemplary vehicle including a system according to disclosed embodiments, in which the vehicle navigates at an intersection using two or more landmarks. [Figure 41] 1 is a flowchart illustrating an example process for navigating an example vehicle through an intersection using two or more landmarks. [Figure 42] 1 is a top view representation of an exemplary vehicle including a system according to disclosed embodiments, in which the vehicle is navigated using overlapping maps. [Figure 43A] 4 is a flowchart illustrating an example process for navigating an example vehicle using overlapping maps. [Figure 43B]4 is a flowchart illustrating an example process for navigating an example vehicle using overlapping maps. [Figure 43C] 4 is a flowchart illustrating an example process for navigating an example vehicle using overlapping maps. [Figure 44] 1 illustrates an exemplary remote server in communication with a vehicle, according to disclosed embodiments. [Figure 45] 1 illustrates a vehicle navigating along a multi-lane road according to a disclosed embodiment. [Figure 46] 1 illustrates a vehicle being navigated using a target trajectory along a multi-lane road in accordance with a disclosed embodiment. [Figure 47] 1 illustrates an example road signature profile according to the disclosed embodiments. [Figure 48] 1 illustrates an exemplary environment in accordance with disclosed embodiments. [Figure 49] 1 is a flowchart illustrating an exemplary process for sparse map autonomous vehicle navigation, according to disclosed embodiments. [Figure 50] 1 illustrates an example environment for autonomous navigation based on expected landmark locations, according to disclosed embodiments. [Figure 51] 1 illustrates an arrangement for autonomous navigation according to a disclosed embodiment. [Figure 52] 1 illustrates another example environment for autonomous navigation based on expected landmark locations, according to the disclosed embodiments. [Figure 53] 1 illustrates another example environment for autonomous navigation based on expected landmark locations, according to the disclosed embodiments. [Figure 54] 1 is a flowchart illustrating an exemplary process for autonomous navigation based on expected landmark positions, according to disclosed embodiments. [Figure 55] 1 is a diagrammatic representation of an exemplary vehicle control system according to the disclosed embodiments. [Figure 56] 1 is a top view representation of an exemplary vehicle including a system according to disclosed embodiments, in which the vehicle is navigated using a lane width profile or road width profile. [Figure 57] 1 is a graph illustrating an example profile that may be used by a vehicle control system according to the disclosed embodiments. [Figure 58] 1 is a top view representation of an exemplary vehicle including a system according to the disclosed embodiments, in which the vehicle is navigated using the length or spacing of road marks on a road segment. [Figure 59] 1 is a top view representation of an exemplary vehicle including a system according to a disclosed embodiment, in which the vehicle is navigated using information about the curvature of road segments. [Figure 60] 4 is a flowchart illustrating an example process for navigating an example vehicle using road signatures. [Figure 61A] 1 is a side view representation of an exemplary vehicle according to disclosed embodiments. [Figure 61B] 1 is a side view representation of an exemplary vehicle according to disclosed embodiments. [Figure 62] 1 is a top view representation of an exemplary vehicle autonomously navigated on a road, according to a disclosed embodiment; [Figure 63] 1 is a flowchart illustrating an exemplary process for autonomously navigating a vehicle according to disclosed embodiments. [Figure 64] 1 is a perspective view of an environment captured by a forward-facing image capture device of an exemplary vehicle in accordance with the disclosed embodiments; [Figure 65] 1 is an exemplary image received from a forward-facing image capture device of a vehicle in accordance with disclosed embodiments. [Figure 66] 1 is a flowchart illustrating an exemplary process for navigating a vehicle by identifying a free space region in which the vehicle may travel, according to disclosed embodiments. [Figure 67] 1 is a top view representation of an exemplary vehicle navigating a road with snow covering at least some lane markings and road edges, according to a disclosed embodiment; [Figure 68]1 is a flowchart illustrating an exemplary process for navigating a vehicle on a road with snow covering at least some lane markings and road edges, according to disclosed embodiments. [Figure 69] 1 is a top view representation of an exemplary vehicle including a system for calibrating vehicle speed, according to disclosed embodiments; [Figure 70] 1 is a flowchart illustrating an exemplary process for calibrating the speed of a vehicle, according to disclosed embodiments. [Figure 71] 1 is another top view representation of an exemplary vehicle including a system for calibrating vehicle speed, according to disclosed embodiments. [Figure 72] 10 is a flowchart illustrating another exemplary process for calibrating the speed of a vehicle, according to disclosed embodiments. [Figure 73] FIG. 1 is an illustration of a street view of an exemplary road segment in accordance with the disclosed embodiments; [Figure 74] FIG. 1 is an illustration of a bird's-eye view of an exemplary road segment in accordance with the disclosed embodiments; [Figure 75] 1 is a flowchart illustrating an example process for determining a lane assignment for a vehicle, according to disclosed embodiments. [Figure 76] FIG. 1 is an illustration of a street view of an exemplary road segment in accordance with the disclosed embodiments; [Figure 77A] FIG. 1 is an illustration of a bird's-eye view of an exemplary road segment in accordance with the disclosed embodiments; [Figure 77B] FIG. 1 is an illustration of a street view of an exemplary road segment in accordance with the disclosed embodiments; [Figure 78] 1 is a flowchart illustrating an exemplary process for autonomously navigating a vehicle along a road segment, according to disclosed embodiments. [Figure 79A] 1 illustrates a top view of a vehicle traveling on a road approaching winter icy road conditions at a particular location, according to a disclosed embodiment; [Figure 79B] 1 illustrates a top view of a vehicle traveling on a road approaching a pedestrian, according to a disclosed embodiment; [Figure 79C] 1 illustrates a top view of a vehicle traveling on a road near another vehicle, according to a disclosed embodiment; [Figure 79D] 1 illustrates a top view of a vehicle traveling in a terminating lane on a road, according to a disclosed embodiment; [Figure 80] 1 illustrates a side view representation of an exemplary vehicle including a system according to the disclosed embodiments. [Figure 81] 1 shows an example flowchart illustrating a method for adaptively navigating a vehicle based on user intervention, according to disclosed embodiments. [Figure 82A] 1 illustrates a top view of a vehicle traveling on a road with parked vehicles, according to a disclosed embodiment; [Figure 82B] 1 illustrates a top view of a vehicle traveling in a terminating lane on a road, according to a disclosed embodiment; [Figure 82C] 1 illustrates a top view of a vehicle traveling on a road approaching a pedestrian, according to a disclosed embodiment; [Figure 82D] 1 illustrates a top view of a vehicle traveling on a road approaching a construction area, according to a disclosed embodiment; [Figure 83] 1 shows an example flowchart illustrating a method for self-aware navigation of a vehicle, according to disclosed embodiments. [Figure 84A] 1 illustrates a top view of a vehicle traveling on a road with multiple parked vehicles, according to a disclosed embodiment; [Figure 84B] 1 illustrates a top view of a vehicle traveling on a road with an intervening vehicle immediately in front of the vehicle, according to a disclosed embodiment; [Figure 84C] 1 illustrates a top view of a vehicle traveling on a road with a temporary barrier directly in front of the vehicle, according to a disclosed embodiment; [Figure 84D] 1 illustrates a top view of a vehicle traveling on a road with temporary roadworks directly in front of the vehicle, according to a disclosed embodiment; [Figure 85A] 1 illustrates a top view of a vehicle traveling on a road with a pothole directly in front of the vehicle, according to a disclosed embodiment; [Figure 85B]1 illustrates a top view of a vehicle traveling on a road with animals and pedestrians crossing in front of the vehicle, according to a disclosed embodiment; [Figure 86] 1 illustrates an example flowchart depicting a method for an adaptive road model manager, according to the disclosed embodiments. [Figure 87A] 1 illustrates a top view of a vehicle traveling on an interstate highway, according to a disclosed embodiment. [Figure 87B] 1 illustrates a top view of a group of vehicles traveling on city roads, according to a disclosed embodiment; [Figure 87C] 1 illustrates a plan view of a vehicle traveling on a road within a particular rural geographic area, according to a disclosed embodiment; [Figure 87D] 1 illustrates a top view of a vehicle traveling on a road with lane shifting, according to a disclosed embodiment; [Figure 88] 1 illustrates an example flowchart depicting a road model management method based on selective feedback, according to disclosed embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0135] Detailed Description The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substituting, reordering, deleting, or adding steps of the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the appended claims.

[0136] Autonomous Vehicle Overview As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle capable of making at least one navigation change without driver input. A "navigation change" refers to a change in one or more of the vehicle's steering, braking, or acceleration. To be autonomous, a vehicle need not be fully automatic (e.g., operating completely without a driver or driver input). Rather, an autonomous vehicle includes a vehicle that can operate under driver control during certain periods of time and without driver control during other periods of time. An autonomous vehicle can also include a vehicle that controls only some aspects of vehicle navigation, such as steering (e.g., to maintain the vehicle course between vehicle lane constraints), but leaves other aspects (e.g., braking) to the driver. In some cases, an autonomous vehicle may handle some or all aspects of the vehicle's braking, speed control, and / or steering.

[0137] Because human drivers typically rely on visual cues and observations to control their vehicles, transportation infrastructure is built accordingly, with lane markings, traffic signs, and traffic lights all designed to provide visual information to drivers. In light of these design features of transportation infrastructure, autonomous vehicles may include cameras and processing units that analyze visual information captured from the vehicle's environment. Visual information may include, for example, transportation infrastructure components observable by the driver (e.g., lane markings, traffic signs, traffic lights, etc.) and other obstacles (e.g., other vehicles, pedestrians, debris, etc.). Furthermore, automated vehicles may use stored information when navigating, such as information that provides a model of the vehicle's environment. For example, the vehicle may use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information related to the vehicle's environment while the vehicle is traveling, and the vehicle (and other vehicles) may use that information to locate itself within the model.

[0138] In some embodiments of the present disclosure, an autonomous vehicle may use information obtained while navigating (e.g., from cameras, GPS devices, accelerometers, speed sensors, suspension sensors, etc.). In other embodiments, an autonomous vehicle may use information obtained from past navigation by the vehicle (or other vehicles) while navigating. In still other embodiments, an autonomous vehicle may use a combination of information obtained while navigating and information obtained from past navigation. In the following sections, an overview of systems according to disclosed embodiments is provided, followed by an overview of forward-looking imaging systems and methods according to the systems. The following sections disclose systems and methods for building, using, and updating sparse maps for autonomous vehicle navigation.

[0139] System Overview FIG. 1 is a block diagram representation of system 100 according to a disclosed exemplary embodiment. System 100 may include various components depending on particular implementation requirements. In some embodiments, system 100 may include processing unit 110, image acquisition unit 120, position sensor 130, one or more memory units 140, 150, map database 160, user interface 170, and wireless transceiver 172. Processing unit 110 may include one or more processing devices. In some embodiments, processing unit 110 may include application processor 180, image processor 190, or any other suitable processing device. Similarly, image acquisition unit 120 may include any number of image acquisition devices and components depending on the requirements of a particular application. In some embodiments, image acquisition unit 120 may include one or more image capture devices (e.g., cameras), such as image capture device 122, image capture device 124, and image capture device 126. System 100 may also include a data interface 128 that communicatively connects processing device 110 to image acquisition device 120. For example, data interface 128 may include any one or more wired and / or wireless links that transmit image data acquired by image acquisition device 120 to processing unit 110.

[0140] Wireless transceiver 172 may include one or more devices configured to transmit and receive data to and from one or more networks (e.g., cellular, Internet, etc.) over a wireless interface through the use of radio frequencies, infrared frequencies, magnetic fields, or electric fields. Wireless transceiver 172 may transmit and / or receive data using any known standard (e.g., Wi-Fi, Bluetooth, Bluetooth Smart, 802.15.4, ZigBee, etc.).

[0141] Both application processor 180 and image processor 190 may include various types of processing devices. For example, either or both of application processor 180 and image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, memory, or any other type of device suitable for running applications and processing and analyzing images. In some embodiments, application processor 180 and / or image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. Various processing devices may be used, including, for example, processors available from manufacturers such as Intel®, AMD®, etc., and may include various architectures (e.g., x86 processor, ARM®, etc.).

[0142] In some embodiments, application processor 180 and / or image processor 190 may include any of the EyeQ series of processors available from Mobileye®. These processor designs include multiple processing units, each with its own local memory and instruction set. Such processors may include video inputs that receive image data from multiple image sensors and may also include video output capabilities. In one example, the EyeQ2® uses 90 nm-micron technology operating at 332 MHz. The EyeQ2® architecture consists of two floating-point, hyper-threaded 32-bit RISC CPUs (MIPS32® 34K® cores), five Vision Computation Engines (VCEs), three Vector Microcode Processors (VMP®), a Denali 64-bit mobile DDR controller, a 128-bit internal audio interconnect, dual 16-bit video input and 18-bit video output controllers, a 16-channel DMA, and several peripherals. The MIPS34K CPU manages five VCEs, three VMPs™ and DMA, a second MIPS34K CPU and multi-channel DMA, and other peripherals. The five VCEs, three VMPs, and the MIPS34K CPU can perform intensive vision calculations required by feature-rich bundled applications. In another example, the disclosed embodiment may use the EyeQ3™, a third-generation processor that is six times more powerful than the EyeQ2™.

[0143] Any processing device disclosed herein may be configured to perform a particular function. Configuring a processing device, such as any of the described EyeQ processors or other controllers or microprocessors, to perform a particular function may include programming computer-executable instructions and providing those instructions to the processing device for execution during operation of the processing device. In some embodiments, configuring a processing device may include programming architectural instructions directly into the processing device. In other embodiments, configuring a processing device may include storing executable instructions in memory accessible by the processing device during operation. For example, the processing device may access the memory to retrieve and execute the stored instructions during operation.

[0144] 1 shows two separate processing devices included in processing unit 110, more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to accomplish the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by three or more processing devices. Furthermore, in some embodiments, system 100 may include one or more of processing units 110 without including other components, such as image acquisition unit 120.

[0145] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices such as a controller, an image preprocessor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, memory, or any other type of device that processes and analyzes images. The image preprocessor may include a video processor that captures, digitizes, and processes images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The support circuits may be any number of circuits commonly known in the art, including cache, power supplies, clocks, and input / output circuits. The memory may store software that, when executed by the processor, controls the operation of the system. The memory may include databases and image processing software. The memory may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage devices, tape storage devices, removable storage devices, and other types of storage devices. In one example, the memory may be separate from the processing unit 110. In another example, the memory may be integrated into the processing unit 110.

[0146] Each memory 140, 150 may contain software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may include various databases and image processing software. The memory units may include random access memory, read-only memory, flash memory, disk drives, optical storage devices, tape storage devices, removable storage devices, and / or any other type of storage device. In some embodiments, memory units 140, 150 may be separate from application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into application processor 180 and / or image processor 190.

[0147] Position sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, position sensor 130 may include a GPS receiver. Such a receiver may determine the location and velocity of a user by processing signals broadcast by Global Positioning System satellites. Position information from position sensor 130 may be provided to application processor 180 and / or image processor 190.

[0148] In some embodiments, the system 100 may include components such as a speed sensor (e.g., a tachometer) to measure the speed of the vehicle 200 and / or an accelerometer to measure the acceleration of the vehicle 200 .

[0149] User interface 170 may include any device suitable for providing information or receiving input from one or more users of system 100. In some embodiments, user interface 170 may include user input devices including, for example, a touchscreen, a microphone, a keyboard, a pointer device, a track wheel, a camera, a knob, buttons, etc. Using such input devices, a user may be able to provide information input or commands to system 100 by typing instructions or information, providing voice commands, selecting on-screen menu options using buttons, a pointer, or eye tracking, or through any other suitable technique for communicating information to system 100.

[0150] User interface 170 may comprise one or more processing devices configured to provide information to or receive information from a user and process that information, for example, for use by application processor 180. In some embodiments, such processing devices may execute instructions to recognize and track eye movements, receive and interpret voice commands, recognize and interpret touches and / or gestures made on a touchscreen, respond to keyboard entries or menu selections, etc. In some embodiments, user interface 170 may include a display, a speaker, a tactile device, and / or any other device that provides output information to a user.

[0151] Map database 160 may include any type of database that stores map data useful to system 100. In some embodiments, map database 160 may include data related to the locations in a reference coordinate system of various items, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc. Map database 160 may store not only the locations of such items, but also descriptors related to those items, including, for example, names associated with any of the stored features. In some embodiments, map database 160 may be physically located with other components of system 100. Alternatively, or in addition, map database 160, or portions thereof, may be located remotely with respect to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 may be downloaded to a network via a wired or wireless data connection (e.g., via a cellular network and / or the Internet, etc.).

[0152] Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from an environment. Furthermore, any number of image capture devices may be used to obtain images for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four or more image capture devices. Image capture devices 122, 124, and 126 are further described below with reference to Figures 2B-2E.

[0153] System 100 or various components of system 100 may be incorporated into a variety of different platforms. In some embodiments, system 100 may be included in vehicle 200, as shown in FIG. 2A. For example, vehicle 200 may include processing unit 110 and any other components of system 100, as described above with respect to FIG. 1. In some embodiments, vehicle 200 may include only a single image capture device (e.g., a camera), while in other embodiments, such as those discussed in connection with FIGS. 2B-2E, multiple image capture devices may be used. For example, as shown in FIG. 2A, either of image capture devices 122 and 124 of vehicle 200 may be part of an ADAS (Advanced Driver Assistance Systems) imaging suite.

[0154] The image capture device included in vehicle 200 as part of image acquisition unit 120 may be located in any suitable location. In some embodiments, as shown in Figures 2A-2E and 3A-3C, image capture device 122 may be located near the rearview mirror. This location may provide a line of sight similar to that of the driver of vehicle 200 and may assist in determining what the driver can and cannot see. While image capture device 122 may be located anywhere near the rearview mirror, placing image capture device 122 on the driver's side of the mirror may further assist in capturing images representative of the driver's field of view and / or line of sight.

[0155] Other locations for the image capture devices of image acquisition unit 120 can also be used. For example, image capture device 124 can be located on or within the bumper of vehicle 200. Such a location may be particularly suitable for image capture devices with a wide field of view. The line of sight of an image capture device located on the bumper can be different from the line of sight of the driver, and thus the bumper image capture device and the driver do not always see the same object. Image capture devices (e.g., image capture devices 122, 124, and 126) can also be located in other locations. For example, the image capture devices can be located on one or both of the side mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on the sides of vehicle 200, mounted on, located behind, or located in front of any window of vehicle 200, mounted near or at the front and / or rear lights of vehicle 200, etc.

[0156] In addition to the image capture device, vehicle 200 may include various other components of system 100. For example, processing unit 110 may be integrated into the vehicle's engine control unit (ECU) or may be included in vehicle 200 separate from the ECU. Vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver, and vehicle 200 may also include a map database 160 and memory units 140 and 150.

[0157] As described above, wireless receiver 172 may receive data via one or more networks (e.g., a cellular network, the Internet, etc.). For example, wireless transceiver 172 may upload data collected by system 100 to one or more servers and download data from one or more servers. Via wireless transceiver 172, system 100 may receive updates to data stored in map database 160, memory 140, and / or memory 150, for example, periodically or on demand. Similarly, wireless transceiver 172 may upload any data from system 100 (e.g., images captured by image acquisition unit 120, data received by position sensor 130, other sensors, or vehicle control systems, etc.) and / or any data processed by processing unit 110 to one or more servers.

[0158] System 100 may upload data to a server (e.g., the cloud) based on a privacy level setting. For example, system 100 may implement a privacy level setting that regulates or limits the types of data (including metadata) that may uniquely identify the vehicle and / or the driver / owner of the vehicle that are sent to the server. Such settings may be set by a user via wireless transceiver 172, initialized by factory default settings, or set by data received by wireless transceiver 172, for example.

[0159] In some embodiments, system 100 may upload data according to a "high" privacy level, under a configuration setting, system 100 may transmit data (e.g., location information associated with a route, captured images, etc.) without any details about the specific vehicle and / or driver / owner. For example, when uploading data according to a "high" privacy level, system 100 may not include the vehicle identification number (VIN) or the name of the vehicle's driver or owner, but instead transmit data such as captured images and / or limited location information associated with a route.

[0160] Other privacy levels are also contemplated. For example, system 100 may transmit data to a server according to a “medium” privacy level, which may include additional information not included under a “high” privacy level, such as the make and / or model of the vehicle and / or vehicle type (e.g., passenger car, sport utility vehicle, truck, etc.). In some embodiments, system 100 may upload data according to a “low” privacy level. Under the “low” privacy level setting, system 100 may upload data including sufficient information to uniquely identify a particular vehicle, owner / driver, and / or part or all of the route traveled by the vehicle. Such “low” privacy level data may include, for example, one or more of the VIN, driver / owner name, vehicle's starting point prior to departure, vehicle's intended destination, vehicle make and / or model, vehicle type, etc.

[0161] Figure 2A is a side view representation of an exemplary vehicle imaging system according to a disclosed embodiment. Figure 2B is a top view representation of the embodiment shown in Figure 2A. As shown in Figure 2B, the disclosed embodiment may show a vehicle 200 including within its body a system 100 having a first image capture device 122 positioned near a rearview mirror and / or near a driver of the vehicle 200, a second image capture device 124 positioned on or within a bumper area (e.g., one of bumper areas 210) of the vehicle 200, and a processing unit 110.

[0162] As shown in Figure 2C, image capture devices 122 and 124 may both be positioned near the rearview mirror and / or near the driver of vehicle 200. Furthermore, while two image capture devices 122 and 124 are shown in Figures 2B and 2C, it should be understood that other embodiments may include three or more image capture devices. For example, in the embodiment shown in Figures 2D and 2E, a first image capture device 122, a second image capture device 124, and a third image capture device 126 are included in system 100 of vehicle 200.

[0163] 2D , image capture device 122 may be positioned near the rearview mirror and / or near the driver of vehicle 200, and image capture devices 124 and 126 may be positioned on a bumper area (e.g., one of bumper areas 210) of vehicle 200. Also, as shown in FIG. 2E , image capture devices 122, 124, and 126 may be positioned near the rearview mirror and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number or configuration of image capture devices, and image capture devices may be positioned in any suitable location within and / or on vehicle 200.

[0164] It should be understood that the disclosed embodiments are not limited to vehicles and may be applicable in other contexts. It should also be understood that the disclosed embodiments are not limited to a particular type of vehicle 200 and may be applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0165] First image capture device 122 may include any suitable type of image capture device. Image capture device 122 may include an optical axis. In one example, image capture device 122 may include an Aptina M9V024 WVGA sensor with a global shutter. In other embodiments, image capture device 122 may provide a resolution of 1280 x 960 pixels and may include a rolling shutter. Image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included to provide, for example, a desired focal length and field of view for the image capture device. In some embodiments, image capture device 122 may be associated with a 6 mm lens or a 12 mm lens. In some embodiments, image capture device 122 may be configured to capture an image having a desired field of view (FOV) 202, as shown in FIG. 2D . For example, image capture device 122 may be configured to have a conventional FOV, such as within the range of 40 degrees to 56 degrees, including a 46 degree FOV, a 50 degree FOV, a 52 degree FOV, or degrees greater than 52 degrees. Alternatively, image capture device 122 may be configured to have a narrower FOV, such as a 23-40 degree FOV, such as a 28 degree FOV or a 36 degree FOV. Additionally, image capture device 122 may be configured to have a wider FOV, such as a 100-180 degree FOV. In some embodiments, image capture device 122 may include a wide-angle bumper camera or a bumper camera with an FOV of up to 180 degrees. In some embodiments, image capture device 122 may be a 7.2 Mpixel image capture device with an aspect ratio of approximately 2:1 (e.g., H×V=3800×1900 pixels) and a horizontal FOV of approximately 100 degrees. Such an image capture device may be used in place of a configuration with three image capture devices. Due to large lens distortion, the vertical FOV of such image capture devices can be much less than 50 degrees in implementations in which the image capture device uses a radially symmetric lens, for example, such lenses are not radially symmetric, thereby allowing a vertical FOV greater than 50 degrees with a horizontal FOV of 100 degrees.

[0166] First image capture device 122 may acquire multiple first images of a scene associated with vehicle 200. The multiple first images may each be acquired as a series of image scan lines, which may be captured using a rolling shutter. Each scan line may include multiple pixels.

[0167] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor can acquire image data associated with each pixel included in a particular scan line.

[0168] Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be utilized with a rolling shutter, whereby each pixel in a row is read one at a time, and the scanning of the rows proceeds row by row until the entire image frame is captured. In some embodiments, the rows may be captured sequentially from top to bottom relative to the frame.

[0169] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute high-resolution imagers and may have a resolution of greater than 5M pixels, greater than 7M pixels, greater than 10M pixels, or more.

[0170] The use of a rolling shutter can result in pixels in different rows being exposed and captured at different times, which can cause skew and other image artifacts in the captured image frame. On the other hand, if image capture device 122 is configured to operate using a global or synchronous shutter, all pixels can be exposed for the same amount of time during a common exposure period. As a result, image data in a frame collected from a system utilizing a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a particular time. Conversely, when applying a rolling shutter, each row in a frame is exposed at a different time, and data is captured at a different time. Therefore, moving objects can appear distorted with an image capture device having a rolling shutter. This phenomenon is described in more detail below.

[0171] Second image capture device 124 and third image capture device 126 may be any type of image capture device. Like first image capture device 122, each of image capture devices 124 and 126 may include an optical axis. In one embodiment, each of image capture devices 124 and 126 may include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of image capture devices 124 and 126 may include a rolling shutter. Like image capture device 122, image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, the lenses associated with image capture devices 124 and 126 may provide the same FOV as that associated with image capture device 122 (such as FOV 202) or a narrower FOV (such as FOVs 204 and 206). For example, image capture devices 124 and 126 may have an FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less than 20 degrees.

[0172] Image capture devices 124 and 126 may acquire second and third multiple images of a scene associated with vehicle 200. Each of the second and third multiple images may be acquired as second and third series of image scan lines, which may be captured using a rolling shutter. Each scan line or row may have a plurality of pixels. Image capture devices 124 and 126 may have second and third scan rates associated with acquiring each image scan line included in the second and third series.

[0173] Each image capture device 122, 124, and 126 may be positioned in any suitable location and in any suitable orientation relative to vehicle 200. The relative positions of image capture devices 122, 124, and 126 may be selected to facilitate fusing together information obtained from the image capture devices. For example, in some embodiments, the FOV associated with image capture device 124 (FOV 204) may partially or completely overlap with the FOV associated with image capture device 122 (e.g., FOV 202) and the FOV associated with image capture device 126 (e.g., FOV 206).

[0174] Image capture devices 122, 124, and 126 may be positioned on vehicle 200 at any suitable relative height. In one example, there may be a height difference between image capture devices 122, 124, and 126, which may provide sufficient parallax information to enable stereo analysis. For example, as shown in FIG. 2A , two image capture devices 122 and 124 are at different heights. There may also be a lateral displacement difference between image capture devices 122, 124, and 126, which may provide additional parallax information for stereo analysis by processing unit 110, for example. The lateral displacement difference may be denoted by dx, as shown in FIGS. 2C and 2D . In some embodiments, a front displacement or a rear displacement (e.g., a range displacement) may exist between image capture devices 122, 124, and 126. For example, image capture device 122 may be positioned 0.5 to 2 meters or more behind image capture device 124 and / or image capture device 126. This type of displacement may allow one of the image capture devices to cover a potential blind spot of the other image capture device.

[0175] Image capture device 122 may have any suitable resolution capability (e.g., number of pixels associated with the image sensor), and the resolution of the image sensor associated with image capture device 122 may be higher, lower, or the same as the resolution of the image sensors associated with image capture devices 124 and 126. In some embodiments, the image sensors associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640x480, 1024x768, 1280x960, or any other suitable resolution.

[0176] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data for one image frame before moving on to acquire pixel data associated with the next image frame) may be controllable. The frame rate associated with image capture device 122 may be higher, lower, or the same as the frame rate associated with image capture devices 124 and 126. The frame rates associated with image capture devices 122, 124, and 126 may depend on various factors that may affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include a selectable pixel delay period imposed before or after acquisition of image data associated with one or more pixels of an image sensor within image capture devices 122, 124, and / or 126. Generally, image data corresponding to each pixel may be acquired according to the device's clock rate (e.g., one pixel per clock cycle). Furthermore, in embodiments including a rolling shutter, one or more of image capture devices 122, 124, and 126 may include a selectable horizontal blanking period imposed before or after acquisition of image data associated with a row of pixels of an image sensor in image capture devices 122, 124, and / or 126. Furthermore, one or more of image capture devices 122, 124, and / or 126 may include a selectable vertical blanking period imposed before or after acquisition of image data associated with an image frame of image capture devices 122, 124, and 126.

[0177] These timing controls may enable the frame rates associated with image capture devices 122, 124, and 126 to be synchronized even if the line scan rate of each image capture device is different. Additionally, as discussed in more detail below, among other factors (e.g., image sensor resolution, maximum line scan rate, etc.), these selectable timing controls may enable the synchronization of image capture from areas where the FOV of image capture device 122 overlaps with the FOV of one or more of image capture devices 124 and 126, even if the field of view of image capture device 122 differs from the FOV of image capture devices 124 and 126.

[0178] The frame rate timing for image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensors. For example, assuming both devices have similar line scan rates, if one device includes an image sensor with a resolution of 640x480 and the other device includes an image sensor with a resolution of 1280x960, it will take longer to capture a frame of image data from the sensor with the higher resolution.

[0179] Another factor that may affect the timing of image data acquisition at image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquisition of a row of image data from the image sensors included in image capture devices 122, 124, and 126 requires some minimum amount of time. Assuming no pixel delay period is added, this minimum amount of time to acquire a row of image data will be related to the maximum line scan rate of a particular device. Devices offering higher maximum line scan rates have the potential to provide higher frame rates than devices with lower maximum line scan rates. In some embodiments, one or both of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 or more times the maximum line scan rate of image capture device 122.

[0180] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate that is equal to or less than that maximum scan rate. The system may be configured so that one or both of image capture devices 124 and 126 operate at a line scan rate that is equal to the line scan rate of image capture device 122. In other examples, the system may be configured so that the line scan rate of image capture devices 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 times or more the line scan rate of image capture device 122.

[0181] In some embodiments, image capture devices 122, 124, and 126 may be asymmetric. That is, the image capture devices may include cameras with different fields of view (FOV) and focal lengths. The fields of view of image capture devices 122, 124, and 126 may include, for example, any desired area of ​​the environment of vehicle 200. In some embodiments, one or more of image capture devices 122, 124, and 126 may be configured to acquire image data from the environment in front of vehicle 200, the environment behind vehicle 200, the environment on either side of vehicle 200, or a combination thereof.

[0182] Additionally, the focal length associated with each image capture device 122, 124, and / or 126 may be selectable (e.g., by inclusion of an appropriate lens, etc.) so that each device captures images of objects at a desired distance range from vehicle 200. For example, in some embodiments, image capture devices 122, 124, and 126 may capture images of close-up objects within a few meters of the vehicle. Image capture devices 122, 124, 126 may also be configured to capture images of objects at greater distances from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more). Furthermore, the focal lengths of image capture devices 122, 124, and 126 may be selected such that one image capture device (e.g., image capture device 122) can capture images of objects relatively close to the vehicle (e.g., within 10 m or within 20 m), while the other image capture devices (e.g., image capture devices 124 and 126) can capture images of objects farther away from vehicle 200 (e.g., more than 20 m, more than 50 m, more than 100 m, more than 150 m, etc.).

[0183] According to some embodiments, the FOV of one or more of image capture devices 122, 124, and 126 may have a wide angle. For example, it may be advantageous to have an FOV of 140 degrees, particularly for image capture devices 122, 124, and 126 that may be used to capture images of areas near vehicle 200. For example, image capture device 122 may be used to capture images of areas to the right or left of vehicle 200, and in such embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0184] The field of view associated with each of image capture devices 122, 124, and 126 may depend on the respective focal lengths. For example, as the focal lengths increase, the corresponding field of view decreases.

[0185] Image capture devices 122, 124, and 126 may be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV between 23 and 46 degrees. In another example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV between 26 and 52 degrees. In some embodiments, the ratio between the FOV of image capture device 122 and the FOV of image capture device 124 and / or image capture device 126 may vary between 1.5 and 2.0. In other embodiments, this ratio may vary between 1.25 and 2.25.

[0186] System 100 may be configured so that the field of view of image capture device 122 at least partially or completely overlaps the field of view of image capture device 124 and / or image capture device 126. In some embodiments, system 100 may be configured so that the fields of view of image capture devices 124 and 126, for example, fall within the field of view of image capture device 122 (e.g., are narrower than the field of view of image capture device 122) and share a common center with the field of view of image capture device 122. In other embodiments, image capture devices 122, 124, and 126 may capture adjacent FOVs or may have partially overlapping FOVs. In some embodiments, the fields of view of image capture devices 122, 124, and 126 may be aligned such that the center of image capture device 124 and / or 126, with the narrower FOV, may be located in the bottom half of the field of view of device 122, with the wider FOV.

[0187] 2F is a diagrammatic representation of an exemplary vehicle control system according to disclosed embodiments. As shown in FIG. 2F, vehicle 200 may include a throttle system 220, a braking system 230, and a steering system 240. System 100 may provide inputs (e.g., control signals) to one or more of throttle system 220, braking system 230, and steering system 240 via one or more data links (e.g., any wired and / or wireless link or link that transmits data). For example, based on analysis of images acquired by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of throttle system 220, braking system 230, and steering system 240 to navigate vehicle 200 (e.g., by causing it to accelerate, turn, shift lanes, etc.). Additionally, system 100 may receive inputs indicative of the operating conditions of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or turning, etc.) from one or more of throttle system 220, braking system 230, and steering system 24. Further details are provided below in connection with Figures 4-7.

[0188] As shown in FIG. 3A , vehicle 200 may also include a user interface 170 for interacting with a driver or passenger of vehicle 200. For example, user interface 170 in a vehicle application may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver or passenger of vehicle 200 may also interact with system 100 using a steering wheel (e.g., located on or near the steering column of vehicle 200, including, for example, a turn signal handle), buttons (e.g., located on the steering wheel of vehicle 200), and the like. In some embodiments, microphone 350 may be positioned adjacent to rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, system 100 may provide various notifications (e.g., alerts) via speaker 360.

[0189] 3B-3D are diagrams of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and opposite a vehicle windshield, according to disclosed embodiments. As shown in FIG. 3B , camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind glare shield 380, which may be in direct contact with the windshield and may include a film and / or anti-reflective material composition. For example, glare shield 380 may be positioned to be aligned opposite a windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 may be positioned behind glare shield 380, for example, as shown in FIG. 3D . The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380. FIG. 3C is a view of the camera mount 370 shown in FIG. 3B from the front.

[0190] As will be appreciated by those skilled in the art having the benefit of this disclosure, many variations and / or modifications may be made to the above-disclosed embodiments. For example, not all components are essential to the operation of system 100. Furthermore, any component may be located in any suitable portion of system 100, and the components may be rearranged in various configurations while still providing the functionality of the disclosed embodiments. Accordingly, the above-described configurations are examples, and regardless of the above-described configuration, system 100 may provide a wide range of functionality for analyzing the surroundings of vehicle 200 and navigating vehicle 200 in response to the analysis.

[0191] As discussed in more detail below, according to various disclosed embodiments, system 100 may provide various features related to autonomous driving and / or driver assistance technologies. For example, system 100 may analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 may collect data for analysis from, for example, image acquisition unit 120, location sensor 130, and other sensors. Furthermore, system 100 may analyze the collected data to identify whether vehicle 200 should take a particular action and then automatically take the determined action without human intervention. For example, when vehicle 200 navigates without human participation, system 100 may automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle system 220, braking system 230, and steering system 240). Additionally, system 100 may analyze the collected data and issue warnings and / or alerts to vehicle occupants based on the analysis of the collected data. Further details regarding various embodiments provided by system 100 are provided below.

[0192] Forward-facing multi-imaging system As described above, system 100 may provide driver assistance features using a multi-camera system. The multi-camera system may use one or more cameras facing forward of the vehicle. In other embodiments, the multi-camera system may include one or more cameras facing the side of the vehicle or the rear of the vehicle. In one embodiment, for example, system 100 may use a two-camera imaging system, in which a first camera and a second camera (e.g., image capture devices 122 and 124) may be positioned at the front and / or side of the vehicle (e.g., vehicle 200). The first camera may have a field of view that is larger, smaller, or partially overlaps the field of view of the second camera. Furthermore, the first camera may be connected to a first image processor to perform monocular image analysis of images provided by the first camera, and the second camera may be connected to a second image processor to perform monocular image analysis of images provided by the second camera. The outputs (e.g., processed information) of the first and second image processors may be combined. In some embodiments, the second image processor may receive images from both the first and second cameras and perform stereo analysis. In another embodiment, system 100 may use a three-camera imaging system, where each camera has a different field of view. Thus, such a system may make decisions based on information derived from objects at various distances both in front of and to the sides of the vehicle. References to monocular image analysis may refer to cases where image analysis is performed based on images captured from a single viewpoint (e.g., a single camera). Stereo image analysis may refer to cases where image analysis is performed based on two or more images captured with one or more image capture parameters changed. For example, captured images suitable for performing stereo analysis may include images captured from two or more different positions, images captured from different fields of view, images captured using different focal lengths, images captured with parallax information, etc.

[0193] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122-126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or other value selected from the range of approximately 20 to 45 degrees), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other value selected from the range of approximately 100 to approximately 180 degrees), and image capture device 126 may provide a medium field of view (e.g., 46 degrees or other value selected from the range of approximately 35 to approximately 60 degrees). In some embodiments, image capture device 126 may operate as the main or primary camera. Image capture devices 122-126 may be positioned substantially side-by-side (e.g., 6 cm apart) behind rearview mirror 310. Furthermore, in some embodiments, as described above, one or more of image capture devices 122-126 may be mounted behind a glare shield 380 that is flush with the windshield of vehicle 200. Such a shield may operate to minimize the effect of any reflections from the interior of the vehicle on the image capture devices 122-126.

[0194] 3B and 3C, the wide field of view camera (e.g., image capture device 124 in the example above) can be mounted lower than the narrow main field of view camera (e.g., image capture devices 122 and 126 in the example above). This configuration can provide a free line of sight from the wide field of view camera. To reduce reflections, the camera can be mounted near the windshield of vehicle 200 and can include a polarizer to attenuate reflected light.

[0195] A three-camera system may offer certain performance features. For example, some embodiments may include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the three-camera configuration described above, processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips, as described above), each dedicated to processing images captured by one or more of image capture devices 122-126.

[0196] In a three-camera system, a first processing device may receive images from both the primary camera and the narrow FOV camera and perform vision processing for the narrow FOV camera to detect, for example, other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the first processing device may calculate pixel disparity between images from the primary camera and the narrow camera to create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or 3D information calculated based on information from another camera.

[0197] The second processing device may receive images from the primary camera and perform vision processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Furthermore, the second processing device may calculate camera displacement, calculate pixel disparity between successive images based on the displacement, and create a 3D reconstruction (e.g., structure-from-motion) of the scene. The second processing device may send the structure-from-motion based on the 3D reconstruction to the first processing device and combine the structure-from-motion with the stereoscopic 3D image.

[0198] The third processing device may receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device may further execute additional processing instructions to analyze the images and identify moving objects in the images, such as vehicles changing lanes, pedestrians, etc.

[0199] In some embodiments, having image-based information streams captured and processed independently may provide an opportunity for redundancy in the system, such as using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0200] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) in providing navigation assistance to vehicle 200, and may use a third image capture device (e.g., image capture device 126) to provide redundancy and verify the analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for stereo analysis by system 100 for navigating vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and verification of information derived based on images captured from image capture device 123 and / or image capture device 124. That is, image capture device 126 (and corresponding processing device) may be considered to provide a redundant subsystem that provides a check on the analysis derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system).

[0201] Those skilled in the art will recognize that the above camera configurations, camera placements, camera numbers, camera locations, etc. are merely exemplary. These components, etc., described for an overall system, may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of multi-camera systems to provide driver assistance and / or autonomous vehicle functionality follow below.

[0202] 4 is an example functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions to perform one or more operations in accordance with the disclosed embodiments. While reference is made below to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0203] 4, memory 140 may store a monocular image analysis module 402, a stereo image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 402-408 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps of the following processes may be performed by one or more processing devices.

[0204] In one embodiment, monocular image analysis module 402 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform monocular image analysis of a set of images acquired by one of image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the set of images with additional sensor information (e.g., information from radar) to perform the monocular image analysis. As described in connection with FIGS. 5A-5D below, monocular image analysis module 402 may include instructions to detect a set of features in the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazards, and any other features associated with the vehicle's environment. Based on the analysis, system 100 (e.g., via processing unit 110) may cause one or more navigation responses in vehicle 200, such as a turn, a lane shift, and an acceleration change, as discussed below in connection with navigation response module 408.

[0205] In one embodiment, stereo image analysis module 404 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform stereo image analysis of first and second sets of images acquired by a combination of image capture devices selected from image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the first and second sets of images with additional sensor information (e.g., information from radar) to perform stereo image analysis. For example, stereo image analysis module 404 may include instructions to perform stereo image analysis based on the first set of images acquired by image capture device 124 and the second set of images acquired by image capture device 126. As described below in connection with FIG. 6 , stereo image analysis module 404 may include instructions to detect a set of features in the first and second sets of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and hazards. Based on the analysis, the processing unit 110 may cause one or more navigation responses in the vehicle 200, such as turns, lane shifts, and acceleration changes, as described below in connection with the navigation response module 408.

[0206] In one embodiment, speed and acceleration module 406 may store software configured to analyze data received from one or more computational and electromechanical devices within vehicle 200 configured to alter the speed and / or acceleration of vehicle 200. For example, processing unit 110 may execute instructions associated with speed and acceleration module 406 to calculate a target speed of vehicle 200 based on data derived from the execution of monocular image analysis module 402 and / or stereo image analysis module 404. Such data may include, for example, target position, speed, and / or acceleration, the position and / or speed of vehicle 200 relative to nearby vehicles, pedestrians, or road objects, and the position information of vehicle 200 relative to lane markings on the road. Additionally, processing unit 110 may calculate a target speed of vehicle 200 based on sensor input (e.g., information from radar) and input from other systems of vehicle 200, such as throttle system 220, braking system 230, and / or steering system 240 of vehicle 200. Based on the calculated target speed, the processing unit 110 may send electronic signals to the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration, for example, by physically releasing the brakes or easing the accelerator of the vehicle 200.

[0207] In one embodiment, the navigation response module 408 may be executable by the processing unit 110 and may store software that determines a desired navigation response based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and speed information associated with nearby vehicles, pedestrians, and road objects, as well as target position information for the vehicle 200. Furthermore, in some embodiments, the navigation response may be based (partially or fully) on map data, a predetermined position of the vehicle 200, and / or a relative speed or relative acceleration between the vehicle 200 and one or more objects detected from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine a desired navigation response based on sensor inputs (e.g., information from radar) and inputs from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and the steering system 240 of the vehicle 200. Based on the desired navigation response, processing unit 110 may send electronic signals to throttle system 220, braking system 230, and steering system 240 of vehicle 200 to trigger the desired navigation response, for example, by turning the steering wheel of vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, processing unit 110 may use the output of navigation response module 408 (e.g., the desired navigation response) as input to the execution of velocity and acceleration module 406 to calculate a change in velocity of vehicle 200.

[0208] 5A is a flowchart illustrating an example process 500A for generating one or more navigational responses based on monocular image analysis, according to a disclosed embodiment. At step 510, processing unit 110 may receive a plurality of images via data interface 128 between processing unit 110 and image acquisition unit 120. For example, a camera (such as image capture device 122 having field of view 202) included in image acquisition unit 120 may capture a plurality of images of an area in front of vehicle 200 (or, for example, to the side or rear of the vehicle) and transmit them to processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). Processing unit 110 may execute monocular image analysis module 402 to analyze the plurality of images at step 520, as described in further detail below in connection with FIGS. 5B-5D . By performing the analysis, processing unit 110 may detect a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, and traffic lights.

[0209] Processing unit 110 may also execute monocular image analysis module 402 in step 520 to detect various road hazards, such as, for example, truck tire parts, fallen road signs, loose cargo, and small animals. Road hazards may vary in structure, shape, size, and color, making such hazards more difficult to detect. In some embodiments, processing unit 110 may execute monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera movement between consecutive image frames and calculate pixel disparity between frames to construct a 3D map of the road. Processing unit 110 may then use the 3D map to detect the road surface and hazards present on the road surface.

[0210] In step 530, processing unit 110 may execute navigation response module 408 to cause vehicle 200 to make one or more navigational responses based on the analysis performed in step 520 and the techniques described above in connection with FIG. 4 . Navigational responses may include, for example, a turn, a lane shift, an acceleration change, and the like. In some embodiments, processing unit 110 may use data derived from execution of speed and acceleration module 406 to cause one or more navigational responses. Furthermore, multiple navigational responses may occur simultaneously, sequentially, or any combination thereof. For example, processing unit 110 may cause vehicle 200 to cross a lane and then, for example, accelerate, for example, by sequentially sending control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 may cause vehicle 200 to brake and simultaneously shift lanes, for example, by simultaneously sending control signals to braking system 230 and steering system 240 of vehicle 200.

[0211] FIG. 5B is a flowchart illustrating an example process 500B for detecting one or more vehicles and / or pedestrians in a set of images according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500B. In step 540, processing unit 110 may identify a set of candidate objects representing potentially present vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify locations within each image that may contain a target object (e.g., a vehicle, a pedestrian, or portions thereof). The predetermined patterns may be specified to achieve a low rate of “false hits” and a low rate of “misses.” For example, processing unit 110 may use a low similarity threshold to the predetermined patterns to identify candidate objects as possible vehicles or pedestrians. By doing so, processing unit 110 may reduce the probability of missing (e.g., not identifying) a candidate object representing a vehicle or pedestrian.

[0212] At step 542, processing unit 110 may filter the set of candidate objects to eliminate certain candidates (e.g., irrelevant or less relevant objects) based on classification criteria. Such criteria may be derived from various characteristics associated with object types stored in a database (e.g., a database stored in memory 140). The characteristics may include the object's shape, dimensions, texture, and location (e.g., relative to vehicle 200), etc. Thus, processing unit 110 may use one or more sets of criteria to reject false candidates from the set of candidate objects.

[0213] At step 544, processing unit 110 may analyze multiple image frames to identify whether objects in the set of candidate images represent vehicles and / or pedestrians. For example, processing unit 110 may track detected candidate objects across successive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). Additionally, processing unit 110 may estimate parameters of the detected objects and compare the object's frame-by-frame position data to predicted positions.

[0214] In step 546, processing unit 110 may construct a set of measurements of the detected objects. Such measurements may include, for example, position, velocity, and acceleration values ​​(relative to vehicle 200) associated with the detected objects. In some embodiments, processing unit 110 may construct the measurements based on estimation techniques using a series of time-based observations, such as a Kalman filter or linear quadratic estimation (LQE), and / or modeling data available for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). A Kalman filter may be based on measurements of the scale of the objects, where the scale measurement is proportional to the time to impact (e.g., the amount of time it takes vehicle 200 to reach the object). Thus, by performing steps 540-546, processing unit 110 may identify vehicles and pedestrians appearing in the set of captured images and derive information (e.g., position, velocity, size) associated with the vehicles and pedestrians. Based on the identification and derived information, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in connection with FIG. 5A .

[0215] In step 548, processing unit 110 may perform optical flow analysis of one or more images to reduce the probability of detecting “false hits” and missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to analyzing movement patterns associated with other vehicles and pedestrians, for vehicle 200 in one or more images, separate from road surface movement. Processing unit 110 may calculate the movement of candidate objects by observing different positions of the object across multiple image frames captured at different times. Processing unit 110 may use the position and time values ​​as inputs to a mathematical model to calculate the movement of candidate objects. Thus, optical flow analysis may provide another method for detecting vehicles and pedestrians in the vicinity of vehicle 200. Processing unit 110 may perform optical flow analysis in combination with steps 540-546 to provide redundancy for detecting vehicles and pedestrians and increase the reliability of system 100.

[0216] 5C is a flowchart illustrating an example process 500C for detecting road marks and / or lane geometry information in a set of images according to the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500C. In step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect lane mark segments, lane geometry information, and other related road marks, processing unit 110 may filter the set of objects to exclude those determined to be irrelevant (e.g., small holes, small rocks, etc.). In step 552, processing unit 110 may group together segments detected in step 550 that belong to the same road or lane mark. Based on the grouping, processing unit 110 may develop a model, such as a mathematical model, to represent the detected segments.

[0217] At step 554, processing unit 110 may construct a set of measurements associated with the detected segment. In some embodiments, processing unit 110 may create a projection of the detected segment from the image plane onto the real-world plane. The projection may be characterized using a third-order polynomial with coefficients corresponding to physical properties of the detected road, such as its position, slope, curvature, and curvature derivative. In generating the projection, processing unit 110 may take into account road surface variations and pitch and roll rates associated with vehicle 200. Additionally, processing unit 110 may model road height by analyzing motion cues present in the position and road surface. Furthermore, processing unit 110 may estimate pitch and roll rates associated with vehicle 200 by tracking a set of feature points in one or more images.

[0218] In step 556, processing unit 110 may perform a multi-frame analysis, for example, by tracking the detected segment across successive image frames and accumulating frame-by-frame data associated with the detected segment. When processing unit 110 performs a multi-frame analysis, the set of measurements constructed in step 554 may become more reliable and may be associated with an increasingly higher degree of confidence. Thus, by performing steps 550-556, processing unit 110 may identify road marks appearing in the set of captured images and derive lane geometry information. Based on the identification and derived information, processing unit 110 may cause one or more navigational responses in vehicle 200, as described above in connection with FIG. 5A .

[0219] In step 558, processing unit 110 may consider additional information sources to further develop a safety model of vehicle 200 in the vehicle's surroundings. Processing unit 110 may use the safety model to define situations in which system 100 may safely perform autonomous control of vehicle 200. To develop the safety model, in some embodiments, processing unit 110 may consider the positions and movements of other vehicles, detected road edges and barriers, and / or general road shape descriptions extracted from map data (such as data from map database 160). By considering additional information sources, processing unit 110 may provide redundancy in detecting road marks and lane geometry and increase the reliability of system 100.

[0220] FIG. 5D is a flowchart illustrating an example process 500D for detecting traffic lights in a set of images according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500D. In step 560, processing unit 110 may scan the set of images and identify objects that appear at locations in the images that are likely to contain traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects that excludes objects that are unlikely to correspond to traffic lights. The filtering may be based on various characteristics associated with traffic lights, such as shape, size, texture, and location (e.g., relative to vehicle 200). Such characteristics may be based on many examples of traffic lights and traffic control signals and may be stored in a database. In some embodiments, processing unit 110 may perform multi-frame analysis on the set of candidate objects that reflect possible traffic lights. For example, processing unit 110 may track the candidate objects across consecutive image frames, estimate the real-world locations of the candidate objects, and filter out moving objects (which are unlikely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate object to identify the relative position of the detected color appearing within the potential traffic light.

[0221] In step 562, processing unit 110 may analyze the geometry of the intersection. The analysis may be based on any combination of (i) the number of lanes detected on either side of vehicle 200, (ii) marks (such as arrow marks) detected on the road, and (iii) a description of the intersection extracted from map data (such as data from map database 160). Processing unit 110 may perform the analysis using information derived from execution of monocular analysis module 402. In addition, processing unit 110 may identify correspondences between traffic lights detected in step 560 and lanes appearing near vehicle 200.

[0222] As vehicle 200 approaches the intersection, in step 564, processing unit 110 may update the confidence associated with the analyzed intersection geometry and the detected traffic lights. For example, the number of traffic lights estimated to appear at the intersection compared to the number that actually appear at the intersection may affect the confidence. Thus, based on the confidence, processing unit 110 may delegate control to the driver of vehicle 200 to improve the safety situation. By performing steps 560-564, processing unit 110 may identify traffic lights appearing in the set of captured images and analyze the intersection geometry information. Based on the identification and analysis, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in connection with FIG. 5A .

[0223] FIG. 5E is a flowchart of an example process 500E for generating one or more navigation responses in vehicle 200 based on a vehicle path, according to a disclosed embodiment. In step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path may be represented using a set of points represented by coordinates (x, y), and the distance d i between any two points in the set of points may be in the range of 1 to 5 meters. In one embodiment, processing unit 110 may construct the initial vehicle path using two polynomials, such as left and right road polynomials. Processing unit 110 may calculate the geometric midpoint between the two polynomials and offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset) if there is a predetermined offset (an offset of 0 may correspond to driving in the center of the lane). The offset may be in a direction perpendicular to the segment between any two points in the vehicle path. In another embodiment, processing unit 110 may use one polynomial and the estimated lane width to offset each point in the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0224] In step 572, processing unit 110 may update the vehicle path constructed in step 570. Processing unit 110 may reconstruct the vehicle path constructed in 570 using a higher resolution such that the distance dk between two points in the set of points representing the vehicle path is shorter than the distance di described above. For example, the distance dk may be in the range of 0.1 to 0.3 meters. Processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may result in a cumulative distance vector S corresponding to the total length of the vehicle path (i.e., based on the set of points representing the vehicle path).

[0225] In step 574, processing unit 110 may identify look-ahead points (expressed in coordinates as (xl, zl)) based on the updated vehicle path performed in step 572. Processing unit 110 may extract the look-ahead points from the cumulative distance vector S, and the look-ahead points may be associated with a look-ahead distance and a look-ahead time. The look-ahead distance may have a lower bound range of 10 to 20 meters and may be calculated as the product of the speed of vehicle 200 and the look-ahead time. For example, as the speed of vehicle 200 decreases, the look-ahead distance may also decrease (e.g., until a lower bound is reached). The look-ahead time, which may range from 0.5 to 1.5 seconds, may be inversely proportional to the gain of one or more control loops associated with producing a navigation response in vehicle 200, such as a heading error tracking control loop. For example, the gain of the heading error tracking control loop may depend on the bandwidth of the yaw rate loop, the steering actuator loop, and vehicle lateral dynamics, etc. Thus, the higher the gain of the heading error tracking control loop, the shorter the look-ahead time.

[0226] In step 576, processing unit 110 may determine a heading error and a yaw rate command based on the look ahead point identified in step 574. Processing unit 110 may determine the heading error by calculating the arctangent of the look ahead point, e.g., arctan(xl / zl). Processing unit 110 may determine the yaw rate command as the product of the heading error and a high-level control gain. The high-level control gain may be equal to (2 / look ahead time) if the look ahead distance is not at a lower limit. If the look ahead distance is at a lower limit, the high-level control gain may be equal to (2*velocity of vehicle 200 / look ahead distance).

[0227] 5F is a flowchart illustrating an example process 500F for determining whether a leading vehicle is changing lanes, according to the disclosed embodiments. In step 580, processing unit 110 may determine navigation information associated with the leading vehicle (e.g., a vehicle traveling in front of vehicle 200). For example, processing unit 110 may determine the position, velocity (e.g., direction and speed), and / or acceleration of the leading vehicle using the techniques described above in connection with FIGS. 5A and 5B. Processing unit 110 may also determine one or more road polynomials, lookahead points (associated with vehicle 200), and / or snail trails (e.g., a set of points describing the path taken by the leading vehicle) using the techniques described above in connection with FIG. 5E.

[0228] In step 582, processing unit 110 may analyze the navigation information identified in step 580. In one embodiment, processing unit 110 may calculate the distance (e.g., along the trail) between the snail trail and the road polynomial. If the difference in this distance along the trail exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters for straight roads, 0.3 to 0.4 meters for gently curving roads, and 0.5 to 0.6 meters for sharply curving roads), processing unit 110 may determine that the leading vehicle is likely changing lanes. If multiple vehicles are detected traveling in front of vehicle 200, processing unit 110 may compare the snail trails associated with each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose snail trail does not match the snail trails of the other vehicles is likely changing lanes. Processing unit 110 may further compare the curvature of the snail trail (associated with the leading vehicle) with the expected curvature of the road segment along which the leading vehicle is traveling. The expected curvature may be extracted from map data (e.g., data from map database 160), road polynomials, snail trails of other vehicles, prior knowledge about the road, etc. If the difference between the curvature of the snail trail and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the leading vehicle is likely changing lanes.

[0229] In another embodiment, processing unit 110 may compare the instantaneous position of the leading vehicle with a look-ahead point (associated with vehicle 200) over a specific time period (e.g., 0.5 to 1.5 seconds). If the difference in distance and the cumulative sum of discrepancies between the instantaneous position of the leading vehicle and the look-ahead point during the specific time period exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a gently curving road, or 1.3 to 1.7 meters on a sharply curving road), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the snail trail by comparing the lateral distance traveled along the trail to the expected curvature of the snail trail. The expected radius of curvature may be determined according to the calculation: (δz² + δx²) / 2 / (δx), where σx represents the lateral travel distance and σz represents the longitudinal travel distance. If the difference between the lateral movement distance and the expected curvature exceeds a predetermined threshold (e.g., 500-700 meters), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the position of the leading vehicle. If the position of the leading vehicle obscures the road polynomial (e.g., the leading vehicle overlies the road polynomial), processing unit 110 may determine that the leading vehicle is likely changing lanes. If the position of the leading vehicle is such that another vehicle is detected ahead of the leading vehicle and the snail trails of the two vehicles are not parallel, processing unit 110 may determine that the (closer) leading vehicle is likely changing lanes.

[0230] In step 584, processing unit 110 may determine whether leading vehicle 200 is changing lanes based on the analysis performed in step 582. For example, processing unit 110 may make that determination based on a weighted average of the individual analyses performed in step 582. Under such a scheme, for example, a determination by processing unit 110 that the leading vehicle is likely to be changing lanes based on a particular type of analysis may be assigned a value of “1” (with a “0” representing a determination that the leading vehicle is unlikely to be changing lanes). Different weights may be assigned to different analyses performed in step 582, and the disclosed embodiments are not limited to any particular combination of analyses and weights.

[0231] 6 is a flowchart illustrating an example process 600 for generating one or more navigational responses based on stereo image analysis, according to disclosed embodiments. In step 610, processing unit 110 may receive first and second pluralities of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) may capture first and second pluralities of images of an area ahead of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first and second pluralities of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0232] At step 620, processing unit 110 may execute stereo image analysis module 404 to perform stereo image analysis of the first and second plurality of images to create a 3D map of the road in front of the vehicle and detect features in the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards. The stereo image analysis may be performed similarly to the steps described above in connection with FIGS. 5A-5D . For example, processing unit 110 may execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road marks, traffic lights, road hazards, etc.) in the first and second plurality of images, filter out a subset of the candidate objects based on various criteria, perform multi-frame analysis, construct measurements, and identify confidence levels for the remaining candidate objects. In performing the above steps, processing unit 110 may consider information from both the first and second plurality of images, rather than information from only one set of images. For example, processing unit 110 may analyze differences in pixel-level data (or other subsets of data from the two streams of captured images) of a candidate object that appears in both the first and second plurality of images. As another example, processing unit 110 may estimate the position and / or velocity (e.g., relative to vehicle 200) of a candidate object by observing that the object appears in one of the plurality of images but not in the other, or other differences that may exist for objects that appear in the two image streams. For example, the position, velocity, and / or acceleration relative to vehicle 200 may be determined based on the trajectory, location, movement characteristics, etc. of features associated with the object that appears in one or both of the image streams.

[0233] In step 630, processing unit 110 may execute navigation response module 408 to generate one or more navigational responses in vehicle 200 based on the analysis performed in step 620 and the techniques described above in connection with FIG. 4. The navigational responses may include, for example, turns, lane shifts, acceleration changes, speed changes, braking, etc. In some embodiments, processing unit 110 may generate one or more navigational responses using data derived from execution of speed and acceleration module 406. Furthermore, multiple navigational responses may be generated simultaneously, sequentially, or any combination thereof.

[0234] 7 is a flowchart illustrating an example process 700 for generating one or more navigational responses based on the analysis of three sets of images, according to disclosed embodiments. In step 710, processing unit 110 may receive first, second, and third pluralities of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture first, second, and third pluralities of images of an area in front of and / or to the side of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first, second, and third pluralities of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface that communicates data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0235] At step 720, processing unit 110 may analyze the first, second, and third plurality of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards. The analysis may be performed similar to the steps described above in connection with FIGS. 5A-5D and 6. For example, processing unit 110 may perform monocular image analysis on each of the first, second, and third plurality of images (e.g., via execution of monocular image analysis module 402 and based on the steps described above in connection with FIGS. 5A-5D). Alternatively, processing unit 110 may perform stereo image analysis on the first and second plurality of images, the second and third plurality of images, and / or the first and third plurality of images (e.g., via execution of stereo image analysis module 404 and based on the steps described above in connection with FIG. 6). Processed information corresponding to the analysis of the first, second, and / or third plurality of images may be combined. In some embodiments, processing unit 110 may perform a combination of monocular image analysis and stereo image analysis. For example, processing unit 110 may perform monocular image analysis on the first plurality of images (e.g., via execution of monocular image analysis module 402) and stereo image analysis on the second and third plurality of images (e.g., via execution of stereo image analysis module 404). The configuration of image capture devices 122, 124, and 126—including their respective positions and fields of view 202, 204, and 206—may affect the type of analysis performed on the first, second, and third plurality of images. The disclosed embodiments are not limited to a particular configuration of image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third plurality of images.

[0236] In some embodiments, processing unit 110 may perform tests on system 100 based on the images acquired and analyzed in steps 710 and 720. Such tests may provide an indicator of the overall performance of system 100 with a particular configuration of image capture devices 122, 124, and 126. For example, processing unit 110 may identify the rate of "false hits" (e.g., when system 100 incorrectly determines the presence of a vehicle or pedestrian) and "misses."

[0237] In step 730, processing unit 110 may cause one or more navigation responses in vehicle 200 based on information derived from two of the first, second, and third plurality of images. The selection of two of the first, second, and third plurality of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the plurality of images. Processing unit 110 may make the selection based on the quality and resolution of the images, the effective field of view reflected in the images, the number of captured frames, the degree to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the proportion in which the object appears in each such frame, etc.), etc.

[0238] In some embodiments, processing unit 110 may select information derived from two of the first, second, and third plurality of images by identifying the extent to which information derived from one image source is consistent with information derived from the other image sources. For example, processing unit 110 may combine processed information (whether monocular analysis, stereo analysis, or any combination of the two) derived from each of image capture devices 122, 124, and 126 to identify visual indicators (e.g., lane markings, detected vehicles and / or their positions and / or paths, detected traffic lights, etc.) that are consistent across images captured from each of image capture devices 122, 124, and 126. Processing unit 110 may also filter out information that is inconsistent across captured images (e.g., a vehicle changing lanes, a lane model showing a vehicle too close to vehicle 200, etc.). Thus, processing unit 110 may select information derived from two of the first, second, and third plurality of images based on the identification of consistent and inconsistent information.

[0239] The navigational responses may include, for example, turns, lane shifts, and acceleration changes. Processing unit 110 may generate one or more navigational responses based on the analysis performed in step 720 and the techniques described above in connection with FIG. 4 . Processing unit 110 may also generate one or more navigational responses using data derived from execution of velocity and acceleration module 406. In some embodiments, processing unit 110 may generate one or more navigational responses based on the relative position, relative velocity, and / or relative acceleration between vehicle 200 and an object detected in any of the first, second, and third plurality of images. The multiple navigational responses may be generated simultaneously, sequentially, or any combination thereof.

[0240] Sparse Road Models for Autonomous Vehicle Navigation In some embodiments, the disclosed systems and methods may use sparse maps for autonomous vehicle navigation. For example, a sparse map may provide sufficient information to navigate an autonomous vehicle without storing and / or updating large amounts of data. As discussed in more detail below, an autonomous vehicle may use a sparse map to navigate one or more roads based on one or more stored trajectories.

[0241] Sparse Maps for Autonomous Vehicle Navigation In some embodiments, the disclosed systems and methods may use sparse maps for autonomous vehicle navigation. For example, a sparse map may provide sufficient information for navigation without requiring excessive data storage or data transfer rates. As discussed in more detail below, a vehicle (which may be an autonomous vehicle) may use a sparse map to navigate one or more roads. For example, in some embodiments, a sparse map may include data related to roads and possible landmarks along the roads, which are sufficient for vehicle navigation while also exhibiting a small data footprint. For example, the sparse data maps described in more detail below require much less storage space and data transfer bandwidth than digital maps that include detailed map information, such as image data collected along roads. For example, rather than storing detailed representations of road segments, a sparse data map may store three-dimensional polynomial representations of preferred vehicle paths along roads. These paths may require negligible data storage space. Furthermore, in the described sparse data maps, landmarks may be identified and included in the sparse map road model to assist in navigation. These landmarks may be spaced at any suitable interval to enable vehicle navigation, although in some cases, such landmarks need not be identified and included in the model at a high density and at close intervals. Rather, in some cases, navigation can be based on landmarks spaced at least 50 meters, at least 100 meters, at least 500 meters, at least 1 kilometer, or at least 2 kilometers apart. As discussed in more detail in other sections, sparse maps may be generated based on data collected or measured by vehicles equipped with various sensors and devices, such as image capture devices, global positioning system sensors, and motion sensors, as the vehicles travel along roads. In some cases, sparse maps may be generated based on data collected during multiple drives of one or more vehicles along a particular road.

[0242] According to disclosed embodiments, an autonomous vehicle system may use a sparse map for navigation. At the core of the sparse map, one or more three-dimensional contours may represent predetermined trajectories that an autonomous vehicle may traverse as it travels along the associated road segment. The sparse map may also include data representing one or more road features. Such road features may include recognized landmarks, road signature profiles, and any other road-related features useful in navigating the vehicle. The sparse map may enable autonomous navigation of the vehicle based on the relatively small amount of data contained in the sparse map. For example, rather than including detailed representations of roads, such as data detailing road edges, road curvature, images associated with road segments, or other physical features associated with road segments, disclosed embodiments of the sparse map may require relatively small storage space (and bandwidth, if portions of the sparse map are transferred to the vehicle) while still adequately providing autonomous vehicle navigation. The small data footprint of the disclosed sparse maps, discussed in more detail below, may be achieved in some embodiments by storing representations of road-related elements that require a small amount of data, yet still enable autonomous navigation. For example, rather than storing detailed representations of various aspects of a road, the disclosed sparse maps may store polynomial representations of one or more trajectories that a vehicle may follow along a road. Thus, rather than storing (or needing to transfer) details about the physical properties of the road to enable navigation along the road, the disclosed sparse maps may be used to navigate a vehicle along a particular road segment, in some cases without having to interpret the physical aspects of the road, but rather by aligning a driving path to a trajectory (e.g., a polynomial spline) along the particular road segment. In this way, a vehicle may be navigated primarily based on the stored trajectory (e.g., a polynomial spline), which may require much less storage space than methods that include storing road images, road parameters, road layouts, etc.

[0243] In addition to stored polynomial representations of trajectories along road segments, the disclosed sparse maps can also include small data objects that can represent road features. In some embodiments, the small data objects can include digital signatures derived from digital images (or digital signals) acquired by sensors (e.g., cameras or other sensors, such as suspension sensors) mounted on vehicles traveling along road segments. The digital signatures can have a reduced size compared to the signals acquired by the sensors. In some embodiments, the digital signatures can be created to be compatible with a classification function configured to detect and identify road features from signals acquired by sensors, for example, during ongoing driving. In some embodiments, the digital signatures can be created based on images of road features captured by cameras mounted on vehicles traveling along the same road segment at successive times (or digital signals generated by sensors, if the stored signatures are not image-based and / or include other data) to have as small a footprint as possible while preserving the ability to correlate or match road features with stored signatures. In some embodiments, the size of the data object may further be associated with the uniqueness of the road feature, for example, a road feature detectable by an on-board camera, where the on-board camera system is coupled to a classifier that is able to distinguish image data corresponding to the road feature as being associated with a particular type of road feature, e.g., a road sign, and where such road sign is locally unique in the area (e.g., there are no identical or same type of road signs nearby), it may be sufficient to store data indicating the type of road feature and its location.

[0244] As discussed in more detail below, road features (e.g., landmarks along a road segment) may be stored as small data objects that can represent the road features in a relatively small number of bytes while providing sufficient information to recognize and use such features for navigation. In just one example, road signs may be identified as recognized landmarks upon which vehicle navigation may be based. Representations of road signs may be stored in a sparse map, for example, to include a few bytes of data indicating the type of landmark (e.g., a stop sign) and a few bytes of data indicating the location of the landmark. Navigation based on such data-light representations of landmarks (e.g., using representations sufficient for locating, recognizing, and navigating based on the landmark) may provide a desired level of navigation functionality associated with sparse maps without significantly increasing the data overhead associated with sparse maps. This lean representation of landmarks (and other road features) may take advantage of sensors and processors included onboard the vehicle that are configured to detect, identify, and / or classify specific road features. For example, if in a given area a sign, or even a particular type of sign, is locally unique (e.g., there are no other signs or no other signs of the same type), the sparse map may use data indicating the type of landmark (the sign or the particular type of sign), and during navigation (e.g., autonomous navigation), when a camera on board the autonomous vehicle captures an image of an area containing a sign (or the particular type of sign), the processor may process the image, detect the sign (if in fact present in the image), classify it as a sign (or the particular type of sign), and correlate the location of the sign stored in the sparse map to its location.

[0245] In some embodiments, an autonomous vehicle may include a vehicle body and a processor configured to receive data included in a sparse map and generate navigation instructions to navigate the vehicle along a road segment based on the data in the sparse map.

[0246] 8 shows a sparse map 800 that vehicle 200 (which may be an autonomous vehicle) may access to provide autonomous vehicle navigation. Sparse map 800 may be stored in a memory, such as memory 140 or 150. Such a memory device may include any type of non-transitory storage device or computer-readable medium. For example, in some embodiments, memory 140 or 150 may include a hard drive, a compact disc, flash memory, a magnetic-based memory device, an optical-based memory device, etc. In some embodiments, sparse map 800 may be stored in a database (e.g., map database 160), which may be stored in memory 140 or 150 or other types of storage devices.

[0247] In some embodiments, sparse map 800 may be stored on a storage device or non-transitory computer-readable medium provided onboard vehicle 200 (e.g., storage device included in a navigation system provided onboard vehicle 200). A processor provided on vehicle 200 (e.g., processing unit 110) may access sparse map 800 stored on a storage device or computer-readable medium provided onboard vehicle 200 to generate navigation instructions that guide autonomous vehicle 200 as it travels road segments.

[0248] However, sparse map 800 need not be stored locally on the vehicle. In some embodiments, sparse map 800 may be stored on a storage device or computer-readable medium provided on a remote server that communicates with vehicle 200 or a device associated with vehicle 200. A processor (e.g., processing unit 110) provided on vehicle 200 may receive the data included in sparse map 800 from the remote server and execute the data to guide the autonomous driving of vehicle 200. In such embodiments, sparse map 800 may be accessible to multiple vehicles (e.g., tens, hundreds, thousands, or millions of vehicles, etc.) traveling various road segments. It should also be noted that sparse map 800 may include multiple submaps. For example, in some embodiments, sparse map 800 may include hundreds, thousands, millions, or more submaps that can be used to navigate the vehicle. Such submaps may be referred to as local maps, and a vehicle traveling along a road may have access to any number of local maps associated with the location at which the vehicle is traveling. The local map sections of the sparse map 800 may be stored with a global navigation satellite system (GNSS) key as an index into the database of the sparse map 800. Thus, calculations of steering angles to navigate a host vehicle in the system may be performed without relying on the GNSS positions of the host vehicle, road features, or landmarks, while such GNSS information may be used to look up the associated local map.

[0249] The collection of data and the generation of sparse map 800 are covered in detail in other sections. However, in general, sparse map 800 may be generated based on data collected from one or more vehicles as they travel along a road. For example, sensors (e.g., cameras, speedometers, GPS, accelerometers, etc.) mounted on one or more vehicles may be used to record the trajectories of one or more vehicles traveling along a road, and a polynomial representation of the preferred trajectories of vehicles continuing to travel along the road may be determined based on the collected trajectories traveled by the one or more vehicles. Similarly, data collected by one or more vehicles may help identify possible landmarks along a particular road. Data collected from traveling vehicles may also be used to identify road profile information, such as a road width profile, a road roughness profile, a traffic line spacing profile, etc. Using the collected information, sparse map 800 may be generated and distributed (e.g., to local storage or via wireless data transmission) for use in navigating one or more autonomous vehicles. However, map generation may not end with the initial generation of the map. As discussed in more detail in other sections, the sparse map 800 may be updated continuously or periodically based on data collected from the vehicle as it continues to travel the roads included in the sparse map 800.

[0250] The data recorded in the sparse map 800 may include location information based on Global Positioning System (GPS) data. For example, location information may be included in the sparse map 800 for various map elements, including, for example, landmark locations, road profile locations, etc. The locations of map elements included in the sparse map 800 may be obtained using GPS data collected from vehicles traveling on roads. For example, a vehicle passing an identified landmark may locate the identified landmark using GPS location information associated with the vehicle and a determination of the identified landmark's location relative to the vehicle (e.g., based on image analysis of data collected from one or more cameras on the vehicle). Such a location determination of the identified landmark (or any other feature included in the sparse map 800) may be repeated as additional vehicles pass the identified landmark location. Some or all of the additional location determinations may be used to refine the location information stored in the sparse map 800 for the identified landmark. For example, in some embodiments, multiple location measurements for a particular feature stored in the sparse map 800 may be averaged together. However, any other mathematical operation may be used to refine the stored location of a map element based on multiple locations specified for the map element.

[0251] The sparse maps of the disclosed embodiments may enable autonomous navigation of a vehicle using a relatively small amount of stored data. In some embodiments, the sparse map 800 may have a data density (e.g., including data representing target trajectories, landmarks, and any other stored road features) of less than 2 MB per kilometer of road, less than 1 MB per kilometer of road, less than 500 kB per kilometer of road, or less than 100 kB per kilometer of road. In some embodiments, the data density of the sparse map 800 may be less than 10 kB per kilometer of road, or even less than 2 kB per kilometer of road (e.g., 1.6 kB per kilometer), or 10 kB per kilometer or less. In some embodiments, substantially all, if not all, roads in the United States may be navigated autonomously using a sparse map having a total of 4 GB or less of data. These data density values ​​may represent averages across the entire sparse map 800, local maps within the sparse map 800, and / or specific road segments within the sparse map 800.

[0252] As noted, the sparse map 800 may include representations of multiple target trajectories 810 to guide autonomous driving or navigation along road segments. Such target trajectories may be stored as cubic splines. The target trajectories stored in the sparse map 800 may be determined based on two or more reconstructed trajectories of the vehicle's previous travels along a particular road segment. A road segment may have a single target trajectory or multiple target trajectories associated with it. For example, a two-lane road may store a first target trajectory representing the intended travel path along the road in a first direction, and a second target trajectory representing the intended travel path along the road in another direction (e.g., opposite the first direction). Additional target trajectories for particular road segments may be stored. For example, a multi-lane road may store one or more target trajectories representing the intended travel path of the vehicle in one or more lanes associated with the multi-lane road. In some embodiments, each lane of a multi-lane road may have its own target trajectory associated with it. In other embodiments, fewer target trajectories may be stored than there are lanes on the multi-lane road. In such cases, a vehicle navigating a multi-lane road may use any of the stored target trajectories to guide navigation by taking into account the lane offset from the lane in which the target trajectory is stored (e.g., if the vehicle is traveling in the leftmost lane of a three-lane highway and only the target trajectory for the center lane of the highway is stored, the vehicle may navigate using the target trajectory for the center lane by accounting for the lane offset between the center lane and the leftmost lane when generating navigation instructions).

[0253] In some embodiments, the target trajectory may represent an ideal path that the vehicle should follow as it travels. The target trajectory may, for example, be positioned approximately in the center of the driving lane. In other cases, the target trajectory may be positioned elsewhere relative to the road segment. For example, the target trajectory may generally coincide with the center of the road, the edge of the road, the edge of the lane, or the like. In such cases, navigation based on the target trajectory may include an offset amount determined to maintain relative to the position of the target trajectory. Furthermore, in some embodiments, the offset amount determined to maintain relative to the position of the target trajectory may vary depending on the type of vehicle (e.g., a passenger car including two axles may have a different offset along at least a portion of the target trajectory than a truck including three or more axles).

[0254] The sparse map 800 may also include data related to a number of predetermined landmarks 820 associated with particular road segments, local maps, etc. As discussed in detail in other sections, these landmarks may be used for navigating an autonomous vehicle. For example, in some embodiments, the landmarks may be used to identify the vehicle's current position relative to a stored target trajectory. Using this position information, the autonomous vehicle may be able to adjust its heading to match the direction of the target trajectory at the identified location.

[0255] Multiple landmarks 820 may be identified and stored in the sparse map 800 at any suitable interval. In some embodiments, landmarks may be stored relatively densely (e.g., every few meters or more). However, some embodiments may utilize much wider landmark spacing values. For example, in the sparse map 800, identified (or recognized) landmarks may be spaced 10 meters, 20 meters, 50 meters, 100 meters, 1 kilometer, or 2 kilometers apart. In some cases, identified landmarks may even be spaced apart by distances greater than 2 kilometers. Between landmarks, and thus between vehicle positions determined relative to the target trajectory, the vehicle may use sensors to determine its motion, estimate its position relative to the target trajectory, and navigate based on autonomous navigation. During autonomous navigation, errors may accumulate, so that location relative to the target trajectory may become less and less accurate over time. The vehicle may use landmarks (and their known positions) in the sparse map 800 to eliminate autonomous navigation-induced errors in location determination. In this way, identified landmarks included in sparse map 800 can serve as navigation anchors, from which the vehicle's precise position relative to the target trajectory can be determined. Because a certain amount of error in localization may be tolerable, it is not necessary to always provide identified landmarks to the autonomous vehicle. Rather, as described above, suitable navigation may be possible based on landmark spacing of 10 meters, 20 meters, 50 meters, 100 meters, 500 meters, 1 kilometer, 2 kilometers, or even more. In some embodiments, a density of one identified landmark per kilometer of road may be sufficient to maintain longitudinal localization accuracy within 1 meter. Therefore, it is not necessary to store every potential landmark that appears along a road segment in sparse map 800.

[0256] In addition to the target trajectory and identified landmarks, the sparse map 800 may include information related to various other road features. For example, FIG. 9A shows a representation of curves along a particular road segment that may be stored in the sparse map 800. In some embodiments, a single lane of a road may be modeled by a three-dimensional polynomial description of the left and right sides of the road. Such polynomials representing the left and right sides of a single lane are shown in FIG. 9A. Regardless of how many lanes a road may have, the road may be represented using polynomials in a manner similar to that shown in FIG. 9A. For example, the left and right sides of a multi-lane road may be represented by polynomials similar to those shown in FIG. 9A, and intermediate lane markings included on a multi-lane road (e.g., dashed line marks representing lane boundaries, solid yellow lines representing boundaries between lanes traveling in different directions, etc.) may also be represented using polynomials such as those shown in FIG. 9A.

[0257] As shown in FIG. 9A , lane 900 may be represented using a polynomial (e.g., a first-order, second-order, third-order, or any suitable order polynomial). For illustrative purposes, lane 900 is shown as a two-dimensional lane, and the polynomials are shown as two-dimensional polynomials. Lane 900 includes a left side 910 and a right side 920. In some embodiments, more than one polynomial may be used to represent the location of each side of a road or lane boundary. For example, left side 910 and right side 920 may each be represented by multiple polynomials of any suitable length. In some cases, the polynomials may have a length of approximately 100 meters, although other lengths greater than or less than 100 meters may also be used. Furthermore, polynomials may overlap with each other to facilitate seamless transitions in polynomial-based navigation that a host vehicle encounters as it travels along a road. For example, each of the left side 910 and the right side 920 may be divided into segments approximately 100 meters long (an example first predetermined range) and represented by multiple third-order polynomials that overlap each other by approximately 50 meters. The polynomials representing the left side 910 and the right side 920 may or may not have the same degree. For example, in some embodiments, some polynomials are second-order polynomials, some third-order polynomials, and some fourth-order polynomials.

[0258] In the example shown in FIG. 9A , the left side 910 of lane 900 is represented by two groups of third-order polynomials. The first group includes polynomial segments 911, 912, and 913. The second group includes polynomial segments 914, 915, and 916. The two groups are generally parallel to each other and trace the location of the corresponding sides of the road. Polynomial segments 911-916 are approximately 100 meters long and overlap adjacent segments in the series by approximately 50 meters. However, as noted above, polynomials of different lengths and overlap amounts can also be used. For example, the polynomials can have lengths of 500 meters, 1 km, or more than 1 km, and the overlap amount can vary from 0 to 50 meters, 50 to 100 meters, or more than 100 meters. Furthermore, while FIG. 9A is shown as representing polynomials that extend in 2D space (e.g., the surface of a piece of paper), it should be understood that these polynomials may represent curves that extend in three dimensions (e.g., include a height component) to represent elevation changes in road segments in addition to XY curvature.

[0259] Returning to the target trajectories of the sparse map 800, FIG. 9B shows a cubic polynomial that represents a target trajectory for a vehicle traveling along a particular road segment. The target trajectory represents not only the XY path that the host vehicle should travel along a particular road segment, but also the elevation changes that the host vehicle will experience as it travels the road segment. Thus, each target trajectory in the sparse map 800 may be represented by one or more cubic polynomials, such as cubic polynomial 950 shown in FIG. 9B. The sparse map 800 may include multiple trajectories (e.g., millions or billions or more to represent vehicle trajectories along various road segments along roads around the world). In some embodiments, each target trajectory may correspond to a spline connecting the cubic polynomial segments.

[0260] With respect to the data footprint of the polynomial curves stored in the sparse map 800, in some embodiments, each third-order polynomial may be represented by four parameters, with each parameter requiring four bytes of data. A suitable representation may be obtained using a third-order polynomial requiring approximately 192 bytes of data per 100 meters. This translates to a data usage / transfer requirement of approximately 200 kB / hour for a host vehicle traveling at approximately 100 km / hour.

[0261] The sparse map 800 may describe the lane network using a combination of geometry descriptors and metadata. The geometry may be described by polynomials or splines as described above. The metadata may describe the number of lanes, special features (such as carpool lanes), and other sparse labels where possible. The total footprint of such indicators may be negligible.

[0262] As described above, the sparse map 800 may include a plurality of predetermined landmarks associated with a road segment. Rather than storing actual images of the landmarks and relying, for example, on image recognition analysis based on captured and stored images, each landmark in the sparse map 800 may be represented and recognized using less data than would be required for a stored actual image. The data representing the landmarks may include sufficient information to describe or identify the landmarks along the road. By storing data describing the features of the landmarks rather than actual images of the landmarks, the size of the sparse map 800 may be reduced.

[0263] FIG. 10 shows examples of types of landmarks that may be represented in the sparse map 800. Landmarks may include any visible and identifiable object along a road segment. Landmarks may be selected to be fixed and not change location and / or content frequently. Landmarks included in the sparse map 800 may be useful in locating the vehicle 200 relative to a target trajectory as the vehicle travels a particular road segment. Examples of landmarks may include traffic signs, directional signs, general-purpose signs (e.g., rectangular signs), roadside fixtures (e.g., lampposts, reflectors, etc.), and any other suitable category. In some embodiments, lane markings on roads may also be included as landmarks in the sparse map 800.

[0264] 10 includes traffic signs, directional signs, roadside fixtures, and general-purpose signs. Traffic signs may include, for example, speed limit signs (e.g., speed limit sign 1000), yield signs (e.g., yield sign 1005), route number signs (e.g., route number sign 1010), traffic light signs (e.g., traffic light sign 1015), and stop signs (e.g., stop sign 1020). Directional signs may include signs including one or more arrows indicating one or more directions to different locations. For example, directional signs may include a highway sign 1025 with an arrow directing vehicles to a different road or location, an exit sign 1030 with an arrow directing vehicles to another road, etc.

[0265] Generic signs can be non-traffic related. For example, generic signs can include billboards used for advertising or welcome signs adjacent to the boundary between two countries, states, counties, cities, or towns. Figure 10 shows generic sign 1040 ("Joe's Restaurant"). Generic sign 1040 can have a rectangular shape as shown in Figure 10, but generic sign 1040 can also have other shapes, such as a square, circle, triangle, etc.

[0266] Landmarks can also include roadside fixtures. Roadside fixtures may be objects that are not signs and may not be traffic or directional related. For example, roadside fixtures may include lampposts (e.g., lamppost 1035), utility poles, traffic light poles, etc.

[0267] Landmarks may also include beacons specifically designed for use in autonomous vehicle navigation systems. For example, such beacons may include stand-alone structures placed at predetermined intervals to assist a host vehicle in navigating. Such beacons may also include visual / graphical information (e.g., icons, emblems, bar codes, etc.) that is added to existing road signs that may be identified or recognized by a vehicle traveling along a road segment. Such beacons may also include electronic components. In such embodiments, electronic beacons (e.g., RFID tags, etc.) may be used to transmit non-visual information to the host vehicle. Such information may include, for example, landmark-specific and / or landmark location information that the host vehicle may use to determine its location along the target trajectory.

[0268] In some embodiments, landmarks included in the sparse map 800 may be represented by data objects of a predetermined size. Data representing landmarks may include any parameters suitable for identifying a particular landmark. For example, in some embodiments, landmarks stored in the sparse map 800 may include parameters such as the landmark's physical size (e.g., to support estimation of the distance to the landmark based on a known size / scale), distance to the previous landmark, lateral offset, height, type code (e.g., landmark type—any type of directional sign, traffic sign, etc.), GPS coordinates (e.g., to support global positioning), and any other suitable parameters. Each parameter may have an associated data size. For example, landmark size may be stored using 8 bytes of data. The distance to the previous landmark, lateral offset, and height may be specified using 12 bytes of data. A type code associated with a landmark such as a directional sign or traffic sign may require approximately 2 bytes of data. In the case of a generic landmark, an image signature that allows for identification of the generic landmark may be stored using 50 bytes of data storage. A landmark GPS position may have 16 bytes of data storage associated with it. These data sizes for each parameter are merely examples and other data sizes may be used.

[0269] Representing landmarks in the sparse map 800 in this manner provides a lean solution for efficiently representing landmarks in a database. In some embodiments, signs may be referred to as semantic or non-semantic signs. Semantic signs may include any class of signs with a standardized meaning (e.g., speed limit signs, warning signs, directional signs, etc.). Non-semantic signs may include any signs with no associated standardized meaning (e.g., general advertising signs, signs identifying businesses, etc.). For example, each semantic sign may be represented using 38 bytes of data (e.g., 8 bytes for size, 12 bytes for distance to the previous landmark, lateral offset, and height, 2 bytes for a type code, and 16 bytes for GPS coordinates). The sparse map 800 may represent landmark types using a tag system. In some cases, each traffic or directional sign may be associated with its own tag, which may be stored in the database as part of the landmark-specific information. For example, the database may include on the order of 1,000 different tags representing various traffic signs, and on the order of about 10,000 different tags representing directional signs. Of course, any suitable number of tags may be used, and additional tags may be created as needed. A general-purpose sign may, in some embodiments, be represented using less than about 100 bytes (e.g., about 86 bytes, including 8 bytes for size, 12 bytes for distance to previous landmark, lateral offset, and height, 50 bytes for image signature, and 16 bytes for GPS coordinates).

[0270] Thus, for semantic road signs that do not require image signatures, even with a relatively high landmark density of approximately one every 50 meters, the data density impact on sparse map 800 can be on the order of approximately 760 bytes per kilometer (e.g., 20 landmarks per km × 38 bytes per landmark = 760 bytes). Even for generic signs that include an image signature component, the data density impact is approximately 1.72 kB per km (e.g., 20 landmarks per km × 86 bytes per landmark = 1,720 bytes). For semantic road signs, this equates to a data usage of approximately 76 kB per hour for a vehicle traveling at 100 km / h. For generic signs, this equates to a data usage of approximately 170 kB per hour for a vehicle traveling at 100 km / h.

[0271] In some embodiments, a generally rectangular object, such as a rectangular sign, may be represented in the sparse map 800 with 100 bytes of data or less. A representation of a generally rectangular object (e.g., generic sign 1040) in the sparse map 800 may include a condensed image signature (e.g., condensed image signature 1045) associated with the generally rectangular object. This condensed image signature may be used, for example, to aid in the identification of the generic sign, e.g., a recognized landmark. Such a condensed image signature (e.g., image information derived from actual image data representing the object) may avoid the need to store actual images of the object or the need for comparative image analysis to be performed on the actual image to recognize the landmark.

[0272] 10 , sparse map 800 may include or store a simplified image signature 1045 associated with generic sign 1040 rather than an actual image of generic sign 1040. For example, after an image capture device (e.g., image capture device 122, 124, or 126) captures an image of generic sign 1040, a processor (e.g., image processor 190 or a processor capable of processing images located on-board or remotely relative to the host vehicle) may perform image analysis to extract / create simplified image signature 1045 that includes a unique signature or pattern associated with generic sign 1040. In one embodiment, simplified image signature 1045 may include a shape, color pattern, brightness pattern, or any other feature that can be extracted from an image of generic sign 1040 that describes generic sign 1040. For example, in FIG. 10 , the circles, triangles, and stars shown in simplified image signature 1045 may represent areas of different colors. Patterns represented by circles, triangles, and stars may be stored in the sparse map 800, for example, within the 50 bytes designated to contain an image signature. In particular, the circles, triangles, and stars are not meant to necessarily indicate that such shapes are stored as part of the image signature. Rather, these shapes are meant to conceptually represent recognizable areas having discernible color differences, textured areas, graphical shapes, or other differences in characteristics that may be associated with generic signs. Such simplified image signatures may be used to identify landmarks in the form of generic signs. For example, the simplified image signatures may be used to perform a same-not-same analysis based on a comparison of the stored simplified image signatures with image data captured, for example, using a camera mounted on an autonomous vehicle.

[0273] Returning to the target trajectory, the host vehicle may use a particular road segment to navigate, and FIG. 11A shows a polynomial representation trajectory captured during the process of building or maintaining sparse map 800. The polynomial representation of the target trajectory included in sparse map 800 may be determined based on two or more reconstructed trajectories of the vehicle's previous travels along the same road segment. In some embodiments, the polynomial representation of the target trajectory included in sparse map 800 may be an aggregation of two or more reconstructed trajectories of the vehicle's previous travels along the same road segment. In some embodiments, the polynomial representation of the target trajectory included in sparse map 800 may be an average of two or more reconstructed trajectories of the vehicle's previous travels along the same road segment. Other mathematical operations may also be used in constructing the target trajectory along a road path based on reconstructed trajectories collected from vehicles traveling along road segments.

[0274] As shown in FIG. 11A , a road segment 1100 may be traveled by several vehicles 200 at different times. Each vehicle 200 may collect data related to the path taken along the road segment. The path traveled by a particular vehicle may be identified based on camera data, accelerometer information, speed sensor information, and / or GPS information, among other potential sources. Such data may be used to reconstruct the trajectories of the vehicles traveling along the road segment, and based on these reconstructed trajectories, a target trajectory (or multiple target trajectories) for the particular road segment may be determined. Such target trajectories may represent a preferred path for a host vehicle (e.g., guided by an autonomous navigation system) as it travels along the road segment.

[0275] 11A , a first reconstructed trajectory 1101 may be determined based on data received from a first vehicle traveling along road segment 1100 during a first time period (e.g., day 1), a second reconstructed trajectory 1102 may be obtained from a second vehicle traveling along road segment 1100 during a second time period (e.g., day 2), and a third reconstructed trajectory 1103 may be obtained from a third vehicle traveling along road segment 1100 during a third time period (e.g., day 3). Each of trajectories 1101, 1102, and 1103 may be represented by a polynomial, such as a cubic polynomial. Note that in some embodiments, any of the reconstructed trajectories may be generated onboard a vehicle traveling along road segment 1100.

[0276] Alternatively or additionally, such a reconstructed trajectory may be determined on the server side based on information received from the road segment 1100. For example, in some embodiments, the vehicle 200 may transmit data related to its travel along the road segment 1100 (e.g., steering angle, heading, position, speed, detected road segments, and / or detected landmarks, among others) to one or more servers. The server may reconstruct a trajectory of the vehicle 200 based on the received data. The server may reconstruct a trajectory to guide the navigation of an autonomous vehicle traveling along the same road segment 1100 at a later time based on the first trajectory 1101, the second trajectory 1102, and the third trajectory 1103. While a target trajectory may be associated with a single previous travel of a road segment, in some embodiments, each target trajectory included in the sparse map 800 may be determined based on two or more reconstructed trajectories of vehicles traveling the same road segment. In FIG. 11A , the target trajectory may be represented by 1110. In some embodiments, the target trajectory 1110 may be generated based on an average of the first trajectory 1101, the second trajectory 1102, and the third trajectory 1103. In some embodiments, the target trajectory 1110 included in the sparse map 800 may be an aggregation (e.g., a weighted combination) of two or more reconstructed trajectories.

[0277] 11B and 11C further illustrate the concept of a target trajectory associated with road segments present within a geographic area 1111. As shown in FIG. 11B, a first road segment 1120 within the geographic area 1111 may include multiple roads, including two lanes 1122 designated for vehicles to travel in a first direction and two additional lanes 1124 designated for vehicles to travel in a second direction opposite the first direction. The lanes 1122 and 1124 may be separated by a double yellow line 1123. The geographic area 1111 may also include a branch road segment 1130 that intersects with the road segment 1120. The road segment 1130 may include a two-lane road, with each lane designated for a different direction of travel. The geographic area 1111 may also include other road features, such as a stop line 1132, a stop sign 1134, a speed limit sign 1136, and a hazard sign 1138.

[0278] 11C , sparse map 800 may include a local map 1140 including a road model that assists in the autonomous navigation of a vehicle within geographic area 1111. For example, local map 1140 may include target trajectories for one or more lanes associated with road segments 1120 and / or 1130 within geographic area 1111. For example, local map 1140 may include target trajectories 1141 and / or 1142 that an autonomous vehicle may access or rely on when traveling in lane 1122. Similarly, local map 1140 may include target trajectories 1143 and / or 1144 that an autonomous vehicle may access or rely on when traveling in lane 1124. Additionally, local map 1140 may include target trajectories 1145 and / or 1146 that an autonomous vehicle may access or rely on when traveling in road segment 1130. Target trajectory 1147 represents a preferred path for the autonomous vehicle to follow when transitioning from lane 1120 (particularly, target trajectory 1141 associated with the rightmost lane of lane 1120) to road segment 1130 (particularly, target trajectory 1145 associated with a first side of road segment 1130). Similarly, target trajectory 1148 represents a preferred path for the autonomous vehicle to follow when transitioning from road segment 1130 (particularly, target trajectory 1146) to a portion of road segment 1124 (particularly, target trajectory 1143 associated with the left lane of lane 1124, as shown).

[0279] The sparse map 800 may also include representations of other road-related features associated with the geographic area 1111. For example, the sparse map 800 may also include a representation of one or more landmarks identified in the geographic area 1111. Such landmarks may include a first landmark 1150 associated with a stop line 1132, a second landmark 1152 associated with a stop sign 1134, a third landmark 1154 associated with a speed limit sign, and a fourth landmark 1156 associated with a hazard sign 1138. Such landmarks may be used, for example, to assist the autonomous vehicle in determining its current position relative to any of the depicted target trajectories, so that the vehicle may adjust its heading to match the direction of the target trajectory at the determined location.

[0280] In some embodiments, the sparse map 800 may also include a road signature profile. Such a road signature profile may be associated with any identifiable / measurable variation in at least one parameter associated with a road. For example, in some cases, such a profile may be associated with variations in surface roughness of a particular road segment, variations in road width across a particular road segment, variations in the distance between painted dashed lines along a particular road segment, variations in road curvature along a particular road segment, etc. FIG. 11D illustrates an example road signature profile 1160. While the profile 1160 may represent any of the parameters described above or other parameters, in one example, the profile 1160 may represent measurements of road surface roughness, such as obtained by monitoring one or more sensors that provide an output indicative of suspension displacement as the vehicle travels along a particular road segment. Alternatively, the profile 1160 may represent variations in road width, as determined based on image data acquired via a camera mounted on the vehicle while traveling along a particular road segment. Such a profile may be useful, for example, in identifying a particular position of an autonomous vehicle relative to a particular target trajectory. That is, as the autonomous vehicle travels along a road segment, it may measure a profile associated with one or more parameters associated with the road segment. If the measured profile can be correlated / matched to a predetermined profile that plots parameter variation with position along the road segment, the measured profile and the predetermined profile may be used (e.g., by overlaying corresponding sections of the measured profile and the predetermined profile) to determine the current position along the road segment, and thus the current position relative to the target trajectory of the road segment.

[0281] In some embodiments, sparse map 800 may include different trajectories based on different characteristics associated with the user of the autonomous vehicle, environmental conditions, and / or other parameters related to driving. For example, in some embodiments, different trajectories may be generated based on different user preferences and / or profiles. Sparse map 800 including such different trajectories may be provided to different autonomous vehicles of different users. For example, some users may prefer to avoid toll roads, while others may prefer to take the shortest or fastest route, regardless of whether the route includes toll roads. The disclosed system may generate different sparse maps with different trajectories based on such different user preferences or profiles. As another example, some users may prefer to drive in the fast lane, while others may prefer to maintain a center lane position at all times.

[0282] Different trajectories may be generated and included in the sparse map 800 based on different environmental conditions, such as day and night, snow, rain, fog, etc. An autonomous vehicle operating under such different environmental conditions may be provided with the sparse map 800, which may be generated based on the different environmental conditions. In some embodiments, a camera provided on the autonomous vehicle may detect the environmental conditions and provide such information to a server that generates and provides the sparse map. For example, the server may generate the sparse map 800 or update an already generated sparse map 800 to include trajectories that may be more suitable or safer for autonomous driving under the detected environmental conditions. Updating the sparse map 800 based on environmental information may be performed dynamically as the autonomous vehicle travels along the road.

[0283] Other different parameters related to driving may also be used as the basis for generating and providing different sparse maps to different autonomous vehicles. For example, if an autonomous vehicle is traveling at a high speed, turns may be tighter. Trajectories associated with particular lanes rather than roads may be included in sparse map 800 so that the autonomous vehicle may stay within a particular lane while following a particular trajectory. If images captured by a camera mounted on the autonomous vehicle indicate that the vehicle has drifted outside of its lane (e.g., crossed a lane marking), an action may be triggered within the vehicle to return the vehicle to the designated lane by following the particular trajectory.

[0284] Building a road model for autonomous vehicle navigation In some embodiments, the disclosed systems and methods may construct a road model for autonomous vehicle navigation. For example, the road model may include crowd-sourced data. The disclosed systems and methods may refine the crowd-sourced data based on observed local conditions. Additionally, the disclosed systems and methods may determine an improved trajectory for the autonomous vehicle based on sensor information. Furthermore, the disclosed systems and methods may identify landmarks for use in the road model and further refine the positions of the landmarks in the road model. These systems and methods are disclosed in further detail in the following sections.

[0285] Crowdsourced Data for Autonomous Vehicle Navigation In some embodiments, the disclosed systems and methods may build a road model for autonomous vehicle navigation. For example, the disclosed systems and methods may use crowd-sourced data to generate an autonomous vehicle road model that can be used to navigate one or more autonomous vehicles along a road system. Crowdsourcing means that data is received from various vehicles (e.g., autonomous vehicles) traveling a road segment at different times, and such data is used to generate and / or update the road model. The model may also be transmitted to those vehicles or other vehicles that subsequently travel along the road segment to assist autonomous vehicle navigation. The road model may include multiple target trajectories that represent preferred trajectories that the autonomous vehicle should follow as it travels the road segment. The target trajectories may be the same as reconstructed actual trajectories collected from vehicles traveling the road segment, which may be transmitted from the vehicles to a server. In some embodiments, the target trajectories may differ from the actual trajectories previously taken by one or more vehicles as they travel the road segment. The target trajectories may be generated based on the actual trajectories (e.g., through averaging or any other suitable calculation).

[0286] The vehicle trajectory data that the vehicle may upload to the server may match the vehicle's actual reconstructed trajectory, or may correspond to a recommended trajectory that may be based on or related to the vehicle's actual reconstructed trajectory, but may differ from the actual reconstructed trajectory. For example, the vehicle may modify the actual reconstructed trajectory and submit (e.g., recommend) the modified actual trajectory to the server. The road model may use the recommended modified trajectory as a target trajectory for autonomous navigation of other vehicles.

[0287] In addition to trajectory information, other information potentially used in constructing the sparse data map 800 may include information related to possible landmark candidates. For example, through crowdsourcing of information, the disclosed systems and methods may identify possible landmarks in the environment and refine landmark locations. The landmarks may be used by the autonomous vehicle's navigation system to identify and / or adjust the vehicle's position along the desired trajectory.

[0288] The reconstructed trajectory that a vehicle may generate as it travels along a road may be obtained by any suitable method. In some embodiments, the reconstructed trajectory may be developed by stitching together moving segments of the vehicle using, for example, ego-motion estimates (e.g., three-dimensional translation and three-dimensional rotation of the camera, and therefore of the vehicle body). Rotation and translation estimates may be determined based on analysis of images captured by one or more image capture devices, along with information from other sensors or devices, such as inertial sensors and speed sensors. For example, the inertial sensors may include accelerometers or other suitable sensors configured to measure changes in translation and / or rotation of the vehicle body. The vehicle may include a speed sensor that measures the speed of the vehicle.

[0289] In some embodiments, the ego-motion of the camera (and therefore the vehicle body) may be estimated based on optical flow analysis of captured images. Optical flow analysis of a sequence of images identifies pixel movements from the sequence of images and, based on the identified movements, identifies the vehicle's behavior. The ego-motion may be integrated over time and along road segments to reconstruct a trajectory associated with the road segments traveled by the vehicle.

[0290] Data (e.g., reconstructed trajectories) collected by multiple vehicles on multiple drives along a road segment at different times may be used to construct a road model (e.g., including a target trajectory, etc.) included in sparse map 800. Data collected by multiple vehicles on multiple drives along a road segment at different times may also be averaged to improve the accuracy of the model. In some embodiments, data regarding road geometry and / or landmarks may be received from multiple vehicles traveling through a common road segment at different times. Such data received from different vehicles may be combined to generate and / or update the road model.

[0291] The disclosed systems and methods may enable autonomous vehicle navigation (e.g., steering control) using low-footprint models that may be collected by the autonomous vehicle itself without the assistance of expensive survey equipment. To support autonomous navigation (e.g., steering applications), the road model may include road geometry, road lane structure, and landmarks that may be used to identify the location or position of a vehicle along a trajectory included in the model. The generation of the road model may be performed by a remote server that communicates with and receives data from vehicles traveling on the road. The data may include sensed data, reconstructed trajectories based on the sensed data, and / or recommended trajectories that may represent modified reconstructed trajectories. The server may transmit the model to those vehicles or to other vehicles subsequently traveling on the road to assist in autonomous navigation.

[0292] The geometry of the reconstructed trajectory along the road segment (and also the geometry of the target trajectory) can be represented by a curve in three-dimensional space, which can be a spline connecting cubic polynomials. The reconstructed trajectory curve can be determined from an analysis of a video stream or multiple images captured by a camera mounted on the vehicle. In some embodiments, in each frame or image, a position several meters ahead of the vehicle's current position is identified. This position is the position to which the vehicle is expected to travel within a predetermined time period. This operation can be repeated for each frame, and simultaneously, the vehicle can calculate the ego-motion (rotation and translation) of the camera. In each frame or image, a short-range model of the desired path is generated by the vehicle in the reference frame attached to the camera. The short-range models can be stitched together to obtain a three-dimensional model of the road in some coordinate system, which can be arbitrary or predetermined. The three-dimensional model of the road can then be fitted by a spline, which can include or connect one or more polynomials of a suitable order.

[0293] One or more detection modules may be used to determine a short-distance road model for each frame. For example, a bottom-up lane detection module may be used. The bottom-up lane detection module may be useful when lane markings are painted on the road. This module locates edges in the image and assembles them to form lane markings. A second module may be used together with the bottom-up lane detection module. The second module may be an end-to-end deep neural network that may be trained to predict an accurate short-distance path from the input image. In both modules, the road model may be detected in the image coordinate system and transformed into a three-dimensional space virtually attached to the camera.

[0294] Although the reconstruction trajectory modeling method may result in the accumulation of errors due to the integration of egomotion over long periods of time, which may include noise components, the generated models may provide sufficient accuracy for navigation over regional scales, making such errors insignificant. Additionally, it is possible to offset the integration errors by using information from external sources, such as satellite imagery or geodetic surveys. For example, the disclosed systems and methods may use GNSS receivers to offset the accumulated errors. However, GNSS positioning signals may not always be available and accurate. The disclosed systems and methods may enable steering applications that are weakly dependent on the availability and accuracy of GNSS positioning. In such systems, the use of GNSS signals may be limited. For example, in some embodiments, the disclosed systems may use GNSS signals only for database indexing purposes.

[0295] In some embodiments, range scales (e.g., local scales) that may be relevant to autonomous vehicle navigation and steering applications may be on the order of 50 meters, 100 meters, 200 meters, 300 meters, etc. Such distances may be used because geometric road models are primarily used for two purposes: to pre-plan a trajectory and to localize a vehicle on the road model. In some embodiments, the planning task may use the model over a typical range of 40 meters ahead (or any other suitable distance ahead, such as 20 meters, 30 meters, 50 meters, etc.) when the control algorithm steers the vehicle according to a target point 1.3 seconds ahead (or any other time point, such as 1.5 seconds, 1.7 seconds, 2 seconds, etc.). The localization task uses the road model over a typical range of 60 meters behind the vehicle (or any other suitable distance, such as 50 meters, 100 meters, 150 meters, etc.) by a method called “tail alignment,” which is described in more detail in another section. The disclosed systems and methods can generate geometric models that are accurate enough over a certain range, such as 100 meters, so that the planned trajectory does not deviate from the lane center by more than 30 cm, for example.

[0296] As mentioned above, a three-dimensional road model can be constructed by detecting short-distance sections and stitching them together. Stitching can be made possible by calculating a sixth-order egomotion model using video and / or images captured by cameras, data from inertial sensors reflecting the vehicle's motion, and the host vehicle's velocity signal. The accumulated error can be sufficiently small over a geographical area scale, such as on the order of 100 meters. This can all be completed in a single drive along a particular road segment.

[0297] In some embodiments, multiple runs may be used to average the resulting model to further improve its accuracy. The same vehicle may drive the same route multiple times, or multiple vehicles may send collected model data to a central server. In either case, a matching procedure may be performed to identify duplicate models so that they can be averaged to generate the target trajectory. The constructed model (e.g., including the target trajectory) may be used for steering once convergence criteria are met. Subsequent runs may be used for further model improvement and to adapt to infrastructure changes.

[0298] Sharing driving experience (such as sensed data) between multiple vehicles becomes feasible when multiple vehicles are connected to a central server. Each vehicle client may store a partial copy of the universal road model that may be relevant to the vehicle client's current location. A bidirectional update procedure between the vehicle and the server may be performed by the vehicle and the server. The small footprint concept described above allows the disclosed system and method to perform bidirectional updates using very little bandwidth.

[0299] Information related to potentially existing landmarks may also be identified and transferred to the central server. For example, the disclosed systems and methods may identify one or more physical characteristics of a potentially existing landmark based on one or more images including the landmark. The physical characteristics may include the landmark's physical size (e.g., height, width), the landmark's distance from the vehicle, the distance between the landmark and a previous landmark, the landmark's lateral position (e.g., the landmark's position relative to the driving lane), the landmark's GPS coordinates, the landmark's type, textual identification information for the landmark, etc. For example, the vehicle may analyze one or more images captured by a camera to detect potentially existing landmarks, such as speed limit signs. The vehicle may determine the distance from the vehicle to the landmark based on an analysis of the one or more images. In some embodiments, the distance may be determined based on an analysis of the image of the landmark using suitable image analysis methods, such as scaling and / or optical flow methods. In some embodiments, the disclosed systems and methods may be configured to identify the type or classification of a potentially existing landmark. If a vehicle determines that a particular potential landmark corresponds to a predetermined type or classification stored in the sparse map, it may be sufficient for the vehicle to communicate to the server an indication of the landmark's type or classification along with the landmark's location. The server may store such an indication. Later, other vehicles may capture an image of the landmark, process the image (e.g., using a classifier), and compare the results of the image processing with the server-stored indication regarding the landmark's type. There may be various types of landmarks, and different types of landmarks may be associated with different types of data that are uploaded and stored on the server; different processes onboard the vehicle may detect the landmarks and communicate information about the landmarks to the server; an in-vehicle system may receive the landmark data from the server and use the landmark data to identify landmarks in autonomous navigation.

[0300] In some embodiments, multiple autonomous vehicles traveling a road segment may communicate with a server. The vehicle (or client) may generate a curve describing its maneuver in any coordinate system (e.g., through egomotion integration). The vehicle may detect landmarks and locate the landmarks in the same system. The vehicle may upload the curve and landmarks to the server. The server may collect data from the vehicles over multiple maneuvers and generate an integrated road model. The server may distribute the model to clients (e.g., vehicles). The server may update the model continuously or periodically as it receives new data from the vehicles. For example, the server may process the new data to evaluate whether it contains information that should trigger an update or the creation of new data at the server. The server may distribute the updated model or updates to the vehicles to provide autonomous vehicle navigation.

[0301] The server may use one or more criteria to determine whether new data received from a vehicle should trigger an update to the model or the creation of new data. For example, if the new data indicates that a previously recognized landmark at a particular location is no longer present or has been replaced with another landmark, the server may determine that the new data should trigger an update to the model. As another example, if the new data indicates that a road segment is closed and this is corroborated by data received from other vehicles, the server may determine that the new data should trigger an update to the model.

[0302] The server may distribute the updated model (or an updated portion of the model) to one or more vehicles traveling on a road segment with which the update to the model is associated. The server can also distribute the updated model to vehicles that are about to travel on a road segment with which the update to the model is associated or whose planned journey includes that road segment. For example, while the autonomous vehicle is traveling along another road segment before reaching the road segment with which the update is associated, the server may distribute the update or the updated model to the autonomous vehicle before the autonomous vehicle reaches that road segment.

[0303] In some embodiments, a remote server may collect trajectories and landmarks from multiple clients (e.g., vehicles traveling a common road segment). The server may use the landmarks to match curves and create an average road model based on the trajectories collected from multiple vehicles. The server may also calculate the road graph and the most likely path at each node or junction of the road segment.

[0304] The server may average landmark characteristics received from multiple vehicles that traveled along a common road segment, such as the distance from one landmark to another (e.g., the previous landmark along the road segment) as measured by the multiple vehicles, to determine arc-length parameters and support along-route positioning and speed calibration for each client vehicle. The server may average physical dimensions of landmarks measured by multiple vehicles that traveled along a common road segment and recognized as the same landmark. The average physical dimensions may be used to support distance estimation, such as the distance from the vehicle to the landmark. The server may average lateral positions (e.g., the position of the landmark from the lane in which the vehicle is traveling) of landmarks measured by multiple vehicles that traveled along the common road segment and recognized as the same landmark. The average lateral positions may be used to support lane assignment. The server may average GPS coordinates of landmarks measured by multiple vehicles that traveled along the same road segment and recognized as the same landmark. The average GPS coordinates of the landmarks may be used to support global positioning or location of the landmark within the road model.

[0305] In some embodiments, the server may identify model changes, such as construction, detours, new signs, sign removal, etc., based on data received from the vehicle. The server may update the model continuously, periodically, or immediately as it receives new data from the vehicle. The server may distribute updates to the model or updated models to the vehicle to provide autonomous navigation.

[0306] In some embodiments, the server may analyze driver intervention during autonomous driving. The server may analyze data received from the vehicle at the time and location where the intervention occurred and / or data received before the time the intervention occurred. The server may identify specific portions of data that caused or are closely related to the intervention, such as data indicating a temporary lane closure setup or data indicating a pedestrian on the road. The server may update the model based on the identified data. For example, the server may modify one or more trajectories stored in the model.

[0307] According to disclosed embodiments, the system can store information obtained during autonomous navigation (or regular driver-controlled navigation) for use in subsequent trips along the same road. The system can share that information with other vehicles as they navigate along that road. Each client system can then further refine the crowd-sourced data based on observed local conditions.

[0308] FIG. 12 is a schematic diagram of a system using crowdsourced data for autonomous vehicle navigation. FIG. 12 shows a road segment 1200 including one or more lanes. Multiple vehicles 1205, 1210, 1215, 1220, and 1225 may travel on the road segment 1200 at the same time or at different times (although they are shown in FIG. 12 as appearing on the road segment 1200 simultaneously). At least one of the vehicles 1205-1225 may be an autonomous vehicle. For simplicity of this example, it is assumed that all of the vehicles 1205-1225 are autonomous vehicles. Each vehicle may be similar to a vehicle disclosed in other embodiments (e.g., vehicle 200) and may include components or devices included in or associated with a vehicle disclosed in other embodiments. Each vehicle may be equipped with an image capture device or camera (e.g., image capture device 122 or camera 122). Each vehicle may communicate with remote server 1230 over one or more networks (e.g., via a cellular network and / or the Internet, etc.) through wireless communication path 1235, as shown by the dashed lines. Each vehicle may send data to server 1230 and receive data from server 1230. For example, server 1230 may collect data from multiple vehicles traveling road segment 1200 at different times and process the collected data to generate an autonomous vehicle road navigation model or updates to the model. Server 1230 may send updates to the autonomous vehicle road navigation model or models to vehicles that sent data to server 1230. Server 1230 may later send updates to the autonomous vehicle road navigation model or models to other vehicles that travel road segment 1200.

[0309] Navigation information collected (e.g., detected, sensed, or measured) by the vehicles 1205-1225 as they travel along the road segment 1200 may be transmitted to the server 1230. In some embodiments, the navigation information may be associated with a common road segment 1200. The navigation information may include a trajectory associated with each vehicle 1205-1225 as it travels along the road segment 1200. In some embodiments, the trajectory may be reconstructed based on data sensed by various sensors and devices provided on the vehicles 1205. For example, the trajectory may be reconstructed based on at least one of accelerometer data, speed data, landmark data, road geometry or profile data, vehicle positioning data, and egomotion data. In some embodiments, the trajectory may be reconstructed based on data from inertial sensors, such as accelerometers, and the speed of the vehicle 1205 as sensed by a speed sensor. Additionally, in some embodiments, the trajectory may be determined based on the detected ego-motion of the camera (e.g., by an on-board processor of each of the vehicles 1205-1225), where the ego-motion may represent a three-dimensional translation and / or a three-dimensional rotation (or rotational movement). The ego-motion of the camera (and thus the vehicle body) may be determined from an analysis of one or more images captured by the camera.

[0310] In some embodiments, the trajectory of vehicle 1205 may be determined by a processor provided onboard vehicle 1205 and transmitted to server 1230. In other embodiments, server 1230 may receive data sensed by various sensors and devices provided on vehicle 1205 and determine the trajectory based on the data received from vehicle 1205.

[0311] In some embodiments, navigation information transmitted from vehicles 1205-1225 to server 1230 may include data regarding road geometry or profile. The geometry of road segment 1200 may include lane structure and / or landmarks. Lane structure may include the total number of lanes in road segment 1200, the type of lane (e.g., one-way lane, two-way lane, driving lane, passing lane, etc.), markings on the lane, lane width, etc. In some embodiments, navigation information may include lane assignments, such as which lane out of multiple lanes the vehicle is traveling in. For example, a lane assignment may be associated with the numeric value "3" indicating that the vehicle is traveling in the third lane from the left or right. As another example, a lane assignment may be associated with the text value "center lane" indicating that the vehicle is traveling in the center lane.

[0312] The server 1230 may store the navigation information on a non-transitory computer-readable medium, such as a hard drive, compact disc, tape, memory, etc. The server 1230 may generate at least a portion of an autonomous vehicle road navigation model for the common road segment 1200 (e.g., through a processor included in the server 1230) based on navigation information received from multiple vehicles 1205-1225. The server 1230 may determine a trajectory associated with each lane based on crowd-sourced data (e.g., navigation information) received from multiple vehicles (e.g., 1205-1225) traveling on the lanes of the road segment at different times. The server 1230 may generate an autonomous vehicle road navigation model or a portion of the model (e.g., an updated portion) based on the multiple trajectories determined based on the crowd-sourced navigation data. The server 1230 may transmit the model or an updated portion of the model to one or more of the autonomous vehicles 1205-1225 currently traveling on the road segment 1200 or any other autonomous vehicles that later travel the road segment to update an existing autonomous vehicle road navigation model provided in the vehicle's navigation system. The autonomous vehicle road navigation model may be used by an autonomous vehicle in autonomously navigating along common road segments 1200.

[0313] In some embodiments, the autonomous vehicle road navigation model may be included in a sparse map (e.g., sparse map 800 shown in FIG. 8 ). Sparse map 800 may include a sparse record of data related to road geometry and / or landmarks along roads, which may provide sufficient information to guide the autonomous navigation of the autonomous vehicle, yet without requiring excessive data storage. In some embodiments, the autonomous vehicle road navigation model may be stored separately from sparse map 800, and may use map data from sparse map 800 when the model is implemented for navigation. In some embodiments, the autonomous vehicle road navigation model may use the map data included in sparse map 800 to determine a target trajectory along road segment 1200 to guide the autonomous navigation of autonomous vehicles 1205-1225 or other vehicles that subsequently travel along road segment 1200. For example, if the autonomous vehicle road navigation model is executed by a processor included in the navigation system of vehicle 1205, the model may cause the processor to compare a trajectory determined based on navigation information received from vehicle 1205 with a predetermined trajectory included in sparse map 800 to verify and / or correct the current course of travel of vehicle 1205. ...

Claims

1. 1. An apparatus for controlling operation of a vehicle based on crowd-sourced data, comprising: A processor; a memory containing instructions; When executed by the processor, the processor: accessing an autonomous vehicle road navigation model for navigation of a route by the vehicle, the autonomous vehicle road navigation model utilizing data relating to a speed along the route; calculating a target speed of the vehicle along the route based on the autonomous vehicle road navigation model, the target speed of the vehicle along the route being calculated for use in a predicted route of navigation of the vehicle along the route using a trajectory established for a roadway located on the route, the trajectory including a three-dimensional spline representing a predetermined path of travel along the roadway, the predicted route of navigation including one or more curves of the roadway; obtaining navigation information for the route, the navigation information being based on data collected from one or more other vehicles on the route, and the navigation information indicating current conditions on the route that affect the target speed of the vehicle along the route; driving said vehicle at a reduced speed around said one or more curves on said route; Run the command, The vehicle is driven at the reduced speed along a trajectory established with respect to the roadway; the reduced velocity is based on the calculated target velocity and the navigation information; the one or more curves are identified based on a position of the vehicle along the trajectory before at least a portion of the one or more curves are captured in an image; a deceleration of the vehicle is automatically applied to change to the reduced speed prior to the one or more curves. Device.

2. The apparatus of claim 1 , wherein the navigation information further indicates changed conditions for the navigation of the route.

3. The apparatus of claim 1 or 2, wherein the navigation information indicates hazards detected on the route that have been observed by the one or more other vehicles along the route.

4. An apparatus according to claim 1 , wherein the navigation information is obtained from a server during navigation of the vehicle along the route.

5. An apparatus according to claim 1 , wherein the navigation information indicates changes in traffic conditions that cause a change in speed of the one or more other vehicles on the route.

6. An apparatus according to claim 1 , wherein the target speed of the vehicle along the route is updated in response to environmental conditions encountered by the one or more other vehicles along the route.

7. An apparatus according to any preceding claim, wherein the target speed of the vehicle along the route is updated without driver input.

8. 8. The apparatus of claim 1 , wherein operating the vehicle at the reduced speed along the path is performed by a control system of the vehicle, the control system being configured to control acceleration and deceleration of the vehicle.

9. 9. An apparatus according to claim 1, wherein the position of the vehicle along the route is determined based on a determined distance between the vehicle and the one or more curves.

10. 1. A method performed by a navigation control system of a vehicle based on crowd-sourced data, comprising: accessing an autonomous vehicle road navigation model for navigation of a route by the vehicle, the autonomous vehicle road navigation model including data utilizing a speed along the route; calculating a target speed of the vehicle along the route based on the autonomous vehicle road navigation model, the target speed of the vehicle along the route being calculated for use in a predicted route of navigation of the vehicle along the route using a trajectory established for roadways located on the route, the trajectory including a three-dimensional spline representing a predetermined path of travel along the roadway, the predicted route of navigation including one or more curves in the roadway; obtaining navigation information for the route, the navigation information being based on data collected from one or more other vehicles on the route, and the navigation information indicating current conditions on the route that affect the target speed of the vehicle along the route; driving the vehicle at a reduced speed along the one or more curves on the route, the vehicle being driven at the reduced speed along a trajectory established for the roadway, the reduced speed being based on the calculated target speed and the navigation information, the one or more curves being identified based on a position of the vehicle along the trajectory before at least a portion of the one or more curves are captured in an image, and a deceleration of the vehicle is automatically applied to change to the reduced speed prior to the one or more curves; A method for providing the above.

11. The method of claim 10 , wherein the navigation information further indicates changed conditions for the navigation of the route.

12. The method of claim 10 or 11, wherein the navigation information indicates hazards detected on the route that have been observed by the one or more other vehicles along the route.

13. 13. A method according to any one of claims 10 to 12, wherein the navigation information is obtained from a server during navigation of the vehicle along the route.

14. 14. The method of claim 10, wherein the navigation information indicates a change in traffic conditions that causes a change in speed of the one or more other vehicles on the route.

15. 15. A method according to any one of claims 10 to 14, wherein the target speed of the vehicle along the route is updated in response to environmental conditions encountered by the one or more other vehicles along the route.

16. 16. A method according to any one of claims 10 to 15, wherein the target speed of the vehicle along the route is updated without driver input.

17. 17. The method of claim 10, wherein operating the vehicle at the reduced speed along the path is performed by a control system of the vehicle, the control system being configured to control acceleration and deceleration of the vehicle.

18. 18. The method of any one of claims 10 to 17, wherein the position of the vehicle along the route is identified based on a determined distance between the vehicle and the one or more curves.

19. A vehicle, An autonomous driving control system for controlling the operation of the vehicle; a navigation control unit operably coupled to the autonomous driving control system; A vehicle, wherein the navigation control unit includes circuitry configured to carry out the method of any one of claims 10 to 18.

20. A program causing an electromechanical device to carry out the method of any one of claims 10 to 18.

21. At least one computer readable storage medium with stored instructions, the at least one computer readable storage medium storing the program of claim 20.

Images