Accident liability tracking system for autonomous host vehicle

Abstract

An accident liability tracking system for an autonomous host vehicle is provided. The system includes a processing device including circuitry and a memory, where the memory includes instructions that, when executed by the circuitry, cause the processing device to: receive at least one image; analyzing the at least one image to identify a target vehicle in the environment of the host vehicle; determining one or more characteristics of the identified navigation state of the target vehicle; determining a collision between the expected host vehicle and the target vehicle; based on the determination of the expected collision, determining whether the host vehicle can avoid the collision based on a navigation state of the host vehicle; comparing the determined one or more characteristics of the identified navigation state of the target vehicle with a set of accident liability rules related to a change in the lateral position of the vehicle based on the determination that the collision cannot be avoided; based on the comparison, storing at least one value indicative of potential accident liability on the identified portion of the target vehicle; and outputting the stored at least one value for determining the accident liability.

Description

[0001] This invention is a divisional application of the invention patent application with application date of December 21, 2017, application number 201780084515.6, and invention name “Navigation system with imposed liability constraints”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 438,563, filed December 23, 2016, U.S. Provisional Patent Application No. 62 / 546,343, filed August 16, 2017, U.S. Provisional Patent Application No. 62 / 565,244, filed September 29, 2017, and U.S. Provisional Patent Application No. 62 / 582,687, filed November 7, 2017. All of the foregoing applications are incorporated herein by reference in their entirety. Technical Field

[0004] The present disclosure generally relates to autonomous vehicle navigation. In addition, the present disclosure relates to systems and methods for navigating within potential accident liability constraints. Background Art

[0005] As technology continues to advance, the goal of fully autonomous vehicles capable of navigating roadways is imminent. Autonomous vehicles may need to consider a variety of factors and make appropriate decisions based on those factors to safely and accurately reach a desired destination. For example, autonomous vehicles may need to process and interpret visual information (e.g., captured by cameras), information from radar or lidar, and may also use information obtained from other sources (e.g., GPS devices, speed sensors, accelerometers, suspension sensors, etc.). Furthermore, to navigate to a destination, autonomous vehicles may also need to identify their location within a specific roadway (e.g., a specific lane on a multi-lane road), navigate alongside other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, cross from one road to another at appropriate intersections or junctions, and respond to any other circumstances that occur or develop during vehicle operation. Furthermore, navigation systems may need to adhere to certain imposed constraints. In some cases, these constraints may involve interactions between the host vehicle and one or more other objects (such as other vehicles, pedestrians, etc.). In other cases, the constraints may involve responsibility rules to be followed when performing one or more navigation actions on behalf of the host vehicle.

[0006] In the autonomous driving field, there are two key considerations for viable autonomous vehicle systems. The first is standardization of safety assurance, including the requirements that each autonomous vehicle must meet to guarantee safety, and how to verify these requirements. The second is scalability, as engineering solutions that incur significant costs will not scale to millions of vehicles and could prevent widespread adoption of autonomous vehicles, or even prevent widespread adoption. Therefore, there is a need for an interpretable mathematical model for safety assurance and a system design that adheres to safety assurance requirements while being scalable to millions of vehicles. Summary of the Invention

[0007] Embodiments consistent with the present disclosure provide systems and methods for autonomous vehicle navigation. The disclosed embodiments may use cameras to provide autonomous vehicle navigation features. For example, consistent with the disclosed embodiments, the disclosed system may include one, two, or more cameras that monitor the vehicle environment. The disclosed system may provide a navigation response based on, for example, analysis of images captured by one or more cameras. The navigation response may also take into account other data, including, for example, global positioning system (GPS) data, sensor data (e.g., from accelerometers, velocity sensors, suspension sensors, etc.), and / or other map data.

[0008] At least one embodiment of the present disclosure provides an accident responsibility tracking system for an autonomous host vehicle traveling in a multi-lane corridor, the system comprising: at least one processing device comprising circuitry and memory, wherein the memory comprises instructions that, when executed by the circuitry, cause the at least one processing device to: receive at least one image captured by an image capture device via a data connection, the at least one image representing an environment of the host vehicle; analyze the at least one image to identify a target vehicle in the environment of the host vehicle, the target vehicle traveling in a lane adjacent to a lane in which the host vehicle is traveling, the analysis comprising applying at least one of computer vision software or a trained learning model; determine one or more characteristics of a navigational state of the identified target vehicle based on the analysis of the at least one image, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; determine an expected collision between the host vehicle and the target vehicle based on the analysis of the at least one image; and determine, based on the determination of the expected collision, whether a collision occurred based on the navigational state of the host vehicle. Determining whether a host vehicle can avoid a collision, wherein the determination of whether a collision can be avoided includes generating a plurality of planned navigation actions for the host vehicle and determining whether there is a predicted future state predicted to avoid a collision, the predicted future state corresponding to at least one of the plurality of planned navigation actions; based on the determination that a collision cannot be avoided, comparing one or more characteristics of the determined navigation state of the identified target vehicle with a set of accident liability rules related to changes in the lateral position of the vehicles, the rules including a time to responsibility for an expected collision, the time to responsibility including an earliest time before the collision at which the target vehicle enters a lane of travel of the host vehicle, wherein a longitudinal distance between the host vehicle and the target vehicle is unsafe; based on the comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the set of accident liability rules, storing at least one value indicating potential accident liability on a portion of the identified target vehicle, the value indicating potential accident liability being determined at least in part based on changes in the lateral position of the target vehicle; and outputting the at least one stored value for determining accident liability.

[0009] At least one embodiment of the present disclosure provides a system for navigating an autonomous host vehicle traveling in a multi-lane corridor, the system comprising: at least one processing device comprising circuitry and memory, wherein the memory comprises instructions that, when executed by the circuitry, cause the at least one processing device to: receive, via a data connection, a plurality of images captured by an image capture device, the plurality of images representing an environment of the host vehicle; determine, based on at least one driving strategy, a planned navigation action for completing a navigation goal of the host vehicle; determine a trajectory of the host vehicle to complete the planned navigation goal; analyze the plurality of images to identify a target vehicle in the environment of the host vehicle and a predicted trajectory of the target vehicle, the target vehicle being traveling in a lane adjacent to a lane of travel of the host vehicle, the analysis comprising applying a computational At least one of machine vision software or a trained learning model; determining an expected collision between a host vehicle and a target vehicle based on a trajectory of the host vehicle and a trajectory of the target vehicle; based on the determination of the expected collision, testing a planned navigation action of the host vehicle against an accident responsibility rule set associated with a change in the lateral position of the vehicles, the rule including a responsibility time for the expected collision, the responsibility time including an earliest time before the collision at which the target vehicle enters a lane of travel of the host vehicle, wherein a longitudinal distance between the host vehicle and the target vehicle is unsafe, wherein potential accident responsibility is determined based on the predicted trajectory of the target vehicle and the trajectory of the host vehicle; if the testing of the planned navigation action against the accident responsibility rule set indicates, if the planned navigation action is taken , there is potential accident liability for the main vehicle, then the main vehicle is not implemented to perform the planned navigation action; and if the test indication for the planned navigation action for the accident liability rule set is that if the planned navigation action is taken, no accident liability will be generated for the main vehicle, then the main vehicle is implemented to perform the planned navigation action; and wherein at least one processing device is further programmed to: determine one or more characteristics of the navigation state of the identified target vehicle based on the analysis of multiple images, the one or more characteristics including a change in the lateral position of the target vehicle relative to at least one of the center of the lane in which the target vehicle is traveling or the lateral position of the main vehicle; based on the determination of the expected collision, determine whether the main vehicle can avoid the collision between the main vehicle and the target vehicle based on the navigation state of the main vehicle. A collision is detected, wherein the determination of whether the collision can be avoided includes generating a plurality of additional planned navigation actions for the host vehicle and determining whether there is a predicted future state predicted to avoid the collision, the predicted future state corresponding to at least one of the plurality of planned additional navigation actions; based on the determination that the collision cannot be avoided, comparing one or more characteristics of the determined navigation state of the identified target vehicle with an accident responsibility rule set; and based on the comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the accident responsibility rule set, storing at least one value indicating potential accident responsibility on a portion of the identified target vehicle, the value indicating potential accident responsibility being determined at least in part based on a change in a lateral position of the target vehicle.

[0010] At least one embodiment of the present disclosure provides a method for tracking accident responsibility of an autonomous host vehicle traveling in a multi-lane corridor, the method comprising: receiving at least one image captured by an image capture device via a data connection, the at least one image representing an environment of the host vehicle; analyzing the at least one image to identify a target vehicle in the environment of the host vehicle, the target vehicle traveling in a lane adjacent to a lane in which the host vehicle is traveling, the analysis comprising applying at least one of computer vision software or a trained learning model; determining one or more characteristics of a navigational state of the identified target vehicle based on the analysis of the at least one image, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; determining an expected collision between the host vehicle and the target vehicle based on the analysis of the at least one image; and determining whether the host vehicle can avoid the collision based on the navigational state of the host vehicle based on the determination of the expected collision, wherein the determination of whether the collision can be avoided is determined. The method includes generating a plurality of planned navigation actions for a host vehicle and determining whether there is a predicted future state predicted to avoid a collision, the predicted future state corresponding to at least one of the plurality of planned navigation actions; based on a determination that a collision cannot be avoided, comparing one or more characteristics of the determined navigation state of the identified target vehicle with a set of accident liability rules related to a change in a lateral position of the vehicle, the rules including a time to responsibility for an expected collision, the time to responsibility including an earliest time before the collision at which the target vehicle enters a lane of travel of the host vehicle, wherein a longitudinal distance between the host vehicle and the target vehicle is unsafe; based on a comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the set of accident liability rules, storing at least one value indicating potential accident liability on a portion of the identified target vehicle, the value indicating potential accident liability being determined at least in part based on a change in the lateral position of the target vehicle; and outputting the at least one stored value for determining accident liability.

[0011] At least one embodiment of the present disclosure provides a method for navigating an autonomous host vehicle traveling in a multi-lane corridor, the method comprising: receiving a plurality of images captured by an image capture device via a data connection, the plurality of images representing an environment of the host vehicle; determining a planned navigation action for completing a navigation goal of the host vehicle based on at least one driving strategy; determining a trajectory of the host vehicle to complete the planned navigation goal; analyzing the plurality of images to identify a target vehicle in the environment of the host vehicle and a predicted trajectory of the target vehicle, the target vehicle traveling in a lane adjacent to a lane in which the host vehicle is traveling, the analyzing comprising applying at least one of computer vision software or a trained learning model; and determining a trajectory of the host vehicle to complete the planned navigation goal based on at least one driving strategy. The method includes determining a trajectory of the host vehicle and a trajectory of the target vehicle to determine an expected collision between the host vehicle and the target vehicle; based on the determination of the expected collision, testing the planned navigation action of the host vehicle against an accident responsibility rule set related to a change in the lateral position of the vehicle, the rule including a responsibility time for the expected collision, the responsibility time including an earliest time before the collision, at which the target vehicle enters the lane of travel of the host vehicle, wherein the longitudinal distance between the host vehicle and the target vehicle is unsafe, wherein potential accident responsibility is determined based on the predicted trajectory of the target vehicle and the trajectory of the host vehicle; if the test indication of the planned navigation action against the accident responsibility rule set is, if the planned navigation action is taken if the host vehicle is potentially responsible for the accident if the planned navigation action is taken, causing the host vehicle to not perform the planned navigation action; if the test of the planned navigation action against the accident responsibility rule set indicates that the host vehicle would not be responsible for the accident if the planned navigation action is taken, causing the host vehicle to perform the planned navigation action; determining one or more characteristics of a navigation state of the identified target vehicle based on the analysis of the plurality of images, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; determining whether the host vehicle can avoid a collision between the host vehicle and the target vehicle based on the navigation state of the host vehicle based on the determination of an expected collision, wherein the determination of whether the collision can be avoided includes generating a plurality of additional planned navigation actions for the host vehicle and determining whether there is a predicted future state predicted to avoid the collision, the predicted future state corresponding to at least one of the plurality of planned additional navigation actions; based on a determination that the collision cannot be avoided, comparing the determined one or more characteristics of the navigation state of the identified target vehicle to the accident responsibility rule set; and storing at least one value indicating potential accident responsibility on a portion of the identified target vehicle based on the comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the accident responsibility rule set.

[0012] At least one embodiment of the present disclosure provides a non-transitory computer-readable medium containing instructions that, when executed by at least one processor, cause the at least one processor to perform a method for navigating an autonomous host vehicle traveling in a multi-lane corridor, the method comprising: receiving at least one image captured by an image capture device via a data connection, the at least one image representing an environment of the host vehicle; analyzing the at least one image to identify a target vehicle in the environment of the host vehicle, the target vehicle traveling in a lane adjacent to a lane in which the host vehicle is traveling, the analyzing comprising applying at least one of computer vision software or a trained learning model; determining one or more characteristics of a navigational state of the identified target vehicle based on the analysis of the at least one image, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; determining an expected collision between the host vehicle and the target vehicle based on the analysis of the at least one image; and determining, based on the determination of the expected collision, whether the host vehicle is near the navigational state of the host vehicle. Whether a collision can be avoided, wherein the determination of whether a collision can be avoided includes generating a plurality of planned navigation actions for a host vehicle and determining whether there is a predicted future state predicted to avoid a collision, the predicted future state corresponding to at least one of the plurality of planned navigation actions; based on the determination that a collision cannot be avoided, comparing one or more characteristics of the determined navigation state of the identified target vehicle with a set of accident liability rules related to changes in the lateral position of the vehicle, the rules including a time of responsibility for an expected collision, the time of responsibility including an earliest time before the collision at which the target vehicle enters a lane of travel of the host vehicle, wherein a longitudinal distance between the host vehicle and the target vehicle is unsafe; based on the comparison of the one or more characteristics of the determined navigation state of the identified target vehicle with the set of accident liability rules, storing at least one value indicating potential accident liability on a portion of the identified target vehicle, the value indicating potential accident liability being determined at least in part based on changes in the lateral position of the target vehicle; and outputting the at least one stored value for determining accident liability.

[0013] Systems and methods for navigating a host vehicle are provided. In some embodiments, the system may include at least one processing device programmed to: receive at least one image representing an environment of the host vehicle from an image capture device; determine, based on at least one driving strategy, a planned navigation action for achieving a navigation goal of the host vehicle; analyze the at least one image to identify a target vehicle in the environment of the host vehicle; test the planned navigation action against at least one accident liability rule to determine potential accident liability of the host vehicle relative to the identified target vehicle; cause the host vehicle to not implement the planned navigation action if the testing of the planned navigation action against the at least one accident liability rule indicates that potential accident liability of the host vehicle exists if the planned navigation action is implemented; and cause the host vehicle to implement the planned navigation action if the testing of the planned navigation action against the at least one accident liability rule indicates that no accident liability exists for the host vehicle if the planned navigation action is implemented.

[0014] In some embodiments, a navigation system for a main vehicle may include at least one processing device programmed to: receive at least one image representing an environment of the main vehicle from an image capture device; determine a plurality of potential navigation actions for the main vehicle based on at least one driving strategy; analyze the at least one image to identify a target vehicle in the environment of the main vehicle; test the plurality of potential navigation actions against at least one accident liability rule to determine the potential accident liability of the main vehicle relative to the identified target vehicle; select one of the potential navigation actions for which a test indicates that no accident liability would result for the main vehicle if the selected potential navigation action were taken; and cause the main vehicle to implement the selected potential navigation action.

[0015] In some embodiments, a system for navigating a main vehicle may include at least one processing device programmed to: receive at least one image representing an environment of the main vehicle from an image capture device; determine two or more planned navigation actions for achieving a navigation goal of the main vehicle based on at least one driving strategy; analyze at least one image to identify a target vehicle in the environment of the main vehicle; test each of the two or more planned navigation actions against at least one accident liability rule to determine potential accident liability; for each of the two or more planned navigation actions, if the test indicates that there is potential accident liability of the main vehicle if a particular one of the two or more planned navigation actions is taken, cause the main vehicle not to implement the particular one of the planned navigation actions; and for each of the two or more planned navigation actions, if the test indicates that no accident liability of the main vehicle would result if the particular one of the two or more planned navigation actions were taken, identify the particular one of the two or more planned navigation actions as a feasible candidate for implementation; select a navigation action to be taken from the feasible candidates for implementation based on at least one cost function; and cause the main vehicle to implement the selected navigation action.

[0016] In some embodiments, an accident responsibility tracking system for a host vehicle may include at least one processing device programmed to: receive at least one image representing the host vehicle's environment from an image capture device; analyze the at least one image to identify a target vehicle in the host vehicle's environment; determine one or more characteristics of a navigation state of the identified target vehicle based on the analysis of the at least one image; compare the determined one or more characteristics of the navigation state of the identified target vehicle with at least one accident responsibility rule; store at least one value indicating potential accident responsibility of the identified target vehicle based on the comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with at least one accident responsibility rule; and, after an accident occurs between the host vehicle and at least one target vehicle, output the stored at least one value for determining responsibility for the accident.

[0017] In some embodiments, a system for navigating a host vehicle may include at least one processing device programmed to: receive at least one image representing an environment of the host vehicle from an image capture device; determine a planned navigation action for achieving a navigation goal of the host vehicle based on at least one driving strategy; analyze the at least one image to identify a target vehicle in the environment of the host vehicle; test the planned navigation action against at least one accident liability rule to determine potential accident liability; if the testing of the planned navigation action against at least one accident liability rule indicates that there is potential accident liability of the host vehicle if the planned navigation action is taken, cause the host vehicle to not implement the planned navigation action; and if the testing of the planned navigation action against at least one accident liability rule indicates that there is no accident liability of the host vehicle if the planned navigation action is taken, cause the host vehicle to implement the planned navigation action; and wherein the at least one processing device is further programmed to: determine one or more characteristics of a navigation state of the identified target vehicle based on the analysis of the at least one image; compare the determined one or more characteristics of the navigation state of the identified target vehicle with at least one accident liability rule; and store at least one value indicative of potential accident liability on the part of the identified target vehicle based on the comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the at least one accident liability rule.

[0018] In some embodiments, a system for navigating a host vehicle may include at least one processing device programmed to: receive at least one image representing the host vehicle's environment from an image capture device; determine a planned navigation action for achieving the host vehicle's navigation goal based on at least one driving strategy; analyze the at least one image to identify a target vehicle in the host vehicle's environment; determine a next-state distance between the host vehicle and the target vehicle that would result if the planned navigation action were taken; determine a current maximum braking capacity of the host vehicle and a current speed of the host vehicle; determine the current speed of the target vehicle and assume a maximum braking capacity of the target vehicle based on at least one identified characteristic of the target vehicle; and implement the planned navigation action if, given the host vehicle's maximum braking capacity and the host vehicle's current speed, the host vehicle can stop within a stopping distance that is less than the sum of the determined next-state distance and a distance traveled by the target vehicle determined based on the target vehicle's current speed and the assumed maximum braking capacity of the target vehicle.

[0019] In some embodiments, a system for navigating a main vehicle may include at least one processing device programmed to: receive at least one image representing an environment of the main vehicle from an image capture device; determine a planned navigation action for achieving a navigation goal of the main vehicle based on at least one driving strategy; analyze at least one image to identify a first target vehicle in front of the main vehicle and a second target vehicle in front of the first target vehicle; determine a next-state distance between the main vehicle and the second target vehicle that would result if the planned navigation action were taken; determine a current maximum braking capacity of the main vehicle and a current speed of the main vehicle; and implement the planned navigation action if, given the maximum braking capacity of the main vehicle and the current speed of the main vehicle, the main vehicle can stop within a stopping distance that is less than the determined next-state distance between the main vehicle and the second target vehicle.

[0020] In some embodiments, a system for navigating a host vehicle may include at least one processing device programmed to: receive at least one image representing an environment of the host vehicle from an image capture device; analyze the at least one image to identify a target vehicle in the environment of the host vehicle; determine two or more potential navigation actions for achieving a navigation goal of the host vehicle; test each of the two or more potential navigation actions against at least one accident liability rule to determine, for each of the two or more potential navigation actions, an indicator of potential accident liability between the host vehicle and the identified target vehicle; and select one of the two or more potential navigation actions to implement only if the indicator of potential accident liability associated with the selected action indicates that no potential accident liability will be attributed to the host vehicle as a result of implementing the selected action.

[0021] In some embodiments, a system for navigating a main vehicle may include at least one processing device programmed to: receive at least one image representing an environment of the main vehicle from an image capture device; receive an indicator of a current navigation state of the main vehicle from at least one sensor; determine that a collision between the main vehicle and one or more objects is inevitable based on analysis of the at least one image and based on the indicator of the current navigation state of the main vehicle; determine, based on at least one driving strategy, a first planned navigation action of the main vehicle involving an expected collision with a first object and a second planned navigation action of the main vehicle involving an expected collision with a second object; test the first and second planned navigation actions against at least one accident liability rule to determine potential accident liability; if the test of the first planned navigation action against at least one accident liability rule indicates that there is potential accident liability for the main vehicle if the first planned navigation action is taken, causing the main vehicle to not implement the first planned navigation action; and if the test of the second planned navigation action against at least one accident liability rule indicates that there will be no accident liability for the main vehicle if the second planned navigation action is taken, causing the main vehicle to implement the second planned navigation action.

[0022] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executable by at least one processing device to perform any of the steps and / or methods described herein.

[0023] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments.

[0025] In the attached figure:

[0026] Figure 1 is a pictorial representation of an example system consistent with the disclosed embodiments.

[0027] Figure 2A is a diagrammatic side view representation of an example vehicle including systems consistent with the disclosed embodiments.

[0028] Figure 2B is consistent with the disclosed embodiments Figure 2A Diagrammatic top view representation of the vehicle and systems shown in .

[0029] Figure 2C is a diagrammatic top view representation of another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0030] Figure 2Dis a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0031] Figure 2E is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0032] Figure 2F is a pictorial representation of an example vehicle control system consistent with the disclosed embodiments.

[0033] Figure 3A is a pictorial representation of the interior of a vehicle including a rearview mirror and a user interface for a vehicle imaging system, consistent with the disclosed embodiments.

[0034] Figure 3B is an illustration of an example of a camera installation configured to be positioned behind a rearview mirror and against a vehicle windshield, consistent with the disclosed embodiments.

[0035] Figure 3C is consistent with the disclosed embodiments Figure 3B Illustrations of the camera installation from different viewing angles are shown in FIG.

[0036] Figure 3D is an illustration of an example of a camera installation configured to be located behind a rearview mirror and against a vehicle windshield, consistent with the disclosed embodiments.

[0037] Figure 4 is an example block diagram of a memory configured to store instructions for performing one or more operations, consistent with the disclosed embodiments.

[0038] Figure 5A is a flow chart illustrating an example process for eliciting one or more navigation responses based on monocular image analysis, consistent with the disclosed embodiments.

[0039] Figure 5B is a flow chart illustrating an example process for detecting one or more vehicles and / or pedestrians in a set of images, consistent with the disclosed embodiments.

[0040] Figure 5C is a flow chart illustrating an example process for detecting road markings and / or lane geometry information in a set of images, consistent with the disclosed embodiments.

[0041] Figure 5D is a flow chart illustrating an example process for detecting traffic lights in a set of images, consistent with the disclosed embodiments.

[0042] Figure 5Eis a flow chart illustrating an example process for eliciting one or more navigation responses based on a vehicle path, consistent with the disclosed embodiments.

[0043] Figure 5F is a flow chart illustrating an example process for determining whether a leading vehicle is changing lanes, consistent with the disclosed embodiments.

[0044] Figure 6 is a flow chart illustrating an example process for eliciting one or more navigation responses based on stereo image analysis, consistent with the disclosed embodiments.

[0045] Figure 7 is a flow chart illustrating an example process for eliciting one or more navigation responses based on analysis of three sets of images, consistent with the disclosed embodiments.

[0046] Figure 8 is a block diagram representation of modules that may be implemented by one or more specially programmed processing devices of a navigation system of an autonomous vehicle, consistent with the disclosed embodiments.

[0047] Figure 9 is a diagram of navigation options consistent with the disclosed embodiments.

[0048] Figure 10 is a diagram of navigation options consistent with the disclosed embodiments.

[0049] Figure 11A 、 Figure 11B and Figure 11C A schematic representation of navigation options for a host vehicle in a merge zone consistent with the disclosed embodiments is provided.

[0050] Figure 11D An illustrative description of a dual-lane merging scenario consistent with the disclosed embodiments is provided.

[0051] Figure 11E A diagram is provided of options that are potentially useful in a double-merge scenario consistent with the disclosed embodiments.

[0052] Figure 12 A diagram is provided that captures a representative image of a host vehicle's environment and potential navigation constraints consistent with the disclosed embodiments.

[0053] Figure 13 A flow chart of an algorithm for navigating a vehicle consistent with the disclosed embodiments is provided.

[0054] Figure 14 A flow chart of an algorithm for navigating a vehicle consistent with the disclosed embodiments is provided.

[0055] Figure 15A flow chart of an algorithm for navigating a vehicle consistent with the disclosed embodiments is provided.

[0056] Figure 16 A flow chart of an algorithm for navigating a vehicle consistent with the disclosed embodiments is provided.

[0057] Figure 17A and 17B An illustrative illustration of a host vehicle navigating into a roundabout is provided, consistent with the disclosed embodiments.

[0058] Figure 18 A flow chart of an algorithm for navigating a vehicle consistent with the disclosed embodiments is provided.

[0059] Figure 19 An example of a host vehicle traveling on a multi-lane highway is shown, consistent with the disclosed embodiments.

[0060] Figure 20A and 20B An example of a vehicle cutting in front of another vehicle is shown, consistent with the disclosed embodiments.

[0061] Figure 21 An example of a vehicle following another vehicle is shown, consistent with the disclosed embodiments.

[0062] Figure 22 An example of a vehicle exiting a parking lot and merging onto a potentially busy road is shown, consistent with the disclosed embodiments.

[0063] Figure 23 A vehicle is shown traveling on a road, consistent with the disclosed embodiments.

[0064] Figure 24 Four example scenarios consistent with the disclosed embodiments are shown.

[0065] Figure 25 An example scenario consistent with the disclosed embodiments is shown.

[0066] Figure 26 An example scenario consistent with the disclosed embodiments is shown.

[0067] Figure 27 An example scenario consistent with the disclosed embodiments is shown.

[0068] Figure 28A and 28B An example of a scenario in which a vehicle is following another vehicle is shown, consistent with the disclosed embodiments.

[0069] Figure 29A and 29BExample attribution of blame in a cut-through scenario consistent with disclosed embodiments is shown.

[0070] Figure 30A and 30B Example attribution in a cut-through scenario consistent with the disclosed embodiments is shown.

[0071] Figures 31A-31D Example attribution in a drift scenario consistent with the disclosed embodiments is shown.

[0072] Figure 32A and 32B Example attribution in a two-way traffic scenario consistent with the disclosed embodiments is shown.

[0073] Figure 33A and 33B Example attribution in a two-way traffic scenario consistent with the disclosed embodiments is shown.

[0074] Figure 34A and 34B Example attribution in a route priority scenario consistent with the disclosed embodiments is shown.

[0075] Figure 35A and 35B Example attribution in a route priority scenario consistent with the disclosed embodiments is shown.

[0076] Figure 36A and 36B Example attribution in a route priority scenario consistent with the disclosed embodiments is shown.

[0077] Figure 37A and 37B Example attribution in a route priority scenario consistent with the disclosed embodiments is shown.

[0078] Figure 38A and 38B Example attribution in a route priority scenario consistent with the disclosed embodiments is shown.

[0079] Figure 39A and 39B Example attribution in a route priority scenario consistent with the disclosed embodiments is shown.

[0080] Figure 40A and 40B Example attribution in a traffic light scenario consistent with disclosed embodiments is shown.

[0081] Figure 41A and 41B Example attribution in a traffic light scenario consistent with disclosed embodiments is shown.

[0082] Figure 42A and 42B Example attribution in a traffic light scenario consistent with disclosed embodiments is shown.

[0083] Figures 43A-43C An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown.

[0084] Figures 44A-44C An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown.

[0085] Figures 45A-45C An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown.

[0086] Figures 46A-46D An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown. DETAILED DESCRIPTION

[0087] The following detailed description refers to the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar parts. Although several exemplary embodiments are described herein, modifications, adjustments, and other implementations are possible. For example, replacements, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by replacing, reordering, removing, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the proper scope is defined by the appended claims.

[0088] Autonomous Vehicle Overview

[0089] As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle that is capable of implementing at least one navigation change without driver input. A "navigation change" refers to one or more of the vehicle's steering, braking, or acceleration / deceleration. By autonomous, it is meant that the vehicle need not be fully automatic (e.g., fully operable without a driver or without driver input). Instead, autonomous vehicles include those that are capable of operating under driver control during certain time periods and are capable of operating without driver control during other time periods. Autonomous vehicles may also include vehicles that only control certain aspects of vehicle navigation, such as steering (e.g., maintaining the vehicle's route between vehicle lane constraints) or certain steering operations in certain circumstances (but not all circumstances), but may leave other aspects to the driver (e.g., braking or braking in certain circumstances). In some cases, an autonomous vehicle may handle some or all aspects of the vehicle's braking, rate control, and / or steering.

[0090] Because human drivers often rely on visual cues and observation to control their vehicles, the transportation infrastructure has been built with lane markings, traffic signs, and traffic lights designed to provide visual information to drivers. Given these design characteristics of the transportation infrastructure, autonomous vehicles can include cameras and processing units that analyze visual information captured from the vehicle's environment. This visual information can include, for example, images representing components of the transportation infrastructure observable by the driver (e.g., lane markings, traffic signs, traffic lights, etc.) as well as other obstacles (e.g., other vehicles, pedestrians, debris, etc.). Autonomous vehicles can also use stored information, such as information that provides a model of the vehicle's environment when navigating. For example, a vehicle can use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information about its environment while the vehicle is traveling, and the vehicle (and other vehicles) can use this information to locate itself on the model. Some vehicles are also capable of communicating with each other, sharing information, and alerting fellow vehicles to hazards or changes in the vehicle's surroundings.

[0091] System Overview

[0092] Figure 1is a block diagram representation of a system 100 consistent with the disclosed example embodiments. Depending on the requirements of a particular implementation, system 100 may include various components. In some embodiments, system 100 may include a processing unit 110, an image acquisition unit 120, a location sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. Processing unit 110 may include one or more processing devices. In some embodiments, processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, image acquisition unit 120 may include any number of image acquisition devices and components, depending on the requirements of a particular application. In some embodiments, image acquisition unit 120 may include one or more image capture devices (e.g., a camera, a charge-coupled device (CCD), or any other type of image sensor), such as image capture device 122, image capture device 124, and image capture device 126. System 100 may also include a data interface 128 that communicatively connects processing unit 110 to image acquisition unit 120. For example, the data interface 128 may include any wired and / or wireless link or links for transmitting image data acquired by the image acquisition unit 120 to the processing unit 110 .

[0093] The wireless transceiver 172 may include one or more devices configured to exchange transmissions over an air interface to one or more networks (e.g., cellular, Internet, etc.) using radio frequencies, infrared frequencies, magnetic fields, or electric fields. The wireless transceiver 172 may use any well-known standard to send and / or receive data (e.g., Wi-Fi, Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmissions may include communications from the host vehicle to one or more remotely located servers. Such transmissions may also include communications (one-way or two-way) between the host vehicle and one or more target vehicles in the host vehicle's environment (e.g., to facilitate coordinating the host vehicle's navigation with or in conjunction with target vehicles in the host vehicle's environment), or even broadcast transmissions to unspecified recipients in the vicinity of the sending vehicle.

[0094] Both the application processor 180 and the image processor 190 may include various types of hardware-based processing devices. For example, either or both of the application processor 180 and the image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), auxiliary circuits, a digital signal processor, an integrated circuit, memory, or any other type of device suitable for running applications and suitable for image processing and analysis. In some embodiments, the application processor 180 and / or the image processor 190 may include any type of single-core or multi-core processor, a mobile device microcontroller, a central processing unit, etc. Various processing devices may be used, including, for example, processors available from, for example, processors from manufacturers such as , and can include various architectures (e.g., x86 processors, wait).

[0095] In some embodiments, the application processor 180 and / or the image processor 190 may include a processor that can be Any of the EyeQ series processor chips available. These processor designs include multiple processing units with local memory and instruction sets. Such processors may include video inputs for receiving image data from multiple image sensors and may also include video output capabilities. In one example, 90 nanometer-micron technology operating at 332 MHz was used. The architecture consists of two floating-point hyperthreaded 32-bit RISC CPUs ( cores), five Visual Computing Engines (VCEs), and three vector microcode processors The MIPS34K CPU manages the five VCEs, three VMPs, and a series of peripherals. TM and DMA, a second MIPS34KCPU and multi-channel DMA and other peripherals. These five VCEs, three and MIPS34K CPU can perform the intensive visual calculations required by multi-function bundled applications. In another example, as a third-generation processor and Six times stronger can be used in the disclosed embodiments. In other examples, can be used in the disclosed embodiments and / or Of course, any newer or future EyeQ processing devices may also be used with the disclosed embodiments.

[0096] Any of the processing devices disclosed herein can be configured to perform certain functions. Configuring a processing device (such as any described EyeQ processor or other controller or microprocessor) to perform certain functions can include programming computer-executable instructions and making these instructions available to the processing device for execution during its operation. In some embodiments, configuring the processing device can include programming the processing device directly with architectural instructions. In other embodiments, configuring the processing device can include storing executable instructions on a memory accessible to the processing device during operation. For example, the processing device can access the memory during operation to obtain and execute the stored instructions. In either case, a processing device configured to perform the sensing, image analysis, and / or navigation functions disclosed herein represents a dedicated hardware-based system that controls multiple hardware-based components of a host vehicle.

[0097] although Figure 1 Two separate processing devices are depicted as being included in processing unit 110, but more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to perform the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Additionally, in some embodiments, system 100 may include one or more processing units 110 without including other components such as image acquisition unit 120.

[0098] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices such as a controller, an image preprocessor, a central processing unit (CPU), auxiliary circuits, a digital signal processor, an integrated circuit, memory, or any other type of device for image processing and analysis. The image preprocessor may include a video processor for capturing, digitizing, and processing images from the image sensor. The CPU may include any number of microcontrollers or microprocessors. The auxiliary circuits may include any number of circuits known in the art, including caches, power supplies, clocks, and input / output circuits. The memory may store software that, when executed by the processor, controls the operation of the system. The memory may include a database and image processing software. The memory may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage, tape storage, removable storage, and other types of storage. In one embodiment, the memory may be separate from the processing unit 110. In another embodiment, the memory may be integrated into the processing unit 110.

[0099] Each memory 140, 150 may include software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. For example, these memory units may include various databases and image processing software, as well as trained systems such as neural networks and deep neural networks. The memory units may include random access memory, read-only memory, flash memory, disk drives, optical storage, tape storage, removable storage, and / or any other type of storage. In some embodiments, the memory units 140, 150 may be separate from the application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into the application processor 180 and / or image processor 190.

[0100] Position sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, position sensor 130 may include a GPS receiver. Such a receiver may determine user location and velocity by processing signals broadcast by global positioning system satellites. Position information from position sensor 130 may be made available to application processor 180 and / or image processor 190.

[0101] In some embodiments, system 100 may include components such as a speed sensor (e.g., a speedometer) for measuring the velocity of vehicle 200. System 100 may also include one or more accelerometers (single-axis or multi-axis) for measuring the acceleration of vehicle 200 along one or more axes.

[0102] The memory units 140, 150 may include a database, or data organized in any other form, indicating the locations of known landmarks. Sensor information of the environment (such as images from lidar or stereo processing of two or more images, radar signals, depth information) may be processed together with position information (such as GPS coordinates, the vehicle's ego motion, etc.) to determine the vehicle's current position relative to known landmarks and to improve the vehicle's position. Certain aspects of this technology are included in what is known as REM. TM The positioning technology is sold by the assignee of this application.

[0103] The user interface 170 may include any device suitable for providing information to or receiving input from one or more users of the system 100. In some embodiments, the user interface 170 may include user input devices including, for example, a touch screen, a microphone, a keyboard, a pointing device, a tracking wheel, a camera, knobs, buttons, etc. Using such input devices, a user can provide information input or commands to the system 100 by typing instructions or information, providing voice commands, selecting menu options on a screen using buttons, a pointer, or eye tracking capabilities, or by any other suitable technique for communicating information to the system 100.

[0104] The user interface 170 may be equipped with one or more processing devices configured to provide and receive information to and from a user, and to process that information for use by, for example, the application processor 180. In some embodiments, such processing devices may execute instructions to recognize and track eye movements, receive and interpret voice commands, recognize and interpret touches and / or gestures made on a touch screen, respond to keyboard input or menu selections, etc. In some embodiments, the user interface 170 may include a display, a speaker, a haptic device, and / or any other device for providing output information to a user.

[0105] Map database 160 may comprise any type of database for storing map data useful to system 100. In some embodiments, map database 160 may include data relating to the locations of various items within a reference coordinate system, including roads, water features, geographic features, commercial areas, points of interest, restaurants, gas stations, and the like. Map database 160 may store not only the locations of these items but also descriptors associated with these items, including, for example, names associated with any stored features. In some embodiments, map database 160 may be physically located with other components of system 100. Alternatively or additionally, map database 160 or a portion thereof may be remotely located relative to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 may be downloaded via a wired or wireless data connection to a network (e.g., via a cellular network and / or the Internet, etc.). In some cases, map database 160 may store a sparse data model comprising a polynomial representation of certain road features (e.g., lane markings) or a target trajectory of the host vehicle. The map database 160 may also include stored representations of various recognized landmarks that may be used to determine or update the known position of the host vehicle relative to the target trajectory. The landmark representation may include data fields such as landmark type, landmark location, and other potential identifiers.

[0106] Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from the environment. Furthermore, any number of image capture devices may be used to acquire images for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four, or more image capture devices. Figures 2B to 2E Image capture devices 122, 124, and 126 are further described.

[0107] One or more cameras (e.g., image capture devices 122, 124, and 126) may be part of a sensing block included on the vehicle. The sensing block may include various other sensors, and any or all of these sensors may be relied upon to form the vehicle's sensed navigation state. In addition to cameras (front, side, rear, etc.), other sensors (such as radar, lidar, and acoustic sensors) may be included in the sensing block. Furthermore, the sensing block may include one or more components configured to transmit and / or receive information related to the vehicle's environment. For example, such components may include wireless transceivers (RF, etc.) that can receive sensor-based information or any other type of information related to the host vehicle's environment from a source remotely located relative to the host vehicle. This information may include sensor output or related information received from vehicle systems external to the host vehicle. In some embodiments, this information may include information received from a remote computing device, a central server, or the like. Furthermore, cameras may be configured in many different ways: a single camera unit, multiple cameras, a camera cluster, with a long field of view (FOV), a short field of view (FOV), a wide angle, a fisheye, and the like.

[0108] The system 100 or its various components may be incorporated into a variety of different platforms. In some embodiments, the system 100 may be included on a vehicle 200, such as Figure 2A For example, the vehicle 200 may be equipped with Figure 1 In some embodiments, the vehicle 200 may be equipped with only a single image capture device (e.g., a camera), while in other embodiments, such as a combination of Figures 2B to 2E For those embodiments discussed, multiple image capture devices may be used. For example, Figure 2A Either of the image capture devices 122 and 124 of the vehicle 200 shown in FIG. 2 may be part of an ADAS (Advanced Driver Assistance System) imaging set.

[0109] The image capture device included on the vehicle 200 as part of the image acquisition unit 120 may be placed in any suitable location. Figures 2A to 2E ,as well as Figures 3A to 3C As shown in , the image capture device 122 can be located near the rearview mirror. This location can provide a line of sight similar to the line of sight of the driver of the vehicle 200, which can assist in determining what is visible and invisible to the driver. The image capture device 122 can be placed anywhere near the rearview mirror, and placing the image capture device 122 on the driver's side of the mirror can also assist in obtaining an image representative of the driver's field of view and / or line of sight.

[0110] Other locations of the image capture device of image acquisition unit 120 can also be used. For example, image capture device 124 can be located on or in the bumper of vehicle 200. This location is particularly suitable for image capture devices with a wide field of view. The line of sight of the image capture device located on the bumper can be different from the driver's line of sight, and therefore, the bumper image capture device and the driver may not always see the same object. Image capture devices (e.g., image capture devices 122, 124, and 126) can also be located in other locations. For example, the image capture device can be located on or in one or both of the sideview mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on the side of vehicle 200, mounted on any window of vehicle 200, placed behind any window of vehicle 200, or placed in front of any window, and in or near the lighting devices installed on the front and / or rear of vehicle 200.

[0111] In addition to the image capture device, the vehicle 200 may also include various other components of the system 100. For example, the processing unit 110 may be included on the vehicle 200, either integrated with or separate from the vehicle's engine control unit (ECU). The vehicle 200 may also be equipped with a location sensor 130, such as a GPS receiver, and may also include a map database 160 and memory units 140 and 150.

[0112] As discussed earlier, the wireless transceiver 172 can transmit and / or receive data over one or more networks (e.g., a cellular network, the Internet, etc.). For example, the wireless transceiver 172 can upload data collected by the system 100 to one or more servers and download data from one or more servers. Via the wireless transceiver 172, the system 100 can receive, for example, periodic or on-demand updates to data stored in the map database 160, the memory 140, and / or the storage 150. Similarly, the wireless transceiver 172 can upload any data from the system 100 (e.g., images captured by the image acquisition unit 120, data received by the position sensor 130 or other sensors, the vehicle control system, etc.) and / or any data processed by the processing unit 110 to one or more servers.

[0113] The system 100 can upload data to a server (e.g., to the cloud) based on privacy level settings. For example, the system 100 can implement privacy level settings to specify or limit the type of data (including metadata) sent to the server that can uniquely identify the vehicle and / or the driver / owner of the vehicle. These settings can be set by the user via, for example, the wireless transceiver 172, can be initialized by factory default settings, or by data received by the wireless transceiver 172.

[0114] In some embodiments, the system 100 may upload data according to a "high" privacy level, and if the setting is set, the system 100 may transmit data (e.g., location information related to the route, captured images, etc.) without any details about a specific vehicle and / or driver / owner. For example, when uploading data according to the "high" privacy setting, the system 100 may not include the vehicle identification number (VIN) or the name of the driver or owner of the vehicle, and may instead transmit data (such as captured images and / or limited location information related to the route).

[0115] Other privacy levels may also be considered. For example, the system 100 may transmit data to the server according to a "medium" privacy level and may include additional information that is not included at a "high" privacy level, such as the make and / or model of the vehicle and / or the type of vehicle (e.g., passenger vehicle, sport utility vehicle, truck, etc.). In some embodiments, the system 100 may upload data according to a "low" privacy level. At the "low" privacy level setting, the system 100 may upload data and include information sufficient to uniquely identify a specific vehicle, owner / driver, and / or portion or entire route traveled by the vehicle. For example, such "low" privacy level data may include one or more of the following: VIN, driver / owner name, point of origin of the vehicle prior to departure, the vehicle's desired destination, make and / or model of the vehicle, vehicle type, etc.

[0116] Figure 2A is a diagrammatic side view representation of an example vehicle imaging system consistent with the disclosed embodiments. Figure 2B yes Figure 2A A diagrammatic top view illustration of the embodiment shown in FIG. Figure 2B As shown, the disclosed embodiments may include a vehicle 200 including a system 100 in its body with a first image capture device 122 located near a rearview mirror of the vehicle 200 and / or near a driver, a second image capture device 124 located on or in a bumper area (e.g., one of the bumper areas 210) of the vehicle 200, and a processing unit 110.

[0117] like Figure 2C As shown, both image capture devices 122 and 124 may be located near the rearview mirror of vehicle 200 and / or near the driver. Figure 2B and Figure 2C Two image capture devices 122 and 124 are shown, it should be understood that other embodiments may include more than two image capture devices. Figure 2D and Figure 2E In the embodiment shown in , a first image capture device 122 , a second image capture device 124 , and a third image capture device 126 are included in the system 100 of a vehicle 200 .

[0118] like Figure 2D As shown, image capture device 122 may be located near a rearview mirror of vehicle 200 and / or near the driver, and image capture devices 124 and 126 may be located on or in a bumper area (e.g., one of bumper areas 210) of vehicle 200. Figure 2EAs shown, image capture devices 122, 124, and 126 can be located near the rearview mirror and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, and the image capture devices can be located in any suitable location within or on vehicle 200.

[0119] It should be understood that the disclosed embodiments are not limited to vehicles and can be applied in other scenarios. It should also be understood that the disclosed embodiments are not limited to a specific type of vehicle 200 and can be applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0120] The first image capture device 122 may include any suitable type of image capture device. The image capture device 122 may include an optical axis. In one example, the image capture device 122 may include an Aptina M9V024WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280×960 pixels and may include a rolling shutter. The image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included, for example, to provide a desired focal length and field of view for the image capture device. In some embodiments, the image capture device 122 may be associated with a 6 mm lens or a 12 mm lens. In some embodiments, as Figure 2D As shown, image capture device 122 can be configured to capture images with a desired field of view (FOV) 202. For example, image capture device 122 can be configured with a conventional FOV, such as in the range of 40 to 56 degrees, including a 46-degree FOV, a 50-degree FOV, a 52-degree FOV, or a larger FOV. Alternatively, image capture device 122 can be configured with a narrow FOV in the range of 23 to 40 degrees, such as a 28-degree FOV or a 36-degree FOV. Furthermore, image capture device 122 can be configured with a wide FOV in the range of 100 to 180 degrees. In some embodiments, image capture device 122 can include a wide-angle bumper camera or a camera with a FOV of up to 180 degrees. In some embodiments, image capture device 122 can be a 7.2 megapixel image capture device with an aspect ratio of approximately 2:1 (e.g., H×V=3800×1900 pixels) and a horizontal FOV of approximately 100 degrees. Such an image capture device can be used in place of a three-image capture device configuration. Due to significant lens distortion, in embodiments where the image capture device uses a radially symmetric lens, the vertical FOV of such an image capture device may be significantly less than 50 degrees. For example, such a lens may not be radially symmetric, which would allow a vertical FOV greater than 50 degrees with a 100 degree horizontal FOV.

[0121] The first image capture device 122 can acquire a plurality of first images of a scene associated with the vehicle 200. Each of the plurality of first images can be acquired as a series of image scan lines, which can be captured using a rolling shutter. Each scan line can include a plurality of pixels.

[0122] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. The scan rate may refer to the scan rate at which the image sensor may acquire image data associated with each pixel included in a particular scan line.

[0123] Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor and a rolling shutter may be employed such that each pixel in a row is read one at a time, and scanning of the rows continues on a row-by-row basis until the entire image frame has been captured. In some embodiments, the rows may be captured sequentially from top to bottom relative to the frame.

[0124] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute a high-resolution imager and may have a resolution greater than 5 M pixels, 7 M pixels, 10 M pixels, or more.

[0125] The use of a rolling shutter may cause pixels in different rows to be exposed and captured at different times, which may cause distortions and other image artifacts in the captured image frame. On the other hand, when image capture device 122 is configured to operate with a global or synchronized shutter, all pixels can be exposed for the same amount of time and during a common exposure period. As a result, the image data in a frame collected from a system employing a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a specific time. In contrast, in a rolling shutter application, each row in the frame is exposed and data is captured at a different time. As a result, moving objects may appear distorted in an image capture device with a rolling shutter. This phenomenon will be described in more detail below.

[0126] Second image capture device 124 and third image capture device 126 can be any type of image capture device. Similar to first image capture device 122, each of image capture devices 124 and 126 can include an optical axis. In one embodiment, each of image capture devices 124 and 126 can include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of image capture devices 124 and 126 can include a rolling shutter. Similar to image capture device 122, image capture devices 124 and 126 can be configured to include various lenses and optical elements. In some embodiments, the lenses associated with image capture devices 124 and 126 can provide a FOV (such as FOVs 204 and 206) that is equal to or narrower than the FOV associated with image capture device 122 (such as FOV 202). For example, image capture devices 124 and 126 can have a FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.

[0127] Image capture devices 124 and 126 can acquire a plurality of second and third images of a scene associated with vehicle 200. Each of the plurality of second and third images can be acquired as a second series of image scan lines and a third series of image scan lines, which can be captured using a rolling shutter. Each scan line or row can have a plurality of pixels. Image capture devices 124 and 126 can have a second scan rate and a third scan rate associated with the acquisition of each image scan line included in the second and third series.

[0128] Each image capture device 122, 124, and 126 can be placed at any suitable location and orientation relative to vehicle 200. The relative positions of image capture devices 122, 124, and 126 can be selected to facilitate fusing information obtained from the image capture devices. For example, in some embodiments, the FOV associated with image capture device 124 (such as FOV 204) may partially or completely overlap with the FOV associated with image capture device 122 (such as FOV 202) and the FOV associated with image capture device 126 (such as FOV 206).

[0129] Image capture devices 122, 124, and 126 may be located at any suitable relative heights on vehicle 200. In one example, there may be height differences between image capture devices 122, 124, and 126 that may provide sufficient parallax information to enable stereo analysis. Figure 2AAs shown, the two image capture devices 122 and 124 are at different heights. For example, there may also be lateral displacement differences between the image capture devices 122, 124, and 126, providing additional parallax information for the stereo analysis of the processing unit 110. Figure 2C and Figure 2D As shown, the difference in lateral displacement can be expressed by d x In some embodiments, there may be a forward or backward displacement (e.g., a range displacement) between image capture devices 122, 124, and 126. For example, image capture device 122 may be positioned 0.5 to 2 meters or more behind image capture device 124 and / or image capture device 126. This type of displacement may enable one of the image capture devices to cover a potential blind spot of the other(s).

[0130] Image capture device 122 may have any suitable resolution capability (e.g., the number of pixels associated with the image sensor), and the resolution of the image sensor(s) associated with image capture device 122 may be higher, lower, or the same as the resolution of the image sensor(s) associated with image capture devices 124 and 126. In some embodiments, the image sensor(s) associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640×480, 1024×768, 1280×960, or any other suitable resolution.

[0131] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data for one image frame and then proceeds to capture pixel data associated with the next image frame) can be controllable. The frame rate associated with image capture device 122 can be higher, lower, or the same as the frame rates associated with image capture devices 124 and 126. The frame rates associated with image capture devices 122, 124, and 126 can depend on various factors that may affect the timing of the frame rates. For example, one or more of image capture devices 122, 124, and 126 can include a selectable pixel delay period that is applied before or after acquiring image data associated with one or more pixels of an image sensor in image capture devices 122, 124, and / or 126. Typically, image data corresponding to each pixel can be acquired based on the clock rate used for the device (e.g., one pixel per clock cycle). Additionally, in embodiments including a rolling shutter, one or more of image capture devices 122, 124, and 126 may include a selectable horizontal blanking period that is applied before or after acquiring image data associated with a row of pixels of an image sensor in image capture devices 122, 124, and / or 126. Additionally, one or more of image capture devices 122, 124, and 126 may include a selectable vertical blanking period that is applied before or after acquiring image data associated with an image frame of image capture devices 122, 124, and 126.

[0132] These timing controls can enable synchronization of the frame rates associated with image capture devices 122, 124, and 126, even if the line scan rate of each is different. Furthermore, as will be discussed in more detail below, these selectable timing controls, along with other factors (e.g., image sensor resolution, maximum line scan rate, etc.), can enable synchronization of image capture from areas where the FOV of image capture device 122 overlaps with one or more of the FOVs of image capture devices 124 and 126, even if the field of view of image capture device 122 is different from the FOVs of image capture devices 124 and 126.

[0133] The frame rate timing in image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensors. For example, assuming similar line scan rates for both devices, if one device includes an image sensor with a resolution of 640×480 and the other device includes an image sensor with a resolution of 1280×960, more time will be required to acquire a frame of image data from the sensor with the higher resolution.

[0134] Another factor that may affect the timing of image data acquisition in image capture devices 122, 124, and 126 is the maximum line scan rate. For example, a certain minimum amount of time will be required to acquire a line of image data from the image sensors included in image capture devices 122, 124, and 126. Assuming no pixel delay period is added, this minimum amount of time for acquiring a line of image data will be related to the maximum line scan rate for the particular device. Devices that offer higher maximum line scan rates have the potential to provide higher frame rates than devices with lower maximum line scan rates. In some embodiments, one or more of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 times, or more, the maximum line scan rate of image capture device 122.

[0135] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate less than or equal to its maximum scan rate. The system may be configured such that one or more of image capture devices 124 and 126 operate at a line scan rate equal to the line scan rate of image capture device 122. In other examples, the system may be configured such that the line scan rate of image capture device 124 and / or image capture device 126 may be 1.25, 1.5, 1.75, or 2 or more times the line scan rate of image capture device 122.

[0136] In some embodiments, image capture devices 122, 124, and 126 may be asymmetric. That is, they may include cameras with different fields of view (FOVs) and focal lengths. For example, the fields of view of image capture devices 122, 124, and 126 may include any desired area of ​​the environment surrounding vehicle 200. In some embodiments, one or more of image capture devices 122, 124, and 126 may be configured to acquire image data from the environment in front of vehicle 200, behind vehicle 200, to the sides of vehicle 200, or a combination thereof.

[0137] Furthermore, the focal length associated with each image capture device 122, 124, and / or 126 can be selectable (e.g., by including an appropriate lens, etc.) so that each device captures images of objects at a desired range of distances relative to the vehicle 200. For example, in some embodiments, the image capture devices 122, 124, and 126 can capture images of objects that are within a few meters of the vehicle. The image capture devices 122, 124, and 126 can also be configured to capture images of objects at greater ranges from the vehicle (e.g., 25 meters, 50 meters, 100 meters, 150 meters, or more). In addition, the focal lengths of image capture devices 122, 124, and 126 may be selected so that one image capture device (e.g., image capture device 122) may capture images of objects that are relatively close to the vehicle (e.g., within 10 meters or within 20 meters), while other image capture devices (e.g., image capture devices 124 and 126) may capture images of objects that are farther away from the vehicle 200 (e.g., greater than 20 meters, 50 meters, 100 meters, 150 meters, etc.).

[0138] According to some embodiments, the FOV of one or more image capture devices 122, 124, and 126 may have a wide angle. For example, having a FOV of 140 degrees may be advantageous, particularly for image capture devices 122, 124, and 126 that may be used to capture images of areas near vehicle 200. For example, image capture device 122 may be used to capture images of areas to the right or left of vehicle 200, and in these embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0139] The field of view associated with each of image capture devices 122, 124, and 126 may depend on the respective focal length. For example, as the focal length increases, the corresponding field of view decreases.

[0140] Image capture devices 122, 124, and 126 can be configured to have any suitable field of view. In one specific example, image capture device 122 can have a horizontal FOV of 46 degrees, image capture device 124 can have a horizontal FOV of 23 degrees, and image capture device 126 can have a horizontal FOV between 23 and 46 degrees. In another example, image capture device 122 can have a horizontal FOV of 52 degrees, image capture device 124 can have a horizontal FOV of 26 degrees, and image capture device 126 can have a horizontal FOV between 26 and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 can vary from 1.5 to 2.0. In other embodiments, the ratio can vary between 1.25 and 2.25.

[0141] System 100 can be configured such that the field of view of image capture device 122 at least partially or completely overlaps the field of view of image capture device 124 and / or image capture device 126. In some embodiments, system 100 can be configured such that the fields of view of image capture devices 124 and 126, for example, fall within (e.g., are narrower than) the field of view of image capture device 122 and share a common center with the field of view of image capture device 122. In other embodiments, image capture devices 122, 124, and 126 can capture adjacent FOVs or can have partial overlap in their FOVs. In some embodiments, the fields of view of image capture devices 122, 124, and 126 can be aligned such that the center of the narrower FOV image capture devices 124 and / or 126 can be located in the lower half of the field of view of the wider FOV device 122.

[0142] Figure 2F is a pictorial representation of an example vehicle control system consistent with the disclosed embodiments. Figure 2F As indicated, the vehicle 200 may include a throttle adjustment system 220, a braking system 230, and a steering system 240. The system 100 may provide input (e.g., control signals) to one or more of the throttle adjustment system 220, the braking system 230, and the steering system 240 via one or more data links (e.g., any wired and / or wireless links for transmitting data). For example, based on analysis of images acquired by the image capture devices 122, 124, and / or 126, the system 100 may provide control signals to one or more of the throttle adjustment system 220, the braking system 230, and the steering system 240 to navigate the vehicle 200 (e.g., by causing acceleration, steering, lane changes, etc.). In addition, the system 100 may receive input from one or more of the throttle adjustment system 220, the braking system 230, and the steering system 240 indicating an operating condition of the vehicle 200 (e.g., speed, whether the vehicle 200 is braking and / or steering, etc.). The following is combined Figures 4 to 7 Provide further details.

[0143] like Figure 3AAs shown, vehicle 200 may also include a user interface 170 for interacting with the driver or passengers of vehicle 200. For example, user interface 170 in a vehicle application may include a touch screen 320, a knob 330, a button 340, and a microphone 350. The driver or passenger of vehicle 200 may also interact with system 100 using handles (e.g., located on or near the steering column of vehicle 200, including, for example, a turn signal handle), buttons (e.g., located on the steering wheel of vehicle 200), and the like. In some embodiments, microphone 350 may be located adjacent to rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, system 100 may provide various notifications (e.g., alarms) via speakers 360.

[0144] Figures 3B to 3D is an illustration of an example camera mount 370 configured to be located behind a rearview mirror (e.g., rearview mirror 310) and opposite a vehicle windshield, consistent with the disclosed embodiments. Figure 3B As shown, camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be located behind a sun visor 380, wherein sun visor 380 may be flush with the vehicle windshield and include a film and / or a composition of anti-reflective materials. For example, sun visor 380 may be positioned so that it is aligned with the vehicle's windshield having a matching bevel. In some embodiments, each of image capture devices 122, 124, and 126 may be located behind sun visor 380, such as at Figure 3D The disclosed embodiments are not limited to any particular configuration of image capture devices 122 , 124 , and 126 , camera mount 370 , and light shield 380 . Figure 3C yes Figure 3B An illustration of camera mount 370 is shown from a front perspective.

[0145] As will be appreciated by those skilled in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the aforementioned disclosed embodiments. For example, not all components are necessary for the operation of system 100. Furthermore, any component may be located in any suitable portion of system 100 and the components may be rearranged into various configurations while still providing the functionality of the disclosed embodiments. Therefore, the aforementioned configurations are exemplary, and regardless of the configurations discussed above, system 100 may provide a wide range of functionality for analyzing the surroundings of vehicle 200 and navigating vehicle 200 in response to that analysis.

[0146] As discussed in greater detail below and in accordance with various disclosed embodiments, system 100 can provide various features related to autonomous driving and / or driver assistance technologies. For example, system 100 can analyze image data, location data (e.g., GPS location information), map data, velocity data, and / or data from sensors included in vehicle 200. System 100 can collect data for analysis from, for example, image acquisition unit 120, location sensor 130, and other sensors. Furthermore, system 100 can analyze the collected data to determine whether vehicle 200 should take a certain action, and then automatically take the determined action without human intervention. For example, while vehicle 200 is navigating without human intervention, system 100 can automatically control braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle control system 220, braking system 230, and steering system 240). Furthermore, system 100 can analyze the collected data and, based on the analysis of the collected data, issue warnings and / or alerts to vehicle occupants. Additional details regarding various embodiments provided by system 100 are provided below.

[0147] Forward multi-imaging system

[0148] As discussed above, system 100 can provide driver assistance functionality using a multi-camera system. A multi-camera system can use one or more cameras facing the front of the vehicle. In other embodiments, the multi-camera system can include one or more cameras facing the sides or rear of the vehicle. In one embodiment, for example, system 100 can use a dual-camera imaging system, where the first and second cameras (e.g., image capture devices 122 and 124) can be located in front of and / or on the sides of a vehicle (e.g., vehicle 200). Other camera configurations are consistent with the disclosed embodiments, and the configurations disclosed herein are exemplary. For example, system 100 can include a configuration with any number of cameras (e.g., one, two, three, four, five, six, seven, eight, etc.). Furthermore, system 100 can include camera "clusters." For example, a camera cluster (including any appropriate number of cameras, such as one, four, eight, etc.) can be forward-facing relative to the vehicle, or can face any other direction (e.g., rearward, sideways, angled, etc.). Thus, system 100 can include multiple camera clusters, each oriented in a specific direction to capture images from a specific area of ​​the vehicle's environment.

[0149] The first camera may have a field of view that is larger than, smaller than, or partially overlaps with the field of view of the second camera. Furthermore, the first camera may be connected to a first image processor to perform monocular image analysis on images provided by the first camera, and the second camera may be connected to a second image processor to perform monocular image analysis on images provided by the second camera. The outputs of the first and second image processors (e.g., processed information) may be combined. In some embodiments, the second image processor may receive images from both the first and second cameras to perform stereo analysis. In another embodiment, system 100 may utilize a three-camera imaging system, each camera having a different field of view. Thus, such a system may make decisions based on information derived from objects located at varying distances in front of and to the sides of the vehicle. References to monocular image analysis may refer to instances where image analysis is performed based on images captured from a single viewpoint (e.g., from a single camera). Stereo image analysis may refer to instances where image analysis is performed based on two or more images captured using one or more variations in image capture parameters. For example, captured images suitable for performing stereo image analysis may include images captured from two or more different positions, from different fields of view, using different focal lengths, with disparity information, and the like.

[0150] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122 through 126. In this configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or another value selected from a range of approximately 20 to 45 degrees), image capture device 124 may provide a wide field of view (e.g., 150 degrees or another value selected from a range of approximately 100 to approximately 180 degrees), and image capture device 126 may provide an intermediate field of view (e.g., 46 degrees or another value selected from a range of approximately 35 to approximately 60 degrees). In some embodiments, image capture device 126 may serve as the primary or base camera. Image capture devices 122 through 126 may be located behind rearview mirror 310 and substantially side-by-side (e.g., 6 centimeters apart). Furthermore, in some embodiments, as discussed above, one or more of image capture devices 122 through 126 may be mounted behind sun visor 380 flush with the windshield of vehicle 200. This shielding may act to reduce the effect of any reflections from the interior of the vehicle on the image capture devices 122 - 126 .

[0151] In another embodiment, as above combined Figure 3B and 3CAs discussed, the wide field of view camera (e.g., image capture device 124 in the above example) can be mounted lower than the narrow field of view camera and the main field of view camera (e.g., image capture devices 122 and 126 in the above example). This configuration can provide a clear line of sight from the wide field of view camera. To reduce reflections, the camera can be mounted closer to the windshield of vehicle 200, and a polarizer can be included on the camera to dampen reflected light.

[0152] A three-camera system can provide certain performance characteristics. For example, some embodiments may include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the three-camera configuration discussed above, the processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips as discussed above), where each processing device is dedicated to processing images captured by one or more of the image capture devices 122 to 126.

[0153] In a three-camera system, a first processing device can receive images from both the main camera and the narrow field of view camera and perform vision processing on the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the first processing device can calculate the disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the vehicle 200's environment. The first processing device can then combine the 3D reconstruction with 3D map data, or with 3D information calculated based on information from another camera.

[0154] The second processing device can receive images from the primary camera and perform visual processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the second processing device can calculate camera displacement and, based on this displacement, calculate the disparity of pixels between consecutive images and create a 3D reconstruction of the scene (e.g., structure from motion). The second processing device can send the structure from motion based on the 3D reconstruction to the first processing device for combination with the stereoscopic 3D image.

[0155] The third processing device can receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device can also execute additional processing instructions to analyze the images to identify moving objects in the images, such as vehicles changing lanes, pedestrians, etc.

[0156] In some embodiments, having streams of image-based information captured and processed independently can provide an opportunity to provide redundancy in the system. Such redundancy can include, for example, using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0157] In some embodiments, system 100 utilizes two image capture devices (e.g., image capture devices 122 and 124) to provide navigation assistance for vehicle 200, and utilizes a third image capture device (e.g., image capture device 126) to provide redundancy and validate the analysis of data received from the other two image capture devices. For example, in this configuration, image capture devices 122 and 124 may provide images for stereo analysis performed by system 100 to navigate vehicle 200, while image capture device 126 may provide images for monocular analysis performed by system 100 to provide redundancy and validation of information obtained based on images captured from image capture devices 122 and / or 124. That is, image capture device 126 (and a corresponding processing device) may be considered to provide a redundant subsystem for providing a check on the analysis obtained from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system). Additionally, in some embodiments, redundancy and verification of the received data may be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers outside the vehicle, etc.).

[0158] Those skilled in the art will recognize that the above-described camera configurations, camera placements, number of cameras, camera positions, etc. are merely examples. These components and other components described with respect to the overall system can be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functionality are as follows.

[0159] Figure 4 1 is an exemplary functional block diagram of memory 140 and / or storage 150 that may be stored / programmed with instructions for performing one or more operations consistent with embodiments of the present disclosure. Although reference is made below to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or storage 150.

[0160] like Figure 4As shown, memory 140 may store a monocular image analysis module 402, a stereo image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 402 to 408 included in memory 140. Those skilled in the art will understand that in the following discussion, references to processing unit 110 may refer individually or collectively to application processor 180 and image processor 190. Thus, any of the following processing steps may be performed by one or more processing devices.

[0161] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform monocular image analysis on a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensory information (e.g., information from radar) to perform the monocular image analysis. 5A to 5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on this analysis, the system 100 (e.g., via the processing unit 110) may cause one or more navigation responses in the vehicle 200, such as steering, lane changes, changes in acceleration, etc., as discussed below in conjunction with the navigation response module 408.

[0162] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform monocular image analysis on a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensory information (e.g., information from radar, lidar, etc.) to perform monocular image analysis. 5A to 5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on this analysis, the system 100 (e.g., via the processing unit 110) may cause one or more navigation responses in the vehicle 200, such as steering, lane changes, changes in acceleration, etc., as discussed below in conjunction with determining navigation responses.

[0163] In one embodiment, the stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform stereo image analysis on a first set of images and a second set of images acquired by a combination of image capture devices selected from any of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the first set of images and the second set of images with additional sensory information (e.g., information from radar) to perform stereo image analysis. For example, the stereo image analysis module 404 may include instructions for performing stereo image analysis based on a first set of images acquired by the image capture device 124 and a second set of images acquired by the image capture device 126. As described below in conjunction with Figure 6 As described, the stereo image analysis module 404 may include instructions for detecting a set of features within the first and second sets of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, etc. Based on this analysis, the processing unit 110 may cause one or more navigation responses in the vehicle 200, such as steering, lane changes, changes in acceleration, etc., as discussed below in conjunction with the navigation response module 408. Furthermore, in some embodiments, the stereo image analysis module 404 may implement techniques associated with a trained system (such as a neural network or a deep neural network) or an untrained system.

[0164] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computing and electromechanical devices in the vehicle 200 configured to cause changes in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target velocity for the vehicle 200 based on data resulting from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include, for example, target location, velocity, and / or acceleration, location and / or velocity of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects, location information of the vehicle 200 relative to lane markings on the road, etc. Furthermore, the processing unit 110 may calculate the target velocity for the vehicle 200 based on sensory input (e.g., information from radar) and input from other systems of the vehicle 200, such as the throttle control system 220, the braking system 230, and / or the steering system 240. Based on the calculated target rate, the processing unit 110 can transmit electronic signals to the throttle regulation system 220, the braking system 230 and / or the steering system 240 of the vehicle 200, for example by physically pressing the brake or releasing the accelerator of the vehicle 200, to trigger a change in speed and / or acceleration.

[0165] In one embodiment, the navigation response module 408 may store software that is executable by the processing unit 110 to determine a desired navigation response based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and velocity information associated with nearby vehicles, pedestrians, and road objects, target position information for the vehicle 200, and the like. Additionally, in some embodiments, the navigation response may be based (in part or in whole) on map data, a predetermined position of the vehicle 200, and / or a relative velocity or acceleration between the vehicle 200 and one or more objects detected from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine the desired navigation response based on sensory input (e.g., information from radar) and input from other systems of the vehicle 200, such as the throttle control system 220, the braking system 230, and the steering system 240 of the vehicle 200. Based on the desired navigation response, the processing unit 110 may transmit electronic signals to the throttle adjustment system 220, the braking system 230, and the steering system 240 of the vehicle 200 to trigger the desired navigation response, such as by turning the steering wheel of the vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, the processing unit 110 may use the output of the navigation response module 408 (e.g., the desired navigation response) as input to the execution of the speed and acceleration module 406 for calculating the change in the velocity of the vehicle 200.

[0166] Furthermore, any modules disclosed herein (e.g., modules 402, 404, and 406) may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems.

[0167] Figure 5A is a flow chart illustrating an example process 500A for causing one or more navigation responses based on monocular image analysis, consistent with the disclosed embodiments. At step 510, the processing unit 110 may receive a plurality of images via the data interface 128 between the processing unit 110 and the image acquisition unit 120. For example, a camera (such as the image capture device 122 having the field of view 202) included in the image acquisition unit 120 may capture a plurality of images of an area in front of the vehicle 200 (e.g., or to the side or rear of the vehicle) and transmit them to the processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). At step 520, the processing unit 110 may execute the monocular image analysis module 402 to analyze the plurality of images, as described below in conjunction with Figures 5B to 5D By performing this analysis, processing unit 110 may detect a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, etc.

[0168] At step 520, the processing unit 110 may also execute the monocular image analysis module 402 to detect various road hazards, such as, for example, parts of a truck tire, fallen road signs, loose cargo, small animals, etc. Road hazards may vary in structure, shape, size, and color, which may make the detection of these hazards more difficult. In some embodiments, the processing unit 110 may execute the monocular image analysis module 402 to perform multi-frame analysis on the multiple images to detect road hazards. For example, the processing unit 110 may estimate the camera motion between consecutive image frames and calculate the disparity in pixels between frames to construct a 3D map of the road. The processing unit 110 may then use the 3D map to detect the road surface and the hazards present on the road surface.

[0169] At step 530, the processing unit 110 may execute the navigation response module 408 to perform the navigation response based on the analysis performed at step 520 and the above combination. Figure 4 The described techniques cause one or more navigation responses. Navigation responses may include, for example, steering, lane changes, acceleration changes, etc. In some embodiments, processing unit 110 may use data obtained from the execution of velocity and acceleration module 406 to cause one or more navigation responses. In addition, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof. For example, processing unit 110 may cause vehicle 200 to change lanes and then accelerate by, for example, sequentially transmitting control signals to steering system 240 and throttle adjustment system 220 of vehicle 200. Alternatively, processing unit 110 may cause vehicle 200 to brake and change lanes simultaneously by, for example, simultaneously transmitting control signals to braking system 230 and steering system 240 of vehicle 200.

[0170] Figure 5B is a flow chart illustrating an example process 500B for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500B. At step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify possible locations within each image that may contain objects of interest (e.g., vehicles, pedestrians, or parts thereof). The predetermined patterns may be designed in such a way as to achieve a high "false hit" rate and a low "miss" rate. For example, processing unit 110 may apply a low similarity threshold to the predetermined patterns to identify candidate objects as possible vehicles or pedestrians. Doing so may allow processing unit 110 to reduce the likelihood of missing (e.g., failing to identify) candidate objects representing vehicles or pedestrians.

[0171] At step 542, processing unit 110 may filter the set of candidate objects based on classification criteria to exclude certain candidates (e.g., irrelevant or less relevant objects). Such criteria may be derived from various attributes associated with object types stored in a database (e.g., a database stored in memory 140). Attributes may include object shape, size, texture, location (e.g., relative to vehicle 200), etc. Thus, processing unit 110 may use one or more sets of criteria to reject false candidates from the set of candidate objects.

[0172] At step 544, processing unit 110 may analyze multiple frames of imagery to determine whether an object in the set of candidate objects represents a vehicle and / or a pedestrian. For example, processing unit 110 may track detected candidate objects across consecutive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). In addition, processing unit 110 may estimate parameters of the detected objects and compare the frame-by-frame position data of the objects with the predicted position.

[0173] In step 546, the processing unit 110 may construct a set of measurements for the detected object. Such measurements may include, for example, position, velocity, and acceleration values ​​(relative to the vehicle 200) associated with the detected object. In some embodiments, the processing unit 110 may construct the measurements based on estimation techniques such as a Kalman filter or linear quadratic estimation (LQE) using a series of time-based observations and / or based on modeling data available for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on a measure of the scale of the object, where the measure of scale is proportional to the time to collision (e.g., the amount of time it takes for the vehicle 200 to reach the object). Thus, by executing steps 540 to 546, the processing unit 110 may identify vehicles and pedestrians that appear within the set of captured images and obtain information associated with the vehicles and pedestrians (e.g., position, velocity, size). Based on this identification and the obtained information, the processing unit 110 may cause one or more navigation responses in the vehicle 200, as described above in conjunction with Figure 5A described.

[0174] At step 548, the processing unit 110 may perform an optical flow analysis on the one or more images to reduce the likelihood of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to, for example, analyzing motion patterns relative to the vehicle 200, associated with other vehicles and pedestrians, and distinct from road motion in one or more images. The processing unit 110 may calculate the motion of the candidate objects by observing the different positions of the objects across multiple image frames captured at different times. The processing unit 110 may use the position and time values ​​as input to a mathematical model for calculating the motion of the candidate objects. Thus, optical flow analysis may provide another method for detecting vehicles and pedestrians near the vehicle 200. The processing unit 110 may perform the optical flow analysis in conjunction with steps 540 to 546 to provide redundancy in detecting vehicles and pedestrians and improve the reliability of the system 100.

[0175] Figure 5C is a flow chart illustrating an example process 500C for detecting road markings and / or lane geometry information in a set of images consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. In order to detect segments of lane markings, lane geometry information, and other relevant road markings, processing unit 110 may filter the set of objects to exclude those determined to be irrelevant (e.g., small potholes, small rocks, etc.). At step 552, processing unit 110 may group together the segments detected in step 550 that belong to the same road marking or lane marking. Based on the grouping, processing unit 110 may generate a model, such as a mathematical model, representing the detected segments.

[0176] In step 554, the processing unit 110 may construct a set of measurements associated with the detected segment. In some embodiments, the processing unit 110 may create a projection of the detected segment from the image plane onto a real-world plane. The projection may be characterized using a cubic polynomial having coefficients corresponding to physical properties such as the position, slope, curvature, and curvature derivatives of the detected road. In generating the projection, the processing unit 110 may take into account variations in the road surface, as well as the pitch and roll rates associated with the vehicle 200. In addition, the processing unit 110 may model the road elevation by analyzing position and motion cues that appear on the road surface. Furthermore, the processing unit 110 may estimate the pitch and roll rates associated with the vehicle 200 by tracking a set of feature points in one or more images.

[0177] At step 556, processing unit 110 may perform a multi-frame analysis by, for example, tracking the detected segments across consecutive image frames and accumulating frame-by-frame data associated with the detected segments. As processing unit 110 performs multi-frame analysis, the set of measurements constructed in step 554 may become more reliable and associated with increasingly higher confidence levels. Thus, by executing steps 550 to 556, processing unit 110 may identify road markings present in the set of captured images and derive lane geometry information. Based on this identification and the derived information, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in conjunction with Figure 5A described.

[0178] At step 558, processing unit 110 may consider additional information sources to further generate a safety model of vehicle 200 in its surrounding environment. Processing unit 110 may use this safety model to define an environment in which system 100 can safely perform autonomous control of vehicle 200. To generate this safety model, in some embodiments, processing unit 110 may consider the position and motion of other vehicles, detected curbs and guardrails, and / or a general road shape description extracted from map data (such as data from map database 160). By considering additional information sources, processing unit 110 can provide redundancy for detecting road markings and lane geometry and increase the reliability of system 100.

[0179] Figure 5D FIG5 is a flow chart illustrating an example process 500D for detecting traffic lights in a set of images consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500D. At step 560, processing unit 110 may scan the set of images and identify objects appearing in the images at locations that may contain traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects, excluding those that are unlikely to correspond to traffic lights. Filtering may be based on various attributes associated with traffic lights, such as shape, size, texture, and position (e.g., relative to vehicle 200). Such attributes may be based on multiple examples of traffic lights and traffic control signals and stored in a database. In some embodiments, processing unit 110 may perform multi-frame analysis on the set of candidate objects that reflect possible traffic lights. For example, processing unit 110 may track candidate objects across consecutive image frames, estimate the real-world positions of the candidate objects, and filter out moving objects (which are unlikely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate objects and identify the relative positions of the detected colors within the possible traffic lights.

[0180] At step 562, processing unit 110 may analyze the geometry of the intersection. This analysis may be based on any combination of: (i) the number of lanes detected on either side of vehicle 200, (ii) detected markings on the road (e.g., arrows), and (iii) a description of the intersection extracted from map data (e.g., data from map database 160). Processing unit 110 may use information derived from execution of monocular analysis module 402 to perform the analysis. Furthermore, processing unit 110 may determine the correspondence between the traffic lights detected in step 560 and the lanes present near vehicle 200.

[0181] At step 564, as the vehicle 200 approaches the intersection, the processing unit 110 may update the confidence level associated with the analyzed intersection geometry and the detected traffic lights. For example, the number of traffic lights estimated to be present at the intersection compared to the number of traffic lights actually present at the intersection may affect the confidence level. Therefore, based on the confidence level, the processing unit 110 may delegate control to the driver of the vehicle 200 in order to improve safety conditions. By executing steps 560 to 564, the processing unit 110 may identify the traffic lights that appear within the set of captured images and analyze the intersection geometry information. Based on this identification and analysis, the processing unit 110 may cause one or more navigation responses in the vehicle 200, as described above in conjunction with Figure 5A described.

[0182] Figure 5E FIG2 is a flow chart illustrating an example process 500E for inducing one or more navigation responses in a vehicle based on a vehicle path, consistent with the disclosed embodiments. At step 570, the processing unit 110 may construct an initial vehicle path associated with the vehicle 200. The vehicle path may be represented by a set of points expressed in coordinates (x, z), and the distance d between two points in the set of points may be i May fall within the range of 1 to 5 meters. In one embodiment, the processing unit 110 may construct an initial vehicle path using two polynomials, such as a left road polynomial and a right road polynomial. The processing unit 110 may calculate the geometric midpoint between the two polynomials and offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset), if any (a zero offset may correspond to driving in the middle of the lane). The offset may be in a direction perpendicular to the line segment between any two points in the vehicle path. In another embodiment, the processing unit 110 may use a polynomial and an estimated lane width to offset each point of the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0183] At step 572, the processing unit 110 may update the vehicle path constructed at step 570. The processing unit 110 may reconstruct the vehicle path constructed at step 570 using a higher resolution so that the distance d between two points in the set of points representing the vehicle path is less than d. k Smaller than the above distance d i For example, the distance d k The processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may produce a cumulative distance vector S corresponding to the total length of the vehicle path (ie, based on the set of points representing the vehicle path).

[0184] At step 574, the processing unit 110 may determine a look-ahead point (expressed in coordinates (x l ,z l )). The processing unit 110 can extract a look-ahead point from the accumulated distance vector S, and the look-ahead point can be associated with a look-ahead distance and a look-ahead time. The look-ahead distance can have a lower limit ranging from 10 meters to 20 meters and can be calculated as the product of the velocity of the vehicle 200 and the look-ahead time. For example, as the velocity of the vehicle 200 decreases, the look-ahead distance can also decrease (e.g., until it reaches the lower limit). The look-ahead time can range from 0.5 to 1.5 seconds and can be inversely proportional to the gain of one or more control loops associated with causing a navigation response in the vehicle 200, such as a heading error tracking control loop. For example, the gain of the heading error tracking control loop can depend on the bandwidth of the yaw rate loop, the steering actuator loop, the lateral dynamics of the vehicle, etc. Therefore, the higher the gain of the heading error tracking control loop, the shorter the look-ahead time.

[0185] At step 576, the processing unit 110 may determine the heading error and the yaw rate command based on the foresight point determined in step 574. The processing unit 110 may calculate the arc tangent of the foresight point, such as arctan(x l ,z l ) to determine the heading error. Processing unit 110 may determine the yaw rate command as the product of the heading error and the high-level control gain. If the look-ahead distance is not at the lower limit, the high-level control gain may be equal to: (2 / look-ahead time). Otherwise, the high-level control gain may be equal to: (2×velocity of vehicle 200 / look-ahead distance).

[0186] Figure 5Fis a flow chart illustrating an example process 500F for determining whether a leading vehicle is changing lanes consistent with the disclosed embodiments. In step 580, the processing unit 110 may determine navigation information associated with a leading vehicle (e.g., a vehicle traveling in front of the vehicle 200). For example, the processing unit 110 may use the above combined Figure 5A and Figure 5B The described techniques can be used to determine the position, velocity (e.g., direction and speed), and / or acceleration of the vehicle ahead. The processing unit 110 can also use the above combined Figure 5E The described techniques determine one or more road polynomials, look-ahead points (associated with vehicle 200 ), and / or snail trails (eg, a set of points describing a path taken by a leading vehicle).

[0187] In step 582, processing unit 110 may analyze the navigation information determined in step 580. In one embodiment, processing unit 110 may calculate the distance between the tracking trajectory and the road polynomial (e.g., along the trajectory). If the variance of this distance along the trajectory exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters on a straight road, 0.3 to 0.4 meters on a moderately curved road, and 0.5 to 0.6 meters on a sharply curved road), processing unit 110 may determine that the leading vehicle is likely changing lanes. In the event that multiple vehicles are detected traveling ahead of vehicle 200, processing unit 110 may compare the tracking trajectory associated with each vehicle. Based on this comparison, processing unit 110 may determine that a vehicle whose tracking trajectory does not match the tracking trajectory of the other vehicles is likely changing lanes. Processing unit 110 may additionally compare the curvature of the tracking trajectory (associated with the leading vehicle) with the expected curvature of the road segment in which the leading vehicle is traveling. The expected curvature may be extracted from map data (e.g., data from map database 160), from a road polynomial, from tracked trajectories of other vehicles, from prior knowledge about the road, etc. If the difference between the curvature of the tracked trajectory and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the leading vehicle is likely changing lanes.

[0188] In another embodiment, the processing unit 110 may compare the instantaneous position of the preceding vehicle with the forward view point (associated with the vehicle 200) over a specific time period (e.g., 0.5 to 1.5 seconds). If the distance between the instantaneous position of the preceding vehicle and the forward view point changes during the specific time period, and the cumulative sum of the changes exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a moderately curved road, and 1.3 to 1.7 meters on a sharp curve), the processing unit 110 may determine that the preceding vehicle is likely changing lanes. In another embodiment, the processing unit 110 may analyze the geometry of the tracking track by comparing the lateral distance traveled along the tracking track with the expected curvature of the tracking track. The expected curvature radius may be determined according to the formula: (δ z 2 +δ x 2 ) / 2 / (δ x ), where δ x represents the lateral distance traveled and δ z The longitudinal distance traveled is represented by the curvature. If the difference between the lateral distance traveled and the expected curvature exceeds a predetermined threshold (e.g., 500 to 700 meters), the processing unit 110 can determine that the leading vehicle is likely to be changing lanes. In another embodiment, the processing unit 110 can analyze the position of the leading vehicle. If the position of the leading vehicle obscures the road polynomial (e.g., the leading vehicle is overlaid on top of the road polynomial), the processing unit 110 can determine that the leading vehicle is likely to be changing lanes. In the event that the position of the leading vehicle is such that another vehicle is detected in front of the leading vehicle and the tracking trajectories of the two vehicles are not parallel, the processing unit 110 can determine that the (closer) leading vehicle is likely to be changing lanes.

[0189] In step 584, the processing unit 110 may determine whether the leading vehicle 200 is changing lanes based on the analysis performed in step 582. For example, the processing unit 110 may make this determination based on a weighted average of the individual analyses performed in step 582. Under such an approach, for example, a determination by the processing unit 110 that the leading vehicle is likely changing lanes based on a particular type of analysis may be assigned a value of "1" (and "0" used to represent a determination that the leading vehicle is unlikely to be changing lanes). The different analyses performed in step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analyses and weights. Furthermore, in some embodiments, the analysis may utilize a trained system (e.g., a machine learning or deep learning system) that may, for example, estimate a future path ahead of the vehicle's current location based on images captured at the current location.

[0190] Figure 6is a flow chart illustrating an example process 600 for inducing one or more navigation responses based on stereo image analysis consistent with the disclosed embodiments. At step 610, processing unit 110 may receive a first and second plurality of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) may capture a first and second plurality of images of an area in front of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first and second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0191] At step 620, the processing unit 110 may execute the stereo image analysis module 404 to perform stereo image analysis on the first and second plurality of images to create a 3D map of the road ahead of the vehicle and detect features within the image, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. The stereo image analysis may be performed in a manner similar to the above combined Figures 5A-5D The steps described above may be performed in the manner described above. For example, processing unit 110 may execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road signs, traffic lights, road hazards, etc.) within the first and second pluralities of images, filter out a subset of candidate objects based on various criteria, and perform multi-frame analysis, construct measurements, and determine confidence levels for the remaining candidate objects. In performing the above steps, processing unit 110 may consider information from both the first and second pluralities of images, rather than from a single set of images. For example, processing unit 110 may analyze differences in pixel-level data (or other subsets of data from the two streams of captured images) for candidate objects appearing in both the first and second pluralities of images. As another example, processing unit 110 may estimate the position and / or velocity of a candidate object (e.g., relative to vehicle 200) by observing whether a candidate object appears in one of the multiple images but not in the other, or other differences that may exist relative to objects appearing in both image streams. For example, the position, velocity, and / or acceleration relative to vehicle 200 may be determined based on features such as trajectory, position, movement characteristics, etc. associated with an object appearing in one or both image streams.

[0192] In step 630, the processing unit 110 may execute the navigation response module 408 to respond based on the analysis performed in step 620 and the above combination. Figure 4The described techniques may be used to cause one or more navigation responses in the vehicle 200. The navigation responses may include, for example, steering, lane changes, changes in acceleration, changes in speed, braking, etc. In some embodiments, the processing unit 110 may use data obtained from the execution of the speed and acceleration module 406 to cause the one or more navigation responses. Furthermore, multiple navigation responses may occur simultaneously, sequentially, or any combination thereof.

[0193] Figure 7 FIG2 is a flow chart illustrating an example process 700 for inducing one or more navigation responses based on analysis of three sets of images, consistent with the disclosed embodiments. In step 710, processing unit 110 may receive first, second, and third pluralities of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture first, second, and third pluralities of images of areas in front of and / or to the sides of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first, second, and third pluralities of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface for transmitting data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0194] At step 720, the processing unit 110 may analyze the first, second, and third plurality of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. This analysis may be performed similarly to the above combined Figures 5A-5D and Figure 6 For example, the processing unit 110 may perform monocular image analysis on each of the first, second, and third plurality of images (e.g., via execution of the monocular image analysis module 402 and based on the above combination). Figures 5A-5D Alternatively, the processing unit 110 may perform stereoscopic image analysis on the first and second pluralities of images, the second and third pluralities of images, and / or the first and third pluralities of images (e.g., via execution of the stereoscopic image analysis module 404 and based on the above combination). Figure 64 and 3 pluralities of images). The processed information corresponding to the analysis of the first, second, and / or third pluralities of images may be combined. In some embodiments, the processing unit 110 may perform a combination of monocular and stereo image analysis. For example, the processing unit 110 may perform monocular image analysis on the first pluralities of images (e.g., via execution by the monocular image analysis module 402) and perform stereo image analysis on the second and third pluralities of images (e.g., via execution by the stereo image analysis module 404). The configuration of the image capture devices 122, 124, and 126—including their respective positions and fields of view 202, 204, and 206—may affect the type of analysis performed on the first, second, and third pluralities of images. The disclosed embodiments are not limited to a particular configuration of the image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third pluralities of images.

[0195] In some embodiments, processing unit 110 may perform testing on system 100 based on the images acquired and analyzed in steps 710 and 720. Such testing may provide an indicator of the overall performance of system 100 for certain configurations of image acquisition devices 122, 124, and 126. For example, processing unit 110 may determine the rate of "false hits" (e.g., instances where system 100 incorrectly determines the presence of a vehicle or pedestrian) and "misses."

[0196] At step 730, the processing unit 110 may cause one or more navigation responses in the vehicle 200 based on information obtained from two of the first, second, and third pluralities of images. The selection of two of the first, second, and third pluralities of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the plurality of images. The processing unit 110 may also make a selection based on image quality and resolution, the effective field of view reflected in the image, the number of frames captured, the degree to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the proportion of the object appearing in each such frame), etc.

[0197] In some embodiments, processing unit 110 may select information derived from two of the first, second, and third pluralities of images by determining how consistent the information derived from one image source is with information derived from the other image sources. For example, processing unit 110 may combine processed information derived from each of image capture devices 122, 124, and 126 (whether through monocular analysis, stereo analysis, or any combination thereof) and determine visual indicators (e.g., lane markings, detected vehicles and their positions and / or paths, detected traffic lights, etc.) that are consistent between each captured image from image capture devices 122, 124, and 126. Processing unit 110 may also exclude information that is inconsistent between the captured images (e.g., vehicles changing lanes, lane models indicating that a vehicle is too close to vehicle 200, etc.). Thus, processing unit 110 may select information derived from two of the first, second, and third pluralities of images based on the determination of consistent and inconsistent information.

[0198] The navigation response may include, for example, steering, lane change, acceleration change, etc. The processing unit 110 may perform the following actions based on the analysis performed in step 720 and the above combination. Figure 4 The described techniques cause one or more navigation responses. Processing unit 110 may also cause one or more navigation responses using data derived from execution of velocity and acceleration module 406. In some embodiments, processing unit 110 may cause one or more navigation responses based on the relative position, relative velocity, and / or relative acceleration between vehicle 200 and an object detected within any of the first, second, and third pluralities of images. Multiple navigation responses may occur simultaneously, sequentially, or any combination thereof.

[0199] Reinforcement learning and trained navigation systems

[0200] The following section discusses autonomous driving and systems and methods for achieving autonomous control of a vehicle, whether that control is fully autonomous (a self-driving vehicle) or partially autonomous (e.g., one or more driver assistance systems or functions). Figure 8As shown, the autonomous driving task can be divided into three main modules, including a sensing module 801, a driving strategy module 803, and a control module 805. In some embodiments, modules 801, 803, and 805 can be stored in the memory unit 140 and / or the memory unit 150 of the system 100, or modules 801, 803, and 805 (or portions thereof) can be stored remotely from the system 100 (e.g., stored in a server accessible to the system 100 via, for example, the wireless transceiver 172). In addition, any module disclosed herein (e.g., modules 801, 803, and 805) can implement techniques associated with a trained system (such as a neural network or a deep neural network) or an untrained system.

[0201] The sensing module 801, which may be implemented using the processing unit 110, can handle various tasks related to sensing the navigational state of the host vehicle's environment. These tasks may rely on input from various sensors and sensing systems associated with the host vehicle. These inputs may include images or image streams from one or more onboard cameras, GPS location information, accelerometer output, user feedback or user input to one or more user interface devices, radar, lidar, and the like. Sensing, which may include data from cameras and / or any other available sensors, as well as map information, can be collected, analyzed, and formulated into a "sensed state," which describes information extracted from the scene in the host vehicle's environment. This sensed state may include sensed information related to target vehicles, lane markings, pedestrians, traffic lights, road geometry, lane shape, obstacles, distance to other objects / vehicles, relative velocity, relative acceleration, and any other potential sensed information. Supervised machine learning may be implemented to generate a sensed state output based on the sensed data provided to the sensing module 801. The output of the sensing module may represent the sensed navigational "state" of the host vehicle, which may be passed to the driving strategy module 803.

[0202] Although the sensed state can be generated based on image data received from one or more cameras or image sensors associated with the host vehicle, any suitable sensor or combination of sensors can also be used to generate the sensed state used in navigation. In some embodiments, the sensed state can be generated without relying on captured image data. In fact, any navigation principles described herein can be applied to sensed states generated based on captured image data as well as sensed states generated using other non-image-based sensors. The sensed state can also be determined via a source external to the host vehicle. For example, the sensed state can be generated in whole or in part based on information received from a source remote from the host vehicle (e.g., based on sensor information, processed state information, etc. shared from other vehicles, shared from a central server, or from any other information source relevant to the navigation state of the host vehicle).

[0203] The driving strategy module 803 (discussed in more detail below and may be implemented using the processing unit 110) may implement a desired driving strategy to determine one or more navigation actions to be taken by the host vehicle in response to the sensed navigation state. If there are no other agents (e.g., target vehicles or pedestrians) in the host vehicle's environment, the sensed state input to the driving strategy module 803 may be processed in a relatively straightforward manner. The task becomes more complex when the sensed state requires negotiation with one or more other agents. Techniques for generating the output of the driving strategy module 803 may include reinforcement learning (discussed in more detail below). The output of the driving strategy module 803 may include at least one navigation action for the host vehicle and may include a desired acceleration (which may be converted into an updated velocity for the host vehicle), a desired yaw rate for the host vehicle, a desired trajectory, and other potential desired navigation actions.

[0204] Based on the output from the driving strategy module 803, a control module 805, which may also be implemented using the processing unit 110, may generate control instructions for one or more actuators or controlled devices associated with the host vehicle. Such actuators and devices may include an accelerator, one or more steering controls, brakes, signal transmitters, displays, or any other actuators or devices that may be controlled as part of navigation operations associated with the host vehicle. Aspects of control theory may be used to generate the outputs of the control module 805. The control module 805 may be responsible for generating and outputting instructions to the controllable components of the host vehicle in order to implement the desired navigation goals or requirements of the driving strategy module 803.

[0205] Returning to the driving policy module 803, in some embodiments, the driving policy module 803 can be implemented using a trained system trained via reinforcement learning. In other embodiments, the driving policy module 803 can be implemented without machine learning methods by using a specified algorithm to "manually" solve various scenarios that may arise during autonomous navigation. However, while feasible, this approach can lead to oversimplified driving policies and may lack the flexibility of a trained system based on machine learning. For example, a trained system may be better equipped to handle complex navigation states and may be better able to determine whether a taxi is stopping or stopping to pick up / drop off a passenger; determine whether a pedestrian intends to cross the street in front of the host vehicle; defensively balance unexpected actions of other drivers; negotiate dense traffic involving target vehicles and / or pedestrians; decide when to suspend certain navigation rules or augment others; anticipate unsensed but anticipated conditions (e.g., whether a pedestrian will emerge from behind a car or obstacle); and so on. A trained system based on reinforcement learning may also be better equipped to solve continuous and high-dimensional state spaces and continuous action spaces.

[0206] Training the system using reinforcement learning can involve learning a driving policy to map from sensed states to navigation actions. The driving policy is a function π: S→A, where S is a set of states and is the action space (e.g., desired velocity, acceleration, yaw command, etc.). The state space is S = S s ×S p , where S s is the sensing state, and S p is additional information about the state saved by the policy. Working in discrete time intervals, at time t, the current state s can be observed t ∈S, and the policy can be applied to obtain the desired action a t =π(s t ).

[0207] The system can be trained by exposing it to various navigation states, causing it to apply the policy, and providing rewards (based on a reward function designed to reward the desired navigation behavior). Based on the reward feedback, the system can "learn" the policy and become trained in producing the desired navigation actions. For example, the learning system can observe the current state s t ∈S, and based on the strategy To decide action a t ∈A. Based on the action decided (and the execution of that action), the environment moves to the next state s t+1∈ S, to be observed by the learning system. For each action taken in response to the observed state, the feedback to the learning system is a reward signal r1, r2, ...

[0208] The goal of reinforcement learning (RL) is to find a policy π. It is usually assumed that at time t, there exists a reward function r t , whose measurement is in state s t and take action a t However, taking action a at time t t affects the environment and, therefore, the value of future states. Therefore, when deciding which action to take, not only the current reward but also future rewards should be considered. In some cases, when the system determines that a higher reward may be achieved in the future if a lower reward option is taken now, then the system should take an action even if that action is associated with a lower reward than another available option. To formalize this, observe that the policy π and the initial state s are summarized in distribution over , where if the agent starts at state s0 = s and follows policy π from there, then the vector (r1, ..., r T ) is the probability of observing rewards r1,…,r T The probability of the initial state s can be defined as:

[0209]

[0210] Instead of limiting the time horizon to T, one can discount future rewards to define, for some fixed γ∈(0,1),

[0211]

[0212] In any case, the best strategy is the solution to the following equation:

[0213]

[0214] Here, the expectation is above the initial state s.

[0215] There are several possible approaches for training a driving policy system. For example, one could use imitation methods (e.g., behavioral cloning), in which the system learns from state / action pairs, where the actions are those that a good actor (e.g., a human) would choose in response to a particular observed state. Suppose that a human driver is observed. From this observation, many forms of (s t ,a t ) (where s t is the state, and a tis the action of the human driver) can be obtained, observed and used as the basis for training a driving policy system. For example, supervised learning can be used to learn a policy π such that π(s t )≈a t This approach has many potential advantages. First, it does not require the definition of a reward function. Second, the learning is supervised and occurs offline (no actors need to be applied during the learning process). A disadvantage of this approach is that different human drivers, or even the same human driver, are not deterministic in their choice of strategy. Therefore, for ||π(s t )-a t Learning very small functions is often infeasible. Moreover, even small errors can accumulate to produce larger errors over time.

[0216] Another technique that can be used is policy-based learning. Here, the policy can be expressed in parameter form and optimized directly using a suitable optimization technique (e.g., stochastic gradient descent). There are many other ways to solve this problem. One advantage of this approach is that it directly solves the problem, and therefore often leads to good practical results. A potential disadvantage is that it often requires "online policy training", that is, the learning of π is an iterative process, where at iteration j, there is a non-perfect policy π j , and in order to construct the next strategy π j , must be based on π j Interact with the environment while performing actions.

[0217] The system can also be trained using value-based learning (learning Q or V functions). Assume that a good approximation can be learned to the optimal value function V*. An optimal policy can be constructed (e.g., relying on the Bellman equation). Some versions of value-based learning can be implemented offline (called "off-policy" training). Some disadvantages of value-based methods may be due to their strong reliance on Markov assumptions and the need to approximate complex functions (approximating the value function can be more difficult than directly approximating the policy).

[0218] Another technique can include model-based learning and planning (learning the probabilities of state transitions and solving the optimization problem of finding the optimal V). A combination of these techniques can also be used to train a learning system. In this approach, the dynamics of the process can be learned, i.e., taking (s t ,a t ) and in the next state s t+1Once this function is learned, the optimization problem can be solved to find the policy π that is optimal. This is called "planning". One advantage of this approach may be that the learning is partially supervised and can be achieved by observing the triples (s t ,a t ,s t+1 ) and is applied offline. Similar to the “imitation” approach, a drawback of this approach may be that small errors in the learning process may accumulate and produce an underperforming policy.

[0219] Another approach for training the driving policy module 803 can include decomposing the driving policy function into semantically meaningful components. This allows for manual implementation of certain parts of the policy, which can ensure policy safety, while other parts can be implemented using reinforcement learning techniques, which can adapt to many scenarios, achieve a human-like balance between defensive / aggressive behavior, and negotiate with other drivers. From a technical perspective, reinforcement learning methods can combine several approaches and provide an easy-to-use training program, where most training can be performed using recorded data or a custom simulator.

[0220] In some embodiments, the training of the driving policy module 803 can rely on an "option" mechanism. To illustrate, consider the simple scenario of a driving policy on a two-lane highway. In a direct RL approach, the policy π maps the state to Where the first component of π(s) is the desired acceleration command and the second component of π(s) is the yaw rate. In the modified approach, the following strategy can be constructed:

[0221] Automatic cruise control (ACC) strategy, O ACC : S→A: This policy always outputs a yaw rate of 0 and only changes the rate to achieve smooth and accident-free driving.

[0222] ACC+left strategy, o L : S → A: The longitudinal commands for this strategy are the same as the ACC commands. The yaw rate is a direct implementation of centering the vehicle toward the middle of the left lane while ensuring safe lateral movement (e.g., not moving left if there is a car on the left).

[0223] ACC+right strategy, o R :S→A:with o L Same, but the vehicle may center toward the center of the right lane.

[0224] These strategies can be called "options". Based on these "options", we can learn the strategy π that selects the option o: S→O, where O is a set of available options. In one case, O={o ACC , o L , o R}. Option selector strategy π o By setting To define the actual policy π: S→A.

[0225] In practice, the policy function can be decomposed into an option graph 901, such as Figure 9 shown. Figure 10 Another example option graph 1000 is shown in FIG. An option graph can represent a hierarchical decision set organized as a directed acyclic graph (DAG). There is a special node called the root node 903 of the graph. This node has no incoming nodes. The decision process starts at the root node and traverses the entire graph until it reaches a "leaf" node, which is a node with no outgoing decision lines. Figure 9 As shown, the leaf nodes may include, for example, nodes 905, 907, and 909. Upon encountering a leaf node, the driving strategy module 803 may output acceleration and steering commands associated with the desired navigation action associated with the leaf node.

[0226] Internal nodes (such as nodes 911, 913, and 915), for example, may cause the implementation of a policy that selects children from among its available options. The set of available children of an internal node includes all nodes that are associated with the particular internal node via a decision line. For example, Figure 9 The internal node 913 designated as "merge" includes three child nodes 909, 915 and 917 ("keep", "pass right" and "pass left"), each connected to the node 913 by a decision line.

[0227] Flexibility in the decision making system can be achieved by enabling nodes to adjust their position in the hierarchy of the options graph. For example, any node can be allowed to declare itself as "critical". Each node can implement an "is critical" function that outputs "true" if the node is in a critical part of its policy implementation. For example, a node responsible for overtaking can declare itself as critical in the middle of a maneuver. This can impose a constraint on the set of available children of a node u, which can include all nodes v that are children of node u, and there is a path from v to a leaf node that passes through all nodes designated as critical. On the one hand, this approach can allow the desired path to be declared on the graph at each time step, while on the other hand, the stability of the policy can be preserved, especially when the critical part of the policy is being implemented.

[0228] By defining an option graph, the problem of learning a driving policy π: S→A can be decomposed into the problem of defining a policy for each node in the graph, where the policy at the internal node should be selected from among the available child nodes. For some nodes, the corresponding policy can be implemented manually (for example, by specifying a set of actions in response to the observed state through an if-then type algorithm), while for other policies a trained system built by reinforcement learning can be used for implementation. The choice between manual or trained / learned approaches can depend on the safety aspects associated with the task and on its relative simplicity. The option graph can be constructed in such a way that some nodes are implemented directly, while other nodes can rely on trained models. This approach can ensure the safe operation of the system.

[0229] The following discussion provides information on how to Figure 9 Further details on the role of the option map in [ 803 ] are provided. As described above, the input to the driving policy module is a "sensed state," which summarizes the environment map, e.g., obtained from available sensors. The output of the driving policy module 803 is a set of expectations (optionally along with a set of hard constraints) that define the trajectory as a solution to the optimization problem.

[0230] As described above, an option graph represents a hierarchical set of decisions organized as a DAG. There is a special node called the "root" of the graph. The root node is the only node with no incoming edges (e.g., decision lines). The decision process traverses the graph starting from the root node until it reaches a "leaf" node, which is a node with no outgoing edges. Each internal node should implement a policy that selects a child from its available children. Each leaf node should implement a policy that defines a set of expectations (e.g., a set of navigation goals for the host vehicle) based on the entire path from root to leaf. This set of expectations, defined directly based on the sensed states, together with a set of hard constraints, establishes an optimization problem whose solution is the vehicle's trajectory. Hard constraints can be employed to further improve the safety of the system, and the expectations can be used to provide the system with driving comfort and human-like driving behavior. The trajectory provided as the solution to the optimization problem, in turn, defines the commands that should be provided to the steering, braking, and / or engine actuators in order to complete the trajectory.

[0231] return Figure 9, option graph 901 represents an option graph for a two-lane highway, including a merge lane (meaning that at some point, a third lane merges into the right or left lane of the highway). Root node 903 first determines whether the host vehicle is in a normal road scenario or an approaching merge scenario. This is an example of a decision that can be made based on the sensed state. Normal road node 911 includes three child nodes: hold node 909, left pass node 917, and right pass node 915. Hold refers to a situation where the host vehicle wants to continue traveling in the same lane. The hold node is a leaf node (no outgoing edges / lines). Therefore, the hold node defines a set of expectations. The first expectation it defines can include a desired lateral position - for example, as close to the center of the current lane as possible. It can also be expected to navigate smoothly (for example, within a predetermined or allowed acceleration maximum). The hold node can also define how the host vehicle reacts to other vehicles. For example, the hold node can view the sensed target vehicles and assign semantic meaning to each vehicle, which can be converted into components of the trajectory.

[0232] Various semantic meanings can be assigned to target vehicles in the host vehicle's environment. For example, in some embodiments, semantic meanings can include any of the following designations: 1) Not relevant: indicates that the sensed vehicle in the scene is currently irrelevant; 2) Next lane: indicates that the sensed vehicle is in an adjacent lane and should maintain an appropriate offset relative to the vehicle (the exact offset can be calculated in an optimization problem that constructs a trajectory given a desired and hard constraints, and the exact offset can potentially be vehicle-dependent—the hold leaf of the option graph sets the semantic type of the target vehicle, which defines the desired relative to the target vehicle); 3) Yield: The host vehicle will attempt to yield to the sensed target vehicle by, for example, reducing speed (particularly if the host vehicle determines that the target vehicle may cut into the host vehicle's lane); 4) Takeway: The host vehicle will attempt to take the right of way by, for example, increasing speed; 5) Follow: The host vehicle desires to follow the target vehicle and maintain a steady course; 6) Left / right pass: This means that the host vehicle intends to initiate a lane change to the left or right lane. Left pass node 917 and left pass node 915 are internal nodes where the desired behavior is not yet defined.

[0233] The next node in option graph 901 is select gap node 919. This node may be responsible for selecting a gap between two target vehicles in a particular target lane that the subject vehicle desires to enter. By selecting a node of the form IDj, for some value of j, the subject vehicle reaches a leaf that specifies a desire for the trajectory optimization problem—for example, the subject vehicle desires to maneuver in order to reach the selected gap. This maneuver may involve first accelerating / braking in the current lane, then moving to the target lane at an appropriate time to enter the selected gap. If select gap node 919 cannot find a suitable gap, it moves to abort node 921, which defines the desire to return to the center of the current lane and cancel the overtaking.

[0234] Returning to the merge node 913, when the host vehicle approaches the merge, it has several options that may depend on the specific situation. Figure 11A As shown, the host vehicle 1105 is traveling along a two-lane road, with no other target vehicles detected in the main lane or merging lane 1111 of the two-lane road. In this case, the driving strategy module 803 may select the hold node 909 upon reaching the merge node 913. That is, it may be desirable to remain in its current lane if no target vehicles are sensed merging onto the road.

[0235] exist Figure 11B Here, the host vehicle 1105 senses one or more target vehicles 1107 entering the main road 1112 from the merging lane 1111. In this case, once the driving strategy module 803 encounters the merging node 913, it may choose to initiate a left passing maneuver to avoid the merging situation.

[0236] exist Figure 11C In Figure 1, host vehicle 1105 encounters one or more target vehicles 1107 entering primary road 1112 from merging lane 1111. Host vehicle 1105 also detects target vehicle 1109 traveling in a lane adjacent to the host vehicle's lane. Host vehicle 1105 also detects one or more target vehicles 1110 traveling in the same lane as host vehicle 1105. In this case, driving policy module 803 may decide to adjust the speed of host vehicle 1105 to yield to target vehicle 1107 and to proceed ahead of target vehicle 1115. This may be achieved, for example, by proceeding to select gap node 919, which in turn selects the gap between ID0 (vehicle 1107) and ID1 (vehicle 1115) as the appropriate merging gap. In this case, the appropriate gap for the merging situation defines the objective of the trajectory planner optimization problem.

[0237] As described above, nodes of the option graph can declare themselves as "critical", which ensures that the selected option passes through critical nodes. Formally, each node can implement an IsCritical function. After performing a forward pass from the root to the leaf on the option graph and solving the trajectory planner's optimization problem, a backward pass can be performed from the leaf back to the root. Along this backward pass, the IsCritical function of all nodes in the pass can be called, and a list of all critical nodes can be saved. In the forward path corresponding to the next time frame, the driving policy module 803 may be required to select a path from the root node to the leaf that passes through all critical nodes.

[0238] Figures 11A to 11C This can be used to illustrate the potential benefits of this approach. For example, when an overtaking maneuver is initiated and the driving strategy module 803 reaches the leaf corresponding to IDk, such as when the host vehicle is in the middle of an overtaking maneuver, it would be undesirable to select the hold node 909. To avoid this jump, the IDj node can designate itself as critical. During the maneuver, the success of the trajectory planner can be monitored, and if the overtaking maneuver proceeds as expected, the function IsCritical will return a "true" value. This approach can ensure that the overtaking maneuver will continue in the next time frame (rather than jumping to another potentially inconsistent maneuver before completing the originally selected maneuver). On the other hand, if the monitoring of the maneuver indicates that the selected maneuver did not proceed as expected, or if the maneuver becomes unnecessary or impossible, the IsCritical function can return a "false" value. This can allow the selection gap node to select a different gap in the next time frame, or to abort the overtaking maneuver entirely. This approach can, on the one hand, allow the desired path to be declared on the option graph at each time step, while on the other hand, it can help improve the stability of the strategy during the critical portion of execution.

[0239] Hard constraints can be different from navigation expectations, which will be discussed in more detail below. For example, hard constraints can ensure safe driving by applying an additional layer of filtering to the planned navigation actions. The hard constraints involved can be determined based on the sensed state, and the hard constraints can be manually programmed and defined, rather than by using a trained system built on reinforcement learning. However, in some embodiments, the trained system can learn the applicable hard constraints to apply and follow. This approach can prompt the driving policy module 803 to arrive at a selected action that already complies with the applicable hard constraints, which can reduce or eliminate the selected action that may need to be modified later to comply with the applicable hard constraints. Nevertheless, as a redundant safety measure, hard constraints can be applied to the output of the driving policy module 803 even if the driving policy module 803 has been trained to cope with predetermined hard constraints.

[0240] There are many potential examples of hard constraints. For example, a hard constraint can be defined in conjunction with a guardrail at the edge of the road. The host vehicle is not allowed to pass the guardrail under any circumstances. Such a rule creates a hard lateral constraint on the trajectory of the host vehicle. Another example of a hard constraint can include a bump in the road (e.g., a rate control bump), which can cause a hard constraint on the driving speed before and while traversing the bump. Hard constraints can be considered safety-critical and therefore can be defined manually rather than relying solely on a trained system that learns the constraints during training.

[0241] In contrast to hard constraints, a desired goal may be to achieve or attain comfortable driving. As discussed above, an example of a desire may include a goal to place the host vehicle within a lane at a lateral position corresponding to the center of the host vehicle's lane. Another desire may include the ID of a gap that is suitable to enter. Note that the host vehicle does not need to be exactly in the center of the lane, but rather, a desire to be as close to it as possible can ensure that the host vehicle tends to migrate to the center of the lane even when drifting off the lane center. The desire may not be safety critical. In some embodiments, the desire may require negotiation with other drivers and pedestrians. One approach to constructing the desire may rely on an option graph, and the policies implemented in at least some nodes of the graph may be based on reinforcement learning.

[0242] For nodes of option graph 901 or 1000 implemented as nodes based on learning training, the training process may include decomposing the problem into a supervised learning phase and a reinforcement learning phase. In the supervised learning phase, it is possible to learn from (s t ,a t )arrive A differentiable map of This can be similar to "model-based" reinforcement learning. However, in the forward loop of the network, s t+1 The actual value is replaced by This eliminates the problem of error accumulation. The role of prediction is to propagate messages from the future back to past actions. In this sense, the algorithm can be a combination of "model-based" reinforcement learning and "policy-based learning."

[0243] An important element that can be provided in some scenarios is a differentiable path from future loss / reward back to the decision on the action. Due to the option graph structure, the implementation of options involving safety constraints is usually not differentiable. To overcome this problem, the selection of children in the learned policy node can be random. That is, the node can output a probability vector p that assigns the probability of selecting each child of a particular node. Suppose the node has k children, and let a (1) ,...,a (k)is the action of the path from each child to the leaf. The resulting predicted action is thus This may lead to a differentiable path from the action to p. In practice, the action a can be chosen to be a for i~p (i) , and a and The difference between them can be called additive noise.

[0244] For a given s t ,a t Down For training, supervised learning can be used with real data. For training of node policies, a simulator can be used. Afterwards, fine-tuning of the policy can be done using real data. Two concepts can make the simulation more realistic. First, using imitation, an initial policy can be built using a "behavioral cloning" paradigm using large real-world datasets. In some cases, the resulting agent can be suitable. In other cases, the resulting agent at least forms a very good initial policy for other agents on the road. Second, using self-play, our own policy can be used to reinforce the training. For example, given an initial implementation of other agents (cars / pedestrians), which can be experienced, the policy can be trained based on a simulator. Some of the other agents can be replaced by the new policy and the process can be repeated. As a result, the policy can continue to improve as it should respond to a wider variety of other agents with different levels of complexity.

[0245] Furthermore, in some embodiments, the system can implement a multi-actor approach. For example, the system can consider data from various sources and / or images captured from multiple angles. Furthermore, some disclosed embodiments can provide energy economy because the anticipation of events that do not directly involve the host vehicle but may have an impact on the host vehicle can be factored in, or even the anticipation of events that may lead to unpredictable situations involving other vehicles can be factored in (e.g., radar may "see through" the vehicle ahead and unavoidably anticipate or even have a high probability of an event that will affect the host vehicle).

[0246] Trained system with imposed navigation constraints

[0247] In the context of autonomous driving, a key concern is ensuring that the learned policy of a trained navigation network is safe. In some embodiments, constraints can be used to train the driving policy system so that the actions selected by the trained system already account for applicable safety constraints. Furthermore, in some embodiments, an additional layer of safety can be provided by passing the selected actions of the trained system through one or more hard constraints related to a specific sensed scenario in the host vehicle's environment. This approach can ensure that the actions taken by the host vehicle are limited to those actions that are confirmed to satisfy the applicable safety constraints.

[0248] At its core, the navigation system may include a learning algorithm based on a policy function that maps observed states to one or more desired actions. In some embodiments, the learning algorithm is a deep learning algorithm. The desired action may include at least one action that is expected to maximize the vehicle's expected reward. While in some cases, the actual action taken by the vehicle may correspond to one of the desired actions, in other cases, the actual action taken may be determined based on the observed state, one or more desired actions, and non-learned hard constraints (e.g., safety constraints) imposed on the learning navigation engine. These constraints may include no-drive zones around various types of detected objects (e.g., target vehicles, pedestrians, stationary objects on or beside the road, moving objects on or beside the road, guardrails, etc.). In some cases, the size of the zone may vary based on the detected motion (e.g., rate and / or direction) of the detected object. Other constraints may include a maximum travel speed when traveling within a pedestrian's influence zone, a maximum deceleration rate (to account for target vehicle spacing behind the host vehicle), mandatory stops at sensed crosswalks or railroad crossings, and the like.

[0249] Hard constraints used in conjunction with systems trained via machine learning can provide a degree of safety in autonomous driving that may exceed the degree of safety achievable based solely on the outputs of the trained system. For example, a machine learning system may be trained using a desired set of constraints as training guidance, and thus, the trained system may select actions in response to a sensed navigation state that address and adhere to the limitations of applicable navigation constraints. However, the trained system still has some flexibility in selecting navigation actions, and thus, at least in some cases, the actions selected by the trained system may not strictly adhere to the relevant navigation constraints. Therefore, in order to require that the selected actions strictly adhere to the relevant navigation constraints, the outputs of the trained system may be combined, compared, filtered, adjusted, modified, etc. using non-machine learning components outside of the learning / trained framework that ensure the strict application of the relevant navigation constraints.

[0250] The following discussion provides additional details about the trained system and the potential benefits (particularly from a safety perspective) gleaned from combining the trained system with algorithmic components outside the trained / learning framework. As previously mentioned, the reinforcement learning objective of the policy can be optimized via stochastic gradient ascent. The objective (e.g., expected reward) can be defined as

[0251] Goals involving expectations can be used in machine learning contexts. However, such goals, when not constrained by navigation constraints, may not return actions that are strictly constrained by those constraints. For example, consider a reward function where represents trajectories of rare “corner” events (e.g., such as accidents) to be avoided, while Denote the remaining trajectories, one goal of the learning system can be to learn to perform overtaking maneuvers. Typically, in accident-free trajectories, will reward successful, smooth overtakes and penalize staying in the lane without completing the overtake - hence the range [-1,1]. represents an accident, then the reward -r should provide a high enough penalty to prevent such an event from happening. The question is what value r should be to ensure accident-free driving.

[0252] Observed that the accident The effect of is an additional term -pr, where p is the probability mass of a trajectory with an accident event. If this term is negligible, i.e., p < < 1 / r, then the learning system can more often prefer a strategy that resulted in an accident (or generally adopts a reckless driving strategy) in order to successfully perform an overtaking maneuver, rather than a more defensive strategy at the expense of some overtaking maneuvers not being successfully completed. In other words, if the probability of an accident is at most p, then r must be set so that r>> 1 / p. It is desirable to make p very small (e.g., at p = 10 -9 of the order of magnitude). Therefore, r should be large. In the policy gradient, it can be estimated The following lemma shows that the random variable The variance of pr 2 As it gets larger, the variance is larger than r for r>>1 / p. Therefore, estimating the target may be difficult, and estimating its gradient may be even more difficult.

[0253] Lemma: Let π o Let p and r be scalars so that we get And with probability 1-p we get So,

[0254]

[0255] Here, the final approximation applies to the case r ≥ 1 / p.

[0256] This discussion shows that the form The goal may not ensure functional safety without causing variance problems. Baseline subtraction methods for reducing variance may not provide an adequate remedy for this problem because the problem will The high variance of is transferred to the equally high variance of the baseline constant, the estimate of which is also subject to numerical instability. Moreover, if the probability of an accident is p, then on average at least 1 / p sequences should be sampled before an accident event is obtained. This means that the The minimum number of samples in the learning algorithm sequence is lower bounded by 1 / p. The solution to this problem can be found in the architectural design described in this paper, rather than through numerical tuning techniques. The approach here is based on the idea that hard constraints should be injected outside the learning framework. In other words, the policy function can be decomposed into a learnable part and a non-learnable part. Formally, the policy function can be constructed as in maps the (unknowable) state space to a set of expectations (e.g., desired navigation goals, etc.), and π (T) Map this expectation to a trajectory (which determines how the car should move over a short distance). Function π (T) Responsible for driving comfort and making strategic decisions, such as which other cars should be overtaken or given way, and what is the desired position of the host vehicle within its lane. The mapping from the sensed navigation state to the desired state is the policy It can learn from experience by maximizing expected rewards. The generated expectation can be converted into a cost function for the driving trajectory. Function π (T) Rather than learning a function, it can be achieved by finding a trajectory that minimizes the cost subject to hard constraints on functional safety. This decomposition can ensure functional safety while providing a comfortable ride.

[0257] like Figure 11D As shown, a double merging navigation scenario provides an example that further illustrates these concepts. In a double merging, vehicles approach the merging area 1130 from both the left and right sides. Also, vehicles from each side (such as vehicle 1133 or vehicle 1135) can decide whether to merge into the lane on the other side of the merging area 1130. Successfully executing a double merging in busy traffic may require significant negotiation skills and experience, and may be difficult to perform with a heuristic or brute force approach by enumerating all possible trajectories that all actors in the scene may take. In this double merging example, a set of desired paths suitable for a double merging maneuver can be defined. Can be the Cartesian product of the following sets:

[0258] Among them, [0,v max ] is the desired target velocity of the host vehicle, L = {1, 1.5, 2, 2.5, 3, 3.5, 4} is the desired lateral position in lanes, where integers represent lane centers and fractions represent lane edges, and {g, t, o} is the classification label assigned to each of the n other vehicles. The other vehicle may be assigned “g” if the host vehicle is to yield to the other vehicle, “t” if the host vehicle is to occupy the lane relative to the other vehicle, or “o” if the host vehicle is to maintain an offset distance relative to the other vehicle.

[0259] The following describes a set of expectations How can it be transformed into a cost function for the driving trajectory? The driving trajectory can be composed of (x1, y1), ..., (x k ,y k ) means, where (x i ,y i ) is the (lateral, longitudinal) position of the subject vehicle at time τ·i (in egocentric units), where in some experiments τ = 0.1 seconds and k = 10. Of course, other values ​​can also be chosen. The cost assigned to the trajectory can include a weighted sum of the costs assigned to the desired velocity, the lateral position, and the label assigned to each of the other n vehicles.

[0260] Given the desired rate v∈[0,v max ], the cost of the trajectory associated with the rate is

[0261]

[0262] Given a desired lateral position l∈L, the cost associated with the desired lateral position is

[0263]

[0264] where dist(x,y,l) is the distance from point (x,y) to lane position l. Regarding the cost due to other vehicles, for any other vehicle, (x′1, y′1), ..., (x′ k , y′ k ) may represent other vehicles in the egocentric unit of the host vehicle, and i may be the earliest point for which there exists j such that (x i ,y i ) and (x' j ,y' j) is small. If there is no such point, then i can be set to i=∞. If the other vehicle is classified as "yield", then it can be expected that τi>τj+0.5, which means that the host vehicle will arrive at the trajectory intersection point at least 0.5 seconds after the other vehicle arrives at the same point. The formula used to convert the above constraints into costs can be [τ(ji)+0.5] + .

[0265] Likewise, if another car is classified as “in lane”, we can expect τj>τi+0.5, which can be translated into a cost [τ(ij)+0.5] + If the other car is classified as “offset”, then we can expect i = ∞, which means that the trajectory of the host vehicle and the trajectory of the offset car do not intersect. This situation can be converted into a cost by penalizing relative to the distance between the trajectories.

[0266] Assigning weights to each of these costs provides a single objective function π for the trajectory planner (T) A cost that encourages smooth driving can be added to this target. Furthermore, to ensure the functional safety of the trajectory, hard constraints can be added to this target. For example, (x i, y i ) leaves the road, and if |i–j| is small, then for any trajectory point (x' j ,y' j ), prohibit (x i, y i ) is close to (x' j ,y' j ).

[0267] In summary, strategy π θ It can be decomposed into a mapping from an unknowable state to a set of expectations and a mapping from expectations to actual trajectories. The latter mapping is not based on learning and can be achieved by solving an optimization problem whose cost depends on the expectations and whose hard constraints can guarantee the functional safety of the policy.

[0268] The following discussion describes the mapping from unknowable states to this set of expectations. As mentioned above, in order to comply with functional safety, a system that relies solely on reinforcement learning may suffer from This result can be avoided by using policy gradients to iteratively decompose the problem into a mapping from an (unknowable) state space to a set of desired trajectories, which is then mapped to actual trajectories without involving a machine learning-based trained system.

[0269] Decision making can be further broken down into semantically meaningful components for various reasons. For example, The size of may be large or even continuous. Figure 11D In the double lane merging scenario described above, Additionally, the gradient estimator may involve the term In such an expression, the variance can grow over the time range T. In some cases, the value of T can be approximately 250, which may be sufficient to produce significant variance. Assuming a sampling rate in the range of 10 Hz and a merging area 1130 of 100 meters, preparation for the merge can begin approximately 300 meters before the merging area. If the host vehicle is traveling at a speed of 16 meters per second (approximately 60 kilometers per hour), the value of T for the episode can be approximately 250.

[0270] Back to the concept of option maps, Figure 11E It is shown that it can be expressed Figure 11D As previously described, an option graph may represent a hierarchical set of decisions organized as a directed acyclic graph (DAG). In the graph, there may be a special node called a "root" node 1140, which may be the only node with no incoming edges (e.g., decision lines). The decision process may traverse the graph starting from the root node until it reaches a "leaf" node, i.e., a node with no outgoing edges. Each internal node may implement a policy function that selects a child from its available children. There may be a set of traversals on the option graph to a set of desired choices. In other words, traversals on the option graph can be automatically converted to Given a node v in the graph, the parameter vector θ v A strategy for selecting the children of v can be specified. If θ is all θ v The concatenation of , then, can be defined by traversing from the root of the graph to the leaves At each node v, the v Defines the strategy for selecting child nodes.

[0271] exist Figure 11E In the double merging option diagram 1139, the root node 1140 may first determine whether the host vehicle is within the merging area (e.g., Figure 11D1130), or whether the host vehicle is approaching a merging area and needs to prepare for a possible merge. In both cases, the host vehicle may need to decide whether to change lanes (e.g., to the left or right) or whether to remain in the current lane. If the host vehicle has decided to change lanes, the host vehicle may need to decide whether conditions are suitable to continue and execute the lane change maneuver (e.g., at a “Continue” node 1142). If a lane change is not possible, the host vehicle may attempt to “advance” toward the desired lane by aiming to be on the lane markings (e.g., at a node 1144 as part of a negotiation with the vehicle in the desired lane). Alternatively, the host vehicle may choose to “remain” in the same lane (e.g., at a node 1146). This process may determine the lateral position of the host vehicle in a natural manner. For example,

[0272] This allows for a natural way to determine the desired lateral position. For example, if the host vehicle changes lanes from lane 2 to lane 3, the "continue" node may set the desired lateral position to 3, the "hold" node may set the desired lateral position to 2, and the "advance" node may set the desired lateral position to 2.5. Next, the host vehicle may decide whether to maintain the "same" rate (node ​​1148), "accelerate" (node ​​1150), or "slow down" (node ​​1152). Next, the host vehicle may enter a "chain-like" structure 1154 of overtaking other vehicles and setting their semantic meaning to values ​​in the set {g, t, o}. This process may set expectations relative to other vehicles. The parameters of all nodes in the chain may be shared (similar to a recurrent neural network).

[0273] A potential benefit of this option is the interpretability of the results. Another potential benefit is the ability to rely on the set The decomposable structure of , and therefore, the policy at each node can be selected from a small number of probabilities. In addition, this structure can allow reducing the variance of the policy gradient estimator.

[0274] As described above, the length of a segment in a double-merge scenario can be approximately T = 250 steps. This value (or any other suitable value depending on the specific navigation scenario) provides sufficient time to see the consequences of the host vehicle's actions (for example, if the host vehicle decides to change lanes in preparation for a merge, the host vehicle will only see the benefits after successfully completing the merge). On the other hand, due to the dynamic nature of driving, the host vehicle must make decisions at a sufficiently high frequency (for example, 10 Hz in the above case).

[0275] The option graph can achieve a lower effective value of T in at least two ways. First, given higher-level decisions, rewards can be defined for lower-level decisions while considering shorter segments. For example, when the host vehicle has selected the "Lane Change" and "Continue" nodes, a policy for assigning semantic meaning to the vehicle can be learned by observing segments of 2 to 3 seconds (meaning T becomes 20-30 instead of 250). Second, for high-level decisions (such as whether to change lanes or stay in the same lane), the host vehicle may not need to make a decision every 0.1 seconds. Alternatively, the host vehicle can make decisions less frequently (e.g., every second) or implement an "option expiration" function, and then calculate the gradient only after each option expiration. In both cases, the effective value of T can be an order of magnitude smaller than its original value. In summary, the estimator for each node can depend on a value of T that is an order of magnitude smaller than the original 250 steps, which may immediately lead to a lower variance.

[0276] As described above, hard constraints can promote safer driving, and there can be several different types of constraints. For example, static hard constraints can be defined directly based on the sensed state. These hard constraints can include speed bumps, rate limits, road bends, intersections, etc. within the environment of the host vehicle, which may involve one or more constraints on vehicle speed, heading, acceleration, braking (deceleration), etc. Static hard constraints may also include semantic free space, where, for example, the host vehicle is prohibited from driving outside the free space and is prohibited from driving too close to physical obstacles. Static hard constraints can also restrict (e.g., prohibit) maneuvers that are inconsistent with various aspects of the vehicle's kinematic motion. For example, static hard constraints can be used to prohibit maneuvers that may cause the host vehicle to flip, slide, or otherwise lose control.

[0277] Hard constraints can also be associated with the vehicle. For example, a constraint can be employed that requires the vehicle to maintain a longitudinal distance of at least one meter to other vehicles and a lateral distance of at least 0.5 meters to other vehicles. Constraints can also be applied such that the host vehicle will avoid maintaining a collision course with one or more other vehicles. For example, time τ can be a time metric based on a particular scenario. The predicted trajectories of the host vehicle and one or more other vehicles can be considered from the current time to time τ. In the event that the two trajectories intersect, Let be the time for vehicle i to arrive at and leave the intersection. That is, each car will arrive at the intersection when the first part of the car passes through it, and it will take a certain amount of time before the last part of the car passes through it. This amount of time separates the arrival time from the departure time. Assume (i.e., the arrival time of vehicle 1 is less than the arrival time of vehicle 2), then we will want to ensure that vehicle 1 has left the intersection before vehicle 2 arrives. Otherwise, a collision will result. Therefore, we can implement Furthermore, to ensure that vehicle 1 and vehicle 2 miss each other by a minimum amount, an additional safety margin can be obtained by including a buffer time (e.g., 0.5 seconds or another appropriate value) in the constraint. The hard constraint related to the predicted intersection trajectory of the two vehicles can be expressed as

[0278] The amount of time τ that the trajectories of the host vehicle and one or more other vehicles are tracked can vary. However, in an intersection scenario, the rate may be lower, τ may be longer, and τ may be defined such that the host vehicle will enter and exit the intersection in less than τ seconds.

[0279] Of course, applying hard constraints to vehicle trajectories requires predicting the trajectories of those vehicles. For the host vehicle, trajectory prediction may be relatively straightforward, as the host vehicle typically already understands and is in fact planning the desired trajectory at any given time. Relative to other vehicles, predicting their trajectories may be less straightforward. For other vehicles, the baseline calculation used to determine the predicted trajectory may rely on the current velocity and heading of the other vehicles, for example, as determined based on analysis of image streams captured by one or more cameras and / or other sensors (radar, lidar, acoustics, etc.) on the host vehicle.

[0280] However, there may be some exceptions that may simplify the problem or at least provide increased confidence in the trajectory predicted for the other vehicle. For example, for structured roads where lane indications are present and where give-way rules may be present, the trajectory of the other vehicle may be based at least in part on the position of the other vehicle relative to the lane and on the applicable give-way rules. Thus, in some cases, when the lane structure is observed, it may be assumed that the vehicle in the next lane will respect the lane boundaries. That is, the host vehicle may assume that the vehicle in the next lane will remain in its lane unless evidence is observed (e.g., a signal light, strong lateral motion, movement across a lane boundary) indicating that the vehicle in the next lane will cut into the host vehicle's lane.

[0281] Other situations can also provide clues about the intended trajectory of other vehicles. For example, at stop signs, traffic lights, roundabouts, and so on, where the host vehicle may have the right of way, it can be assumed that other vehicles will obey that right of way. Therefore, unless evidence of a violation is observed, it can be assumed that other vehicles will continue along a trajectory that obeys the host vehicle's right of way.

[0282] Hard constraints can also be applied to pedestrians in the host vehicle's environment. For example, a buffer distance can be established with respect to pedestrians, prohibiting the host vehicle from traveling closer than a specified buffer distance to any observed pedestrian. The pedestrian buffer distance can be any suitable distance. In some embodiments, the buffer distance can be at least one meter relative to an observed pedestrian.

[0283] Similar to the case with vehicles, hard constraints can also be applied with respect to the relative motion between pedestrians and the host vehicle. For example, the trajectory of a pedestrian (based on heading and velocity) can be monitored relative to the projected trajectory of the host vehicle. Given a specific pedestrian trajectory, where for each point p on the trajectory, t(p) can represent the time required for the pedestrian to reach point p. In order to maintain the required buffer distance of at least 1 meter from the pedestrian, t(p) must be greater than the time the host vehicle will reach point p (with enough time difference so that the host vehicle passes at least one meter in front of the pedestrian), or t(p) must be less than the time the host vehicle will reach point p (for example, if the host vehicle brakes to make way for the pedestrian). Nevertheless, in the latter example, the hard constraint may require the host vehicle to arrive at point p enough time after the pedestrian so that the host vehicle can pass behind the pedestrian and maintain the required buffer distance of at least one meter. Of course, there may be exceptions to the hard constraints for pedestrians. For example, in situations where the host vehicle has the right of way or is moving very slowly, and there is no observed evidence that the pedestrian will refuse to yield to the host vehicle or will otherwise navigate toward the host vehicle, the hard pedestrian constraint can be relaxed (e.g., to a smaller buffer of at least 0.75 meters or 0.50 meters).

[0284] In some examples, if it is determined that not all constraints can be satisfied, the constraints can be relaxed. For example, if the road is too narrow to leave two curbs or the required spacing between the curb and the parked vehicle (e.g., 0.5 meters), if there is a mitigating situation, one or more constraints can be relaxed. For example, if there are no pedestrians (or other objects) on the sidewalk, you can drive slowly at 0.1 meters from the curb. In some embodiments, if relaxing the constraints will improve the user experience, the constraints can be relaxed. For example, to avoid potholes, constraints can be relaxed to allow vehicles to get closer to the edge of the lane, curb, or pedestrians than would normally be allowed. In addition, when determining which constraints to relax, in some embodiments, the one or more constraints selected to be relaxed are those that are considered to have the smallest available negative impact on safety. For example, before relaxing a constraint involving proximity to other vehicles, constraints on how close a vehicle can get to a curb or concrete barrier can be relaxed. In some embodiments, pedestrian constraints may be the last to be relaxed, or in some cases may never be relaxed.

[0285] Figure 12 Examples of scenes that may be captured and analyzed during navigation of a host vehicle are shown. For example, the host vehicle may include a navigation system as described above (e.g., system 100), which may receive a plurality of images representing the host vehicle's environment from a camera associated with the host vehicle (e.g., at least one of image capture device 122, image capture device 124, and image capture device 126). Figure 12The scene shown in is an example of one of the images that may be captured at time t from the environment of a host vehicle traveling in lane 1210 along a predicted trajectory 1212. The navigation system may include at least one processing device (e.g., including any EyeQ processor described above or other device) that is specifically programmed to receive multiple images and analyze the images to determine an action in response to the scene. Specifically, as Figure 8 As shown, at least one processing device may implement a sensing module 801, a driving strategy module 803, and a control module 805. The sensing module 801 may be responsible for collecting and outputting image information collected from a camera, and providing this information in the form of a recognized navigation state to the driving strategy module 803, which may constitute a trained navigation system that has been trained through machine learning techniques (such as supervised learning, reinforcement learning, etc.). Based on the navigation state information provided by the sensing module 801 to the driving strategy module 803, the driving strategy module 803 (e.g., by implementing the option map method described above) may generate a desired navigation action for the host vehicle to perform in response to the recognized navigation state.

[0286] In some embodiments, at least one processing device may convert the desired navigation action directly into a navigation command using, for example, a control module 805. However, in other embodiments, hard constraints may be applied such that the desired navigation action provided by the driving strategy module 803 is tested against various predetermined navigation constraints that may be involved in the scenario and the desired navigation action. For example, where the driving strategy module 803 outputs a desired navigation action that will cause the host vehicle to follow trajectory 1212, the navigation action may be tested against one or more hard constraints associated with various aspects of the host vehicle's environment. For example, the captured image 1201 may show a curb 1213, a pedestrian 1215, a target vehicle 1217, and a stationary object (e.g., an overturned box) present in the scene. Each of these may be associated with one or more hard constraints. For example, the curb 1213 may be associated with a static constraint that prohibits the host vehicle from navigating into or past the curb and onto the sidewalk 1214. The curb 1213 may also be associated with a barrier envelope that defines a distance (e.g., a buffer zone) extending away from and along the curb (e.g., 0.1 m, 0.25 m, 0.5 m, 1 m, etc.) that defines a prohibited navigation zone for the host vehicle. Of course, static constraints may also be associated with other types of roadside boundaries (e.g., guardrails, concrete bollards, traffic cones, bridge pylons, or any other type of roadside obstacle).

[0287] It should be noted that distance and range can be determined by any suitable method. For example, in some embodiments, distance information can be provided by an onboard radar and / or lidar system. Alternatively or additionally, distance information can be derived from an analysis of one or more images captured from the environment of the host vehicle. For example, a number of pixels of an identified object represented in an image can be determined and compared to the known field of view and focal length geometry of the image capture device to determine scale and distance. For example, speed and acceleration can be determined by observing the change in scale between objects from image to image over a known time interval. The analysis can indicate the direction of movement of the object toward or away from the host vehicle, and how fast the object is moving away from or toward the host vehicle. The crossing speed can be determined by analyzing the change in the X-coordinate position of the object from one image to another over a known time period.

[0288] Pedestrian 1215 may be associated with a pedestrian envelope that defines a buffer zone 1216. In some cases, a hard constraint may be imposed that prohibits the host vehicle from navigating within one meter of pedestrian 1215 (in any direction relative to pedestrian 1215). Pedestrian 1215 may also define the location of a pedestrian influence zone 1220. This influence zone may be associated with a constraint that limits the host vehicle's speed within the influence zone. The influence zone may extend from pedestrian 1215 to 5 meters, 10 meters, 20 meters, and so on. Each graduation of the influence zone may be associated with a different speed limit. For example, within an area from one to five meters from pedestrian 1215, the host vehicle may be limited to a first speed (e.g., 10 mph, 20 mph, etc.), which may be less than the speed limit in the pedestrian influence zone extending from 5 to 10 meters. Any graduation may be used for each level of the influence zone. In some embodiments, the first graduation may be narrower than from one to five meters and may extend only from one to two meters. In other embodiments, a first level of influence zone may extend from one meter (the boundary of the no-navigation zone surrounding pedestrians) to a distance of at least 10 meters. A second level may extend from ten meters to at least about twenty meters. The second level may be associated with a maximum travel speed of the host vehicle that is greater than the maximum travel speed associated with the first level of pedestrian influence zone.

[0289] One or more stationary object constraints may also be associated with a scene detected in the host vehicle's environment. For example, in image 1201, at least one processing device may detect a stationary object, such as a box 1219 located in the road. The detected stationary object may include various objects, such as a tree, a pole, a road sign, or at least one of the objects in the road. One or more predetermined navigation constraints may be associated with the detected stationary object. For example, such a constraint may include a stationary object envelope, wherein the stationary object envelope defines a buffer zone about the object within which the host vehicle may be prohibited from navigating. At least a portion of the buffer zone may extend a predetermined distance from the edge of the detected stationary object. For example, in the scene represented by image 1201, a buffer zone of at least 0.1 meters, 0.25 meters, 0.5 meters, or more may be associated with box 1219, such that the host vehicle will pass to the right or left of the box by at least a certain distance (e.g., the buffer zone distance) to avoid colliding with the detected stationary object.

[0290] Predefined hard constraints may also include one or more target vehicle constraints. For example, target vehicle 1217 may be detected in image 1201. To ensure that the host vehicle does not collide with target vehicle 1217, one or more hard constraints may be employed. In some cases, the target vehicle envelope may be associated with a single buffer zone distance. For example, a buffer zone may be defined by a distance of 1 meter around the target vehicle in all directions. The buffer zone may define an area extending at least one meter from the target vehicle, into which the host vehicle is prohibited from navigating.

[0291] However, the envelope around target vehicle 1217 need not be defined by a fixed buffer distance. In some cases, the predetermined hard constraints associated with the target vehicle (or any other movable object detected in the host vehicle's environment) may depend on the orientation of the host vehicle relative to the detected target vehicle. For example, in some cases, the longitudinal buffer distance (e.g., the distance extending from the target vehicle to the front or rear of the host vehicle—such as when the host vehicle is traveling toward the target vehicle) may be at least one meter. The lateral buffer distance (e.g., the distance extending from the target vehicle to either side of the host vehicle—such as when the host vehicle is traveling in the same or opposite direction as the target vehicle so that one side of the host vehicle will pass adjacent to one side of the target vehicle) may be at least 0.5 meters.

[0292] As described above, other constraints may also be involved by detecting target vehicles or pedestrians in the host vehicle's environment. For example, the predicted trajectories of the host vehicle and the target vehicle 1217 may be considered, and in the event that the two trajectories intersect (e.g., at the intersection point 1230), a hard constraint may be required. or The host vehicle is vehicle 1, and the target vehicle 1217 is vehicle 2. Similarly, the trajectory of the pedestrian 1215 (based on heading and velocity) can be monitored relative to the projected trajectory of the host vehicle. Given a particular pedestrian trajectory, for each point p on the trajectory, t(p) will represent the pedestrian's arrival at point p (i.e., Figure 12 In order to maintain the required buffer distance of at least 1 meter from the pedestrian, t(p) must be greater than the time the host vehicle will arrive at point p (with enough time difference so that the host vehicle passes at least one meter in front of the pedestrian), or t(p) must be less than the time the host vehicle will arrive at point p (for example, if the host vehicle brakes to make way for the pedestrian). Nevertheless, in the latter example, the hard constraint will require the host vehicle to arrive at point p enough after the pedestrian so that the host vehicle can pass behind the pedestrian and maintain the required buffer distance of at least one meter.

[0293] Other hard constraints may also be employed. For example, at least in some cases, a maximum deceleration rate for the host vehicle may be employed. This maximum deceleration rate may be determined based on the detected distance to a target vehicle following the host vehicle (e.g., using images collected from a rear-facing camera). Hard constraints may include mandatory stops at sensed crosswalks or railroad crossings, or other applicable constraints.

[0294] In the event that the analysis of the scene in the host vehicle's environment indicates that one or more predetermined navigation constraints may be involved, these constraints may be imposed relative to one or more planned navigation actions of the host vehicle. For example, in the event that the analysis of the scene causes the driving strategy module 803 to return a desired navigation action, the desired navigation action may be tested against one or more involved constraints. If the desired navigation action is determined to violate any aspect of the involved constraints (e.g., if the desired navigation action will drive the host vehicle within 0.7 meters of a pedestrian 1215, where a predetermined hard constraint requires the host vehicle to maintain a distance of at least 1.0 meters from the pedestrian 1215), at least one modification may be made to the desired navigation action based on the one or more predetermined navigation constraints. Adjusting the desired navigation action in this manner can provide an actual navigation action for the host vehicle in accordance with the constraints involved in the particular scene detected in the host vehicle's environment.

[0295] After determining the actual navigational maneuver of the host vehicle, the navigational maneuver may be implemented by causing at least one adjustment of a navigational actuator of the host vehicle in response to the determined actual navigational maneuver of the host vehicle. Such a navigational actuator may include at least one of a steering mechanism, a brake, or an accelerator of the host vehicle.

[0296] Precedence Constraints

[0297] As described above, the navigation system may employ various hard constraints to ensure safe operation of the host vehicle. Constraints may include a minimum safe driving distance relative to pedestrians, target vehicles, roadblocks, or detected objects, a maximum travel speed when passing within the influence zone of a detected pedestrian, or a maximum deceleration rate for the host vehicle, among others. These constraints may be imposed on trained systems based on machine learning (supervised, reinforcement, or a combination thereof), but they may also be useful for untrained systems (e.g., those that employ algorithms to directly process expected situations occurring in scenarios from the host vehicle's environment).

[0298] In either case, there may be a hierarchy of constraints. In other words, some navigation constraints take precedence over others. Therefore, if a situation arises where no navigation action is available that would satisfy all relevant constraints, the navigation system may first determine the available navigation action that implements the highest-priority constraint. For example, the system may cause the vehicle to avoid a pedestrian first, even if navigating to avoid the pedestrian would result in a collision with another vehicle or object detected in the road. In another example, the system may cause the vehicle to ride on a curb to avoid a pedestrian.

[0299] Figure 13 A flow chart illustrating an algorithm for implementing a hierarchy of associated constraints determined based on an analysis of a scene in the host vehicle's environment is provided. For example, at step 1301, at least one processing device associated with a navigation system (e.g., an EyeQ processor, etc.) may receive a plurality of images representing the host vehicle's environment from a camera mounted on the host vehicle. By analyzing the images representing the scene in the host vehicle's environment at step 1303, a navigation state associated with the host vehicle may be identified. For example, the navigation state may indicate that the host vehicle is traveling along a two-lane road 1210, such as Figure 12 12, wherein a target vehicle 1217 is moving through an intersection in front of the host vehicle, a pedestrian 1215 is waiting to cross the path of the host vehicle, an object 1219 exists in front of the host vehicle's lane, and various other attributes of the scene.

[0300] In step 1305, one or more navigation constraints related to the navigation state of the host vehicle may be determined. For example, after analyzing a scene in the environment of the host vehicle represented by one or more captured images, at least one processing device may determine one or more navigation constraints related to objects, vehicles, pedestrians, etc. identified through image analysis of the captured images. In some embodiments, the at least one processing device may determine at least a first predetermined navigation constraint and a second predetermined navigation constraint related to the navigation state, and the first predetermined navigation constraint may be different from the second predetermined navigation constraint. For example, the first navigation constraint may relate to one or more target vehicles detected in the environment of the host vehicle, while the second navigation constraint may relate to pedestrians detected in the environment of the host vehicle.

[0301] In step 1307, at least one processing device may determine the priority associated with the constraints identified in step 1305. In the depicted example, a second predetermined navigation constraint involving pedestrians may have a higher priority than a first predetermined navigation constraint involving a target vehicle. While the priority associated with navigation constraints can be determined or assigned based on a variety of factors, in some embodiments, the priority of navigation constraints may be related to their relative importance from a safety perspective. For example, while it may be important to comply with or satisfy all implemented navigation constraints in as many situations as possible, some constraints may be associated with a higher safety risk than others and, therefore, may be assigned a higher priority. For example, a navigation constraint requiring the host vehicle to maintain a spacing of at least 1 meter from a pedestrian may have a higher priority than a constraint requiring the host vehicle to maintain a spacing of at least 1 meter from a target vehicle. This may be because a collision with a pedestrian can have more severe consequences than a collision with another vehicle. Similarly, maintaining a spacing between the host vehicle and the target vehicle may have a higher priority than a constraint requiring the host vehicle to avoid boxes in the road, drive below a certain speed over speed bumps, or expose the host vehicle occupants to no more than a maximum acceleration level.

[0302] Although the driving strategy module 803 is designed to maximize safety by satisfying the navigation constraints involved in a particular scenario or navigation state, in some cases, it is not practically possible to satisfy each of the involved constraints. In such cases, as shown in step 1309, the priority of each involved constraint can be used to determine which involved constraint should be satisfied first. Continuing with the above example, in the case where it is not possible to satisfy both the pedestrian gap constraint and the target vehicle gap constraint, and only one of the constraints can be satisfied, the higher priority pedestrian gap constraint may cause that constraint to be satisfied before attempting to maintain a gap to the target vehicle. Therefore, under normal circumstances, as shown in step 1311, in the case where both the first predetermined navigation constraint and the second predetermined navigation constraint can be satisfied, at least one processing device can determine a first navigation action for the host vehicle that satisfies both the first predetermined navigation constraint and the second predetermined navigation constraint based on the identified navigation state of the host vehicle. However, in other cases, when not all involved constraints can be satisfied, as shown in step 1313, when the first predetermined navigation constraint and the second predetermined navigation constraint cannot be satisfied at the same time, at least one processing device can determine, based on the identified navigation state, a second navigation action of the main vehicle that satisfies the second predetermined navigation constraint (i.e., a higher priority constraint) but does not satisfy the first predetermined navigation constraint (having a lower priority than the second navigation constraint).

[0303] Next, in order to implement the determined navigational maneuver of the host vehicle, at least one processing device may cause at least one adjustment of a navigation actuator of the host vehicle in response to the determined first navigational maneuver of the host vehicle or the determined second navigational maneuver of the host vehicle at step 1315. As described in the previous example, the navigation actuator may include at least one of a steering mechanism, a brake, or an accelerator.

[0304] Constraint relaxation

[0305] As described above, navigation constraints can be imposed for safety purposes. Constraints can include a minimum safe driving distance relative to a pedestrian, target vehicle, roadblock, or detected object, a maximum driving speed when passing within the influence zone of a detected pedestrian, or a maximum deceleration rate for the host vehicle, among others. These constraints can be imposed on either a learning or non-learning navigation system. In certain circumstances, these constraints can be relaxed. For example, in a scenario where the host vehicle slows down or stops near a pedestrian and then slowly advances to communicate its intention to pass the pedestrian, the pedestrian's response can be detected from the acquired imagery. If the pedestrian responds by remaining still or stopping (and / or if eye contact with the pedestrian is sensed), it can be understood that the pedestrian recognizes the navigation system's intention to pass the pedestrian. In such a scenario, the system can relax one or more of the predetermined constraints and implement less stringent constraints (e.g., allowing the vehicle to navigate within 0.5 meters of the pedestrian, rather than within the more stringent 1-meter boundary).

[0306] Figure 14 A flow chart for implementing control of a host vehicle based on the relaxation of one or more navigation constraints is provided. In step 1401, at least one processing device may receive a plurality of images representing an environment of the host vehicle from a camera associated with the host vehicle. In step 1403, analysis of the images may enable identification of a navigation state associated with the host vehicle. In step 1405, at least one processor may determine a navigation constraint associated with the navigation state of the host vehicle. The navigation constraint may include a first predetermined navigation constraint related to at least one aspect of the navigation state. In step 1407, analysis of the plurality of images may reveal the presence of at least one navigation constraint relaxation factor.

[0307] A navigation constraint relaxation factor may include any suitable indicator that one or more navigation constraints may be suspended, changed, or otherwise relaxed in at least one aspect. In some embodiments, at least one navigation constraint relaxation factor may include a determination (based on image analysis) that the pedestrian's eyes are looking in the direction of the host vehicle. In this case, it can be more safely assumed that the pedestrian is aware of the host vehicle. Therefore, the confidence that the pedestrian will not engage in unexpected maneuvers that result in the pedestrian moving into the path of the host vehicle may be higher. Other constraint relaxation factors may also be used. For example, at least one navigation constraint relaxation factor may include: a pedestrian that is determined to be not moving (e.g., a pedestrian that is assumed to be unlikely to enter the path of the host vehicle); or a pedestrian whose movement is determined to be slowing down. Navigation constraint relaxation factors may also include more complex actions, such as a pedestrian that is determined to be not moving after the host vehicle stops and then resumes moving. In this case, it can be assumed that the pedestrian is aware that the host vehicle has the right of way, and the pedestrian's stopping may indicate the pedestrian's intention to give way to the host vehicle. Other situations that may cause one or more constraints to be relaxed include the type of curb (e.g., a low curb or a curb with a gradual slope may allow a relaxed distance constraint), the lack of pedestrians or other objects on the sidewalk, vehicles with their engines not running that may have relaxed distances, or situations where pedestrians are moving towards and / or away from the area where the host vehicle is traveling.

[0308] In the event that a navigation constraint relaxation factor is identified (e.g., in step 1407), a second navigation constraint may be determined or generated in response to the detection of the constraint relaxation factor. The second navigation constraint may be different from the first navigation constraint, and the second navigation constraint may include at least one characteristic that is relaxed relative to the first navigation constraint. The second navigation constraint may include a newly generated constraint based on the first constraint, wherein the newly generated constraint includes at least one modification that relaxes the first constraint in at least one aspect. Alternatively, the second constraint may constitute a predetermined constraint that is less stringent than the first navigation constraint in at least one aspect. In some embodiments, such a second constraint may be retained for use only in situations where a constraint relaxation factor is identified in the environment of the host vehicle. Regardless of whether the second constraint is newly generated or selected from a set of fully or partially available predetermined constraints, applying the second navigation constraint in place of the more stringent first navigation constraint (which may be applied in situations where no relevant navigation constraint relaxation factor is detected) may be referred to as constraint relaxation and may be completed in step 1409.

[0309] If at least one constraint relaxation factor is detected in step 1407 and at least one constraint has been relaxed in step 1409, a navigation action of the host vehicle may be determined in step 1411. The navigation action of the host vehicle may be based on the identified navigation state and may satisfy the second navigation constraint. The navigation action may be implemented in step 1413 by causing at least one adjustment of a navigation actuator of the host vehicle in response to the determined navigation action.

[0310] As described above, the use of navigation constraints and relaxed navigation constraints can be employed with either trained (e.g., through machine learning) or untrained (e.g., a navigation system that is programmed to respond with a predetermined action in response to a specific navigation state). In the case of a trained navigation system, the availability of relaxed navigation constraints for certain navigation situations can represent a paradigm shift from a trained system response to an untrained system response. For example, a trained navigation network can determine an original navigation action for a host vehicle based on a first navigation constraint. However, the action taken by the vehicle can be a different action from the navigation action that satisfies the first navigation constraint. Instead, the action taken can satisfy a second, more relaxed navigation constraint and can be an action generated by an untrained system (e.g., as a response to detecting a specific condition in the host vehicle's environment, such as the presence of a navigation constraint relaxation factor).

[0311] There are many examples of navigation constraints that can be relaxed in response to detecting a constraint relaxation factor in the host vehicle's environment. For example, where a predetermined navigation constraint includes a buffer zone associated with a detected pedestrian, and at least a portion of the buffer zone extends a certain distance from the detected pedestrian, the relaxed navigation constraint (newly generated, recalled from memory from a predetermined set, or generated as a relaxed version of a pre-existing constraint) can include a different or modified buffer zone. For example, the different or modified buffer zone can have a smaller distance relative to the pedestrian than the original or unmodified buffer zone relative to the detected pedestrian. Thus, where an appropriate constraint relaxation factor is detected in the host vehicle's environment, the host vehicle can be allowed to navigate closer to the detected pedestrian in light of the relaxed constraint.

[0312] As described above, the relaxed characteristic of the navigation constraint may include a reduction in the width of a buffer zone associated with at least one pedestrian. However, the relaxed characteristic may also include a reduction in the width of a buffer zone associated with a target vehicle, a detected object, a roadside obstacle, or any other object detected in the host vehicle's environment.

[0313] At least one relaxed characteristic may also include other types of modifications to navigation constraint characteristics. For example, the relaxed characteristic may include an increase in the velocity associated with at least one predetermined navigation constraint. The relaxed characteristic may also include an increase in the maximum allowable deceleration / acceleration associated with at least one predetermined navigation constraint.

[0314] While constraints may be relaxed in some cases as described above, in other cases navigation constraints may be augmented. For example, in some cases, the navigation system may determine conditions that warrant an augmentation to a set of normal navigation constraints. Such augmentation may include adding new constraints to a set of predetermined constraints or adjusting one or more aspects of the predetermined constraints. The additions or adjustments may result in more conservative navigation relative to the set of predetermined constraints that would be applicable under normal driving conditions. Conditions that may warrant a constraint augmentation may include sensor failure, adverse environmental conditions (rain, snow, fog, or other situations associated with reduced visibility or reduced vehicle traction), etc.

[0315] Figure 15 A flow chart is provided for implementing control of a host vehicle based on augmentation of one or more navigation constraints. In step 1501, at least one processing device may receive a plurality of images representing an environment of the host vehicle from a camera associated with the host vehicle. In step 1503, analysis of the images may enable identification of a navigation state associated with the host vehicle. In step 1505, at least one processor may determine a navigation constraint associated with the navigation state of the host vehicle. The navigation constraint may include a first predetermined navigation constraint related to at least one aspect of the navigation state. In step 1507, analysis of the plurality of images may reveal the presence of at least one navigation constraint augmenting factor.

[0316] The navigation constraints involved may include those mentioned above (e.g., Figure 12) any navigation constraint or any other suitable navigation constraint. A navigation constraint enhancement factor may include any indicator that one or more navigation constraints may be supplemented / enhanced in at least one aspect. Supplementing or enhancing navigation constraints may be performed on a per-group basis (e.g., by adding a new navigation constraint to a predetermined set of constraints) or on a per-constraint basis (e.g., by modifying a particular constraint so that the modified constraint is more restrictive than the original constraint, or by adding a new constraint corresponding to a predetermined constraint, wherein the new constraint is more restrictive in at least one aspect than the corresponding constraint). Additionally or alternatively, supplementing or enhancing navigation constraints may involve selecting from a set of predetermined constraints on a hierarchical basis. For example, a set of enhanced constraints may be selected based on whether a navigation enhancement factor is detected in or relative to the host vehicle's environment. In a normal situation where no enhancement factor is detected, the navigation constraints involved may be drawn from the constraints applicable to the normal situation. On the other hand, in the event that one or more constraint enhancement factors are detected, the constraints involved may be drawn from the enhanced constraints generated or predetermined relative to the one or more enhancement factors. The enhanced constraints may be more restrictive in at least one aspect than the corresponding constraints applicable to the normal situation.

[0317] In some embodiments, at least one navigation constraint enhancing factor may include detecting (e.g., based on image analysis) the presence of ice, snow, or water on a road surface in the host vehicle's environment. For example, such a determination may be based on detecting: areas of higher reflectivity than expected for dry roads (e.g., indicating ice or water on the road); white areas on the road indicating the presence of snow; shadows on the road consistent with the presence of longitudinal grooves on the road (e.g., tire tracks in the snow); water droplets or ice / snow particles on the host vehicle's windshield; or any other suitable indicator of the presence of water or ice / snow on the road surface.

[0318] At least one navigation constraint enhancing factor may also include particles detected on an exterior surface of a windshield of the host vehicle. Such particles may impair image quality of one or more image capture devices associated with the host vehicle. Although described with respect to the windshield of the host vehicle, in relation to a camera mounted behind the windshield of the host vehicle, detection of particles on other surfaces (e.g., a lens or lens cover of a camera, a headlight lens, a rear windshield, a taillight lens, or any other surface of the host vehicle that is visible to (or detected by a sensor of) an image capture device associated with the host vehicle) may also indicate the presence of a navigation constraint enhancing factor.

[0319] A navigation constraint enhancing factor may also be detected as a property of one or more image acquisition devices. For example, a detected degradation in the image quality of one or more images captured by an image capture device (e.g., a camera) associated with the host vehicle may also constitute a navigation constraint enhancing factor. The degradation in image quality may be associated with a hardware failure or partial hardware failure associated with the image capture device or a component associated with the image capture device. Such degradation in image quality may also be caused by environmental conditions. For example, the presence of smoke, fog, rain, snow, etc. in the air surrounding the host vehicle may also result in reduced image quality relative to roads, pedestrians, target vehicles, etc. that may be present in the environment of the host vehicle.

[0320] Navigation constraint enhancing factors may also relate to other aspects of the host vehicle. For example, in some cases, a navigation constraint enhancing factor may include a detected failure or partial failure of a system or sensor associated with the host vehicle. Such enhancing factors may include, for example, a detected failure or partial failure of a speed sensor, GPS receiver, accelerometer, camera, radar, lidar, brakes, tires, or any other system associated with the host vehicle that may affect the host vehicle's ability to navigate relative to the navigation constraints associated with the host vehicle's navigational state.

[0321] In the event that the presence of a navigation constraint enhancing factor is identified (e.g., in step 1507), a second navigation constraint may be determined or generated in response to the detection of the constraint enhancing factor. The second navigation constraint may be different from the first navigation constraint, and the second navigation constraint may include at least one characteristic that is enhanced relative to the first navigation constraint. The second navigation constraint may be more restrictive than the first navigation constraint because the detection of the constraint enhancing factor in the environment of the host vehicle or associated with the host vehicle may indicate that the host vehicle may have at least one reduced navigation capability relative to normal operating conditions. Such reduced capability may include lower road traction (e.g., ice, snow, or water on the road; reduced tire pressure, etc.); impaired vision (e.g., rain, snow, dust, smoke, fog, etc. that reduces the quality of captured images); impaired detection capability (e.g., sensor failure or partial failure, reduced sensor performance, etc.), or any other reduction in the ability of the host vehicle to navigate in response to the detected navigation condition.

[0322] If at least one constraint-enhancing factor is detected in step 1507 and at least one constraint is enforced in step 1509, a navigation action for the host vehicle may be determined in step 1511. The navigation action for the host vehicle may be based on the identified navigation state and may satisfy the second navigation (i.e., enforced) constraint. The navigation action may be implemented in step 1513 by causing at least one adjustment to be made to a navigation actuator of the host vehicle in response to the determined navigation action.

[0323] As discussed, the use of navigation constraints and augmented navigation constraints can be employed with either trained (e.g., through machine learning) or untrained (e.g., systems programmed to respond with predetermined actions in response to specific navigation conditions) navigation systems. In the case of using a trained navigation system, the availability of augmented navigation constraints for certain navigation situations can represent a mode switch from a trained system response to an untrained system response. For example, a trained navigation network can determine an original navigation action for a host vehicle based on a first navigation constraint. However, the action taken by the vehicle can be a different action from the navigation action that satisfies the first navigation constraint. Instead, the action taken can satisfy a second augmented navigation constraint and can be an action generated by an untrained system (e.g., as a response to detecting a specific condition in the host vehicle's environment, such as the presence of a navigation constraint augmenting factor).

[0324] There are many examples of navigation constraints that can be generated, supplemented, or enhanced in response to the detection of a constraint-enhancing factor in the host vehicle's environment. For example, where a predetermined navigation constraint includes a buffer zone associated with a detected pedestrian, object, vehicle, etc., and at least a portion of the buffer zone extends a certain distance from the detected pedestrian / object / vehicle, the enhanced navigation constraint (newly generated, recalled from memory from a predetermined set, or generated as an enhanced version of a pre-existing constraint) can include a different or modified buffer zone. For example, the different or modified buffer zone can have a greater distance relative to the detected pedestrian / object / vehicle than the original or unmodified buffer zone relative to the detected pedestrian / object / vehicle. Thus, where an appropriate constraint-enhancing factor is detected in the host vehicle's environment or relative to the host vehicle, the host vehicle can be forced to navigate further away from the detected pedestrian / object / vehicle in light of the enhanced constraint.

[0325] At least one enhanced characteristic may also include other types of modifications to navigation constraint characteristics. For example, the enhanced characteristic may include a reduction in speed associated with at least one predetermined navigation constraint. The enhanced characteristic may also include a reduction in the maximum allowable deceleration / acceleration associated with at least one predetermined navigation constraint.

[0326] Navigation based on long-term planning

[0327] In some embodiments, the disclosed navigation system can not only respond to a navigation state detected in the host vehicle's environment, but can also determine one or more navigation actions based on long-range planning. For example, the system can consider the potential impact of one or more navigation actions available as options for navigating relative to the detected navigation state on future navigation states. Considering the impact of available actions on future states allows the navigation system to determine navigation actions based not only on the currently detected navigation state, but also on long-range planning. Navigation using long-range planning techniques may be particularly useful in situations where the navigation system employs one or more reward functions as a technique for selecting navigation actions from available options. Potential rewards can be analyzed relative to available navigation actions that can be taken in response to the detected current navigation state of the host vehicle. However, further, potential rewards can also be analyzed relative to actions that can be taken in response to future navigation states that are expected to result from available actions in response to the current navigation state. Thus, the disclosed navigation system can, in some cases, select a navigation action from available actions that can be taken in response to the detected navigation state, even when the selected navigation action may not yield the highest reward. This is particularly true when the system determines that a selected action may result in a future navigation state that gives rise to one or more potential navigation actions that offer a higher reward than the selected action or, in some cases, than any action available relative to the current navigation state. This principle can be more simply expressed as taking a less favorable action now in order to yield a higher reward option in the future. Thus, the disclosed navigation system capable of long-term planning can select short-term suboptimal actions where the long-term prediction indicates that a short-term loss of reward will result in a long-term increase in reward.

[0328] Typically, autonomous driving applications may involve a series of planning problems in which the navigation system may decide on immediate actions in order to optimize a longer-term goal. For example, when a vehicle encounters a merging situation at a roundabout, the navigation system may decide on immediate acceleration or braking commands in order to initiate navigation to the roundabout. Although the immediate action on the detected navigation state at the roundabout may involve acceleration or braking commands in response to the detected state, the long-term goal is to successfully merge, and the long-term impact of the selected command is the success / failure of the merge. The planning problem can be solved by breaking the problem into two stages. First, supervised learning can be applied to predict the near future based on the present (assuming that the representation of the predictor with respect to the present will be differentiable). Second, a recurrent neural network can be used to model the complete trajectory of the actor, where unexplained factors are modeled as (additive) input nodes. This can allow the use of supervised learning techniques and direct optimization of the recurrent neural network to determine the solution to the long-term planning problem. This approach can also enable robust policies to be learned by incorporating adversarial elements into the environment.

[0329] The two most fundamental elements of autonomous driving systems are sensing and planning. Sensing deals with finding a compact representation of the current state of the environment, while planning involves deciding which actions to take to optimize future goals. Supervised machine learning techniques are useful for solving sensing problems. Machine learning algorithmic frameworks can also be used for planning, particularly reinforcement learning (RL) frameworks such as those mentioned above.

[0330] RL can be performed in a sequence of consecutive rounds. In round t, the planner (also known as the actor or driving policy module 803) can observe the state s t ∈S, which represents the actor and the environment. Then it should decide the action a t ∈A. After performing the action, the actor receives an immediate reward and move to the new state s t+1 As an example, the host vehicle may include an adaptive cruise control (ACC) system, where the vehicle should autonomously implement acceleration / braking to maintain a proper distance from the vehicle ahead while maintaining a smooth drive. This state can be modeled as a pair of where x t is the distance to the vehicle ahead, v t is the speed of the host vehicle relative to the speed of the preceding vehicle. Will be an acceleration command (if a t <0 if the host vehicle decelerates). The reward can be determined by |a t | (reflects driving stability) and s t(reflecting the function of maintaining a safe distance between the host vehicle and the preceding vehicle). The goal of the planner is to maximize the cumulative reward (which can be the time horizon of future rewards or the discounted sum). To do this, the planner can rely on a policy π: S→A, which maps states to actions.

[0331] Supervised learning (SL) can be viewed as a special case of RL, where s t is sampled from some distribution on S, and the reward function can have in the form of is the loss function, and the learner observes y t The value of y t To view the status t There are several differences between general RL models and special case SL, and these differences make the general RL problem more challenging.

[0332] In some SL situations, the actions (or predictions) taken by the learner may have no effect on the environment. In other words, s t+1 and a t are independent. This has two important implications. First, in SL, samples (s1, y1), ..., (s m ,y m ) can be collected in advance before starting to search for a policy (or predictor) with good accuracy relative to the sample. In contrast, in RL, the state s t+1 Typically depends on the action taken (and the previous state), which in turn depends on the policy used to generate that action. This will make the data generation process bound to the policy learning process. Second, because in SL actions do not affect the environment, choosing a t The contribution to the performance of π is local. Specifically, a t Only affects the value of the immediate reward. In contrast, in RL, the action taken in round t may have a long-term impact on the reward value in future rounds.

[0333] In SL, knowledge of the “correct” answer t , together with rewards The shape together can provide for a t Full knowledge of the rewards of all possible choices of , which can enable the calculation of rewards relative to a tIn contrast, in RL, the "one-time" value of the reward may be all that can be observed for a specific choice of action taken. This can be called "bandit" feedback. This is one of the most important reasons why "exploration" is needed as part of long-term navigation planning, because in RL-based systems, if only "bandit" feedback is available, the system may not always know whether the action taken is the best action to take.

[0334] Many RL algorithms rely at least in part on a mathematical optimization model called the Markov Decision Process (MDP). The Markov assumption is that given s t and a t , s t+1 The distribution of is completely determined. In terms of the stable distribution over the states of the MDP, it gives a closed-form expression for the cumulative reward of a given policy. The stable distribution of policies can be expressed as the solution of a linear programming problem. This gives rise to two classes of algorithms: 1) optimization with respect to the original problem, which is called policy search; 2) optimization with respect to the dual problem, whose variable is called the value function V π If the MDP starts from an initial state s and from there actions are chosen according to π, then the value function determines the expected cumulative reward. The relevant quantity is the state-action value function Q π (s, a), given that we start in state s, choose action a at that instant, and from there choose actions according to π, the state-action value function determines the cumulative reward. The Q function may lead to a characterization of the optimal policy (using the Bellman equation). In particular, the Q function may indicate that the optimal policy is a deterministic function from S to A (in fact, it may be characterized as a "greedy" policy with respect to the optimal Q function).

[0335] One potential advantage of the MDP model is that it allows the future to be coupled to the present using a Q function. For example, suppose the host vehicle is now in state s, Q π The value of (s, a) can indicate the impact of performing action a on the future. Therefore, the Q function can provide a local measure of the quality of action a, making the RL problem more similar to the SL scenario.

[0336] Many RL algorithms approximate the V-function or Q-function in some way. Value iteration algorithms (e.g., Q-learning) can rely on the fact that the V- and Q-functions of the optimal policy can be fixed points of some operators derived from the Bellman equation. Actor-critic policy iteration algorithms aim to learn a policy in an iterative manner, where at iteration t, the “critic” estimates And based on that estimate, the “actor” refines the strategy.

[0337] Despite the mathematical advantages of MDPs and the ease of switching to a Q-function representation, the approach may have several limitations. For example, an approximate notion of a Markovian behavior state may be all that can be found in some cases. Furthermore, transitions between states depend not only on the behavior of the actor, but also on the behavior of other actors in the environment. For example, in the ACC example mentioned above, although the dynamics of the autonomous vehicle may be Markovian, the next state may depend on the behavior of the driver of another vehicle, which is not necessarily Markovian. One possible solution to this problem is to use a partially observed MDP, where it is assumed that a Markovian state exists, but that what can be observed are observations distributed according to the hidden state.

[0338] A more direct approach might consider a game-theoretic generalization of MDPs (e.g., a stochastic game framework). Indeed, algorithms for MDPs can be generalized to multi-actor games (e.g., min-max-Q learning or Nash-Q learning). Other approaches might include explicit modeling of other players and vanishing regret learning algorithms. Learning in a multi-actor setting can be more complex than in a single-actor setting.

[0339] A second limitation of the Q-function representation can arise by deviating from the tabular setting. The tabular setting is when the number of states and actions is small, so Q can be represented as a table with |S| rows and |A| columns. However, if the natural representation of S and A involves Euclidean space, and the state and action spaces are discrete, the number of states / actions may be exponential in dimension. In this case, it may not be practical to adopt a tabular setting. Instead, the Q-function can be approximated by a function from a class of parameter hypotheses (e.g., a neural network of a certain architecture). For example, a deep-Q-network (DQN) learning algorithm can be used. In a DQN, the state space can be continuous, while the action space can still be a small discrete set. There may be ways to handle continuous action spaces, but they may rely on approximating the Q-function. In any case, the Q-function may be complex and sensitive to noise, and therefore may pose challenges to learning.

[0340] A different approach could be to use recurrent neural networks (RNNs) to solve RL problems. In some cases, RNNs can be combined with concepts from multi-actor games and robustness to adversarial environments from game theory. Furthermore, this approach may not explicitly rely on any Markov assumptions.

[0341] The following describes in more detail the method of navigating by planning based on prediction. In this method, it can be assumed that the state space S is A subset of , and the action space A is This may be a natural representation in many applications. As mentioned above, there may be two key differences between RL and SL: (1) because past actions affect future rewards, information from the future may need to be propagated back to the past; (2) the “bandit” nature of rewards may obscure the dependencies between (state, action) and rewards, which can complicate the learning process.

[0342] As a first step in this approach, it can be observed that there are interesting problems where the bandit nature of the reward is not a problem. For example, the reward value of an ACC application (as discussed in more detail below) may be differentiable with respect to the current state and action. In fact, even if the reward is given in a "bandit" manner, learning a differentiable function Make The problem of can be a relatively straightforward SL problem (e.g., a one-dimensional regression problem). Therefore, the first step of the method can be to define the reward as a function The function is differentiable with respect to s and a, or the first step of the method can be to use a regression learning algorithm in order to learn a differentiable function This function minimizes at least some regression loss on at least one sample where the instance vector is and the target scalar is r t In some cases, an element of exploration can be used in order to create a training set.

[0343] To solve the connection between the past and the future, similar ideas can be used. For example, suppose that a differentiable function Make Learning such a function can be formulated as a SL problem. It can be considered as a predictor of the near future. Next, we can use the parameter function π θ : S→A to describe the strategy of mapping from S to A. θ Expressing it as a neural network enables using a recurrent neural network (RNN) to express a segment that runs the actor for T rounds, where the next state is defined as Here, Can be defined by the environment and can express unpredictable aspects of the near future. t+1 Depends on s in a differentiable way t and a t The fact that can make the connection between future reward value and past behavior. Policy function π θ The parameter vector of can be learned by backpropagation on the resulting RNN. Note that there is no need to tExplicit probabilistic assumptions are imposed on . In particular, no requirement for a Markov relation is required. Instead, the recurrent network can be relied upon to propagate “enough” information between the past and the future. Intuitively, can describe the predictable part of the near future, while ν t It can express the unpredictable aspects that may arise due to the behavior of other actors in the environment. The learning system should learn a policy that is robust to the behavior of other actors. If ||ν t || is large, the connection between past actions and future rewards may be too noisy to learn a meaningful policy. Explicitly expressing the dynamics of the system in a transparent way can make it easier to incorporate prior knowledge. For example, prior knowledge can simplify the definition of problem.

[0344] As described above, the learning system may benefit from robustness against an adversarial environment (such as the host vehicle's environment), which may include multiple other drivers who may behave in unexpected ways. t In the model with probabilistic assumptions, we can consider the adversarial selection of ν t In some cases, μ t impose constraints, otherwise the adversary may make the planning problem difficult or even impossible. A natural constraint might be to require a constant bound on ||μ t ||.

[0345] Robustness against adversarial environments may be useful in autonomous driving applications. t It can even speed up the learning process because it allows the learning system to focus on the best strategy that is robust. This concept can be illustrated with a simple game. Action The instantaneous loss function is 0.1|a t |+[|s t |-2] + , where [x] + =max{x,0} is the ReLU (rectified linear unit) function. The next state is s t+1 =s t +a t +ν t , where ν t ∈[-0.5, 0.5] is chosen in an adversarial way for the environment. Here, the optimal strategy can be written as a two-layer network with ReLU: t =-[s t -1.5] + +[-s t -1.5] + . Observe that when |st |∈(1.5, 2], the optimal action may have a larger immediate loss than action a=0. Therefore, the system can plan for the future and can not only rely on the immediate loss. Observe that the loss is about a t The derivative is 0.1sign(a t ), and about s t The derivative of is 1[|s t |>2]sign(s t ). t ∈(1.5, 2], ν t The adversarial selection of t = 0.5, so whenever a t >1.5-s t , there can be non-zero loss at round t+1. In this case, the derivative of the loss can be directly back-propagated to a t Therefore, in a t When the choice is non-optimal, v t Adversarial selection of can help the navigation system obtain non-zero backpropagation messages. This relationship can help the navigation system choose the current action based on the expectation that this current behavior (even if it results in a non-optimal reward or even a loss) will provide the opportunity for a more optimal action with a higher reward in the future.

[0346] This approach can be applied to virtually any navigation situation that may arise. The following description applies to the approach for one example: Adaptive Cruise Control (ACC). In the ACC problem, the host vehicle may attempt to maintain an appropriate distance from a target vehicle ahead (e.g., 1.5 seconds to the target vehicle). Another goal may be to drive as smoothly as possible while maintaining a desired gap. A model representing this situation can be defined as follows. The state space is The action space is The first coordinate of the state is the speed of the target car, the second coordinate is the speed of the host vehicle, and the last coordinate is the distance between the host vehicle and the target vehicle (e.g., the position of the host vehicle minus the position of the target vehicle along the curve of the road). The action taken by the host vehicle is to accelerate and can be expressed as a t The quantity τ may represent the time difference between successive rounds. Although τ may be set to any suitable quantity, in one example, τ may be 0.1 seconds. Position s t It can be expressed as And the (unknown) acceleration of the target vehicle can be expressed as

[0347] The complete dynamics of the system can be described by the following description:

[0348]

[0349] This can be described as the sum of two vectors:

[0350]

[0351] The first vector is the predictable part, while the second vector is the unpredictable part. The reward for round t is defined as follows:

[0352] in

[0353] The first term results in a penalty for non-zero acceleration, thus encouraging smooth driving. The second term depends on the distance x to the target car. t The distance from expectations The ratio between the desired distance is defined as the maximum between a distance of 1 meter and a braking distance of 1.5 seconds. In some cases, this ratio may be exactly 1, but as long as the ratio is in the range [0.7, 1.3], the policy may forgo any penalties, which may allow the host vehicle to relax a bit in navigation - a characteristic that may be important in achieving smooth driving.

[0354] Implementing the method outlined above, the navigation system of the host vehicle (e.g., through operation of the driving strategy module 803 within the processing unit 110 of the navigation system) can select an action in response to the observed state. The selected action can be based not only on an analysis of rewards associated with available responsive actions relative to the sensed navigation state, but also on consideration and analysis of future states, potential actions in response to the future states, and rewards associated with the potential actions.

[0355] Figure 16 An algorithmic approach to navigation based on detection and long-term planning is shown. For example, at step 1601, at least one processing device 110 of a navigation system of a host vehicle may receive a plurality of images. These images may capture a scene representing the host vehicle's environment and may be provided by any of the image capture devices described above (e.g., a camera, a sensor, etc.). Analyzing one or more of these images at step 1603 may enable the at least one processing device 110 to identify a current navigation state associated with the host vehicle (as described above).

[0356] Various potential navigation actions in response to the sensed navigation state may be determined in steps 1605, 1607, and 1609. These potential navigation actions (e.g., a first navigation action through an Nth available navigation action) may be determined based on the sensed state and a long-term goal of the navigation system (e.g., completing a merge, smoothly following a preceding vehicle, passing a target vehicle, avoiding an object in the road, slowing down for a detected stop sign, avoiding an oncoming target vehicle, or any other navigation action that may advance the navigation goal of the system).

[0357] For each of the potential navigation actions determined, the system can determine an expected reward. The expected reward can be determined according to any of the above techniques and can include an analysis of the specific potential action with respect to one or more reward functions. For each potential navigation action determined in steps 1605, 1607, and 1609 (e.g., the first, second, and Nth), an expected reward 1606, 1608, and 1610 can be determined, respectively.

[0358] In some cases, the navigation system of the host vehicle may select from the available potential actions based on the values ​​associated with the expected rewards 1606, 1608, and 1610 (or any other type of indicator of expected reward). For example, in some cases, the action that yields the highest expected reward may be selected.

[0359] In other cases, particularly where the navigation system performs long-term planning to determine the navigation action of the host vehicle, the system may not select the potential action that yields the highest expected reward. Instead, the system may look into the future to analyze whether there is an opportunity to achieve a higher reward later if a lower reward action is selected in response to the current navigation state. For example, a future state may be determined for any or all of the potential actions determined at steps 1605, 1607, and 1609. Each future state determined at steps 1613, 1615, and 1617 may represent a future navigation state that is expected to be modified by the corresponding potential action (e.g., the potential action determined at steps 1605, 1607, and 1609) based on the current navigation state.

[0360] For each of the future states predicted in steps 1613, 1615, and 1617, one or more future actions (as navigation options available in response to the determined future state) may be determined and evaluated. At steps 1619, 1621, and 1623, for example, a value or any other type of indicator of an expected reward associated with the one or more future actions may be generated (e.g., based on one or more reward functions). The expected reward associated with the one or more future actions may be evaluated by comparing the value of the reward function associated with each future action or by comparing any other indicator associated with the expected reward.

[0361] In step 1625, the navigation system of the host vehicle may select a navigation action for the host vehicle based on a comparison of expected rewards based not only on the potential actions identified relative to the current navigation state (e.g., in steps 1605, 1607, and 1609), but also on the expected rewards determined as a result of potential future actions available in response to the predicted future state (e.g., determined in steps 1613, 1615, and 1617). The selection in step 1625 may be based on the option and reward analysis performed in steps 1619, 1621, and 1623.

[0362] The selection of a navigation action at step 1625 may be based solely on a comparison of the expected rewards associated with future action options. In this case, the navigation system may select an action for the current state based solely on a comparison of the expected rewards that would result from actions for potential future navigation states. For example, the system may select the potential action identified at steps 1650, 1610, and 1609 that is associated with the highest future reward value determined by the analysis at steps 1619, 1621, and 1623.

[0363] The selection of a navigation action at step 1625 may also be based solely on a comparison of current action options (as described above). In this case, the navigation system may select the potential action associated with the highest expected reward 1606, 1608, or 1610 identified at steps 1605, 1607, or 1609. This selection may be performed with little or no consideration of future navigation states or future expected rewards for available navigation actions responsive to the expected future navigation states.

[0364] On the other hand, in some cases, the selection of the navigation action in step 1625 can be based on a comparison of the expected rewards associated with both the future action options and the current action options. In fact, this may be one of the navigation principles based on long-term planning. For example, the expected rewards for future actions can be analyzed to determine whether any expected rewards can authorize the selection of a lower reward action in response to the current navigation state, so as to achieve a potential higher reward in response to a subsequent navigation action that is expected to be available in response to the future navigation state. As an example, the value of expected reward 1606 or other indicators can indicate the highest expected reward among rewards 1606, 1608 and 1610. On the other hand, expected reward 1608 can indicate the lowest expected reward among rewards 1606, 1608 and 1610. Rather than simply selecting the potential action determined in step 1605 (i.e., the action that causes the highest expected reward 1606), the analysis of future state, potential future action, and future reward can be used to select a navigation action in step 1625. In one example, it may be determined that the reward identified at step 1621 (in response to at least one future action for the future state determined at step 1615, which is based on the second potential action determined at step 1607) may be higher than the expected reward 1606. Based on this comparison, the second potential action determined at step 1607 may be selected over the first potential action determined at step 1605, even though the expected reward 1606 is higher than the expected reward 1608. In one example, the potential navigation actions determined at step 1605 may include merging in front of a detected target vehicle, while the potential navigation actions determined at step 1607 may include merging behind the target vehicle. Although the expected reward 1606 for merging in front of the target vehicle may be higher than the expected reward 1608 associated with merging behind the target vehicle, it may be determined that merging behind the target vehicle may result in a future state for which action options may exist that yield an even higher potential reward than the expected rewards 1606, 1608, or other rewards based on available actions responsive to the current, sensed navigation state.

[0365] The selection from among the potential actions at step 1625 can be based on any suitable comparison of expected rewards (or comparing any other measure or indicator of benefit associated with one potential action to another). In some cases, as described above, if the second potential action is expected to provide at least one future action associated with an expected reward that is higher than the reward associated with the first potential action, then the second potential action can be selected over the first potential action. In other cases, more complex comparisons can be employed. For example, a reward associated with an action option responsive to a projected future state can be compared to more than one expected reward associated with a determined potential action.

[0366] In some cases, if at least one future action is expected to produce a reward that is higher than any reward expected as a result of a potential action for the current state (e.g., expected rewards 1606, 1608, 1610, etc.), then the action and expected reward based on the predicted future state can influence the selection of the potential action for the current state. In some cases, the future action option that produces the highest expected reward (e.g., from among the expected rewards associated with the potential actions for the sensed current state and from among the expected rewards associated with the potential future action options relative to the potential future navigation state) can be used as a guide for selecting the potential action for the current navigation state. That is, after identifying the future action option that produces the highest expected reward (or a reward above a predetermined threshold, etc.), in step 1625, the potential action that will result in the future state associated with the identified future action that produces the highest expected reward can be selected.

[0367] In other cases, selection of available actions may be made based on the difference determined between the expected rewards. For example, if the difference between the expected reward associated with the future action determined at step 1621 and the expected reward 1606 is greater than the difference between the expected reward 1608 and the expected reward 1606 (assuming a positive-signed difference), then the second potential action determined at step 1607 may be selected. In another example, if the difference between the expected reward associated with the future action determined at step 1621 and the expected reward associated with the future action determined at step 1619 is greater than the difference between the expected reward 1608 and the expected reward 1606, then the second potential action determined at step 1607 may be selected.

[0368] Several examples have been described for selecting from among potential actions for the current navigation state. However, any other suitable comparison technique or criteria may be used for selecting available actions through long-term planning based on action and reward analysis extending into projected future states. Figure 16 While two layers of long-term planning analysis are shown (e.g., a first layer considers rewards resulting from potential actions for the current state, and a second layer considers rewards resulting from future action options in response to projected future states), analysis based on more layers may be possible. For example, rather than basing long-term planning analysis on one or two layers, three, four, or more layers of analysis may be used to select from among available potential actions in response to the current navigational state.

[0369] After selecting from among the potential actions responsive to the sensed navigation state, at least one processor may cause at least one adjustment action of a navigation actuator of the host vehicle in response to the selected potential navigation state in step 1627. The navigation actuator may include any suitable device for controlling at least one aspect of the host vehicle. For example, the navigation actuator may include at least one of a steering mechanism, a brake, or an accelerator.

[0370] Navigation based on inferred aggression from other parties

[0371] A target vehicle can be monitored by analyzing the captured image stream to determine indicators of driving aggression. Aggression is described herein as a qualitative or quantitative parameter, but other characteristics may be used, such as the perceived level of driver attention (potential impairment, distraction caused by a cell phone, falling asleep, etc.). In some cases, the target vehicle may be considered to have a defensive posture, while in other cases, the target vehicle may be determined to have a more aggressive posture. Navigation actions may be selected or generated based on the indicators of aggression. For example, in some cases, the relative speed, relative acceleration, increase in relative acceleration, following distance, etc. relative to the host vehicle may be tracked to determine whether the target vehicle is aggressive or defensive. For example, if the target vehicle is determined to have an aggression level exceeding a threshold, the host vehicle may be inclined to yield to the target vehicle. The target vehicle's aggression level may also be discerned based on the determined behavior of the target vehicle relative to one or more obstacles in or near the target vehicle's path (e.g., a preceding vehicle, an obstacle on the road, a traffic light, etc.).

[0372] As an introduction to this concept, an exemplary experiment will be described with respect to a host vehicle merging onto a roundabout, where the navigation goal is to pass through and exit the roundabout. This scenario may begin with the host vehicle approaching the entrance to the roundabout and may end with the host vehicle reaching the exit (e.g., the second exit) of the roundabout. Success may be measured based on whether the host vehicle consistently maintains a safe distance from all other vehicles, whether the host vehicle completes the route as quickly as possible, and whether the host vehicle follows a smooth acceleration strategy. In this example, N T Target vehicles can be randomly placed on the roundabout. To model a mix of adversarial and typical behavior, with probability p, the target vehicle can be modeled using an "aggressive" driving strategy, causing the aggressive target vehicle to accelerate when the host vehicle attempts to merge in front of the target vehicle. With probability 1-p, the target vehicle can be modeled using a "defensive" driving strategy, causing the target vehicle to slow down and allow the host vehicle to merge. In this experiment, p = 0.5, and the host vehicle's navigation system may not be provided with information about the other driver's type. The other driver's type can be randomly selected at the beginning of the segment.

[0373] The navigation state can be represented as the velocity and position of the main vehicle (actor), and the position, velocity and acceleration of the target vehicles. It may be important to keep an eye on the target acceleration in order to distinguish between aggressive and defensive drivers based on the current state. All target vehicles may move along a one-dimensional curve with a circular path as the outline. The main vehicle may move on its own one-dimensional curve, which intersects the target vehicle's curve at a merge point, and this point is the starting point of both curves. In order to model reasonable driving, the absolute value of the acceleration of all vehicles can be bounded by a constant. Because reverse driving is not allowed, the velocity can also be passed through ReLU. Note that by not allowing reverse driving, long-term planning can become necessary because the actor will not regret its past actions.

[0374] As mentioned above, the next state s t+1 Can be broken down into predictable parts and the unpredictable part v t The sum of . The expression can represent the dynamics of the vehicle position and velocity (which can be well defined in a differentiable way), while ν t It can represent the acceleration of the target vehicle. It can be verified that It can be expressed as a combination of ReLU functions on affine transformations, so it is relatively t and a t is differentiable. The vector v t can be defined by the simulator in a non-differentiable way and can behave aggressively towards some targets and defensively towards others. Figure 17A and 17B Two frames from such a simulator are shown in FIG. In this exemplary experiment, the host vehicle 1701 learns to slow down as it approaches the entrance to the roundabout. It also learns to yield to aggressive vehicles (e.g., vehicles 1703 and 1705) and to safely continue driving when merging in front of defensive vehicles (e.g., vehicles 1706, 1708, and 1710). Figure 17A and 17B In the example shown, the type of target vehicle is not provided to the navigation system of the host vehicle 1701. Instead, whether a particular vehicle is determined to be aggressive or defensive is determined by inference based on the observed position and acceleration (e.g., of the target vehicle). Figure 17A In , based on position, velocity, and / or relative acceleration, host vehicle 1701 may determine that vehicle 1703 has an aggressive tendency, and therefore host vehicle 1701 may stop and wait for target vehicle 1703 to pass rather than attempting to merge in front of target vehicle 1703. However, in Figure 17B, target vehicle 1701 recognizes that target vehicle 1710 traveling behind vehicle 1703 is exhibiting a defensive tendency (also based on the observed position, speed and / or relative acceleration of vehicle 1710) and thereby completes a successful lane merge in front of target vehicle 1710 and behind target vehicle 1703.

[0375] Figure 18 A flow chart is provided that represents an exemplary algorithm for navigating a host vehicle based on predicted aggression from other vehicles. Figure 18 In an example, a level of aggression associated with at least one target vehicle can be inferred based on observed behavior of the target vehicle relative to objects in the target vehicle's environment. For example, in step 1801, at least one processing device (e.g., processing device 110) of a host vehicle navigation system can receive a plurality of images representing the host vehicle's environment from a camera associated with the host vehicle. In step 1803, analysis of the one or more received images can enable the at least one processor to identify a target vehicle (e.g., vehicle 1703) in the host vehicle's environment. In step 1805, analysis of the one or more received images can enable the at least one processing device to identify at least one obstacle to the target vehicle in the host vehicle's environment. The object can include debris in the road, a stop / traffic light, a pedestrian, another vehicle (e.g., a vehicle traveling in front of the target vehicle, a parked vehicle, etc.), a box in the road, a roadblock, a curb, or any other type of object that may be encountered in the host vehicle's environment. In step 1807, analysis of the one or more received images can enable the at least one processing device to determine at least one navigation characteristic of the target vehicle relative to the at least one identified obstacle to the target vehicle.

[0376] Various navigational characteristics may be used to infer the aggression level of a detected target vehicle in order to generate an appropriate navigational response to the target vehicle. For example, such navigational characteristics may include the relative acceleration between the target vehicle and at least one identified obstacle, the distance of the target vehicle from the obstacle (e.g., the following distance of the target vehicle behind another vehicle), and / or the relative speed between the target vehicle and the obstacle.

[0377] In some embodiments, the navigation characteristics of the target vehicle can be determined based on outputs from sensors associated with the host vehicle (e.g., radar, speed sensors, GPS, etc.). However, in some cases, the navigation characteristics of the target vehicle can be determined based in part or in whole based on an analysis of images of the host vehicle's environment. For example, the image analysis techniques described above and in, for example, U.S. Patent No. 9,168,868, incorporated herein by reference, can be used to identify a target vehicle within the host vehicle's environment. Furthermore, monitoring the position of the target vehicle in captured images over time and / or monitoring the position of one or more features associated with the target vehicle (e.g., taillights, headlights, bumpers, wheels, etc.) in captured images can enable determination of the relative distance, speed, and / or acceleration between the target vehicle and the host vehicle or between the target vehicle and one or more other objects in the host vehicle's environment.

[0378] The level of aggression of an identified target vehicle can be inferred from any suitable observed navigation characteristic of the target vehicle, or any combination of observed navigation characteristics. For example, a determination of aggressiveness can be made based on any observed characteristic and one or more predetermined threshold levels, or any other suitable qualitative or quantitative analysis. In some embodiments, a target vehicle may be considered aggressive if it is observed following the host vehicle or another vehicle at a distance less than a predetermined aggressive distance threshold. On the other hand, a target vehicle observed following the host vehicle or another vehicle at a distance greater than a predetermined defensive distance threshold may be considered defensive. The predetermined aggressive distance threshold need not be the same as the predetermined defensive distance threshold. Furthermore, either or both the predetermined aggressive distance threshold and the predetermined defensive distance threshold may include a range of values ​​rather than a bright line value. Furthermore, neither the predetermined aggressive distance threshold nor the predetermined defensive distance threshold need be fixed. Rather, these values ​​or ranges of values ​​may change over time, and different thresholds / ranges of thresholds may be applied based on the observed characteristics of the target vehicle. For example, the applied threshold may depend on one or more other characteristics of the target vehicle. Higher observed relative speeds and / or accelerations may warrant application of larger thresholds / ranges. Conversely, lower relative speeds and / or accelerations (including zero relative speed and / or acceleration) may authorize the application of smaller distance thresholds / ranges when making aggressive / defensive inferences.

[0379] Aggressive / defensive inferences can also be based on relative speed and / or relative acceleration thresholds. If the observed relative speed and / or relative acceleration of a target vehicle relative to another vehicle exceeds a predetermined level or range, the target vehicle can be considered aggressive. If the observed relative speed and / or relative acceleration of a target vehicle relative to another vehicle is below a predetermined level or range, the target vehicle can be considered defensive.

[0380] While an aggressive / defensive determination can be made based solely on any observed navigation characteristic, the determination can also depend on any combination of observed characteristics. For example, as described above, in some cases, a target vehicle may be deemed aggressive based solely on the observation that it is following another vehicle at a distance below a certain threshold or range. However, in other cases, the target vehicle may be deemed aggressive if it is following another vehicle at a distance less than a predetermined amount (which may be the same or different threshold applied when determining based solely on distance) and with a relative speed and / or relative acceleration greater than a predetermined amount or range. Similarly, a target vehicle may be deemed defensive based solely on the observation that it is following another vehicle at a distance greater than a certain threshold or range. However, in other cases, the target vehicle may be deemed defensive if it is following another vehicle at a distance greater than a predetermined amount (which may be the same or different threshold applied when determining based solely on distance) and with a relative speed and / or relative acceleration less than a predetermined amount or range. System 100 may deduce aggressiveness / defensiveness if, for example, the vehicle exceeds 0.5G acceleration or deceleration (e.g., 5m / s3 jerk), the vehicle has 0.5G lateral acceleration in a lane change or curve, a vehicle causes another vehicle to do any of the above, a vehicle changes lanes and causes another vehicle to yield with 0.3G deceleration or 3m / s3 jerk or more, and / or the vehicle changes lanes twice without stopping.

[0381] It should be understood that a reference to a quantity exceeding a range may indicate that the quantity exceeds all values ​​associated with the range or falls within the range. Similarly, a reference to a quantity below a range may indicate that the quantity is below all values ​​associated with the range or falls within the range. In addition, while examples for making aggressive / defensive inferences are described with respect to distance, relative acceleration, and relative speed, any other suitable quantity may be used. For example, a calculation of the time to collision may be used, or any indirect indicator of the distance, acceleration, and / or speed of the target vehicle. It should also be noted that while the above examples focus on a target vehicle relative to other vehicles, aggressive / defensive inferences may be made by observing the navigation characteristics of the target vehicle relative to any other type of obstacle (e.g., pedestrians, roadblocks, traffic lights, debris, etc.).

[0382] Back to Figure 17A and 17B In the illustrated example, as host vehicle 1701 approaches a roundabout, a navigation system (including at least one of its processing devices) may receive an image stream from a camera associated with the host vehicle. Based on analysis of one or more received images, any of target vehicles 1703, 1705, 1706, 1708, and 1710 may be identified. Furthermore, the navigation system may analyze navigational characteristics of one or more identified target vehicles. The navigation system may recognize that the gap between target vehicles 1703 and 1705 represents a potential first opportunity to merge onto the roundabout. The navigation system may analyze target vehicle 1703 to determine an aggression indicator associated with target vehicle 1703. If target vehicle 1703 is deemed aggressive, the host vehicle's navigation system may choose to yield to vehicle 1703 rather than merge ahead of vehicle 1703. On the other hand, if target vehicle 1703 is deemed defensive, the host vehicle's navigation system may attempt to complete the merge ahead of vehicle 1703.

[0383] As host vehicle 1701 approaches the roundabout, at least one processing device of the navigation system may analyze the captured images to determine navigational characteristics associated with target vehicle 1703. For example, based on the images, it may be determined that vehicle 1703 is following vehicle 1705 at a distance that provides sufficient clearance for host vehicle 1701 to safely enter. In fact, it may be determined that vehicle 1703 is following vehicle 1705 at a distance exceeding an aggressive distance threshold, and therefore, based on this information, the host vehicle navigation system may be inclined to identify target vehicle 1703 as defensive. However, in some cases, as described above, more than one navigational characteristic of the target vehicle may be analyzed when making an aggressive / defensive determination. Further analysis may determine that, when target vehicle 1703 is following target vehicle 1705 at a non-aggressive distance, vehicle 1703 has a relative speed and / or relative acceleration relative to vehicle 1705 that exceeds one or more thresholds associated with aggressive behavior. In effect, host vehicle 1701 may determine that target vehicle 1703 is accelerating relative to vehicle 1705 and approaching a gap that exists between vehicles 1703 and 1705. Based on further analysis of relative speed, acceleration, and distance (and even the rate at which the gap between vehicles 1703 and 1705 is closing), host vehicle 1701 may determine that target vehicle 1703 is behaving aggressively. Thus, while there may be a sufficient gap that the host vehicle can safely navigate into, host vehicle 1701 may anticipate that a merge in front of target vehicle 1703 will result in an aggressively navigating vehicle immediately behind the host vehicle. Furthermore, based on behavior observed through image analysis or other sensor outputs, target vehicle 1703 may anticipate that if host vehicle 1701 were to merge in front of vehicle 1703, target vehicle 1703 would continue to accelerate toward host vehicle 1701 or continue to travel toward host vehicle 1701 at a non-zero relative speed. This situation may be undesirable from a safety perspective and may also cause discomfort to the occupants of the host vehicle. For this reason, Figure 17B As shown, host vehicle 1701 may choose to yield to vehicle 1703 and merge onto the roundabout behind vehicle 1703 and in front of vehicle 1710, which is deemed defensive based on an analysis of one or more of its navigational characteristics.

[0384] Back to Figure 18At step 1809, at least one processing device of the navigation system of the host vehicle may determine a navigation action for the host vehicle (e.g., merging in front of vehicle 1710 and behind vehicle 1703) based on at least one navigation characteristic of the identified target vehicle relative to the identified obstacle. To implement the navigation action (at step 1811), at least one processing device may cause at least one adjustment of a navigation actuator of the host vehicle in response to the determined navigation action. For example, a brake may be applied to give way to the target vehicle. Figure 17A and the accelerator may be applied along with steering of the host vehicle's wheels to cause the host vehicle to enter the roundabout behind vehicle 1703, as Figure 17B shown.

[0385] As described in the above examples, the navigation of the host vehicle may be based on the navigation characteristics of the target vehicle relative to another vehicle or object. Alternatively, the navigation of the host vehicle may be based solely on the navigation characteristics of the target vehicle without specific reference to another vehicle or object. For example, in Figure 18 At step 1807, analysis of the plurality of images captured from the host vehicle's environment may enable determination of at least one navigational characteristic of the identified target vehicle, the navigational characteristic being indicative of a level of aggression associated with the target vehicle. Navigational characteristics may include speed, acceleration, and the like, which do not require reference to another object or target vehicle in order to make an aggressive / defensive determination. For example, an observed acceleration and / or speed associated with the target vehicle that exceeds a predetermined threshold or falls within or exceeds a numerical range may indicate aggressive behavior. Conversely, an observed acceleration and / or speed associated with the target vehicle that falls below a predetermined threshold or falls within or exceeds a numerical range may indicate defensive behavior.

[0386] Of course, in some cases, observed navigational characteristics (e.g., position, distance, acceleration, etc.) may be referenced relative to the host vehicle for purposes of making an aggressive / defensive determination. For example, observed navigational characteristics of a target vehicle that indicate a level of aggressiveness associated with the target vehicle may include an increase in relative acceleration between the target vehicle and the host vehicle, a following distance of the target vehicle behind the host vehicle, a relative speed between the target vehicle and the host vehicle, etc.

[0387] Navigation based on accident liability constraints

[0388] As described in the sections above, planned navigation actions can be tested against predetermined constraints to ensure that certain rules are met. In some embodiments, this concept can be extended to considerations of potential accident liability. As described below, a primary goal of autonomous navigation is safety. Since absolute safety may not be possible (e.g., at least because a particular host vehicle under autonomous control cannot control other vehicles around it - it can only control its own actions), using potential accident liability as a consideration in autonomous navigation, and indeed as a constraint on planned actions, can help ensure that a particular autonomous vehicle does not take any actions that are considered unsafe - e.g., those actions for which potential accident liability could be attached to the host vehicle. A desired level of accident avoidance (e.g., fewer than 10 accidents per hour driven) can be achieved if the host vehicle only takes actions that are safe and that are determined not to result in an accident for which the host vehicle is itself at fault (fault) or responsible. -9 ).

[0389] Challenges posed by most current approaches to autonomous driving include a lack of safety guarantees (or at least an inability to provide the desired level of safety), as well as a lack of scalability. Consider the problem of ensuring safe driving for multiple actors. Since society is unlikely to tolerate machine-caused road fatalities, an acceptable level of safety is crucial for the acceptance of autonomous vehicles. While the goal may be to provide zero accidents, this may not be possible because multiple actors are often involved in an accident, and it is conceivable that an accident occurs solely due to the fault of other actors. For example, Figure 19 As shown, host vehicle 1901 is driving on a multi-lane highway. While host vehicle 1901 can control its own movements relative to target vehicles 1903, 1905, 1907, and 1909, it cannot control the movements of the target vehicles around it. As a result, if vehicle 1905, for example, suddenly cuts into the host vehicle's lane on a collision course with the host vehicle, host vehicle 1901 may be unable to avoid an accident with at least one of the target vehicles. To address this challenge, autonomous vehicle practitioners typically respond by adopting a statistically driven approach, in which safety validation becomes increasingly rigorous as more miles of data are collected.

[0390] However, to understand the nature of the problem with data-driven approaches to safety, first consider that the probability of a (human) fatality caused by a driving accident is known to be 10 per hour. -6 It is reasonable to assume that in order for society to accept machines replacing humans in driving tasks, the death rate should be reduced by three orders of magnitude, that is, to 10 per hour. -9 This estimate is similar to the assumed airbag mortality rate and is derived from aviation standards. For example, 10 -9is the probability that the wing will spontaneously separate from the aircraft in mid-air. However, it is not practical to try to guarantee safety using data-driven statistical methods that provide additional confidence by accumulating driving miles. -9 The amount of data required to calculate the probability of death is its reciprocal (i.e. 10 9 hours of data), which is on the order of 30 billion miles. Furthermore, multi-actor systems interact with their environment and may not be able to be validated offline (unless a realistic simulator is available that simulates real human driving in all its richness and complexity (such as reckless driving) - but the problem of validating such a simulator is even more difficult than creating a safe autonomous vehicle actor). Any change to the planning and control software will require the same amount of new data collection, which is clearly impractical and impractical. Furthermore, developing systems from data always suffers from a lack of interpretability and explainability of the actions taken - if an autonomous vehicle (AV) gets into an accident resulting in a fatality, we need to know why. Therefore, a model-based approach to safety is needed, but existing "functional safety" and ASIL requirements in the automotive industry were not designed to cope with multi-actor environments.

[0391] The second major challenge in developing safe driving models for autonomous vehicles is the need for scalability. The premise of AVs isn't just about "building a better world," but rather that mobility without drivers can be maintained at a lower cost than mobility with drivers. This premise is always coupled with the concept of scalability—that is, supporting the mass production of AVs (in the millions) and, more importantly, supporting negligible incremental costs to enable driving in new cities. So, while the cost of computing and sensing is indeed important, the cost of validation and the ability to drive "everywhere" rather than in a select few cities are also necessary requirements to sustain business if AVs are to be manufactured at scale.

[0392] The problem with most current approaches lies in a "brute force" mentality along three axes: (i) the required "computational density," (ii) the way HD maps are defined and created, and (iii) the required specifications of the sensors. A brute force approach runs counter to scalability and shifts the focus to a future of unconstrained, ubiquitous in-vehicle computing, where the cost of building and maintaining HD maps becomes negligible and scalable, and exotic, ultra-advanced sensors are developed, produced, and cost-effectively scaled to automotive grade. A future where any of these scenarios are possible is certainly possible, but achieving all of them is likely a low-probability event. Therefore, there is a need for a formal model that combines safety and scalability into a socially acceptable AV program that is scalable in the sense of supporting millions of cars driving anywhere in the developed world.

[0393] The disclosed embodiments represent a solution that can provide the target safety level (or even exceed the safety goal) and can also be scaled to systems including millions (or more) of autonomous vehicles. In terms of safety, a model called "Responsibility Sensitive Safety (RSS)" is introduced that formalizes the concept of "accident attribution", is interpretable and explainable, and incorporates the concept of "responsibility" into the actions of robotic actors. The definition of RSS is agnostic to the way it is implemented - a key feature that facilitates the goal of creating a convincing global safety model. RSS is motivated by the observation that (e.g. Figure 19 (as shown) actors play asymmetric roles in accidents where typically only one of the actors is responsible and therefore liable for the accident. The RSS model also includes a formal treatment of "careful driving" under limited sensing conditions where not all actors are always visible (e.g., due to occlusions). A major goal of the RSS model is to guarantee that an actor never has an accident for which it is "attributed" or for which it is responsible. The model can only be useful if it comes with an effective policy that complies with RSS (e.g., a function that maps "sensing states" to actions). For example, actions that seem innocent at the moment may lead to catastrophic events in the distant future (a "butterfly effect"). RSS can help to construct a set of local constraints in the short term future that can guarantee (or at least virtually guarantee) that no...

Claims

1. An accident responsibility tracking system for an autonomous host vehicle traveling in a multi-lane corridor, the system comprising: At least one processing device comprising circuitry and memory, wherein the memory comprises instructions that, when executed by the circuitry, cause the at least one processing device to: receiving, via a data connection, at least one image captured by an image capture device, the at least one image representing an environment of the host vehicle; analyzing the at least one image to identify a target vehicle in the host vehicle's environment, the target vehicle traveling in a lane adjacent to a lane of travel of the host vehicle, the analyzing comprising applying at least one of computer vision software or a trained learning model; determining one or more characteristics of a navigational state of the identified target vehicle based on an analysis of the at least one image, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; determining an anticipated collision between the host vehicle and the target vehicle based on an analysis of the at least one image; Based on the determination that the collision is anticipated, determining whether the host vehicle can avoid the collision based on a navigation state of the host vehicle, wherein the determination of whether the collision can be avoided includes generating a plurality of planned navigation maneuvers for the host vehicle and determining whether there is a predicted future state predicted to avoid the collision, the predicted future state corresponding to at least one of the plurality of planned navigation maneuvers; based on a determination that the collision cannot be avoided, comparing the determined one or more characteristics of the navigational state of the identified target vehicle to a set of accident liability rules associated with changes in lateral position of vehicles, the rules including a liability time for an expected collision, the liability time including an earliest time prior to the collision at which the target vehicle enters a lane of travel of the host vehicle with an unsafe longitudinal distance between the host vehicle and the target vehicle; storing at least one value indicative of potential accident liability on a portion of the identified target vehicle based on a comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the accident liability rule set, the value indicative of potential accident liability being determined based at least in part on a change in a lateral position of the target vehicle; and The at least one stored value for determining the accident responsibility is outputted.

2. The tracking system of claim 1 further comprising an interface for accessing and downloading collision liability values ​​after an accident occurs between the host vehicle and the identified target vehicle.

3. The tracking system according to claim 1, wherein: The memory includes instructions that, when executed by the circuit, cause the at least one processing device to detect a plurality of target vehicles in an environment of the host vehicle, determine a navigation state characteristic of each of the plurality of target vehicles, and determine and store a value indicating potential accident liability on a portion of a corresponding target vehicle of the plurality of target vehicles based on a comparison of the corresponding navigation state characteristic of each of the target vehicles with the accident liability rule set.

4. The tracking system according to claim 1, wherein: The accident liability rule set includes a lateral speed rule.

5. The tracking system according to claim 1, wherein: The accident responsibility rule set includes a lateral position rule.

6. The tracking system according to claim 1, wherein: The accident responsibility rule set includes a driving direction priority rule.

7. The tracking system according to claim 1, wherein: The accident responsibility rule set includes traffic light based rules.

8. The tracking system according to claim 1, wherein: The accident responsibility rule set includes rules based on traffic signs.

9. The tracking system according to claim 1, wherein: The accident responsibility rule set includes a route priority rule.

10. A system for navigating an autonomous host vehicle traveling in a multi-lane corridor, the system comprising: At least one processing device comprising circuitry and memory, wherein the memory comprises instructions that, when executed by the circuitry, cause the at least one processing device to: receiving, via a data connection, a plurality of images captured by an image capture device, the plurality of images representing an environment of the host vehicle; Determining a planned navigation action for completing a navigation goal of the host vehicle based on at least one driving strategy; Determining a trajectory of the host vehicle to complete the planned navigation target; analyzing the plurality of images to identify a target vehicle in the host vehicle's environment and a predicted trajectory of the target vehicle, the target vehicle traveling in a lane adjacent to a lane of travel of the host vehicle, the analyzing comprising applying at least one of computer vision software or a trained learning model; determining an expected collision between the host vehicle and the target vehicle based on the trajectory of the host vehicle and the trajectory of the target vehicle; testing the planned navigation maneuvers of the host vehicle against a set of accident liability rules related to changes in lateral position of vehicles based on a determination that the collision is expected, the rules including a liability time for the expected collision, the liability time including an earliest time prior to the collision at which the target vehicle enters a lane of travel of the host vehicle wherein a longitudinal distance between the host vehicle and the target vehicle is unsafe, wherein potential accident liability is determined based on a predicted trajectory of the target vehicle and a trajectory of the host vehicle; If the test of the planned navigation action for the accident liability rule set indicates that there is potential accident liability for the host vehicle if the planned navigation action is taken, causing the host vehicle to not implement the planned navigation action; and If the test of the planned navigation action for the accident responsibility rule set indicates that if the planned navigation action is taken, no accident responsibility will be incurred for the host vehicle, causing the host vehicle to implement the planned navigation action; and Wherein, the at least one processing device is further programmed to: determining one or more characteristics of a navigation state of the identified target vehicle based on an analysis of the plurality of images, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; based on the determination that the collision is anticipated, determining whether the host vehicle can avoid a collision between the host vehicle and the target vehicle based on a navigational state of the host vehicle, wherein the determination of whether the collision can be avoided includes generating a plurality of additional planned navigational maneuvers for the host vehicle and determining whether there is a predicted future state predicted to avoid the collision, the predicted future state corresponding to at least one of the plurality of planned additional navigational maneuvers; Based on a determination that the collision cannot be avoided, comparing the determined one or more characteristics of the navigation state of the identified target vehicle to a set of accident liability rules; and Based on a comparison of one or more characteristics of the determined navigation state of the identified target vehicle with the accident liability rule set, at least one value indicating potential accident liability on a portion of the identified target vehicle is stored, the value indicating potential accident liability being determined at least in part based on a change in a lateral position of the target vehicle.

11. The system according to claim 10, wherein: The memory includes instructions that, when executed by the circuitry, cause the at least one processing device to output the stored at least one value for determining responsibility for an incident between the host vehicle and an identified target vehicle after the incident occurs.

12. The system according to claim 10, wherein: The accident liability rule set includes a lateral speed rule.

13. The system according to claim 10, wherein: The accident responsibility rule set includes a lateral position rule.

14. The system according to claim 10, wherein: The accident responsibility rule set includes a driving direction priority speed rule.

15. The system according to claim 10, wherein: The accident responsibility rule set includes traffic light based rules.

16. The system of claim 10, wherein: The accident responsibility rule set includes rules based on traffic signs.

17. The system according to claim 10, wherein: The accident responsibility rule set includes a route priority rule.

18. A method for tracking accident liability of an autonomous host vehicle traveling in a multi-lane corridor, the method comprising: receiving, via a data connection, at least one image captured by an image capture device, the at least one image representing an environment of the host vehicle; analyzing the at least one image to identify a target vehicle in the host vehicle's environment, the target vehicle traveling in a lane adjacent to a lane of travel of the host vehicle, the analyzing comprising applying at least one of computer vision software or a trained learning model; determining one or more characteristics of a navigational state of the identified target vehicle based on an analysis of the at least one image, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; determining an anticipated collision between the host vehicle and the target vehicle based on an analysis of the at least one image; Based on the determination that the collision is anticipated, determining whether the host vehicle can avoid the collision based on a navigation state of the host vehicle, wherein the determination of whether the collision can be avoided includes generating a plurality of planned navigation maneuvers for the host vehicle and determining whether there is a predicted future state predicted to avoid the collision, the predicted future state corresponding to at least one of the plurality of planned navigation maneuvers; based on a determination that the collision cannot be avoided, comparing the determined one or more characteristics of the navigational state of the identified target vehicle to a set of accident liability rules associated with changes in lateral position of vehicles, the rules including a liability time for an expected collision, the liability time including an earliest time prior to the collision at which the target vehicle enters a lane of travel of the host vehicle with an unsafe longitudinal distance between the host vehicle and the target vehicle; storing at least one value indicative of potential accident liability on a portion of the identified target vehicle based on a comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the accident liability rule set, the value indicative of potential accident liability being determined based at least in part on a change in a lateral position of the target vehicle; as well as The at least one stored value for determining the accident responsibility is outputted.

19. The method according to claim 18, further comprising: accessing, via an interface, a collision liability value following an incident between the host vehicle and the identified target vehicle; as well as The collision responsibility value is downloaded via the interface.

20. The method of claim 18, further comprising: detecting a plurality of target vehicles in an environment of the host vehicle; determining a navigation state characteristic of each target vehicle in the plurality of target vehicles; as well as Based on a comparison of the corresponding navigation state characteristic of each of the target vehicles with the accident liability rule set, a value indicative of potential accident liability on a portion of a corresponding target vehicle of the plurality of target vehicles is determined and stored.

21. The method according to claim 18, wherein: The accident liability rule set includes a lateral speed rule.

22. The method according to claim 18, wherein: The accident responsibility rule set includes a lateral position rule.

23. The method according to claim 18, wherein: The accident responsibility rule set includes a driving direction priority rule.

24. The method of claim 18, wherein: The accident responsibility rule set includes traffic light based rules.

25. The method of claim 18, wherein: The accident responsibility rule set includes rules based on traffic signs.

26. The method of claim 18, wherein: The accident responsibility rule set includes a route priority rule.

27. A method for navigating an autonomous host vehicle traveling in a multi-lane corridor, the method comprising: receiving, via a data connection, a plurality of images captured by an image capture device, the plurality of images representing an environment of the host vehicle; Determining a planned navigation action for completing a navigation goal of the host vehicle based on at least one driving strategy; Determining a trajectory of the host vehicle to complete the planned navigation target; analyzing the plurality of images to identify a target vehicle in the host vehicle's environment and a predicted trajectory of the target vehicle, the target vehicle traveling in a lane adjacent to a lane of travel of the host vehicle, the analyzing comprising applying at least one of computer vision software or a trained learning model; determining an expected collision between the host vehicle and the target vehicle based on the trajectory of the host vehicle and the trajectory of the target vehicle; testing the planned navigation maneuvers of the host vehicle against a set of accident liability rules related to changes in lateral position of vehicles based on a determination that the collision is expected, the rules including a liability time for the expected collision, the liability time including an earliest time prior to the collision at which the target vehicle enters a lane of travel of the host vehicle wherein a longitudinal distance between the host vehicle and the target vehicle is unsafe, wherein potential accident liability is determined based on a predicted trajectory of the target vehicle and a trajectory of the host vehicle; If the test of the planned navigation action for the accident responsibility rule set indicates that there is a potential accident responsibility of the host vehicle if the planned navigation action is taken, causing the host vehicle not to perform the planned navigation action; If the test indication of the planned navigation action for the accident responsibility rule set indicates that if the planned navigation action is taken, no accident responsibility will be generated for the host vehicle, causing the host vehicle to implement the planned navigation action; determining one or more characteristics of a navigation state of the identified target vehicle based on an analysis of the plurality of images, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; based on the determination that the collision is anticipated, determining whether the host vehicle can avoid a collision between the host vehicle and the target vehicle based on a navigational state of the host vehicle, wherein the determination of whether the collision can be avoided includes generating a plurality of additional planned navigational maneuvers for the host vehicle and determining whether there is a predicted future state predicted to avoid the collision, the predicted future state corresponding to at least one of the plurality of planned additional navigational maneuvers; based on a determination that the collision cannot be avoided, comparing the determined one or more characteristics of the navigation state of the identified target vehicle to a set of accident liability rules; and At least one value indicative of potential accident liability on a portion of the identified target vehicle is stored based on a comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the accident liability rule set.

28. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform a method for navigating an autonomous host vehicle traveling in a multi-lane corridor, the method comprising: receiving, via a data connection, at least one image captured by an image capture device, the at least one image representing an environment of the host vehicle; analyzing the at least one image to identify a target vehicle in the host vehicle's environment, the target vehicle traveling in a lane adjacent to a lane of travel of the host vehicle, the analyzing comprising applying at least one of computer vision software or a trained learning model; determining one or more characteristics of a navigational state of the identified target vehicle based on an analysis of the at least one image, the one or more characteristics comprising a change in a lateral position of the target vehicle relative to at least one of a center of a lane in which the target vehicle is traveling or a lateral position of the host vehicle; determining an anticipated collision between the host vehicle and the target vehicle based on an analysis of the at least one image; Based on the determination that the collision is anticipated, determining whether the host vehicle can avoid the collision based on a navigation state of the host vehicle, wherein the determination of whether the collision can be avoided includes generating a plurality of planned navigation maneuvers for the host vehicle and determining whether there is a predicted future state predicted to avoid the collision, the predicted future state corresponding to at least one of the plurality of planned navigation maneuvers; based on a determination that the collision cannot be avoided, comparing the determined one or more characteristics of the navigational state of the identified target vehicle to a set of accident liability rules associated with changes in lateral position of vehicles, the rules including a liability time for an expected collision, the liability time including an earliest time prior to the collision at which the target vehicle enters a lane of travel of the host vehicle with an unsafe longitudinal distance between the host vehicle and the target vehicle; storing at least one value indicative of potential accident liability on a portion of the identified target vehicle based on a comparison of the determined one or more characteristics of the navigation state of the identified target vehicle with the accident liability rule set, the value indicative of potential accident liability being determined based at least in part on a change in a lateral position of the target vehicle; and The at least one stored value for determining the accident responsibility is outputted.

Images