Predicting driver path based on gaze direction and applications thereof

Abstract

Provided herein are system, device, apparatus, method, and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for predicting a driver path of a vehicle. In the method, an image of a driver of the vehicle is received. Based on the image, a gaze direction of the driver is determined. Based at least in part on the gaze direction, a path of the vehicle is predicted. The vehicle is controlled based on the predicted path.

Description

[0001] Cross-references to related applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 727,403, filed December 3, 2024. The foregoing application is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure generally relates to autonomous vehicle navigation. Additionally, this disclosure relates to systems and methods for navigation based on potential accident liability constraints. Background Technology

[0003] With continuous technological advancements, the goal of fully autonomous vehicles capable of navigating on roads is fast approaching. Autonomous vehicles may need to consider a variety of factors and make appropriate decisions based on those factors to safely and accurately reach their intended destination. For example, autonomous vehicles may need to process and interpret visual information (e.g., information captured from cameras), information from radar or lidar, and may also use information from other sources (e.g., from GPS devices, speed sensors, accelerometers, suspension sensors, etc.). Simultaneously, to navigate to their destination, autonomous vehicles may also need to identify their location within a specific roadway (e.g., a specific lane in a multi-lane road), navigate alongside other vehicles, avoid obstacles and pedestrians, obey traffic signals and signs, move from one road to another at appropriate intersections or junctions, and respond to any other situations that occur or develop during vehicle operation. Furthermore, navigation systems may need to adhere to certain imposed constraints. In some cases, these constraints may relate to the interaction between the primary vehicle and one or more other objects (such as other vehicles, pedestrians, etc.). In other cases, the constraints may relate to rules of responsibility that the primary vehicle should follow when performing one or more navigation actions.

[0004] In the field of autonomous driving, there are two important considerations for feasible autonomous vehicle systems. The first is the standardization of safety assurance, including the requirements that each self-driving car must meet to ensure safety, and how to verify these requirements. The second is scalability, because engineering solutions that lead to exacerbated costs will not scale to millions of vehicles and may hinder the widespread adoption of autonomous vehicles, or even less widespread adoption. Therefore, interpretable mathematical models for safety assurance and the design of systems that adhere to safety assurance requirements while scalable to millions of vehicles are needed. Summary of the Invention

[0005] Embodiments consistent with this disclosure provide systems and methods for autonomous vehicle navigation. The disclosed embodiments may use cameras to provide autonomous vehicle navigation features. For example, consistent with the disclosed embodiments, the disclosed system may include one, two, or more cameras monitoring the vehicle's environment. The disclosed system may provide a navigation response based, for example, analysis of images captured by one or more of the cameras. The navigation response may also take into account other data, including, for example, Global Positioning System (GPS) data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data.

[0006] In some embodiments, a non-transitory computer-readable device stores instructions that, when executed by at least one computing device, cause the at least one computing device to perform the following operations: receive an image of a driver of a vehicle; determine, based on the image, the driver's gaze direction; predict a path of the vehicle, at least in part based on the gaze direction; and control the vehicle based on the predicted path.

[0007] In some embodiments, a system for predicting a driver's path for a vehicle includes one or more externally facing sensors that monitor the external environment of the vehicle; one or more internally facing cameras that monitor the driver of the vehicle; a memory; and at least one processor coupled to the memory and configured to receive an image of the driver of the vehicle; determine the driver's gaze direction based on the image; predict the path of the vehicle based at least in part on the gaze direction; and control the vehicle based on the predicted path.

[0008] In some embodiments, a system for predicting a driver's path for a vehicle includes a vehicle controller; a memory; and at least one processor coupled to the memory and configured to receive an image of the driver of the vehicle; determine the driver's gaze direction based on the image; predict the path of the vehicle based at least in part on the gaze direction; and control the vehicle based on the predicted path.

[0009] In some embodiments, a computer-implemented method for predicting a driver's path for a vehicle includes receiving an image of the driver of the vehicle; determining, based on the image, the driver's gaze direction; predicting a path for the vehicle based at least in part on the gaze direction; and controlling the vehicle based on the predicted path.

[0010] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that can be executed by at least one processing device to perform any of the steps and / or methods described herein.

[0011] The foregoing general description and the following detailed description are merely exemplary and illustrative, and do not limit the scope of the claims. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings: Figure 1 This is a schematic representation of an exemplary system consistent with the disclosed embodiments.

[0013] Figure 2A A schematic side view representation of an exemplary vehicle that includes a system consistent with the disclosed embodiments.

[0014] Figure 2B To be consistent with the disclosed embodiments Figure 2A The schematic top view of the vehicle and system is shown.

[0015] Figure 2C A schematic top view representation of another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0016] Figure 2D A schematic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0017] Figure 2E A schematic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0018] Figure 2F This is a schematic representation of an exemplary vehicle control system consistent with the disclosed embodiments.

[0019] Figure 3A A schematic representation of the interior of a vehicle, including a rearview mirror and user interface for a vehicle imaging system, consistent with the disclosed embodiments.

[0020] Figure 3B An illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield, consistent with the disclosed embodiments.

[0021] Figure 3C To be viewed from a different perspective in accordance with the disclosed embodiments. Figure 3B The diagram shows the camera mounting components.

[0022] Figure 3DAn illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield, consistent with the disclosed embodiments.

[0023] Figure 4 An exemplary block diagram of a memory configured to store instructions for performing one or more operations, consistent with the disclosed embodiments.

[0024] Figure 5A A flowchart is provided to illustrate an exemplary process for evoking one or more navigation responses based on monocular image analysis, consistent with the disclosed embodiments.

[0025] Figure 5B A flowchart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images, consistent with the disclosed embodiments.

[0026] Figure 5C A flowchart illustrating an exemplary process for detecting road markings and / or lane geometry information in a set of images, consistent with the disclosed embodiments.

[0027] Figure 5D A flowchart illustrating an exemplary process for detecting traffic lights in a set of images, consistent with the disclosed embodiments.

[0028] Figure 5E A flowchart is provided to illustrate an exemplary process for evoking one or more navigation responses based on a vehicle path, consistent with the disclosed embodiments.

[0029] Figure 5F A flowchart illustrating an exemplary process for determining whether a vehicle ahead is changing lanes, consistent with the disclosed embodiments.

[0030] Figure 6 A flowchart illustrating an exemplary process for evoking one or more navigation responses based on stereo image analysis, consistent with the disclosed embodiments.

[0031] Figure 7 A flowchart illustrating an exemplary process consistent with the disclosed embodiments for evoking one or more navigation responses based on the analysis of three sets of images.

[0032] Figure 8 A block diagram representation of a module that can be implemented by one or more specially programmed processing devices for a navigation system for autonomous vehicles, consistent with the disclosed embodiments.

[0033] Figure 9 A navigation options diagram consistent with the disclosed embodiments.

[0034] Figure 10A navigation options diagram consistent with the disclosed embodiments.

[0035] Figure 11A , 11B 11C provides a schematic representation of navigation options for the primary vehicle in a lane-changing zone consistent with the disclosed embodiments.

[0036] Figure 11D An illustrative description of a dual lane-changing scenario consistent with the disclosed embodiments is provided.

[0037] Figure 11E An options diagram consistent with the disclosed embodiments is provided that may be useful in dual-lane merging scenarios.

[0038] Figure 12 A representational image of the host vehicle’s environment capture and a graph of potential navigation constraints, consistent with the disclosed embodiments, are provided.

[0039] Figure 13 An algorithm flowchart for navigating vehicles, consistent with the disclosed embodiments, is provided.

[0040] Figure 14 An algorithm flowchart for navigating vehicles, consistent with the disclosed embodiments, is provided.

[0041] Figure 15 An algorithm flowchart for navigating vehicles, consistent with the disclosed embodiments, is provided.

[0042] Figure 16 An algorithm flowchart for navigating vehicles, consistent with the disclosed embodiments, is provided.

[0043] Figure 17A and 17B A schematic illustration of the main vehicle navigating into the roundabout, consistent with the disclosed embodiments, is provided.

[0044] Figure 18 An algorithm flowchart for navigating vehicles, consistent with the disclosed embodiments, is provided.

[0045] Figure 19 An example of a main vehicle driving on a multi-lane highway, consistent with the disclosed embodiments, is shown.

[0046] Figure 20A and 20B An example of a vehicle cutting in front of another vehicle, consistent with the disclosed embodiments, is shown.

[0047] Figure 21 An example of a vehicle following another vehicle, consistent with the disclosed embodiments, is shown.

[0048] Figure 22An example is shown where a vehicle, consistent with the disclosed embodiments, leaves a parking lot and merges into a potentially busy road.

[0049] Figure 23 A vehicle traveling on a road, consistent with the disclosed embodiments, is shown.

[0050] Figure 24 A-24D illustrates four example scenarios consistent with the disclosed embodiments.

[0051] Figure 25 An example scenario consistent with the disclosed embodiments is shown.

[0052] Figure 26 An example scenario consistent with the disclosed embodiments is shown.

[0053] Figure 27 An example scenario consistent with the disclosed embodiments is shown.

[0054] Figure 28A and 28B Examples of scenarios where a vehicle follows another vehicle, consistent with the disclosed embodiments, are shown.

[0055] Figure 29A and 29B Example attribution is shown in an entry scenario consistent with the disclosed embodiments.

[0056] Figure 30A and 30B Example attribution is shown in an entry scenario consistent with the disclosed embodiments.

[0057] Figures 31A-31D An example attribution is shown in a drift scenario consistent with the disclosed embodiments.

[0058] Figure 32A and 32B An example attribution of responsibility in a two-way traffic scenario consistent with the disclosed embodiments is shown.

[0059] Figure 33A and 33B An example attribution of responsibility in a two-way traffic scenario consistent with the disclosed embodiments is shown.

[0060] Figure 34A and 34B An example attribution is shown in a route prioritization scenario consistent with the disclosed embodiments.

[0061] Figure 35A and 35B An example attribution is shown in a route prioritization scenario consistent with the disclosed embodiments.

[0062] Figure 36A and 36B An example attribution is shown in a route prioritization scenario consistent with the disclosed embodiments.

[0063] Figure 37A and 37B An example attribution is shown in a route prioritization scenario consistent with the disclosed embodiments.

[0064] Figure 38A and 38B An example attribution is shown in a route prioritization scenario consistent with the disclosed embodiments.

[0065] Figure 39A and 39B An example attribution is shown in a route prioritization scenario consistent with the disclosed embodiments.

[0066] Figure 40A and 40B An example attribution is shown in a traffic light scenario consistent with the disclosed embodiments.

[0067] Figure 41A and 41B An example attribution is shown in a traffic light scenario consistent with the disclosed embodiments.

[0068] Figure 42A and 42B An example attribution is shown in a traffic light scenario consistent with the disclosed embodiments.

[0069] Figures 43A-43C An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown.

[0070] Figures 44A-44C An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown.

[0071] Figure 45A-45C An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown.

[0072] Figures 46A-46D An example vulnerable road user (VRU) scenario consistent with the disclosed embodiments is shown.

[0073] Figure 47A and 47B An example scenario is shown, consistent with the disclosed embodiments, in which a vehicle follows another vehicle.

[0074] Figure 48 A flowchart illustrating an exemplary process for navigating a master vehicle, consistent with the disclosed embodiments.

[0075] Figures 49A-49D An example scenario is shown, consistent with the disclosed embodiments, in which a vehicle follows another vehicle.

[0076] Figure 50 A flowchart illustrating an exemplary process for braking a master vehicle consistent with the disclosed embodiments.

[0077] Figure 51 A flowchart illustrating an exemplary process for navigating a master vehicle, consistent with the disclosed embodiments.

[0078] Figures 52A-52D An example proximity buffer for the master vehicle, consistent with the disclosed embodiments, is shown.

[0079] Figure 53A and 53B An example scenario, including proximity to a buffer, is shown, consistent with the disclosed embodiments.

[0080] Figure 54A and 54B An example scenario, including proximity to a buffer, is shown, consistent with the disclosed embodiments.

[0081] Figure 55 A flowchart for selectively replacing human driver control of the main vehicle, consistent with the disclosed embodiments, is provided.

[0082] Figure 56 A flowchart illustrating an exemplary process for navigating a master vehicle, consistent with the disclosed embodiments.

[0083] Figures 57A-57C An example scenario consistent with the disclosed embodiments is shown.

[0084] Figure 58 A flowchart illustrating an exemplary process for navigating a master vehicle, consistent with the disclosed embodiments.

[0085] Figure 59 An environment illustrating how gaze direction affects path prediction is shown according to some embodiments.

[0086] Figure 60 A flowchart illustrating a gaze-oriented method for predicting paths and controlling vehicles is provided.

[0087] Figure 61 An illustration of one embodiment shows the capture of the driver's gaze via an inward-facing camera.

[0088] Figure 62 A model of a Purkinje image according to some embodiments is shown.

[0089] Figure 63A -C illustrates head and eye pose detection for classifying driver gaze according to one embodiment.

[0090] Figure 64 Object detection is demonstrated using an outward-facing camera according to one embodiment.

[0091] Figure 65 The illustration shows a scene captured by an outward-facing sensor according to one embodiment, and a heat map representing the driver's gaze direction captured by an inward-facing sensor, as shown in one embodiment.

[0092] Figure 66 A flowchart illustrating a gaze-oriented method for predicting paths and controlling vehicles is provided.

[0093] Figure 67 A flowchart is shown for a method of predicting paths and controlling vehicles based on gaze direction.

[0094] Figure 68 This is a block diagram of an ADAS according to some embodiments.

[0095] Figure 69 A computer system according to exemplary embodiments of the present disclosure is shown. Detailed Implementation

[0096] The following detailed description refers to the accompanying drawings. Where possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar components. While several illustrative embodiments are described herein, modifications, adaptations, and other embodiments are possible. For example, components shown in the drawings may be replaced, added, or modified, and the illustrative methods described herein may be modified by replacing, reordering, removing, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Rather, the appropriate scope is defined by the appended claims.

[0097] Overview of Autonomous Vehicles As used throughout this disclosure, the term "autonomous vehicle" means a vehicle capable of implementing at least one navigation change without driver input. "Navigation change" refers to a change in one or more of the vehicle's steering, braking, or acceleration / deceleration. For autonomy to be achieved, the vehicle does not need to be fully automatic (e.g., fully operational without a driver or driver input). Rather, autonomous vehicles include those that can operate under driver control during certain time periods and without driver control during other time periods. Autonomous vehicles may also include those that control only some aspects of vehicle navigation (such as steering (e.g., to maintain vehicle alignment between lane constraints) or some steering operations in certain conditions (but not all conditions), but leave other aspects to the driver (e.g., braking or braking in certain conditions). In some cases, an autonomous vehicle may handle some or all aspects of the vehicle's braking, rate control, and / or steering.

[0098] Since human drivers typically rely on visual cues and observation to control vehicles, traffic infrastructure is constructed accordingly, with lane markings, traffic signs, and traffic lights designed to provide visual information to the driver. Given these design characteristics of traffic infrastructure, autonomous vehicles may include cameras and processing units that analyze visual information captured from the vehicle's environment. Visual information may include, for example, images representing components of traffic infrastructure observable by the driver (e.g., lane markings, traffic signs, traffic lights, etc.) and other obstacles (e.g., other vehicles, pedestrians, debris, etc.). Additionally, autonomous vehicles may use stored information, such as information providing a model of the vehicle's environment during navigation. For example, a vehicle may use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information related to its environment while driving, and the vehicle (and other vehicles) may use said information to locate itself on the model. Some vehicles may also be able to communicate with each other, share information, modify hazards or alterations in the vehicle's surroundings for peer vehicles, etc.

[0099] System Overview Figure 1This is a block diagram representation of system 100 consistent with the exemplary embodiments disclosed. System 100 may include various components depending on the requirements of a particular implementation. In some embodiments, system 100 may include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. Processing unit 110 may include one or more processing means. In some embodiments, processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing means. Similarly, depending on the requirements of a particular application, image acquisition unit 120 may include any number of image acquisition means and components. In some embodiments, image acquisition unit 120 may include one or more image capture means (e.g., a camera, CCD, or any other type of image sensor), such as image capture means 122, image capture means 124, and image capture means 126. System 100 may also include a data interface 128 that communicatively connects processing unit 110 to image acquisition unit 120. For example, data interface 128 may include any one or more wired and / or wireless links for transmitting image data acquired by image acquisition unit 120 to processing unit 110.

[0100] Wireless transceiver 172 may include one or more devices configured to exchange transmissions with one or more networks (e.g., cellular networks, the Internet, etc.) via an air interface using radio frequency, infrared frequency, magnetic field, or electric field. Wireless transceiver 172 may use any known standard to transmit and / or receive data (e.g., Wi-Fi, Bluetooth®, Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmissions may include communication from a master vehicle to one or more remotely located servers. Such transmissions may also include (one-way or two-way) communication between the master vehicle and one or more target vehicles in the master vehicle's environment (e.g., to facilitate coordination of navigation of the master vehicle in light of or in relation to target vehicles in the master vehicle's environment), or even broadcast transmissions to unspecified receivers in the vicinity of the transmitting vehicle.

[0101] Both application processor 180 and image processor 190 may include various types of hardware-based processing devices. For example, either or both of application processor 180 and image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), supporting circuitry, a digital signal processor, an integrated circuit, memory, or any other type of device suitable for running applications and suitable for image processing and analysis. In some embodiments, application processor 180 and / or image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. Various processing devices may be used, including processors available from manufacturers such as Intel®, AMD®, etc., and may include various architectures (e.g., x86 processors, ARM®, etc.).

[0102] In some embodiments, application processor 180 and / or image processor 190 may include any processor chip from the EyeQ family of processor chips available from Mobileye®. These processor designs each include multiple processing units with local memory and instruction sets. Such processors may include video inputs for receiving image data from multiple image sensors and may also include video output capabilities. In one example, the EyeQ2® uses 90 nm micrometer technology operating at 332 MHz. The EyeQ2® architecture consists of two floating-point hyper-threaded 32-bit RISC CPUs (MIPS32® 34K® cores), five Visual Computing Engines (VCEs), three Vector Microcode Processors (VMPs®), a Denali 64-bit mobile DDR controller, a 128-bit internal Sonics interconnect, dual 16-bit video input and 18-bit video output controllers, a 16-channel DMA, and several peripherals. The MIPS34K CPU manages the five VCEs, three VMPs™ and DMA, a second MIPS34K CPU and multi-channel DMA, and other peripherals. Five VCEs, three VMP® processors, and a MIPS34K CPU enable the intensive vision computing required for versatile bundled applications. In another instance, the EyeQ3® (a third-generation processor with six times the performance of the EyeQ2®) can be used in the disclosed embodiments. In other instances, the EyeQ4® and / or EyeQ5® can be used in the disclosed embodiments. Of course, any newer or future EyeQ processing devices can also be used with the disclosed embodiments.

[0103] Any of the processing devices disclosed herein can be configured to perform certain functions. Configuring a processing device (such as the described EyeQ processor or any other controller or microprocessor) to perform certain functions may include programming computer-executable instructions and making those instructions available for execution by the processing device during operation of the processing device. In some embodiments, configuring the processing device may include programming the processing device directly using architectural instructions. In other embodiments, configuring the processing device may include storing executable instructions on memory accessible to the processing device during operation. For example, the processing device may access the memory during operation to obtain and execute the stored instructions. In any case, a processing device configured to perform the sensing, image analysis, and / or navigation functions disclosed herein represents a specialized hardware-based system that controls multiple hardware-based components of a master vehicle.

[0104] Although Figure 1 Two separate processing devices included in processing unit 110 are depicted, but more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to perform the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Furthermore, in some embodiments, system 100 may include one or more processing units 110 without other components such as image acquisition unit 120.

[0105] Processing unit 110 can include various types of devices. For example, processing unit 110 can include various devices such as controllers, image preprocessors, central processing units (CPUs), support circuitry, digital signal processors, integrated circuits, memory, or any other type of device for image processing and analysis. An image preprocessor can include a video processor for capturing, digitizing, and processing images from an image sensor. A CPU can include any number of microcontrollers or microprocessors. Support circuitry can be any number of circuits well known in the art, including caches, power supplies, clocks, and input / output circuits. Memory can store software that controls the operation of the system when executed by the processor. Memory can include databases and image processing software. Memory can include any number of random access memories, read-only memories, flash memory, disk drives, optical storage devices, magnetic tape storage devices, removable storage devices, and other types of storage devices. In one instance, the memory can be separate from processing unit 110. In another instance, the memory can be integrated into processing unit 110.

[0106] Each memory unit 140, 150 may include software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), can control various aspects of the operation of system 100. For example, these memory units may include various database and image processing software, as well as trained systems such as neural networks or deep neural networks. Memory units may include random access memory, read-only memory, flash memory, disk drives, optical storage devices, magnetic tape storage devices, removable storage devices, and / or any other type of storage device. In some embodiments, memory units 140, 150 may be decoupled from application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into application processor 180 and / or image processor 190.

[0107] The location sensor 130 may include any type of device adapted to determine a location associated with at least one component of the system 100. In some embodiments, the location sensor 130 may include a GPS receiver. Such a receiver can determine a user's location and speed by processing signals broadcast by Global Positioning System satellites. Location information from the location sensor 130 may be used by the application processor 180 and / or the image processor 190.

[0108] In some embodiments, system 100 may include components such as speed sensors (e.g., speedometers) for measuring the speed of vehicle 200. System 100 may also include one or more accelerometers (single-axis or multi-axis) for measuring the acceleration of vehicle 200 along one or more axes.

[0109] Memory units 140 and 150 may include a database or data organized in any other form that indicates the location of known landmarks. Sensory information about the environment (such as images, radar signals, depth information from lidar, or stereo processing of two or more images) can be processed together with location information (such as GPS coordinates, vehicle self-motion, etc.) to determine the vehicle's current location relative to known landmarks and to refine the vehicle's location. Certain aspects of this technology are included in what is called REM. TM In the positioning technology, the positioning technology is sold by the assignee of this application.

[0110] User interface 170 may include any means adapted to provide information to or receive input from one or more users of system 100. In some embodiments, user interface 170 may include user input devices, such as touchscreens, microphones, keyboards, pointer devices, scroll wheels, cameras, knobs, buttons, etc. Using such input devices, users can provide information input or commands to system 100 by typing instructions or information, providing voice commands, selecting menu options on the screen (using buttons, pointers, or eye-tracking capabilities), or by any other suitable technology for transmitting information to system 100.

[0111] User interface 170 may be equipped with one or more processing devices configured to provide and receive information from the user, and to process the information for use by, for example, application processor 180. In some embodiments, such processing devices may execute instructions to recognize and track eye movements, receive and interpret voice commands, recognize and interpret touches and / or gestures made on a touchscreen, respond to keyboard input or menu selection, etc. In some embodiments, user interface 170 may include a display, speakers, haptic devices, and / or any other means for providing output information to the user.

[0112] Map database 160 may include any type of database for storing map data useful to system 100. In some embodiments, map database 160 may include data relating to the location of various items (including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations) in a reference coordinate system. Map database 160 may not only store the locations of such items but also descriptors associated with these items, including, for example, names associated with any of the stored features. In some embodiments, map database 160 may be physically located along with other components of system 100. Alternatively or additionally, map database 160 or a portion thereof may be remotely located relative to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 may be downloaded via a wired or wireless data connection to a network (e.g., via cellular networks and / or the Internet). In some cases, map database 160 may store a sparse data model including a polynomial representation of certain road features (e.g., lane markings) or target trajectories for the primary vehicle. Map database 160 may also include stored representations of various identified landmarks, which can be used to determine or update the known position of the master vehicle relative to the target trajectory. Landmark representations may include data fields such as landmark type, landmark location, and other possible identifiers.

[0113] Image capture devices 122, 124, and 126 may each include any type of means suitable for capturing at least one image from the environment. Furthermore, any number of image capture devices can be used to acquire images for input to an image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four or more image capture devices. Reference will be made below. Figure 2B-2E The image capture devices 122, 124 and 126 are further described.

[0114] One or more cameras (e.g., image capture devices 122, 124, and 126) may be part of a sensing block included on the vehicle. Various other sensors may be included in the sensing block, and the vehicle's sensed navigation state may be developed relying on any or all of these sensors. In addition to cameras (forward, side, rear, etc.), the sensing block may also include other sensors such as radar, lidar, and acoustic sensors. Furthermore, the sensing block may include one or more components configured to transmit and receive information related to the vehicle's environment. For example, such components may include a wireless transceiver (RF, etc.) that can receive sensor-based information or any other type of information related to the environment of the main vehicle from a source remotely located relative to the main vehicle. This information may include sensor output information or related information received from vehicle systems other than the main vehicle. In some embodiments, this information may include information received from a remote computing device, a centralized server, etc. Moreover, cameras may take many different configurations: single camera unit, multiple cameras, camera clusters, long FOV, short FOV, wide-angle, fisheye, etc.

[0115] System 100 or its various components can be incorporated into a variety of different platforms. In some embodiments, system 100 may be included on vehicle 200, such as... Figure 2A As shown. For example, vehicle 200 may be equipped with processing unit 110 and any other components of system 100, as described above relative to... Figure 1 As described. While in some embodiments, vehicle 200 may be equipped with only a single image capture device (e.g., a camera), in other embodiments (such as combined with...) Figure 2B-2E Among those discussed, multiple image capture devices can be used. For example, such as Figure 2A As shown, either of the image capture devices 122 and 124 of the vehicle 200 can be part of an ADAS (Advanced Driver Assistance System) imaging set.

[0116] The image capture device, which is part of the image acquisition unit 120 and is included on the vehicle 200, can be positioned at any suitable location. In some embodiments, such as Figures 2A-2E and Figures 3A-3CAs shown, the image capturing device 122 can be located in the vicinity of the rearview mirror. This location provides a line of sight similar to that of the driver of vehicle 200, which can help determine what is visible and invisible to the driver. The image capturing device 122 can be positioned anywhere near the rearview mirror, but placing the image capturing device 122 on the driver's side of the mirror can further assist in obtaining an image representing the driver's field of view and / or line of sight.

[0117] Other positioning of the image capture device for image acquisition unit 120 may also be used. For example, image capture device 124 may be positioned on or therein on the bumper of vehicle 200. Such positioning may be particularly suitable for image capture devices with a wide field of view. The line of sight of the image capture device positioned on the bumper may be different from that of the driver, and therefore the bumper image capture device and the driver may not always see the same object. Image capture devices (e.g., image capture devices 122, 124, and 126) may also be positioned in other locations. For example, the image capture device may be located on or in one or both of the side mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on the side of vehicle 200, mounted on any of the windows of vehicle 200, positioned behind or in front of them, and mounted in or near the lights at the front and / or rear of vehicle 200, etc.

[0118] In addition to the image capture device, vehicle 200 may also include various other components of system 100. For example, processing unit 110 may be included on vehicle 200, or integrated with or separate from the vehicle's engine control unit (ECU). Vehicle 200 may also be equipped with position sensor 130 such as a GPS receiver, and may also include map database 160 and memory units 140 and 150.

[0119] As discussed above, the wireless transceiver 172 can receive data via one or more networks (e.g., cellular networks, the Internet, etc.) and / or receive data. For example, the wireless transceiver 172 can upload data collected by the system 100 to one or more servers and download data from said one or more servers. Via the wireless transceiver 172, the system 100 can receive, for example, periodic or on-demand updates to data stored in map database 160, memory 140, and / or memory 150. Similarly, the wireless transceiver 172 can upload any data from the system 100 (e.g., images captured by image acquisition unit 120, data received by position sensor 130 or other sensors, vehicle control system, etc.) and / or any data processed by processing unit 110 to one or more servers.

[0120] System 100 may upload data to a server (e.g., to the cloud) based on privacy level settings. For example, System 100 may implement privacy level settings to regulate or restrict the data types (including metadata) sent to a server that can uniquely identify the vehicle and / or the vehicle's driver / owner. Such settings may be configured by the user via, for example, wireless transceiver 172, through factory default settings, or through data received by wireless transceiver 172.

[0121] In some embodiments, system 100 may upload data according to a “high” privacy level, and under a setting, system 100 may transmit data (e.g., location information related to routes, captured images, etc.) without any details about the specific vehicle and / or driver / owner. For example, when uploading data under a “high” privacy setting, system 100 may not include the vehicle identification number (VIN) or the name of the driver or vehicle owner, but may instead transmit data such as captured images and / or restricted location information related to routes.

[0122] Other privacy levels are also envisioned. For example, system 100 may transmit data to the server at a “medium” privacy level, including additional information not included at the “high” privacy level, such as the vehicle’s brand and / or model and / or vehicle type (e.g., passenger vehicle, SUV, truck, etc.). In some embodiments, system 100 may upload data at a “low” privacy level. At the “low” privacy level setting, system 100 may upload data and include information sufficient to uniquely identify a specific vehicle, owner / driver, and / or part or all of the route traveled by the vehicle. Such “low” privacy level data may include, for example, one or more of the following: VIN, driver / owner name, vehicle’s origin point before departure, vehicle’s intended destination, vehicle’s brand and / or model, vehicle type, etc.

[0123] Figure 2A A schematic side view representation of an exemplary vehicle imaging system consistent with the disclosed embodiments. Figure 2B for Figure 2A The illustrated embodiment is shown in a schematic top view diagram. Figure 2B The illustrated and disclosed embodiments may include a vehicle 200, which includes a system 100 in its body, the system having a first image capture device 122 positioned in the vicinity of a rearview mirror and / or near the driver of the vehicle 200, a second image capture device 124 positioned on or in a bumper area of ​​the vehicle 200 (e.g., one of the bumper areas 210), and a processing unit 110.

[0124] like Figure 2CAs shown, both image capture devices 122 and 124 can be positioned in the vicinity of the rearview mirror and / or near the driver of vehicle 200. Additionally, although... Figure 2B and 2C Two image capture devices 122 and 124 are shown, but it should be understood that other embodiments may include more than two image capture devices. For example, in Figure 2D and 2E In the illustrated embodiment, the first, second, and third image capture devices 122, 124, and 126 are included in the system 100 of the vehicle 200.

[0125] like Figure 2D As shown, image capture device 122 can be positioned in the vicinity of the rearview mirror and / or near the driver of vehicle 200, and image capture devices 124 and 126 can be positioned above or in the bumper area of ​​vehicle 200 (e.g., one of bumper areas 210). And as... Figure 2E As shown, image capturing devices 122, 124, and 126 can be positioned in the vicinity of the rearview mirror and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capturing devices, and the image capturing devices can be positioned within vehicle 200 and / or in any suitable location on said vehicle.

[0126] It should be understood that the disclosed embodiments are not limited to vehicles and can be applied to other situations. It should also be understood that the disclosed embodiments are not limited to a specific type of vehicle 200 and can be applied to all types of vehicles, including cars, trucks, trailers and other types of vehicles.

[0127] The first image capture device 122 may include any suitable type of image capture device. The image capture device 122 may include an optical axis. In one example, the image capture device 122 may include an Aptina M9V024 WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280 x 960 pixels and may include a rolling shutter. The image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included, for example, to provide a desired focal length and field of view for the image capture device. In some embodiments, the image capture device 122 may be associated with a 6mm lens or a 12mm lens. In some embodiments, the image capture device 122 may be configured to capture an image with a desired field of view (FOV) 202, such as... Figure 2DAs shown. For example, image capture device 122 can be configured to have a regular FOV, such as 46 degrees, 50 degrees, 52 degrees, or greater, within the range of 40 to 56 degrees. Alternatively, image capture device 122 can be configured to have a narrow FOV, such as 28 degrees or 36 degrees, within the range of 23 to 40 degrees. Furthermore, image capture device 122 can be configured to have a wide FOV within the range of 100 to 180 degrees. In some embodiments, image capture device 122 may include a wide-angle bumper camera or a camera with a maximum FOV of 180 degrees. In some embodiments, image capture device 122 may be a 7.2 M-pixel image capture device with an aspect ratio of approximately 2:1 (e.g., HxV = 3800 × 1900 pixels) and a horizontal FOV of approximately 100 degrees. Such image capture devices can be used to replace a three-image capture device configuration. Due to significant lens distortion, in embodiments where the image capture device uses radially symmetrical lenses, the vertical FOV of such image capture devices may be significantly less than 50 degrees. For example, such lenses may not be radially symmetrical, which would allow a vertical FOV greater than 50 degrees and a horizontal FOV of 100 degrees.

[0128] The first image capturing device 122 can acquire a plurality of first images relative to a scene associated with the vehicle 200. Each of the plurality of first images can be acquired as a series of image scan lines that can be captured using a rolling shutter. Each scan line may include a plurality of pixels.

[0129] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor can acquire image data associated with each pixel included in a particular scan line.

[0130] For example, image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be used in conjunction with a rolling shutter, such that each pixel in a row is read one at a time, and the scanning of rows is performed on a line-by-line basis until the entire image frame has been captured. In some embodiments, rows may be captured sequentially from top to bottom relative to the frame.

[0131] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124 and 126) may constitute a high-resolution imager and may have a resolution greater than 5M pixels, 7M pixels, 10M pixels or more.

[0132] The use of a rolling shutter can cause pixels in different rows to be exposed and captured at different times, which can lead to skew and other image artifacts in the captured image frame. On the other hand, when the image capture device 122 is configured to operate using a global or synchronous shutter, all pixels can be exposed for the same amount of time and during a common exposure period. Therefore, the image data in a frame collected from a system using a global shutter represents a snapshot of the entire FOV (e.g., FOV 202) at a specific time. Conversely, in a rolling shutter application, each row in the frame is exposed, and the data is captured at different times. Therefore, moving objects may appear distorted in an image capture device with a rolling shutter. This phenomenon will be described in more detail below.

[0133] The second image capture device 124 and the third image capture device 126 can be any type of image capture device. Similar to the first image capture device 122, each of the image capture devices 124 and 126 may include an optical axis. In one embodiment, each of the image capture devices 124 and 126 may include an Aptina M9V024WVGA sensor with a global shutter. Alternatively, each of the image capture devices 124 and 126 may include a rolling shutter. Similar to image capture device 122, image capture devices 124 and 126 can be configured to include various lenses and optical elements. In some embodiments, the lenses associated with image capture devices 124 and 126 may provide an FOV (such as FOV 202) that is the same as or narrower than that associated with image capture device 122 (such as FOV 204 and 206). For example, image capture devices 124 and 126 may have an FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.

[0134] Image capture devices 124 and 126 can acquire multiple second and third images of a scene associated with vehicle 200. Each of the multiple second and third images can be acquired as a second series of image scan lines and a third series of image scan lines that can be captured using a rolling shutter. Each scan line or row can have multiple pixels. Image capture devices 124 and 126 can have a second scan rate and a third scan rate associated with the acquisition of each of the image scan lines included in the second and third series.

[0135] Each image capture device 122, 124, and 126 can be positioned relative to the vehicle 200 at any suitable location and orientation. The relative positioning of the image capture devices 122, 124, and 126 can be selected to facilitate the fusion of information acquired from the image capture devices. For example, in some embodiments, the field of view (FOV) associated with image capture device 124 (such as FOV 204) can partially or completely overlap with the FOV associated with image capture device 122 (such as FOV 202) and the FOV associated with image capture device 126 (such as FOV 206).

[0136] Image capture devices 122, 124, and 126 can be positioned at any suitable relative height on the vehicle 200. In one example, a height difference may exist between image capture devices 122, 124, and 126, which can provide sufficient parallax information for stereoscopic analysis. For example, as Figure 2A As shown, the two image capture devices 122 and 124 are located at different heights. For example, a lateral displacement difference may also exist between image capture devices 122, 124, and 126, providing additional parallax information for stereo analysis by the processing unit 110. The difference in lateral displacement can be determined by d... x Indicates, such as Figure 2C and 2D As shown. In some embodiments, there may be forward or backward displacement (e.g., range displacement) between image capture devices 122, 124, and 126. For example, image capture device 122 may be positioned 0.5 meters to 2 meters or more behind image capture devices 124 and / or 126. This type of displacement allows one of the image capture devices to cover potential blind spots of the other image capture devices.

[0137] Image capture device 122 may have any suitable resolution capability (e.g., the number of pixels associated with an image sensor), and the resolution of the image sensor associated with image capture device 122 may be higher, lower, or the same as the resolution of the image sensors associated with image capture devices 124 and 126. In some embodiments, the image sensors associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640×480, 1024×768, 1280×960, or any other suitable resolution.

[0138] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data for an image frame before continuing to capture pixel data associated with the next image frame) can be controllable. The frame rate associated with image capture device 122 can be higher, lower, or the same as the frame rates associated with image capture devices 124 and 126. The frame rates associated with image capture devices 122, 124, and 126 can depend on a variety of factors that may affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include a selectable pixel delay period applied before or after the acquisition of image data associated with one or more pixels of the image sensors in image capture devices 122, 124, and / or 126. Generally, image data corresponding to each pixel can be acquired according to the clock rate for the device (e.g., one pixel per clock cycle). Additionally, in embodiments including a rolling shutter, one or more of the image capture devices 122, 124, and 126 may include a selectable horizontal blanking period applied before or after the acquisition of image data associated with pixel rows of the image sensors in the image capture devices 122, 124, and / or 126. Furthermore, one or more of the image capture devices 122, 124, and / or 126 may include a selectable vertical blanking period applied before or after the acquisition of image data associated with image frames of the image capture devices 122, 124, and 126.

[0139] These timing controls enable synchronization of the frame rates associated with image capture devices 122, 124, and 126, even when the line scan rates of each image capture device differ. Furthermore, as will be discussed in more detail below, these selectable timing controls and other factors (e.g., image sensor resolution, maximum line scan rate, etc.) can enable synchronization of image capture from areas where the field of view (FOV) of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even when the field of view of image capture device 122 differs from the FOVs of image capture devices 124 and 126.

[0140] The frame rate timing in the image capture devices 122, 124, and 126 can depend on the resolution of the associated image sensor. For example, assuming similar line scan rates for two devices, if one device includes an image sensor with a resolution of 640×480 and the other device includes an image sensor with a resolution of 1280×960, it will take more time to acquire one frame of image data from the sensor with the higher resolution.

[0141] Another factor that can affect the timing of image data acquisition in image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquiring one line of image data from an image sensor included in image capture devices 122, 124, and 126 will require a certain minimum amount of time. Assuming no pixel delay period is added, this minimum amount of time for acquiring one line of image data will be related to the maximum line scan rate of a particular device. A device that provides a higher maximum line scan rate is likely to provide a higher frame rate than a device with a lower maximum line scan rate. In some embodiments, one or more of image capture devices 124 and 126 may have a higher maximum line scan rate than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25 times, 1.5 times, 1.75 times, or 2 times or more of the maximum line scan rate of image capture device 122.

[0142] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may be operated at a scan rate less than or equal to its maximum scan rate. The system may be configured such that one or more of image capture devices 124 and 126 operate at a line scan rate equal to the line scan rate of image capture device 122. In other instances, the system may be configured such that the line scan rate of image capture device 124 and / or image capture device 126 may be 1.25 times, 1.5 times, 1.75 times, or 2 times or more of the line scan rate of image capture device 122.

[0143] In some embodiments, image capture devices 122, 124, and 126 may be asymmetrical. That is, they may include cameras with different fields of view (FOV) and focal lengths. For example, the fields of view of image capture devices 122, 124, and 126 may include any desired area relative to the environment of vehicle 200. In some embodiments, one or more of image capture devices 122, 124, and 126 may be configured to acquire image data from the environment of the front of vehicle 200, the rear of vehicle 200, the sides of vehicle 200, or a combination thereof.

[0144] Furthermore, the focal length associated with each image capturing device 122, 124, and / or 126 can be selectable (e.g., by including appropriate lenses, etc.) so that each device acquires an image of an object relative to the vehicle 200 at a desired distance range. For example, in some embodiments, image capturing devices 122, 124, and 126 can acquire images of close-up objects within a few meters of the vehicle. Image capturing devices 122, 124, and 126 can also be configured to acquire images of objects at a greater distance from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more). Furthermore, the focal lengths of image capturing devices 122, 124, and 126 can be selected such that one image capturing device (e.g., image capturing device 122) can acquire images of objects relatively close to the vehicle (e.g., within 10 m or 20 m), while other image capturing devices (e.g., image capturing devices 124 and 126) can acquire images of objects further away from the vehicle (e.g., greater than 20 m, 50 m, 100 m, 150 m, etc.).

[0145] According to some embodiments, the field of view (FOV) of one or more image capture devices 122, 124, and 126 may have a wide angle. For example, having a 140-degree FOV may be advantageous, particularly for image capture devices 122, 124, and 126 that can be used to capture images of areas adjacent to the vehicle 200. For example, image capture device 122 may be used to capture images of areas to the right or left of the vehicle 200, and in such embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0146] The field of view associated with each of the image capturing devices 122, 124, and 126 can depend on the corresponding focal length. For example, as the focal length increases, the corresponding field of view decreases.

[0147] Image capture devices 122, 124, and 126 can be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV between 23 degrees and 46 degrees. In another example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV between 26 degrees and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 may vary between 1.5 and 2.0. In other embodiments, this ratio may vary between 1.25 and 2.25.

[0148] System 100 can be configured such that the field of view of image capture device 122 at least partially or completely overlaps with the field of view of image capture devices 124 and / or 126. In some embodiments, system 100 can be configured such that the field of view of image capture devices 124 and 126, for example, falls within the field of view of image capture device 122 (e.g., narrower than said field of view) and shares a common center with said field of view. In other embodiments, image capture devices 122, 124, and 126 can capture adjacent FOVs or can have partial overlap in their FOVs. In some embodiments, the field of view of image capture devices 122, 124, and 126 can be aligned such that the center of the narrower FOV image capture device 124 and / or 126 is located in the lower half of the field of view of the wider FOV device 122.

[0149] Figure 2F This is a schematic representation of an exemplary vehicle control system consistent with the disclosed embodiments. Figure 2F As indicated, vehicle 200 may include a throttle system 220, a braking system 230, and a steering system 240. System 100 may provide inputs (e.g., control signals) to one or more of the throttle system 220, braking system 230, and steering system 240 via one or more data links (e.g., any wired and / or wireless link or link used for data transmission). For example, based on analysis of images acquired by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of the throttle system 220, braking system 230, and steering system 240 to navigate vehicle 200 (e.g., by inducing acceleration, turning, lane changing, etc.). Furthermore, system 100 may receive inputs from one or more of the throttle system 220, braking system 230, and steering system 240 indicative of operating conditions of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or turning, etc.). The following is in conjunction with... Figures 4 to 7 Further details will be provided.

[0150] like Figure 3AAs shown, vehicle 200 may also include a user interface 170 for interaction with the driver or passengers of vehicle 200. For example, the user interface 170 in a vehicle application may include a touchscreen 320, a knob 330, a button 340, and a microphone 350. The driver or passengers of vehicle 200 may also interact with system 100 using handles (e.g., located on or near the steering column of vehicle 200, including, for example, a turn signal handle), buttons (e.g., located on the steering wheel of vehicle 200), etc. In some embodiments, microphone 350 may be located adjacent to rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, system 100 may provide various notifications (e.g., alarms) via speaker 360.

[0151] Figure 3B-3D An illustration of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and against the vehicle windshield, consistent with the disclosed embodiments. Figure 3B As shown, the camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind a glare shield 380, which may be flush with a vehicle windshield and comprises a composition of a film and / or anti-reflective material. For example, the glare shield 380 may be positioned such that it is aligned against a vehicle windshield having a matching slope. In some embodiments, each of the image capture devices 122, 124, and 126 may be positioned behind the glare shield 380, such as, for example... Figure 3D The embodiments described herein are not limited to any particular configuration of the image capture devices 122, 124 and 126, the camera mount 370 and the glare shield 380. Figure 3C Viewed from the front Figure 3B The illustration shows the camera mounting component 370.

[0152] As those skilled in the art who benefit from this disclosure will understand, various variations and / or modifications can be made to the disclosed embodiments. For example, not all components are essential for the operation of system 100. Furthermore, any component can be located in any suitable part of system 100, and these components can be rearranged into various configurations while providing the functionality of the disclosed embodiments. Thus, the foregoing configuration is an example, and regardless of the configuration discussed above, system 100 can provide a wide range of functions to analyze the environment surrounding vehicle 200 and navigate vehicle 200 in response to said analysis.

[0153] As discussed in further detail below and consistent with the various disclosed embodiments, system 100 can provide various features related to autonomous driving and / or driver assistance technologies. For example, system 100 can analyze image data, location data (e.g., GPS positioning information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 can collect data for analysis from, for example, image acquisition unit 120, position sensor 130, and other sensors. Furthermore, system 100 can analyze the collected data to determine whether vehicle 200 should take certain actions, and then automatically take those actions without human intervention. For example, when vehicle 200 is navigating without human intervention, system 100 can automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle system 220, braking system 230, and steering system 240). Additionally, system 100 can analyze the collected data and issue warnings and / or alerts to vehicle occupants based on the analysis of the collected data. Further details regarding various embodiments provided by system 100 are provided below.

[0154] Forward multiple imaging system As discussed above, system 100 can provide driver assistance functions using a multi-camera system. The multi-camera system can use one or more cameras facing forward toward the vehicle. In other embodiments, the multi-camera system may include one or more cameras facing the side or rear of the vehicle. In one embodiment, for example, system 100 may use a dual-camera imaging system, wherein a first camera and a second camera (e.g., image capture devices 122 and 124) may be positioned at the front and / or side of the vehicle (e.g., vehicle 200). Other camera configurations are consistent with the disclosed embodiments, and the configurations disclosed herein are examples. For example, system 100 may include any number of cameras (e.g., one, two, three, four, five, six, seven, eight, etc.). Furthermore, system 100 may include a “cluster” of cameras. For example, a cluster of cameras (including any suitable number of cameras, e.g., one, four, eight, etc.) may be forward-facing relative to the vehicle, or may be oriented in any other direction (e.g., rearward, sideways, angled, etc.). Therefore, system 100 may include a cluster of multiple cameras, each oriented in a specific direction to capture images from a specific area of ​​the vehicle environment.

[0155] The first camera may have a field of view larger than, smaller than, or partially overlapping with that of the second camera. Furthermore, the first camera may be connected to a first image processor for monocular image analysis of the images provided by the first camera, and the second camera may be connected to a second image processor for monocular image analysis of the images provided by the second camera. The outputs of the first and second image processors (e.g., processed information) may be combined. In some embodiments, the second image processor may receive images from both the first and second cameras for stereoscopic analysis. In another embodiment, system 100 may use a three-camera imaging system, wherein each of the cameras has a different field of view. Therefore, such a system can make decisions based on information derived from objects positioned at varying distances, both in front of and to the side of the vehicle. A reference to monocular image analysis may refer to an instance where image analysis is performed based on an image captured from a single viewpoint (e.g., from a single camera). Stereoscopic image analysis may refer to an instance where image analysis is performed based on two or more images captured using one or more variations of image capture parameters. For example, captured images suitable for stereo image analysis may include: images captured from two or more different locations, images captured from different fields of view, images captured using different focal lengths and parallax information, etc.

[0156] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122 to 126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or other values ​​selected from the range of about 20 to 45 degrees), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other values ​​selected from the range of about 100 to about 180 degrees), and image capture device 126 may provide an intermediate field of view (e.g., 46 degrees or other values ​​selected from the range of about 35 to about 60 degrees). In some embodiments, image capture device 126 may act as a primary or main camera. Image capture devices 122-126 may be positioned behind rearview mirror 310 and substantially side-by-side (e.g., spaced 6 cm apart). Furthermore, in some embodiments, as discussed above, one or more of image capture devices 122 to 126 may be mounted behind a glare shield 380 flush with the windshield of vehicle 200. This mask can minimize the impact of any reflections from inside the car on the image capture devices 122 to 126.

[0157] In another embodiment, such as in combination Figure 3B and 3CAs discussed, a wide field-of-view camera (e.g., image capture device 124 in the examples above) can be mounted below the narrow field-of-view and main field-of-view cameras (e.g., image devices 122 and 126 in the examples above). This configuration provides a free line of sight from the wide field-of-view camera. To reduce reflections, the camera can be mounted close to the windshield of the vehicle 200 and may include a polarizer on the camera to reduce reflected light.

[0158] A three-camera system can provide certain performance characteristics. For example, some embodiments may include verifying the ability of one camera to detect an object based on detection results from another camera. In the three-camera configuration discussed above, processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips, as discussed above), wherein each processing device is dedicated to processing images captured by one or more of the image capture devices 122-126.

[0159] In a three-camera system, the first processing unit can receive images from both the main camera and the narrow field-of-view (FOV) camera, and perform visual processing on the narrow FOV camera to detect, for example, other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Furthermore, the first processing unit can calculate the pixel parallax between the images from the main camera and the narrow camera and create a 3D reconstruction of the vehicle's environment. Then, the first processing unit can combine the 3D reconstruction with 3D... Data or a combination of 3D information calculated based on information from another camera.

[0160] The second processing unit can receive images from the main camera and perform visual processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the second processing unit can calculate camera displacement and, based on said displacement, calculate pixel parallax between consecutive images and create a 3D reconstruction of the scene (e.g., from moving structures). The second processing unit can then send the structure from the motion-based 3D reconstruction to the first processing unit for combination with the stereoscopic 3D image.

[0161] The third processing unit can receive images from a wide field of view (FOV) camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing unit can further execute additional processing instructions to analyze the images to identify moving objects in the images, such as vehicles changing lanes or pedestrians.

[0162] In some embodiments, enabling image-based information streams to be captured and processed independently can provide opportunities for redundancy in the system. Such redundancy may include, for example, using a first image capture device and images processed from said device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0163] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) to provide navigation assistance to vehicle 200, and a third image capture device (e.g., image capture device 126) to provide redundancy and verify the analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for stereo analysis by system 100 for navigation of vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and verification based on information obtained from images captured by image capture devices 122 and / or 124. That is, image capture device 126 (and corresponding processing device) may be considered to provide a redundant subsystem to provide checks on the analyses derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system). Furthermore, in some embodiments, the redundancy and verification of the received data can be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors), information received from one or more transceivers outside the vehicle, etc.

[0164] Those skilled in the art will recognize that the camera configurations, placements, number of cameras, and positioning described above are merely exemplary. These components, and other components described relative to the overall system, can be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functionality are as follows.

[0165] Figure 4 An exemplary functional block diagram of a memory 140 and / or 150, consistent with the disclosed embodiments, that can store / program instructions for performing one or more operations. Although memory 140 is mentioned below, those skilled in the art will recognize that instructions can be stored in memory 140 and / or 150.

[0166] like Figure 4As shown, memory 140 may store monocular image analysis module 402, stereo image analysis module 404, rate and acceleration module 406, and navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of the modules 402-408 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may individually or collectively refer to application processor 180 and image processor 190. Therefore, steps in any of the following processes may be performed by one or more processing devices.

[0167] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform monocular image analysis on a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensor information (e.g., information from radar) to perform monocular image analysis. Figures 5A-5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on this analysis, system 100 (e.g., via processing unit 110) may induce one or more navigation responses in vehicle 200, such as turning, lane changing, or changes in acceleration, as discussed below in conjunction with navigation response module 408.

[0168] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform monocular image analysis on a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensor information (e.g., information from radar, lidar, etc.) to perform monocular image analysis. (See also...) Figures 5A-5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on this analysis, system 100 (e.g., via processing unit 110) may induce one or more navigation responses in vehicle 200, such as turning, lane changing, changes in acceleration, etc., as discussed below in conjunction with determining the navigation response.

[0169] In one embodiment, the stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform stereo image analysis on a first set of images and a second set of images acquired by a combination of image capture devices selected from any of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the first set of images and the second set of images with additional sensing information (e.g., information from radar) to perform stereo image analysis. For example, the stereo image analysis module 404 may include instructions for performing stereo image analysis based on the first set of images acquired by the image capture device 124 and the second set of images acquired by the image capture device 126. (The following is a continuation of the previous paragraph.) Figure 6 As described, the stereo image analysis module 404 may include instructions for detecting a set of features (such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, etc.) within a first set of images and a second set of images. Based on this analysis, the processing unit 110 may induce one or more navigation responses (such as turning, lane changing, changes in acceleration) in the vehicle 200, as discussed below in conjunction with the navigation response module 408. Furthermore, in some embodiments, the stereo image analysis module 404 may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems.

[0170] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computing and electromechanical devices in the vehicle 200, which are configured to cause changes in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target speed for the vehicle 200 based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include, for example, target position, speed and / or acceleration, position and / or speed of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects, position information of the vehicle 200 relative to lane markings on the road, etc. In addition, the processing unit 110 may calculate the target speed for the vehicle 200 based on sensor inputs (e.g., information from radar) and inputs from other systems of the vehicle 200 (such as the vehicle 200's throttle system 220, braking system 230, and / or steering system 240). Based on the calculated target rate, the processing unit 110 can transmit electronic signals to the throttle system 220, braking system 230 and / or steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration by, for example, physically pressing down the brakes or releasing the accelerator of the vehicle 200.

[0171] In one embodiment, the navigation response module 408 may store software executable by the processing unit 110 to determine the desired navigation response based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and rate information associated with nearby vehicles, pedestrians, and road objects, target position information for vehicle 200, etc. Additionally, in some embodiments, the navigation response may be (partially or entirely) based on map data, the predetermined position of vehicle 200, and / or the relative velocity or relative acceleration between vehicle 200 and one or more objects detected from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine the desired navigation response based on sensor inputs (e.g., information from radar) and inputs from other systems of vehicle 200 (such as the throttle system 220, braking system 230, and steering system 240 of vehicle 200). Based on the desired navigation response, the processing unit 110 can transmit electronic signals to the throttle system 220, braking system 230, and steering system 240 of the vehicle 200 to trigger the desired navigation response by, for example, turning the steering wheel of the vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, the processing unit 110 can use the output of the navigation response module 408 (e.g., the desired navigation response) as input to the execution of the speed and acceleration module 406 to calculate the change in the rate of the vehicle 200.

[0172] Furthermore, any module disclosed herein (e.g., modules 402, 404, and 406) may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems.

[0173] Figure 5A A flowchart illustrating an exemplary process 500A for inducing one or more navigation responses based on monocular image analysis, consistent with the disclosed embodiments, is provided. At step 510, processing unit 110 may receive multiple images via data interface 128 between processing unit 110 and image acquisition unit 120. For example, a camera included in image acquisition unit 120 (such as image capture device 122 with field of view 202) may capture multiple images of an area in front of vehicle 200 (or, for example, the side or rear of vehicle) and transmit the multiple images to processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). At step 520, processing unit 110 may execute monocular image analysis module 402 to analyze the multiple images, as described below. Figures 5B-5D Further details are provided. By performing the analysis, the processing unit 110 can detect a set of features within the group of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, etc.

[0174] At step 520, processing unit 110 may also execute monocular image analysis module 402 to detect various road hazards, such as truck tire components, fallen road signs, loose cargo, small animals, etc. Road hazards can have different structures, shapes, sizes, and colors, which can make the detection of such hazards more challenging. In some embodiments, processing unit 110 may execute monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera motion between consecutive image frames and calculate pixel parallax between frames to construct a 3D map of the road. Processing unit 110 can then use the 3D map to detect hazards on the road surface and above the road surface.

[0175] At step 530, processing unit 110 may execute navigation response module 408 based on the analysis performed at step 520 and as described above. Figure 4 The described techniques induce one or more navigation responses in vehicle 200. Navigation responses may include, for example, turning, lane changing, changes in acceleration, etc. In some embodiments, processing unit 110 may induce one or more navigation responses using data derived from the execution of speed and acceleration module 406. Additionally, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof. For example, processing unit 110 may induce vehicle 200 to change lanes and then accelerate by, for example, sequentially transmitting control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 may induce vehicle 200 to brake while changing lanes by, for example, simultaneously transmitting control signals to braking system 230 and steering system 240 of vehicle 200.

[0176] Figure 5B A flowchart illustrating an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a set of images, consistent with the disclosed embodiments, is provided. Processing unit 110 may execute monocular image analysis module 402 to implement process 500B. At step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify possible locations within each image that may contain the object of interest (e.g., a vehicle, pedestrian, or a portion thereof). The predetermined patterns may be designed to achieve a high “false hit” rate and a low “miss” rate. For example, processing unit 110 may use a low threshold of similarity to the predetermined pattern to identify candidate objects as possible vehicles or pedestrians. Doing so allows processing unit 110 to reduce the probability of missing (e.g., not identifying) candidate objects representing vehicles or pedestrians.

[0177] At step 542, processing unit 110 may filter a set of candidate objects based on classification criteria to exclude certain candidates (e.g., irrelevant or less relevant objects). Such criteria can be derived from various attributes associated with object types stored in a database (e.g., a database stored in memory 140). Attributes may include object shape, size, texture, location (e.g., relative to vehicle 200), etc. Therefore, processing unit 110 may use one or more sets of criteria to reject erroneous candidates from a set of candidate objects.

[0178] At step 544, processing unit 110 may analyze multiple image frames to determine whether an object in a set of candidate objects represents a vehicle and / or a pedestrian. For example, processing unit 110 may track detected candidate objects across consecutive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). Additionally, processing unit 110 may estimate parameters for the detected objects and compare the frame-by-frame position data of the objects with predicted positions.

[0179] At step 546, processing unit 110 can construct a set of measurements for the detected object. Such measurements may include, for example, position, velocity, and acceleration values ​​(relative to vehicle 200) associated with the detected object. In some embodiments, processing unit 110 can construct the measurements based on estimation techniques using a series of time-based observations such as a Kalman filter or linear quadratic estimation (LQE) and / or based on available modeling data for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on a measurement of the object's scale, where the scale measurement is proportional to the collision time (e.g., the amount of time it takes for vehicle 200 to arrive at the object). Thus, by performing steps 540 through 546, processing unit 110 can identify vehicles and pedestrians appearing within a set of captured images and derive information associated with the vehicles and pedestrians (e.g., position, speed, size). Based on the identification and derived information, processing unit 110 can induce one or more navigation responses in vehicle 200, as described above. Figure 5A As described.

[0180] At step 548, processing unit 110 may perform optical flow analysis on one or more images to reduce the probability of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to, for example, analyzing motion patterns relative to vehicle 200 in one or more images associated with other vehicles and pedestrians, and said motion patterns are different from road surface motion. Processing unit 110 can calculate the motion of candidate objects by observing different positions of objects across multiple image frames captured at different times. Processing unit 110 can use position and time values ​​as inputs to a mathematical model to calculate the motion of candidate objects. Therefore, optical flow analysis can provide another method for detecting vehicles and pedestrians near vehicle 200. Processing unit 110 may combine steps 540 to 546 to perform optical flow analysis to provide redundancy for detecting vehicles and pedestrians and increase the reliability of system 100.

[0181] Figure 5C A flowchart illustrating an exemplary process 500C for detecting road markings and / or lane geometry information in a set of images, consistent with the disclosed embodiments, is provided. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect segments of lane markings, lane geometry information, and other relevant road markings, processing unit 110 may filter the group of objects to exclude those determined to be irrelevant (e.g., potholes, small rocks, etc.). At step 552, processing unit 110 may group segments detected in step 550 that belong to the same road marking or lane marking together. Based on this grouping, processing unit 110 may develop a model, such as a mathematical model, representing the detected segments.

[0182] At step 554, processing unit 110 may construct a set of measurements associated with the detected segment. In some embodiments, processing unit 110 may create a projection of the detected segment from the image plane to the real-world plane. This projection may be characterized using a cubic polynomial with coefficients corresponding to physical properties such as the detected road position, slope, curvature, and derivative of curvature. In generating the projection, processing unit 110 may consider changes in the road surface and the pitch and roll rates associated with vehicle 200. Furthermore, processing unit 110 may model the road elevation by analyzing position and motion cues present on the road surface. Additionally, processing unit 110 may estimate the pitch and roll rates associated with vehicle 200 by tracking a set of feature points in one or more images.

[0183] At step 556, processing unit 110 can perform multi-frame analysis, for example, by tracking detected segments across consecutive image frames and accumulating frame-by-frame data associated with the detected segments. As processing unit 110 performs multi-frame analysis, the set of measurements constructed at step 554 can become more reliable and correlated with increasingly higher confidence levels. Therefore, by performing steps 550 to 556, processing unit 110 can identify road markings appearing within a set of captured images and derive lane geometry information. Based on the identified and derived information, processing unit 110 can induce one or more navigation responses in vehicle 200, as described above. Figure 5A As described.

[0184] At step 558, processing unit 110 may consider additional information sources to further develop a safety model for scenarios involving vehicle 200 in its surrounding environment. Processing unit 110 may use the safety model to define scenarios in which system 100 can safely perform autonomous control of vehicle 200. To develop the safety model, in some embodiments, processing unit 110 may consider the positions and movements of other vehicles, detected road edges and barriers, and / or general road shape descriptions extracted from map data (such as data from map database 160). By considering additional information sources, processing unit 110 can provide redundancy for detecting road markings and lane geometry features and increase the reliability of system 100.

[0185] Figure 5D A flowchart illustrating an exemplary process 500D for detecting traffic lights in a set of images, consistent with the disclosed embodiments, is provided. Processing unit 110 may execute monocular image analysis module 402 to implement process 500D. At step 560, processing unit 110 may scan the set of images and identify objects appearing in the images at locations that may contain traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects, excluding those objects that are unlikely to correspond to traffic lights. This filtering may be done based on various attributes associated with traffic lights, such as shape, size, texture, location (e.g., relative to vehicle 200), etc. Such attributes may be based on multiple instances of traffic lights and traffic control signals and stored in a database. In some embodiments, processing unit 110 may perform multi-frame analysis on a set of candidate objects reflecting possible traffic lights. For example, processing unit 110 may track candidate objects across consecutive image frames, estimate the real-world location of the candidate objects, and filter out those objects that are moving (which are unlikely to be traffic lights). In some embodiments, the processing unit 110 may perform color analysis on candidate objects and identify the relative positions of detected colors that appear inside possible traffic lights.

[0186] At step 562, processing unit 110 may analyze the geometric features of the intersection. This analysis may be based on any combination of: (i) the number of lanes detected on either side of vehicle 200, (ii) markings detected on the road (such as arrow markings), and (iii) a description of the intersection extracted from map data (such as data from map database 160). Processing unit 110 may use information derived from the execution of monocular analysis module 402 for this analysis. In addition, processing unit 110 may determine the correspondence between traffic lights detected at step 560 and lanes appearing near vehicle 200.

[0187] At step 564, as vehicle 200 approaches the intersection, processing unit 110 can update the confidence level associated with the analyzed intersection geometry and detected traffic lights. For example, the estimated number of traffic lights at the intersection may affect the confidence level compared to the actual number present at the intersection. Therefore, based on the confidence level, processing unit 110 can delegate control to the driver of vehicle 200 to improve safety conditions. By performing steps 560 to 564, processing unit 110 can identify traffic lights appearing in the set of captured images and analyze intersection geometry information. Based on the identification and analysis, processing unit 110 can trigger one or more navigation responses in vehicle 200, as described above. Figure 5A As described.

[0188] Figure 5E A flowchart illustrating an exemplary process 500E for inducing one or more navigation responses in vehicle 200 based on a vehicle path, consistent with the disclosed embodiments, is provided. At step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path can be represented using coordinates (…). x , z The set of points is represented by a number of points, and the distance between any two points in the set is... d i The distance can fall within the range of 1 to 5 meters. In one embodiment, processing unit 110 can use two polynomials (such as a left-road polynomial and a right-road polynomial) to construct an initial vehicle path. Processing unit 110 can calculate the geometric midpoint between the two polynomials and, if applicable, include an offset of a predetermined amount (e.g., a smart lane offset) at each point in the resulting vehicle path (zero offset may correspond to driving in the middle of the lane). The offset may be along a direction perpendicular to the segment between any two points in the vehicle path. In another embodiment, processing unit 110 can use a polynomial and an estimated lane width to offset each point of the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0189] At step 572, processing unit 110 may update the vehicle path constructed at step 570. Processing unit 110 may use a higher resolution to reconstruct the vehicle path constructed at step 570, such that the distance between two points in the set of points representing the vehicle path is... d k Less than the distance mentioned above d i For example, distance d k The distance can fall within the range of 0.1 meters to 0.3 meters. Processing unit 110 can use a parabolic spline algorithm to reconstruct the vehicle path, which generates a cumulative distance vector corresponding to the total length of the vehicle path. S (That is, based on a set of points representing the vehicle path).

[0190] At step 574, processing unit 110 may determine the look-ahead point (represented in coordinates) based on the updated vehicle path constructed at step 572. x l , z l The processing unit 110 can process the accumulated distance vector. S A look-ahead point is extracted, and this look-ahead point can be associated with a look-ahead distance and a look-ahead time. The look-ahead distance (which can have a lower bound in the range of 10 to 20 meters) can be calculated as the product of the vehicle 200's speed and the look-ahead time. For example, as the vehicle 200's speed decreases, the look-ahead distance can also decrease (e.g., until it reaches the lower bound). The look-ahead time (which can be in the range of 0.5 to 1.5 seconds) can be inversely proportional to the gain of one or more control loops (such as a heading error tracking control loop) associated with causing the navigation response in the vehicle 200. For example, the gain of the heading error tracking control loop can depend on the bandwidth of the yaw rate loop, the steering actuator loop, the vehicle's lateral dynamics, etc. Therefore, the higher the gain of the heading error tracking control loop, the shorter the look-ahead time.

[0191] At step 576, processing unit 110 can determine the heading error and yaw rate commands based on the look-ahead point determined at step 574. Processing unit 110 can do this by calculating the arctangent of the look-ahead point, for example, arctan( x l / z l The heading error is determined by the processing unit 110. The processing unit 110 can determine the yaw rate command as the product of the heading error and the high-level control gain. If the lookout distance is not at the lower bound, the high-level control gain can be equal to: (2 / lookout time). Otherwise, the high-level control gain can be equal to: (2 * vehicle 200 speed / lookout distance).

[0192] Figure 5F A flowchart is provided to illustrate an exemplary process 500F for determining whether a vehicle ahead is changing lanes, consistent with the disclosed embodiments. At step 580, processing unit 110 may determine navigation information associated with the vehicle ahead (e.g., a vehicle traveling in front of vehicle 200). For example, processing unit 110 may use the above-described combination of... Figure 5A and 5B The described technique determines the position, speed (e.g., direction and velocity), and / or acceleration of a vehicle ahead. Processing unit 110 may also use the techniques described above. Figure 5E The described technique determines one or more road polynomials, look-ahead points (associated with vehicle 200), and / or snail tracks (e.g., a set of points describing the path taken by the vehicle ahead).

[0193] At step 582, processing unit 110 may analyze the navigation information determined at step 580. In one embodiment, processing unit 110 may calculate the distance between the snail track and the road polynomial (e.g., along the track). If the variance of this distance along the track exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters on a straight road, 0.3 to 0.4 meters on a moderately curved road, and 0.5 to 0.6 meters on a road with sharp curves), processing unit 110 may determine that the vehicle ahead may be changing lanes. In the case of multiple vehicles detected traveling in front of vehicle 200, processing unit 110 may compare the snail track associated with each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose snail track does not match the snail tracks of other vehicles may be changing lanes. Processing unit 110 may additionally compare the curvature of the snail track (associated with the vehicle ahead) with the expected curvature of the road segment the vehicle ahead is traveling on. The expected curvature can be extracted from map data (e.g., data from map database 160), from road polynomials, from the snail tracks of other vehicles, from prior knowledge about the road, etc. If the difference between the curvature of the snail track and the expected curvature of the road segment exceeds a predetermined threshold, the processing unit 110 can determine that the vehicle ahead may be changing lanes.

[0194] In another embodiment, processing unit 110 may compare the instantaneous position of the vehicle ahead with a look-ahead point (associated with vehicle 200) over a specific time period (e.g., 0.5 to 1.5 seconds). If the distance between the instantaneous position of the vehicle ahead and the look-ahead point changes over the specific time period, and the cumulative sum of the changes exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a moderately curved road, and 1.3 to 1.7 meters on a road with sharp curves), processing unit 110 may determine that the vehicle ahead may be changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the track by comparing the lateral distance traveled along the snail's path with the expected curvature of the snail's path. The expected radius of curvature can be calculated based on (δ... z 2 + δ x 2 ) / 2 / (δ x ) is determined, where δ x δ represents the lateral distance traveled. z This represents the longitudinal distance traveled. If the difference between the lateral distance traveled and the expected curvature exceeds a predetermined threshold (e.g., 500 meters to 700 meters), processing unit 110 can determine that the vehicle ahead may be changing lanes. In another embodiment, processing unit 110 can analyze the position of the vehicle ahead. If the position of the vehicle ahead obscures the road polynomial (e.g., the vehicle ahead is covered on top of the road polynomial), processing unit 110 can determine that the vehicle ahead may be changing lanes. If the position of the vehicle ahead is such that another vehicle is detected in front of the vehicle ahead and the snail tracks of the two vehicles are not parallel, processing unit 110 can determine that the (closer) vehicle ahead may be changing lanes.

[0195] At step 584, processing unit 110 may determine whether the vehicle 200 ahead is changing lanes based on the analysis performed at step 582. For example, processing unit 110 may make the determination based on a weighted average of the various analyses performed at step 582. In such an approach, for example, a decision made by processing unit 110 based on a particular type of analysis that the vehicle ahead may be changing lanes may be assigned a value "1" (and "0" to indicate that the vehicle ahead is unlikely to be changing lanes). Different analyses performed at step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analysis and weights. Furthermore, in some embodiments, the analysis may utilize a trained system (e.g., a machine learning or deep learning system) that can estimate the future path ahead of the vehicle's current location, for example, based on images captured at the current location.

[0196] Figure 6A flowchart illustrating an exemplary process 600 for evoking one or more navigation responses based on stereoscopic image analysis, consistent with the disclosed embodiments, is provided. At step 610, processing unit 110 may receive a first plurality of images and a second plurality of images via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) may capture the first plurality of images and the second plurality of images of an area in front of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first plurality of images and the second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0197] At step 620, processing unit 110 may execute stereo image analysis module 404 to perform stereo image analysis on the first plurality of images and the second plurality of images to create a 3D map of the road in front of the vehicle and detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. Stereo image analysis can be performed in conjunction with the above. Figures 5A-5D The steps described are performed in a similar manner. For example, processing unit 110 may execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road markings, traffic lights, road hazards, etc.) in a first plurality of images and a second plurality of images, filter subsets of candidate objects based on various criteria, and perform multi-frame analysis to construct measurement results and determine confidence levels for the remaining candidate objects. In performing the steps described above, processing unit 110 may consider information from both the first plurality of images and the second plurality of images, rather than information from a single set of images. For example, processing unit 110 may analyze the differences in pixel-level data (or other subsets of data from the two streams of captured images) of candidate objects appearing in both the first plurality of images and the second plurality of images. As another example, processing unit 110 may estimate the position and / or velocity (e.g., relative to vehicle 200) of an object by observing that a candidate object appears in one of the plurality of images but not in the other, or relative to other differences that may exist relative to objects appearing in the two image streams. For example, the position, velocity, and / or acceleration relative to vehicle 200 can be determined based on the trajectory, position, motion characteristics, etc., of features associated with one or both objects appearing in the image stream.

[0198] At step 630, processing unit 110 may execute navigation response module 408 based on the analysis performed at step 620 and as described above. Figure 4The described techniques are used to induce one or more navigation responses in vehicle 200. Navigation responses may include, for example, turning, lane changing, changes in acceleration, changes in speed, braking, etc. In some embodiments, processing unit 110 may use data derived from the execution of speed and acceleration module 406 to induce one or more navigation responses. Additionally, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof.

[0199] Figure 7 A flowchart illustrating an exemplary process 700 consistent with the disclosed embodiments for evoking one or more navigation responses based on the analysis of three sets of images is provided. At step 710, processing unit 110 may receive a first plurality of images, a second plurality of images, and a third plurality of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture the first plurality of images, the second plurality of images, and the third plurality of images of areas in front of and / or to the sides of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first plurality of images, the second plurality of images, and the third plurality of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface for transmitting data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0200] In step 720, processing unit 110 can analyze the first plurality of images, the second plurality of images, and the third plurality of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. The analysis can be performed in conjunction with the above. Figures 5A-5D and Figure 6 The steps described are performed in a similar manner. For example, processing unit 110 can perform monocular image analysis on each of the first plurality of images, the second plurality of images, and the third plurality of images (e.g., via execution by monocular image analysis module 402 and based on the above combination). Figures 5A-5D The steps described above). Alternatively, processing unit 110 may perform stereoscopic image analysis on the first plurality of images and the second plurality of images, the second plurality of images and the third plurality of images and / or the first plurality of images and the third plurality of images (e.g., via execution by stereoscopic image analysis module 404 and based on the above). Figure 6The steps described herein. Processed information corresponding to the analysis of a first plurality of images, a second plurality of images, and / or a third plurality of images can be combined. In some embodiments, processing unit 110 may perform a combination of monocular image analysis and stereoscopic image analysis. For example, processing unit 110 may perform monocular image analysis on the first plurality of images (e.g., via execution of monocular image analysis module 402) and stereoscopic image analysis on the second plurality of images and the third plurality of images (e.g., via execution of stereoscopic image analysis module 404). The configuration of image capture devices 122, 124, and 126—including their respective positioning and fields of view 202, 204, and 206—can affect the type of analysis performed on the first plurality of images, the second plurality of images, and the third plurality of images. The disclosed embodiments are not limited to a specific configuration of image capture devices 122, 124, and 126, or the type of analysis performed on the first plurality of images, the second plurality of images, and the third plurality of images.

[0201] In some embodiments, processing unit 110 may test system 100 based on the images acquired and analyzed at steps 710 and 720. Such testing may provide an indicator of the overall performance of system 100 for certain configurations of image capture devices 122, 124, and 126. For example, processing unit 110 may determine the ratio of “false hits” (e.g., situations where system 100 incorrectly determines the presence of a vehicle or pedestrian) and “missed hits”.

[0202] At step 730, processing unit 110 may induce one or more navigation responses in vehicle 200 based on information derived from both of the first plurality of images, the second plurality of images, and the third plurality of images. The selection of either the first plurality of images, the second plurality of images, or the third plurality of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the plurality of images. Processing unit 110 may also base its responses on image quality and resolution, the effective field of view reflected in the images, the number of captured frames, the extent to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the proportion of objects appearing in each such frame, etc.), etc.

[0203] In some embodiments, processing unit 110 can select information derived from both of the first plurality of images, the second plurality of images, and the third plurality of images by determining the degree to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 can combine processed information derived from each of image capture devices 122, 124, and 126 (whether by monocular analysis, stereo analysis, or any combination of both) and determine consistent visual indicators (e.g., lane markings, detected vehicles and their locations and / or paths, detected traffic lights, etc.) across the images captured from each of image capture devices 122, 124, and 126. Processing unit 110 can also exclude inconsistent information across the captured images (e.g., vehicles changing lanes, lane models indicating vehicles too close to vehicle 200, etc.). Therefore, processing unit 110 can select information derived from both of the first plurality of images, the second plurality of images, and the third plurality of images based on the determination of consistent and inconsistent information.

[0204] Navigation responses may include, for example, turning, lane changing, and changes in acceleration. Processing unit 110 can base its responses on the analysis performed in step 720 and the above-mentioned factors. Figure 4 The described techniques are used to induce one or more navigation responses. Processing unit 110 may also use data derived from the execution of velocity and acceleration module 406 to induce one or more navigation responses. In some embodiments, processing unit 110 may induce one or more navigation responses based on the relative position, relative velocity, and / or relative acceleration between vehicle 200 and objects detected in any of a first plurality of images, a second plurality of images, and a third plurality of images. Multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof.

[0205] Reinforcement learning and trained navigation systems Subsequent chapters discuss autonomous driving and the systems and methods for achieving autonomous control of vehicles, whether that control is fully autonomous (self-driving vehicle) or partially autonomous (e.g., one or more driver assistance systems or functions). Figure 8As shown, the autonomous driving task can be divided into three main modules: a sensing module 801, a driving strategy module 803, and a control module 805. In some embodiments, modules 801, 803, and 805 may be stored in memory units 140 and / or 150 of system 100, or modules 801, 803, and 805 (or portions thereof) may be stored remotely from system 100 (e.g., in a server accessible to system 100 via, for example, wireless transceiver 172). Furthermore, any module disclosed herein (e.g., modules 801, 803, and 805) may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems.

[0206] The sensing module 801, implemented using processing unit 110, can handle various tasks related to sensing the navigation state of the host vehicle's environment. These tasks may rely on inputs from various sensors and sensing systems associated with the host vehicle. These inputs may include images or image streams from one or more onboard cameras, GPS positioning information, accelerometer outputs, user feedback, or user input to one or more user interface devices, radar, lidar, etc. Sensing data, including data from cameras and / or any other available sensors, as well as map information, can be collected, analyzed, and configured into a "sensing state" that describes information extracted from the scene in the host vehicle's environment. The sensed state may include sensed information related to target vehicles, lane markings, pedestrians, traffic lights, road geometry, lane shape, obstacles, distance to other objects / vehicles, relative speed, relative acceleration, and any other potential sensed information. Supervised machine learning can be implemented to generate a sensed state output based on the sensed data provided to the sensing module 801. The output of the sensing module can represent the sensed navigation "state" of the host vehicle that can be passed to the driving strategy module 803.

[0207] While a sensed state can be developed based on image data received from one or more cameras or image sensors associated with the host vehicle, any suitable sensor or combination of sensors can be used to develop a sensed state for navigation. In some embodiments, a sensed state can be developed without relying on captured image data. In fact, any of the navigation principles described herein can be applied to sensed states developed based on captured image data as well as sensed states developed using other non-image-based sensors. A sensed state can also be determined via sources external to the host vehicle. For example, a sensed state can be developed entirely or partially based on information received from sources remote from the host vehicle (e.g., sensor information, processed state information, etc., shared from other vehicles, shared from a central server, or from any other information source related to the navigation state of the host vehicle).

[0208] The driving strategy module 803, discussed in more detail below and implemented using processing unit 110, can implement a desired driving strategy to determine one or more navigation actions to be taken by the primary vehicle in response to the sensed navigation state. If no other agents (e.g., target vehicles or pedestrians) are present in the primary vehicle's environment, the sensed state input to the driving strategy module 803 can be handled in a relatively straightforward manner. The task becomes more complex when the sensed state requires negotiation with one or more other agents. Techniques used to generate the output of the driving strategy module 803 can include reinforcement learning (discussed in more detail below). The output of the driving strategy module 803 can include at least one navigation action for the primary vehicle and may include a desired acceleration (which can be converted into an updated velocity of the primary vehicle), a desired yaw rate of the primary vehicle, a desired trajectory, and other potential desired navigation actions.

[0209] Based on the output from the driving strategy module 803, the control module 805, which can also be implemented using the processing unit 110, can develop control commands for one or more actuators or controlled devices associated with the host vehicle. Such actuators and devices may include accelerators, one or more steering controls, brakes, signal transmitters, displays, or any other actuators or devices that can be controlled as part of navigation operations associated with the host vehicle. Aspects of control theory can be used to generate the output of the control module 805. The control module 805 can be responsible for developing commands and outputting them to controllable components of the host vehicle to implement the desired navigation objectives or requirements of the driving strategy module 803.

[0210] Returning to the driving strategy module 803, in some embodiments, a trained system trained through reinforcement learning can be used to implement the driving strategy module 803. In other embodiments, the driving strategy module 803 can be implemented without machine learning methods by using a specified algorithm to "manually" resolve various scenarios that may arise during autonomous navigation. However, while feasible, this approach may lead to an oversimplified driving strategy and may lack the flexibility of a machine learning-based trained system. A trained system can, for example, be better equipped to handle complex navigation states and better determine whether a taxi is stopped or stopped to pick up or drop off passengers; determine whether a pedestrian intends to cross the road before the main vehicle; balance unexpected behavior and defensive actions by other drivers; negotiate in dense traffic involving target vehicles and / or pedestrians; decide when to suspend certain navigation rules or enhance others; anticipate unperceived but predictable situations (e.g., whether a pedestrian will appear from behind a car or obstacle), etc. A reinforcement learning-based trained system can also be better equipped to resolve continuous and high-dimensional state spaces and continuous action spaces.

[0211] Training a system using reinforcement learning can involve learning a driving policy that maps sensed states to navigation actions. The driving policy is a function... ,in S As a set of states, and This is the action space (e.g., desired velocity, acceleration, yaw commands, etc.). The state space is... S = S s x S p ,in S s For sensing state, and S p Additional information about the state stored for the strategy. In the case of operating in discrete time intervals, in time... t At that time, the current state can be observed. Furthermore, strategies can be applied to obtain the desired action. .

[0212] The system can be trained by exposing it to various navigation states, enabling it to apply policies, and providing rewards (based on a reward function designed to reward expected navigational behavior). Based on reward feedback, the system can "learn" policies and be trained to produce the desired navigational actions. For example, the learning system can observe the current state. And based on strategy To decide the action Based on the determined action (and the execution of that action), the environment moves to the next state. This is for the learning system to observe. For each action developed in response to the observed state, the feedback to the learning system is a reward signal. .

[0213] The goal of reinforcement learning (RL) is to discover policy π. It is typically assumed that in time... t At that time, there exists a reward function. r t The reward function measurement is in a state. s t And take action a t The instantaneous mass. However, taking action at time t. a t This affects the environment and therefore the values ​​of future states. Therefore, when deciding what action to take, not only current rewards but also future rewards should be considered. In some cases, when the system determines that taking a lower-reward option now will yield a larger reward in the future, the system should take an action, even if that action is associated with a reward lower than another available option. To formalize this, observe the policy π and the initial state. s Induction The distribution on, where if the agent changes from state s 0 = s Starting from here and following strategy π, the vector ( r 1 … r T The probability of observing a reward is... r 1 … r T The probability of ( ). Initial state s The value can be defined as: .

[0214] For some fixed It can be defined by discounting future rewards, rather than limiting the time range to... T : .

[0215] In any situation, the optimal strategy is the following solution: The expected result exceeds the initial state. s .

[0216] Several possible methods exist for training driving strategy systems. For example, imitation methods (e.g., behavioral cloning) can be used, where the system learns from state / action pairs, where actions are chosen by a competent agent (e.g., a human) in response to a particular observed state. Suppose a human driver is observed. Through this observation, numerous instances of the aforementioned form can be obtained and observed (…). s t , a t ),in s t It is a state and a t These are the actions of a human driver, and they are used as the basis for training driving strategy systems. For example, supervised learning can be used to learn policy π, making... This method has many potential advantages. First, it does not require defining a reward function. Second, the learning is supervised and occurs offline (no agent needs to be applied during the learning process). The disadvantage of this method is that different human drivers, and even the same human driver, are not deterministic in their policy choices. Therefore, learning... Very small functions are often impractical. Moreover, even small errors can accumulate over time, resulting in large errors.

[0217] Another approach is policy-based learning. Here, the policy can be expressed in parametric form and directly optimized using appropriate optimization techniques (e.g., stochastic gradient descent). This method directly solves... The problem is presented in the text. Of course, there are multiple ways to solve this problem. One advantage of this method is that it directly addresses the problem and therefore often yields good practical results. A potential drawback is that it typically requires "online policy" training, meaning that the learning of π is an iterative process, where in iteration... j At that time, there is an imperfect strategy π j And in order to construct the next strategy π j It must be based on π j Interact with the environment while taking action.

[0218] The system can also be trained through value-based learning (learning the Q or V function). This is assuming the optimal value function can be learned. VA good approximation of *. Optimal policies can be constructed (e.g., by relying on the Bellman equation). Some forms of value-based learning (called "offline policy" training) can be implemented offline. Some drawbacks of value-based methods may stem from their strong reliance on Markovian assumptions and the required approximation of complex functions (approximating the value function may be more difficult than directly approximating the policy).

[0219] Another technique may include model-based learning and planning (learning the probabilities of state transitions and solving an optimization problem to find the optimal V). Combinations of these techniques can also be used to train learning systems. In this approach, the dynamics of the process can be learned, i.e., by employing ( s t , a t And generate the next state. s t+1 The distribution on the π is a function. Once this function is learned, optimization problems can be solved to discover the value of the optimal policy π. This is called "planning". One advantage of this method is that the learning part is supervised and can be learned by observing triples ( s t , a t , s t+1 This method can be used for offline applications. Similar to the "imitation" method, one drawback of this method may be that small errors during the learning process may accumulate and produce underperforming strategies.

[0220] Another approach for training the driving policy module 803 may include decomposing the driving policy function into semantically meaningful components. This allows for the manual implementation of parts of the policy, ensuring its safety, and the implementation of other parts using reinforcement learning techniques. This enables adaptation to various scenarios, a human-like balance between defensive and aggressive behavior, and human-like negotiation with other drivers. From a technical perspective, reinforcement learning methods can combine several approaches to provide tractable training procedures, with most training performed using recorded data or a self-built simulator.

[0221] In some embodiments, the training of the driving strategy module 803 can rely on an "options" mechanism. For illustration, consider a simple scenario for a driving strategy on a two-lane highway. In the direct RL method, policy π maps the state to... In this context, the first component of π(s) is the desired acceleration command, and the second component of π(s) is the yaw rate. In the modified method, the following strategy can be constructed: Automatic Cruise Control (ACC) Strategy This strategy always outputs a yaw rate of 0 and only changes the speed in order to implement smooth and accident-free driving.

[0222] ACC + Left Strategy The longitudinal command for this strategy is the same as the ACC command. Yaw rate is simply to center the vehicle in the left lane while ensuring safe lateral movement (e.g., not moving to the left if there is a car on the left).

[0223] ACC+ right-hand policy :and o L The same, but vehicles may align themselves towards the center of the right lane.

[0224] These strategies can be called "options." Using these "options," one can learn strategies for choosing options. ,in O It is a set of available options. In one situation, Option selector strategy π o By setting for each s To define the actual strategy, .

[0225] In practice, the policy function can be decomposed into option graph 901, such as... Figure 9 As shown. Figure 10 Another example, option graph 1000, is shown. An option graph can represent a hierarchical set of decisions organized as a directed acyclic graph (DAG). There exists a special node, called the root node 903 of the graph. This node has no incoming nodes. The decision process traverses the graph starting from the root node until it reaches a "leaf" node, which is a node that has no outgoing decision lines. Figure 9 As shown, leaf nodes may include, for example, nodes 905, 907, and 909. Upon encountering a leaf node, the driving strategy module 803 may output acceleration and steering commands associated with the desired navigation action (associated with the leaf node).

[0226] For example, internal nodes such as nodes 911, 913, and 915 can lead to a strategy that selects a child from its available options. This set of available children for an internal node includes all nodes associated with a particular internal node via a decision line. For example, Figure 9 The internal node 913, which is designated as "Merge", includes three sub-nodes 909, 915 and 917 ("Hold", "Overtake to the Right" and "Overtake to the Left" respectively), and each of the three sub-nodes is connected to node 913 through a decision line.

[0227] The flexibility of a decision-making system can be achieved by allowing nodes to adjust their position within the hierarchical structure of the option graph. For example, any node could be allowed to declare itself "critical." Each node can implement a "critical" function, which outputs "true" if the node is in a critical segment of its policy implementation. For instance, the node responsible for overtaking could declare itself critical during maneuvering. This could potentially benefit the nodes... u A set of available sub-sub ... u All nodes of the child v And for the node, there exists a [from] v The path to the leaf node is the path through all nodes assigned as critical. On the one hand, this approach allows the desired path on the graph to be declared at each time step, and on the other hand, it preserves the stability of the strategy, especially when implementing the critical parts of the strategy.

[0228] By defining an option graph, the problem of learning driving strategy π is: S A The problem can be decomposed into defining a policy for each node in the graph, where the policy at internal nodes should be chosen from the available child nodes. For some nodes, the corresponding policy can be implemented manually (e.g., specifying a set of actions in response to the observed state via an if-then type algorithm), while for others, the policy can be implemented using a trained system constructed through reinforcement learning. The choice between manual or trained / learned methods can depend on task-related safety aspects and the relative simplicity of the task. The options graph can be constructed in such a way that some nodes are easy to implement, while others can rely on a trained model. This approach ensures the safe operation of the system.

[0229] The following discussion provides information on Figure 9 Further details regarding the role of the option graph within the driving strategy module 803. As discussed above, the input to the driving strategy module is the "sensed state," which is summarized, for example, an environmental map obtained from available sensors. The output of the driving strategy module 803 is a set of expectations (optionally, along with a set of hard constraints) that define the trajectory as a solution to the optimization problem.

[0230] As described above, the option graph represents a hierarchical decision set organized as a DAG. There exists a special node called the “root” of the graph. The root node is a node with no incoming edges (e.g., decision lines). The decision process traverses the graph starting from the root node until it reaches a “leaf” node, i.e., a node with no outgoing edges. Each internal node should implement a policy that selects from the available children of that internal node. Each leaf node should implement a policy that defines a set of expectations (e.g., a set of navigation purposes for the primary vehicle) based on the entire path from the root to the leaf. This set of expectations, along with a set of hard constraints defined directly based on the sensed state, establishes its solution as an optimization problem for the vehicle's trajectory. The hard constraints can be used to further improve system safety, and the expectations can be used to provide driving comfort and human-like driving behavior. The trajectory provided as a solution to the optimization problem is then defined as the commands that should be provided to the steering, braking, and / or engine actuators to complete the trajectory.

[0231] Return to Figure 9 Option diagram 901 represents an option diagram for a two-lane highway, including merging lanes (meaning at some points, a third lane merges into the right or left lane of the highway). Root node 903 first determines whether the primary vehicle is in a normal road scenario or an imminent merging scenario. This is an example of a decision that can be implemented based on the sensed state. Normal road node 911 includes three child nodes: hold node 909, left overtaking node 917, and right overtaking node 915. Hold refers to the situation where the primary vehicle wishes to continue driving in the same lane. The hold node is a leaf node (without outgoing edges / lines). Therefore, the hold node defines a set of expectations. The first expectation defined by the hold node may include the desired lateral position—for example, as close as possible to the center of the current driving lane. There may also be an expectation of smooth navigation (for example, within a predefined or allowed maximum acceleration). The hold node may also define how the primary vehicle should react to other vehicles. For example, the hold node may check sensed target vehicles and assign a semantic meaning to each sensed target vehicle, which can be translated into trajectory components.

[0232] Various semantic meanings can be assigned to target vehicles in the environment of the master vehicle. For example, in some embodiments, the semantic meaning may include any of the following designations: 1) Irrelevant: indicating that the sensed vehicle is currently irrelevant in the scene; 2) Next Lane: indicating that the sensed vehicle is in an adjacent lane and should maintain an appropriate offset relative to this vehicle (the exact offset can be calculated in an optimization problem of constructing a trajectory given expectations and hard constraints, and may be a semantic type of the target vehicle set by a hold-alive leaf of an option graph that may be vehicle-related, the semantic type defining the expectations relative to the target vehicle); 3) Give Way: the master vehicle will attempt to give way to the sensed target vehicle by, for example, reducing speed (especially if the master vehicle determines that the target vehicle may cut into the master vehicle's lane); 4) Obstruction: the master vehicle will attempt to gain right-of-way by, for example, increasing speed; 5) Following: the master vehicle expects to maintain smooth driving following this target vehicle; 6) Overtake Left / Right: this means that the master vehicle wants to initiate a lane change to the left or right lane. The Overtake Left node 917 and the Overtake Right node 915 are internal nodes whose expectations have not yet been defined.

[0233] The next node in option diagram 901 is the selection interval node 919. This node is responsible for selecting the interval between two target vehicles that the primary vehicle expects to enter in a specific target lane. By selecting a node of the form IDj, for some value of j, the primary vehicle reaches the leaf of the expected trajectory optimization problem assigned, for example, the primary vehicle wants to maneuver to reach the selected interval. This maneuver may involve first accelerating / braking in the current lane, and then moving to the target lane at the appropriate time to enter the selected interval. If the selection interval node 919 cannot find a suitable interval, the selection interval node moves to the termination node 921, which defines moving back to the center of the current lane and canceling the overtaking expectation.

[0234] Returning to lane merging node 913, when the primary vehicle approaches a lane merge, the primary vehicle has several options that may depend on specific circumstances. For example, such as... Figure 11A As shown, the main vehicle 1105 is traveling along a two-lane road, and no other target vehicles are detected in the main lane or in the merging lane 1111. In this case, the driving strategy module 803 can choose to remain in the lane 909 when it reaches the merging node 913. That is, if no target vehicle is detected when merging into the roadway, it is expected to remain in its current lane.

[0235] exist Figure 11BIn the latter case, the situation is slightly different. Here, the main vehicle 1105 senses one or more target vehicles 1107 entering the main lane 1112 from the merging lane 1111. In this case, once the driving strategy module 803 encounters the merging node 913, the driving strategy module can choose to initiate a left overtaking maneuver to avoid the merging situation.

[0236] exist Figure 11C In this scenario, the primary vehicle 1105 encounters one or more target vehicles 1107 entering the main lane 1112 from the merging lane 1111. The primary vehicle 1105 also detects target vehicles 1109 traveling in lanes adjacent to its own. The primary vehicle also detects one or more target vehicles 1110 traveling in the same lane as the primary vehicle 1105. In this situation, the driving strategy module 803 can decide to adjust the speed of the primary vehicle 1105 to give way to target vehicles 1107 and continue driving ahead of target vehicles 1115. This can be accomplished, for example, by proceeding to a selection interval node 919, which then selects the interval between ID0 (vehicle 1107) and ID1 (vehicle 1115) as the appropriate merging interval. In this case, the appropriate interval for the merging situation defines the objective of the trajectory planner optimization problem.

[0237] As discussed above, nodes in the option graph can declare themselves as "critical," ensuring that the selected option passes through critical nodes. Formally, each node can implement the function IsCritical. After performing a forward pass from root to leaf on the option graph and solving the trajectory planner's optimization problem, a backward pass can be performed from leaf back to root. Along this backward pass, the IsCritical function of all nodes in the pass can be invoked, and a list of all critical nodes can be maintained. In the forward path corresponding to the next time frame, the driving strategy module 803 may need to select a path from the root node to a leaf that passes through all critical nodes.

[0238] Figure 11A-11CThis can be used to illustrate the potential benefits of this approach. For example, when initiating an overtaking maneuver and the driving strategy module 803 reaches the leaf corresponding to IDk, selecting, for example, a hold node 909 is undesirable while the master vehicle is in the process of overtaking. To avoid this jump, the IDj node can be assigned as critical. During the maneuver, the success of the trajectory planner can be monitored, and if the overtaking maneuver progresses as expected, the IsCritical function will return a "true" value. This approach ensures that the overtaking maneuver continues in the next time frame (rather than jumping to another potentially inconsistent maneuver before completing the initially selected maneuver). On the other hand, if the monitoring of the maneuver indicates that the selected maneuver is not progressing as expected, or if the maneuver has become unnecessary or impossible, the IsCritical function can return a "false" value. This allows the selection interval node to choose a different interval in the next time frame, or to terminate the overtaking maneuver entirely. On the one hand, this approach allows the desired path on the option graph to be declared at each time step; on the other hand, it helps improve the stability of the strategy in the critical parts of execution.

[0239] The hard constraints, discussed in more detail below, can be distinguished from navigation expectations. For example, hard constraints can ensure safe driving by applying an additional filtering layer to planned navigation actions. The hard constraints involved can be determined based on sensed states, and these constraints can be manually programmed and defined rather than through a trained system constructed using reinforcement learning. However, in some embodiments, the trained system can learn which applicable hard constraints to apply and follow. This approach can facilitate the driving strategy module 803 reaching actions that already conform to the applicable hard constraints, reducing or eliminating actions that might require later modification to conform to the applicable hard constraints. However, as a redundancy safety measure, hard constraints can also be applied to the output of the driving strategy module 803 even if it has been trained to take into account predetermined hard constraints.

[0240] There are many instances of potential hard constraints. For example, a hard constraint can be defined in conjunction with a guardrail at the edge of a road. Under no circumstances should a primary vehicle be allowed to cross the guardrail. This rule imposes a hard lateral constraint on the trajectory of the primary vehicle. Another instance of a hard constraint can include road bumps (e.g., speed control bumps), which may impose hard constraints on the driving speed before and when crossing the bump. Hard constraints can be considered safety-critical and therefore can be defined manually rather than solely relying on a trained system that learns the constraints during training.

[0241] In contrast to hard constraints, the desired outcome can be to achieve or attain comfortable driving. As discussed above, an example of a desired outcome could include positioning the primary vehicle in a lateral position within the lane corresponding to the center of the lane. Another desired outcome could include the ID of the interval to be matched. It should be noted that it is not required that the primary vehicle be precisely centered in the lane, but rather that the desired outcome of being as close as possible to the center of the lane ensures that the primary vehicle tends to migrate to the center of the lane even if it is off-center. The desired outcome may not be safety-critical. In some embodiments, the desired outcome may require negotiation with other drivers and pedestrians. One method for constructing the desired outcome could rely on an option graph, and the policies implemented at at least some nodes of the graph could be based on reinforcement learning.

[0242] For nodes in option graph 901 or 1000 that are implemented as learning-based training nodes, the training process may include decomposing the problem into a supervised learning phase and a reinforcement learning phase. In the supervised learning phase, the node can learn from (… s t ， a t )to Differentiable mappings, such that 。 This can be similar to "model-based" reinforcement learning. However, in the positive loop of the network, It can be s t+1 The actual value is replaced, thus eliminating the problem of error accumulation. The predictive function is to propagate information from the future back to the past. In this sense, the algorithm can be a combination of "model-based" reinforcement learning and "policy-based learning".

[0243] An important element that can be provided in some scenarios is the differentiable path from future loss / reward back to the decision about the action. Using an option graph structure, the implementation of options involving safety constraints is often not differentiable. To overcome this problem, the selection of children in the learned policy nodes can be randomized. That is, the nodes can output probability vectors. p The probability vector assigns a probability to each child in the selection of a particular node. Suppose a node has k children, and let... This represents the action along the path from each child to the leaf. Therefore, the predicted action is... This can produce a transition from action to... p Differentiable paths. In practice, for i ~ p, action a Can be selected as a (i) ,and a and The difference between them can be called additive noise.

[0244] For a given s t 、a t of Supervised learning training can be used with real-world data. For training, a policy from a node simulator can be used. Later, real-world data can be used to fine-tune the policy. Two ideas can make the simulation more realistic. First, through imitation, a "behavioral cloning" paradigm can be used to build an initial policy using a large real-world dataset. In some cases, the resulting agent may be suitable. In other cases, the resulting agent will at least form a very good initial policy for other agents on the road. Second, using self-play, we can use our own policies to enhance training. For example, given the initial implementation by other agents (cars / pedestrians) that might be encountered, the policy can be trained based on a simulator. Some of these other agents can be replaced with new policies, and the process can be repeated. Therefore, the policy can be continuously improved because it should respond to a wider variety of other agents with different levels of sophistication.

[0245] Furthermore, in some embodiments, the system can implement a multi-agent approach. For example, the system can consider data from various sources and / or images captured from multiple angles. Additionally, some disclosed embodiments can provide energy economy because the anticipation of events that do not directly involve the primary vehicle but may affect it, or even the anticipation of unpredictable events that could lead to situations involving other vehicles, can be taken into consideration (e.g., radar can "see through" vehicles ahead and anticipate unavoidable events, or even anticipate a high probability of events that will affect the primary vehicle).

[0246] A trained system with imposed navigation constraints In the context of autonomous driving, a significant concern is ensuring that the learned strategies of trained navigation networks are safe. In some embodiments, constraints can be used to train the driving strategy system so that actions selected by the trained system may already take into account applicable safety constraints. Additionally, in some embodiments, an extra layer of safety can be provided by subjecting the actions selected by the trained system to one or more hard constraints related to a specific sensed scenario in the host vehicle's environment. This approach ensures that actions taken by the host vehicle are limited to those determined to satisfy applicable safety constraints.

[0247] At its core, the navigation system may include a policy function-based learning algorithm that maps observed states to one or more desired actions. In some implementations, the learning algorithm is a deep learning algorithm. Desired actions may include at least one action expected to maximize a anticipated reward for the vehicle. While in some cases the actual action taken by the vehicle may correspond to one of the desired actions, in others the actual action taken may be determined based on the observed state, one or more desired actions, and non-learned hard constraints (e.g., safety constraints) imposed on the learning navigation engine. These constraints may include a non-driving zone around various types of detected objects (e.g., target vehicles, pedestrians, stationary objects on the side of the road or in the carriageway, moving objects on the side of the road or in the carriageway, guardrails, etc.). In some cases, the size of the zone may vary based on the detected motion (e.g., speed and / or direction) of the detected object. Other constraints may include maximum speed when passing within the pedestrian influence zone, maximum deceleration (to account for the distance between the target vehicle and the main vehicle), mandatory stopping at sensed pedestrian crossings or railway crossings, etc.

[0248] Hard constraints used in conjunction with systems trained through machine learning can provide a level of safety in autonomous driving that may exceed the level of safety available solely based on the output of the trained system. For example, a desired set of constraints can be used as a training guide to train a machine learning system, and thus, the trained system can select actions in response to sensed navigation states that take into account and comply with restrictions on applicable navigation constraints. However, the trained system still retains some flexibility in selecting navigation actions, and therefore, there may be at least some situations where the actions selected by the trained system may not strictly adhere to the relevant navigation constraints. Therefore, to require selected actions to strictly adhere to the relevant navigation constraints, non-machine learning components outside the learning / training framework that guarantees strict application of the relevant navigation constraints can be used to combine, compare, filter, adjust, modify, etc., the output of the trained system.

[0249] The following discussion provides additional details about trained systems and the potential benefits (particularly from a security perspective) that can be obtained by combining trained systems with algorithmic components outside of trained / learning frameworks. As discussed, the reinforcement learning objective of the policy can be optimized using stochastic gradient ascent. The objective (e.g., expected reward) can be defined as... .

[0250] The intended purpose can be used in machine learning scenarios. However, without being bound by navigational constraints, such a purpose may not return the action strictly bound by those constraints. For example, consider a reward function, for which... This represents the trajectory of rare "dilemma" events (e.g., accidents) that need to be avoided, and Representing the remaining trajectories, one goal of the learning system could be to learn overtaking maneuvers. Typically, in accident-free trajectories, Successful, smooth overtaking will be rewarded, while failing to overtake while remaining in the lane will be penalized—therefore the range is [-1, 1]. If the sequence Reporting an accident will result in a reward. The punishment should be high enough to deter this from happening. The problem is... What value should be used to ensure accident-free driving?

[0251] The accident was observed to have an impact on The impact is an additional item ,in p Let be the probabilistic quality of the trajectory of the accident event. If this term is negligible, i.e., If the probability of an accident is at most 100%, the learning system may prefer a strategy that leads to accidents (or generally adopts a reckless driving strategy) to successfully perform overtaking maneuvers more frequently than a more cautious strategy, which at the cost of failing to complete some overtaking maneuvers. In other words, if the probability of an accident is at most 100%, then the learning system may prefer a strategy that leads to accidents (or generally adopts a reckless driving strategy) to successfully perform overtaking maneuvers more frequently than a more cautious strategy, which may fail to complete some overtaking maneuvers. p ,but It must be set to make It may be expected that... p Extremely small (e.g., approximately) p = 10 -9 ).therefore, It should be relatively large. Regarding the policy gradient, it can be estimated... The gradient of the random variable. The following lemma shows that the gradient of the random variable. The variance with Growth, for In other words, the variance is greater than Therefore, estimating the objective may be difficult, and estimating its gradient may be even more difficult.

[0252] Lemma: Let π o As a strategy, and to make p and r Let be a scalar, such that in probability p In the case of obtaining And with a probability of 1- p In the case of obtaining .but, The last approximation value applies to the case r ≥ 1 / p.

[0253] This discussion shows that, regarding form Objections may not be able to ensure functional safety without introducing variance problems. Baseline subtraction methods used for variance reduction may not provide sufficient remedy for the problem, as it will stem from... The high variance transformation of its estimate will also suffer from the same high variance of the baseline constant, which is also numerically unstable. Furthermore, if the probability of the accident is... p Therefore, on average, at least 1 / p Each sequence is sampled. This means that the aim is to make The lower bound of the minimization learning algorithm is 1 / p The solution to this problem can be found in the architectural design described in this paper, rather than through numerical tuning techniques. The approach here is based on the concept that hard constraints should be injected outside the learning framework. In other words, the policy function can be decomposed into learnable and non-learnable parts. Formally, the policy function can be structured as follows: ,in Map the (agnostic) state space to a set of desires (e.g., desired navigational purpose, etc.), Mapping expectations to a trajectory (which determines how the car should move over short distances). Function It is responsible for driving comfort and making strategic decisions, such as which other vehicles to overtake or yield to and the desired position of the main vehicle within its lane. The mapping from the sensed navigation state to the desired outcome is the strategy. The strategy can be learned from experience by maximizing the expected reward. The resulting expectation can be transformed into a cost function along the driving trajectory. (Not a learned function) This can be implemented by discovering the trajectory that minimizes cost under hard constraints on functional safety. This decomposition ensures functional safety while providing a comfortable driving experience.

[0254] like Figure 11D The depicted dual lane merging navigation scenario provides an example further illustrating these concepts. In dual lane merging, vehicles approach merging area 1130 from both the left and right sides. Vehicles such as vehicle 1133 or vehicle 1135 can decide from each side whether to merge into the lane on the other side of merging area 1130. Successfully executing dual lane merging in busy traffic may require considerable negotiation skills and experience, and may be difficult to perform heuristically or brute-force by enumerating all possible trajectories that can be taken by all agents in the scenario. In this dual lane merging example, a set of expectations suitable for dual lane merging maneuvering can be defined. . It can be the Cartesian product of the following sets: , where [0,v max Let ] be the desired target speed of the main vehicle, and L = {1, 1.5, 2, 2.5, 3, 3.5, 4} be the desired lateral position in lane units, where integers indicate the lane center and decimals indicate the lane boundaries, and { g, t, o} is assigned to n The classification label for each of the other vehicles. Other vehicles can be assigned a "classification label" if the primary vehicle must yield to another vehicle. g "When the main vehicle needs to occupy the lane relative to other vehicles, it will be allocated" t "or assigned when the primary vehicle needs to maintain an offset distance relative to other vehicles." o ".

[0255] The following is about how to express a set of expectations This is transformed into a description of the cost function along the driving trajectory. The driving trajectory can be described by... It means that, among them It is in time The (lateral, longitudinal) position of the master vehicle (the egocentric unit). In some experiments, and k = 10. Of course, other values ​​can also be chosen. The cost assigned to the trajectory can include a weighted sum of individual costs assigned to: desired velocity, lateral position, and costs assigned to other factors. n The labeling of each vehicle in the vehicle group.

[0256] Given the expected speed The cost of the trajectory associated with speed is .

[0257] Given the desired lateral position The cost associated with the desired lateral position is dist( x, y, l) is the distance from point (x, y) to the lane position. l The distance. Regarding costs incurred due to other vehicles, for any other vehicle, It can represent another vehicle in the egocentric unit of the main vehicle, and i can be an existing entity. j Make ( x i , y i )and( x' j , y' j The earliest point with the smaller distance between them. If no such point exists, theni It can be set to i = ∞. If another car is classified as "yielding", then it can be expected that... This means that the primary vehicle will arrive at the trajectory intersection at least 0.5 seconds after the other vehicle arrives at the same point. A possible formula to translate the above constraints into cost is... .

[0258] Similarly, if another car is classified as "obstructing the road," then one can expect... This can be translated into costs. If another car is classified as "offset", then one can expect... i = ∞, meaning the trajectory of the main vehicle and the trajectory of the offset car do not intersect. This condition can be translated into a cost by penalizing the distance between the trajectories.

[0259] Assigning weights to each of these costs can provide a single objective function for the trajectory planner. The cost of encouraging smooth driving can be added to the objective. Furthermore, to ensure the functional safety of the trajectory, hard constraints can be added to the objective. For example, if | ij If the value is smaller, then for any trajectory point of any other vehicle ( x' j , y' j ), can prohibit ( x i, y i )Leave the roadway and may prohibit ( x i, y i )near( x' j , y' j ).

[0260] In summary, strategy It can be decomposed into a mapping from irrelevant states to a set of expectations and a mapping from expectations to actual trajectories. The latter mapping is not based on learning and can be implemented by solving an optimization problem whose cost depends on the expectations and whose hard constraints guarantee the functional safety of the policy.

[0261] The following discussion describes the mapping from irrelevant states to the set of desired outcomes. As described above, to meet functional safety requirements, systems relying solely on reinforcement learning may suffer from rewards. The high and difficult-to-handle changes. This result can be avoided by using policy gradient iteration to decompose the problem into a mapping from the (irrelevant) state space to a set of expectations, and then mapping to the actual trajectory of the system that is not involved in machine learning-based training.

[0262] For various reasons, decision-making can be further broken down into semantically meaningful components. For example, The size can be large, and may even be continuous. (See above regarding...) Figure 11D In the described dual lane-changing scenario, Additionally, gradient estimators can involve terms In such expressions, variance can vary over a time range. T And growth. In some cases, T The value can be approximately 250, which may be high enough to produce significant variance. If the sampling rate is in the range of 10 Hz and the merging zone 1130 is 100 meters, preparation for merging can begin approximately 300 meters before the merging zone. If the main vehicle is traveling at 16 m / s (approximately 60 km / h), then the round... T The value can be approximately 250.

[0263] Returning to the concept of the options diagram, Figure 11E The text shows what can be represented. Figure 11D The depicted dual-path scenario is represented by an option graph. As previously discussed, an option graph can represent a hierarchical set of decisions organized as a directed acyclic graph (DAG). A special node, called the "root" node 1140, may exist in the graph; this special node can be a node with no incoming edges (e.g., decision lines). The decision process can traverse the graph starting from the root node until it reaches a "leaf" node, i.e., a node with no outgoing edges. Each internal node can implement a policy function that selects a child from its available children. There can be a set of traversals on the option graph leading to the desired set. The predefined mapping. In other words, traversal on the option graph can be automatically transformed into... The expected outcome. Given the nodes in the graph. v parameter vector You can specify the selection v The strategy of the child. If Ө It is all Ө v The cascade, then This can be defined as follows: traversing from the root of the graph to a leaf, while at each node... v Used by Ө v Define a strategy to select child nodes.

[0264] exist Figure 11EIn the dual lane merging option diagram 1139, the root node 1140 can first determine whether the primary vehicle is in the lane merging area (e.g., Figure 11D Within area 1130, or whether the primary vehicle is adjacent to the merging area and needs to prepare for a possible lane change. In both cases, the primary vehicle may need to decide whether to change lanes (e.g., to the left or right) or to remain in its current lane. If the primary vehicle has decided to change lanes, it may need to determine whether conditions are suitable to continue and perform a lane change maneuver (e.g., at “pass” node 1142). If changing lanes is not possible, the primary vehicle may (e.g., at node 1144, as part of negotiation with vehicles in the desired lane) attempt to “advance” toward the desired lane with the aim of being on the lane markings. Alternatively, the primary vehicle may choose to “remain” in the same lane (e.g., at node 1146). This process can determine the lateral position of the primary vehicle in a natural way. For example, This allows the desired lateral position to be determined naturally. For example, if the master vehicle changes lanes from lane 2 to lane 3, the "Pass" node can set the desired lateral position to 3, the "Hold" node can set the desired lateral position to 2, and the "Propel" node can set the desired lateral position to 2.5. Next, the master vehicle can decide whether to maintain the "same" speed (node ​​1148), "accelerate" (node ​​1150), or "decelerate" (node ​​1152). Then, the master vehicle can enter a "chain" structure 1154, which passes through other vehicles and sets its semantic meaning to a set { g, t, o The values ​​in}. This process can set expectations relative to other vehicles. Parameters can be shared across all nodes in this chain (similar to a recurrent neural network).

[0265] A potential benefit of the options is the interpretability of the results. Another potential benefit is the ability to rely on sets. The decomposable structure allows for the selection of a policy at each node from a limited number of possibilities. Furthermore, this structure can reduce the variance of the policy gradient estimator.

[0266] As discussed above, the length of a turn in a dual lane-changing scenario can be approximately T = 250 steps. This value (or any other suitable value depending on the specific navigation scenario) provides sufficient time to observe the consequences of the primary vehicle's actions (e.g., if the primary vehicle decides to change lanes in preparation for a lane change, the primary vehicle will only see the benefit after successfully completing the lane change). On the other hand, due to the dynamics of driving, the primary vehicle must make decisions at a sufficiently high frequency (e.g., 10 Hz in the case described above).

[0267] Option diagrams can be implemented in at least two ways.T The effective value is reduced. First, given the higher-level decision-making, rewards can be defined for lower-level decisions, while considering shorter rounds. For example, when the primary vehicle has already selected the "lane change" and "pass" nodes, a 2-3 second observation period can be observed (meaning...). T Instead of 250, the primary vehicle learns a strategy for assigning semantic meaning to vehicles (by changing the number of options to 20-30). Secondly, for high-level decisions (such as changing lanes or staying in the same lane), the primary vehicle may not need to make a decision every 0.1 seconds. Instead, the primary vehicle can be able to make decisions at a lower frequency (e.g., once per second), or implement an "option termination" function, and then only compute gradients after each option termination. In both cases, T The effective values ​​of each node may be an order of magnitude smaller than their original values. In summary, the estimator at each node can depend on... T The value, which is an order of magnitude smaller than the original 250 steps, may instantly shift to a smaller variance.

[0268] As discussed above, hard constraints can promote safer driving and can exist in several different types. For example, static hard constraints can be defined directly from the sensing state. These can include speed bumps, speed limits, road curvature, intersections, etc., in the host vehicle's environment that may impose one or more constraints on vehicle speed, heading, acceleration, braking (deceleration), etc. Static hard constraints can also include semantic free space, such as prohibiting the host vehicle from leaving free space and prohibiting the navigating host vehicle from getting too close to physical barriers. Static hard constraints can also restrict (e.g., prohibit) various maneuvers that do not conform to the vehicle's kinematics; for example, static hard constraints can be used to prohibit maneuvers that may cause the host vehicle to roll over, skid, or otherwise lose control.

[0269] Hard constraints can also be associated with vehicles. For example, constraints could require a vehicle to maintain a longitudinal distance of at least one meter and a lateral distance of at least 0.5 meters from other vehicles. Constraints can also be applied to prevent the primary vehicle from maintaining a collision path with one or more other vehicles. For example, time τ can be a time measure based on a specific scenario. Predicted trajectories of the primary vehicle and one or more other vehicles from the current time to time τ can be considered. In the case of intersecting trajectories, It can represent a vehicle i Arrival and departure times at the intersection. That is, each car arrives at the intersection when the first part of the car passes the intersection, and a certain amount of time is required before the last part of the car passes the intersection. This amount of time separates the arrival and departure times. Assume... (That is, if vehicle 1 arrives at a later time than vehicle 2), then we need to ensure that vehicle 1 has left the intersection before vehicle 2 arrives. Otherwise, a collision will occur. Therefore, a hard constraint can be implemented to ensure that... Furthermore, to ensure that vehicles 1 and 2 do not miss each other by a minimum, an additional safety margin (e.g., 0.5 seconds or another suitable value) can be obtained by including a buffer time in the constraints. The hard constraints associated with the predicted intersection trajectories of the two vehicles can be expressed as follows: .

[0270] The amount of time τ that tracks the trajectories of the primary vehicle and one or more other vehicles can vary. However, in intersection scenarios where speeds may be low, τ can be longer, and τ can be defined as such that the primary vehicle will enter and leave the intersection in less than τ seconds.

[0271] Of course, applying hard constraints to vehicle trajectories requires predicting the trajectories of these vehicles. For the master vehicle, trajectory prediction can be relatively simple, as it typically already understands and is actually planning its expected trajectory for any given time. Predicting the trajectories of other vehicles may be less straightforward. For other vehicles, the baseline calculations used to determine the predicted trajectories can rely on their current speed and heading, for example, based on analysis of image streams captured by one or more cameras and / or other sensors (radar, lidar, acoustic sensors, etc.) on the master vehicle.

[0272] However, there can be exceptions that can simplify the problem or at least provide additional confidence in the predicted trajectory of another vehicle. For example, in structured roads where lane markings exist and yield rules may be in place, the trajectories of other vehicles can be based at least in part on their positions relative to the lanes and on applicable yield rules. Therefore, in some cases, when an observed lane structure exists, it can be assumed that the vehicle in the next lane will adhere to lane boundaries. That is, the primary vehicle may assume that the vehicle in the next lane will remain in its lane unless there is observed evidence (e.g., traffic lights, strong lateral movement, movement across lane boundaries) indicating that the vehicle in the next lane will cut into the primary vehicle's lane.

[0273] Other situations can also provide clues about the expected trajectories of other vehicles. For example, at stop signs, traffic lights, roundabouts, etc., where the primary vehicle may have the right-of-way, it can be assumed that other vehicles will obey this right-of-way. Therefore, unless there is observed evidence of rule violations, it can be assumed that other vehicles are proceeding along the trajectory of those who obey the primary vehicle's right-of-way.

[0274] Hard constraints can also be applied to pedestrians in the environment surrounding the main vehicle. For example, a buffer distance can be established for pedestrians, preventing the main vehicle from navigating closer than a predetermined buffer distance relative to any observed pedestrian. The pedestrian buffer distance can be any suitable distance. In some embodiments, the buffer distance can be at least one meter relative to an observed pedestrian.

[0275] Similar to the situation with vehicles, hard constraints can also be applied to the relative motion between pedestrians and main vehicles. For example, a pedestrian's trajectory (based on heading and speed) can be monitored relative to the expected trajectory of the main vehicle. Given a specific pedestrian trajectory, for each point p on the trajectory, t(p) can represent the time required for the pedestrian to reach point p. To maintain the required buffer distance of at least 1 meter from the pedestrian, t(p) must be greater than the time it takes for the main vehicle to reach point p (with a sufficient time difference so that the main vehicle passes in front of the pedestrian by at least 1 meter) or t(p) must be less than the time it takes for the main vehicle to reach point p (e.g., if the main vehicle brakes to yield to the pedestrian). In the latter example, the hard constraint can still require the main vehicle to arrive at point p a sufficient time later than the pedestrian so that the main vehicle can pass behind the pedestrian and maintain the required buffer distance of at least 1 meter. Of course, exceptions to the pedestrian hard constraints can also exist. For example, in cases where the primary vehicle has the right-of-way or is traveling very slowly and there is no observed evidence that the pedestrian will refuse to give way to the primary vehicle or will otherwise walk toward the primary vehicle, the pedestrian hard constraint may be relaxed (e.g., relaxed to a smaller buffer zone of at least 0.75 meters or 0.50 meters).

[0276] In some instances, constraints may be relaxed when it is determined that not all constraints can be met. For example, if a road is too narrow to allow the desired clearance (e.g., 0.5 meters) between two curbs or between the curb and parked vehicles, one or more constraints may be relaxed if mitigating circumstances exist. For example, if there are no pedestrians (or other objects) on the sidewalk, one may proceed slowly at a distance of 0.1 meters from the curb. In some embodiments, constraints may be relaxed if doing so improves the user experience. For example, to avoid potholes, constraints may be relaxed to allow vehicles to navigate closer to the edge of the lane, curb, or pedestrians than normally permitted. Furthermore, when determining which constraints to relax, in some embodiments, one or more constraints are selected for relaxation that are considered to have the least effective negative impact on safety. For example, constraints related to how close a vehicle can travel to the curb or concrete barrier may be relaxed before relaxing constraints on proximity to other vehicles. In some embodiments, pedestrian constraints may be the last constraints to be relaxed, or in some cases may never be relaxed.

[0277] Figure 12Examples of scenes that can be captured and analyzed during navigation of a primary vehicle are shown. For example, the primary vehicle may include a navigation system as described above (e.g., system 100) that can receive multiple images representing the environment of the primary vehicle from cameras associated with the primary vehicle (e.g., at least one of image capture device 122, image capture device 124, and image capture device 126). Figure 12 The scene shown is available in time. t An instance of an image captured from the environment of a primary vehicle traveling along a predicted trajectory 1212 in lane 1210. The navigation system may include at least one processing device (e.g., including an EyeQ processor or any of the other devices described above) specifically programmed to receive multiple images and analyze them to determine actions in response to the scene. Specifically, the at least one processing device may implement a sensing module 801, a driving strategy module 803, and a control module 805, such as... Figure 8 As shown. The sensing module 801 can be responsible for collecting and outputting image information collected from the camera, and providing the information to the driving strategy module 803 in the form of a recognized navigation state. The driving strategy module can constitute a trained navigation system that has been trained through machine learning techniques (such as supervised learning, reinforcement learning, etc.). Based on the navigation state information provided to the driving strategy module 803 by the sensing module 801, the driving strategy module 803 (e.g., by implementing the option graph method described above) can generate the desired navigation actions for the main vehicle to perform in response to the recognized navigation state.

[0278] In some embodiments, at least one processing device may use, for example, control module 805 to directly translate the desired navigation action into a navigation command. However, in other embodiments, hard constraints may be applied to test the desired navigation action provided by driving strategy module 803 against various predetermined navigation constraints that may be involved in the scene and the desired navigation action. For example, if driving strategy module 803 outputs a desired navigation action that would cause the master vehicle to follow trajectory 1212, this navigation action may be tested against one or more hard constraints associated with various aspects of the master vehicle's environment. For example, captured image 1201 may reveal curbs 1213, pedestrians 1215, target vehicles 1217, and stationary objects (e.g., overturned boxes) present in the scene. Each of these may be associated with one or more hard constraints. For example, curb 1213 may be associated with a static constraint that prevents the master vehicle from navigating into or across the curb onto sidewalk 1214. The curb 1213 can also be associated with a road barrier envelope, which defines a distance (e.g., a buffer strip) extending away from (e.g., 0.1 m, 0.25 m, 0.5 m, 1 m, etc.) the curb and along it, the distance being defined as a non-navigation strip for primary vehicles. Of course, static constraints can also be associated with other types of roadside boundaries (e.g., guardrails, concrete posts, traffic cones, towers, or any other type of roadside barrier).

[0279] It should be noted that distance and ranging can be determined by any suitable method. For example, in some embodiments, distance information can be provided by an onboard radar and / or lidar system. Alternatively or additionally, distance information can be derived from the analysis of one or more images captured from the environment of the host vehicle. For example, the number of pixels representing an identified object in an image can be determined and compared with the known field of view and focal length geometry of the image capturing device to determine scale and distance. Velocity and acceleration can be determined, for example, by observing changes in the scale between objects between images within a known time interval. This analysis can indicate the direction of movement toward or away from the host vehicle and how quickly an object moves away from or toward the host vehicle. Traverse velocity can be determined by analyzing changes in the X-coordinate position of an object from one image to another within a known time period.

[0280] Pedestrian 1215 can be associated with a pedestrian envelope defining a buffer zone 1216. In some cases, the imposed hard constraints can prohibit a host vehicle from navigating within 1 meter of pedestrian 1215 (in any direction relative to the pedestrian). Pedestrian 1215 can also define the location of a pedestrian influence zone 1220. This influence zone can be associated with constraints that limit the speed of host vehicles within the influence zone. The influence zone can extend from pedestrian 1215 to 5 meters, 10 meters, 20 meters, etc. Each division of the influence zone can be associated with a different speed limit. For example, within the zone from 1 meter to 5 meters of pedestrian 1215, host vehicles can be limited to a first speed (e.g., 10 mph, 20 mph, etc.), which can be less than the speed limit in a pedestrian influence zone extending from 5 meters to 10 meters. Any division can be used for the various levels of the influence zone. In some embodiments, the first level can be narrower than 1 meter to 5 meters and can extend only from 1 meter to 2 meters. In other embodiments, the first level of the pedestrian impact zone can extend from 1 meter (the boundary of the non-navigation zone around the pedestrian) to a distance of at least 10 meters. The second level can then extend from 10 meters to at least approximately 20 meters. The second level can be associated with the maximum speed of the main vehicle, which is greater than the maximum speed associated with the first level of the pedestrian impact zone.

[0281] One or more stationary object constraints may also be involved in a scene detected in the environment of the primary vehicle. For example, in image 1201, at least one processing device may detect a stationary object, such as box 1219 present in a driveway. The detected stationary object may include at least one of various objects, such as trees, poles, road signs, or objects in the driveway. One or more predefined navigation constraints may be associated with the detected stationary object. For example, such constraints may include a stationary object envelope, wherein the stationary object envelope defines a buffer zone around the object, within which navigation of the primary vehicle may be prohibited. At least a portion of the buffer zone may extend a predetermined distance from the edge of the detected stationary object. For example, in the scene represented by image 1201, a buffer zone of at least 0.1 meters, 0.25 meters, 0.5 meters, or greater may be associated with box 1219, such that the primary vehicle will pass to the right or left of the box at least a certain distance (e.g., buffer zone distance) to avoid a collision with the detected stationary object.

[0282] Predefined hard constraints may also include one or more target vehicle constraints. For example, a target vehicle 1217 may be detected in image 1201. To ensure that the primary vehicle does not collide with the target vehicle 1217, one or more hard constraints may be employed. In some cases, the target vehicle envelope may be associated with a single buffer zone distance. For example, the buffer zone may be defined by a distance of 1 meter around the target vehicle in all directions. The buffer zone may define an area extending at least one meter from the target vehicle, into which the primary vehicle is prohibited from navigating.

[0283] However, the envelope around the target vehicle 1217 does not need to be defined by a fixed buffer distance. In some cases, the predefined hard constraints associated with the target vehicle (or any other movable object detected in the environment of the master vehicle) may depend on the orientation of the master vehicle relative to the detected target vehicle. For example, in some cases, the longitudinal buffer distance (e.g., the distance extending from the front or rear of the target vehicle toward the master vehicle—such as when the master vehicle is driving directly toward the target vehicle) may be at least one meter. The lateral buffer distance (e.g., the distance extending from either side of the target vehicle toward the master vehicle—such as when the master vehicle is traveling in the same or opposite direction as the target vehicle such that one side of the master vehicle will pass adjacent to one side of the target vehicle) may be at least 0.5 meters.

[0284] As described above, other constraints may also apply to the detection of target vehicles or pedestrians in the environment of the primary vehicle. For example, the predicted trajectories of the primary vehicle and target vehicle 1217 can be considered, and in the case where the two trajectories intersect (e.g., at intersection point 1230), hard constraints may require... or Wherein, the primary vehicle is vehicle 1 and the target vehicle 1217 is vehicle 2. Similarly, the trajectory of pedestrian 1215 (based on heading direction and speed) can be monitored relative to the expected trajectory of the primary vehicle. Given a specific pedestrian trajectory, for each point p on the trajectory, t(p) will represent the pedestrian's arrival point p (i.e., Figure 12 The time required for the main vehicle to reach point p (points 1, 2, 3, and 1) is as follows. To maintain the required buffer distance of at least 1 meter from the pedestrian, t(p) must be greater than the time it takes for the main vehicle to reach point p (with a sufficient time difference so that the main vehicle passes in front of the pedestrian by at least 1 meter) or t(p) must be less than the time it takes for the main vehicle to reach point p (e.g., if the main vehicle brakes to yield to the pedestrian). In the latter example, the hard constraint still requires the main vehicle to arrive at point p a sufficient time later than the pedestrian so that the main vehicle can pass behind the pedestrian and maintain the required buffer distance of at least 1 meter.

[0285] Other hard constraints may also be used. For example, in at least some cases, a maximum deceleration rate of the lead vehicle may be used. This maximum deceleration rate may be determined based on the detected distance to the target vehicle following the lead vehicle (e.g., using images collected from a rear-view camera). Hard constraints may include mandatory stopping at sensed pedestrian crossings or railway crossings, or other applicable constraints.

[0286] In cases where the analysis of a scenario within the host vehicle's environment indicates that one or more predefined navigation constraints may be involved, these constraints can be imposed relative to one or more planned navigation actions for the host vehicle. For example, if the analysis of a scenario causes the driving strategy module 803 to return a desired navigation action, this desired navigation action can be tested against one or more of the involved constraints. If the desired navigation action is determined to violate any aspect of the involved constraints (e.g., if the desired navigation action would carry the host vehicle within 0.7 meters of pedestrian 1215, where a predefined hard constraint requires the host vehicle to maintain a distance of at least 1.0 meter from pedestrian 1215), at least one modification to the desired navigation action can be made based on one or more predefined navigation constraints. Adjusting the desired navigation action in this way can provide an actual navigation action for the host vehicle that conforms to the constraints involved in the specific scenario detected in the host vehicle's environment.

[0287] After determining the actual navigation action for the host vehicle, the navigation action can be implemented by causing at least one adjustment to the navigation actuator of the host vehicle in response to the determined actual navigation action for the host vehicle. Such navigation actuator may include at least one of the host vehicle's steering mechanism, brake, or accelerator.

[0288] Prioritized constraints As described above, various hard constraints can be applied to navigation systems to ensure the safe operation of the primary vehicle. Constraints may include minimum safe driving distances relative to pedestrians, target vehicles, road barriers, or detected objects; maximum driving speed when passing within the influence zone of a detected pedestrian; or the maximum deceleration rate of the primary vehicle. These constraints can be imposed on trained systems trained based on machine learning (supervised, reinforcement, or combination), but they can also be useful for untrained systems (e.g., those employing algorithms to directly address anticipated situations arising from the primary vehicle's environment).

[0289] In either case, a hierarchical structure of constraints can exist. In other words, some navigation constraints can have higher priority than others. Therefore, if a situation arises where a navigation action that satisfies all the involved constraints becomes unavailable, the navigation system can determine the available navigation action that first fulfills the highest priority constraint. For example, the system might allow the vehicle to avoid a pedestrian first, even if avoiding the pedestrian would result in a collision with another vehicle or object detected in the road. In another instance, the system might allow the vehicle to drive onto the curb to avoid a pedestrian.

[0290] Figure 13 A flowchart illustrating an algorithm for implementing a hierarchical structure of constraints determined based on analysis of a scene in the host vehicle's environment is provided. For example, at step 1301, at least one processing device associated with the navigation system (e.g., an EyeQ processor, etc.) can receive multiple images representing the host vehicle's environment from a camera mounted on the host vehicle. Through analysis of one or more images representing the scene of the host vehicle's environment at step 1303, a navigation state associated with the host vehicle can be identified. For example, the navigation state might indicate that the host vehicle is traveling along a path such as... Figure 12 The scene includes a two-lane road 1210, a target vehicle 1217 moving across an intersection in front of the main vehicle, a pedestrian 1215 waiting to cross the road on which the main vehicle is traveling, an object 1219 in front of the main vehicle in its lane, and various other scene attributes.

[0291] At step 1305, one or more navigation constraints related to the navigation state of the primary vehicle can be determined. For example, after analyzing a scene represented by one or more captured images in the environment of the primary vehicle, at least one processing device can determine one or more navigation constraints related to objects, vehicles, pedestrians, etc., identified through image analysis of the captured images. In some embodiments, at least one processing device can determine at least a first predefined navigation constraint and a second predefined navigation constraint related to the navigation state, and the first predefined navigation constraint may be different from the second predefined navigation constraint. For example, the first navigation constraint may be related to one or more target vehicles detected in the environment of the primary vehicle, and the second navigation constraint may be related to pedestrians detected in the environment of the primary vehicle.

[0292] At step 1307, at least one processing device may determine the priority associated with the constraints identified in step 1305. In the described example, a second predefined navigation constraint associated with a pedestrian may have a higher priority than a first predefined navigation constraint associated with a target vehicle. While the priority associated with navigation constraints can be determined or assigned based on various factors, in some embodiments, the priority of a navigation constraint may be related to its relative importance from a safety perspective. For example, while it may be important to comply with or satisfy all implemented navigation constraints as many times as possible, some constraints may be associated with a greater safety risk than others and may therefore be assigned a higher priority. For example, a navigation constraint requiring the primary vehicle to maintain a distance of at least 1 meter from a pedestrian may have a higher priority than a constraint requiring the primary vehicle to maintain a distance of at least 1 meter from a target vehicle. This may be because a collision with a pedestrian may have more serious consequences than a collision with another vehicle. Similarly, maintaining space between the primary vehicle and the target vehicle may have a higher priority than constraints requiring the primary vehicle to avoid boxes in the road, drive over speed bumps at a speed below a certain level, or expose the primary vehicle occupants to no more than the maximum acceleration level.

[0293] While the driving strategy module 803 is designed to maximize safety by satisfying navigation constraints involved in a specific scenario or navigation state, in some cases, satisfying every involved constraint may be physically impossible. In such cases, the priority of each involved constraint can be used to determine which of the involved constraints should be satisfied first, as shown in step 1309. Continuing with the example above, in cases where it is impossible to satisfy both the pedestrian spacing constraint and the target vehicle spacing constraint, but only one of the constraints can be satisfied, the higher priority of the pedestrian spacing constraint may lead to satisfying this constraint before attempting to maintain the distance from the target vehicle. Therefore, under normal circumstances, at least one processing device can determine a first navigation action for the master vehicle based on the identified navigation state of the master vehicle, satisfying both the first predefined navigation constraint and the second predefined navigation constraint, where both the first predefined navigation constraint and the second predefined navigation constraint can be satisfied, as shown in step 1311. However, in other cases where all the constraints involved cannot be satisfied, at least one processing device may determine a second navigation action for the master vehicle based on the identified navigation state that satisfies the second predefined navigation constraint (i.e., the higher priority constraint) but does not satisfy the first predefined navigation constraint (which has a lower priority than the second navigation constraint), wherein both the first and second predefined navigation constraints cannot be satisfied, as shown in step 1313.

[0294] Next, at step 1315, in order to implement a determined navigation action for the main vehicle, at least one processing device may cause at least one adjustment to the navigation actuator of the main vehicle in response to a determined first navigation action or a determined second navigation action for the main vehicle. As in the previous examples, the navigation actuator may include at least one of a steering mechanism, a brake, or an accelerator.

[0295] Relaxing constraints As discussed above, navigation constraints can be imposed for safety purposes. Constraints may include a minimum safe driving distance relative to pedestrians, target vehicles, road barriers, or detected objects; a maximum driving speed when passing within the influence zone of a detected pedestrian; or a maximum deceleration rate for the primary vehicle. These constraints can be imposed in both learning and non-learning navigation systems. In some cases, these constraints can be relaxed. For example, in cases where the primary vehicle slows down or stops near a pedestrian and then proceeds slowly to convey an intention to pass by the pedestrian, the pedestrian's response can be detected from acquired images. If the pedestrian's response is to remain still or stop moving (and / or if eye contact with the pedestrian is sensed), it can be understood that the pedestrian recognizes the navigation system's intention to pass by the pedestrian. In such cases, the system can relax one or more predefined constraints and enforce less stringent constraints (e.g., allowing the vehicle to navigate within 0.5 meters of the pedestrian instead of within a more stringent 1-meter boundary).

[0296] Figure 14 A flowchart is provided for implementing control of a master vehicle based on the relaxation of one or more navigation constraints. At step 1401, at least one processing device may receive multiple images representing the environment of the master vehicle from a camera associated with the master vehicle. At step 1403, analysis of the images may enable the identification of a navigation state associated with the master vehicle. At step 1405, at least one processor may determine the navigation constraints associated with the navigation state of the master vehicle. The navigation constraints may include a first predefined navigation constraint relating to at least one aspect of the navigation state. At step 1407, analysis of the multiple images may reveal the presence of at least one navigation constraint relaxation factor.

[0297] Navigation constraint relaxation factors may include any suitable indication that at least one or more navigation constraints can be suspended, modified, or otherwise relaxed. In some embodiments, at least one navigation constraint relaxation factor may include the determination (based on image analysis) that a pedestrian's eyes are looking in the direction of the primary vehicle. In such cases, it can be safely assumed that the pedestrian is aware of the primary vehicle. Therefore, the level of confidence that the pedestrian will not make an unintended action that would cause the pedestrian to move into the path of the primary vehicle may be high. Other constraint relaxation factors may also be used. For example, at least one navigation constraint relaxation factor may include: a pedestrian determined to be stationary (e.g., a pedestrian presumed to be unlikely to enter the path of the primary vehicle); or a pedestrian determined to be slowing down. Navigation constraint relaxation factors may also include more complex actions, such as a pedestrian determined to be stationary after the primary vehicle has stopped and then resumed movement. In this case, it can be assumed that the pedestrian understands that the primary vehicle has the right-of-way, and that the pedestrian's stopping may indicate the pedestrian's intention to yield to the primary vehicle. Other circumstances that may lead to a relaxation of one or more constraints include the type of curb (e.g., a low curb or a curb with a gentle slope may allow for a relaxation of distance constraints), the lack of pedestrians or other objects on the sidewalk, vehicles whose engines are not running may have a relaxed distance, or situations in which pedestrians are moving away from and / or away from the area that the main vehicle is facing.

[0298] If, for example, a navigation constraint relaxation factor is identified (at step 1407), a second navigation constraint can be determined or developed in response to the detection of the constraint relaxation factor. This second navigation constraint may differ from the first navigation constraint and may include at least one characteristic that is relaxed relative to the first navigation constraint. The second navigation constraint may include a newly generated constraint based on the first constraint, wherein the newly generated constraint includes at least one modification that relaxes the first constraint in at least one aspect. Alternatively, the second constraint may constitute a predetermined constraint that is less stringent than the first navigation constraint in at least one aspect. In some embodiments, such a second constraint may be reserved only for cases where a constraint relaxation factor is identified in the environment of the primary vehicle. Whether the second constraint is newly generated or selected from a set of fully or partially available predetermined constraints, applying the second navigation constraint in place of the more stringent first navigation constraint (which may be applied even without the detection of a relevant navigation constraint relaxation factor) can be termed constraint relaxation and can be completed in step 1409.

[0299] If at least one constraint relaxation factor is detected at step 1407 and at least one constraint has been relaxed in step 1409, a navigation action for the master vehicle can be determined at step 1411. The navigation action for the master vehicle can be based on the identified navigation state and can satisfy a second navigation constraint. The navigation action can be implemented at step 1413 by inducing at least one adjustment to the navigation actuator of the master vehicle in response to the determined navigation action.

[0300] As discussed above, the use of navigation constraints and relaxed navigation constraints can be employed in conjunction with trained (e.g., through machine learning) or untrained navigation systems (e.g., systems programmed to respond with predetermined actions in response to specific navigation states). In the case of a trained navigation system, the availability of relaxed navigation constraints for certain navigation situations can represent a mode switch from trained system responses to untrained system responses. For example, a trained navigation network can determine the initial navigation action for the primary vehicle based on a first navigation constraint. However, the action taken by the vehicle can be different from the navigation action that satisfies the first navigation constraint. Instead, the action taken can satisfy a more relaxed second navigation constraint and can be an action developed by an untrained system (e.g., as a response to the detection of a specific condition in the primary vehicle's environment, such as the presence of a navigation constraint relaxation factor).

[0301] There are numerous instances of navigation constraints that can be relaxed in response to the detection of a constraint relaxation factor in the environment of the master vehicle. For example, where the predefined navigation constraints include a buffer zone associated with a detected pedestrian, and at least a portion of the buffer zone extends a certain distance from the detected pedestrian, the relaxed navigation constraints (newly developed, self-recalled from a predetermined set, or generated as a relaxation of a pre-existing constraint) can include different or modified buffer zones. For example, the different or modified buffer zone may have a distance relative to the pedestrian that is smaller than the distance of the original or unmodified buffer zone relative to the detected pedestrian. Thus, given the relaxed constraints, the master vehicle can be allowed to navigate closer to the detected pedestrian, where an appropriate constraint relaxation factor is detected in the environment of the master vehicle.

[0302] The relaxed characteristics of navigation constraints may include a reduced width in the buffer zone associated with at least one pedestrian, as noted above. However, the relaxed characteristics may also include a reduced width in the buffer zone associated with: a target vehicle, a detected object, a roadside barrier, or any other object detected in the environment of the primary vehicle.

[0303] At least one relaxed characteristic may also include other types of modifications to the navigation constraint characteristics. For example, a relaxed characteristic may include an increase in velocity associated with at least one predefined navigation constraint. A relaxed characteristic may also include an increase in the maximum permissible deceleration / acceleration associated with at least one predefined navigation constraint.

[0304] While constraints can be relaxed in certain situations, as described above, navigation constraints can be strengthened in others. For example, in some cases, the navigation system can determine conditions that guarantee an enhanced set of navigation constraints. This enhancement may include adding new constraints to a predefined set or adjusting one or more aspects of predefined constraints. Additions or adjustments may produce more conservative navigation relative to the predefined set of constraints applicable under normal driving conditions. Conditions that guarantee constraint enhancement may include sensor malfunctions, adverse environmental conditions (rain, snow, fog, or other conditions associated with reduced visibility or reduced vehicle traction), etc.

[0305] Figure 15 A flowchart is provided to implement control of a master vehicle based on enhancements to one or more navigation constraints. At step 1501, at least one processing device may receive multiple images representing the environment of the master vehicle from a camera associated with the master vehicle. At step 1503, analysis of the images may enable the identification of a navigation state associated with the master vehicle. At step 1505, at least one processor may determine navigation constraints associated with the navigation state of the master vehicle. Navigation constraints may include a first predefined navigation constraint relating to at least one aspect of the navigation state. At step 1507, analysis of the multiple images may reveal the presence of at least one navigation constraint enhancement factor.

[0306] The navigation constraints involved may include those mentioned above (e.g., relative to...). Figure 12This refers to any one or any other suitable navigation constraint discussed. Navigation constraint enhancement factors may include any indication that can supplement / enhance one or more navigation constraints in at least one respect. Supplementation or enhancement of navigation constraints may be performed on a set-by-set basis (e.g., by adding a new navigation constraint to a predetermined set of constraints), or on a constraint-by-constraint basis (e.g., modifying a particular constraint to make the modified constraint more restrictive than the original constraint, or adding a new constraint corresponding to a predetermined constraint, wherein the new constraint is more restrictive than the corresponding constraint in at least one respect). Alternatively or additionally, supplementation or enhancement of navigation constraints may refer to selection from a set of predetermined constraints based on a hierarchical structure. For example, a set of enhanced constraints may be used for selection based on whether a navigation enhancement factor is detected in or relative to the host vehicle's environment. Under normal conditions where no enhancement factor is detected, the relevant navigation constraints may be drawn from constraints applicable to normal conditions. On the other hand, in the case where one or more constraint enhancement factors are detected, the relevant constraints may be drawn from enhanced constraints generated or predefined relative to one or more enhancement factors. The enhanced constraints may be more restrictive than the corresponding constraints applicable under normal conditions in at least one respect.

[0307] In some embodiments, at least one navigation constraint enhancement factor may include (e.g., image analysis-based) detecting the presence of ice, snow, or water on the surface of a road in the environment of the primary vehicle. Such determination may be based on, for example, the detection of areas with higher reflectivity than expected on a dry roadway (e.g., indicating the presence of ice or water on the roadway); white areas on the roadway indicating the presence of snow; shadows on the roadway consistent with longitudinal grooves on the roadway (e.g., tire tracks in snow); water droplets or ice / snow particles on the windshield of the primary vehicle; or any other suitable indication of the presence of water or ice / snow on the road surface.

[0308] At least one navigation constraint enhancement factor may also include the detection of particles on the outer surface of the windshield of the host vehicle. Such particles may impair the image quality of one or more image capture devices associated with the host vehicle. While described in relation to the windshield of the host vehicle (which relates to a camera mounted behind the windshield of the host vehicle), the detection of particles on other surfaces (e.g., camera lenses or lens caps, headlight lenses, rear windshields, taillight lenses, or any other surface of the host vehicle that is visible (or detected by sensors) to the image capture devices associated with the host vehicle may also indicate the presence of a navigation constraint enhancement factor.

[0309] Navigation constraint enhancement factors can also be detected as attributes of one or more image acquisition devices. For example, a detected reduction in image quality of one or more images captured by an image capture device (e.g., a camera) associated with the primary vehicle can also constitute a navigation constraint enhancement factor. Image quality degradation can be associated with hardware failures or partial hardware failures associated with the image capture device or assemblies associated with it. Such image quality degradation can also be caused by environmental conditions. For example, the presence of smoke, fog, rain, snow, etc., in the air around the primary vehicle can also result in reduced image quality relative to roads, pedestrians, target vehicles, etc., that may be present in the environment of the primary vehicle.

[0310] Navigation constraint enhancement factors can also be related to other aspects of the primary vehicle. For example, in some cases, navigation constraint enhancement factors may include detected faults or partial faults in systems or sensors associated with the primary vehicle. Such enhancement factors may include, for example, the detection of faults or partial faults in the following: speed sensor, GPS receiver, accelerometer, camera, radar, lidar, brakes, tires, or any other system associated with the primary vehicle that may affect the primary vehicle's ability to navigate relative to navigation constraints associated with the primary vehicle's navigation state.

[0311] If, for example, at step 1507, a navigation constraint enhancement factor is identified, a second navigation constraint can be determined or developed in response to the detection of the constraint enhancement factor. This second navigation constraint may differ from the first navigation constraint and may include at least one characteristic that is enhanced relative to the first navigation constraint. The second navigation constraint may be more restrictive than the first navigation constraint because detecting constraint enhancement factors in or associated with the primary vehicle's environment may indicate that the primary vehicle may have at least one reduced navigation capability relative to normal operating conditions. Such reduced capability may include reduced road traction (e.g., ice, snow, or water on the roadway; reduced tire pressure, etc.); impaired vision (e.g., rain, snow, dust, smoke, fog, etc., reducing the quality of captured images); impaired detection capability (e.g., sensor malfunction or partial malfunction, reduced sensor performance, etc.), or any other reduction in the primary vehicle's ability to navigate in response to a detected navigation state.

[0312] If at least one constraint enhancement factor is detected at step 1507 and at least one constraint has been enhanced in step 1509, a navigation action for the master vehicle can be determined at step 1511. The navigation action for the master vehicle can be based on the identified navigation state and can satisfy a second navigation (i.e., enhanced) constraint. The navigation action can be implemented at step 1513 by causing at least one adjustment to the navigation actuator of the master vehicle in response to the determined navigation action.

[0313] As discussed, the use of navigation constraints and enhanced navigation constraints can be employed in conjunction with trained (e.g., through machine learning) or untrained navigation systems (e.g., systems programmed to respond with predetermined actions in response to specific navigation states). In the case of a trained navigation system, the availability of enhanced navigation constraints for certain navigation situations can represent a mode switch from trained system responses to untrained system responses. For example, a trained navigation network can determine the initial navigation action for the primary vehicle based on a first navigation constraint. However, the action taken by the vehicle can be different from the navigation action that satisfies the first navigation constraint. Instead, the action taken can satisfy an enhanced second navigation constraint and can be an action developed by an untrained system (e.g., as a response to the detection of specific conditions (such as the presence of factors enhancing the navigation constraints) in the primary vehicle's environment).

[0314] There are numerous instances where navigation constraints can be generated, supplemented, or enhanced in response to the detection of constraint enhancement factors in the environment of the primary vehicle. For example, where predefined navigation constraints include buffer zones associated with detected pedestrians, objects, vehicles, etc., and at least a portion of the buffer zone extends a certain distance from the detected pedestrian / object / vehicle, enhanced navigation constraints (newly developed, self-recalled from a predetermined set, or generated as an enhancement of a pre-existing constraint) can include different or modified buffer zones. For example, the distance of the different or modified buffer zone relative to the pedestrian / object / vehicle may be greater than the distance of the original or unmodified buffer zone relative to the detected pedestrian / object / vehicle. Therefore, given the enhanced constraints, the primary vehicle can be forced to navigate further away from the detected pedestrian / object / vehicle, where appropriate constraint enhancement factors are detected in or relative to the primary vehicle's environment.

[0315] At least one enhanced feature may also include other types of modifications to the navigation constraint features. For example, an enhanced feature may include a reduction in velocity associated with at least one predefined navigation constraint. An enhanced feature may also include a reduction in the maximum permissible deceleration / acceleration associated with at least one predefined navigation constraint.

[0316] Navigation based on long-range planning In some embodiments, the disclosed navigation system can determine one or more navigation actions not only in response to a detected navigation state in the environment of the host vehicle, but also based on long-range planning. For example, the system can consider the potential impact of one or more navigation actions, which are available as options for navigating relative to the detected navigation state, on future navigation states. Considering the impact of available actions on future states allows the navigation system to determine navigation actions not only based on the currently detected navigation state but also based on long-range planning. Navigation using long-range planning techniques is particularly suitable for situations where the navigation system employs one or more reward functions as techniques for selecting navigation actions from available options. Potential rewards can be analyzed relative to available navigation actions that can be taken in response to the detected current navigation state of the host vehicle. However, furthermore, potential rewards can also be analyzed relative to actions that can be taken in response to future navigation states expected to result from available actions for the current navigation state. Therefore, in some cases, the disclosed navigation system can select a navigation action in response to a detected navigation state, even if the selected navigation action may not produce the highest reward from the available actions taken in response to the current navigation state. This is especially true when the system determines that the chosen action can lead to one or more potential navigation actions that offer a higher reward in the future navigation state compared to the chosen action or, in some cases, compared to any action available in the current navigation state. The principle can be simply expressed as taking a less advantageous action now in order to produce a higher reward option in the future. Therefore, a publicly available navigation system capable of long-term planning can select suboptimal short-term actions, where long-term predictions indicate that a short-term reward loss can lead to a long-term reward gain.

[0317] Generally, autonomous driving applications can involve a range of planning problems where the navigation system makes decisions about immediate actions to optimize long-term objectives. For example, when a vehicle faces a lane-changing situation at a roundabout, the navigation system can decide on an immediate acceleration or braking command to initiate navigation into the roundabout. While the immediate action to the detected navigation state at the roundabout could involve acceleration or braking commands in response to the detected state, the long-term objective is a successful lane change, and the long-term effect of the chosen command is the success or failure of the lane change. The planning problem can be solved by decomposing it into two phases. First, supervised learning can be applied to predict the near future based on the present (assuming the predictor is differentiable with respect to the current representation). Second, a recurrent neural network can be used to model the agent's complete trajectory, where unexplained factors are modeled as (additional) input nodes. This allows for the use of supervised learning techniques and direct optimization of the recurrent neural network to determine solutions to the long-term planning problem. This approach can also enable the learning of robust policies by incorporating adversarial elements into the environment.

[0318] Two of the most fundamental elements of an autonomous driving system are sensing and planning. Sensing involves discovering a compact representation of the current state of the environment, while planning involves deciding which actions to take to optimize future objectives. Supervised machine learning techniques are useful for addressing the sensing problem. Machine learning algorithm frameworks can also be used for the planning part, especially reinforcement learning (RL) frameworks, such as those described above.

[0319] RL can be performed in a series of consecutive rounds. In each round... t At that time, the planner (also known as the intelligent agent or driving strategy module 803) can observe the state. The state represents both the agent and the environment. The planner should then decide on the action. After performing the action, the agent receives an immediate reward. And move to the new state. s t+1 As an example, the primary vehicle may include an adaptive cruise control (ACC) system, where the vehicle should autonomously accelerate / brake to maintain a sufficient distance from the vehicle in front while maintaining smooth driving. The state can be modeled as a response to... ,in x t It is the distance to the vehicle in front, and v t It is the speed of the main vehicle relative to the vehicle in front. (Action) It will be the acceleration command (if) a t <0, meaning the main vehicle slows down. The reward can depend on... (Reflects driving smoothness) and depends on s t The planner's goal is to maximize the cumulative reward (which may be determined by the time frame or the sum of discounts on future rewards). To achieve this, the planner can rely on strategies. The strategy maps states to actions.

[0320] Supervised learning (SL) can be viewed as a special case of RL, where s t From S It is sampled from a certain distribution on the spectrum, and the reward function can have the form... ,in It is the loss function, and the learner observes... y t The value, which is the value when the viewing state is... s tThe optimal (potentially noisy) value of the action to be taken. There may be several differences between general RL models and special cases of SL, and these differences may make general RL problems more challenging.

[0321] In some cases of SolidWorks (SL), the actions (or predictions) taken by the learner may have no impact on the environment. In other words, s t+1 and a t It is independent. This likely has two important implications. First, in SL, samples can be collected in advance. And only then can the search for a strategy (or predictor) that will have good accuracy relative to the sample begin. Conversely, in RL, the state... s t+1 Typically, it depends on the action taken (and also on the previous state), which in turn depends on the policy used to generate the action. This links the data generation process to the policy learning process. Secondly, since actions do not affect the environment in SL, [the following is unclear and likely incomplete: "for..."] a t The choice of has a local contribution to the performance of π. Specifically, a t It only affects the value of the immediate reward. Conversely, in RL, it affects the value of the reward in each round. t Actions taken at that time may have a long-term impact on the value of rewards in future rounds.

[0322] In SL, the "correct" answer y t Understanding along with the shape of the reward It can provide all possible options a t A complete understanding of the rewards allows for the calculation of rewards relative to... a t The derivative of . Conversely, in RL, the "one-off" value of the reward can be all the observable values ​​for a particular action taken. This can be called "bandit" feedback. This is one of the most important reasons why "exploration" is needed as part of long-term navigation planning, because in RL-based systems, if only "bandit" feedback is available, the system may not always know whether the action taken is the best action to take.

[0323] Many RL algorithms rely, at least in part, on mathematically elegant models of Markov decision processes (MDPs). The Markov assumption is that, given... s t and a t In this case, s t+1The distribution is completely deterministic. This yields a closed-form expression for the cumulative reward of a given policy based on the stable distribution of the states in the MDP. The stable distribution of the policy can be expressed as a solution to a linear programming problem. This leads to two families of algorithms: 1) optimization of the primal problem, which can be called policy search, and 2) optimization of the dual problem, whose variables are called... value function , If the MDP starts from the initial state s Starting from this point, and with the action chosen based on π, the value function determines the expected cumulative reward. The relevant number is the state-action value function. Its determination assumption is based on the state s Initial cumulative rewards, actions selected instantly. a And the actions to be chosen from here onward based on π. Q The function can (using Bellman's equation) produce a representation of the optimal strategy. Specifically, Q The function can indicate that the optimal strategy is from S arrive A The deterministic function (in fact, the optimal strategy can be characterized as about the optimal) Q (The "greedy" strategy of functions).

[0324] One potential advantage of the MDP model is that it allows for the use of... Q The function couples the future into the present. For example, given that the main vehicle is currently in state... s , The value can indicate the action to be performed. a The impact on the future. Therefore, the Q-function can provide action. a The quality of a local metric makes the RL problem more similar to the SL scenario.

[0325] Many RL algorithms approximate one or another. V function or Q Functions. Value iteration algorithms, for example, Q Learning algorithms can rely on optimal strategies. V functions and Q The fact that the function can be a fixed point of some operators derived from the Bellman equations is a key factor. The actor-evaluator policy iterative algorithm aims to learn the policy iteratively, where in iteration... t At that time, the "evaluator" estimated And based on this estimate, the "actor" improves its strategy.

[0326] Although MDP is mathematically elegant and switching to QFunction representations are convenient, but this approach can have several limitations. For example, in some cases, the concept of approximating Markov behavioral states may be all that can be discovered. Furthermore, state transitions may depend not only on the agent's actions but also on the actions of other actors in the environment. For instance, in the ACC example mentioned above, while the autonomous vehicle's dynamics may be Markovian, the next state could depend on the actions of another car's driver, which are not necessarily Markovian. One possible solution to this problem is to use a partially observed MDP, where the existence of Markov states is assumed, but what can be seen is determined by observations of the hidden state distribution.

[0327] A more direct approach could be to consider game-theoretic generalizations of MDPs (e.g., stochastic game frameworks). In fact, algorithms for MDPs can generalize to multi-agent games (e.g., minimax-Q learning or Nash-Q learning). Other approaches could include explicit modeling of other players and vanishing regret learning algorithms. Learning in multi-agent settings can be more complex than in single-agent settings.

[0328] The second limitation of the Q-function representation may arise from deviations from the table setting. The table setting is used when the number of states and actions is small, and therefore... Q It can be expressed as having individual lines and A table with columns. However, if S and A The natural representation of a space includes Euclidean space, and since the state and action spaces are discretized, the number of states / actions can be exponential in dimensionality. In such cases, using a tabular representation may be impractical. Instead, Q A function can be approximated by some function derived from a class of parameter assumptions (e.g., a neural network of a certain architecture). For example, a Deep Q-Network (DQN) learning algorithm can be used. In DQN, the state space can be continuous, but the action space may still be a small discrete set. Methods for dealing with continuous action spaces may exist, but these methods may rely on approximation. Q Function. In any case, Q Functions can be complex and sensitive to noise, and therefore can be challenging to learn.

[0329] A different approach could be to use recurrent neural networks (RNNs) to solve RL problems. In some cases, RNNs can be combined with concepts from game theory, such as multi-agent games and robustness to adversarial environments. Furthermore, this approach can be one that does not explicitly rely on any Markov assumptions.

[0330] The method for navigation via prediction-based planning is described in more detail below. In this method, a state space can be assumed. S yes A subset of, and action space A yes A subset of. In many applications, this can be a natural representation. As noted above, there may be two key differences between RL and SL: (1) because past actions affect future rewards, information for the future may need to be propagated back to the past; (2) the “robbery” nature of rewards can obscure the dependency between (state, action) and reward, thus complicating the learning process.

[0331] As a first step in the method described, the following observation can be made: There is an interesting problem where the slot machine nature of the reward is not the issue. For example, the reward value for the ACC application (discussed in more detail below) can be differentiable with respect to the current state and action. In fact, even if the reward is given in a "slot machine" manner, learning a differentiable function... Make The problem can also be a relatively simple SL problem (e.g., a one-dimensional regression problem). Therefore, the first step of the method could be to define the reward as a function. The function is relative to s and a It is differentiable, or a regression learning algorithm can be used to learn the differentiable function. The differentiable function makes the instance vectors be and the target scalar is The goal is to minimize at least some regression losses on the samples. In some cases, exploratory elements can be used to create a training set.

[0332] A similar idea can be used to address the connection between the past and the future. For example, suppose we could learn differentiable functions. Make Learning this type of function can be characterized as a learning problem (SL). This can be viewed as a predictor for the near future. Next, a parametric function can be used. To describe from S Mapped to A The strategy. Representing it as a neural network allows the use of recurrent neural networks (RNNs) to represent the continuous operation of an agent. T The number of rounds, where the next state is defined as... .here, It can be defined by the environment and can express unpredictable aspects of the near future. s t+1 Depends on in a differentiable ways t and a t The fact that this establishes a link between future reward value and past actions is demonstrated by the parameter vector of the policy function. It can be learned through backpropagation on the obtained RNN. Note that it is not necessary to... Explicit probability assumptions are imposed. In particular, no Markov relation is required. Instead, recursive networks can be relied upon to propagate “sufficient” information between the past and future. Intuitively, It can describe the predictable parts of the near future, while Unpredictable aspects can be expressed, which may arise due to the behavior of other participants in the environment. The learning system should learn strategies that are robust to the behavior of other participants. If If the value is too large, the connection between past actions and future rewards may be too noisy for learning meaningful strategies. Explicitly expressing the system's dynamics in a transparent manner makes it easier to incorporate prior knowledge. For example, prior knowledge can be simplified in definition. The problem.

[0333] As discussed above, learning systems can benefit from robustness relative to adversarial environments, such as the environment of the primary vehicle, which could include multiple other drivers who might act in unintended ways. In the absence of... In models that impose probabilistic assumptions, adversarial selection can be considered. The environment. In some cases, it is possible to... Imposing limitations, otherwise the opponent will make the planning problem difficult or even impossible. A natural limitation might be a requirement. Bounded by constants.

[0334] Robustness in adversarial environments can be useful in autonomous driving applications. Choosing an adversarial approach... It can even accelerate the learning process because it allows the learning system to focus on robust, optimal strategies. This concept can be illustrated using a simple game theory approach. The state is... The action is And the instantaneous loss function is ,in This is the ReLU (Modified Linear Unit) function. The next state is... Among them, the selection of the environment in an adversarial manner Here, the optimal strategy can be written as a two-layer network with ReLU: It was observed that when At that time, the best movement compared to the movementa = 0 has a large immediate loss. Therefore, the system can plan for the future and can rely not only on immediate losses. The observed loss is relative to... a t The derivative is And relative to s t The derivative is In it In this case, The choice of confrontation will determine the setting , and therefore, whenever round t + 1 Non-zero losses may exist in all cases. In such cases, the derivative of the loss can be directly backpropagated to... a t 。 Therefore, in a t In the case where the choice is the second best, Adversarial choices can help navigation systems obtain non-zero backpropagation messages. This relationship can assist navigation systems in choosing current actions based on such expectations that this current action (even if it leads to a suboptimal reward or even a loss) will provide opportunities to obtain more optimal actions in the future that will bring higher rewards.

[0335] This method can be applied to virtually any navigation situation that may arise. The following describes an application of this method to an example: Adaptive Cruise Control (ACC). In the ACC problem, the primary vehicle may attempt to maintain a sufficient distance from a target vehicle ahead (e.g., 1.5 seconds from the target car). Another objective could be to drive as smoothly as possible while maintaining the desired distance. The model representing this situation can be defined as follows. The state space is... And the action space is The first coordinate of the state is the target car's speed, the second coordinate is the master vehicle's speed, and the last coordinate is the distance between the master vehicle and the target vehicle (e.g., the master vehicle's position minus the target vehicle's position along a road bend). The master vehicle's action is to accelerate, and this can be achieved by... a t Indicates. Number. It can refer to the time difference between consecutive rounds. Although It can be set to any suitable number, but in one instance, It can be 0.1 seconds. Location. s t It can be represented as And the (unknown) acceleration of the target vehicle can be expressed as .

[0336] The complete dynamics of the system can be described as follows: This can be described as the sum of two vectors: The first vector represents the predictable portion, and the second vector represents the unpredictable portion. For each round... t The reward is defined as follows: in The first factor might result in a penalty for non-zero acceleration, thus encouraging smooth driving. The second factor depends on the distance to the target vehicle. x t Distance from Expected The ratio between the distance of 1 meter and the braking distance of 1.5 seconds is defined as the maximum value between the distance of 1 meter and the braking distance of 1.5 seconds. In some cases, this ratio may be exactly 1, but as long as this ratio is within [0.7, 1.3], the strategy can waive any penalty, which allows the primary vehicle to have some slack in navigation—a characteristic that may be important in achieving smooth driving.

[0337] Implementing the method outlined above, the main vehicle's navigation system (e.g., through the operation of the driving strategy module 803 within the navigation system's processing unit 110) can select an action in response to the observed state. The selected action can be based not only on the analysis of the reward associated with the responsive action available relative to the sensed navigation state, but also on consideration and analysis of future states, potential actions in response to future states, and the rewards associated with those potential actions.

[0338] Figure 16 An algorithmic approach for navigation based on detection and long-range planning is demonstrated. For example, at step 1601, at least one processing unit 110 of the main vehicle's navigation system can receive multiple images. These images can capture scenes representing the environment of the main vehicle and can be provided by any image capture device described above (e.g., a camera, sensor, etc.). Analysis of one or more of these images at step 1603 enables at least one processing unit 110 to identify the current navigation state associated with the main vehicle (as described above).

[0339] At steps 1605, 1607, and 1609, various potential navigation actions in response to the sensed navigation state can be determined. These potential navigation actions (e.g., first navigation action to Nth available navigation action) can be determined based on the sensed state of the navigation system and long-term objectives (e.g., completing a lane change, smoothly following a leading vehicle, overtaking a target vehicle, avoiding an object in the roadway, slowing down in response to a detected stop sign, avoiding an intruding target vehicle, or any other navigation action that can advance the navigation objectives of the system).

[0340] For each of the identified potential navigation actions, the system can determine an expected reward. The expected reward can be determined according to any of the techniques described above and can include analysis of the specific potential action relative to one or more reward functions. Expected rewards 1606, 1608, and 1610 can be determined for each of the potential navigation actions (e.g., first, second, and Nth) identified in steps 1605, 1607, and 1609, respectively.

[0341] In some cases, the primary vehicle's navigation system can select from available potential actions based on the values ​​associated with expected rewards 1606, 1608, and 1610 (or any other type of indication of expected rewards). For example, in some situations, the action that produces the highest expected reward can be selected.

[0342] In other situations, particularly when the navigation system is performing long-range planning to determine navigation actions for the primary vehicle, the system may not select the potential action that yields the highest expected reward. Instead, the system can look to the future to analyze whether there is an opportunity to achieve a higher reward later if a lower-reward action is selected in response to the current navigation state. For example, for any or all of the potential actions determined at steps 1605, 1607, and 1609, future states can be determined. Each future state determined at steps 1613, 1615, and 1617 can represent a future navigation state expected to be generated based on the current navigation state as modified by the corresponding potential action (e.g., the potential actions determined at steps 1605, 1607, and 1609).

[0343] For each of the future states predicted in steps 1613, 1615, and 1617, one or more future actions (as navigation options available in response to the determined future state) can be identified and evaluated. At steps 1619, 1621, and 1623, for example, the value of the expected reward or any other type of indication associated with one or more of the future actions can be developed (e.g., based on one or more reward functions). The expected reward associated with one or more future actions can be evaluated by comparing the value of the reward function associated with each future action or by comparing any other indication associated with the expected reward.

[0344] At step 1625, the primary vehicle's navigation system may select a navigation action for the primary vehicle based on a comparison of anticipated rewards, rather than solely on potential actions identified relative to the current navigation state (e.g., at steps 1605, 1607, and 1609), but also on anticipated rewards determined due to potential future actions (available in response to predicted future states, e.g., determined at steps 1613, 1615, and 1617). The selection at step 1625 may be based on the option and reward analysis performed at steps 1619, 1621, and 1623.

[0345] The selection of the navigation action at step 1625 can be based solely on a comparison of the expected rewards associated with future action options. In this case, the navigation system can select an action for the current state based solely on a comparison of the expected rewards generated by actions for potential future navigation states. For example, the system can select a potential action identified at steps 1605, 1607, or 1609 that is associated with the highest future reward value determined by the analysis at steps 1619, 1621, and 1623.

[0346] The selection of the navigation action at step 1625 can also be based solely on a comparison of current action options (as noted above). In this case, the navigation system can select the potential action identified at steps 1605, 1607, or 1609 that is associated with the highest expected reward 1606, 1608, or 1610. This selection can be made with little or no consideration of future navigation states or the future expected reward of the navigation action available in response to an expected future navigation state.

[0347] On the other hand, in some cases, the selection of the navigation action at step 1625 can be based on a comparison of expected rewards associated with both future action options and current action options. In fact, this can be based on one of the navigation principles of long-range planning. For example, the expected reward for a future action can be analyzed to determine if any action can guarantee the selection of a lower reward action in response to the current navigation state in order to achieve a potentially higher reward in response to a subsequent navigation action expected to be available in response to the future navigation state. As an example, the value or other indication of expected reward 1606 can indicate the highest expected reward among rewards 1606, 1608, and 1610. On the other hand, expected reward 1608 can indicate the lowest expected reward among rewards 1606, 1608, and 1610. Instead of simply selecting the potential action determined in step 1605 (i.e., the action that produces the highest expected reward 1606), the navigation action selection at step 1625 can be made using an analysis of the future state, potential future actions, and future rewards. In one instance, it can be determined that the reward identified at step 1621 (in response to at least one future action for a future state, determined at step 1615 based on a second potential action determined at step 1607) may be higher than the expected reward 1606. Based on this comparison, the second potential action determined at step 1607 may be selected instead of the first potential action determined at step 1605, even though the expected reward 1606 is higher than the expected reward 1608. In one instance, the potential navigation action determined at step 1605 may include merging in front of the detected target vehicle, while the potential navigation action determined at step 1607 may include merging in behind the target vehicle. Although the expected reward 1606 for merging in front of the target vehicle may be higher than the expected reward 1608 associated with merging in behind the target vehicle, it can be determined that merging in behind the target vehicle may result in a future state where action options exist that produce a potential reward even higher than the expected rewards 1606, 1608, or other rewards based on actions available in response to the current sensed navigation state.

[0348] The selection from potential actions at step 1625 can be based on any suitable comparison of the anticipated reward (or any other measure or indication of the benefit of one potential action over another). In some cases, as described above, the second potential action may be selected instead of the first if it is expected to provide at least one future action associated with an anticipated reward higher than that associated with the first potential action. In other cases, more complex comparisons may be employed. For example, the reward associated with an action option in response to a predicted future state may be compared with more than one anticipated reward associated with a identified potential action.

[0349] In some scenarios, if at least one of the expected future actions will produce a reward higher than any reward expected due to a potential action to the current state (e.g., expected rewards 1606, 1608, 1610, etc.), then the action based on the expected future state and the expected reward may influence the selection of a potential action to the current state. In some cases, the future action option that produces the highest expected reward (e.g., from the expected rewards associated with a potential action to the sensed current state and from the expected rewards associated with potential future action options relative to a potential future navigation state) can be used as guidance for selecting a potential action to the current navigation state. That is, after identifying the future action option that produces the highest expected reward (or a reward above a predetermined threshold, etc.), a potential action that would lead to a future state associated with the identified future action that produces the highest expected reward can be selected at step 1625.

[0350] In other cases, the selection of available actions can be based on a determined difference between expected rewards. For example, if the difference between the expected reward associated with the future action determined in step 1621 and expected reward 1606 is greater than the difference between expected reward 1608 and expected reward 1606 (assuming sign + difference), then the second potential action determined in step 1607 can be selected. In another instance, if the difference between the expected reward associated with the future action determined in step 1621 and the expected reward associated with the future action determined in step 1619 is greater than the difference between expected reward 1608 and expected reward 1606, then the second potential action determined in step 1607 can be selected.

[0351] Several instances have been described for selecting from potential actions based on the current navigation state. However, any other suitable comparison technique or criterion can be used to select available actions through long-range planning based on action and reward analysis extending to anticipated future states. Additionally, although Figure 16 This represents two layers in long-range planning analysis (e.g., the first layer considers the rewards generated by potential actions in response to the current state, and the second layer considers the rewards generated by future action options in response to the anticipated future state), but analysis based on more layers is also possible. For example, instead of basing long-range planning analysis on one or two layers, analysis with three, four, or more layers can be used to select from available potential actions in response to the current navigation state.

[0352] After selecting from potential actions in response to the sensed navigation state, at step 1627, at least one processor may cause at least one adjustment to the navigation actuator of the host vehicle in response to the selected potential navigation action. The navigation actuator may include any suitable means for controlling at least one aspect of the host vehicle. For example, the navigation actuator may include at least one of a steering mechanism, a brake, or an accelerator.

[0353] Based on other inferred attack navigation Target vehicles can be monitored by analyzing acquired image streams to determine indications of driving aggression. Aggression is described herein as a qualitative or quantitative parameter, but other characteristics can also be used: perceived level of attention (potential harm to the driver, distraction—cell phone, drowsiness, etc.). In some cases, a target vehicle can be perceived as having a defensive posture, while in others it can be identified as having a more aggressive posture. Navigation actions can be selected or developed based on aggression indications. For example, in some cases, relative speed, relative acceleration, increase in relative acceleration, following distance, etc., relative to the host vehicle can be tracked to determine whether the target vehicle is aggressive or defensive. For example, if the target vehicle is determined to have an aggression level exceeding a threshold, the host vehicle may tend to yield to the target vehicle. The level of aggression of a target vehicle can also be identified based on determined behavior of the target vehicle relative to one or more obstacles in or near its path (e.g., vehicles ahead, obstacles in the road, traffic lights, etc.).

[0354] As an introduction to this concept, an example experiment will be described regarding a primary vehicle merging into a roundabout, where the navigation objective is to pass through and exit the roundabout. The process can begin when the primary vehicle approaches the roundabout's entrance and end when the primary vehicle reaches the roundabout's exit (e.g., the second exit). Success can be measured based on whether the primary vehicle consistently maintains a safe distance from all other vehicles, whether the primary vehicle completes the route as quickly as possible, and whether the primary vehicle follows a smooth acceleration strategy. In this illustration, [the following can be...] N T Target vehicles are randomly placed on a roundabout. To model a mixture of adversarial and typical behaviors, probability is used. p The target vehicle can be modeled using an "aggressive" driving strategy, such that when the primary vehicle attempts to merge into front of the aggressive target vehicle, the target vehicle accelerates. At probability 1- p In this scenario, the target vehicle can be modeled using a "defensive" driving strategy, causing it to decelerate and the host vehicle to merge. In this experiment, p = 0.5, and the main vehicle's navigation system may not provide information about other driver types. Other driver types can be randomly selected at the start of the round.

[0355] Navigation state can be represented as the speed and position of the primary vehicle (agent), and the position, speed, and acceleration of the target vehicles. Maintaining target acceleration observations may be important to differentiate between aggressive and defensive drivers based on the current state. All target vehicles can move on a one-dimensional curve that outlines the roundabout path. The primary vehicle can move on its own one-dimensional curve, which intersects the target vehicles' curves at a merge point, the origin of both curves. To model rational driving, the absolute values ​​of acceleration for all vehicles can be bounded to a constant. Speed ​​can also be represented using ReLU, as backward driving is not allowed. Note that by disallowing backward driving, long-term planning can become essential, as the agent will not regret its past actions.

[0356] As described above, the next state s t+1 It can be broken down into predictable parts and unpredictable parts The sum. Expression It can represent the dynamics of a vehicle's position and speed (which can be explicitly defined in a differentiable manner), while It can represent the acceleration of the target vehicle. This can be verified. It can be expressed as a combination of ReLU functions over affine transformations, therefore its relative... s t and a t It is differentiable. Vector It can be defined in a non-differentiable manner by the simulator, and can exhibit aggressive behavior towards some targets while exhibiting defensive behavior towards others. Two frameworks from this type of simulator are as follows: Figure 17A and 17B As shown. In this example experiment, the primary vehicle 1701 learned to slow down when it approached the roundabout entrance. The primary vehicle also learned to yield to attacking vehicles (e.g., vehicles 1703 and 1705) and to safely continue when merging in front of defensive vehicles (e.g., vehicles 1706, 1708, and 1710). Figure 17A and 17B In the illustrated example, the type of the target vehicle was not provided to the navigation system of the primary vehicle 1701. Instead, whether a particular vehicle was identified as aggressive or defensive was determined by inference based on factors such as the target vehicle's observed position and acceleration. Figure 17ABased on position, speed, and / or relative acceleration, the primary vehicle 1701 can determine that the target vehicle 1703 has an aggressive tendency, and therefore, the primary vehicle 1701 can stop and wait for the target vehicle 1703 to pass rather than attempting to merge in front of the target vehicle 1703. However, in Figure 17B In the process, target vehicle 1701 (again based on the observed position, speed and / or relative acceleration of vehicle 1710) recognizes that target vehicle 1710 traveling behind vehicle 1703 is exhibiting a defensive tendency, and thus completes a successful lane change in front of target vehicle 1710 and behind target vehicle 1703.

[0357] Figure 18 A flowchart is provided representing an example algorithm for navigating a master vehicle based on predictions of aggression from other vehicles. Figure 18 In this example, the level of aggression associated with at least one target vehicle can be inferred based on observed behavior of the target vehicle relative to objects in its environment. For example, at step 1801, at least one processing unit of the main vehicle navigation system (e.g., processing unit 110) can receive multiple images representing the environment of the main vehicle from a camera associated with the main vehicle. At step 1803, analysis of one or more of the received images enables at least one processor to identify a target vehicle (e.g., vehicle 1703) in the environment of the main vehicle 1701. At step 1805, analysis of one or more of the received images enables at least one processing unit to identify at least one obstacle to the target vehicle in the environment of the main vehicle. The object may include debris in the roadway, stop lights / traffic lights, pedestrians, another vehicle (e.g., a vehicle traveling in front of the target vehicle, a parked vehicle, etc.), boxes in the roadway, road barriers, curbs, or any other type of object that may be encountered in the environment of the main vehicle. At step 1807, analysis of one or more of the received images enables at least one processing unit to determine at least one navigation characteristic of the target vehicle relative to at least one identified obstacle to the target vehicle.

[0358] Various navigation characteristics can be used to infer the aggressiveness level of a detected target vehicle in order to develop an appropriate navigation response to the target vehicle. For example, such navigation characteristics may include the relative acceleration between the target vehicle and at least one identified obstacle, the distance of the target vehicle from the obstacle (e.g., the following distance of the target vehicle behind another vehicle), and / or the relative speed between the target vehicle and the obstacle.

[0359] In some embodiments, the navigation characteristics of a target vehicle can be determined based on outputs from sensors associated with the host vehicle (e.g., radar, speed sensors, GPS, etc.). However, in some cases, the navigation characteristics of a target vehicle can be determined, in part or in whole, based on the analysis of images of the host vehicle's environment. For example, image analysis techniques described above and, for example, in U.S. Patent No. 9,168,868 (which is incorporated herein by reference) can be used to identify a target vehicle within the host vehicle's environment. Furthermore, monitoring the position of the target vehicle in captured images over time and / or monitoring the position of one or more features associated with the target vehicle (e.g., taillights, headlights, bumpers, wheels, etc.) in captured images can make it possible to determine the relative distance, velocity, and / or acceleration between the target vehicle and the host vehicle or between the target vehicle and one or more other objects in the host vehicle's environment.

[0360] The level of aggression of a target vehicle can be inferred from any suitable observed navigation characteristics or any combination of observed navigation characteristics. For example, a determination of aggression can be made based on any observed characteristic and one or more predetermined threshold levels or any other suitable qualitative or quantitative analysis. In some embodiments, a target vehicle may be considered aggressive if it is observed following a host vehicle or another vehicle at a distance less than a predetermined aggression distance threshold. On the other hand, a target vehicle observed following a host vehicle or another vehicle at a distance greater than a predetermined defensive distance threshold may be considered defensive. The predetermined aggression distance threshold does not need to be the same as the predetermined defensive distance threshold. Furthermore, either or both of the predetermined aggression distance threshold and the predetermined defensive distance threshold may include a range of values, rather than a fixed value. Moreover, neither the predetermined aggression distance threshold nor the predetermined defensive distance threshold needs to be fixed. Instead, these values ​​or ranges can vary over time, and different thresholds / threshold ranges can be applied based on the observed characteristics of the target vehicle. For example, the applied threshold may depend on one or more other characteristics of the target vehicle. Higher observed relative speeds and / or accelerations may warrant the application of a larger threshold / range. Conversely, lower relative velocity and / or acceleration, including zero relative velocity and / or acceleration, can ensure the application of smaller distance thresholds / ranges when making offensive / defensive inferences.

[0361] Aggression / defensivity inference can also be based on relative speed and / or relative acceleration thresholds. A target vehicle may be considered aggressive if its observed relative speed and / or relative acceleration relative to another vehicle exceeds a predetermined level or range. A target vehicle may be considered defensive if its observed relative speed and / or relative acceleration relative to another vehicle falls below a predetermined level or range.

[0362] While an aggressive / defensive stance can be determined solely based on any observed navigational characteristic, the determination can also depend on any combination of the observed characteristics. For example, as noted above, in some cases, a target vehicle can be considered aggressive simply because it is observed following another vehicle at a distance below a certain threshold or range. However, in other cases, a target vehicle can be considered aggressive if it is following another vehicle at a distance less than a predetermined amount (which may be the same as or different from the threshold applied in distance-only determinations) and has a relative speed and / or relative acceleration greater than a predetermined amount or range. Similarly, a target vehicle can be considered defensive simply because it is observed following another vehicle at a distance greater than a certain threshold or range. However, in other cases, a target vehicle can be considered defensive if it is following another vehicle at a distance greater than a predetermined amount (which may be the same as or different from the threshold applied in distance-only determinations) and has a relative speed and / or relative acceleration less than a predetermined amount or range. System 100 may take aggressive / defensive actions in situations such as: when a vehicle accelerates or decelerates by more than 0.5G (e.g., a dynamism of 5 m / s³), when a vehicle changes lanes or has lateral acceleration of 0.5G on a curve, when a vehicle causes another vehicle to do any of the above, when a vehicle changes lanes and causes another vehicle to give way with a deceleration of more than 0.3G or a dynamism of 3 m / s³, and / or when a vehicle changes two lanes without stopping.

[0363] It should be understood that a reference to a number exceeding a range can indicate that the number exceeds all values ​​associated with the range or falls within the range. Similarly, a reference to a number falling below a range can indicate that the number falls below all values ​​associated with the range or falls within the range. Furthermore, while the examples described for making aggression / defense inferences relate to distance, relative acceleration, and relative velocity, any other suitable numbers can be used. For example, any indirect indication of the distance, acceleration, and / or velocity of the target vehicle can be calculated using the collision time or the target vehicle's distance. It should also be noted that while the examples above focus on a target vehicle relative to other vehicles, aggression / defense inferences can be made by observing the target vehicle's navigational characteristics relative to any other type of obstacle (e.g., pedestrians, road barriers, traffic lights, debris, etc.).

[0364] Return to Figure 17A and 17BIn the example shown, when the primary vehicle 1701 approaches a roundabout, a navigation system including at least one of its processing units can receive an image stream from a camera associated with the primary vehicle. Based on the analysis of one or more of the received images, any one of target vehicles 1703, 1705, 1706, 1708, and 1710 can be identified. Furthermore, the navigation system can analyze the navigation characteristics of one or more of the identified target vehicles. The navigation system can recognize that the gap between target vehicles 1703 and 1705 represents the first potential opportunity to merge into the roundabout. The navigation system can analyze target vehicle 1703 to determine any aggressive indications associated with it. If target vehicle 1703 is perceived as aggressive, the primary vehicle navigation system may choose to yield to vehicle 1703 instead of merging in front of it. On the other hand, if target vehicle 1703 is perceived as defensive, the primary vehicle navigation system may attempt to complete the merge in front of vehicle 1703.

[0365] When the primary vehicle 1701 approaches a roundabout, at least one processing unit of the navigation system can analyze the captured image to determine navigation characteristics associated with the target vehicle 1703. For example, based on the image, it can be determined that vehicle 1703 is following vehicle 1705 at a distance that provides sufficient clearance for the primary vehicle 1701 to safely enter. In practice, it can be determined that vehicle 1703 is following vehicle 1705 at a distance exceeding an aggressive distance threshold, and therefore, based on this information, the primary vehicle navigation system may tend to identify the target vehicle 1703 as defensive. However, in some cases, as discussed above, more than one navigation characteristic of the target vehicle can be analyzed when making an aggressive / defensive determination. Further analysis may allow the primary vehicle navigation system to determine that, when the target vehicle 1703 is following behind the target vehicle 1705 at a non-aggressive distance, vehicle 1703 has a relative velocity and / or relative acceleration relative to vehicle 1705 exceeding one or more thresholds associated with aggressive behavior. In practice, the primary vehicle 1701 can determine that the target vehicle 1703 is accelerating relative to vehicle 1705 and reducing the gap between them. Based on further analysis of relative speed, acceleration, and distance (and even the rate at which the gap between vehicles 1703 and 1705 is reduced), the primary vehicle 1701 can determine that the target vehicle 1703 is exhibiting aggressive behavior. Therefore, although there may be a sufficient gap that the primary vehicle can safely navigate to, the primary vehicle 1701 can anticipate that changing lanes in front of the target vehicle 1703 will result in the aggressively navigating vehicle being directly behind the primary vehicle. Furthermore, based on behavior observed through image analysis or other sensor outputs, it can be anticipated that the target vehicle 1703 will continue to accelerate toward the primary vehicle 1701 or continue moving toward the primary vehicle 1701 at a non-zero relative speed if the primary vehicle 1701 were to change lanes in front of the target vehicle 1703. From a safety perspective, this situation is likely undesirable and could also cause discomfort to the passengers of the primary vehicle. For these reasons, vehicle 1701 may choose to give way to vehicle 1703, such as Figure 17B As shown, it merges into the roundabout behind vehicle 1703 and in front of vehicle 1710 (which is considered defensive based on analysis of one or more of its navigation characteristics).

[0366] Return to Figure 18At step 1809, at least one processing device of the main vehicle's navigation system can determine a navigation action for the main vehicle (e.g., merging in front of vehicle 1710 and behind vehicle 1703) based on at least one navigation characteristic of the target vehicle relative to the identified obstacles. To implement the navigation action (at step 1811), at least one processing device can cause at least one adjustment to the navigation actuators of the main vehicle in response to the determined navigation action. For example, brakes can be applied to... Figure 17A Vehicle 1703 gives way, and accelerators and steering of the main vehicle's wheels can be applied to allow the main vehicle to enter the roundabout behind vehicle 1703, as... Figure 17B As shown.

[0367] As described in the examples above, the navigation of the primary vehicle can be based on the navigation characteristics of the target vehicle relative to another vehicle or object. Alternatively, the navigation of the primary vehicle can be based solely on the navigation characteristics of the target vehicle, without specifically referencing another vehicle or object. For example, in... Figure 18 At step 1807, analysis of multiple images captured from the environment of the main vehicle enables the determination of at least one navigation characteristic of the identified target vehicle, said at least one navigation characteristic indicating the level of aggression associated with the target vehicle. Navigation characteristics may include speeds, accelerations, etc., that do not require reference to another object or target vehicle to make an aggression / defense determination. For example, observed acceleration and / or speed associated with the target vehicle exceeding a predetermined threshold or falling within or exceeding a value range may indicate aggressive behavior. Conversely, observed acceleration and / or speed associated with the target vehicle falling below a predetermined threshold or falling within or exceeding a value range may indicate defensive behavior.

[0368] Of course, in some cases, observed navigation characteristics (e.g., position, distance, acceleration, etc.) can be referenced relative to the primary vehicle to make an aggressive / defensive determination. For example, observed navigation characteristics that indicate the level of aggression associated with the target vehicle may include the increase in relative acceleration between the target vehicle and the primary vehicle, the following distance of the target vehicle behind the primary vehicle, the relative speed between the target vehicle and the primary vehicle, etc.

[0369] Navigation based on accident liability constraints As described in the preceding sections, planned navigation actions can be tested against predetermined constraints to ensure compliance with certain rules. In some embodiments, this concept can be extended to consideration of potential accident liability. As discussed below, the primary objective of autonomous navigation is safety. Since absolute safety may be impossible (e.g., at least because a particular master vehicle under autonomous control cannot control other vehicles in its surrounding environment—the particular master vehicle can only control its own actions), using potential accident liability as a consideration in autonomous navigation, and in effect as a constraint on planned actions, can help ensure that a particular autonomous vehicle does not take any actions deemed unsafe, such as those actions for which potential accident liability might be attributed to the master vehicle. If the master vehicle only takes safe actions that are determined not to cause an accident that would result in the master vehicle's own fault or responsibility, a desired level of accident avoidance (e.g., less than 10) can be achieved. -9 / driving hours).

[0370] The challenges of most current autonomous driving approaches include a lack of safety guarantees (or at least an inability to provide the desired level of safety) and a lack of scalability. Consider the problem of ensuring safe driving for multiple agents. Since society is unlikely to tolerate deaths in road accidents caused by machines, an acceptable level of safety is crucial for the acceptance of autonomous vehicles. While the goal could be to achieve zero accidents, this may be impossible because accidents typically involve multiple agents, and one might envision situations where accidents occur solely due to the attribution of blame to other agents. For example, as... Figure 19 As shown, the master vehicle 1901 is driving on a multi-lane highway, and while the master vehicle 1901 can control its own actions relative to target vehicles 1903, 1905, 1907, and 1909, it cannot control the actions of the target vehicles around it. Therefore, if vehicle 1905 suddenly cuts into the master vehicle's lane, for example, along the collision path, the master vehicle 1901 may be unable to avoid an accident with at least one of the target vehicles. To address this challenge, a typical response from autonomous vehicle operators is to resort to a statistics-driven approach, where safety verification becomes more rigorous as more mileage data is collected.

[0371] However, in order to understand the problematic nature of data-driven safety approaches, we must first consider that the probability of death from an accident per hour of (human) driving is known to be 10. -6 It is reasonable to assume that for society to accept the replacement of humans with machines in driving tasks, the mortality rate should decrease by three orders of magnitude, i.e., to 10%. -9 The probability per hour. This estimate is similar to the assumed airbag fatality rate based on aviation standards. For example, 10 -9This is the probability that the wing will spontaneously detach from the aircraft in mid-air. However, it is impractical to attempt to guarantee safety using data-driven statistical methods (which provide additional confidence levels by summing up the miles flown). Guarantee 10 -9 The amount of data required to determine the probability of death per driving hour is the reciprocal of that probability (i.e., 10). 9 The data volume is approximately 30 billion miles, proportional to the hourly data. Furthermore, multi-agent systems interact with their environment and may not be verifiable offline (unless a realistic simulator is available, mimicking real human driving and its richness and complexity, such as reckless driving; however, validating a simulator would be far more difficult than creating a safe autonomous vehicle agent). Any changes to the planning and control software would require the same amount of new data collection, which is clearly cumbersome and impractical. Moreover, developing systems through data always suffers from a lack of interpretability and explainability of the actions taken—if an autonomous vehicle (AV) causes a fatal accident, the cause needs to be understood. Therefore, a model-based safety approach is needed, but existing "functional safety" and ASIL requirements in the automotive industry are not designed for multi-agent environments.

[0372] The second major challenge in developing safe driving models for autonomous vehicles is the need for scalability. The underlying premise of AVs is not merely "building a better world," but rather the assumption that driverless mobility can be supported at a lower cost than with a driver. This premise is always accompanied by the concept of scalability—in terms of supporting the mass production of AVs (in the millions), and more importantly, in terms of supporting negligible incremental costs for driving in new cities. Therefore, the costs of computation and sensing are indeed significant, as are the costs of calibration and the ability to drive "everywhere" rather than in a select few cities, if AVs are to be mass-produced.

[0373] The problem with most current approaches lies in a “brute-force” mindset along three axes: (i) the required “computational density,” (ii) the way high-definition maps are defined and created, and (iii) the required sensor specifications. This brute-force approach violates scalability and shifts the focus to a future where ubiquitous, infinitely invasive in-vehicle computing makes the cost of constructing and maintaining HD maps negligible and scalable, and where exotic, ultra-advanced sensors are developed and commercialized at automotive-grade levels at negligible cost. While a future where any of these conditions are met is indeed plausible, making all of them applicable is likely a low-probability event. Therefore, a formal model is needed that integrates security and scalability into a socially acceptable AV procedure and is scalable in the sense of supporting millions of cars driving anywhere in developed countries.

[0374] The disclosed embodiments represent solutions that can provide a target level of safety (or even exceed that target) and can be scaled to systems comprising millions (or more) of autonomous vehicles. At the forefront of safety, a model called “Responsible Sensitive Safety” (RSS) is introduced, which formalizes the concept of “accident attribution” in an interpretable and demonstrable way and incorporates a sense of “responsibility” into the actions of robotic agents. The definition of RSS is independent of how it is implemented—a key feature that contributes to the goal of creating a compelling global safety model. The motivation for RSS is observation (such as…) Figure 19 (As shown) Agents play asymmetric roles in accidents, where typically only one agent is responsible for the accident and therefore liable for it. RSS models also include a formalized treatment of "cautious driving" under limited sensing conditions, where not all agents are always visible (e.g., due to occlusion). A primary objective of RSS models is to guarantee that an agent will never create an accident "attributed" to or responsible for it. A model is only useful if it has an efficient policy that conforms to RSS (e.g., mapping "sensing states" to a function of actions). For example, an action that seems innocent at the present moment could lead to a catastrophic event in the distant future ("butterfly effect"). RSS can be useful for constructing a set of local constraints for the short-term future that guarantee (or at least practically guarantee) that an accident will not occur in the future due to the actions of the primary vehicle.

[0375] Another contribution revolves around the following: the introduction of a “semantic” language composed of units, measurements, and action spaces, and specifications on how to incorporate them into the planning, sensing, and actuation of AV. To clarify the semantics, in this context, consider how to instruct someone taking a driving course to think about a “driving strategy.” These instructions are not geometric—the instructions do not take the form of “drive 13.7 meters at the current speed, and then at 0.8 meters…” m / s 2 Instead of the form of "acceleration at a certain rate," instructions have a semantic essence—"follow the car in front of you" or "overtake the car on your left." Typical language of human driving strategies concerns longitudinal and lateral objectives, not acceleration vectors in geometric units. Formal semantic language could be useful in several frontiers related to: the computational complexity of planning (which does not scale exponentially with time and the number of agents), how safety and comfort interact, how to define computations for sensing, and the specification of sensor modalities and how they interact in fusion methods. Fusion methods (based on semantic language) can ensure that RSS models achieve the 10 required per hour of driving. -9 The probability of death, and only for about 10 5 Offline verification of the hourly driving data dataset.

[0376] For example, in reinforcement learning environments, the Q-function (e.g., when the agent is in a state) s S At the same time, assess the implementation of actions a A A function of long-term quality; given such a Q-function, the natural selection of actions could be to select the action with the highest quality, π( s )= argmax a Q ( s , a The number of trajectories to be examined at any given time can be defined in a semantic space, where the number of trajectories to be examined is always 10, regardless of the time range used in the planning. 4 The boundary is defined. The signal-to-noise ratio in this space can be high, allowing efficient machine learning methods to successfully model the Q-function. In the case of computation of sensing, semantics can allow the distinction between errors affecting safety and errors affecting driving comfort. A PAC model (Approximately Correct (PAC)) for sensing, associated with the Q-function, is defined (borrowing Valiant's PAC learning terminology), and it is shown how measurement errors can be incorporated into planning in a way that conforms to RSS but allows for optimization of driving comfort. Semantic language may be important for the success of some aspects of this model, as other standard error measures (such as errors relative to the global coordinate system) may not be suitable for the PAC sensing model. Furthermore, semantic language can be an important implementer for defining HD maps, which can be constructed using low-bandwidth sensing data and therefore can be built through crowdsourcing and support scalability.

[0377] In summary, the disclosed embodiments may include formal models covering the following key components of an AV: sensing, planning, and motion. These models help ensure that, from a planning perspective, no incidents of AV responsibility occur. Furthermore, through the PAC sensing model, even with sensing errors, the described fusion method can meet the described safety model with only a very reasonable amount of offline data collection. Moreover, the model can link security and scalability through a semantic language, thus providing a complete approach to secure and scalable AVs. Finally, it is worth noting that developing an accepted safety model adopted by industry and regulatory bodies may be a prerequisite for the success of AVs.

[0378] RSS models typically follow the classic sense-plan-action robot control approach. The sensing system is responsible for understanding the current state of the host vehicle's environment. The planning component (which may be called a "driving strategy" and can be implemented via a set of hard-coded instructions, a trained system (e.g., a neural network), or a combination thereof) is responsible for determining the best next action (e.g., how to move from the left lane to the right lane to exit a highway) considering the available options for achieving the driving objective. The action component is responsible for implementing the plan (e.g., a system of actuators and one or more controllers for actions such as steering, accelerating, and / or braking of the vehicle to implement the selected navigational actions). The embodiments described below focus primarily on the sensing and planning components.

[0379] Accidents can stem from sensing or planning errors. Planning is a difficult task for multi-agent systems because other road users (humans and machines) react to the actions of the AV. The described RSS model is designed to address safety issues in the planning component, among others. This can be termed multi-agent safety. In statistical methods, the probability of planning errors can be estimated "online." That is, after each software update, billions of miles must be driven with the new version to provide an acceptable level of estimation of the frequency of planning errors. This is clearly impractical. As an alternative, the RSS model provides a 100% (or nearly 100%) guarantee that the planning module will not commit errors attributable to the AV (the concept of "attribution" has been formally defined). The RSS model also provides an efficient way to validate it without relying on online testing.

[0380] Errors in sensing systems are often easier to verify because sensing can be independent of vehicle movement, and therefore the probability of serious sensing errors can be verified using "offline" data. However, even when collecting more than 10 9 Offline data from a single driving hour is also challenging. As part of the description of the disclosed sensing system, a fusion method that can be validated using a significantly smaller amount of data is described.

[0381] The described RSS system can also be scaled to millions of vehicles. For example, the described semantic driving strategy and the applied safety constraints can be aligned with sensing and mapping requirements that can be scaled to millions of vehicles even with today's technology.

[0382] The fundamental building block of this system is a thorough security definition, that is, the minimum standards that the AV system may need to comply with. In the following technical lemma, statistical methods for verifying AV systems are proven infeasible, even for verifying simple statements such as "the system has..." N "Number of incidents per hour." This means that model-based safety definitions are the only viable tool for verifying AV systems.

[0383] Lemma 1 Let X be a probability space, and let A be a probability space. Pr(A) = p 1<0.1 The event. Suppose we extract from X. Independent The sample is distributed, and let .but .

[0384] Proof using inequality 1 - x ≥ e -2x (Completeness was proven in Appendix A.1), thus we obtain .

[0385] Inference 1 Assuming AV system AV 1 An accident occurs, with probability p. 1 Small but insufficient. Given 1 / p 1 Any of the samples The deterministic verification procedure will (with constant probability) not be in AV 1 Unlike AV systems that never experience accidents 0 Between point.

[0386] To understand the typical value of this type of probability, assume the expected accident probability is 10. -9 / hour, while a certain AV system provides only 10 -8 The probability. Even if the system obtains 10 8 Even with a given driving hour, there is a constant probability that the verification process cannot indicate a danger in the system.

[0387] Finally, it should be noted that this challenge is designed to disable a single, specific, and hazardous AV system. (Complete) Solution It cannot be viewed as a single system because new versions, bug fixes, and updates will be necessary. From the validator's perspective, every change, even for a single line of code, will generate... New system Therefore, statistically validated solutions must be validated online with new samples after each small fix or change to account for changes in the state distribution observed and achieved by the new system. It is impractical to repeatedly and systematically obtain such a large number of samples (even then, with constant probability, it would be impossible to validate the system).

[0388] Furthermore, any statistical statement must be formalized to be measurable. A statement about the number of accidents that occur in a system is significantly weaker in terms of statistical nature than a statement that "it is driven safely." To say this, what constitutes safety must be formally defined.

[0389] Absolute safety is impossible If at some future time, cars c Actions takena If no accidents occur afterward, the action can be considered as... Absolutely safe It can be seen that by observing simple driving scenarios (e.g., such as...) Figure 19 Achieving absolute safety (as described) is impossible. From vehicle 1901's perspective, no action can guarantee that surrounding cars won't collide with it. Solving this problem by prohibiting autonomous vehicles from being in such situations is also impossible. Since every highway with more than two lanes will lead to this situation at some point, prohibiting this scenario is tantamount to requiring drivers to stay in their garages. At first glance, the consequences may seem disappointing. Nothing is absolutely safe. However, as defined above, such a requirement for absolute safety may be overly demanding, as evidenced by the fact that human drivers do not adhere to the requirement of absolute safety. Instead, human behavior follows a responsible safety concept.

[0390] Responsible for sensitive security (RSS) A crucial aspect missing from the concept of absolute safety is the asymmetry of most accidents—usually one of the drivers is responsible for the crash and will be held liable. Figure 19 In the example, if the car on the left, 1909, suddenly drives towards the car in the center, 1901, the car in the center will not be held responsible. To formalize this fact, considering that it is not responsible, the behavior of AV 1901 keeping in its own lane can be considered safe. For this purpose, a formal conception of "accident attribution" or accident responsibility is described, which can serve as a premise for a safe driving method.

[0391] As an example, consider the following simple scenario: two cars c f , c r Driving at the same speed along a straight road, one car is behind another. Assume... c f The car in front braked suddenly because of an obstacle on the road and managed to avoid it. Unfortunately, c r Not with c f Maintaining a sufficient distance, failing to react in time, and colliding. c f The rear side. Clearly, the blame lies with... c r The car behind has an obligation to maintain a safe distance from the car in front and be prepared to make reasonable emergency braking at any time.

[0392] Next, consider a broader series of scenarios: driving on a multi-lane road, where cars can freely change lanes, merge into the paths of other cars, and drive at different speeds. To simplify the following discussion, assume a straight road on a flat surface, where the lateral axis and longitudinal axis are respectively... x axis, y Axis. Under mild conditions, this can be achieved by defining a homomorphism between a real curved road and a straight road. Additionally, consider discrete time space. The definition can help distinguish between two intuitively different sets of cases: the simple case where no significant lateral manipulation occurs; and the more complex case involving lateral movement.

[0393] Definition 1 (Car Corridor) Car C corridor For range ,in For c The leftmost and rightmost corner positions.

[0394] Definition 2 (Entry Point) Car C 1 (for example, Figure 20A and 20B The car in 2003) cuts into the scene at time t. Car C 0 of (For example, Figure 20A and 20B (Cars in 2001) The corridor, provided that the car did not collide with c at time t-1. 0 The corridors intersect It intersects with it at time t.

[0395] The front and rear sections of a corridor can be further distinguished. The term "direction of entry" can describe movement in the direction of the relevant corridor boundary. These definitions can define situations involving lateral movement. For simple cases where this does not exist, such as one car following another, a safe longitudinal distance is defined: Definition 3 (Safe longitudinal distance) Car C r (Automotive 2103) With located at c r Another car in the front of the corridor f (Automotive 2105) Between Longitudinal distance 2101 ( Figure 21 ) Relative to the response time p is Safe The condition is that for c f The task to be carried out What braking command a, |a| if c max,brake , Starting from time p, its maximum braking will be applied until it comes to a complete stop. r The car will not be related to c A collision occurred. f ρ

[0396] The following Lemma 2 will d The function is calculated as follows: c r , c f speed, response time ρ and maximum acceleration a max,brake . ρ and a max,brake Both are constants, and their values ​​should be determined through adjustment to some reasonable level. In another instance, response time... Let c and maximum acceleration a max,brake Any of these can be set for a specific vehicle or vehicle type, or can be adapted / adjusted based on measurements of vehicle conditions, road conditions, user (e.g., driver or passenger) preferences, or parameters entered in other ways.

[0397] Lemma 2 To be located at c on the longitudinal axis r The vehicles behind. Let a f For maximum braking and max,brake 、a max,accel Accelerate the command and set ρ to c. The response time. Let υ r υ r Let l be the longitudinal speed of the car, and let l f Let be the length of the vehicle. f 、l r Define υ =υ p,max + ρ ∙ a r and define max,accel Let L = (l and . ) / 2. Then, c r +l f The minimum safe longitudinal distance is: r ρ Proof Order d t In time t The distance at that time. To prevent accidents, each must be kept at a safe distance. t time dt > L In order to build d min It is necessary to d Find the tightest lower bound required at 0. Clearly, d 0 must be at least L As long as the two cars are T ≥ υ If the car does not stop after a few seconds, the speed of the car in front will be... T a f – υ max,brake ,and c r The upper bound of the speed will be ρ ρ,max – (T – υ ) a max,accel .therefore, T The lower bound of the distance between the cars after 1 second will be: It should be noted that T r yes c r The time it takes to reach a complete stop (speed is 0), and T f This is the time it takes for the other vehicle to come to a complete stop. Note that... a max,brake ( T r – T f ) = – υ ρ,max ρa f + µ of two numbers a and b max,brake Therefore, if T r ≤ T f Then it requires d 0> L That's enough. If T r >T f ,but .

[0398] Requirement d Tr > L The proof is obtained by rearranging the terms.

[0399] Finally, we define comparison operators that allow comparisons with a certain "margin": when comparing lengths, speeds, etc., it is necessary to accept very similar numbers as "equal".

[0400] Definition 4 ( µ -Compare) The comparison is a> when a>b + µ. - b, in a b, and µ When |a – b| ≤ µ µ accidental It is the earliest time before the accident, including: At that time a> µ b .

[0401] The comparisons below (argmin, argmax, etc.) are for some appropriate µ of µ - Comparison. Assuming in a car c 1 、c An accident occurred between points 2 and 3. To determine who is at fault for the accident, we define the relevant point in time that needs to be checked. This is a point in time before the accident, and intuitively, it is the "point of no return"; after that, nothing can prevent the accident from happening.

[0402] Definition 5 (Time of Liability) There is an intersection between the corridors of one car and another car, and Time of Attribution Longitudinal distance is unsafe ● Assume car c ● An intersection occurs between them. That is, c .

[0403] Clearly, such a timeframe exists because both conditions are met at the moment the accident occurs. The time of liability can be broken down into two separate categories: ● There are also times of entry into the line of liability, that is, the first moment when a car and another car cross a corridor and are at an unsafe distance.

[0404] ● The time of no entry occurs, i.e., when the corridor has already intersected, the distance is within a safe longitudinal distance, and the distance changes to unsafe at the time of the attribution.

[0405] Definition 6 (due to lateral velocity) µ - Defeat) µ-failure: Its lateral velocity relative to the direction of entry is greater than c. 1 、c 2 The lateral velocity is high µ. 1. In the following cases, due to lateral velocity Assume car c 2 An intersection occurs between them. That is, c

[0406] It should be noted that the direction of speed is important: for example, a speed of -1, 1 (both cars collide with each other) results in a tie, but if the speed is 1, 1 + ... µIf the value is 2, then the blame lies with the car moving towards another car in a positive direction. Intuitively, this definition would allow us to blame a car that is driving very fast in a sideways direction towards another car.

[0407] Definition 7 (in lateral position) µ 1, µ 2) - Winning Its relative to the center of the cutting lane (the middle of the lane closest to the cutting-in corridor) 1 、c 2 The lateral position of the heart is less than µ 1. In the following situations, in a lateral position ( µ 1, µ 2) Win (In absolute value), and more than c The attribution or responsibility for the accident is... 1 A function of time state, and 2 has a small lateral position µ 2.

[0408] Intuitively, it cannot be attributed to being very close to the center of the lane. µ 1), and closer than the other car (difference) µ 2) The car.

[0409] Definition 8 (Attribution) of Automobiles c 1 、c 2 The definition is as follows: Time of Attribution If the time of attribution is not the time of entry, then the responsibility lies with the car behind. If the time of attribution is also the time of entry, then both cars are at fault, unless for one of the cars... ● The car, without loss of generality, is c

[0410] ● For some predefined µ, the following two conditions hold: The car will not lose speed due to lateral velocity. 1 The car won by virtue of its lateral position. ● Figure 22 ● object .

[0411] In this situation, C1 would be exempt. In other words, if an unsafe cut occurs, both vehicles would be at fault unless one of the vehicles was not (significantly) faster laterally and (significantly) closer to the center of the lane. This captures the desired behavior: if following a vehicle, maintain a safe distance; and if cutting into the corridor of a vehicle that is only driving in its own lane, do so only at a safe distance. Systems based on automated controllers used to comply with the above safety guidelines should not lead to overly defensive driving, as discussed further below.

[0412] Addressing Limited Sensing After considering the highway example, the second example addresses the problem of limited sensing. A very common human response when being held responsible for an accident falls into the category of "but I can't see him." This is often true. Human sensing capabilities are limited, sometimes due to unconsciously focusing on different parts of the road, sometimes due to carelessness, and sometimes due to physical limitations—the inability to see a pedestrian hidden behind a parked car. Of these human limitations, advanced automated sensing systems may only be affected by the latter: a 360° field of vision of the road, and the fact that computers are never careless, allowing these advanced automated sensing systems to surpass human sensing capabilities. Returning to the "but I can't see him" example, the appropriate answer is "Okay, you should be more careful." To formalize the need for caution regarding limited sensing, consider... The first time we see the object. The scene depicted. Car 2201 ( c 0) An individual is attempting to exit a parking lot and merge into a (potentially) busy road, but cannot see if there are any cars on the street because their view is obstructed by building 2203. Assume this is a narrow city street with a speed limit of 30 km / h. The human driver's behavior is to slowly merge into the road, gaining more and more field of view until the sensing limitation is eliminated. A temporally significant moment should be defined—the moment when the obstructed object is first exposed to us; after exposure, the obstructed object can be treated like any other sensorable object.

[0413] Definition 9 (Exposure Time) Assuming that at or after the exposure time, car c Exposure time With speed υ>υ .

[0414] Definition 10 (Attribution due to unreasonable speed) Driving, and c 1 (Car 2205) If this was not done, then the blame lies solely with c. limit In other words, c 0 υ 1 υ 1. Blamed for unreasonable speed.

[0415] This extension allows C0 to safely exit the parking lo...

Claims

1. A non-transitory computer-readable device storing instructions that, when executed by at least one computing device, cause at least one computing device to perform an operation, the operation comprising: Receive an image of the vehicle's driver; Based on the image, determine the driver's gaze direction; The path of the vehicle is predicted, at least in part, based on the gaze direction; as well as The vehicle is controlled based on the predicted path.

2. The apparatus of claim 1, wherein the image is a first image, and the operation further comprises: Receive a second image of the vehicle's surrounding environment; Identify objects in the second image; and Determine whether the gaze direction points to the object, wherein the prediction includes predicting the path based on whether the gaze direction points to the object.

3. The apparatus of claim 2, wherein the operation further comprises classifying the object as a type that a vehicle has driven through or a type that a vehicle has not driven through; wherein, based on classifying the object as a type that a vehicle has driven through, predicting the path comprises predicting that the path should point to the object.

4. The apparatus of claim 2, wherein the operation further comprises classifying the object as either a type that a vehicle passes through or a type that a vehicle does not pass through; wherein, based on classifying the object as a type that a vehicle does not pass through, predicting the path comprises predicting that the path will move away from the object's direction.

5. The apparatus of claim 1, further comprising acquiring a series of photographic images captured from at least one camera, the at least one camera being fixed to the vehicle and positioned to capture the surrounding environment of the vehicle, wherein predicting the path comprises predicting the path based on self-motion estimated from the series of photographic images.

6. The apparatus of claim 1, wherein predicting the path comprises predicting the path based on input from sensors on the vehicle.

7. The apparatus of claim 1, wherein the operation further comprises: The operation of the vehicle's advanced driver assistance system (ADAS) is adjusted based on the predicted path.

8. The apparatus of claim 1, wherein the control comprises: The degree of automatic braking is adjusted based on the fact that the object does not exist in the predicted path.

9. The apparatus of claim 1, wherein the control comprises: The driver is notified based on the predicted path.

10. A system for predicting a driver's path for a vehicle, the system comprising: One or more inward-facing cameras that monitor the driver of the vehicle; Memory; as well as At least one processor, said at least one processor being coupled to the memory and configured to: The image of the driver of the vehicle is received from an inward-facing camera that monitors the driver of the vehicle; Based on the image, determine the driver's gaze direction; The path of the vehicle is predicted, at least in part, based on the gaze direction; as well as The vehicle is controlled based on the predicted path.

11. The system of claim 10, wherein the image is a first image, the system further comprising one or more externally facing sensors that monitor the external environment of the vehicle, and the at least one processor is further configured to: Receive a second image of the vehicle's surrounding environment from one or more outward-facing sensors; Identify objects in the second image; and Determine whether the gaze direction points to the object, wherein the prediction includes predicting the path based on whether the gaze direction points to the object.

12. The system of claim 11, wherein the at least one processor is further configured to classify the object as a type that a vehicle has driven through or a type that a vehicle has not driven through; wherein, based on classifying the object as a type that a vehicle has driven through, predicting the path includes predicting that the path should point to the object.

13. The system of claim 11, wherein the at least one processor is further configured to acquire a series of photographic images captured from at least one camera, the at least one camera being fixed to the vehicle and positioned to capture the surrounding environment of the vehicle, wherein predicting the path comprises predicting the path based on self-motion estimated from the series of photographic images.

14. The system of claim 10, wherein predicting the path comprises predicting the path based on input from sensors on the vehicle.

15. The system of claim 10, wherein the at least one processor is further configured to: The operation of the vehicle's ADAS system is adjusted based on the predicted path.

16. The system of claim 10, wherein the control comprises: The degree of automatic braking is adjusted based on the fact that the object does not exist in the predicted path.

17. The system of claim 10, wherein the control comprises: The driver is notified based on the predicted path.

18. A computer-implemented method for predicting a driver's path for a vehicle, the computer-implemented method comprising: Receive an image of the vehicle's driver; Based on the image, determine the driver's gaze direction; The path of the vehicle is predicted, at least in part, based on the gaze direction; as well as The vehicle is controlled based on the predicted path.

19. The method of claim 18, wherein the image is a first image, the method further comprising: Receive a second image of the vehicle's surrounding environment; Identify objects in the second image; and Determine whether the gaze direction points to the object, wherein the prediction includes predicting the path based on whether the gaze direction points to the object.

20. The method of claim 19, further comprising classifying the object into a type that a vehicle has driven through or a type that a vehicle has not driven through; wherein, based on classifying the object into a type that a vehicle has driven through, predicting the path comprises predicting that the path should point to the object.

Images