Real-time trajectory planning system for autonomous vehicles with dynamic modeling of component-level system latency

Abstract

A system and method for autonomous driving that calculates a route trajectory incorporating the latency and calculation of vehicle components such that when a vehicle is in an autonomous driving mode and uses the trajectory to calculate a specified route, the latency of each vehicle component can be taken into account to provide a more accurate vehicle route.

Description

[Technical Field]

[0001] [Cross-reference to related patent applications] This application claims priority to U.S. Provisional Patent Application No. 63 / 373,039, filed August 19, 2022, entitled "TRAJECTORY PLANNER REAL TIME TRAJECTORY PLANNING SYSTEM WITH DYNAMIC MODELLING OF COMPONENT LEVEL SYSTEM LATENCY FOR SELF DRIVING VEHICLES," the disclosure of which is incorporated herein by reference in its entirety.

[0002] Embodiments of the present disclosure relate to systems and methods for providing a driving trajectory planner system with optimized trajectories, and more particularly, to systems and methods for optimizing driving paths, such as driving trajectories, as part of an autonomous driving system. [Background technology]

[0003] Generally described, computing devices and communication networks can be utilized to exchange data and / or information. In a typical application, a computing device can request content from another computing device over a communication network. For example, a computing device can collect various data and, utilizing a software application, exchange content with a server computing device over a network (e.g., the Internet).

[0004] Various vehicles, such as electric vehicles, internal combustion engine vehicles, and hybrid vehicles, can be configured with various sensors and components to facilitate operation of the vehicle and management of one or more systems included in the vehicle, such as an autonomous driving system (e.g., a self-driving system). In certain scenarios, a vehicle owner or vehicle user may desire to utilize a sensor-based system to facilitate operation of the vehicle's autonomous driving system. For example, vehicles often include hardware and software capabilities that facilitate autonomous driving by utilizing location-based services or accessing computing devices that provide location-based services. In another example, a vehicle may also include a navigation system or access navigation component that can generate information related to navigation and direction information provided to a vehicle occupant or user. In yet another example, a vehicle may include a vision system that facilitates autonomous driving by utilizing navigation and location-based services, safety services, or other operational services / components. [Brief explanation of the drawings]

[0005] The present disclosure is described herein with reference to drawings of specific embodiments, which are intended to illustrate, but not to limit, the disclosure. It should be understood that the accompanying drawings, which are incorporated in and constitute a part of this specification, are for the purpose of illustrating the concepts disclosed herein and may not be to scale.

[0006] [Figure 1] FIG. 1 illustrates a block diagram of an exemplary environment of a vision system in a vehicle, in accordance with one or more aspects of the present application.

[0007] [Figure 2A] FIG. 1 illustrates a block diagram of an exemplary environment corresponding to a vehicle, in accordance with one or more aspects of the present application.

[0008] [Figure 2B]1 is an exemplary vision system for a vehicle in accordance with one or more aspects of the present application.

[0009] [Figure 3] FIG. 1 illustrates a block diagram of an exemplary architecture for implementing a planning component, according to aspects of the present application.

[0010] [Figure 4A] FIG. 2 is a block diagram of the example environment of FIG. 1 illustrating the generation of simulated model content and the subsequent generation of a set of vision system training data for a machine learning process based on the simulated model content. [Figure 4B] FIG. 2 is a block diagram of the example environment of FIG. 1 illustrating the generation of simulated model content and the subsequent generation of a set of vision system training data for a machine learning process based on the simulated model content. [Figure 4C] FIG. 2 is a block diagram of the example environment of FIG. 1 illustrating the generation of simulated model content and the subsequent generation of a set of vision system training data for a machine learning process based on the simulated model content.

[0011] [Figure 5A] FIG. 2 is a block diagram of an exemplary interaction for controlling a vehicle.

[0012] [Figure 5B] 1 is an example of a data pipeline illustrating the latency in issuing commands to a vehicle.

[0013] [Figure 5C] 10 is an illustration of using estimated latencies of lateral and longitudinal control in trajectory representation.

[0014] [Figure 5D] 10 is a line graph showing an example of a run by modeling an optimized trajectory with and without considering system delays. DETAILED DESCRIPTION OF THE INVENTION

[0015] Although certain preferred embodiments and examples are disclosed below, the subject matter of the present invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Therefore, the claims appended hereto are not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described sequentially as multiple discrete operations, in a manner that may be helpful in understanding particular embodiments; however, the order of description should not be construed to imply that these operations are order-dependent. In addition, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not all such aspects or advantages are achieved by a particular embodiment. Thus, for example, various embodiments may be practiced in a manner that achieves or optimizes one advantage or advantages as taught herein without necessarily achieving other aspects or advantages that may be taught or suggested herein.

[0016] Autonomous vehicle software and hardware components have some inherent latency when a sensor measurement is evaluated from the initial timestamp at which it is taken to the final actuation (steering, accelerator pedal, braking), due to the time spent on calculations, communication across different modules, hardware actuation, etc. To provide a system capable of autonomously or semi-autonomously driving an autonomous vehicle like a human and making accurate decisions in safety-critical scenarios, the planning and decision-making system must factor the latency of each such component into the temporal decision-making process. Therefore, the systems described herein are configured to accurately model the latency of final actuations, such as steering, braking, or acceleration, independently. Additionally, the design should be efficient enough to avoid performing unnecessary millisecond-level calculations to make new decisions; otherwise, system latency and reaction times will be worse than if the unnecessary calculations were not performed.

[0017] Embodiments of the present invention relate to a trajectory planning system that can be run in real time on safety-critical systems such as autonomous vehicles and that is capable of accurately modeling vehicle behavior when accounting for the latency of individual actuator components.

[0018] In summary, some embodiments of the disclosed system may be incorporated into an autonomous vehicle and may perform one or more of the following processes to control operating parameters for autonomously driving the vehicle: 1. Sensor measurements from a specific set of sensors are processed by a perception system to generate information about the surroundings (e.g., world) around the vehicle, including but not limited to surrounding object information, road type information, driving direction information, etc. 2. A planning (e.g., decision-making) system may utilize the output of the perception system to derive several semantic decisions for the autonomous vehicle to follow. Based on the derived semantic decisions, the planning system may generate a high-fidelity trajectory plan (position, velocity, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk, lateral jerk). 3. The control built into the vehicle system may utilize trajectory planning to generate autonomous driving commands, such as speed and steering commands, that can be used to control the vehicle. 4. The generated steering command can be utilized by a vehicle steering command unit, such as an electronic power assist ECU, which acts as a motor controller for the steering motor, to steer the direction of the vehicle. 5. The speed, acceleration and jerk commands are sent to the vehicle motor controller, which converts the commands into final torque of the motor or brake pressure of the vehicle brake system, thus controlling the speed, position and direction of the vehicle.

[0019] In some embodiments, a vehicle planning and control system may perform one or more of these processes. Using this methodology, an autonomous vehicle's planning and control system receives a representation of the surrounding world measured by a sensor group (e.g., a set of sensors implemented in the vehicle) and processed by a perception system. The purpose of the planning system is to make decisions and plan a trajectory to achieve a specific goal of the autonomous vehicle, such as progressing along a desired navigation route while maintaining a safe and comfortable ride. The trajectory generated by the planning system is sent to a controller, which calculates actuator rates and steering commands to accurately track the output trajectory of the planning system. The vehicle planning and control system may be generally referred to as the planning system, and the planning system may be implemented as a planning component of the vehicle.

[0020] Embodiments of the present application correspond to systems and methods for providing a trajectory planner system within a vehicle by optimizing the trajectory of an autonomous vehicle while taking into account the latency required to process data within the system when the vehicle is in an autonomous driving mode. Optimizing or optimization, as referred to herein, may include classical computational techniques as well as machine learning (e.g., neural network)-based techniques. A trajectory may generally refer to a vehicle's expected driving trajectory. More specifically, the systems and methods are used to optimize a trajectory in an autonomous driving system before it enters the trajectory. Trajectory information may be determined based on data received from cameras or other sensors mounted on the vehicle using navigation services such as a global positioning system. For example, a user (e.g., a driver) may input a destination location into the vehicle's navigation system, and then the vehicle's autonomous driving system may determine one or more driving routes to the destination. In this example, the vehicle's autonomous system may determine a route to reach the destination. As the autonomous vehicle travels toward its destination, the vehicle continuously calculates an optimal trajectory based on road conditions, other vehicles, traffic signals, pedestrians, etc., to safely steer the vehicle to the destination. The trajectory may be calculated, for example, every few milliseconds, allowing the autonomous vehicle to directly change its route or change its speed to take current road conditions into account.

[0021] Traditionally, a vehicle's autonomous driving system is associated with physical sensors that provide input to the autonomous system. For example, physical sensors may include imaging systems and hardware processors and memory to support those imaging systems. Additionally, an autonomous system may include radar systems, LIDAR systems, etc., that can detect distances to objects and characterize attributes of detected objects. These components may have inherent latencies in evaluating data generated from these sensors for use in the autonomous driving (e.g., self-driving) of the vehicle. Each of these hardware components may also have different latencies reflecting the time it takes for the component to manage its data, transmit that data to the system, and calculate the vehicle's current trajectory. In addition, when a vehicle is autonomously driving, the system itself introduces latencies in calculating the vehicle's appropriate path. These latencies can pose challenges in determining the driving path and optimal trajectory because sensors, navigation systems, communication paths (e.g., communication buses), or processors may each be operating on data acquired at slightly different times during the vehicle's travel. In addition, the vehicle or surrounding environment may have changed during the few milliseconds that the system was calculating a preferred driving path, causing the system to calculate a path based on outdated information. For example, if other vehicles in the vicinity are changing their driving paths, the autonomous system may not be able to simultaneously determine the optimal trajectory in the changing environment due to latency in detecting and calculating data to detect the changes in the driving paths of other vehicles.

[0022] In some applications, these latencies are problematic when determining a vehicle path, such as a vehicle trajectory. For example, a vehicle's autonomous system may pre-plan a travel trajectory based on processed data received from various sensors before the vehicle travels along the calculated trajectory. In this example, by the time trajectory planning is complete and control commands are actuated to move the vehicle to a desired state (e.g., speed and steering angle), the vehicle may be moving in a state different from the initial state at which the planning system began planning due to latency. Latency can generally be referred to as a time delay due to a portion of the data used as input data for the vehicle's planning component. For example, the delay in a portion of the data may be due to sensor operation delays, communication delays, sensor data processing delays, etc. For example, if the planning system generates a desired trajectory that requires the vehicle to accelerate, the vehicle may not be able to achieve this acceleration until downstream systems process and act on this request. Subsequently, the vehicle's sensor system may not measure the vehicle acceleration corresponding to the desired trajectory acceleration until downstream systems process and act on this request. If this latency to the desired vehicle behavior encoded in the planning system's trajectory is large compared to the duration and time criticality of the maneuver the planning system is trying to achieve, it can lead to inaccurate decision making and trajectory planning.

[0023] To address at least some of the above shortcomings, aspects of the present application relate to using a planning system trained to consider the latencies of various components within a vehicle. When each vehicle uses a planning system configured to incorporate the latencies of various components when generating route information, the system can properly calculate the vehicle's trajectory as it travels along a specified route to a destination.

[0024] A general aspect of the present application relates to determining (or estimating) latency for each of the hardware components and software models implemented in the system and using the determined latency in decision-making logic. In these aspects, the system may utilize various neural network-based planning systems.

[0025] In some embodiments, the system is configured to determine the latency corresponding to each hardware component and software model used in the autonomous driving of the vehicle. For example, if the vehicle uses a front camera, a side camera, and a rear camera for autonomous driving, the system determines the respective latency corresponding to each of these cameras. In this example, the system can also determine any computational latency due to software or systems processing data received from these cameras for use in autonomous driving. In some embodiments, these latencies are utilized in the temporal decision-making process of autonomous driving. In these embodiments, based on these latencies, the system determines the exact time sequence of data received or processed from each hardware component or software model. For example, if the latencies of the front camera, the side camera, and the rear camera are 1 ms, 2 ms, and 3 ms, respectively, images captured by the front camera, the side camera, and the rear camera 1 ms, 2 ms, and 3 ms before the reference time are used as images for autonomous driving. These hardware component and software models are provided merely as examples, and various hardware component configurations and software models can be used based on the application.

[0026] In some embodiments, the system provides for determining an optimal trajectory in real time while operating a vehicle, such as in an autonomous driving environment. In these embodiments, the system may provide for real-time calculation of the optimal trajectory while taking into account the latency of each component when combining one or more data received from a navigation system and / or sensors onboard the vehicle. For example, a deep neural network (DNN) may be trained as part of the autonomous system and may output the latency of the components.

[0027] In some embodiments, the optimal trajectory is derived by utilizing a trained planning system or model. The optimal trajectory refers to the optimal driving path and associated vehicle operating parameters, such as position, speed, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk, and lateral jerk, that can achieve a comfortable, safe, and efficient driving experience. The planning model implemented in the vehicle's planning component can determine the optimal trajectory path using data from one or more sensors on the vehicle or by utilizing a navigation system or service. In some embodiments, the planning model can simulate multiple trajectories and determine the optimal trajectory for the vehicle's operating parameters. In these embodiments, the determination is based on determining the vehicle's operating parameters corresponding to each simulated trajectory. For example, the planning model may generate a vectorization model for each simulated result and determine the vehicle operating parameters for each vectorization model. The planning model may then determine the optimal trajectory that can achieve a comfortable, safe, and efficient driving experience. In some embodiments, the planning model may predict any latencies that may be due to sensors and computing processors. In these embodiments, based on the predicted latencies, the planning model may allocate computing resources taking these latencies into account.

[0028] In some embodiments, an autonomous system onboard a vehicle relies solely on a vision system to determine an optimal trajectory. Illustratively, a vision-only system is in contrast to a vehicle that may combine a vision-based system with one or more additional sensor systems, such as a radar-based system, a LIDAR-based system, or a SONAR system. In some embodiments, a vision-only system may consist of a machine-learned process that can process inputs solely from a vision system that includes multiple cameras mounted on the vehicle. The machine-learned process can identify objects and generate outputs specifying characteristics / attributes of the identified objects, such as position, velocity, and acceleration, as measured relative to the vehicle. The outputs from the machine-learned process can then be utilized for further processing in navigation systems, positioning systems, safety systems, and the like.

[0029] According to aspects of the present application, a network service can configure a machine-learned process according to a supervised learning model in which the machine-learned process is trained with labeled data including identified objects and specified characteristics / attributes, such as position, velocity, acceleration, etc. A first portion of the training dataset corresponds to data collected from a target vehicle that includes a vision system, such as a vision system included in an in-vehicle vision-only system. Additionally, a second portion of the training data corresponds to additional information obtained from another system, i.e., a simulated content system, capable of generating video images and associated attribute information.

[0030] Illustratively, the network service can receive a combined set of inputs (e.g., a first data set and a second data set) from the target vehicles, including both visual and simulated content. The network service then combines the data based on a standardized data format. Illustratively, the combined data set enables previously collected visual data to be supplemented with additional information or attributes / characteristics that may not have been obtained from processing the visual data. The combined data set can result in a data set that tracks objects for a defined set of times based on the first and second data sets. The network service can then process the combined data set using various techniques. Such techniques include smoothing, extrapolating missing information, applying kinematic models, applying confidence values, etc. The network service then generates an updated machine-learned process based on training on the combined data set. The trained machine-learned process can be sent to a vision-only vehicle or to a target vehicle equipped with both a vision and detection system, where the process can be repeated to further update / refine the machine-learned process.

[0031] While various aspects are described according to example embodiments and feature combinations, those skilled in the relevant art will understand that the examples and feature combinations are exemplary in nature and should not be construed as limiting. More specifically, aspects of the present application may be applicable to various types of vehicles, including vehicles having different propulsion systems, such as combined engines, hybrid engines, electric engines, etc. Furthermore, aspects of the present application may be applicable to various types of vehicles that may incorporate different types of sensors, sensing systems, navigation systems, or location systems. Thus, examples should not be construed as limiting. Similarly, aspects of the present application may be combined with or implemented in conjunction with other types of components that may facilitate vehicle operation, including autonomous driving applications, driver convenience applications, etc.

[0032] FIG. 1 illustrates a block diagram of an exemplary environment 100 for generating simulated content models and training set data for an in-vehicle vision system in accordance with one or more aspects of the present application. System 100 may include a network connecting a first set of vehicles 102, a network service 110, and a simulated content system 120. Illustratively, various aspects related to network service 110 and simulated content system 120 may be implemented as one or more components associated with one or more functions or services. The components may correspond to software modules implemented or executed by one or more external computing devices, which may be separate, standalone external computing devices. Thus, the components of network service 110 and simulated content system 120 should be considered logical representations of services and do not require any specific implementation on one or more external computing devices.

[0033] The network 106 connects the devices and modules of the system as shown in FIG. 1. The network can connect any number of devices. In some embodiments, a network service provider offers network-based services to client devices over the network. A network service provider implements network-based services and refers to a large, shared pool of network-accessible computing resources (such as compute, storage, or networking resources, applications, or services), which may be virtualized or bare metal. The network service provider can offer on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to match fluctuating loads. Thus, the concepts of "cloud computing" or "network-based computing" can be thought of as both applications delivered as services over a network and the network service provider's hardware and software that provides those services. In some embodiments, the network can be a content delivery network.

[0034] Illustratively, the set of vehicles 102 corresponds to one or more vehicles configured with a vision-only based system for identifying objects and characterizing one or more attributes of the identified objects. The set of vehicles 102 is configured with a machine-learned process, e.g., implemented in a supervised learning model, configured to identify objects and characterize attributes of the identified objects, such as position, velocity, and acceleration attributes, utilizing only vision system input. The set of vehicles 102 may also be configured without an additional detection system, such as a radar detection system, a LIDAR detection system, or the like.

[0035] Illustratively, network service 110 may include multiple network-based services that can provide functionality for configuring / responding to requests for machine-learned processes for vision-only-based systems, as applied to aspects of the present application. As shown in FIG. 1 , network-based service 110 may acquire datasets from vehicle 102 and simulated content system 120, process the datasets to form training material for the machine-learned processes, and generate machine-learned processes for vision-only-based vehicle 102. Visual Information Processing Component 112 The network-based service may include multiple data stores for maintaining various information related to aspects of the present application, including a vehicle data store 114 and a machine-learned process data store 116. The data stores in FIG. 1 are logical in nature and may be implemented in the network service 110 in a variety of ways.

[0036] Similar to network services 110, simulated content system 120 may include a number of network-based services that can provide functionality related to providing a visual frame of data and associated data labels for machine learning applications as applied to aspects of the present application. As shown in FIG. system 120 can create various scenarios according to a set of defined attributes / variables, Simulated Content Services A scenario generation component 122 may be included. Simulated content system 120 may include multiple data stores for maintaining various information related to aspects of the present application, including a scenario clip data store 124 and a ground truth attribute data store 126. The data stores of FIG. 1 are logical in nature and may be implemented in a simulated content service in a variety of ways.

[0037] For illustrative purposes, FIG. 2A illustrates an environment corresponding to a vehicle in the set of vehicles 102, in accordance with one or more aspects of the present application. The environment includes a collection of local sensors 225 that can provide input for the operation of the vehicle and for the collection of information, as described herein. The collection of local sensors 225 can include one or more sensor systems or sensor-based systems mounted on the vehicle or accessible to the vehicle during operation. The local sensors 225 or sensor systems may be integrated into the vehicle. Alternatively, the local sensors or sensor systems may be provided by an interface associated with the vehicle, such as a physical connection, a wireless connection, or a combination thereof.

[0038] In one aspect, the local sensors 225 may include a vision system that provides input to the vehicle, such as object detection, attributes of the detected objects (e.g., position, velocity, acceleration), the presence of environmental conditions (e.g., snow, rain, ice, fog, smoke, etc.), etc. An exemplary collection of cameras mounted on the vehicle to form a vision system is described with reference to FIG. 2B . As previously mentioned, the vehicle 102 relies on such a vision system for defined vehicle operation functions without assistance from or in place of other conventional detection systems.

[0039] In yet another aspect, local sensors 225 may include one or more positioning systems capable of obtaining reference information from external sources, enabling various levels of accuracy in determining the vehicle's positioning information. For example, the positioning system may include various hardware and software components for processing information from GPS sources, wireless local area network (WLAN) access point information sources, Bluetooth information sources, radio frequency identification (RFID) sources, etc. In some embodiments, the positioning system may obtain a combination of information from multiple sources. Illustratively, the positioning system may obtain information from various input sources to determine the vehicle's positioning information, particularly altitude at its current location. In other embodiments, the positioning system may also determine movement-related operating parameters, such as direction of travel, speed, and acceleration. The positioning system may be configured as part of the vehicle for multiple purposes, including automated driving applications, augmented driving, or user-assisted navigation, etc. Illustratively, the positioning system may include a planning component 212 linked to data 230 that facilitates identification of various vehicle parameters or process information.

[0040] In yet another aspect, local sensors 225 may include one or more navigation systems 235 for identifying navigation-related information. Illustratively, the navigation system may obtain positioning information from a positioning system and identify characteristics or information related to identified location data (e.g., received from a global positioning system (GPS) 245), such as altitude, road grade, etc. The navigation system 235 may also identify recommended or intended lane locations on multi-lane roads based on directions provided or expected to the vehicle user. Similar to a positioning system, a navigation system may be configured as part of the vehicle for multiple purposes, including automated driving applications, augmented driving, or user-assisted navigation. The navigation system may be combined or integrated with planning component 212, which is part of the positioning system.

[0041] The local resources may include one or more planning components 212, which may be hosted on the vehicle or on a computing device (e.g., a mobile computing device) accessible from the vehicle. The planning component 212 may illustratively access input from a local sensor 225 or sensor system and process the input data as described herein. For purposes of this application, the planning component 212 will be described with respect to one or more functions relevant to the illustrative embodiment. For example, the planning component 212 in the vehicle 102 collects and transmits a first data set corresponding to collected visual information.

[0042] The environment may further include various additional sensor components or sensing systems operable to provide information regarding various operating parameters for use according to one or more operating conditions. The environment may further include one or more control components 228 for processing the output, such as transmitting the data via a communication output 240, generating the data in a memory, or transmitting the output to other planning components.

[0043] According to one or more aspects described herein, the planning component 212 can be configured to provide vehicle control commands (e.g., steering, acceleration, pedal, brake, speed, etc.) by estimating inherent latencies such as data computation latency, internal data communication latency, and hardware (e.g., steering actuator, motor, etc.) actuation latency to process the sensor data. In some embodiments, the planning component 212 can include a perception system 212. The perception system 212 can be configured to receive sensor data from local sensors 225, navigation 235, and GPS 245. After receiving the sensor data, the perception system 212 can process the received data to determine vehicle states such as position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration. The planning component 212 can utilize the determined vehicle states to estimate latency and generate vehicle operating parameter control commands. For example, the planning component 212 can classify vehicle states into lateral states and longitudinal states. The lateral states can refer to vehicle operation variables related to vehicle position, heading, steering angle, and steering angle rate. The longitudinal states can refer to vehicle operation variables related to vehicle speed, acceleration, longitudinal jerk, and lateral snap. The lateral states can be associated with controlling vehicle steering actuators, and the longitudinal states can be associated with controlling vehicle motor actuators. The planning component 212 can determine latencies for controlling individual steering and motor actuators. For example, the planning component 212 can determine the time latency from measuring vehicle states to controlling individual steering and motor actuators. Exemplary latencies can include time delays in receiving input data from local sensors (e.g., measurement latency), processing the local sensor data (e.g., computation latency), sending and receiving data between components (e.g., communication latency), and operating the individual steering and motor actuators (e.g., hardware latency). The planning component 212 can accurately plan vehicle operations by optimizing the vehicle trajectory based on the determined latencies.

[0044] Referring now to FIG. 2B , an exemplary vision system 200 for a vehicle will be described. The vision system 200 includes a set of cameras capable of capturing image data during operation of the vehicle. As described above, individual image information may be received at specific frequencies such that the depicted image represents a specific timestamp of the image. In some embodiments, the image information may represent a high dynamic range (HDR) image. For example, different exposures may be combined to form an HDR image. As another example, images from an image sensor may be preprocessed (e.g., using a planning model) to convert them into an HDR image.

[0045] As shown in FIG. 2B, the set of cameras may include a set of forward-facing cameras 202 that capture image data. The forward-facing cameras may be mounted slightly elevated on the windshield portion of the vehicle. As shown in FIG. 2B, the forward-facing cameras 202 may include multiple individual cameras configured to generate a composite image. For example, the camera housing may include three image sensors pointing forward. In this example, the first image sensor may have a wide-angle (e.g., fisheye) lens. The second image sensor may have a normal or standard lens (e.g., a 35mm equivalent focal length, a 50mm equivalent focal length, etc.). The third image sensor may have a zoom or narrow lens. In this manner, the vehicle can capture three images of various focal lengths in the forward direction. The vision system 200 further includes a set of cameras 204 mounted on the door pillars of the vehicle. The vision system 200 further includes two cameras 206 mounted on the front bumper of the vehicle. Additionally, the vision system 200 may include a rear-facing camera 208 mounted on the rear bumper, trunk, or license plate holder.

[0046] The set of cameras 202, 204, 206, and 208 may all provide captured images to one or more planning components 212, such as a dedicated controller / embedded system. For example, the planning component 212 may include one or more matrix processors configured to rapidly process information related to the planning model. In some embodiments, the planning component 212 may be used to perform convolutions related to forwarding paths through a convolutional neural network. For example, the planning component 212 may convolve input data with weight data. The planning component 212 may include multiple multiply-accumulate units to perform the convolutions. As an example, the matrix processor may use organized or formatted input data and weight data to facilitate larger convolution operations. Alternatively, the image data may be sent to a general-purpose planning component. The planning component 212 may be referred to as an autonomous driving planning component.

[0047] Illustratively, each camera may operate individually and be considered a separate input of visual data for processing. In other embodiments, one or more subsets of camera data may be combined to form composite image data, such as for a trio of forward-facing cameras 202.

[0048] In some embodiments, the planning component 212 can process captured images to determine a driving path, and the vehicle 200 is configured to automatically drive to a destination specified by a user (e.g., a driver) using a navigation system. In these embodiments, the planning component can simulate multiple driving paths over a specific distance (e.g., over a specific number of time frames) and determine an optimized driving path in real time. For example, if the vehicle 200 needs to change lanes, the planning component 212 simulates multiple driving paths for the lane change based on the lanes and surrounding objects, such as surrounding vehicles and any objects. Each simulated result can be vectorized based on vehicle operation parameters such as acceleration and speed. The planning component 212 can then utilize a planning system to determine an optimal driving path for the lane change. The planning system can be configured to analyze the vectorized results to determine an optimal driving path for the lane change.

[0049] In some embodiments, planning component 212 can determine an optimal trajectory when vehicle 200 needs to move along a trajectory during its autonomous driving. In these embodiments, when vehicle 200 needs to move within a trajectory, such as turning in a particular direction, planning component 212 can determine an optimal trajectory that can provide the driver with a comfortable, safe, and efficient driving experience. For example, when vehicle 200 needs to turn left, planning component 212 can simulate multiple trajectories for making the left turn. Each simulated trajectory can be vectorized based on vehicle operation parameters, such as acceleration and speed, before the vehicle turns in the specified direction. Planning component 212 can then utilize a planning model to determine an optimal trajectory based on driving comfort, safety, and efficiency, and the vehicle utilizes the optimal trajectory during its autonomous driving.

[0050] An exemplary architecture for implementing the planning component 212 on one or more local resources or network services will now be described with reference to Figure 3. The planning component 212 may be part of a component / system that can provide functionality related to autonomous driving.

[0051] 3 is exemplary in nature and should not be construed as requiring a particular hardware or software configuration for the planning component 212. The general architecture of the planning component 212 shown in FIG. 3 includes an arrangement of computer hardware and software components that may be used to implement aspects of the present disclosure. As shown, the planning component 212 includes a processing unit 302, a network interface 304, a computer-readable media drive 306, and an input / output device interface 308, all of which may communicate with each other via a communication bus. The components of the planning component 212 may be physical hardware components or may be implemented in a virtualized environment.

[0052] The network interface 304 may provide connectivity to one or more networks or computing systems, such as the network 106 of Figure 1. Thus, the processing unit 302 may receive information or instructions from other computing systems or services over a network. The processing unit 302 also includes a memory 3 10 and may further provide output information for autonomous driving via an input / output device interface. In some embodiments, planning component 212 may include more (or fewer) components than those shown in FIG.

[0053] The memory 310 may include computer program instructions that the processing unit 302 executes to implement one or more embodiments. The memory 310 generally includes RAM, ROM, or other persistent or non-transitory memory. The memory 310 may store an operating system 312 that provides computer program instructions for use by the processing unit 302 in the general management and operation of the planning component 212. The memory 310 may further include computer program instructions and other information for implementing aspects of the present disclosure. For example, in one embodiment, the memory 310 stores a sensor interface component 312 that obtains information from the vehicle 102, a data store, other services, etc. 4 Includes:

[0054] The memory 310 stores visual information for acquiring and processing one or more data sets, including simulated content according to various vehicle operating conditions, as described herein. process It further includes a component 316 .

[0055] The memory 310 may further include a planning model 318 for generating or training a machine-learned process for use in autonomous driving of the vehicle 102. In some embodiments, the planning model 318 uses data from vehicle systems to determine optimal driving paths and optimal vehicle operating parameters that can achieve comfortable, safe, and efficient autonomous driving. In these embodiments, the planning model 318 is integrated into the trajectory planner component 32. 2 and trajectory optimization component 32 4 For example, the trajectory optimization component 32 may determine optimal operating parameters for the vehicle based on inputs received from 4may provide multiple possible trajectories in the form of multiple vectored data that the vehicle can utilize to turn in a particular direction. In this example, the planning model 318 may analyze the vectored data for each trajectory based on vehicle operating parameters such as vehicle speed, acceleration, etc. The planning model 318 then determines an optimal trajectory that can achieve an optimal trip in terms of comfort, safety, and driving efficiency.

[0056] The memory 310 may further include a latency calculation component 320 for determining latency due to hardware components or software models used for autonomous driving. The latency calculation component 320 may be implemented by utilizing the planning model 318. In some embodiments, the latency calculation component 320 may determine each latency due to hardware components, such as cameras, sensors, or any system components related to decision-making for autonomous driving. The latency calculation component 320 may further determine latency due to any computational software models that process data related to decision-making for autonomous driving. In some embodiments, the calculated latency may be used by individual components or combinations of components in the memory 310. In some embodiments, the calculated latency may be updated in real time.

[0057] The memory 310 may further include a trajectory planner component 322 for generating multiple travel paths usable for autonomous navigation. In some embodiments, while the vehicle is operating autonomously, the trajectory planner component 322 2The trajectory planner component 322 searches for all possible driving paths in real time. For example, when a vehicle needs to change lanes, the trajectory planner component 322 can generate multiple driving paths for the lane change. In this example, the vehicle can analyze its surrounding environment, such as other vehicles or any objects surrounding the vehicle. After determining the driving paths, the trajectory planner component 322 can convert each driving path into vectorized data, which is sent to the planning model 318.

[0058] The memory 310 may further include a trajectory optimization component 324 for generating multiple trajectories that can be used in autonomous driving when the vehicle needs to move in a certain direction. In some embodiments, when the vehicle needs to move in a certain direction, the trajectory optimization component 324 searches all possible trajectories in real time. For example, when the vehicle needs to turn right, the trajectory optimization component 324 may generate multiple trajectories for making the right turn. In this example, the trajectory optimization component 324 may convert each driving path into vectorized data, and the vectorized data is sent to the planning model 318. In some embodiments, the trajectory optimization component 324 may represent a neural network that outputs a trajectory. For example, the neural network may be trained to output a trajectory based on an input (e.g., the input may represent at least the output of a vision stack that outputs objects positioned around the vehicle).

[0059] In some embodiments, the trajectory optimization component 324 may simulate the vehicle's travel on a trajectory. In these embodiments, for each simulation on a particular trajectory, the trajectory optimization component 324 may generate simulation results including data related to, for example, position, speed, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk, or lateral jerk while traveling on the travel path. The trajectory optimization component 324 may evaluate the suitability of the trajectory using an objective function whose value is maximized. In some embodiments, the memory 310 may further include a vehicle controller component 326 for controlling the vehicle to travel on the trajectory path determined by the trajectory optimization component 324. In some embodiments, the vehicle controller component 326 generates commands related to vehicle operation, such as steering, acceleration, and braking. The commands may include, for example, steering commands to control a steering motor, velocity and acceleration commands to control a motor, and jerk commands to control a braking system. For example, controller component 326 generates these commands based on data related to the trajectory path determined in trajectory optimization component 324 (e.g., data related to position, velocity, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk, or lateral jerk). In one embodiment, controller component 326 can be implemented as a standalone component such as a controller. In some embodiments, controller component 326 may implement classical and / or modern control techniques (e.g., PID controllers, nonlinear control techniques, adaptive and learning techniques, etc.). In some embodiments, as illustrated in FIG. 5A , controller component 326 may be omitted so that component 324 can directly output commands (e.g., lower-level commands) such as steering, acceleration, braking, etc.

[0060] Although shown as combined components within planning component 212, those skilled in the art will understand that one or more of the components within memory 310 may be implemented in separate computing environments, including both physical and virtualized computing environments.

[0061] 4A-4C, an exemplary interaction of components of an environment that processes vision system data and generates simulated content system data to update a training model for a machine learning process is described. At (1), one or more vehicles 102 can collect and transmit a set of inputs (e.g., a first data set). The first data set illustratively corresponds to video image data and any associated metadata or other attributes collected by the vision system 200 of the vehicle 102.

[0062] Illustratively, the vehicle 102 may be configured to collect vision system data and transmit the collected data. For example, the collected vision system data may be transmitted based on a periodic time frame or various collection / transmission criteria. Furthermore, in some embodiments, the vehicle 102 may also be configured to identify a particular scenario or location that results in the collection and transmission of the collected data, such as via geographic coordinates or other identifiers. As shown in FIG. 4A, at (2), the collected vision system data may be transmitted to the simulated content service 122 directly from the vehicle 102 or indirectly through the network service 110.

[0063] At (3), the simulated content service 122 receives and processes the vision system data collected from the vehicle 102. Illustratively, the simulated content service 122 may process the vision-based data to complete lost frames of video data, update version information, correct errors, and the like. Additionally, at (3), the simulated content service 122 may further process the collected vision system data to identify ground truth labels for the captured video data. Illustratively, the ground truth labels may correspond to any one of various detectable objects that may be depicted in the video data. In one embodiment, the ground truth label data may include information identifying a road edge. Additionally, the ground truth label data may include information dependent on the identified road edge, such as a lane, a road center, and one or more stationary objects (e.g., road signs, landmarks, etc.). Furthermore, in some embodiments, the ground truth label data may include dynamic object data associated with one or more identified objects, such as a vehicle, a dynamic obstacle, an environmental object, etc.

[0064] In (4), the simulated content service 122 can process the ground truth label data. Illustratively, the simulated content service 122 can process the ground truth labels according to a priority to identify / extract core ground truth label data to be used as the basis for the simulated content. Illustratively, lane edge ground truth labels can be considered to have a high or higher priority. Additional ground truth label data, such as lane labels, lane center labels, static object labels, or dynamic object labels, can be associated with a low or lower priority with respect to the lane label data or with respect to each other. In some embodiments, the label data can be filtered to exclude one or more labels (e.g., dynamic objects) that can be replaced by the simulated content or that are not required to generate the simulated content. For illustrative purposes, the set of processed ground truth labels can be considered content model attributes for the simulated content.

[0065] At (5), the simulated content service 122 generates a model for future generation of simulated content. Illustratively, the simulated content service 122 can process content attributes in a manner such as error adjustment, extrapolation, or variation.

[0066] At (6), the simulated content service 122 can generate index or attribute data (e.g., metadata) for each clip or simulated content data that facilitates data selection, sorting, or maintenance. The index or attribute data can include location identification, type of simulated object, number of generated / available variants, simulated environmental conditions, tracking information, origin information, etc. For purposes of FIG. 4A, simulated content can be generated without the specific requirements / needs for the training scenarios described with respect to FIG. 4B.

[0067] Referring to FIG. 4B , illustratively, the stored and indexed simulated content information can be provided to the network service 110 as part of the training data. In (1), the simulated content service 122 can receive a selection or criteria for selecting data. Illustratively, the computing device 104 can be utilized to provide criteria, such as classification criteria. In some embodiments, a request for simulated content is utilized to provide attributes of the simulated content. Thus, the generation of the simulated content can be considered responsive to the request for simulated content in itself. Thus, the generation of the simulated content can be considered to be synchronous in nature or dependent in nature. In other embodiments, the request can be a simple selection of an index value or attribute, such that the simulated content service 122 can generate simulated content based on pre-configured attributes or configurations that are independent of the individual request for simulated content. Thus, the generation of the simulated content can be considered request-independent.

[0068] At (2), the network service 110 can then process the request and identify the generated simulated content model, such as via the index data. At (3), the simulated content service 122 generates supplemental video image data and associated attribute data. Illustratively, the simulated content system 120 can utilize a set of modifiable variables or attributes to create different scenarios or scenes for use as supplemental content. For example, the simulated content system 120 can utilize color attributes, object type attributes, acceleration attributes, motion attributes, data time attributes, location / position attributes, weather condition attributes, and vehicle density attributes to create various scenarios related to the identified objects. Illustratively, the supplemental content can be utilized to emulate real-world scenarios that are unlikely to occur or be observed by a set of vehicles 102. For example, the supplemental content can emulate various scenarios corresponding to unsafe or dangerous conditions.

[0069] The simulated content system 120 can illustratively utilize statistical selection of scenarios to avoid repetition based on minor differences (e.g., similar scenarios that differ only by object color) that could otherwise bias the machine learning process. 120 The content service 122 may be configured to provide a number of supplemental content frames and a distribution of differences in one or more variables. offerIllustratively, output from simulated content service 122 may include labels (e.g., ground truth information) that identify one or more attributes (e.g., position, velocity, and acceleration) that can be detected or processed by network service 110. In this regard, the simulated content dataset may facilitate detailed labeling, which may be dynamically adjusted as needed for different machine-learned training sets. At (4), the simulated content training set is transmitted to network service 110.

[0070] 4C, once network service 110 receives the training set, at (1), network service 110 processes the training set. At (2), network service 110 1 0 generates an updated machine-learned process based on training on the combined dataset. Illustratively, network service 110 can utilize various planning models to generate the updated machine-learned process.

[0071] Referring to FIG. 5A, a block diagram of an exemplary interaction of the controlled vehicles is described. As shown in FIG. 5A, the planning component 212 (FIGS. 2A, 2B, and 3) to2A and 2B , as well as a processor for processing the output data of these sensors. In some embodiments, the planning component 212 is configured to make decisions and plan a trajectory (e.g., a driving trajectory) to achieve certain driving criteria for the autonomous vehicle, such as progressing along a desired navigation route while maintaining a safe and comfortable ride. In some embodiments, the trajectory generated by the planning component 212 is sent to a controller, which calculates actuator velocity and steering commands to accurately track the output trajectory of the planning system. In these embodiments, the controller generates commands related to vehicle operations, such as steering, acceleration, and braking. The commands include, for example, steering commands to control the steering motor, velocity and acceleration commands to control the motor, and jerk commands to control the braking system. For example, the controller component 326 generates these commands based on the trajectory generated by the planning component 212.

[0072] 5A illustrates an example of vehicle components for controlling vehicle operation parameters for autonomous driving. As shown in FIG. 5A, the vehicle may include some or all of a sensor suite 512, a perception system 514, a planning system 516, a controller 518, and actuators 520. These components are exemplary and provided for purposes of example, and the present application is not intended to limit the systems or components of the vehicle. In some embodiments, the sensor suite 512 via the perception system 514, the planning system 516, and the controller 518 may be implemented as the planning component 212. In some embodiments, the autonomous driving may be performed based on a set of instructions that an autonomous processor implemented in the vehicle executes in real time while the vehicle is traveling.

[0073] The sensor suite 512 can collect sensor data measured by various sensors implemented in the vehicle. For example, the sensor suite 512 can collect data from the local sensors 205, the navigation 235, and the GPS 245, as shown in FIG. 2A . The collected sensor data can be transmitted to the perception system 514. In some embodiments, the perception system 514 processes the collected data and determines a current vehicle state, such as position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration. In some embodiments, each of the current vehicle states can be used as input data to an autonomous processor (not shown in FIG. 5A ), which can generate vehicle control commands based on the current vehicle states. In some embodiments, the generated commands are not used directly to control the vehicle. Instead, the planning system 516 can further process the generated commands to optimize vehicle operation. For example, the planning system 516 can determine a time latency for controlling vehicle actuators. The planning system 516 can determine time latencies to estimate overall system latency; for example, the time latencies may be selected as a measure of maximum, minimum, or central tendency of actuator time latencies, or may be selected as a measure of maximum, minimum, or central tendency of longitudinal or lateral latencies (e.g., actuators associated with longitudinal or lateral control). The system 516 can also determine the time latencies of each vehicle actuator (e.g., steering and motor actuators). For example, each data measured by the local sensors 225, navigation 235, and GPS 245 (shown in FIG. 2A) can include a timestamp, and the measured data can be used to control the steering and motor actuators. The planning system 516 can determine latencies from measurements of data for actuating the steering and motor actuators. In some embodiments, the latencies vary for each actuator, even if the data is measured simultaneously.For example, latency can include the time delay in receiving input data from local sensors (e.g., measurement latency), processing the local sensor data (e.g., computation latency), sending and receiving data between components (e.g., communication latency), and operating individual steering and motor actuators (e.g., hardware latency).

[0074] After estimating the latency, the planning system 516 may begin command calculations. For example, if the latency for controlling a steering actuator is longer, the planning system 516 may wait the determined latency period after receiving a command to control a motor actuator to receive a command related to the steering actuator. In some embodiments, the planning system 516 may update the received commands for each steering and motor actuator. For example, if a motor actuator control command is received earlier than a steering actuator control command (due to different latencies), the planning system may reduce the motor speed if the later-received steering actuator control command includes a higher steering tilt angle. Once the planning system 516's calculations are complete, the results, in some embodiments, are sent to the controller 518 to calculate control commands. In some embodiments, if the measured environment changes, the planning system 516 may restart a new calculation cycle by receiving new measurements from the perception system 514 and sending updated vehicle operation commands to the controller 518. For example, the cycle may restart when the vehicle's driving trajectory changes. These cycles or trajectory updates may be performed in real time, such as repeatedly running in a calculation loop. For example, the loop may receive new measurements from the sensor suite 512, collect data measured by the perception system 514, process the collected data, and send it to the planning system 511. 6and translating the planning system 516 into control commands by the controller 518. In some embodiments, the planning system 516 can output the control commands directly to the actuators 520. For example, the planning system 516 can include a neural network trained to output the control commands. Thus, in some embodiments, the controller 518 may be optional or omitted.

[0075] In some embodiments, at the beginning of each planning cycle (e.g., at the beginning of each loop), planning system 516 can receive perception information about the state of the vehicle (e.g., position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration) from perception system 514. Planning system 516 can use this state information to determine trajectory information, such as the start of the trajectory planned for the current cycle. In some embodiments, this state information can be used to estimate how far the vehicle has traveled along the trajectory of the previous cycle based on the state information of the vehicle at that point along the trajectory of the previous cycle, using a process called “feedforward.” In these embodiments, this state along the trajectory of the previous cycle, where the trajectory planning for the current cycle begins, is called the feedforward state. Feedforward can align the trajectory plan for the current cycle with the trajectory previously planned by the planning system. In some embodiments, in an autonomous vehicle system having an architecture including a trajectory planning system and a controller, the output trajectory from the planning system can be considered a “reference” for the controller to track, such that the controller can use feedback from vehicle state information to account for unmodeled disturbances and minimize tracking error with respect to a reference trajectory. In these embodiments, the controller can minimize the deviation between the planned trajectory and the measured state of the vehicle, while the trajectory planning system can generate a trajectory that gradually progresses toward a higher-level planning goal (e.g., staying centered in the lane, completing a turning maneuver, etc.).

[0076] In some embodiments, planning component 212 can use nonlinear optimization or machine learning models (e.g., neural networks) to find a trajectory that is (1) feasible (e.g., respecting constraints imposed by vehicle kinematics, such as maximum turning radius, or operating limits, such as minimum and maximum acceleration and jerk); (2) safe (e.g., respecting constraints imposed by road geometry by avoiding collisions with curbs, obeying lane markings, maintaining a safe distance from leading vehicles, and reacting appropriately to any objects, vehicles, people, or animals along the planned trajectory by avoiding or slowing down); and (3) comfortable for the end user (e.g., exhibiting characteristics similar to a human driving trajectory, such as preferring shorter, smoother routes requiring minimal steering or acceleration). In these embodiments, the mathematical formulation of this optimization involves expressing the trajectory as a function of several optimizable parameters. For example, to accurately model the different delays of the steering system and drive motor control system, state variables and control inputs along the trajectory can, in some embodiments, be separated into lateral and longitudinal components. In some embodiments, planning component 212 can generate a trajectory by utilizing two separate mathematical functions that describe the evolution along the path of (1) lateral control-related state variables and control inputs (position, heading, steering angle, steering angle rate, steering angle angular acceleration) and (2) longitudinal control-related state variables (time, velocity, acceleration, longitudinal jerk, longitudinal snap at the current distance along the path). A nonlinear optimization that refines an initial guess for the trajectory parameters to arrive at an optimized solution can be used to iteratively maximize the value of an objective function that scores the trajectory according to the criteria described above. Because the lateral and longitudinal components of the trajectory are separated into two separate functions, their initial portions can be accurately constrained in a way that models the estimated latencies of the lateral and longitudinal actuators.

[0077] Referring to FIG. 5B, an example of a data pipeline representing latency in issuing commands to a vehicle is described. In this example, as shown in FIG. 5B, the horizontal axis represents time, and therefore the width of each block can represent the latency corresponding to that block. In some embodiments, the planning component 212 can generate a trajectory in a "Planner Execution" block 532 and then send it to a "Controller Execution" block 534, which sends commands to "Steering Control & Actuation" and "Motor Control / Brake Actuation" blocks 536-540. In some embodiments, block 532 can send commands to blocks 536-540 without block 534 (e.g., component 532 can output commands via a neural network trained to control actuators). In these embodiments, because the actuator-specific blocks 536-540 represent different actuators, their actuation latencies to achieve desired control values ​​can differ, as represented by the different widths of these control blocks along the time axis. In some embodiments, in addition to these operational latencies, the planning component 212 can highlight algorithm execution times and phase delays between different components due to the scheduling of calculations, all of which may contribute to latency downstream of the start of planner execution.

[0078] Referring to FIG. 5C , an example of using estimated lateral and longitudinal control latencies in a trajectory representation is described. In some embodiments, planning component 212 can maintain estimates (as shown in FIG. 5B ) to anticipate latencies and generate a trajectory that more accurately represents how the future state of the vehicle will unfold. For example, in a current planning cycle, planning component 212 can recognize that the output trajectory of a previous planning cycle may still be in the process of being actuated to achieve a desired control value (e.g., the vehicle issuing a desired command). Planning component 212 can set an initial portion of the trajectory to match the portion from the previous cycle's trajectory output. In doing so, planning component 212 can model a process in which the current and near-term evolution of the vehicle's state may not be immediately affected by the output of planning component 212. Instead, planning component 212 can begin its decision-making and trajectory optimization from a point in time in the near future, estimating that these are the points in time when downstream processes and actuators can move the vehicle in response to the current planning component 212 output. In some embodiments, planning component 212 can begin its decision-making process from estimated future states at the end of the estimated latency period, such as beginning trajectory optimization at these estimated future states. In these embodiments, because the trajectory formulation can be divided into lateral and longitudinal components 552, 554, planning component 212 can also incorporate estimated lateral and longitudinal delay segments in a decoupled manner, as shown in FIG. 5C. The decoupling of lateral and longitudinal components 552, 554 according to the different latencies of different actuators can be utilized to achieve better accuracy of the trajectory optimization model while maintaining the responsiveness of lower latency actuator paths.

[0079] In some embodiments, a quantity of the trajectory optimization objective function that depends on both the lateral and longitudinal components of the trajectory (e.g., lateral acceleration, which depends on both the vehicle's speed and the path curvature) can be evaluated. In these embodiments, the trajectory optimization can consider possibly different lateral and longitudinal delay segment lengths to ensure that optimizable quantities along the two trajectory components are evaluated at aligned points along the trajectory evolution. In some embodiments, delay segments in the initial portions of the lateral and longitudinal trajectory components, which were modeled as fixed based on the trajectory of the previous cycle, may not be optimizable by the trajectory optimization. While the description of FIG. 5C focused on the latency associated with the lateral and longitudinal components, in some embodiments, the planning component 212 can estimate the overall system latency using one of the components. As an example, the component 212 may output a control signal that combines lateral and longitudinal control (e.g., the component 212 may represent a machine learning model such as a neural network). Thus, in this example, the component 212 may determine the latency associated with this overall control. Optionally, the planning component 212 may estimate the latency of the entire system based on selecting the highest, lowest, or central tendency measure of the components 552, 554.

[0080] Referring to FIG. 5D , an example of a journey by modeling an optimized trajectory with accounting system delays is described. In some embodiments, the accuracy of the system model in both the lateral and longitudinal components 552, 554 (shown in FIG. 5C ) can result in commanding the vehicle for optimal safety and comfort. As shown in FIG. 5D , without accounting for estimated feedforward system delays, trajectory optimization 562 may be less accurate because it may plan maneuvers such as sudden decelerations that it may attempt to complete with less deceleration and jerk than would be required to stop within a desired distance along the path. As shown in FIG. 5D , planning component 212, which accounts for its estimated longitudinal actuation delays, can recognize that it may need to use more severe deceleration and jerk to achieve the desired behavior, and more accurate modeling of system delays allows the controller to plan a trajectory 564 that tracks with less tracking error. In this example, trajectory 566 represents the absence of sudden decelerations. In some embodiments, using a longitudinal delay segment that is separate from a lateral delay segment can avoid the possibility of introducing artificial latency into the system response, for example, when lateral control latency is longer than longitudinal control latency. In addition to improving vehicle safety by more accurately modeling the system response in planning component 212, this modeling of system latency can improve ride comfort; with actuator outputs that may generally more closely match what planning component 212 desires and modeled via its generated trajectory, planning component 212 can control ride smoothness and generally expect the actuators to more accurately follow the desired trajectory.

[0081] The foregoing disclosure is not intended to limit the disclosure to the precise form or particular field of use disclosed. Accordingly, various alternative embodiments and / or modifications to the disclosure, whether expressly described or implied herein, are contemplated in light of the present disclosure. While embodiments of the present disclosure have been described in this manner, those skilled in the art will recognize that changes can be made in form and detail without departing from the scope of the present disclosure. Accordingly, the present disclosure is limited only by the scope of the claims.

[0082] In the foregoing specification, the present disclosure has been described with reference to specific embodiments. However, as will be understood by those skilled in the art, the various embodiments disclosed herein can be modified or implemented in a variety of other ways without departing from the spirit and scope of the present disclosure. Accordingly, the present description is to be considered illustrative and is intended to teach those skilled in the art how to make and use various embodiments of the disclosed decision and control processes. It should be understood that the forms of the disclosure shown and described herein are to be taken as representative embodiments. Equivalent elements, materials, processes, or steps may be substituted for those typically shown and described herein. Furthermore, certain features of the present disclosure may be utilized independently of the use of other features, all of which will be apparent to those skilled in the art after having the benefit of this description of the present disclosure. The terms "including," "comprising," "incorporating," "consisting of," "have," "is," and the like, used to describe and claim the present disclosure, are intended to be construed in a non-exclusive manner, i.e., permitting the presence of items, components, or elements not expressly recited. Reference to the singular is to be construed as relating to the plural as well.

[0083] Furthermore, the various embodiments disclosed herein should be taken in an illustrative and descriptive sense and should not be construed as limiting the present disclosure in any way. All joining references (e.g., attached, fastened, coupled, connected, etc.) are used solely to aid the reader's understanding of the present disclosure and do not create limitations on the position, orientation, or use of the systems and / or methods disclosed herein, among other things. Accordingly, any joining references should be interpreted broadly. Furthermore, such joining references do not necessarily infer that two elements are directly connected to each other.

[0084] Additionally, all numerical terms, such as, but not limited to, "first," "second," "third," "primary," "secondary," "main," "principal," or any other conventional and / or numerical term, should also be construed merely as identifiers to aid the reader's understanding of the various elements, embodiments, variations and / or modifications of the present disclosure, and cannot impose any limitation on or beyond another element, embodiment, variation and / or modification, particularly with respect to order or priority, in any element, embodiment, variation and / or modification.

[0085] It will also be understood that one or more of the elements shown in the drawings / diagrams may be implemented in a more separated or integrated manner, as may be useful depending on the particular application, or may even be excluded or presented as inoperative in certain cases.

Claims

1. 1. A system for planning a trajectory of a vehicle for autonomous driving, the system including one or more processors and a non-transitory computer storage medium having stored thereon instructions, the instructions, when executed by the one or more processors, causing the one or more processors to, during a cycle: acquiring vehicle data from a set of sensors; identifying vehicle trajectory information and vehicle status by processing the acquired vehicle data; identifying a vehicle control command based on the identified vehicle trajectory information and the vehicle state, the vehicle state being related to operation of at least one actuator; determining the vehicle control commands for each actuator; characterizing a latency for each actuator, the latency being an estimated time delay between acquiring the vehicle data and controlling the corresponding actuator; optimizing the vehicle control commands based on a characterized latency of each actuator based on the trajectory information; causing control of the vehicle based on the optimized vehicle control command; Causes the system to perform operations, including

2. The system of claim 1 , wherein the cycle corresponds to a vehicle's travel trajectory, and a new cycle begins when the vehicle enters a new trajectory.

3. The system of claim 1 , wherein the trajectory information indicates a start of a trajectory, and a vehicle state at the start of the trajectory matches a vehicle state controlled by a vehicle control command optimized in a previous cycle.

4. The system of claim 1 , wherein the operation further comprises identifying an optimal trajectory path before optimizing the vehicle control commands, and the vehicle control commands are generated based on the optimal trajectory path.

5. The system of claim 1 , wherein the actuator includes a lateral component and a longitudinal component, the lateral component being associated with operation of a steering actuator and the longitudinal component being associated with operation of a motor actuator.

6. 6. The system of claim 5, wherein the estimated latency corresponds to the longitudinal component being lower than the lateral component, and a system delay in optimizing the vehicle control command over a period of the estimated latency corresponds to the lateral component.

7. 2. The system of claim 1, wherein the vehicle trajectory information is identified by identifying geographic coordinates and collecting vision system data, the vision system data including ground truth labels of video images captured by a camera mounted on the vehicle.

8. The system of claim 1 , wherein the vehicle conditions include current vehicle position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration.

9. The system of claim 1 , wherein each of the acquired vehicle data includes a timestamp representing the time at which each of the vehicle data was measured.

10. The system of claim 9 , wherein the latency is estimated by determining the time difference between the timestamp of each vehicle data and a particular time of a corresponding actuator operation.

11. The system of claim 1 , wherein the latency includes a measurement latency, a computation latency, a communication latency, and a hardware latency.

12. The system of claim 1 , wherein the system is configured to determine the trajectory information based on the identified vehicle state.

13. The system of claim 1 , wherein the optimization is based on predetermined criteria, the criteria including vehicle kinematics, safety constraints, and comfort constraints.

14. 1. A system for planning a vehicle trajectory, comprising: a sensor suite configured to collect vehicle data from vehicle sensors implemented on the vehicle; a perception system configured to generate a vehicle state by receiving and processing the collected vehicle data from the sensor suite; 1. A planning system comprising: identifying a vehicle control command for the generated vehicle state; determining a latency for controlling each actuator of the vehicle; a planning system configured to optimize a trajectory of the vehicle based on the determined latency; Including, the system.

15. The system of claim 14 , wherein the vehicle state includes vehicle trajectory information, and the planning system optimizes the vehicle trajectory information based on the determined latency of each actuator.

16. The system of claim 14 , wherein the optimization is based on predetermined criteria, the criteria including vehicle kinematics, safety constraints, and comfort constraints.

17. 15. The system of claim 14, wherein the system plans a vehicle trajectory for each cycle, and a new cycle begins when the vehicle enters a new trajectory.

18. 15. The system of claim 14, wherein the vehicle includes two actuations corresponding to a lateral component and a longitudinal component of the vehicle, the lateral component being associated with operation of a steering actuator and the longitudinal component being associated with operation of a motor actuator.

19. 1. A method for planning a trajectory, comprising: acquiring vehicle data from a set of sensors; processing the acquired vehicle data to identify vehicle trajectory information and vehicle status; identifying a vehicle control command based on the identified vehicle trajectory information and the vehicle state, the state relating to operation of at least one actuator; determining the vehicle control commands for each actuator; characterizing a latency for each actuator, the latency being an estimated time delay between acquiring the vehicle data and controlling the corresponding actuator; optimizing the trajectory information based on a characterized latency of each actuator; triggering control of the vehicle based on the optimized trajectory; and A method comprising:

20. 20. The method of claim 19, wherein controlling the vehicle is based on identifying vehicle control commands for driving the vehicle using the optimized trajectory information.

21. The method of claim 19 , wherein the optimization is based on predetermined criteria, the criteria including vehicle kinematics, safety constraints, and comfort constraints.

Images