Real-time trajectory planning system with dynamic modeling of component-level system delays for autonomous vehicles

Abstract

Systems and methods for autonomous driving compute a path trajectory by combining delays and calculations of vehicle components such that when the vehicle is in an autonomous driving mode and using the trajectory to compute a specified path, the delays of each vehicle component may be considered to provide a more accurate vehicle path.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 373,039, filed on August 19, 2022, entitled “TRAJECTORY PLANNER REAL TIMETRAJECTORY PLANNING SYSTEM WITH DYNAMIC MODELLING OF COMPONENT LEVEL SYSTEM LATENCY FOR SELF DRIVING VEHICLES,” the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure relate to systems and methods for providing optimized trajectories for a driving trajectory planner system. More specifically, embodiments of the present disclosure relate to systems and methods for optimizing driving paths, such as driving trajectories as part of an autonomous driving system. Background Art

[0004] Generally speaking, computing devices and communication networks can be used to exchange data and / or information. In common applications, a computing device can request content from another computing device via a communication network. For example, a computing device can collect various data and use software applications to exchange content with a server computing device via a network (e.g., the Internet).

[0005] Various vehicles, such as electric vehicles, internal combustion engine vehicles, hybrid vehicles, etc., can be configured with various sensors and components to facilitate the operation of the vehicle or the management of one or more systems included in the vehicle, such as an autonomous driving system (e.g., an automatic driving system). In some scenarios, a vehicle owner or vehicle user may wish to utilize a sensor-based system to facilitate the operation of the vehicle's autonomous driving system. For example, a vehicle may typically include hardware and software functionality that facilitates autonomous driving by utilizing location services or accessing a computing device that provides location services. In another example, a vehicle may also include a navigation system or access a navigation component that can generate information related to navigation or direction information provided to vehicle occupants and users. In another example, a vehicle may include a visual system to facilitate autonomous driving by utilizing navigation and location services, safety services, or other operational services / components. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present disclosure is described herein with reference to the accompanying drawings of certain embodiments, which are intended to illustrate the present disclosure but not to limit the present disclosure. It should be understood that the accompanying drawings are incorporated into and constitute a part of this specification, are used to illustrate the concepts disclosed herein, and may not be drawn to scale.

[0007] Figure 1 A block diagram depicting an illustrative environment for a vision system in a vehicle according to one or more aspects of the present application;

[0008] Figure 2A depicts a block diagram of an illustrative environment corresponding to a vehicle according to one or more aspects of the present application;

[0009] Figure 2B depicts an illustrative vision system for a vehicle in accordance with one or more aspects of the present application;

[0010] Figure 3 depicts a block diagram of an illustrative architecture for implementing a planning component in accordance with aspects of the present application;

[0011] Figure 4A , Figure 4B and Figure 4C yes Figure 1 A block diagram of an illustrative environment for generating simulation model content and subsequently generating a vision system training data set for a machine learning process based on the simulation model content;

[0012] Figure 5A is a block diagram of illustrative interactions for controlling a vehicle;

[0013] Figure 5B is an example of a data pipeline representing the delay in commanding a vehicle;

[0014] Figure 5C is an illustrative example of using estimated delays for lateral and longitudinal control in trajectory representation; and

[0015] Figure 5D is a line graph showing an illustrative example of driving by modeling an optimized trajectory with and without considering system delays. DETAILED DESCRIPTION

[0016] Although certain preferred embodiments and examples are disclosed below, the subject matter of the present invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses and modifications and equivalents thereof. Therefore, the scope of the appended claims is not limited to any specific embodiment described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable order and are not necessarily limited to any specific disclosed order. Various operations may be described as multiple discrete operations in sequence in a manner that may help understand certain embodiments; however, the order of description should not be interpreted as implying that these operations are sequence-dependent. In addition, the structures, systems and / or devices described herein may be embodied as integrated components or separate components. In order to compare various embodiments, certain aspects and advantages of these embodiments are described. Not all such aspects or advantages are necessarily implemented by any particular embodiment. Therefore, for example, various embodiments may be performed in a manner that implements or optimizes an advantage or group of advantages taught herein without having to implement other aspects or advantages taught or suggested herein.

[0017] The software as well as the hardware components of an autonomous vehicle have some inherent delays from the first timestamp of the sensor measurement to the evaluation of the final actuation (steering, gas pedal, brake) due to the time taken for computation, communication across different modules, hardware actuation, etc. In order to provide a system that can autonomously or semi-autonomously drive an autonomous vehicle like a human and make accurate decisions in safety-critical scenarios, the planning and decision-making system should take the delay of each such component into account in the temporal decision-making process. Therefore, the system described in this article is configured to accurately simulate the delay of the final actuation (such as steering, braking or acceleration) independently and accurately. In addition, the design should be efficient enough to avoid spending unnecessary milliseconds performing calculations to determine a new decision; otherwise, it will make the system latency and reaction time worse than if the unnecessary calculations were not performed.

[0018] Embodiments of the present invention are directed to a trajectory planning system that can run in real-time on a safety-critical system, such as an autonomous vehicle, and that is capable of accurately modeling vehicle behavior while taking into account delays in individual actuator components.

[0019] In short, some embodiments of the disclosed system may be embedded in an autonomous vehicle and may perform one or more of the following processes to control operating parameters for autonomous driving of the vehicle:

[0020] 1. Sensor measurements from a specific sensor suite are processed by a perception system to generate information about surrounding information (e.g., world information) around the vehicle. The surrounding information may include, but is not limited to, surrounding object information, road type information, driving direction information, etc.

[0021] 2. The planning (e.g., decision making) system can use the output of the perception system to derive several semantic decisions that the autonomous vehicle should follow. The planning system can generate a high-fidelity trajectory plan (position, velocity, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk, lateral jerk) based on the derived semantic decisions.

[0022] 3. Controls embedded in the vehicle system can utilize trajectory planning and generate autonomous driving commands, such as speed and steering commands that can be used to control the vehicle.

[0023] 4. The generated steering command can be utilized by a vehicle steering command unit, such as an electronic power steering ECU, which acts as a motor controller for steering the motor to steer the direction of the vehicle.

[0024] 5. Speed, acceleration and jerk commands are sent to the vehicle motor controller to convert the commands into final torque for the motor or brake pressure for the vehicle braking system to control the speed, position and direction of the vehicle.

[0025] In some embodiments, a vehicle planning and control system may perform one or more of these processes. Using this approach, a planning and control system for an autonomous vehicle receives a representation of the world around it measured by a sensor suite (e.g., a set of sensors implemented in a vehicle) and processes it with a perception system. The purpose of the planning system is to make decisions and plan trajectories to achieve certain goals of the autonomous vehicle, such as making decisions along a desired navigation route while maintaining safe and comfortable driving. The trajectory generated by the planning system is sent to a controller, which calculates the speed and steering commands of the actuators to accurately track the output trajectory of the planning system. The vehicle planning and control system may generally be referred to as a planning system, and the planning system may be implemented as a planning component of a vehicle.

[0026] Embodiments of the present application correspond to systems and methods for providing a driving trajectory planning system in a vehicle by optimizing the trajectory of an autonomous vehicle while taking into account the latency required to process data within the system when the vehicle is in autonomous driving mode. Optimization or optimization as referred to herein may include classical computing techniques as well as techniques based on machine learning (e.g., neural networks). Trajectory may generally be referred to as the expected driving trajectory of a vehicle. More specifically, the system and method are used to optimize the trajectory in an autonomous driving system before entering the trajectory. Trajectory information may be determined by utilizing navigation services (such as global positioning systems) and based on data received from cameras and other sensors installed in the vehicle. For example, a user (e.g., a driver) may enter a destination location on a vehicle navigation system, and then an autonomous driving system installed in the vehicle may determine one or more driving paths to the destination. In this example, the autonomous system of the vehicle may determine a path to the destination. When the autonomous vehicle is heading to the destination, the vehicle will continuously calculate the best trajectory based on road conditions, other vehicles, traffic signals, pedestrians, etc., to safely guide the vehicle to the destination. For example, the trajectory may be calculated every few milliseconds so that the autonomous vehicle may directly change its path or speed to take into account the current road conditions.

[0027] Traditionally, the autonomous driving system of a vehicle is associated with physical sensors that provide input to the autonomous system. For example, the physical sensors may include imaging systems and hardware processors, as well as memory for supporting these imaging systems. In addition, the autonomous system may include radar systems, LIDAR systems, etc., which are capable of detecting the distance to an object and characterizing the properties of the detected object. These components may have inherent delays when evaluating the data generated from these sensors for autonomous driving (e.g., autonomous driving) of the vehicle. Each of these hardware components may also have different delays, which reflect the time for the component to manage its data and send the data to the system to calculate the current trajectory of the vehicle. In addition, the system itself also has delays when calculating the correct path of the vehicle when the vehicle is driving autonomously. These delays may bring challenges to determining the driving path or optimal trajectory, because each of the sensors, navigation systems, communication paths (e.g., communication buses), or processors may act based on data captured at slightly different times when the vehicle is driving. In addition, the system may calculate the route based on outdated information because the vehicle or the surrounding environment may have changed within a few milliseconds when the system is calculating the preferred driving path. For example, when other vehicles around are changing their driving paths, the autonomous system may not be able to simultaneously determine the optimal trajectory in the changing environment due to the delay in detecting and calculating the data detecting the changes in the driving paths of other vehicles.

[0028] In one application, these delays are problematic when determining a vehicle path, such as a vehicle trajectory. For example, based on processed data received from various sensors, the vehicle's autonomous system may plan a driving trajectory before the vehicle travels on the calculated trajectory. In this example, when trajectory planning is complete and control commands are actuated to move the vehicle to a desired state (e.g., speed and steering angle), due to the delay, the vehicle may have moved to a state different from the initial state at which the planning system began its planning. Delays can generally be referred to as time delays caused by a portion of the data used as input data for the planning component of the vehicle. For example, the delay of a portion of the data may be caused by an operational delay of a sensor, a communication delay, a sensor data processing delay, and the like. For example, if the planning system generates a desired trajectory that requires vehicle acceleration, the vehicle may not be able to achieve the acceleration until the downstream system processes and responds to the request. Subsequently, the vehicle's sensor system may not be able to measure the vehicle acceleration corresponding to the acceleration of the desired trajectory until the downstream system processes and executes the request. If the delay between the desired vehicle behaviors encoded in the trajectory of the planning system is greater than the duration and time criticality of the maneuver that the planning system wants to achieve, inaccurate decision making and trajectory planning may result.

[0029] To address at least a portion of the above-mentioned deficiencies, aspects of the present application are directed to using a planning system that is trained to account for the delays of various components within a vehicle. If each vehicle uses a planning system that is configured to incorporate the delays of various components when it generates path information, then the system can correctly calculate the trajectory of the vehicle as it travels along its designated path to its destination.

[0030] General aspects of the present application involve determining (or estimating) the delay of each hardware component and software model implemented in the system, and using the determined delays in the decision making logic. In these aspects, the system can utilize various neural network-based planning systems.

[0031] In some embodiments, the system is configured to determine the delay corresponding to each hardware component and software model utilized in the autonomous driving of the vehicle. For example, if the vehicle utilizes front, side, and rear cameras in autonomous driving, the system determines each delay corresponding to each of these cameras. In this example, the system can also determine any computational delay caused by the software or system that processes the data received from these cameras for autonomous driving. In some embodiments, these delays are utilized in the time decision-making process of autonomous driving. In these embodiments, based on these delays, the system determines the accurate time sequence of the data received or processed from each hardware component or software model. For example, if the front camera, the side camera, and the rear camera have delays of 1ms, 2ms, and 3ms, respectively, the images captured by the front camera, the side camera, and the rear camera at 1ms, 2ms, and 3ms before the reference time can be utilized as images at the reference time, wherein these images are used in autonomous driving. These hardware components and software models are provided only as examples, and various hardware component configurations and software models can be used based on applications.

[0032] In some embodiments, the system provides real-time determination of an optimal trajectory while operating a vehicle, such as in an autonomous driving environment. In these embodiments, the system can provide real-time calculation of the optimal trajectory while taking into account the latency of each component in combining one or more data received from a navigation system and / or sensors installed in the vehicle. For example, a deep neural network (DNN) can be trained as part of an autonomous system and can output component latency.

[0033] In some embodiments, a trained planning system or model can be used to provide an optimal trajectory. The optimal trajectory can be referred to as the optimal driving path and its associated vehicle operating parameters that can provide comfortable, safe and efficient driving, such as position, speed, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk, lateral jerk. The planning model implemented in the planning component of the vehicle can determine the optimal trajectory path by utilizing the data of one or more sensors installed in the vehicle and by utilizing a navigation system or service. In some embodiments, the planning model can simulate multiple trajectories and determine the optimal trajectory of the operating parameters of the vehicle. In these embodiments, the determination is based on determining the operating parameters of the vehicle corresponding to each simulated trajectory. For example, the planning model can generate a vectorized model for each simulation result and determine the vehicle operating parameters for each vectorized model. Then, the planning model can determine the optimal trajectory that can provide comfortable, safe and efficient driving. In some embodiments, the planning model can predict any possible delays caused by sensors and computing processors. In these embodiments, based on the predicted delays, the planning model can allocate computing resources to take into account these delays.

[0034] In some embodiments, the autonomous system installed in the vehicle relies solely on the vision system to determine the optimal trajectory. Illustratively, a vision-only system is contrasted with a vehicle that can combine a vision-based system with one or more additional sensor systems (such as a radar-based system, a lidar-based system, a sonar system, etc.). In some embodiments, the vision-only system can be configured with a machine learning process that can only process inputs from the vision system, including multiple cameras installed on the vehicle. The machine learning process can generate outputs that identify objects and specify the characteristics / attributes of the identified objects, such as the position, velocity, and acceleration measured relative to the vehicle. The output from the machine learning process can then be used for further processing, such as for navigation systems, positioning systems, safety systems, etc.

[0035] According to aspects of the present application, a network service can configure a machine learning process according to a supervised learning model, wherein the machine learning process is trained with labeled data, including identified objects and specified characteristics / attributes, such as position, velocity, acceleration, etc. A first portion of the training data set corresponds to data collected from a target vehicle including a vision system, such as a vision system included in a vision-only system in a vehicle. Additionally, a second portion of the training data corresponds to additional information obtained from other systems, i.e., a simulated content system that can generate video images and associated attribute information.

[0036] Illustratively, the network service may receive a combined set of inputs (e.g., a first data set and a second data set) from a target vehicle, including visual and simulated content. The network service then combines the data based on a standardized data format. Illustratively, the combined data set allows previously collected visual data to be supplemented with additional information or attributes / characteristics that may not be available by processing the visual data. The combined data set may produce a data set that tracks an object within a defined time range based on the first data set and the second data set. The network service may then process the combined data set using a variety of techniques. These techniques may include smoothing, extrapolation of missing information, applying a dynamic model, applying a confidence value, and the like. Thereafter, the network service generates an updated machine learning process based on training the combined data set. The trained machine learning process may be transmitted to a target vehicle based on a vision-only vehicle or a vehicle having both vision and detection systems to repeat the process and further update / refine the machine learning process.

[0037] Although various aspects will be described according to illustrative embodiments and feature combinations, those skilled in the relevant art will appreciate that the examples and feature combinations are illustrative in nature and should not be construed as limiting. More specifically, various aspects of the present application can be applicable to various types of vehicles, including vehicles with different propulsion systems, such as combination engines, hybrid engines, electric engines, etc. In addition, various aspects of the present application can be applicable to various types of vehicles that can incorporate different types of sensors, sensing systems, navigation systems, or positioning systems. Therefore, the illustrative examples should not be construed as limiting. Similarly, various aspects of the present application can be combined with or implemented in conjunction with other types of components that can facilitate the operation of the vehicle, including autonomous driving applications, driver convenience applications, etc.

[0038] Figure 1 A block diagram of an illustrative environment 100 for generating a simulation content model and training set data for a visual system in a vehicle according to one or more aspects of the present application is depicted. The system 100 may include a network that connects a first group of vehicles 102, a network service 110, and a simulation content system 120. Illustratively, various aspects associated with the network service 110 and the simulation content system 120 may be implemented as one or more components associated with one or more functions or services. These components may correspond to software modules implemented or executed by one or more external computing devices, which may be separate, independent external computing devices. Therefore, the components of the network service 110 and the simulation content system 120 should be viewed as logical representations of the services without requiring any specific implementation on one or more external computing devices.

[0039] like Figure 1 As depicted in , the network 106 connects the devices and modules of the system. The network can connect any number of devices. In some embodiments, a network service provider provides network-based services to client devices via a network. A network service provider implements network-based services and refers to a large shared pool of network-accessible computing resources (such as computing, storage or networking resources, applications or services), which can be virtualized or bare metal. A network service provider can provide on-demand network access to a shared pool of configurable computing resources, which can be programmatically prepared and released in response to customer commands. These resources can be dynamically prepared and reconfigured to adapt to variable loads. Therefore, the concept of "cloud computing" or "network-based computing" can be viewed as an application delivered as a service over a network and the hardware and software in the network service provider that provides these services. In some embodiments, the network can be a content delivery network.

[0040] Illustratively, the vehicle group 102 corresponds to one or more vehicles configured with a vision-only based system for identifying objects and characterizing one or more attributes of the identified objects. The vehicle group 102 is configured with a machine learning process, such as a machine learning process implemented in a supervised learning model, configured to identify objects and characterize attributes of the identified objects, such as position, velocity, and acceleration attributes, using only vision system inputs. The vehicle group 102 may be configured without any additional detection systems, such as a radar detection system, a LIDAR detection system, etc.

[0041] Illustratively, the network services 110 may include a plurality of network-based services that may provide functionality responsive to configuration / requests of machine learning processes for vision-only based systems, as applied to aspects of the present application. Figure 1 As illustrated in FIG. 1 , the web-based service 110 may include a planning component 212 that may obtain data sets from the vehicle 102 and the simulation content system 120, process the data sets to form training material for the machine learning process, and generate a machine learning process for the vision-only based vehicle 102. The web-based service may include a plurality of data repositories for maintaining various information associated with various aspects of the present application, including a vehicle data repository 114 and a machine learning process data repository 116. Figure 1 The data repositories in are logical in nature and can be implemented in network service 110 in a variety of ways.

[0042] Similar to the network service 110, the simulation content service 120 may include a plurality of network-based services that may provide functionality related to providing data visual frames and associated data labels for machine learning applications, such as those used in various aspects of the present application. Figure 1 As shown in FIG. 1 , the network-based service 120 may include a scenario generation component 122 that can create various scenarios based on a set of defined attributes / variables. The simulation content service 120 may include multiple data repositories for maintaining various information associated with various aspects of the present application, including a scenario clip data repository 124 and a ground truth attribute data repository 126. Figure 1 The data repository in is logical in nature and can be implemented in a variety of ways in the simulation content service.

[0043] For illustration purposes, Figure 2AAn environment corresponding to a vehicle in the vehicle group 102 is illustrated according to one or more aspects of the present application. The environment includes a collection of local sensors 225 that can provide input or information collection as described herein for the operation of the vehicle. The collection of local sensors 225 may include one or more sensors or sensor-based systems that are included in the vehicle or otherwise accessible by the vehicle during operation. The local sensors 225 or sensor systems can be integrated into the vehicle. Alternatively, the local sensors or sensor systems can be provided by an interface associated with the vehicle, such as a physical connection, a wireless connection, or a combination thereof.

[0044] In one aspect, local sensors 225 may include a vision system that provides input to the vehicle, such as detection of objects, properties of detected objects (e.g., position, velocity, acceleration), presence of environmental conditions (e.g., snow, rain, ice, fog, smoke, etc.), etc. Figure 2B An illustrative collection of cameras mounted on a vehicle to form a vision system is described. As previously described, the vehicle 102 will rely on such a vision system to achieve defined vehicle operating functions without assistance from or in lieu of other conventional detection systems.

[0045] On the other hand, the local sensor 225 may include one or more positioning systems that can obtain reference information from external sources, thereby allowing various levels of accuracy in determining the positioning information of the vehicle. For example, the positioning system may include various hardware and software components for processing information from a GPS source, a wireless local area network (WLAN) access point information source, a Bluetooth information source, a radio frequency identification (RFID) source, etc. In some embodiments, the positioning system may obtain a combination of information from multiple sources. Illustratively, the positioning system may obtain information from various input sources and determine the positioning information of the vehicle, particularly at the altitude of the current location. In other embodiments, the positioning system may also determine operating parameters related to driving, such as the direction, speed, acceleration, etc. of driving. The positioning system may be configured as part of a vehicle for a variety of purposes, including autonomous driving applications, enhanced driving, or user-assisted navigation, etc. Illustratively, the positioning system may include a planning component 212 linked to data 230 that facilitates identification of various vehicle parameters or process information.

[0046] In another aspect, local sensors 225 may include one or more navigation systems 235 for identifying navigation-related information. Illustratively, the navigation system may obtain positioning information from a positioning system and identify characteristics or information about the identified position data (e.g., received from a global positioning system (GPS) 245), such as altitude, road grade, etc. The navigation system 235 may also identify suggested or expected lane positions in a multi-lane road based on directions provided or expected for a vehicle user. Similar to the positioning system, the navigation system may be configured as part of the vehicle for a variety of purposes, including autonomous driving applications, enhanced driving, or user-assisted navigation, etc. The navigation system may be combined or integrated with a planning component 212 that is part of the positioning system.

[0047] The local resources may include one or more planning components 212, which may be hosted on the vehicle or a computing device accessible to the vehicle (e.g., a mobile computing device). The planning component 212 may illustratively access input from a local sensor 225 or sensor system and process the input data, as described herein. For purposes of this application, the planning component 212 will be described with respect to one or more functions related to the illustrative aspects. For example, the planning component 212 in the vehicle 102 will collect and transmit a first data set corresponding to the collected visual information.

[0048] The environment may also include various additional sensor components or sensing systems operable to provide information about various operating parameters for use according to one or more of the operating states. The environment may also include one or more control components 228 for processing outputs, such as transmitting data via communication outputs 240, generating data in memory, transmitting outputs to other planning components, etc.

[0049] According to one or more aspects described herein, the planning component 212 may be configured to provide vehicle control instructions (e.g., steering, acceleration, pedals, brakes, speed, etc.) by processing sensor data and estimating inherent delays (such as data calculation delays, internal data communication delays, and hardware (e.g., steering actuators, motors, etc.) actuation delays). In some embodiments, the planning component 212 may include a perception system 212. The perception system 212 may be configured to receive sensor data from local sensors 225, navigation 235, and GPS 245. After receiving the sensor data, the perception system 212 may process the received data to determine the state of the vehicle, such as position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration. The planning component 212 may use the determined vehicle state to estimate delays and generate vehicle operating parameter control instructions. For example, the planning component 212 may classify the vehicle state into a lateral state and a longitudinal state. The lateral state may refer to vehicle operating variables related to vehicle position, heading, steering angle, and steering angle rate. The longitudinal state may refer to vehicle operating variables related to vehicle speed, acceleration, longitudinal jerk, and longitudinal snap. The lateral state can be associated with controlling the vehicle steering actuator, while the longitudinal state can be associated with controlling the vehicle motor actuator. The planning component 212 can determine the delays used to control the individual steering and motor actuators. For example, the planning component 212 can determine the time delay between measuring the vehicle state and controlling the individual steering and motor actuators. Example delays can include time delays in receiving input data from local sensors (e.g., measurement delays), time delays in processing local sensor data (e.g., calculation delays), time delays in transmitting and receiving data between components (e.g., communication delays), and time delays in operating individual steering and motor actuators (e.g., hardware delays). The planning component 212 can accurately plan vehicle operations and optimize vehicle trajectories based on the determined delays.

[0050] Reference now Figure 2B , an illustrative vision system 200 for a vehicle will be described. The vision system 200 includes a set of cameras that can capture image data during vehicle operation. As described above, individual image information can be received at a specific frequency so that the illustrated image represents a specific timestamp of the image. In some embodiments, the image information can represent a high dynamic range (HDR) image. For example, different exposures can be combined to form an HDR image. As another example, images from the image sensor can be pre-processed to convert them into HDR images (e.g., using a planning model).

[0051] like Figure 2B As shown in FIG. 1 , the set of cameras may include a set of forward-facing cameras 202 that capture image data. The forward-facing cameras may be mounted in the windshield area of ​​the vehicle to have a slightly higher height. Figure 2B As illustrated in , the forward camera 202 may include multiple individual cameras configured to generate a composite image. For example, the camera housing may include three image sensors pointing forward. In this example, the first image sensor may have a wide-angle (e.g., fisheye) lens. The second image sensor may have a normal or standard lens (e.g., 35mm equivalent focal length, 50mm equivalent focal length, etc.). The third image sensor may have a zoom or narrow lens. In this way, the vehicle can obtain images of three different focal lengths in the forward direction. The visual system 200 also includes a group of cameras 204 mounted on the door pillars of the vehicle. The visual system 200 may also include two cameras 206 mounted on the front bumper of the vehicle. In addition, the visual system 200 may also include a rear-facing camera 208 mounted on the rear bumper, trunk, or license plate holder.

[0052] Camera groups 202, 204, 206, and 208 can all provide captured images to one or more planning components 212, such as a dedicated controller / embedded system. For example, planning component 212 may include one or more matrix processors configured to quickly process information associated with a planning model. In some embodiments, planning component 212 may be used to perform convolutions associated with forwarding passes through a convolutional neural network. For example, input data and weight data may be convolved. Planning component 212 may include several multiplication-accumulation units that perform convolutions. For example, a matrix processor may use weight data and inputs that have been organized or formatted to facilitate larger convolution operations. Alternatively, image data may be transmitted to a general planning component. Planning component 212 may be referred to as an autonomous driving planning component.

[0053] Illustratively, each camera may be individually operated or treated as a separate visual data input for processing.In other embodiments, one or more camera data subsets may be combined to form composite image data, such as three front-facing cameras 202 .

[0054] In some embodiments, the planning component 212 can process the captured images to determine a driving path, wherein the vehicle 200 is set to automatically drive to a destination specified by a user (e.g., a driver) using a navigation system. In these embodiments, the planning component can simulate multiple driving paths of a certain distance (e.g., within a certain number of time frames) and determine the optimized driving path in real time. For example, if the vehicle 200 needs to change lanes, the planning component 212 simulates multiple driving paths for changing lanes based on traffic lanes and surrounding objects (such as surrounding vehicles or any objects). Each simulation result can be vectorized based on vehicle operating parameters (such as acceleration, speed, and rate). Then, the planning component 212 can determine the best driving path for changing lanes by utilizing the planning system. The planning system can be configured to analyze the vectorization results to determine the best driving path when changing lanes.

[0055] In some embodiments, when the vehicle 200 needs to move along a trajectory direction during its autonomous driving, the planning component 212 can determine the optimal trajectory. In these embodiments, when the vehicle 200 needs to move along a trajectory (such as turning in a specific direction), the planning component 212 can determine the optimal trajectory that can provide the driver with a comfortable, safe and efficient driving experience. For example, when the vehicle 200 needs to turn left, the planning component 212 can simulate multiple trajectories to complete the turn. Each simulated trajectory can be vectored based on vehicle operating parameters such as acceleration, speed, and rate before the vehicle turns to the specified direction. The planning component 212 can then use the planning model to determine the optimal trajectory based on driving comfort, safety, and efficiency, and the vehicle uses the optimal trajectory during its autonomous driving.

[0056] Reference now Figure 3 , an illustrative architecture for implementing a planning component 212 on one or more local resources or network services will be described. The planning component 212 can be part of a component / system capable of providing functionality associated with autonomous driving.

[0057] Figure 3 The architecture is illustrative in nature and should not be construed as requiring any particular hardware or software configuration for the planning component 212. Figure 3 The general architecture of the planning component 212 depicted in includes an arrangement of computer hardware and software components that can be used to implement various aspects of the present disclosure. As illustrated, the planning component 212 includes a processing unit 302, a network interface 304, a computer readable medium driver 306, and an input / output device interface 308, all of which can communicate with each other via a communication bus. The components of the planning component 212 can be physical hardware components or implemented in a virtualized environment.

[0058] Network interface 304 may provide connectivity to one or more networks or computing systems, such as Figure 1 The processing unit 302 can thus receive information and instructions from other computing systems or services via the network. The processing unit 302 can also communicate with the memory 210 and further provide output information for autonomous driving via the input / output device interface. In some embodiments, the planning component 212 may include Figure 3 More (or fewer) components than those shown in .

[0059] Memory 310 may include computer program instructions that processing unit 302 executes to implement one or more embodiments. Memory 310 typically includes RAM, ROM, or other persistent or non-transitory memory. Memory 310 may store an operating system 312 that provides computer program instructions for use by processing unit 302 in general management and operation of planning component 212. Memory 310 may also include computer program instructions and other information for implementing various aspects of the present disclosure. For example, in one embodiment, memory 310 includes a sensor interface component 316 that obtains information from a vehicle (such as vehicle 102), a data repository, other services, etc.

[0060] The memory 310 also includes a visual information planning component 316 for obtaining and processing one or more data sets including simulated content according to various operating states of the vehicle as described herein.

[0061] The memory 310 may also include a planning model 318 for generating or training a machine learning process for use in the autonomous driving of the vehicle 102. In some embodiments, the planning model 318 uses data from the vehicle system to determine the optimal driving path and the optimal operating parameters of the vehicle that can provide comfortable, safe and efficient autonomous driving. In these embodiments, the planning model 318 can determine the optimal operating parameters of the vehicle based on the input received from the driving trajectory planner component 320 and the trajectory optimization component 322. For example, the trajectory optimization component 322 can provide multiple possible trajectories in the form of multiple vectorized data, and the vehicle can turn in a specific direction using multiple possible trajectories. In this example, the planning model 318 can analyze the vectorized data of each trajectory based on the operating parameters of the vehicle (such as the speed, rate, acceleration, etc. of the vehicle). Then, the planning model 318 determines the optimal trajectory that can provide the best driving in terms of comfort, safety and driving efficiency.

[0062] The memory 310 may also include a delay calculation component 320 for determining the delays caused by the hardware components or software models for autonomous driving. The delay calculation component 320 may be implemented by utilizing the planning model 318. In some embodiments, the delay calculation component 320 may determine each delay caused by a hardware component such as a camera, a sensor, or any system component associated with decision making for autonomous driving. The delay calculation component 320 may also determine the delays caused by any computing software model that processes data associated with decision making for autonomous driving. In some embodiments, the calculated delays may be used in a single component or combination of components in the memory 310. In some embodiments, the calculated delays may be updated in real time.

[0063] The memory 310 may also include a driving trajectory planner component 322 for generating multiple driving paths that can be used in autonomous driving. In some embodiments, when the vehicle is operating in an autonomous driving state, the driving trajectory planner component 320 searches all possible driving paths in real time. For example, when the vehicle needs to change lanes, the driving trajectory planner component 322 can generate multiple driving paths for changing lanes. In this example, the vehicle can analyze its surroundings, such as other vehicles or any objects around the vehicle. After determining the driving path, the driving trajectory planner component 322 can convert each driving path into vectorized data, where the vectorized data is transmitted to the planning model 318.

[0064] The memory 310 may also include a trajectory optimization component 324 for generating multiple trajectories that can be used in autonomous driving when the vehicle needs to move in a certain direction. In some embodiments, when the vehicle needs to move in a certain direction, the trajectory optimization component 324 searches all possible trajectories in real time. For example, when the vehicle needs to turn right, the trajectory optimization component 324 can generate multiple right turn trajectories. In this example, the trajectory optimization component 324 can convert each driving path into vectorized data, wherein the vectorized data is transmitted to the planning model 318. In some embodiments, the trajectory optimization component 324 can represent a neural network that outputs a trajectory. For example, a neural network can be trained to output a trajectory based on an input (e.g., an input can at least represent an output of a visual stack of an object that is positioned around the vehicle).

[0065] In some embodiments, the trajectory optimization component 324 can simulate the vehicle driving on the trajectory. In these embodiments, for each simulation on the individual trajectory, the trajectory optimization component 324 can generate simulation results, which include, for example, data related to the position, speed, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk or lateral jerk when driving on the driving path. The trajectory optimization component 324 can use an objective function (whose value is maximized) to evaluate the adaptability of the trajectory. In some embodiments, the memory 310 may also include a vehicle controller component 326 to control the driving vehicle on the trajectory path determined in the trajectory optimization component 324. In some embodiments, the vehicle controller component 326 generates commands related to the operation of the vehicle, such as steering, acceleration and braking. These commands include, for example, steering commands for controlling the steering motor and speed and acceleration commands for controlling the motor, and jerk commands for controlling the braking system. For example, based on data related to the trajectory path determined in the trajectory optimization component 324 (such as data related to position, speed, heading, curvature, longitudinal acceleration, lateral acceleration, longitudinal jerk or lateral jerk), the controller component 326 generates these commands. In one embodiment, the controller component 326 can be implemented as a standalone component, such as a controller. In some embodiments, the controller component 326 can implement classical and / or modern control techniques (e.g., PID controllers, nonlinear control techniques, adaptive and learning techniques, etc.). In some embodiments, Figure 5A As described in , controller component 326 can be removed so that component 324 can directly output commands for steering, acceleration, braking, etc. (e.g., lower-level commands).

[0066] Although illustrated as components combined within planning component 212, those skilled in the relevant art will appreciate that one or more of the components in memory 310 may be implemented in a personalized computing environment, including physical and virtualized computing environments.

[0067] Now go to Figure 4A-4C , describing illustrative interactions of components of an environment processing visual system data and generating simulated content system data to update a training model for a machine learning process. At (1), one or more vehicles 102 may collect and transmit a set of inputs (e.g., a first data set). The first data set illustratively corresponds to video image data collected by a vision system 200 of a vehicle 102 and any associated metadata or other attributes.

[0068] Illustratively, the vehicle 102 may be configured to collect vision system data and transmit the collected data. For example, the collected vision system data may be transmitted based on a periodic time frame or various collection / transmission criteria. Additionally, in some embodiments, the vehicle 102 may also be configured to identify a particular scene or location, such as via geographic coordinates or other identifiers, that will result in the collection and transmission of the collected data. Figure 4A As shown in , at (2), the collected vision system data can be transmitted to the simulation content service 122 directly from the vehicle 102 or indirectly via the network service 110.

[0069] At (3), the simulation content service 122 receives and processes the collected visual system data from the vehicle 102. Illustratively, the simulation content service 122 can process the visual-based data to complete lost video data frames, update version information, error correction, etc. In addition, at (3), the simulation content service 122 can further process the collected visual system data to identify ground-truth labels for the captured video data. Illustratively, the ground-truth labels can correspond to any of a variety of detectable objects that may be depicted in the video data. In one embodiment, the ground-truth label data may include information identifying the edge of the road. In addition, the ground-truth label data may include information that depends on the identified edge of the road, such as lane lines, road centers, and one or more fixed objects (e.g., road signs, markings, etc.). In addition, in some embodiments, the ground-truth label data may include dynamic object data associated with one or more identified objects (such as vehicles, dynamic obstacles, environmental objects, etc.).

[0070] At (4), the simulation content service 122 may process the ground truth label data. Illustratively, the simulation content service 122 may process the ground truth labels based on priority for identifying / extracting core ground truth label data that will be used as the basis for the simulation content. For example, lane edge ground truth labels may be considered to have a high or higher priority. Additional ground truth label data, such as lane line labels, lane center labels, static object labels, or dynamic object labels, may be associated with a low or lower priority with respect to the lane label data or with each other. In some embodiments, the label data may be filtered to remove one or more labels (e.g., dynamic objects) that may be replaced by simulation content or otherwise not required for generating the simulation content. For purposes of illustration, the processed ground truth label set may be considered to be a content model attribute that will be used for the simulation content.

[0071] At (5), the simulation content service 122 generates a model for use in future generation of simulated content. Illustratively, the simulation content service 122 may process content attributes in the form of error adjustment, extrapolation, variation, and the like.

[0072] At (6), the simulation content service 122 may generate index data or attribute data (e.g., metadata) for each clip or simulation content data that will facilitate selection, sorting, or maintenance of the data. The index or attribute data may include identification of the location, type of simulation object, number of variations generated / available, simulated environmental conditions, tracking information, original source information, etc. Figure 4A The purpose of this is to generate simulation content without a specific request / need for training scenarios, which will refer to Figure 4B Give a description.

[0073] refer to Figure 4B , illustratively, the stored and indexed simulation content information can be provided to the network service 110 as part of the training data. At (1), the simulation content service 122 can receive a selection or criteria for selecting data. Illustratively, the computing device 104 can be utilized to provide criteria, such as sorting criteria. In some embodiments, a request for simulation content is utilized to provide properties of the simulation content. Therefore, when the simulation content itself is generated, the generation of the simulation content can be considered to be a response to the request for the simulation content. Therefore, the generation of the simulation content can be considered to be synchronous in nature or dependent in nature. In other embodiments, the request can be a simple selection of an index value or attribute, so that the simulation content service 122 can generate the simulation content based on pre-configured attributes or configurations that are independent of a single request for the simulation content. Therefore, the generation of the simulation content can be considered to be independent of the request.

[0074] At (2), the network service 110 can then process the request and identify the generated simulation content model, such as via index data. At (3), the simulation content service 122 generates supplemental video image data and associated attribute data. Illustratively, the simulation content system 120 can utilize a set of variables or attributes that can be changed to create different scenes or scenarios for use as supplemental content. For example, the simulation content system 120 can utilize color attributes, type of object attributes, acceleration attributes, action attributes, time of data attributes, location / position attributes, weather condition attributes, and density of vehicle attributes to create various scenes related to the identified objects. Illustratively, the supplemental content can be utilized to simulate real-world scenarios that may be unlikely to occur or may be unlikely to be measured by the vehicle group 102. For example, the supplemental content can simulate various scenarios corresponding to unsafe or dangerous conditions.

[0075] The simulated content system 120 can illustratively utilize statistical selection of scenes to avoid repetitions based on minor differences (e.g., similar scenes differing only in the color of objects) that might otherwise bias the machine learning process. In addition, the simulated content system simulates the content service 122 as a plurality of supplemental content frames and differential distributions in one or more variables. Illustratively, the output from the simulated content service 122 can include a label (e.g., ground truth information) that identifies one or more attributes (e.g., position, velocity, and acceleration) that can be detected or processed by the network service 110. In this regard, the simulated content data set can facilitate detailed labeling and can be dynamically adjusted as needed for different machine learning training sets. At (4), the simulated content training set is transmitted to the network service 110.

[0076] Now go to Figure 4C Once the network service 110 receives the training set, at (1), the network service 110 processes the training set. At (2), the network service 100 generates an updated machine learning process based on training the combined data set. Illustratively, the network service 110 can utilize various planning models to generate the updated machine learning process.

[0077] refer to Figure 5A , a block diagram of illustrative interactions for controlling a vehicle will be described. Figure 5A As shown in FIG. , the planning component 212 (eg Figure 2A , Figure 2B and Figure 3 A) may receive a representation of the world around the vehicle (e.g., the vehicle's surroundings) measured by a sensor suite 512 and processed by a perception system 514. The sensor suite may provide Figure 2A and Figure 2B , and a processor for processing the output data of these sensors. In some embodiments, planning component 212 is configured to make decisions and plan trajectories (e.g., driving trajectories) to achieve certain driving standards for autonomous vehicles, such as making decision-making processes along a desired navigation route while maintaining safe and comfortable driving. In some embodiments, the trajectory generated by planning component 212 is sent to a controller, which calculates the speed and steering commands of the actuator to accurately track the output trajectory of the planning system. In these embodiments, the controller can generate commands related to the operation of the vehicle, such as steering, acceleration, and braking. These commands include, for example, steering commands for controlling a steering motor and speed and acceleration commands for controlling a motor, as well as jerk commands for controlling a braking system. For example, based on the trajectory generated by planning component 212, controller component 326 generates these commands.

[0078] Figure 5AExamples of vehicle components used to control vehicle operating parameters for autonomous driving are illustrated. Figure 5A As shown in , the vehicle may include some or all of a sensor suite 512, a perception system 514, a planning system 516, a controller 518, and an actuator 520. These components are illustrative and provided only as examples, and the present application does not limit the systems or components of the vehicle. In some embodiments, the sensor suite 512, the perception system 514, the planning system 516, and the controller 518 may be implemented as a planning component 212. In some embodiments, autonomous driving may be performed based on a set of instructions executed in real time by an autonomous processor implemented in the vehicle while the vehicle is driving.

[0079] The sensor suite 512 can collect sensor data measured by various sensors implemented in the vehicle. For example, the sensor suite 512 can collect data from local sensors 205, navigation 235, and GPS 245, such as Figure 2A The collected sensor data may be transmitted to the perception system 514. In some embodiments, the perception system 514 processes the collected data and determines the current vehicle state, such as position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration. In some embodiments, each current vehicle state may be used as an autonomous processor ( Figure 5A ), and the autonomous processor can generate vehicle control instructions based on the current vehicle state. In some embodiments, the generated instructions are not used directly to control the vehicle. Instead, the generated instructions can be further processed by the planning system 516 to optimize vehicle operation. For example, the planning system 516 can determine the time delays used to control vehicle actuators. The planning system 516 can determine the time delays to estimate the overall system delays, for example, the time delays can be selected as the highest value, lowest value, or central tendency measure of the time delays of the actuators, or the highest value, lowest value, or central tendency measure of the longitudinal or lateral delays (for example, actuators associated with longitudinal or lateral control). The system 516 can also determine the time delay for each vehicle actuator (for example, steering and motor actuators). For example, the time delays generated by local sensors 225, navigation 235, and GPS245 (such as Figure 2AEach data measured by the control system 516 (as shown in ) may include a timestamp, and the measured data may be used to control the steering and motor actuators. The planning system 516 may determine delays from the measured data to actuate the steering and motor actuators. In some embodiments, the delays for each actuator are different even if the data is measured at the same time. For example, the delays may include time delays in receiving input data from local sensors (e.g., measurement delays), time delays in processing local sensor data (e.g., computational delays), time delays in transmitting and receiving data between components (e.g., communication delays), and time delays in operating individual steering and motor actuators (e.g., hardware delays).

[0080] After estimating the delay, the planning system 516 can start the instruction calculation. For example, if the delay of controlling the steering actuator is high, the planning system 516 can wait for the duration of the determined delay to receive the instruction related to the steering actuator after receiving the instruction to control the motor actuator. In some embodiments, the planning system 516 can update the received instructions for each steering and motor actuator. For example, the motor actuator control instruction is received earlier than the steering actuator control instruction (due to different delays), and if the steering actuator control instruction received later includes a higher steering tilt angle, the planning system can reduce the motor speed. Once the calculation of the planning system 516 is completed, in some embodiments, the result is sent to the controller 518 to calculate the control command. In some embodiments, when the measured environment changes, the planning system 516 can restart a new calculation cycle by receiving new measurements from the perception system 514 and sending updated vehicle operation instructions to the controller 518. For example, when the vehicle driving trajectory changes, the cycle can be restarted. These loops or updated trajectories can be executed in real time, such as repeatedly running in a calculation cycle. For example, this cycle may include receiving new measurements from sensor suite 512, collecting measurement data via perception system 514, processing the collected data via planning system 514 and determining a trajectory for navigating in the environment, and converting to control commands via controller 518. In some embodiments, planning system 516 may output control commands directly to actuator 520. For example, planning system 516 may include a neural network trained to output control commands. Thus, in some embodiments, controller 518 may be optional or may be removed.

[0081] In some embodiments, at the beginning of each planning cycle (e.g., at the beginning of each round), the planning system 516 can receive perception information about the state of the vehicle (e.g., position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration) from the perception system 514. The planning system 516 can use this state information to determine trajectory information, such as the beginning of the trajectory to be planned in the current cycle. In some embodiments, by using a process called "feedforward", this state information can be used to estimate how far the vehicle has traveled on the trajectory of the previous cycle based on the state information of the vehicle's driving point on the trajectory of the previous cycle. In these embodiments, this state along the trajectory of the previous cycle when the trajectory planning of the current cycle is initiated is called the feedforward state. Feedforward can enable the trajectory planning of the current cycle to be consistent with the trajectory previously planned by the planning system. In some embodiments, the autonomous driving vehicle system has an architecture including a trajectory planning system and a controller, and the output trajectory from the planning system can be regarded as a "reference" for the controller to track, so that the controller can use feedback from the vehicle state information to consider unmodeled disturbances and minimize tracking errors relative to the reference trajectory. In these embodiments, the controller may minimize the deviation between the planned trajectory and the measured state of the vehicle, and the trajectory planning system may generate a trajectory that makes incremental progress toward a higher-level planning goal (e.g., staying in the center of the lane, completing a turn maneuver, etc.).

[0082] In some embodiments, planning component 212 can use nonlinear optimization or machine learning models (e.g., neural networks) to find a trajectory that is: (1) feasible (e.g., obeys constraints imposed by vehicle kinematics, such as a maximum turning radius, or actuation limits, such as minimum and maximum acceleration and jerk), (2) safe (e.g., obeys constraints imposed by road geometry, avoids collisions with curbs, follows lane markings, maintains a safe following distance relative to the vehicle ahead, and reacts appropriately by offsetting or slowing any objects, vehicles, people, or animals along the planned trajectory), and (3) comfortable for the end user (e.g., exhibits characteristics similar to those of human driving, such as preferring shorter, smoother paths that require the least steering action or acceleration). In these embodiments, the mathematical formulation of such optimization can involve expressing the trajectory as a function of several optimizable parameters. For example, in order to accurately model the different delays of the steering system and the drive motor control system, in some embodiments, the state variables and control inputs along the trajectory can be split into lateral and longitudinal components. In some embodiments, the planning component 212 can generate a trajectory by utilizing two separate mathematical functions that describe the evolution along the path: (1) lateral control-related state variables and control inputs (position, heading, steering angle, steering angle rate, steering angle angular acceleration) and (2) longitudinal control-related state variables (time, velocity, acceleration, longitudinal jerk, longitudinal jump at the current distance along the path). Using nonlinear optimization (which refines the initial guess of the trajectory parameters to obtain an optimized solution), the value of the objective function that scores the trajectory according to the previously listed criteria can be iteratively maximized. Since the lateral and longitudinal components of the trajectory are separated into two independent functions, their initial parts can be constrained exactly in a way that models the estimated delays of the lateral and longitudinal actuators.

[0083] refer to Figure 5B , describes an example of a data pipeline that represents the delay in commanding a vehicle. In this example, Figure 5BAs shown in , the horizontal axis represents time, and therefore, the width of each block can represent the delay corresponding to each block. In some embodiments, the planning component 212 can generate a trajectory in the "planner execution" block 532 and then send it to the "controller execution" block 534, which sends commands to the "steering control and actuation" and "motor control / brake actuation" blocks 536-540. In some embodiments, in the absence of block 534, block 532 can send commands to blocks 536-540 (for example, component 532 can output commands via a neural network trained to control actuators). In these embodiments, since the actuator-specific blocks 536-540 represent different actuators, their actuation delays to achieve the desired control values ​​may be different, represented by the different widths of these control blocks along the time axis. In some embodiments, in addition to these actuation delays, the planning component 212 can also highlight the algorithm runtime and phase delays between different components due to the scheduling of the calculations, all of which may cause delays after the start of the planner execution.

[0084] refer to Figure 5C , describes an illustrative example of using estimated delays for lateral and longitudinal control in a trajectory representation. In some embodiments, planning component 212 can maintain estimates of delays and predict delays (e.g., Figure 5B ) to generate a trajectory that more accurately represents how the future state of the vehicle will evolve. For example, in the current planning cycle, the planning component 212 can recognize that the output trajectory of the previous planning cycle may still be in the process of being actuated to achieve the desired control value (e.g., the desired command to the vehicle). The planning component 212 can set the initial portion of its trajectory to match the portion of the trajectory output of the previous cycle. In doing so, the planning component 212 can model the process that the current and recent evolution of the vehicle state may not be immediately affected by the output of the planning component 212. Instead, the planning component 212 can initiate its decision making and trajectory optimization from a point in time in the near future, estimating that at these points in time, downstream processes and actuators will be able to move the vehicle in response to the output of the current planning component 212. In some embodiments, the planning component 212 can initiate its decision making process based on the estimated future states at the end of the estimated delay duration, including starting trajectory optimization at these estimated future states. In these embodiments, since the trajectory formula can be split into lateral and longitudinal components 552, 554, the planning component 212 can also merge the estimated lateral and longitudinal delay segments in a decoupled manner, such as Figure 5C Based on the different delays of the different actuators, the decoupling of the lateral and longitudinal components 552, 554 can be exploited to maintain the responsiveness of the lower delay actuator path while achieving better accuracy of the trajectory optimization model.

[0085] In some embodiments, those quantities in the objective function of trajectory optimization that depend on the lateral and longitudinal components of the trajectory (e.g., lateral acceleration, which depends on the vehicle's speed and the curvature of the path) may be evaluated. In these embodiments, trajectory optimization may take into account potentially different lateral and longitudinal delay segment lengths to ensure that optimizable quantities along both trajectory components are evaluated at aligned points in the trajectory evolution. In some embodiments, delay segments at the initial portions of the lateral and longitudinal trajectory components (modeled as fixed based on the trajectory of the previous cycle) may not be optimized by trajectory optimization. Although Figure 5C The description of focuses on the delays associated with the lateral and longitudinal components, but in some embodiments, the planning component 212 can use one of the components to estimate the overall system delay. As an example, the component 212 can output a control signal that combines lateral and longitudinal control (e.g., the component 212 can represent a machine learning model, such as a neural network). For this example, the component 212 can therefore determine the delay associated with the overall control. Optionally, the planning component 212 can estimate the overall system delay based on selecting the highest value, the lowest value, or the central tendency measure of the components 552, 554.

[0086] refer to Figure 5D , describes an illustrative example of driving by modeling an optimized trajectory taking into account system delays. In some embodiments, the system model is in lateral and longitudinal components 552, 554 (such as Figure 5C The accuracy of the command vehicle can provide the best safety and comfort driving. Figure 5D As shown in , without taking into account the estimated feedforward system delay, trajectory optimization 562 may be less accurate because it may plan a maneuver, such as a sudden deceleration, that it may attempt to accomplish with less deceleration and jerkiness than is necessary to stop within the desired distance along the path. Figure 5DAs shown in , the planning component 212, taking into account its delay estimates for longitudinal actuation, can recognize that it may need to use more dramatic deceleration and jerkiness to achieve the desired behavior, and it can plan such a trajectory 564 that the controller can track with less tracking error, which is caused by more accurate modeling of system delays. In this example, trajectory 566 represents when there is no sudden deceleration. In some embodiments, the use of longitudinal delay segments that are decoupled from the lateral delay segments can avoid the possibility of introducing artificial delays in the system response when the lateral control delay is longer than the longitudinal control delay. In addition to improving the safety of the vehicle by more accurately modeling the system response in the planning component 212, such modeling of system delays can also improve ride comfort: because the actuator output can generally be more consistent with the planning component 212's desired and generated trajectory modeled, the planning component 212 can control ride smoothness and generally expect the actuator to more accurately follow the desired trajectory.

[0087] The foregoing disclosure is not intended to limit the disclosure to the precise form disclosed or to the specific field of use. Thus, it is contemplated that various alternative embodiments of the disclosure and / or modifications to the disclosure, whether explicitly described or implied herein, are possible in light of the present disclosure. Having thus described the embodiments of the disclosure, one of ordinary skill in the art will recognize that changes may be made in form and detail without departing from the scope of the disclosure. Therefore, the disclosure is limited only by the claims.

[0088] In the foregoing description, the present disclosure has been described with reference to specific embodiments. However, as will be appreciated by those skilled in the art, the various embodiments disclosed herein may be modified or otherwise implemented in various other ways without departing from the spirit and scope of the present disclosure. Therefore, this description should be considered illustrative, and the purpose is to teach those skilled in the art to make and use the disclosed various embodiments of the decision-making and control processes. It should be understood that the disclosed forms shown and described herein should be considered as representative embodiments. Equivalent elements, materials, processes or steps may replace those representatively illustrated and described herein. In addition, some features of the present disclosure may be used independently of the use of other features, all of which will be apparent to those skilled in the art after benefiting from the description of the present disclosure. Expressions such as "including", "comprising", "merging", "consisting of", "having", "being" and the like used to describe and claim the present disclosure are intended to be interpreted in a non-exclusive manner, i.e., allowing items, components or elements not explicitly described to also exist. References to the singular should also be interpreted as being related to the plural.

[0089] In addition, various embodiments disclosed herein should be understood in an illustrative and explanatory sense, and should never be interpreted as limiting the present disclosure. All connection references (e.g., attaching, fixing, coupling, connecting, etc.) are only used to help readers understand the present disclosure, and must not cause restrictions, particularly the position, orientation or use restrictions of the system and / or method disclosed herein. Therefore, if there is a connection reference, it should be interpreted broadly. In addition, such connection references do not necessarily mean that the two elements are directly connected to each other.

[0090] In addition, all numerical terms, such as but not limited to "first", "second", "third", "primary", "secondary", "primary" or any other general and / or numerical terms, should also be used merely as identifiers to help the reader understand the various elements, embodiments, variations and / or modifications of the present disclosure, and shall not create any limitations, especially with respect to the order or preference of any element, embodiment, variation and / or modification relative to or over another element, embodiment, variation and / or modification.

[0091] It should also be appreciated that one or more of the elements depicted in the drawings / figures may also be implemented in a more separate or more integrated manner, or even in some cases may be removed or rendered inoperable, as may be useful depending on the particular application.

Claims

1. A system for planning a trajectory of a vehicle for autonomous driving, the system comprising one or more processors and a non-transitory computer storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations during a loop, the operations comprising: Obtain vehicle data from a set of sensors; Identifying vehicle trajectory information and vehicle status by processing the obtained vehicle data; identifying a vehicle control command based on the identified vehicle trajectory information and the vehicle state, wherein the vehicle state is associated with operation of at least one actuator; determining the vehicle control command for each actuator; characterizing a delay for each actuator, wherein the delay is an estimated time delay between obtaining the vehicle data and controlling a corresponding actuator of the vehicle data; optimizing the vehicle control commands based on a characterized delay of each actuator based on the trajectory information; as well as Control of the vehicle is caused based on the optimized vehicle control commands. 2 . The system of claim 1 , wherein the cycle corresponds to a vehicle driving trajectory, and a new cycle is initiated when the vehicle enters a new trajectory. 3 . The system of claim 1 , wherein the trajectory information indicates a start of the trajectory, and wherein the vehicle state at the start of the trajectory matches a vehicle state controlled by the optimized vehicle control instructions in a previous cycle. 4 . The system of claim 1 , wherein the operations further comprise identifying an optimal trajectory path prior to optimizing the vehicle control commands, wherein the vehicle control commands are generated based on the optimal trajectory path.

5. The system of claim 1, wherein the actuator includes a lateral component and a longitudinal component, and wherein the lateral component is associated with operation of a steering actuator, and wherein the longitudinal component is associated with operation of a motor actuator.

6. The system of claim 5, wherein an estimated delay corresponds to the longitudinal component being lower than the lateral component, and wherein the system delays the optimized time of the vehicle control command by a duration of the estimated delay corresponding to the lateral component.

7. The system of claim 1, wherein the vehicle trajectory information is identified by identifying geographic coordinates and collecting vision system data, and wherein the vision system data includes ground truth labels for video images captured by a camera implemented in the vehicle.

8. The system of claim 1, wherein the vehicle state comprises current vehicle position, heading, steering angle, speed, longitudinal acceleration, and lateral acceleration. 9 . The system according to claim 1 , wherein each of the acquired vehicle data includes a time stamp indicating a measurement time of each vehicle data. 10 . The system of claim 9 , wherein the delay is estimated by determining a time difference between the time stamp of each vehicle data and a specific time of a corresponding actuator operation of each vehicle data.

11. The system of claim 1, wherein the delay comprises a measurement delay, a calculation delay, a communication delay, and a hardware delay.

12. The system of claim 1, wherein the system is configured to determine the trajectory information based on the identified vehicle state.

13. The system of claim 1, wherein the optimization is based on predetermined criteria, the criteria comprising: Vehicle kinematics, safety constraints, and comfort constraints.

14. A system for planning a vehicle trajectory, the system comprising: a sensor suite configured to collect vehicle data from vehicle sensors implemented in the vehicle; a perception system configured to generate a vehicle state by receiving and processing the vehicle data collected from the sensor suite; as well as A planning system, the planning system being configured to: identifying vehicle control instructions for the generated vehicle state; determining a delay for each actuator used to control the vehicle; as well as A trajectory of the vehicle is optimized based on the determined delay.

15. The system of claim 14, wherein the vehicle state includes vehicle trajectory information, and wherein the planning system optimizes the vehicle trajectory information based on the delay determined for each actuator.

16. The system of claim 14, wherein the optimization is based on predetermined criteria, the criteria comprising: Vehicle kinematics, safety constraints, and comfort constraints.

17. The system of claim 14, wherein the system plans the vehicle trajectory for each cycle, and wherein a new cycle is initiated when the vehicle enters a new trajectory.

18. The system of claim 14, wherein the vehicle includes two actuations corresponding to a lateral component and a longitudinal component of the vehicle, and wherein the lateral component is associated with operation of a steering actuator, and wherein the longitudinal component is associated with operation of a motor actuator.

19. A method for planning a trajectory, the method comprising: Obtain vehicle data from a set of sensors; Identifying vehicle trajectory information and vehicle status by processing the obtained vehicle data; identifying a vehicle control command based on the identified vehicle trajectory information and the vehicle state, wherein the state is associated with operation of at least one actuator; determining the vehicle control command for each actuator; characterizing a delay for each actuator, wherein the delay is an estimated time delay between obtaining the vehicle data and controlling the corresponding actuator; optimizing the trajectory information based on a characterized delay of each actuator; and Control of the vehicle is caused based on the optimized trajectory.

20. The method of claim 19, wherein causing control of the vehicle is based on identifying vehicle control instructions for driving the vehicle using the optimized trajectory information.

21. The method of claim 19, wherein the optimization is based on predetermined criteria, the criteria comprising: Vehicle kinematics, safety constraints, and comfort constraints.

Images