Computing systems and non-temporary computer-readable media

Abstract

The computing system can receive motion prediction data from the vehicle, generated by a motion prediction model running on the vehicle. Based on the motion prediction data, the system can determine predicted trajectories for multiple entities in the vehicle's surrounding environment. The system can evaluate the predictive performance of the motion prediction model by (i) matching the predicted endpoint of each predicted trajectory in the set of predicted trajectories for each individual entity of the multiple entities to one or more underlying lanes in a road division; (ii) matching the ground truth future position of the entity to one or more underlying lanes in an underlying lane topology; and (iii) determining the distance along one or more lane divisions between the lanes (one or more) that match the predicted endpoint and the lanes (one or more) that match the ground truth future position.

Description

Background Art

[0001] An autonomous vehicle performs motion prediction for an external agent or entity (e.g., other vehicles, pedestrians, cyclists, and other moving objects) to safely navigate across a road network. The motion prediction model is preferably executed on board the autonomous vehicle to plan a sequence of trajectories so that no collisions occur and the autonomous vehicle operates in a safe, stable, and reliable manner.

Summary of the Invention

Means for Solving the Problems

[0002] A computing system for evaluating a motion prediction model for implementation by a semi-autonomous or fully autonomous vehicle is described herein. In various examples, the computing system can receive motion prediction data from a vehicle. The motion prediction data can be generated by a motion prediction model executing on the vehicle as the vehicle operates along a travel route within a road network. Based on the motion prediction data, the computing system can determine a set of predicted trajectories for each of a plurality of entities in the vehicle's surrounding environment. The system then determines the lane topology underlying the road segment on which the vehicle is operating and can evaluate the prediction performance of the motion prediction model based on whether the ground truth future positions of the entities are within an error tolerance region based on aligning the trajectories to the underlying lanes.

[0003] As provided herein, a road division may include multiple lanes, which may include turning lanes, oncoming lanes, parking spaces, and / or lanes with traffic in the same direction as the vehicle. For example, a road division may include the current position of a vehicle and any surrounding vehicles, each of which may be traveling in a specific lane or changing lanes. A road division may further include the predicted trajectories of each of the surrounding vehicles and / or other entities, as well as the future ground truth positions of the vehicle and the surrounding vehicles and / or entities. A lane division includes a portion of a single lane of a road division. According to the examples provided herein, a vehicle and surrounding entities may operate along a road division and occupy a single lane division at any given time, or (for example, when changing lanes) occupy multiple lane divisions.

[0004] As illustrated by various examples described herein, a system can evaluate a motion prediction model by matching the predicted endpoints of a set of predicted trajectories for each individual entity of a plurality of entities to the underlying lanes of the underlying lane topology of the road division. The system can then determine one or more matching lane divisions for each predicted trajectory. In some examples, the system can make this determination by determining the assignment confidence for each lane division of the underlying lane topology within the proximity of the predicted trajectory, and then determining the distance between the predicted endpoint and the ground truth future position of the individual entity along the matching lane division (e.g., recorded by a set of sensors mounted on the vehicle).

[0005] In certain implementations, a computing system can determine the matched lane segment(s) for a predicted trajectory based on the matched lane segment(s) with the highest assignment confidence relative to other adjacent lanes in the underlying lane topology. In further examples, a computing system can match the predicted endpoint of a predicted trajectory by dynamically calculating distance-based and direction-based assignment confidence for each surrounding lane segment of the predicted trajectory. In such examples, a computing system can determine the highest assignment confidence for the underlying lane segment(s) based on dynamically calculating distance-based and direction-based assignment confidence.

[0006] In various examples, the same matching procedure is also performed using the entity's ground truth endpoint. In particular, the vehicle's ground truth endpoint matches one or more underlying lanes (for example, based on calculating distance-based and direction-based assignment confidence to assign the ground truth endpoint to a specific lane division).

[0007] In various examples, a computing system can receive recorded sensor data from a vehicle, which can indicate the ground truth trajectory for each of several entities. Based on the ground truth trajectory for each of the several entities, the computing system can determine the future ground truth position of each of the several entities at a future point in time, corresponding to the predicted trajectory and endpoint of the entities. If the ground truth position is within an error tolerance based on the matching underlying lane, the prediction can include a "hit." If the ground truth position is outside the error tolerance, the prediction can include a "miss."

[0008] Additionally or alternatively, the computing system can evaluate the motion prediction model by dynamically determining the distance along the lane sections connecting the lane sections that match the predicted endpoint and the lane sections that match the ground truth endpoint. These matched lane sections may be the same or different, and the distance between the matched lane sections provides an indicator of the accuracy of the motion prediction model. Thus, the computing system evaluates the motion prediction model by comparing the lane section matches between the predicted endpoint of the entity and the ground truth endpoint of the entity.

[0009] As provided herein, a vehicle may include an autonomous vehicle that runs a motion prediction model, dynamically generates a motion plan based at least in part on a set of predicted trajectories for each of several entities in the autonomous vehicle's surrounding environment, and autonomously operates a set of control mechanisms of the autonomous vehicle to execute the motion plan. These control mechanisms may include a braking system, a steering system, an acceleration system, and / or a signal transmission system, which the vehicle's onboard computing system operates autonomously according to the motion plan.

[0010] The computing system can evaluate the predictive performance of each of several motion prediction models in the manner described herein to determine the most accurate motion prediction model for execution in autonomous vehicles. As provided herein, the determination of distance-based and orientation-based assignment confidence for matching the predicted endpoint of the predicted trajectory to the underlying lane allows for a greater tolerance of longitudinal error compared to lateral error, which is more critical to the safety of autonomous vehicles. Previous methods (e.g., Euclidean-based evaluation methods) treat longitudinal and lateral errors similarly, potentially resulting in motion prediction models with low overall error quality but unacceptable lateral error from a safety standpoint.

[0011] The disclosures herein are illustrative and not limiting, and in the figures of the accompanying drawings, similar reference numbers refer to similar elements. [Brief explanation of the drawing]

[0012] [Figure 1] This block diagram shows an exemplary computing system for evaluating motion prediction models, as illustrated in the examples described herein. [Figure 2] A block diagram shows a computing system including a dedicated module for evaluating motion prediction models, as illustrated in the examples described herein. [Figure 3A] This figure illustrates the use of a Euclidean-based distance metric in evaluating motion prediction models, as described in the examples provided in this specification. [Figure 3B] This figure shows the use of a lane-based distance metric to evaluate motion prediction models, as illustrated in the examples described herein. [Figure 4] This flowchart illustrates an exemplary method for evaluating motion prediction models using various examples described herein. [Figure 5] This flowchart illustrates an exemplary method for evaluating motion prediction models using various examples described herein. [Modes for carrying out the invention]

[0013] Evaluation metrics are required to make decisions regarding motion prediction models implemented by autonomous vehicles. Autonomous vehicle operators will typically need quantitative methods to develop and test multiple motion planning techniques and determine which motion planning models(s) are best suited for deployment in a fleet of semi-autonomous or fully autonomous vehicles. Evaluation metrics are also required to monitor how well machine learning-based motion prediction models scale with data. Embodiments described herein provide a computing system that quantifies whether a particular scenario (e.g., an unprotected left turn) is predicted adequately in terms of accuracy and safety, and whether additional new fleet data improves the predictive performance of the motion prediction model (e.g., as measured by metrics). In implementing the motion prediction model evaluation techniques described herein, the computing system is intended to be able to determine the optimal motion prediction model for deployment in a fleet of semi-autonomous and / or autonomous vehicles.

[0014] Previous methods for predicting the motion of external entities to a vehicle (e.g., other vehicles, pedestrians, cyclists, and other moving objects) involved Euclidean-based metrics (e.g., distance in Euclidean space) or intent prediction models where machine learning was employed to predict the immediate intentions of nearby vehicles. In the case of Euclidean distance methods, lateral errors may be treated as equivalent to or similar to longitudinal errors, even though lateral errors can lead to trajectory predictions that position entities in the opposite lane. In typical road scenarios, the longitudinal prediction delta (e.g., in the direction of vehicle travel) is considerably less important than the lateral prediction delta.

[0015] In intent prediction methods, intent labels (e.g., an entity brakes, turns left, changes lanes, etc.) are determined for each external entity, and prediction accuracy is calculated based on these semantic labels. Inaccurate intent predictions can lead to dangerous scenarios, such as collisions or hazards to vulnerable road users (VRUs) (e.g., pedestrians and cyclists). Furthermore, evaluating intent prediction models involves determining whether the semantic labels match the entity's ground truth intent and generating a set of accuracy scores.

[0016] As described herein, a computing system is provided that evaluates the accuracy of motion prediction models using the underlying lane topology of the road section on which the vehicle operates as a constraint. In particular, the computing system can receive motion prediction data from a fleet of autonomous vehicles that implement motion prediction models to navigate along a road network. In various examples, each motion prediction model may include a machine learning model that predicts the trajectories of external entities, such as detected vehicles operating in the same road section as the vehicle, as well as the trajectories of VRUs, such as pedestrians and cyclists. These motion prediction models may utilize a variety of different techniques when predicting the trajectories of external entities, such as interaction recognition models (e.g., considering the response of external entities to the vehicle's motion plan), coupled motion planners and prediction models, object classification and prediction models, physics-based motion models, and operation-based motion models.

[0017] In embodiments described herein, a backend computing system can process motion prediction data based on the implementation of a specific motion prediction model from a set of vehicles and determine a set of predicted trajectories of external entities relative to those vehicles. These predicted trajectories may include predictions made by the vehicles when generating and executing dynamic motion plans to traverse an underlying road section. The computing system may further receive ground truth information from the vehicles, which may include sensor data indicating the actual trajectories taken by external entities and the vehicles themselves. The computing system may align the endpoints of all predicted trajectories with the underlying lanes of the road section (e.g., the centerlines of the lanes). Based on the distance to surrounding lane sections and the difference in orientation to surrounding lane sections, the computing system may calculate an assignment confidence to select one or more matched lane sections for the predicted trajectories.

[0018] In various examples, a computing system can use the predicted trajectory endpoint in a matched lane division to determine the distance to the future ground truth position of an entity and to determine the accuracy of the prediction. In certain embodiments, the computing system can match the future ground truth position of an external entity to one or more underlying lane divisions of a road division. For example, the computing system can calculate the assignment confidence in the same or similar manner as the predicted endpoint to select one or more matched lane divisions for the ground truth endpoint of an external entity. The computing system can then determine the distance along the lane division connecting the lane division match of the predicted endpoint and the lane division match of the ground truth endpoint.

[0019] In some examples, a computing system can generate or establish an error tolerance zone based on matched lanes and evaluate whether the ground truth future position ultimately falls within the error tolerance zone. Thus, the computing system dynamically implements this lane matching and evaluation technique for all trajectory and endpoint predictions made by motion prediction models for any number of external entities as a vehicle operates across a road network. Using a distance threshold or error tolerance zone as an evaluation metric, as provided herein, the computing system can then evaluate the overall accuracy of the motion prediction model. In various examples, the computing system can implement this lane topology-based evaluation technique for multiple motion prediction models to determine the most accurate and / or robust motion prediction model for execution in a fleet of autonomous or semi-autonomous vehicles. The evaluation methods described herein are intended to be iterated over new or updated motion prediction models to provide up-to-date information on the most accurate motion prediction model(s).

[0020] In certain implementations, a computing system can implement one or more of the functions described herein using a learning-based approach, such as by executing an artificial neural network (e.g., a recurrent neural network, a convolutional neural network, etc.) or one or more machine learning models to process an individual set of trajectories and classify the driving behavior of each human-driven vehicle passing through an intersection. Such a learning-based approach can further correspond to a computing system that stores or includes one or more machine learning models. In one embodiment, the machine learning model may include an unsupervised learning model. In one embodiment, the machine learning model may include a neural network (e.g., a deep neural network), or other types of machine learning models including non-linear models and / or linear models. The neural network may include a feed-forward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some exemplary machine learning models may utilize attention mechanisms such as self-attention. For example, some exemplary machine learning models may include a multi-head self-attention model (e.g., a transformer model).

[0021] As provided herein, "network" or "one or more networks" can include any type of network or combination of networks that enables communication between devices. In one embodiment, the network may include one or more of a local area network, a wide area network, the Internet, a secure network, a cellular network, a mesh network, a peer-to-peer communication link, or combinations thereof, and may include any number of wired or wireless links. Communication via the network(s) may be achieved via a network interface using, for example, any type of protocol, protection scheme, encoding, format, packaging, etc.

[0022] As further described herein, an "autonomous map" or "autonomous driving map" can include a ground truth map recorded by a mapping vehicle using various sensors (e.g., LIDAR sensors and / or a set of cameras or other imaging devices) and labeled (either manually or automatically) to indicate traffic objects and / or right-of-way rules at any given location. In a variant form, the autonomous map can include a scene reconstructed from encoded sensor data recorded and compressed by the vehicle using a decoder. For example, a given autonomous map can be labeled by a human based on observed traffic signs, traffic signals, and lane markings within the ground truth map. In a further example, reference points or other points of interest can be further labeled on the autonomous map for additional assistance to an autonomous vehicle. An autonomous vehicle or self-driving vehicle can then utilize the labeled autonomous map to perform various other operations required for localization, pose determination, change detection, and autonomous driving on a road. For example, an autonomous vehicle can refer to the autonomous map to determine traffic rules (e.g., speed limits) at the vehicle's current location and can dynamically compare live sensor data from the vehicle's onboard sensor suite with the corresponding autonomous map to safely navigate along the current route.

[0023] One or more examples described herein provide that methods, techniques, and operations implemented by a computing device are implemented by a program or as a computer-implemented method. As used herein, "by a program" means via the use of code or computer-executable instructions. These instructions can be stored in one or more memory resources of the computing device. Steps implemented by a program can be automatic or non-automatic.

[0024] One or more examples described herein can be implemented using a programmatic module, engine, or component. A programmatic module, engine, or component may include a program, a subroutine, a portion of a program, or a software or hardware component capable of performing one or more defined tasks or functions. As used herein, a module or component may reside on a hardware component independent of other modules or components. Alternatively, a module or component may be a shared element or shared process of other modules, programs, or machines.

[0025] Some of the examples described herein may generally require the use of a computing device including processing and memory resources. For example, one or more of the examples described herein may be implemented entirely or partially using network equipment (e.g., a router) on a computing device such as a server and / or a personal computer. Memory resources, processing resources, and network resources may be used in connection with the establishment, use, or implementation of any of the examples described herein (including implementation of any method or implementation of any system).

[0026] Furthermore, one or more examples described herein may be implemented through the use of instructions executable by one or more processors. These instructions may be carried on non-temporary computer-readable media. The machines shown in or described using the following figures provide examples of processing resources and computer-readable media capable of carrying and / or executing instructions for implementing the examples disclosed herein. In particular, many of the machines shown with the examples of the present invention include processors and various forms of memory for holding data and instructions. Examples of non-temporary computer-readable media include persistent memory storage devices such as hard drives in personal computers or servers. Other examples of computer storage media include portable storage units such as flash memory or magnetic memory. Computers, terminals, and network-enabled devices are all examples of machines and devices that utilize processors, memory, and instructions stored on computer-readable media. In addition, examples may be implemented in the form of computer programs or in the form of computer-usable carrier media capable of carrying such programs.

[0027] Exemplary computing system

[0028] Figure 1 is a block diagram illustrating an exemplary computing system for generating a fused environment representation of a vehicle, according to examples described herein. In one embodiment, the computing system 100 may include a control circuit 110, which may include one or more processors (e.g., microprocessors), one or more processing cores, a programmable logic circuit (PLC), or a programmable logic / gate array (PLA / PGA), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other control circuit. In some implementations, the control circuit 110 and / or the computing system 100 may be part of, or form part of, a vehicle control unit (also called a vehicle controller) that is embedded in or otherwise located in a vehicle (e.g., a Mercedes-Benz® car or van). For example, the vehicle controller may be an infotainment system controller (e.g., an infotainment head unit), a telematics control unit (TCU), an electronic control unit (ECU), a central powertrain controller (CPC), a central exterior and interior controller (CEIC), a zone controller, or any other controller (the term "or" is used herein interchangeably with "and / or"), or may include them. In a variant, the control circuit 110 and / or computing system 100 may be included in one or more servers (e.g., backend servers).

[0029] In one embodiment, the control circuit 110 may be programmed by one or more computer-readable or computer-executable instructions stored in a non-temporary computer-readable medium 120. The non-temporary computer-readable medium 120 may be a memory device, also called a data storage device, and the memory device may include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination thereof. The non-temporary computer-readable medium 120 may, for example, form a floppy disk, a hard disk drive (HDD), a solid state drive (SDD) or solid state integrated memory, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), dynamic random access memory (DRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), and / or memory stick. In some cases, the non-temporary computer-readable medium 120 may store computer-executable instructions or computer-readable instructions, such as instructions that perform the methods described below in relation to Figures 4 and 5.

[0030] In various embodiments, the terms “computer-readable instructions” and “computer-executable instructions” are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, where computer-readable or computer-executable instructions form a module, the term “module” broadly refers to a set of software instructions or code configured to cause the control circuit 110 to perform one or more functional tasks. Modules and computer-readable / computer-executable instructions may be described as performing various operations or tasks when the control circuit 110 or other hardware components execute the module or computer-readable instructions.

[0031] In further embodiments, the computing system 100 may include a communication interface 140 that enables communication over one or more networks 150 for sending and receiving data. The communication interface 140 may include any circuits, components, software, etc., for communicating over one or more networks 150 (e.g., a local area network, a wide area network, the Internet, a secure network, a cellular network, a mesh network, and / or a peer-to-peer communication link). In some implementations, the communication interface 140 may include, for example, one or more of a communication controller, a receiver, a transceiver, a transmitter, a port, a conductor, software, and / or hardware for communicating data / information.

[0032] In an exemplary embodiment, the computing system 100 may include a backend computing system that receives motion prediction data from vehicles running a motion planning model to operate across an entire road network. As provided herein, the motion prediction data received from the vehicles may indicate the predicted paths of surrounding entities as determined by the motion prediction model installed and running on the vehicles. In further implementations, the motion prediction data may include sensor data (e.g., image data, LIDAR data, radar data, etc.) indicating the ground truth path of each entity (e.g., the future ground truth of related external entities as the prediction is made).

[0033] The computing system 100 can determine the underlying lane topology of the road section on which the vehicle operates (e.g., using map data or ground truth data from the vehicle), and can match (assign) each predicted trajectory of a particular external entity to a specific lane in the underlying topology, or otherwise assign it. Thus, to evaluate the motion planning model, the computing system 100 can generate a much smaller "hit" area for the predicted trajectory compared to a Euclidean-based method. The accuracy of the motion prediction model based on this evaluation method can provide a greater accuracy penalty for lateral errors, thereby intended to result in a safer and more robust motion prediction model (e.g., a model with smaller lateral errors) for distribution to autonomous vehicles.

[0034] System Description

[0035] Figure 2 is a block diagram showing a computing system 200 including a dedicated module for evaluating a motion prediction model, as described in the examples herein. In a particular example, the computing system 200 may include a communication interface 205 for communicating with a vehicle 285 operating across a road network via one or more networks 250. As provided herein, the vehicle 285 may include an autonomous or semi-autonomous vehicle that operates a motion prediction model to navigate the vehicle's surrounding environment along a travel route. The motion prediction model is used to identify external entities to the vehicle, classify external entities, and / or predict the current trajectory and / or future locations or areas where external entities will be located. Based on these predictions, the computing system onboard the vehicle 285 can dynamically generate a motion plan for navigating across the environment.

[0036] Specifically, each vehicle 285 may include a sensor suite comprising a set of sensors such as image sensors, LiDAR sensors, radar sensors, and ultrasonic sensors. Based on real-time sensor data, the onboard computing system of the vehicle 285 can identify other vehicles and external entities such as vulnerable road units (VRUs), run motion prediction models, and dynamically predict the location or area where each external entity will be at a future point in time (e.g., 5 seconds later). The overall prediction made by the onboard computing system may be recorded by the vehicle as motion prediction data (MPD). In a further example, the computing system 200 may receive ground truth information contained in sensor data (SD) from the vehicle, which can indicate the actual trajectory of an external entity, as recorded by the vehicle's sensors.

[0037] In various examples, the computing system 200 can receive motion prediction data from the vehicle 285, which can provide information indicating how the motion prediction model predicts the trajectory of an external entity. The computing system 200 may include a lane matching module 220 that matches the predicted trajectory of each external entity to the underlying lane of the road section on which the vehicle 285 is operating. For example, each predicted trajectory can be associated with a final position or region, which may include a region where the entity is predicted to be located at a future point in time. When evaluating the motion prediction model, this predicted region may include a "hit" region. Thus, if the ground truth future position of the entity ultimately falls within this predicted region, the evaluation module 230 of the computing system 200 evaluates the prediction as a hit. Any ground truth position of an entity outside this predicted region may contain errors or inaccurate predictions.

[0038] In a further example, the lane matching module 220 can match the ground truth endpoint of each external entity to the underlying lane division. For example, the lane matching module 220 can determine the ground truth trajectory and ground truth endpoint of the external entity at a future point in time when the predicted endpoint is created (e.g., 5 seconds after the prediction). In various examples, the lane matching module 220 can assign the ground truth endpoint to the actual lane in which the external entity is located, or it can determine the assignment confidence for the ground truth endpoint in the manner described herein.

[0039] As described above, previous evaluation methods (e.g., Euclidean-based evaluation methods) typically treat longitudinal and lateral errors similarly. In the examples described herein, the lane matching module 220 can assign each predicted trajectory for each detected entity to one or more underlying lanes in order to reduce the tolerance for lateral errors. The methods described herein are intended to provide a more realistic measure of risk and / or accuracy in real-world driving environments, since lateral errors may allow for the placement of trajectory endpoints in oncoming lanes. In various examples, the lane matching module 220 can perform a comparison between (i) the endpoint of the predicted trajectory and the orientation of the predicted trajectory at the endpoint and (ii) the centerlines of the surrounding lanes. Based on the distance from the endpoint to each surrounding lane section and the difference in orientation to the surrounding lane sections, the lane matching module 220 can determine the assignment confidence for each surrounding lane section. In various cases, the lane segment(s) with the highest determined assignment confidence may be selected as the matching lane segment(s) for the predicted trajectory.

[0040] In some embodiments, the lane matching module 220 can refer to map data to match the end of a trajectory to a lane that forms the basis of a road division. For example, the computing system 200 may include a mapping module 215 that generates, stores, or otherwise retrieves map data including the road network on which the vehicle 285 operates. The map data may include details of lane topology, including lane markings (e.g., dot markings) that divide lanes moving in the same direction, lane markings that indicate turning lanes, and lane markings that distinguish traffic going in the same direction from oncoming traffic. In some examples, the lane matching module 220 can use the map data to match the end of a trajectory to a specific lane that has the same direction of travel as the vehicle 285. In a variant, the lane matching module 220 does not distinguish between lanes going in the same direction and oncoming lanes, but performs distance-based and orientation-based assignment confidence calculations that are independent of lane direction.

[0041] In one example, the lane matching module 220 aligns the trajectory endpoint to the underlying lane by comparing the trajectory endpoint and the trajectory orientation at the endpoint with the centerlines of all surrounding lane divisions. Based on the distance to the surrounding lane divisions and the difference in orientation to these surrounding lane divisions, the lane matching module calculates the assignment confidence. Assuming that d is the shortest distance from the trajectory endpoint to the centerlines of the surrounding lane divisions and α is the difference in orientation between the endpoint and the surrounding lane divisions, the assignment confidence can be calculated as follows:

[0042] Distance-based assignment confidence:

number

[0043] Direction-based assignment confidence:

number

[0044] According to these calculations, the surrounding lane division with the highest assignment confidence is then selected as the matched lane division by the lane matching module 220. In a particular implementation, if the trajectory matches the underlying lane, the motion prediction model (MPM) evaluation module 230 can evaluate the accuracy of the motion prediction model. For example, for each matched endpoint of the predicted trajectory, the evaluation module 230 can calculate the distance to the matched endpoint of the ground truth trajectory on the lane topology. The evaluation module 230 can further define a threshold for the distance, which can correspond to the error rate of the motion prediction model (e.g., any predicted trajectory endpoint outside the threshold can be classified as an error). In a further example, the threshold can be further constructed as a function of velocity (e.g., threshold = v ground-truth (const1 + const2, in this case the allowable distance along the lane topology scales linearly with respect to the ground truth trajectory velocity). In one example, the evaluation module 230 can calculate the accuracy metric of the motion prediction model as the ratio of predicted trajectories where the distance is shorter than a threshold. Thus, the accuracy metric can be interpreted as the error rate based on lane topology distance.

[0045] According to the examples provided herein, the evaluation module 230 can establish hit areas on one or more lane divisions that coincide with the predicted trajectory endpoints and / or ground truth endpoints, which in turn coincide with the underlying lanes of the road divisions. The evaluation module 230 can then determine whether the future ground truth position of an external entity (recorded by the vehicle 285, for example, at a future point in time corresponding to the prediction) falls within this established hit area. The evaluation module 230 can perform this dynamic evaluation method for each predicted trajectory of each detected entity outside the vehicle 285 along the course of the vehicle's travel path through the road network. In a particular implementation, the vehicle 285 performs motion predictions for each detected vehicle and / or VRU within a certain proximity (e.g., within 50 meters) of the vehicle 285, regardless of the direction of travel of the external entities.

[0046] In one or more embodiments, the evaluation module 230 can output a set of accuracy scores indicating the error rate of the motion prediction model. In some examples, the set of accuracy scores can distinguish between types of external entities and / or their respective trajectories, which can be classified in any manner. For example, the evaluation module 230 can output individual accuracy scores for individual classifications such as vehicles going in the same direction, oncoming vehicles, VRUs in the same or opposite direction, VRUs in general, cyclists and / or pedestrians, and individual classifications such as heavy trucks, regular trucks, vans, and automobiles. Thus, using the lane-based evaluation technique described herein, the evaluation module 230 can provide a more accurate or useful representation of the error rate of the motion prediction model for each of these classifications.

[0047] In certain cases, the evaluation module 230 can determine the finer level of accuracy of a particular motion prediction model. For example, the evaluation module 230 can determine that a motion prediction model is particularly accurate when predicting the trajectory of other vehicles, or that another motion prediction model is very accurate when predicting the trajectory of a lane-changing vehicle, an oncoming vehicle, a VRU, etc. Thus, when performing the accuracy evaluation techniques described herein, the evaluation module 230 can determine the “optimal” motion prediction model for a given fleet of vehicles that serve one or more purposes. As provided herein, the “optimal” motion prediction model can address autonomous driving tasks or semi-autonomous driving tasks required of the vehicles in the fleet.

[0048] As an example, the evaluation module 230 can evaluate the accuracy of several motion prediction models to determine the optimal model for use in conjunction with the vehicle's advanced driver assistance system (ADAS) (for example, for performing emergency maneuvers or emergency braking actions). As a further example, the evaluation module 230 can identify the most optimal motion prediction model for predicting the trajectories of pedestrians and cyclists (for example, on congested urban roads). As yet another example, the evaluation module 230 can identify the most accurate motion prediction model for general autonomous driving operations across typical road traffic consisting of other vehicles (for example, a motion prediction model that is highly accurate in predicting the trajectories of other vehicles that generally operate along the same direction as vehicle 285).

[0049] Therefore, taking into account a given fleet of vehicles and their respective purposes (one or more), the evaluation module 230 can provide the MPM distribution module 235 with the most optimal motion prediction model, which can then distribute that optimal motion prediction model to the fleet of vehicles accordingly. In some examples, the evaluation module 230 can further take into account computational constraints relating to a given fleet of vehicles. For example, the fleet of vehicles may also have computer hardware constraints, spatial constraints relating to computer hardware, or cost constraints relating to including sensors or additional computer hardware in each vehicle. In such scenarios, taking into account the constraints of the fleet of vehicles, the evaluation module 230 can identify the most optimal motion prediction model for those vehicles.

[0050] As one example, a fleet of trucks may require a motion prediction model with relatively low computational requirements for typical highway driving, involving limited stops near highways (e.g., distribution centers). The evaluation module 230 can identify the most optimal motion prediction model for these identified objectives and constraints. As another example, a fleet of vehicles may have sufficient computer hardware to run any motion prediction model and may require a near-perfect motion prediction model for urban environments with frequent high-density VRU traffic. The evaluation module 230 can accordingly provide the most optimal motion prediction model for these constraints and objectives.

[0051] Incorporating lane matching into the evaluation of motion prediction models is intended to provide more useful accuracy determinations than, for example, evaluations based solely on Euclidean space. A further explanation of the advantages of lane-based evaluation techniques is provided below with reference to Figures 3A and 3B.

[0052] Evaluation of Euclid-based systems

[0053] Figure 3A shows the use of a Euclidean-based distance metric in evaluating a motion prediction model, as illustrated in the examples described herein. In the prior art example shown in Figure 3A, a vehicle 302 running a motion prediction model detects an adjacent entity 305 (e.g., another vehicle) traveling along a road section 300 on which the vehicle 302 is traveling. The detected entity 305 has a current ground truth trajectory 310 that includes the travel path along the road section 300 that the entity 305 has already traversed. Through the execution of the motion prediction model, the vehicle 302 can dynamically determine the predicted trajectory 315 of the entity 305. Thus, at any given time, the vehicle 302 can predict the future trajectory of the entity 305 as well as the predicted endpoint 325 of the entity 305 at a future time (e.g., 5 seconds later).

[0054] Following a Euclidean-based evaluation technique, the evaluation method forms a geometric threshold surrounding the error tolerance region 320 (e.g., a circle as shown in Figure 3A). As the vehicle 302 and entity 305 continue traveling along their respective paths, if the ground truth future position of entity 305 (e.g., the actual position of entity 305 5 seconds after the prediction) is within the error tolerance region 320, the trajectory prediction is counted as an accuracy hit. If the ground truth future position of entity 305 is outside the tolerance region 320, the prediction is treated as an accuracy miss. Thus, the evaluation of the motion prediction model occurs for all detected entities for which trajectory predictions are made by the vehicle 302 along the course of the travel route. In this specification, it is intended that the lateral error tolerance range of the error tolerance region 320 may be unacceptably large, potentially resulting in an accuracy hit for trajectories that could eventually enter the opposite lane. In such cases, a motion prediction model with unacceptably large lateral error predictions may be evaluated as highly accurate using Euclidean-based techniques, and as a result, may prove to be a suboptimal motion prediction model for use in road networks.

[0055] Lane-based evaluation

[0056] Figure 3B shows the use of a lane-based distance metric for evaluating a motion prediction model, as illustrated in the examples described herein. As shown in the figure of Figure 3B, the longitudinal error tolerance in the error tolerance region 365 is substantially preserved, while the lateral error tolerance is significantly reduced. Referring to Figure 3B, a vehicle 352 travels along a road section 350 and detects an external entity 355 currently traveling along a ground truth path 360. In the example shown in Figure 3B, entity 355 may include another vehicle traveling along the same road section. In further examples, vehicle 352 may detect any number of external entities (e.g., other vehicles and / or VRUs) at any given time, and motion prediction can be performed for each detected entity. Vehicle 352 can run a motion prediction model and predict the trajectory 366 of entity 355 and the endpoint 375 of the predicted trajectory 366 at a specified future time (e.g., 5 seconds later).

[0057] As described with respect to Figure 2, in accordance with the examples described herein, the lane matching module 220 of the computing system 200 can perform assignment confidence calculations and match predicted trajectories and ground truth trajectories to one or more underlying lanes. The evaluation module 230 of the computing system 200 can utilize the predicted trajectories or endpoints (one or more) and matched lane divisions (one or more) for the ground truth trajectories or endpoints to generate an error tolerance area 365 based on the underlying lane topology. Thus, if the ground truth future position of entity 355 falls within the error tolerance area 365, the evaluation module 230 counts the prediction as a hit. Conversely, if the ground truth future position of entity 355 falls outside the error tolerance area 365, the evaluation module 230 records the trajectory prediction as a miss.

[0058] It is intended that all trajectory predictions by motion prediction models can be evaluated in the manner described herein, and that a complete set of accuracy scores can be determined for each motion prediction model. As shown in Figure 3B, by matching the predicted trajectory to the underlying lane for evaluation purposes, the lateral error tolerance is significantly reduced, while the longitudinal tolerance is substantially maintained (for example, because longitudinal accuracy is less important for safety). Thus, the computing system 200 can output accuracy scores for motion prediction models that are more beneficial for road network safety in the field of autonomous vehicle operation.

[0059] methodology

[0060] Figures 4 and 5 are flowcharts illustrating exemplary methods for evaluating motion prediction models, as described herein. In the following description of Figures 4 and 5, reference numerals may be used to refer to various features illustrated and described with respect to Figures 1 and 2. Furthermore, the processes described in relation to Figures 4 and 5 may be carried out by an exemplary computing system 200, such as the one described with respect to Figure 2. Moreover, certain steps described with respect to the flowcharts of Figures 4 and 5 may be performed before, simultaneously with, or after any other step, and do not necessarily have to be performed in each illustrated sequence.

[0061] Referring to Figure 4, in block 400, the computing system 200 can receive motion prediction data from a vehicle 285 running a motion prediction model (for example, for autonomous driving purposes). As described herein, the motion prediction data may indicate the trajectory and / or endpoint predictions of external entities of the vehicle 285. These entities may include other vehicles, VRUs, and other moving objects. For example, a motion prediction model running on the vehicle 285 can detect external entities, determine the current direction and speed of the external entities, and predict each of the external entities' trajectories and where the entities will be located at a future point in time (e.g., 5 seconds later). In block 405, the computing system 200 can match the predicted trajectories and ground truth trajectories of the external entities to the underlying lanes of the road sections through which the vehicle will travel. In particular, the endpoint(s) of the predicted trajectories and ground truth trajectories may be assigned to one or more underlying lanes, which can define an error tolerance for the motion prediction model.

[0062] In block 410, the computing system 200 can evaluate the motion prediction model based on the predicted trajectory and the underlying lane assigned to the ground truth trajectory. In evaluation, the computing system 200 can determine in decision block 415 whether the future ground truth position of the entity is within the error tolerance range (hit) or outside the error tolerance range (miss). Based on the evaluation of the motion prediction model, the computing system 200 can generate a set of accuracy scores for the motion prediction model in block 420 for the predicted trajectories of any number of external entities detected by the vehicle 285. As described throughout this disclosure, the evaluation method described herein substantially maintains the longitudinal error tolerance range in the direction of travel (e.g., the direction of the underlying lane) while reducing the lateral error tolerance range.

[0063] Figure 5 is another flowchart illustrating a method for evaluating a motion prediction model according to the examples described herein. Referring to Figure 5, in block 500, the computing system 200 can receive motion prediction data and recorded sensor data from a vehicle 285 running a motion prediction model to navigate and / or operate autonomously across the entire road network. As provided herein, the sensor data may include any combination of image data from one or more onboard cameras, LiDAR data, radar data, ultrasonic sensor data, etc. In block 505, the computing system 200 can determine the underlying lane topology of the road section through which the vehicle 285 is traveling. In block 510, for each predicted trajectory of each entity and the ground truth trajectory of each entity, the computing system 200 can calculate the confidence in assigning distance and direction to match the endpoint(s) of the trajectory to one or more underlying lanes.

[0064] In block 515, in various examples, the computing system 200 can generate error tolerance thresholds based on the assigned lanes (one or more) for each of the predicted trajectories. As described herein, these thresholds can be scaled based on the determined speed of an external entity. For example, the longitudinal error tolerance can be increased for vehicles moving at higher speeds. In block 520, the computing system 200 can evaluate the prediction accuracy of each motion prediction model based on the lane matching technique described herein to determine the prediction performance of the motion prediction models, and generate a set of accuracy scores for each motion prediction model. As described above, the accuracy scores can be based on the hit rate of predicted endpoints within the error tolerance region coupled to each predicted trajectory.

[0065] In a particular implementation, in block 525, the computing system 200 can determine a set of constraints for a vehicle fleet requiring a motion prediction model for autonomous or semi-autonomous driving. In block 527, the set of constraints may correspond to computational constraints (e.g., computing power limits) based on the computer hardware installed in each vehicle of the fleet. In block 529, the set of constraints may correspond to constraints on the purpose of the vehicle fleet. For example, the fleet may be used primarily for urban driving, highway driving, rural driving, delivery, or human transport. In a particular example, in block 530, the computing system 200 may use the accuracy score of the motion prediction model to determine the optimal motion prediction model for the vehicle fleet based on the set of constraints and output the optimal allocation of motion prediction models to the vehicle fleet, or otherwise easily do so.

[0066] It is intended that the examples described herein be extended to the individual elements and concepts described herein, independently of other concepts, ideas, or systems, and that combinations of elements described in any part of this application be included as examples. While examples are described in detail herein with reference to the accompanying drawings, it should be understood that the concepts are not limited to those exact examples. Therefore, many modifications and variations will be apparent to those skilled in the art. Accordingly, the scope of the concepts is intended to be defined by the following claims and their equivalents. Furthermore, specific features described individually or as part of an example are intended to be combined with other features or parts of other examples described individually, even if other features and examples do not refer to those specific features.

Claims

1. A computing system, One or more processors, It includes a memory for storing instructions, When the instruction is executed by one or more processors, it is transmitted to the computing system. When a vehicle operates on a road network, motion prediction data generated by a motion prediction model running on the vehicle is received from the vehicle. Based on the motion prediction data, determine the predicted set of trajectories for each of the multiple entities in the surrounding environment of the vehicle. The lane topology that forms the basis of the road section in which the vehicle operates is determined. (i) For each individual entity of the plurality of entities, the predicted endpoint of each predicted trajectory in the set of predicted trajectories matches one or more underlying lanes in the underlying lane topology of the road section; (ii) The ground truth future position of the individual entity matches one or more underlying lanes in the underlying lane topology; and (iii) The distance along one or more lane sections between the one or more underlying lanes that match the predicted endpoint and the one or more underlying lanes that match the ground truth future position is determined; thereby evaluating the predictive performance of the motion prediction model. Computing system.

2. The computing system according to claim 1, wherein the instruction to be executed causes the computing system to match the predicted endpoint of the set of predicted trajectories to the one or more underlying lanes by calculating an assignment confidence for each of the one or more underlying lanes based on the set of predicted trajectories.

3. The computing system according to claim 2, wherein the instruction to be executed causes the computing system to dynamically calculate the assignment confidence by dynamically calculating (i) a distance-based assignment confidence and (ii) an orientation-based assignment confidence for each lane division around the predicted endpoint, and one or more underlying lanes correspond to the highest assignment confidence based on the dynamic calculation of the distance-based assignment confidence and the orientation-based assignment confidence.

4. The instruction to be executed further to the computing system For each of the aforementioned multiple entities, recorded sensor data indicating the ground truth trajectory is received from the vehicle. Based on the ground truth trajectory for each of the plurality of entities, the future ground truth position of each of the plurality of entities is determined. The computing system according to claim 1.

5. The vehicle includes an autonomous vehicle, and the execution of the motion prediction model causes the autonomous vehicle to dynamically generate a motion plan based at least partially on the set of predicted trajectories for each of the plurality of entities in the surrounding environment of the autonomous vehicle. To execute the aforementioned motion plan, the set of control mechanisms of the autonomous vehicle is made to operate autonomously. The computing system according to claim 1.

6. The computing system according to claim 5, wherein the set of control mechanisms includes a plurality of the vehicle's braking system, the vehicle's steering system, the vehicle's acceleration system, or the vehicle's signal transmission system.

7. The computing system according to claim 1, wherein the instruction to be executed causes the computing system to evaluate the predictive performance of each of a plurality of motion prediction models in order to determine the most accurate motion prediction model for execution in an autonomous vehicle.

8. The computing system according to claim 1, wherein the plurality of entities include other vehicles operating along the road division.

9. A non-temporary computer-readable medium for storing instructions, wherein when an instruction is executed by one or more processors of a computing system, the computing system... When a vehicle operates on a road network, motion prediction data generated by a motion prediction model running on the vehicle is received from the vehicle. Based on the motion prediction data, determine the predicted set of trajectories for each of the multiple entities in the surrounding environment of the vehicle. The lane topology that forms the basis of the road section in which the vehicle operates is determined. (i) For each individual entity of the plurality of entities, the predicted endpoint of each predicted trajectory in the set of predicted trajectories is matched to one or more underlying lanes in the underlying lane topology of the road section; (ii) The future ground truth position of the individual entity is matched to one or more underlying lanes in the underlying lane topology; and (iii) The distance along one or more lane sections between the one or more underlying lanes that match the predicted endpoint and the one or more underlying lanes that match the future ground truth position is determined to evaluate the predictive performance of the motion prediction model. Non-temporary computer-readable media.

10. The non-temporary computer-readable medium according to claim 9, wherein the instruction to be executed causes the computing system to match the predicted endpoint of the set of predicted trajectories to the one or more underlying lanes by calculating an assignment confidence for each of the one or more underlying lanes based on the set of predicted trajectories.

11. The non-temporary computer-readable medium according to claim 10, wherein the instruction to be executed causes the computing system to dynamically calculate the assignment confidence by dynamically calculating (i) a distance-based assignment confidence and (ii) an orientation-based assignment confidence for each lane division around the predicted endpoint, and one or more underlying lanes correspond to the highest assignment confidence based on the dynamic calculation of the distance-based assignment confidence and the orientation-based assignment confidence.

12. The instruction to be executed further to the computing system For each of the aforementioned multiple entities, recorded sensor data indicating the ground truth trajectory is received from the vehicle. Based on the ground truth trajectory for each of the plurality of entities, the future ground truth position of each of the plurality of entities is determined. The non-temporary computer-readable medium according to claim 9.

13. The vehicle includes an autonomous vehicle, and the execution of the motion prediction model causes the autonomous vehicle to dynamically generate a motion plan based at least in part on the set of predicted trajectories for each of the plurality of entities in the surrounding environment of the autonomous vehicle. To execute the aforementioned motion plan, the set of control mechanisms of the autonomous vehicle is made to operate autonomously. The non-temporary computer-readable medium according to claim 9.

14. The non-temporary computer-readable medium according to claim 5, wherein the set of control mechanisms includes a plurality of the vehicle's braking system, the vehicle's steering system, the vehicle's acceleration system, or the vehicle's signal transmission system.

15. The non-temporary computer-readable medium according to claim 9, wherein the instruction to be executed causes the computing system to evaluate the predictive performance of each of a plurality of motion prediction models in order to determine the most accurate motion prediction model for execution in an autonomous vehicle.

16. The non-temporary computer-readable medium according to claim 9, wherein the plurality of entities include other vehicles operating along the road division.

17. A computer implementation method, Receiving motion prediction data from the vehicle, which is generated by a motion prediction model running on the vehicle when the vehicle is operating in the road network, Based on the motion prediction data, determine the predicted set of trajectories for each of the multiple entities in the surrounding environment of the vehicle. Determining the lane topology that forms the basis of the road section in which the vehicle operates, (i) For each individual entity of the plurality of entities, the predicted endpoint of each predicted trajectory in the set of predicted trajectories coincides with one or more underlying lanes in the underlying lane topology of the road section; (ii) The future ground truth position of the individual entity coincides with one or more underlying lanes in the underlying lane topology; and (iii) The distance along one or more lane sections between the one or more underlying lanes that coincide with the predicted endpoint and the one or more underlying lanes that coincide with the future ground truth position, To evaluate the predictive performance of the aforementioned motion prediction model, Methods that include...

18. The method according to claim 17, wherein the instruction to be executed causes a computing system to match the predicted endpoint of the set of predicted trajectories to the one or more underlying lanes by calculating an assignment confidence for each of the one or more underlying lanes based on the set of predicted trajectories.

19. One or more processors dynamically calculate the assignment confidence by dynamically calculating (i) distance-based assignment confidence and (ii) direction-based assignment confidence for each lane segment surrounding the predicted endpoint, The method according to claim 18, wherein the one or more underlying lanes correspond to the highest assignment confidence based on the dynamic calculation of the distance-based assignment confidence and the direction-based assignment confidence.

20. The vehicle receives recorded sensor data indicating the ground truth trajectory for each of the aforementioned multiple entities, Based on the ground truth trajectory for each of the plurality of entities, the future ground truth position of each of the plurality of entities is determined. The method according to claim 17, further comprising:

Images