Performance testing of robotic systems

Abstract

A "Perception Statistical Performance Model" (PSPM) that models a perception slice of the runtime stack of an autonomous vehicle or other robotic system may be used herein, for example, for safety / performance testing. The PSPM is configured to: receive a computed perception ground truth and determine a probabilistic perception uncertainty distribution from the perception ground truth based on a set of learned parameters, the parameters being learned from a set of actual perception outputs generated using the perception slice to be modeled. Simulation scenarios are run based on a time series of such perception outputs (with modeled perception errors), but simulation scenarios may also be re-run directly based on the perception ground truth (without perception errors). This can, for example, be used to determine whether a perception error is the cause of an unexpected decision within the planner by determining whether such a decision was also made in a simulation scenario when the perception error was "turned off."

Description

Technical Field

[0001] The present disclosure relates to performance testing of autonomous vehicles and other robotic systems. Performance testing is critical to ensuring that such systems can operate with a guaranteed level of safety. Background Art

[0002] It is estimated that in order for an autonomous vehicle (AV) to achieve a safety level that matches that of a human driver, a maximum of 1 error per 10^7 autonomous driving decisions must be ensured throughout the AV’s entire operational design domain (ODD).

[0003] This poses a huge challenge given the inherent complexity of AVs and ODDs. A self-driving system is an incredibly complex assembly of cross-dependent and interacting software and hardware components, each susceptible to limitations or errors. Several components use neural networks for object detection, type classification, action prediction, and other critical tasks. The system needs to operate safely in the ODD. In this case, the ODD describes all possible driving scenarios that an AV may encounter, and is therefore inherently infinite in possibilities, with variables including road topologies, users, appearance, lighting, weather, behavior, seasons, speed, randomness, and intentional behavior.

[0004] The industry standard approach to safety testing is based on real-world driving test miles. The fleet of autonomous vehicles is driven by test drivers, and decisions are characterized as unsafe when test driver intervention is required. Once an instance of test driver intervention occurs in a specific real-world driving scenario, the circumstances of that driving scenario can be explored to isolate the factors that caused the AV behavior to be unsafe and take appropriate mitigation actions. Summary of the invention

[0005] Simulation has been used for safety testing, but is only useful if the simulated scenarios are realistic enough (if an AV planner makes unsafe decisions in a completely unrealistic simulated scenario, then that is far less useful in a safety testing context than providing examples of unsafe behavior in real-world scenarios).

[0006] One approach is to run simulations based on real-world scenarios that require test driver intervention. Sensor outputs from the AV are collected and can be used to recreate driving scenarios in the simulator that require test driver intervention. The variables of the scenario can be "fuzzified" at the planning level in order to test variations of real-world scenarios that are still realistic. In this way, more information about the causes of unsafe behaviors can be obtained, analyzed, and used to improve prediction and planning models. However, a significant problem arises because as the number of errors per decision decreases, the number of test miles that need to be driven to find a sufficient number of unsafe behavior instances increases. A typical AV planner may make about 1 decision every two seconds on average. At an average speed of 20 miles per hour, this equates to about 90 decisions per mile driven. This in turn implies less than one error per 10^5 miles driven in order to match human safety levels. Robust safety testing requires multiple tests to adequately test the AV across the AV's ODD. This situation worsens as the perception stack evolves, as more test miles are required with each change in the perception stack. For these reasons, this approach is simply not feasible when testing at safety levels close to humans.

[0007] There are other problems with existing simulation methods.

[0008] One approach is planning-level simulation, but this cannot fully account for the impact of perception errors. Many factors can affect perception errors, such as weather, lighting, distance to another vehicle or the speed of another vehicle, occlusion, etc.

[0009] The alternative would be full “realistic” simulation, where the entire hardware and software stack of the AV is simulated. However, this is a huge challenge in itself. The AV perception pipeline is usually composed of multiple perception components that work together to interpret the AV’s sensor output.

[0010] One problem is that certain perception components, such as Convolutional Neural Networks (CNNs), are particularly sensitive to the quality of simulated data. Although high-quality simulated image data can be generated, CNNs in perception are extremely sensitive to even small deviations from real data. Therefore, these will require exceptionally high-quality simulated image data that covers all possible conditions that an AV may encounter in the real world (e.g., simulated weather conditions, different combinations of lighting conditions, etc.) - otherwise their behavior in simulated scenarios will not fully reflect their behavior in the real world.

[0011] The second problem is that some types of sensor data are particularly difficult to model (simulate). As a result, even perception systems that are not particularly sensitive to the quality of their input data can give poor results. For example, radar belongs to the category of sensor data that is extremely difficult to simulate. This is because the physics of radar is inherently difficult to model.

[0012] The third overriding issue is computational efficiency. Given current hardware constraints, it is estimated that it is possible to achieve realistic simulations in real time as close to real time as possible (even if other issues can be overcome).

[0013] The present disclosure provides a radically different approach to simulation-based safety testing using a model referred to herein as a "Perceptual Statistical Performance Model" (PSPM). The core problem addressed in the present disclosure is to simulate realistic perceptual outputs - i.e., perceptual outputs with realistic errors - in a manner that is not only more robust than photorealistic simulations, but also significantly more efficient.

[0014] The PSPM models perceptual errors in terms of probabilistic uncertainty distributions based on a robust statistical analysis of the actual perceptual outputs computed by the one or more perceptual components being modeled. A unique aspect of the PSPM is that, given a perceptual ground truth (i.e., a "perfect" perceptual output that would be computed by a perfect but unrealistic perceptual component), the PSPM provides a probabilistic uncertainty distribution that represents the realistic perceptual components that can be provided by the perceptual components it is modeling. For example, a PSPM modeling a PSPM of a simulated 3D bounding box detector will provide an uncertainty distribution that represents realistic 3D object detection outputs, given a ground truth 3D bounding box. Even when the perceptual system is deterministic, it can be effectively modeled as stochastic to account for the epistemic uncertainty of its many hidden variables that depend on practice.

[0015] Of course, perception ground truth will not be available at runtime in a real-world AV (which is why complex perception components are needed to reliably interpret imperfect sensor outputs). However, perception ground truth can be derived directly from simulated scenes running in a simulator. For example, given a 3D simulation of a driving scene with an ego vehicle (the simulated AV being tested) in the presence of external actors, ground truth 3D bounding boxes can be computed directly from the simulated scene of the external actors based on their size and pose (position and orientation) relative to the ego vehicle. The PSPM can then be used to derive realistic 3D bounding object detection outputs from these ground truths, which in turn can be processed by the rest of the AV stack just as they would at runtime.

[0016] A PSPM that models a perception slice of the runtime stack of an autonomous vehicle or other robotic system may be used, for example, for safety / performance testing. The PSPM is configured to receive a computed perception ground truth and determine a probabilistic perception uncertainty distribution from the perception ground truth based on a set of learned parameters, the parameters being learned from a set of actual perception outputs generated using the perception slice to be modeled. Simulation scenarios are run based on a time series of such perception outputs (with modeled perception errors), but simulation scenarios may also be re-run directly based on the perception ground truth (without perception errors). This can, for example, be used to determine if a perception error is the cause of some unexpected decision within the planner by determining if such a decision was also made in a simulation scenario when the perception error was "turned off."

[0017] A first aspect of the present invention provides a computer system for testing and / or training a runtime stack of a robotic system, the computer system comprising:

[0018] a simulator configured to run a simulation scenario in which a simulated subject interacts with one or more external objects;

[0019] a planner of the runtime stack, the planner of the runtime stack configured to make autonomous decisions for each simulation scenario based on a time series of perception outputs computed for the simulation scenario; and

[0020] a controller of the runtime stack, the controller of the runtime stack being configured to generate a series of control signals to cause the simulated agent to perform autonomous decisions as the simulated scenario progresses;

[0021] The computer system is configured to calculate each sensory output by:

[0022] Compute perception ground truth based on the current state of the simulated scene;

[0023] applying a perceptual statistical performance model (PSPM) to the perceptual ground truth to determine a probabilistic perceptual uncertainty distribution; and

[0024] sampling perceptual outputs from a probabilistic perceptual uncertainty distribution;

[0025] wherein the PSPM is used to model a perceptual slice of the runtime stack and is configured to determine a probabilistic perceptual uncertainty distribution based on a set of parameters learned from a set of actual perceptual outputs generated using the perceptual slice to be modeled;

[0026] Therein, the computer system is configured to rerun a simulation scenario based on a time series of perceptual ground truths determined for the rerun scenario without applying the PSPM to those perceptual ground truths and thus without perceptual errors.

[0027] The computer system may include a scenario evaluation component configured to evaluate the behavior of the external agent in each simulation scenario by applying a predetermined set of rules.

[0028] At least some of the predetermined rules are related to safety, and the scenario evaluation component may be configured to evaluate the safety of the subject's behavior in each simulated scenario.

[0029] The subject may initially exhibit unsafe behavior in the scenario, and in that event the scenario may be rerun to assess whether the subject still exhibits unsafe behavior in the rerun scenario.

[0030] The assessment component may be configured to automatically flag instances of unsafe behavior by the subject for further analysis and testing.

[0031] The computer system may be configured to record details of each simulation scenario in a test database, wherein the details include decisions made by the planner, the sensory outputs on which those decisions were based, and the behavior of the simulated agent in executing those decisions.

[0032] The PSPM may be applied to the perceived ground truth and one or more confounding factors associated with the simulation scenario, each confounding factor being a variable of the PSPM, the value of the variable characterizing the physical conditions applicable to the simulation scenario, and the probabilistic perception uncertainty distribution depending on the variable.

[0033] The one or more confounding factors may include one or more of the following confounding factors, the confounding factor at least partially determining the probabilistic uncertainty distribution from which the sensory output is sampled:

[0034] an occlusion level of at least one of the external objects;

[0035] one or more lighting conditions;

[0036] an indication of the time of day;

[0037] one or more weather conditions;

[0038] Indications of the seasons;

[0039] a physical characteristic of at least one of the external objects;

[0040] sensor conditions, such as a location of at least one of the external objects in a sensor field of view of the subject;

[0041] the number or density of external objects;

[0042] the distance between two external objects;

[0043] a truncation level of at least one of the external objects;

[0044] the type of at least one of the objects, and

[0045] An indication of whether at least one of the external objects corresponds to any external object from an earlier time in the simulated scene.

[0046] The computer system can be configured to run simulation scenarios multiple times with different confounding factors and without perceptual error.

[0047] Sampling from a probabilistic perceptual uncertainty distribution can be non-uniform and biased towards lower-probability perceptual outputs.

[0048] The computer system may include a scene obfuscation component configured to generate at least one obfuscated scene for running in the simulator by obfuscating at least one existing scene.

[0049] To model false negative detections, a probabilistically aware uncertainty distribution may provide the probability of successfully detecting a visible object among the subjects, which is used to determine whether to provide an object detection output for the object, where the object is visible when within the subject's sensor field of view in a simulated scene, thereby not guaranteeing detection of the visible object.

[0050] Ray tracing may be used to compute perceptual ground truth for one or more external objects.

[0051] At least one of the external objects may be a moving actor, the computer system includes a prediction stack of the runtime stack configured to predict behavior of the external actor based on the sensory output, and the planner is configured to make autonomous decisions based on the predicted behavior.

[0052] The computer system may be configured to record details of each simulation scenario in a test database, wherein the details include decisions made by the planner, the sensory outputs on which those decisions were based, and the behavior of the simulated agent in executing those decisions.

[0053] A second aspect of the present invention provides a computer-implemented method for performance testing a runtime stack of a robot system, the method comprising:

[0054] running a simulation scenario in the simulator, wherein the simulated agent interacts with one or more external objects, wherein a planner of the runtime stack makes autonomous decisions for the simulation scenario based on a temporal sequence of sensory outputs computed for the simulation scenario, and wherein a controller of the runtime stack generates a sequence of control signals to cause the simulated agent to execute the autonomous decisions as the simulation scenario progresses;

[0055] Among them, each perception output is calculated by the following operations:

[0056] Compute perception ground truth based on the current state of the simulated scene;

[0057] applying a perceptual statistical performance model (PSPM) to the perceptual ground truth to determine a probabilistic perceptual uncertainty distribution; and

[0058] sampling perceptual outputs from a probabilistic perceptual uncertainty distribution;

[0059] wherein the PSPM is used to model a perceptual slice of the runtime stack and determine a probabilistic perceptual uncertainty distribution based on a set of parameters learned from a set of actual perceptual outputs generated using the perceptual slice to be modeled;

[0060] Therein, the simulated scenario is rerun based on the time series of perceptual ground truths determined for the rerun scenario without applying the PSPM to those perceptual ground truths and therefore without perceptual errors.

[0061] Another aspect of the present invention provides a computer program for programming one or more computers to implement any of the methods or functions described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] For a better understanding of the present disclosure, and to illustrate how embodiments of the present disclosure may be implemented, reference is made to the accompanying drawings, in which:

[0063] Figure 1 A schematic block diagram of an autonomous vehicle runtime stack is shown;

[0064] Figure 2 An example of a real-world driving scenario is shown;

[0065] Figure 3 shows a test pipeline using photorealistic simulation;

[0066] Figure 4 An alternative PSPM-based test pipeline according to the present disclosure is shown;

[0067] Figure 5 shows how perceived performance is affected by confounding factors;

[0068] Figure 6 Provides a high-level overview of some of the principles of PSPM-based security testing;

[0069] Figure 7 shows a perceptual error dataset that can be used to train PSPM;

[0070] Fig. 7A Shown is applied to Figure 7 Results of the trained PSPM on the perceptual error dataset;

[0071] Figure 8 The engineering pipeline incorporated into the PSPM is shown;

[0072] Fig. 9 An example of a perception stack is shown;

[0073] Figure 9A-9C shows that one or more PSPMs can be used to model Fig. 9 Different ways of perceiving the stack;

[0074] Fig.10 A schematic overview of factors that can contribute to perceived uncertainty is provided;

[0075] Fig.11 An example of analog image data is shown, to which some forms of perceptual components are highly sensitive;

[0076] Fig.12 and Fig.13 A bird's eye view and a driver's view of a roundabout scene are shown;

[0077] Fig.14 The stereo imaging geometry is schematically depicted;

[0078] Fig.15 An example time series of additive error of the position component is shown;

[0079] Fig.16 shows a hysteresis plot of the position error;

[0080] Fig.17 A graphical representation of a time-dependent position error model is shown;

[0081] Fig.18 An example binning scheme showing the confounding factors azimuth and distance;

[0082] Fig.19 shows a hysteresis plot of the position error increment;

[0083] Fig. 20 A histogram showing position error increments for the X, Y, and Z components;

[0084] Fig.21 The PDF fitting position error increments for the X, Y, and Z components are shown;

[0085] Fig. 22 shows an example mean of the error delta distribution in the training data (based on tracking a single object);

[0086] Fig.23 A time series plot of the true perceived error and the simulated error is shown;

[0087] Fig.24 Hysteresis plots for true perceived error versus simulated error are shown;

[0088] Fig.25 The relative importance of certain confounders in a particular (left to right) context for the target association state as determined by MultiSURF Relief analysis is graphically depicted;

[0089] Fig.26 The relative importance of confounders (from left to right) for target metastasis as determined by MultiSURF Relief analysis is graphically depicted;

[0090] Fig. 27 An example node in a neural network is shown;

[0091] Fig.28 shows a high-level overview of the convolutional neural network architecture;

[0092] Fig.29 It is shown that the PSPM is implemented as a neural network during training and inference;

[0093] Fig.30 A neural network PSPM is shown with one or more confounding factor inputs at the input layer;

[0094] Fig.31 An example of a time-dependent neural network architecture is shown;

[0095] Fig.32 A "set-to-set" PSPM implemented as a neural network is shown;

[0096] Fig.33A Schematic depiction of spatial encoding of perceptual outputs that facilitate processing in a convolutional neural network (CNN);

[0097] Fig.33B Schematically depicts the training stages of CNN PSPM;

[0098] Fig.33C Schematic depiction of a trained CNN PSPM at inference time;

[0099] Fig.33D shows how a CNN PSPM can be architected to encode a perceptual output distribution in an output tensor from which realistic perceptual outputs can be sampled; and

[0100] Fig.34It is shown how the PSPM can be configured to model perceptual slices including an online error estimation component. DETAILED DESCRIPTION

[0101] 1. Overview

[0102] The terms "PSPM" and "PRISM" are used interchangeably in the following description.

[0103] When making a safety case for an autonomous vehicle, it is impractical to perform all the required testing in the real world. However, building simulations with such high fidelity that the vehicle's perception system performs equivalently for real and simulated data is an unsolved problem. The method referred to herein as "PRISM" solves this problem by building a surrogate model of a perception system, which includes both sensors and (one or more) perception components that interpret the sensor data captured by the sensors. PRISM is a distribution of credible perception outputs given some low-fidelity scene representation (perception ground truth).

[0104] Expanding on the above, ensuring driverless technology is provably safe requires testing it in a large number of situations. This testing with real cars is expensive and time consuming. In natural scenarios, the majority of miles driven will be uneventful – in 2016, 136,621 people were injured and 1,792 died in UK road accidents, and all motor vehicles travelled 323.7 billion miles, with just one accident for every 2.4 million miles. Simulation must be part of the driverless technology testing strategy. Simulated miles are much cheaper than real miles, and it is easier and safer to increase the number of hazards per mile in simulation than in the real world.

[0105] One approach to generating realistic perception outputs is via a high-fidelity simulation of the world, including sensor measurements. In this approach, realistic sensor readings are generated and fed to the car's software in place of real sensor readings, e.g., a realistic twin of the real world rendered as an image for perception input. Such renderings, such as Fig.11 The vehicle software outputs control signals for the vehicle actuators, which are fed into the physics simulation. New sensor readings are generated based on the output of the physics simulation, closing the loop. This approach requires accurate models for tasks that range from challenging to unsolved:

[0106] Simulation of road surfaces, vehicle dynamics and other physics is possible with current technology but is not well understood.

[0107] • It is possible to simulate GPS, IMU and wheel-encoding, but it is important to get their error statistics correct.

[0108] Visual appearance, camera lenses, and image sensor modeling are reasonably understood, but high-fidelity rendering is slow.

[0109] Lidar modelling is similar to camera modelling, but with different material reflectance properties. The scanning nature of Lidar is an additional challenge.

[0110] • Radar returns are difficult to model accurately with current technology due to the difficulty in modeling the relevant material properties, detailed dependence on shape and multiple reflections.

[0111] Worst of all, state-of-the-art neural networks for visual object detection are extremely sensitive to detailed image statistics, and constructing synthetic images that elicit the same network behavior as equivalent real images is an unsolved problem.

[0112] Inaccurate models of the sensors described above will affect the output of the perception modules in the simulation, resulting in potentially different ego behavior. Such differences in behavior limit the usefulness of these simulations in evaluating real-world performance. Furthermore, running the many miles of photorealistic simulations required to validate the safe behavior of autonomous vehicles is expensive. This is because rendering photorealistic scenes is a slow, computationally intensive task that requires a GPU. High-fidelity simulations are difficult and expensive, and conclusions from testing using high-fidelity simulations are unlikely to generalize to the real world.

[0113] Figure 1A data flow diagram through a decomposed autonomous vehicle stack 100 is shown. A perception system 102 receives sensor readings from the world and outputs a scene representation. A planning and prediction system (represented by reference numerals 104 and 106, respectively) takes the scene representation and plans a trajectory through the scene. A control system 108 outputs control signals to the world that will cause the vehicle to follow the trajectory.

[0114] The perception system 102, planning and prediction systems 104, 106, and control system 108 communicate with each other using well-defined interfaces. The perception system 102 uses raw sensor data and processes the raw sensor data into a more abstract scene representation. The representation includes dynamic object pose, extent, motion, and detection confidence. The planning and prediction system predicts the possible trajectories of other agents in the scene and plans a safe, legal, and comfortable path through the scene. The control system uses the desired trajectory from the planning and prediction system and outputs control signals for actuators.

[0115] In many cases, especially in the case of the interface between perception and planning, these internal interfaces are easier to simulate than sensor readings. These interfaces can be used for a second type of simulation known as low-fidelity simulation. It is possible to simulate only those aspects of the world that are necessary to reconstruct the abstract scene representation used by the planner, and provide that abstract scene representation directly to the planner, taking the perception system out of the loop. While this avoids some of the burden of high-fidelity simulation, it presents a new challenge: replicating the behavior of the perception system. It is well known that perception systems are not perfect, and that errors in perception systems can affect prediction, planning, and control systems in meaningful ways. Because the results tested in simulation should be generalized to the real world, it is necessary to be able to simulate realistic perception outputs.

[0116] We propose a method for simulating realistic perceptual outputs using a model we call PRISM. PRISM is a distribution of plausible perception outputs given some low-fidelity scene representation. We outline the mathematical framework that guided the creation of PRISM, create a prototype, and document the modelling choices. We do this by showing that the modelling approach is sound.

[0117] In summary, in a high-fidelity simulation, the world is replaced by a simulator and the entire vehicle stack is treated as a black box. In a low-fidelity simulation, the world and the perception system 102 are replaced (see Figure 4 and described below).

[0118] Figure 1 A highly schematic block diagram of a runtime stack 100 for an autonomous vehicle (AV) is shown. The runtime stack 100 is shown to include a perception stack 102, a prediction stack 104, a planner 106, and a controller 108.

[0119] The perception stack 102 receives sensor output from an on-board sensor system 110 of the AV.

[0120] The onboard sensor system 110 may take different forms, but typically includes a variety of sensors, such as image capture devices (cameras / optical sensors), LiDAR and / or RADAR units, satellite positioning sensors (GPS, etc.), motion sensors (accelerometers, gyroscopes, etc.), etc., which together provide rich sensor data from which detailed information about the surrounding environment and the status of the AV and any external actors (vehicles, pedestrians, cyclists, etc.) within the environment can be extracted.

[0121] Thus, the sensor output typically includes sensor data of multiple sensor modalities, such as stereo images from one or more stereo optical sensors, lidar, radar, etc.

[0122] The perception stack 102 includes a plurality of perception components that cooperate to interpret sensor outputs and thereby provide perception outputs to the prediction stack 104 .

[0123] The perception output from the perception stack 102 is used by the prediction stack 104 to predict the future behavior of external actors.

[0124] The predictions computed by the prediction stack 104 are provided to the planner 106 , which uses the predictions to make autonomous driving decisions to be performed by the AV in a manner that takes into account the predicted behavior of external actors.

[0125] The controller 108 implements the decisions made by the planner 106 by providing appropriate control signals to the AV's onboard motors 112. In particular, the planner 106 plans maneuvers to be taken by the AV, and the controller 108 generates control signals in order to execute those maneuvers.

[0126] Figure 2 Examples of certain perception components that may form part of the perception stack 102 are shown, namely a 3D object detector 204 and a Kalman filter 206 .

[0127] The depth estimator 202 captures stereo image pairs and applies stereo imaging (such as Semi-Global Matching) to extract depth estimates from the stereo image pairs. Each depth estimate is in the form of a depth map that assigns depth values ​​to pixels of one image of a stereo image pair derived from the depth map (the other image is used as a reference). The depth estimator 202 includes a stereo pair of optical sensors and a stereo processing component (hardware and / or software), which are not shown separately in the figure. According to the terminology used in this document, both the optical sensors and the stereo processing component of the depth estimator 202 are considered to be part of the on-board sensor system 110 (not the perception stack 102). The depth map is a form of sensor output provided to the perception stack 102.

[0128] The 3D object detector 204 receives the depth estimates and uses them to estimate the pose of external actors near the AV (ego vehicle). Two such external actors are shown in the form of two other vehicles. The pose in this context refers to the 6D pose, i.e., (x, y, z, pitch, roll, yaw), representing the position and orientation of each external actor in 3D space.

[0129] For illustration purposes, Figure 2is highly simplified. For example, a 3D object detector may be formed from multiple cooperating sensing components that collectively operate on sensor outputs of multiple sensor modalities. Application of the PSPM to more complex stacks will be described later. For now, to illustrate some of the core principles of the PSPM, consider a simplified example in which the 3D object detector is assumed to operate on sensor outputs of a single modality (stereo depth).

[0130] In real-world scenarios, multiple physical conditions can affect the performance of the perception stack 102. As noted, physical conditions that are considered variable with respect to a particular PSPM are referred to as "confounders." This allows for consideration of variable physical conditions that are statistically correlated with a particular perception slice.

[0131] As mentioned above, one approach to simulation would be to try not only Figure 1 The entire runtime stack 100 of the vehicle is realistically simulated, and an attempt is also made to realistically simulate the onboard sensor system 110 and the motor 112. This is Figure 3 The challenge in this scenario is the simulation of sensor data: some types of sensor data (e.g., RADAR) are inherently difficult to simulate well, while other types of sensor data (image data) are relatively easier to simulate, and some perception components (such as CNN) are highly sensitive to even small deviations from actual sensor data. Another challenge is the large amount of computing resources required to simulate sensors and run complex perception components (such as CNN).

[0132] For example, for Figure 3 For an arrangement such as the one described above, it would be necessary to simulate very high quality depth maps and to run the 3D object detector 204 on those simulated depth maps. During simulation, even very small deviations in the simulated depth maps (compared to the true depth maps provided by the stereo depth estimator 202) would significantly affect the performance of the 3D object detector 204.

[0133] Figure 4 A high-level schematic overview of a PSPM-based simulation is provided. In this case, a "headless" simulator setup is used, no simulated sensor data (e.g., simulated images, depth maps, lidar and / or radar measurements, etc.) needs to be created, and no perception stack 102 needs to be applied (or at least not in its entirety - see below). Instead, one or more PSPMs are used to efficiently compute realistic perception outputs, which in turn are fed into high-level components of the runtime stack 100 and processed as they would be at runtime.

[0134] The PSPM is said to model a "perceptual slice," which may be all or a portion of the perceptual stack 102. A perceptual slice may be a single perceptual component or multiple cooperating perceptual components.

[0135] Mathematically, a perceptual slice can be represented as a function F, where:

[0136] e=F(x),

[0137] e is the sensory output of the sensor slice, and x is the set of sensor outputs operating on the sensor component.

[0138] On the AV at runtime, e is determined by applying F to x, which in turn is given by the sensor.

[0139] The PSPM mapped to the confounder space C can be expressed as a function p, where:

[0140] p(e|t,c) represents a probabilistic uncertainty distribution that provides the probability of F computing the perceptual output e given the perceptual ground truth t and a set of one or more confounding factors c (i.e., given a particular set of possible real-world conditions represented by a point c in the confounding space C).

[0141] For example, for 2D bounding box detection:

[0142] F can be CNN

[0143] x can be an RGB image

[0144] t can be a ground truth bounding box that can be computed directly from the simulation using ray tracing (no need to simulate x and no need to apply F), or a collection of multiple such bounding boxes for multiple ground truth objects (a collection-to-collection approach)

[0145] c could be distance and / or weather etc.

[0146] exist Figure 3 In the example of , e represents one or more 6D pose vectors (posevector) calculated by the 3D object detector 204, and x represents the depth map provided by the stereo depth estimator 202 from which e is derived.

[0147] Figure 5 shows how to use PSPM to simulate Figure 3The perceived output of the scene. In this case, the perceived slice is the 3D object detector 204.

[0148] "Realistic" in the present context refers to a sensory output that is more realistic than the sensory ground truth.

[0149] A PSPM 500 is provided that essentially models the perceptual slice 204 as a noisy "channel" affected by both the characteristics of the stereo depth estimator 202 and the physical environment. The physical environment is characterized by a set of confounding factors c, which in this example are: lighting, weather, occlusion, and distance to each external actor.

[0150] To apply the PSPM 500, the perception ground truth may be computed directly from the simulated scenario under consideration. For example, in a simulated scenario where a simulated AV (ego vehicle) has many external actors in its vicinity, the 6D pose ground truth may be determined by directly computing the 6D pose of each external actor in the ego vehicle's frame of reference.

[0151] The PSPM 500 then uses the computed ground truth t to compute the distribution p(e|t,c). Continuing with the above example, this will provide, for each simulated external actor, the probability that the actual 3D object detector 204 computed the perceived output e [estimated 3D pose of the external actor] given the perceived ground truth t ["actual" 6D pose] in the real world scene characterized by the same confounding factor c.

[0152] Once p(e|t,c) is calculated, it can be used to run multiple simulations over a range of realistic sensory outputs (PSPM samples) obtained by sampling p(e|t,c). Based on sufficiently high probability realistic means of p(e|t,c) - note that it may be highly desirable to test relatively low probability sensory outputs (outliers) if they are still realistic. The extent to which outliers are tested will depend on the level of safety that the AV needs to meet.

[0153] exist Figure 5 In , three reality perception outputs e1, e2, e3 are shown by way of example. These are sampled from p(e|t,c).

[0154] One approach is to sample perceptual outputs from p(e|t,c) in a way that favors the most likely perceptual outputs, for example, using Monte Carlo sampling. Broadly speaking, this will test a greater number of the most likely perceptual outputs and a smaller number of less likely outputs.

[0155] However, while this may be useful in some contexts, in other contexts it may be more useful to intentionally test a larger number of "outliers" (i.e., unlikely but still realistic perceptual outputs), since perhaps outliers are more likely to cause or contribute to unsafe behavior. That is, p(e|t,c) can be sampled in a way that is intentionally biased toward outliers, in order to intentionally make a particular scenario more "challenging" or "interesting" as it progresses. This can be accomplished by transforming the distribution of the PSPM and sampling from the transformed distribution.

[0156] Figure 6 An overview of the process of building a PSPM is provided. A large amount of real sensor output x is collected and annotated with the perception ground truth t. This is exactly the same process as the process of generating training data for training the perception components of the perception stack 102 (represented by box 602) - and the first subset of annotated sensor outputs is used for this proposal. The trained perception slice 204 is shown, which is executed at runtime (in the real world) and used to build the PSPM 500, which will model this perception slice 204 when simulated.

[0157] continue Figure 3 In the example of , the true sensor output would be the depth map, and the ground truth would be the ground truth 6D pose of any object captured in the depth map. This annotated data is not only used to train the 3D object detector 204 (the perception slice in this example), but is also used to build the PSPM 500, which models the 3D object detector 204 during simulation.

[0158] Box 604 represents PSPM construction (training), and uses a second subset of annotated sensor outputs for this purpose. Each sensor output is additionally annotated with a set of confounding factors c that characterize the physical conditions in which it is captured in the real world. Each set of confounding factors c that the PSPM needs to be able to accommodate requires a large number of sensor outputs. For full "Level 4" autonomous driving, this means capturing and annotating sensor outputs throughout the ODD.

[0159] PSPM 500 can take the form of parametric distributions:

[0160] Dist(t,c;θ)

[0161] where t and c are the variables that the distribution depends on, and θ is the set of learning parameters.

[0162] The parameters θ are learned as follows:

[0163] 1) Applying the trained perceptual slice 204 to each sensor output x to calculate the corresponding perceptual output e;

[0164] 2) For each sensory output e, determine the deviation (error) Δ between e and the corresponding ground truth t;

[0165] 3) Each error Δ is associated with the ground truth t and a set of confounding factors c related to the corresponding sensor output x;

[0166] 4) Adjust the parameters θ to fit the distribution to the error Δ, taking into account the associated ground truth and the variable confounding factor c.

[0167] As will be apparent, various known forms of parameter distributions / models can be applied in this context and are therefore not described in detail.

[0168] More generally, the training set used for PSPM training consists of perceptual ground truth (from manual, automatic or semi-automatic annotation) and the corresponding actual perceptual output generated by the perceptual slice 204 to be modeled. The goal of training is to learn a mapping between the perceptual ground truth and a perceptual output distribution that captures the statistics of the actual perceptual output. Thus, for a given ground truth t, the perceptual output sampled from the distribution p(e|t) will be statistically similar to the actual perceptual output used for training.

[0169] As an example, the perceptual slice 204 may be modeled as having zero mean Gaussian noise. However, it is emphasized that the present disclosure is not limited in this respect. The PSPM may well take the form of more complex non-Gaussian models. As an example, the PSPM may take the form of a hidden Markov model, which would allow explicit modeling of temporal dependencies between perceptual outputs at different times.

[0170] In the Gaussian case, for example, the PSPM 500 can be characterized as:

[0171] e=t+ε

[0172] ε~N(0,Σ(c)),

[0173] where N(0,Σ(c)) represents a Gaussian distribution with zero mean and covariance Σ(c) that varies depending on the confounding factor c. During simulation, noise will then be sampled from the Gaussian and added to the perceptual ground truth. This will depend on the variance of the Gaussian and therefore on the confounding factor applicable to the simulation scenario.

[0174] Example PSPM Error Dataset

[0175] Figure 7An example of a raw error plot for a two-dimensional prediction space is shown—for example, each point may correspond to an (x,y) coordinate that may be estimated by a 2D object detector. Each prediction e is represented by a circle, and each ground truth t is represented by a star. Each error Δ is represented by a line segment between the corresponding prediction e and the corresponding ground truth t (longer line segments represent larger errors).

[0176] To construct the PSPM, the goal is to take into account the variable confounder c to accurately capture in terms of probability Figure 7 The error relationship between the data points (in this context, the data points are the errors Δ) is used to adjust the parameter distribution.

[0177] Fig. 7A Shown is applied to Figure 7 Results of the trained PSPM on the error dataset.

[0178] Selection of confounding factors

[0179] The decision on which confounders to incorporate is driven by observation: when a particular physical property / condition can be seen to have a material effect on the perceived uncertainty, this may trigger its introduction as a confounding variable into the applicable PSPM. Only statistically relevant confounders should be introduced.

[0180] One approach to confounding factors is to partition the error dataset according to the confounding factors and train a separate model for each partition of the dataset. As a very simple example, the two confounding factors can be "lighting" and "weather", each of which can take a binary "good / poor" value. In this case, the dataset can be divided into four subsets with (lighting, weather) = (good, good), (good, bad), (bad, good) and (bad, bad), respectively, and four separate models can be trained for each subset. In this case, the PSPM consists of four models, where the confounding factor variable c = (lighting, weather) is used as an index to determine the choice of model.

[0181] Engineering pipeline architecture

[0182] Figure 8 A highly schematic overview of the engineering pipeline incorporating PSPM is shown. The entire pipeline covers everything from data collection, annotation and extraction; training of perception components; PSPM characterisation and simulation-based testing.

[0183] A large amount of sensor output, such as stereo images, depth maps, lidar measurements, and radar measurements, is collected using a fleet of vehicles, each equipped with a sensor system 110 of the type described above. These are collected in the types of environments and driving scenarios that AVs will need to be able to handle in practice, for example, in target urban areas where AVs are expected to be deployed. The collecting vehicles can be AVs themselves, or they can be manually driven vehicles equipped with similar sensor systems.

[0184] For the purpose of annotating captured sensor output with ground truth, a ground truth pipeline 802 is provided. This includes annotating the sensor output with the type of perceptual ground truth described above. The sensor output annotated with the perceptual ground truth is stored in an annotated ground truth database 804. Further details are described below.

[0185] In addition, the sensor outputs captured by the fleet are also used to extract driving scenarios, which can then be recreated in the simulator. The driving scenarios are captured using a high-level structured scenario description language and stored in a scenario database 806.

[0186] Sensor output captured from the fleet is not the only source of information from which driving scenarios can be extracted. In addition, CCTV (closed circuit television) data 800 is used as the basis for scene extraction, CCTV data typically captured in an urban environment, for example, the urban environment shows challenging urban driving scenarios, such as complex roundabouts. This provides a rich source of challenging driving scenarios, and thus provides an excellent basis for safety testing. A collection of back-end perception components 808 is used to process CCTV data 800 to assist in the process of extracting driving scenarios from CCTV data 800, which are then also stored in a scenario database 806 in a scenario description language format.

[0187] Further details of the scene description language and the process of extracting scenes from CCTV data and other data can be found in UK patent application No. 1816852.6, which is incorporated herein by reference in its entirety.

[0188] The driving scene captured in the scene description language format is a high-level description of the driving scene. The driving scene has both static layout (such as road layout (lanes, markings, etc.), buildings, road infrastructure, etc.) and dynamic elements. Figure 8In the pipeline of , the static layout is captured in the scene description as a pointer to an HD (high definition) map stored in a map database 826. The HD map itself can be derived from annotated sensor output collected by the AV fleet and / or from CCTV.

[0189] Dynamic elements include, for example, the position and motion of agents (e.g., vehicles, pedestrians, cyclists, etc.) within a static layout and are captured in a scene description language.

[0190] Running Simulation

[0191] The test suite orchestration component 810 uses the captured driving scenarios to formulate test instance specifications 812, which can then be run as 3D multibody simulations in a 3D simulator 814. The purpose of these simulations is to enable the derivation of accurate perceptual ground truth and then apply PSPM. Therefore, they contain a sufficient level of 3D geometric detail to enable the derivation of, for example, ground truth 3D bounding boxes (size, 6D pose of external actors in the ego vehicle reference frame), odometry and ego-localization outputs, etc. However, they are not photorealistic simulations as that level of detail is not required. They also do not attempt to simulate conditions such as rain, lighting, etc. as they are modeled as confounding factors c.

[0192] To provide greater scene variation, a scene "fuzzer" 820 is provided that can fuzzify a scene in the sense described above. Fuzzing a scene means changing one or more variables of a scene to create a new scene that is still realistic.

[0193] Typically this will involve blurring dynamic elements into a static scene, for example, changing the motion of external actors, removing or adding external actors, etc.

[0194] However, the static layout may also be obfuscated, for example by changing the curvature of the road, changing the position of static objects, changing road / lane markings, etc.

[0195] Figure 8 The training block 602 is shown to have access to an annotated ground truth data database 804, which is used to train the perception slice 204 of the runtime stack 100 as described above.

[0196] As mentioned above and Figure 8 As shown, the perceptual slice 204 is not necessarily the entire perceptual stack 102. In this example, the perceptual stack 102 is a "slice" before a set of final fusion components (filters) that collaborate to fuse the perceptual outputs from below the perceptual stack 102. These form part of one or more remaining prediction slices 205, which are not modeled using PSPM but are applied to PSPM samples. The output of the final (unmodeled) prediction slice 205 is fed directly into the prediction stack 104.

[0197] The PSPMs are shown as being stored in a PSPM database 820 .

[0198] Run the simulation

[0199] A PSPM sampling orchestration component 816 uses the 3D multi-body simulation in the 3D simulator 814 to derive ground truth, which in turn forms the input to one or more PSPMs used to model the perceptual slice 104 PSPM and provides PSPM samples 818 for each simulation. The PSPM samples 818 are fed to the rest of the runtime stack 100 (i.e., in this example, the PSPM samples 818 are fed to the final set of filters 205) and used as the basis for planning and prediction, ultimately causing the controller 108 to generate control signals, which are provided to the set of simulated AV motors.

[0200] Simulated motor not in Figure 8 shown in Figure 4 412. Figure 4 As shown, the 3D multi-body simulation in the 3D simulator is driven in part by simulated motors. These determine how the (in this case simulated) subject moves within the static layout (i.e., they determine the change of the subject's state (which may be referred to herein as the simulated robot state)). In turn, the subject's behavior may also affect the behavior of simulated external actors in response to the AV's motion. As the 3D simulation progresses, new perception ground truth continues to be derived and fed to the PSPM 500 in an iterative manner until the simulation is complete.

[0201] Each completed simulation is recorded as a set of test results stored in the test database 822 .

[0202] Note that the same scenario can be run multiple times and not necessarily produce the same results. This is due to the probabilistic nature of the PSPM: each time you run a scenario, you may get a different PSPM sample from the PSPM. Therefore, a lot of information can be gained by running the same simulated scenario on multiple occasions and observing, for example, the degree to which the simulated subject behaves differently in each instance of the scenario (large differences in subject behavior indicate that the effects of perceptual errors are significant) or the proportion of scenario instances where the subject behaves unsafely. If the same scenario is run a large number of times and the subject behaves safely and very similarly in each scenario, then this indicates that the planner 106 is able to plan correctly under uncertainty in that scenario.

[0203] Test Oracle

[0204] The driving scenarios used as the basis for the simulation are usually based on real-world scenarios or on fuzzy real-world scenarios. This ensures that realistic scenarios are being tested. Note however that these are usually driving scenarios that do not involve any actual autonomous vehicles, i.e., at least in most cases, the tested driving scenarios are derived from real-life examples of human driving. Therefore, there is no way to know which scenarios are likely to result in failures.

[0205] To this end, a scenario assessment component 824 (which is referred to herein as a "test oracle") is provided, which has the role of assessing whether the behavior of the simulated AV in the scenario is acceptable once the simulation is complete. The output of the test oracle 824 may include a simple binary (yes / no) output to mark whether the AV behaved safely, or it may be a more complex output. For example, it may include a risk score.

[0206] To do this, the test oracle 824 applies a predetermined set of rules, which may be referred to herein as a "Digital Highway Code (DHC)". In essence, the rules that define safe driving behavior are hard-coded. If the scenario is completed without violating those rules, the AV is considered to have passed. However, if any of those rules are violated, the AV will be considered a failure and marked as an instance of unsafe behavior that requires further testing and analysis. Those rules are encoded at an ontological level so that they can be applied to an ontological description of the scenario. The concept of ontology is well known in the field of robotics, and in this context, it is intended to characterize the driving scenario and the behavior of the simulated AV in the scenario at the same level of abstraction so that the DHC rules can be applied by the test oracle 824. The results of the analysis can quantify how the subject performs relative to the DHC, for example, the extent to which the rules are violated (for example, a rule may specify that a certain distance must be maintained from a cyclist at all times, and the results can indicate the extent to which the rule is violated and the circumstances in which the rule is violated).

[0207] Instances of unsafe behavior may also be marked as requiring "disengagement." For example, this may be the case where a failover mechanism within the runtime stack 100 is activated to prevent a crash or some other critical failure (as in that scenario in the real world).

[0208] The present technology is not limited to detecting unsafe behaviors. Behaviors can be evaluated based on other metrics (e.g., comfort, progress, etc.).

[0209] Example-aware stack

[0210] Fig. 9A schematic block diagram of a portion of an example perception stack is shown. A 3D object detector is shown and indicated by reference numeral 204, which in turn is shown to include a 2D object detector 902, a 2D tracker filter 904, a size estimation component 906, an orientation estimation component 908, a depth segmentation component 910, and a template fitting component 912. This represents one example architecture of the 3D object detector 204 mentioned above and shown in earlier figures.

[0211] The 2D object detector receives one image of each captured stereo image pair (the right image R in this example) and applies 2D object detection to the image. The output is a 2D bounding box for each object detected in the image. This provides a 2D (x, y) position of each object in the image plane and a bounding box indicating the size of the projection of the object into the image plane. The 2D tracking filter 904 receives the 2D bounding box outputs and applies filtering to them to improve the 2D bounding box estimate. For example, based on an object behavior model, the filtering may take into account the previously detected 2D bounding boxes and the expected behavior of the detected objects. The filtered 2D bounding boxes and the image data of the original image contained therein are then used for many different purposes. The 2D object detector 902 may take the form of a trained CNN.

[0212] The depth segmentation component 910 receives the filtered 2D bounding box and also receives the depth map extracted from the original stereo image pair by the stereo estimator 202. It uses the filtered 2D box to separate the depth points belonging to each object within the depth map. This is a form of depth segmentation.

[0213] The size estimation component 906 also receives the filtered 2D bounding boxes and uses them to estimate the 3D size of each detected object based on the image data of the right image contained within the 2D bounding boxes.

[0214] The direction estimation component 908 similarly receives the filtered 2D bounding boxes and uses them to determine the 3D direction of each detected object using the image data of the right image contained within the applied 2D bounding boxes. The size estimation component 906 and the direction estimation component 908 can take the form of a trained CNN.

[0215] For each detected object, 3D template fitting component 912 receives the separated depth point of the object from depth segmentation component 910, receives the 3D size of the object from size estimation component 906, and receives the 3D direction of the detected object from direction component 908. 3D template fitting component 902 uses those three pieces of information to fit the template to the depth point belonging to the object in the form of a 3D bounding box. The 3D size and 3D direction of the 3D bounding box are known from size and direction estimation components 906, 908 respectively, and the point to which the bounding box must be fitted is also known. Therefore, this is just the situation of finding the best 3D position of the 3D bounding box. Once this operation is completed for each object, the 3D size and 6D pose (3D position and 3D direction) of each detected object at a given moment can be known.

[0216] The output from the 3D template fitting component 912 to the input of the final filter 205 is shown. In addition, the final filter 205 is shown as having inputs that receive perception outputs from lidar and radar, respectively. Lidar and radar perception components are shown and represented by reference numerals 914 and 916, respectively. Each of these provides a perception output that can be fused with the perception output (such as 6D pose) from the 3D object detector 204. This fusion occurs in the final filter 205, and the output of the final filter is shown as connected to the input of the prediction stack 104. For example, this can be a filtered (perfected) 6D pose that takes into account all of these stereo, lidar and radar measurements. It can also take into account the expected object behavior in 3D space as captured in the expected behavior model of the 3D object.

[0217] Stack-aware slices

[0218] Fig. 9A Shows how to Fig. 9 An example of a "slice" of a perception stack, modeled as a PSPM. The perception stack 102 is considered to be sliced ​​after the final perception component modeled by the PSPM, and the perception output of the perception component can be referred to as the "final output" for the PSPM. The distribution of the PSPM will be defined over those final outputs, i.e., the e in p(e|t,c) corresponds to those final outputs of the components of the perception stack 102 after which it is sliced. All perception components and sensors that provide input to the components (directly or indirectly) are modeled by the PSPM (and are referred to as being "wrapped" in the PSPM) according to their contribution to the uncertainty in the final output e.

[0219] In this case, a single PSPM is provided for each sensor modality, i.e., one for stereo imaging, a second for LiDAR, and a third for RADAR. The three PSPMs are represented by reference numerals 500a, 500b, and 500c, respectively. To construct the first PSPM 500a, the perception stack 102 is sliced ​​after the 3D template fitting component 912, so that the distribution of the first PSPM 500a is defined on the perception output of the template fitting component 912. All perception components and sensors fed to the 3D template fitting component 912 are wrapped in the first PSPM 500a. The second and third PSPMs 914, 916 are sliced ​​after the LiDAR and Radar perception components 914, 916, respectively.

[0220] The final filter 205 is not modeled as a PSPM, but is applied to PSPM samples obtained from the three PSPMs 500a, 500b and 500c during testing.

[0221] Fig. 9B A second example slice is shown, where all three sensor modalities are modeled using a single PSPM 500d. In this case, the distribution p(e|t,c) is defined over all three sensor modalities, i.e., e=(e stere o,e lidar e lidar ). Thus, each PSPM sample will include realistic perceived outputs for all three sensor modalities. In this example, the final filter is still not modeled as a PSPM and will be applied at test time to the sampled PSPM obtained using a single PSPM 500d.

[0222] Fig. 9C A third example slice is shown where all three sensor modalities are modeled as a single PSPM 500e along with the final filter 205. In this case, the distribution p(e|t,c) is defined over the filtered perceptual output of the final filter 205. During testing, the PSPM 500e will be applied to the ground truth derived from simulation, and the resulting PSPM samples will be fed directly to the prediction stack 104.

[0223] Slicing considerations

[0224] A factor when deciding where to "slice" the perception stack is the complexity of the desired ground truth (the desired ground truth will correspond to the perception components after the stack is sliced): An underlying motivation for the PSPM approach is to have ground truth that is relatively easy to measure. The lowest parts of the perception stack 102 operate directly on sensor data, but the level of information required for planning and prediction is much higher. In the PSPM approach, the idea is to "bypass" the lower level details while still providing statistically representative perception outputs for prediction and planning during testing. Broadly speaking, the higher the perception stack 102 is sliced, the simpler the ground truth will typically be.

[0225] Another consideration is the complexity of the perception components themselves, since any perception components not wrapped in a PSPM must be executed during testing.

[0226] It is generally expected that slicing will always occur after the CNN in the perception stack, thereby avoiding the need to simulate the input to the CNN and avoiding using computational resources running the CNN at test time.

[0227] In a sense, it is beneficial to wrap as much of the perception stack 102 as possible into a single PSPM. In the extreme case, this means that the entire perception stack 102 is modeled as a single PSPM. The benefit of this is that any correlations between different sensors and / or perception components can be modeled without having to understand these correlations. However, as more and more of the perception stack 102 is wrapped in a single PSPM, this can significantly increase the complexity of the system being modeled.

[0228] for Fig. 9A , each individual PSPM 500a, 500b, 500c can be built independently of the data of a single sensor modality. This has the benefit of modulation - an existing PSPM can be retrained to test different configurations of the sensor slice 204 without retraining. Ultimately, the best PSPM architecture will depend on the context.

[0229] Especially in Fig. 9C In the case of , it may also be necessary to use a time-dependent model to fully capture the dependency on previous measurements / perception outputs introduced by the final filter 205. For example, Fig. 9C The PSPM500e can take the form of a hidden Markov model to capture additional degrees of time dependence. More generally, such time-dependent PSPMs can be used for any of the above. This is the context in which time-dependent models are useful, but in many cases explicit modeling of time dependence may be useful.

[0230] for Fig. 9A and Fig. 9B, cutting off before the final filter 205 has the benefit that no explicit time dependencies may need to be introduced, ie a form of the PSPM may be used which does not have any explicit dependency on PSPM samples previously taken from the PSPM.

[0231] Example of PSPM

[0232] The above description mainly focuses on dynamic objects, but PSPM can also be used in the same way for static scene detectors, classifiers, and other static scene perception components (e.g., traffic light detector, lane offset correction, etc.).

[0233] In fact, a PSPM can be built for any part of the perception stack 102, including:

[0234] - Odometry, for example:

[0235] IMU,

[0236] oVisual-odometry,

[0237] oLIDAR-odometry,

[0238] oRadar-odometry,

[0239] oWheel encoders;

[0240] -(Self-)positioning, for example:

[0241] oVision-based positioning,

[0242] o GPS positioning (or more commonly satellite positioning).

[0243] "Odometry" refers to the measurement of local relative motion, while "localisation" refers to the measurement of global position on a map.

[0244] A PSPM can be constructed in exactly the same way to model the perceptual output of such a perceptual component using a suitable perceptual ground truth.

[0245] These allow realistic odometry and localization errors to be introduced into simulated scenes in the same way as detection errors, classification errors, etc.

[0246] Ground truth pipeline

[0247] As described above, the generation of annotations in the ground truth pipeline 802 may be manual, automatic, or semi-automatic annotations.

[0248] Automatic or semi-automatic ground truth annotation can use high-quality sensor data that is usually not available to the AV at run time (or at least not available all the time). In fact, this can provide a way to test whether such a component is needed.

[0249] Automatic or semi-automatic annotation can use offline processing to obtain more accurate perception output, which can be used as ground truth for PSPM construction. For example, to obtain perception ground truth for localization or odometry components, offline processing such as bundle adjustment can be used to reconstruct the vehicle's path with high accuracy, which in turn can be used as ground truth to measure and model the accuracy of the AV's online processing. Such offline processing may not be feasible on the AV itself at runtime due to computational resource limitations or because the algorithms used are inherently non-real-time.

[0250] Examples of confounding factors

[0251] Fig.10 A high level overview of the various factors that may lead to uncertainty in the perceptual output (i.e. the various sources of potential perceptual error) is shown. This includes further examples of confounding factors c that may be incorporated as variables in the PSPM:

[0252] - Occlusion

[0253] - Lighting / Time of day

[0254] -weather

[0255] -season

[0256] - the (linear and / or angular) distance to the object

[0257] - the (linear and / or angular) velocity of the object

[0258] - Position in the sensor's field of view (e.g., angle from the center of the image)

[0259] - Other object properties, such as reflectivity, or other aspects of its response to different signals and / or frequencies (infrared, ultrasonic, etc.)

[0260] Other examples of possible confounders include maps of the scene (indicating environmental structure) and inter-agent variables such as "business" (a measure of the number or density of agents in the scene), distance between agents, and agent type.

[0261] Each can be characterized numerically or categorically in one or more variable components (dimensions) of a confounder space C.

[0262] Note, however, that a confounder can be any variable that represents something about the physical world that may be relevant to the perception error. This does not necessarily have to be a directly measurable physical quantity, such as speed, occlusion, etc. For example, another example of a confounder related to another agent might be "intent" (e.g., whether a cyclist intends to turn left or continue straight ahead at an upcoming turn at a particular moment, which can be determined by looking ahead at real-world data of the actions actually taken by the cyclist at a given time). In one sense, variables such as intent are latent variables or unobserved variables, in the sense that at a particular moment (in this case, before the cyclist takes an explicit action), intent cannot be measured directly using the perception system 102, but can only be inferred through other measurable quantities; the key point about confounders is that those other measurable physical quantities do not have to be known or measured in order to model the effect of intent on the confounder error. For example, there may be a statistically significant increase in perceptual error associated with a cyclist with the intention to "turn left" compared to a cyclist with the intention to "continue straight", possibly due to multiple, unknown and potentially complex behavioral variations in the behavior of cyclists who want to turn left, meaning that in reality, the perception system perceives them worse. By introducing the "intention" variable as a confounder in the error model, there is no need to try to determine which observable physical manifestations of intention are associated with perceptual error - as long as the "intention" ground truth can be systematically assigned to the training data (e.g., based on hindsight of the last action taken by the cyclist) in a manner consistent with the simulation (in which case the intentions of the simulated cyclists are known in order to simulate their behavior as the scene develops), such data can then be used to construct appropriate behavioral models for different intentions in order to simulate that behavior, as well as perceptual error models that depend on intention, (if any) without having to determine which physical manifestations of intention are actually associated with perceptual error. In other words, in order to model the effect of intention on perceptual error, it is not necessary to understand why intention is associated with perceptual error, as intention itself can be modeled as a perceptual confounder (rather than attempting to model observable manifestations of intention as confounders).

[0263] Low Level Errors

[0264] Examples of low-level sensor errors include:

[0265] -Registration errors

[0266] -Calibration errors

[0267] -Sensor limitations

[0268] Such errors are not explicitly modeled in the simulation, but their effects are wrapped in the PSPM used to model the perceptual slice that interprets the applicable sensor data. That is, these effects will be encoded in the parameters θ that characterize the PSPM. For example, for a Gaussian-type PSPM, such errors will result in larger covariances that indicate greater uncertainty.

[0269] High-level perception errors

[0270] Other errors may occur in the perception pipeline, for example:

[0271] -Tracking errors

[0272] -Classification errors

[0273] -Dynamic object detection failures

[0274] -Fixed scene detection failures

[0275] When it comes to detection, false positives and false negatives may cause the prediction stack 104 and / or planner 106 to behave in unexpected ways.

[0276] Specific PSPMs are constructed to model such errors in a statistically robust manner. These models can also take into account the effects of variable confounders c.

[0277] Using object detection as an example, detection probabilities can be measured and used to construct a detection distribution that depends on, for example, distance, angle, and occlusion level (confounding factor c in this example). Then, when running the simulation, through ray tracing from the camera, it can be determined from the model that the object "might" be detectable. If so, the measured detection probability is checked, and the object is either detected or not. This deliberately introduces the possibility of sensor-sensitive objects not being detected in the simulation in a way that reflects the behavior of the perception stack 102 in real life, because a statistically robust way of modeling detection failures has been done.

[0278] This approach can be extended in Markov models to ensure correctly modeled conditional detection. For example, an object is detected with the appropriate probability only if it has been detected before, otherwise the probability may be different. In this case, the false negatives involve some time dependency on the simulated detection.

[0279] False positives may be generated randomly at a density similar to the density in space and time measured by the PSPM, i.e., in a statistically representative manner.

[0280] 2. Problem Statement

[0281] As further explanation, this section presents the mathematical framework of PRISM and introduces the specific dynamic object detection problem addressed in the subsequent sections. Section 3 discusses the dataset used to train PRISM, the techniques used to identify relevant features, and a description of the evaluation methods. Section 4 describes specific modeling decisions and how data science informs these decisions.

[0282] Please note that in the following description, the symbol x g ,y g ,z g can be used to represent the coordinates of the location-aware ground truth t. Similarly, x s ,y s ,z s can be used to represent the coordinates of the position-aware stack output e. Therefore, the distribution p(x s ,y s ,z s |x g ,y g ,z g ) is a form that the above-mentioned perceptual uncertainty distribution p(e|t) can take. Similarly, x may be used below to refer to a set of confounding factors, which is equivalent to the set of confounding factors c or c' described above.

[0283] Perception systems have difficult-to-simulate inputs such as camera images, lidar scans, and radar returns. Since these inputs cannot be rendered with perfect realism, perception performance in simulation will not match that in the real world.

[0284] The goal is to build a probabilistic surrogate model for the perception stack, called PRISM. PRISM uses a low-fidelity representation of the world state (the perception ground truth) and produces perception outputs in the same format as the vehicle stack (or more accurately, the perception slice 204 being modeled). When the stack is run on real data, samples drawn from the surrogate model in simulation should be similar to the output of the perception stack.

[0285] PRISM sampling should be fast enough to be used as part of a simulation system for validation and development of downstream components such as a planner.

[0286] 2.1 Intuition

[0287] The following sections describe the most general case, with the following considerations in mind:

[0288] There is some stochastic function that maps from the true state of the world to the output of the perception stack.

[0289] The function can be modeled using training data. The function is modeled as a probability distribution.

[0290] Since the world state changes smoothly over time, the sampled sensory outputs should also change smoothly over time. Since the world state is only partially observed, an appropriate way to achieve this is to have the probability distribution depend on the history of the observed world state and sensory outputs.

[0291] The simulator (Genie) is responsible for generating a representation of the world at runtime. The output of Genie is the 6D pose and extent of dynamic objects, as well as some other information such as road geometry and weather conditions.

[0292] For real-world training data, the world representation is obtained from annotations.

[0293] Mathematical Statement

[0294] 2.2.1 Preliminaries

[0295] For any set S, let the set of S's history be An element (t,h)∈histories(S) consists of t (the current time) and h (a function that returns an element of S at any arbitrary time in the past). Notation Represents the analog equivalent of x.

[0296] The perception system is a random function f:histories(World)→histories(Perception). Usually, f will be of the form:

[0297]

[0298] The goal is to simulate some f. The state of the world can be decomposed into a set of features, ObservedWorld, and a set of everything else, UnobservedWorld (the exact pixel values ​​of a camera image, the temperature of every point on every surface), such that there is a bijection between World and ObservedWorld×UnobservedWorld, and the set of features, ObservedWorld, can be reliably measured (this might include the meshes and textures of every object in the scene, the position of light sources, material densities, etc.). In traditional photorealistic simulation methods, simulating f is equivalent to finding some random function It can be combined with perceive to form

[0299]

[0300] Let observe∶World→ObservedWorld be a function that maps world states to their observed counterparts. Note that this function is not one-to-one: there are many world states that map to a single observed world state. An accurate and useful analog of f For all histories (t,h)∈histories(World) we will have

[0301]

[0302] Where map∶((S→T)×histories(S))→histories(T) maps a function on history.

[0303] Then we must conclude that the best realistic simulation has Make

[0304]

[0305] Since sense predicts the history of sensor readings, the joint distribution of the history (SensorReading), and the correlations between different sensor readings allow dependencies on unobserved features of the world to be modeled more effectively, due to the associativity of °, combining equations 1, 2, and 4 gives equation 3. Similar correlations should be observed in .

[0306] Because SensorReading has high dimensionality, and sense is a random function (because it depends very much on unobserved properties of the world), finding It is very important to make equation 4 even approximately true. Therefore, it is straightforward to find

[0307] 2.2.2 Creating a surrogate model

[0308] The creation of a proxy model can be characterized as a random function estimation task. Let S+ be a finite sequence of elements of S. Let For an element s i A sequence of length N. Get a dataset of sensor reading sequences

[0309]

[0310] Among them, each I ij ∈SensorReading is the time t in run i ij The sensor readings, and M i is the number of timestamps in a particular run. A new dataset is constructed using the function annotate:SensorReading→ObservedWorld which recovers the observed scene parameters from the sensor readings.

[0311] x ij =annotate(I ij ), y ij =perceive(I ij ).

[0312] The task of PRISM is then to estimate the The distribution of samples in A realization of can be obtained by drawing samples from this distribution.

[0313] Output of the stack of previous samples The dependence on y is included because the distribution of y depends meaningfully on the unobserved world, and the unobserved world changes smoothly over time. As discussed in Section 2.2.1, this dependence on the unobserved world means that y will vary in ways that are difficult to predict based solely on the distribution of y. The way the dependencies are modeled changes smoothly over time. This time-dependent nature of the stack outputs is explored for the perception system 102 in Section 4.2.3, where strong correlations over time are found.

[0314] Samples from the learned PRISM distribution give plausible perceptual outputs conditioned on a low-fidelity scene representation and a history of previous samples. These factors are the independent variables in the generative model, and the dependent variable is the perceived scene. Independent variables that meaningfully influence the distribution of the dependent variable are referred to in this article as confounders. Part of the process of building a PRISM model is to identify relevant confounders to include in the model and how these confounders should be combined. Section 3.2 explores a method for identifying relevant confounders.

[0315] 2.2.3 Dynamic Object Issues

[0316] A specific example of a perception system is presented - a system that uses RGBD images to detect dynamic objects in a scene. A "dynamic object" is a car, truck, cyclist, pedestrian, or other road user described by an oriented bounding box (6D pose and extent). The observed world is a collection of such dynamic objects. In this setting,

[0317]

[0318]

[0319]

[0320] SensorReading--Image--[O@1]w xhx4 ,

[0321] in, is the set of finite subsets of S, 1 Type represents the object type (Car, Van, Tram, Pedestrian), Spin(3) is a collection of unit quaternions, and Info is an arbitrary collection whose elements describe additional properties of dynamic objects, such as the degree to which the object is occluded by other (possibly static) objects in the scene closer to the camera. Dynamic objects are useful in characterizing the behavior of a perception system.

[0322] This example simplifies the dynamic object problem even further, selecting only the

[0323] 1For simplicity, assume that there are never two identical objects. It is straightforward to extend this teaching to the case where the elements of one or both of ObservedWorld and Perception are multisets. The positions of dynamic objects are modeled. This includes fitting a model for the possibility that an observable object is not perceived (a false negative).

[0324] It was shown in Section 4.2.8 that false negatives are a more common error made by the perception system 102 than false positives (spurious dynamic object detections).

[0325] For simplicity, the following description only considers dangerous targets (poison) in 3D space, and omits the discussion of direction, range, object type or other possible perception outputs. However, the principles are also applicable to such other perception outputs.

[0326] 3 Methods

[0327] 3.1 Data

[0328] A specific driving scenario is presented, the data of which has been recorded multiple times under similar conditions. The scenario mentioned herein by way of example is a roundabout in south-east London, located on a test route. Fig.12 The background of the roundabout and the path of vehicles can be seen in Fig.13 The camera shown is observed.

[0329] By restricting PRISM training data to operations on the same roundabouts under similar climatic conditions, the effects of weather and sunlight as confounding factors in perceived performance are minimized. The potential performance of PRISM tested on similarly collected data is likewise maximized. For example, the performance of PRISM can be tested on out-of-domain data by estimating how PRISM trained on roundabout data would perform in a highway scenario.

[0330] 3.1.1 Dataset Generation

[0331] PRISM training requires a dataset containing enough information to learn the distribution of perceptual errors. For simplicity, this section only considers the errors introduced by the perception system 102 when predicting the center position of a dynamic object in the camera frame that observes the dynamic object. In order to learn such errors, both ground truth centers and perceptual center estimates are required.

[0332] The ground truth center locations are estimated from human-annotated 3d bounding boxes present in each frame of the recorded video sequence of the roundabout and applied to all dynamic objects in the scene. These bounding boxes are fitted to the scene using the ground truth tooling suite. The ground truth tooling suite combines camera images, stereo depth pointclouds, and lidar pointclouds into a 3D representation of the scene to maximize annotation accuracy. It is assumed that the annotation accuracy is good enough to be used as ground truth.

[0333] Fig. 9 The process of obtaining a stack of predicted objects from recorded camera images is shown. Note that this pipeline is stateless and each pair of camera frames is processed independently. This forces any temporal correlations found in the perceptual error data to be attributed to the behavior of the detector on closely related inputs rather than to the internal state of the detection algorithm.

[0334] In general, a collection of object predictions indexed by image timestamps combined with a similarly indexed collection of ground truth data from the ground truth toolkit is sufficient for PRISM training data. However, all models considered in this section are trained on data that has been through an additional processing step to generate associations between ground truth and predicted objects. This limits the space of models to choose from, but simplifies it by splitting the fitting task into the following parts: fitting a model for location errors; fitting a model for generating false negatives; fitting a model for generating false positives. The association algorithm used is run independently on each frame. For each timestamp, the stack prediction set and the ground truth object set are compared using intersection over union (IOU), where the predicted object with the highest confidence score (a measure generated by the perception stack 102 indicating how good the prediction may be) is considered first. For each predicted object, the ground truth object with the highest IOU is associated with it, forming a pair for learning the error distribution. No association is formed for pairs with IOU scores less than 0.5 (an adjustable threshold). After all predicted objects have been considered associated, a set of unassociated ground truth objects and a set of unassociated predicted objects remain. Unassociated ground truth objects are stored as false negative examples, while unassociated predicted objects are stored as false positive examples.

[0335] A "set-to-set" model that does not require such associations will be considered later.

[0336] 3.1.2 Content of training data

[0337] The previous section described how PRISM training data is generated and divided into three sources: associations, false negatives, and false positives. Table 1 specifies the data present in each source and provides the following definitions:

[0338] centre_x,centre_y,centre_z: x,y,z coordinates of the center of the ground truth 3d box.

[0339] orientation_x, orientation_y, orientation_z: x, y, z components of the rotation from the camera frame (front stereo right) to the ground truth 3d box frame.

[0340] height,width,length: The extent of the ground truth 3d box along the z, y, and x axes in the 3d box’s coordinate system.

[0341] manual_visibility: Labels applied by human annotators to annotate which of four visibility categories the ground truth object belongs to. The categories are: fully-occluded (100%), largely-occluded (80-99%), somewhat-occluded (1-79%), and fully-visible (0%).

[0342] occluded: The portion of the area where the ground truth 2d bounding box overlaps with the 2d bounding boxes of other ground truth objects that are closer to the camera.

[0343] occluded_category: A combination of manual_visibility and occluded, which can be thought of as the maximum of the two. It is useful to combine manual_visibility and occluded in this way to maximize the number of correct occlusion labels. To see this, note that objects occluded by static parts of the scene (bushes, trees, traffic lights) will have an occluded score of 0, but will have a manual_visibility correctly set by the human annotator. Objects that are only occluded by other ground truth objects will not have a manual_visibility field set by the human annotator, but will have a correct occluded field. These two cases can be handled by taking the maximum of the two values. Even with this logic, it is possible for the 2d bounding box of a ground truth object to completely obscure the 2d bounding box of an object behind it, even if some of the background objects are visible. This will generate some fully-occluded cases that can be detected by the perception system.

[0344] truncated: The portion of the eight vertices of the ground truth 3d box that lie outside the sensor frustum.

[0345] type: When attached to a ground truth object (false negatives, the ground truth part of an association pair), this is the manually annotated object type, such as Car or Tram. When attached to a predicted object (false positives, the predicted part of an association pair), this is the perception stack's best guess at the object type, which is limited to Pedestrian or Vehicle.

[0346] In addition to the above, this section will also mention the following exports:

[0347] distance: The distance from the center of the object to the camera, calculated as the Euclidean norm of the object center position in the camera frame.

[0348] azimuth: The angle between the projection of the ray connecting the camera and the center of the object on the camera's y=0 plane and the camera's positive z-axis. The polarity is defined by the direction of rotation around the camera's y-axis. Since objects behind the camera cannot be observed, the range is limited to [-π / 2,π / 2].

[0349] Table 1

[0350]

[0351] The composition of the dataset will be discussed in detail in later sections where relevant. A high-level summary of the data is presented below.

[0352] 15 passes through the roundabout scene, spanning approximately 5 minutes of total footage.

[0353] 8600 unique frames containing 96k ground truth object instances visible to the camera.

[0354] Of these 96k instances: 77% were cars; 14% were trucks; 6% were pedestrians; and 3% belonged to smaller groups.

[0355] Of these 96k instances: 29% are fully visible; 43% are somewhat occluded; and 28% are mostly occluded.

[0356] In Table 1, specific data elements are present in each of the three generated PRISM data sources. X indicates that the column exists in the given data source. GT = Ground Truth, FN = False Negative, FP = False Positive. Each of these is actually three separate variables (e.g., center_x, center_y, center_z), but are "compressed" here for readability. * In the case of an asterisk, the type of content can be "Vehicle" or "Pedestrian", which are the only classes predicted by the five perception stacks. In the case without an asterisk, there are more classes (such as "Truck" and "Van"), which are all the classes reported in the ground truth data.

[0357] 3.1.3 Training and Testing Data

[0358] For all modeling experiments described in this paper, the roundabout dataset was split into roughly equal halves to form training and test sets. No hyperparameter optimisation was performed, and therefore, no validation set was required.

[0359] 3.2 Identification of relevant confounding factors

[0360] The PRISM model may take into account many confounders. Rather than optimizing the model for every possible combination of confounders, it is preferred to perform such optimization on a limited set of confounders that are known to be relevant.

[0361] To identify relevant confounders, a Relief-based algorithm is used. An outline of the general Relief-based algorithm is given in Algorithm 1. The Relief algorithm produces an array of feature weights in the range [-1, 1], where weights greater than 0 indicate that the feature is relevant because changes in the feature tend to change the target variable. In practice, some features will have weights greater than 0 by chance, and only features with weights greater than some user-defined cutoff 0<τ<1 are selected.

[0362]

[0363] The algorithm has the following desirable properties:

[0364] It is sensitive to non-linear relationships between features and the target variable. Other feature selection methods such as naive principal component analysis or comparison of Pearson correlations are not sensitive to these kinds of relationships. Not all unrelated things are independent.

[0365] It is sensitive to interactions between features.

[0366] It is conservative. It will include irrelevant or redundant confounders by chance rather than exclude relevant confounders by chance.

[0367] It is important to note the following caveats to this approach:

[0368] It identifies correlations in the data but provides no insight into how or why the target variable is associated with the confounders under investigation.

[0369] The results depend on the parameterization of the confounding variables.

[0370] There are many extensions to the Relief algorithm. An extension called MultiSURF is used here. MultiSURF was found to perform well on a wide range of problem types and is more sensitive to interactions of three or more features than other methods. This implementation is used from scikit-rebate, an open source Python library that provides implementations of many Relief-based algorithms that are extended to cover scalar features and target variables.

[0371] In the experiment, using Where n is the size of the dataset and α = 0.2 is the expected false discovery rate. According to Chebyshev's inequality, we can say that the probability of accepting an irrelevant confounder as relevant is less than α.

[0372] Relief-based methods are useful tools for identifying possible confounders and their relative importance. However, not all features that affect the error characteristics of a perceptual system are captured in annotated training data. A manual process of checking model failures to hypothesize new features to label as confounders is necessary.

[0373] 4 Model

[0374] 4.1 Heuristic Model

[0375] Camera coordinates represent the position of a point in an image in pixel space. In binocular vision, the camera coordinates of points in both images are available. This allows reconstruction of the position of points in a 3D Cartesian world. The camera coordinates of a point p in 3D space are given by:

[0376]

[0377]

[0378]

[0379] Among them, (u1, v1), (u2, v2) are the image pixel coordinates of p in the left camera and the right camera respectively, (x p ,y p ,z p ) is the 3D world coordinate of p relative to the left camera, b is the camera baseline, and f is the focal length of the camera. Fig.14 As shown. The disparity d is defined as:

[0380]

[0381] The 3D world coordinates of p can be written as:

[0382]

[0383]

[0384]

[0385] The heuristic model is obtained by imposing a distribution in camera coordinates and propagating it to 3D coordinates using the above relations. This distribution can be used equally for object centers or object extents. This model allows to take into account the physical sensor uncertainty of the camera when the image is discretized into pixels. The model is given by:

[0386] p(x s ,y s , z s |x g ,y g , z g )=∫∫∫p(x s ,y s , z s |u1,v,d)p(u1,v,d|x g ,y g , z g )du1dvdd,(12)

[0387] Among them, (x g ,y g , z g ) are the coordinates of the ground truth point, and (x s ,y s , z s ) are the coordinates predicted by the stack. Given the world coordinates, the probability distribution over the camera coordinates is:

[0388] p(u1,v,d|x g ,y g , z g )=p(u1|x g ,y g , z g )p(v|x g ,y g , z g )p(d|x g ,y g , z g ), (13)

[0389] Here, independence of the distribution in each camera coordinate is assumed:

[0390]

[0391]

[0392]

[0393] Where σ is a constant, is a normal distribution, and Lognormal is a log-normal distribution, chosen because it only supports positive real numbers. This defines a normally distributed probability density centered at the camera coordinates of a point in 3D space. The normal distribution was chosen based on mathematical simplicity. If only discretisation error was of concern, a uniform distribution might be more appropriate. However, other errors are likely to contribute to uncertainty in stereo vision, so extended tails of the normal distribution are useful for modeling such phenomena in practice. For a frontal stereo camera, α can be determined to be 0.7 by maximum likelihood estimation. p(x s ,y s ,z s |u1,v,d) is obtained from equation 9-equation 11 s ,y s and z s The point value of is given by a Dirac distribution centered on .

[0394] By forming a piecewise constant diagonal multivariate normally distributed approximation of Eq. 12, solving the integral with Monte Carlo simulation and using the mean and variance of the sampled values ​​for different x g ,y g and z g Value estimation p(x s ,y s ,z s |x g ,y g ,z g ) to obtain the runtime model.

[0395] The model can be improved by considering more accurate approximations of the conditional distribution in Eq. 12, or by modeling the uncertainty in the camera parameters f and b (set to their measured values ​​in the model). How to extend the model to include temporal dependencies is an open question.

[0396] 4.2PRISM

[0397] Described below is an attempt to build a data analysis guided rational proxy model (PRISM) of the perception stack / substack 204. The model includes time-dependent position errors and non-zero probabilities of objects not being detected as salient features of the data.

[0398] 4.2.1 Positional errors

[0399] The center position of dynamic objects detected by the perception stack will be modeled using an additive error model given by:

[0400] y k =x k +e k

[0401] Among them, y k is the observation position of the object, x k is the ground truth location of the object, and e k is the error term, both at time t k The phrase "position error" will be used to refer to the additive noise component of this model. k .

[0402] Fig.15 The position error of a specific dynamic object detected by the perception stack relative to the human-marked ground truth is shown. Fig.16 Lag plots of the same data are found in , indicating the strong temporal correlation of these errors. From these plots, it can be concluded that the generative model of the position errors must condition each sample on the previous samples. An autoregressive model is proposed for the time-correlated position errors, in which each error sample depends linearly on the previous error sample and some noise. The proposed model can be written as:

[0403] e k =e k-1 +Δe k (18)

[0404] Among them, e k is the position error sample at time step k, and Δe k is a random term, Δe k It can be a function of one or more confounding factors and is often referred to as “error deltas”. Fig.17 A diagram visualizing the model is shown, including the dependence on the hypothesized confounders C1 and C2.

[0405] The model is based on several assumptions. First, subsequent error increments are independent. This is explored in Section 4.2.3. Second, the empirical distribution of the error increments can be reasonably captured by the parameter distribution. This is explored in Section 4.2.4. Third, the model described is fixed so that the mean error does not change over time. This is explored in Section 4.2.5. 4.2.2 Piecewise Constant Model

[0406] We have shown that modeling position errors requires that subsequent errors be conditioned on previous errors, but how should the first error sample be chosen? Now consider the task of fitting a time-independent position error distribution. If no temporal correlation is found in the data, the approach taken here can also be applied to all samples for each dynamic object, not just the first sample.

[0407] In general, such a model will be a complex joint probability distribution over all confounders. As discussed in Section 2, due to incomplete scene representations (ObservedWorld≠World) and possible uncertainty in the perception stack, a distribution of possible perceptual outputs given a ground truth scene is expected. The variance is expected to be heteroskedastic; it will vary based on the confounder values. As a simple example, it should not be surprising that the error in the position estimate of a dynamic object has a variance that increases with the distance of the object from the detector.

[0408] The conditional distribution modeled by PRISM is expected to have a complex functional form. This functional form can be approximated by discretizing each confounder. In this representation, categorical confounders (e.g., vehicle type) are mapped to bins. Continuous confounders (e.g., distance to the detector) are split into ranges, and each range is mapped to a bin. The combination of these discretizations is a multidimensional table for which the input set of confounders is mapped to bins. It is assumed that within each bin, the variance is homoskedastic and a distribution with a constant parameter can be fitted. Global heteroskedasticity is captured by different parameters in each bin. The model with a distribution with fixed parameters in each bin is referred to as a piecewise constant model (PCM) in this article. General implementation examples of similar models can be found in the literature. Mathematically, this can be written as P(y|x)∼G(α[f(x)],β[f(x)],…), where y is the set of outputs, x is the set of confounders, f(·) is a function mapping confounders to bins, and G is a probability distribution with parameters α[f(x)],β[f(x)],… fixed within each bin.

[0409] In the PCM used for PRISM, the errors are assumed to be additive, i.e. the predicted position, pose and extent of the stack of dynamic objects are equal to the ground truth position, pose and extent plus some noise. The noise is characterized by the distribution in each bin. In this PCM, this noise is assumed to be normally distributed. Mathematically this can be written as:

[0410]

[0411]

[0412] where y is the stack observation, is the ground truth observation, and ∈ is the noise. The distribution in each bin is characterized by mean μ and covariance Σ. μ and Σ can be viewed as functions of the confounding factor bin.

[0413] Fig.18 An example binning scheme is shown. A bin consists of an azimuth and a distance to the center of a ground truth dynamic object.

[0414] Training the model requires ground truth and stack predictions (actual sensory outputs) collected as described in Section 3.1.1. (For example, use the maximum a posteriori method to incorporate priors) Fit a normal distribution's mean and covariance to the observations in that bin. For the normal distribution's mean, use a normal distribution's prior. For the normal distribution's scale, use an inverse gamma prior.

[0415] To set the hyperparameters of the prior, physical knowledge can be used in conjunction with an intuition about how quickly the model should ignore the prior when data becomes available. This intuition can be expressed in terms of the concept of pseudo-observations, which is how strongly the prior distribution is weighted in the posterior distribution compared to actual observations (encapsulated in the likelihood function). Increasing the number of pseudo-observations leads to priors with lower variance. The hyperparameters for a normally distributed prior can be set to μ h =μ p and Among them, μ p and σ p represents the a priori point estimates of the mean and standard deviation of the bin under consideration, and n pseudo represents the number of pseudo observations. The rate and scale hyper-parameters of the Inverse Gamma prior can be set to and For this model, select n pseudo = 1, and use the heuristic model described in Section 4.1 to provide a priori point estimates for the parameters for each bin.

[0416] The advantages of the PCM approach are that it accounts for global heteroskedasticity, provides a unified framework for capturing different types of confounding factors, and it exploits simple probability distributions. In addition, the model is interpretable: the distribution within the bins can be inspected, the training data can be inspected directly, and there are no hidden transformations. Furthermore, the parameters can be fitted analytically, which means that uncertainty from lack of convergence in the optimization routine can be avoided.

[0417] Confounding factor selection

[0418] To select appropriate confounders for PCM, the method described in Section 3.2 and the data described in Section 3.1.2 are used. The results of the investigation applied to position, range, and orientation errors are shown in Table 2.

[0419] Table 2 - shows the confounders identified as important for the target variable considered

[0420]

[0421] As can be seen from Table 2, for d_centre_x and d_centre_z, the relevant confounding factors are some combination of the object's position relative to the camera and the degree to which the object is occluded. Perception system 102 assumes that the detected object exists on the ground plane, y=0, which may be why d_centre_y does not show dependence on distance.

[0422] This analysis identifies position and occlusion as good confounders to start with for models of position errors of dynamic objects detected by the perception system 102. The data does not show a strong preference for position confounders based on a Cartesian grid (centre_x, center_y, center_z) over polar coordinates (distance, azimuth). Distance and azimuth are used in the PRISM prototype described herein, but a more in-depth evaluation of the relative performance of each prototype can be performed.

[0423] 4.2.3 Time correlation analysis of position error increment

[0424] The time correlation analysis performed on the position error can be repeated for the time series of error increments, giving Fig.19 The lag plots shown in . These plots show much less temporal correlation in the error deltas than is found in the position errors. The Pearson correlation coefficients for the error deltas are shown in Table 3. For each dimension, they are quite small in magnitude, with -0.35 being the farthest from zero. From this analysis it can be concluded that a good model for the error deltas can be formed by independent samples from correlated distributions.

[0425] Table 3 - Pearson correlation coefficients of error increment samples and one time step delayed samples

[0426]

[0427] Distribution of position error increment

[0428] Typically, the x, y, and z error delta dimensions are related. Here, they are considered independently, but note that future efforts could consider modeling them jointly. Fig. 20 A histogram of the error delta samples is presented in , from which it can be clearly seen that the error delta is more likely to be close to zero but has a long tail with extreme values. Fig.21The maximum likelihood best-fit to this data for some of the trial distributions is shown. Visual inspection of these plots suggests that the Student's t-distribution may be a good modeling choice for generating the error deltas. Due to the large number of extreme error deltas in the data, the normal distribution does not fit well.

[0429] Bounding the random walk

[0430] The autoregressive error delta model proposed in Section 4.2.1 is generally a non-bounded stochastic process. However, it is well known that the detected positions of dynamic objects do not simply drift away, but remain close to the ground truth. This is an important property that must be captured in a time-dependent model. As a concrete example of this point, consider modeling the position error as a Gaussian random walk, setting This produced Time t k The position error distribution of , where the variance increases with time without being constrained in time. Such properties must not exist in the PRISM model.

[0431] AR(1) is a first-order autoregressive process defined by the following equation:

[0432] y t =a1y t-1 +∈ t (19)

[0433] where ∈t is a sample from a zero-mean noise distribution, and y t is a sample of the variable of interest at time t. It is known that this process is wide-sense stationary for |a1|<1, otherwise the generated time series is non-stationary. Comparing Equation 18 and Equation 19, it can be seen that given the known results for AR(1), if Δe k is zero mean, then the error increment model proposed in Equation 18 will be non-stationary. Therefore, such a model is not sufficient to generate reasonable stack outputs.

[0434] An extension to the model suggestion of Eq. 18 is proposed, motivated by the nature of the error delta data collected. The extension is conditioned on the previous error for Δe k Modeling is performed to find the relationship between P(Δe k |e kBest fit for (e

[0435] Following the piecewise constant modeling method described in Section 4.2.2, P(Δe k |e k -1) is approximated as follows:

[0436] For the space of e k-1 values, form M bins with boundaries {b0, b1,..., b M}.

[0437] Characterize a separate distribution P m (Δe k ) for each bin, where 0 < m < M represents the bin index.

[0438] Given the previous time step position error e k-1 , the next error increment is drawn from P m (Δe k ), where B m-1 < e k-1 < B m .

[0439] Fig. 22 Shows the sample mean calculated on the PRISM training data with M = 5. The revealed trend is consistent with expectations and has the following intuitive explanation. Consider a series of error increment samples with the same polarity that have accumulated an absolute position error away from the ground truth. To make the overall process appear stationary, subsequent error increment samples with the same polarity should be less likely to change in the direction towards the true object position. This observation helps to explain the negative Pearson coefficients presented in Table 3, which indicate that subsequent error increments slightly tend to reverse polarity.

[0440] For the binning scheme of P m (Δe k ), it suffers from the typical PCM drawback of a low sample base in extreme bins. A simple prior can be used to reduce this risk, for example, setting the mean of the distribution in each bin to follow μ m = -ae m , where e m is the center value of the m-th bin and a > 0. It is worth noting that if P m (Δe k ) is chosen to be a Gaussian distribution such that then the time-dependent model becomes:

[0441]

[0442] The time-dependent model is a typical AR(1) process and is stationary for a < 2. In practice, a good prior will require a ~ 0, so that such a model is stationary by construction.

[0443] 4.2.6 Simple Verification

[0444] It is instructive to see whether samples from the proposed time-dependent position error model reproduce the features that motivated its construction. Fig.23 A plot of the position error for a single dynamic object trajectory sampled from the learned distribution is shown in . Fig.24 A hysteresis plot of the same data is shown. In both cases, the true perceived error data is provided for visual comparison. The similarity between the PRISM sample and the observed stack data is encouraging. Clearly, a more quantitative evaluation is needed (which will be the subject of Section 5) to make any meaningful justification statements.

[0445] False Negatives and False Positives

[0446] Perception systems have failure modes beyond noisy position estimates of dynamic objects. A detector may fail to recognize an object in the scene, i.e. a false negative, or it may recognize an object that is not present, i.e. a false positive. A surrogate model like PRISM must emulate the observed false negative and false positive rates of the detector. This section discusses the importance of modeling false negatives, investigates which confounding factors affect the false negative rate, and proposes two simple Markov models. It is shown that using more confounding factors can produce better performing Markov models, and highlights some issues with doing so using piecewise constant methods.

[0447] A survey was conducted to determine the frequency of true positives (TP), false negatives (FN), and false positives (FP). The results are summarized in Table 4. There are significantly more false negative events than false positives. The counts in Table 4 apply to all object distances. It seems unfair to count false negatives at such a distance from the detector that a human would have a difficult time identifying them. Introducing a distance filter on the events reduces the factor that false negatives are more common than false positives, but the differences are still significant. When considering objects at a depth of less than 50m, the number of TP / FN / FP events is 34046 / 13343 / 843. Lowering the distance threshold to 20m depth, the number of TP / FN / FP events is 12626 / 1236 / 201.

[0448] Table 4 - shows numbers describing false positive and false negative events in the dataset

[0449]

[0450] 4.2.9 False Negative Modelling

[0451] Following the approach described in Section 3.2, we explored the importance of different confounding factors on false negatives by using the relief algorithm. 2 For randomly selected 20% of the training data samples. The results are as follows Fig.25 As shown. A 20% random sample of the data allows the algorithm to run with manageable memory usage. The target variable is the association category produced by the detector: association or false negative. The category of the association is called the association state. The same list of confounders as in Section 3.2 is used, where distance and azimuth replace center_x, center_y, and center_z. It has been found that the binning scheme based on distance and azimuth is as good as binning based on center values, but with lower dimensionality. In addition, occluded_category is used, which is the most reliable occlusion variable. In addition, the association state of the object in the previous time step is included as a potential confounder.

[0452] 2 Ryan J. Urbanowicz, Randal S. Olson, Peter Schmitt, Melissa Meeker and Jason H. Moore.

[0453] Benchmarking relief-based feature selection methods for bioinformatics data mining. Journal of biomedical informatics, 85: 168{188, 2018.

[0454] This is Fig.25 t as "from" in . Note that "from" has three possible values: association, false negative, and empty. When an object first becomes visible to the detector, there will be no previous association state; the detector indeed did not detect the object in the previous time step. The association state for such time steps is considered empty, i.e., a true negative. Similarly, for objects that disappear from the field of view, either by exiting the camera frustum or being completely occluded, the empty association state is used as the previous association state for the first frame in which the object reappears.

[0455] from Fig.25As can be seen, the most important confounder is the “from” category. This means that the strongest predictor of the association state is the association state at the previous time step. This relationship makes intuitive sense; if the detector fails to detect an object in one time step, it is expected to be detected in multiple frames. The object may be inherently difficult for the detector to recognize, or some characteristics of the scene (e.g., lens flare from the camera) may affect its perception and persist across multiple frames. The next most important confounder is occluded_category. This again makes intuitive sense - if an object is occluded, it is more difficult to detect and therefore more likely to be a false negative. Distance is also important. Again, this is expected; the farther the object is, the less information there is about it (e.g., in a camera image, a farther car is represented by fewer pixels than a closer car).

[0456] Guided by this evaluation, a false negative model is constructed where the only confounding variable is the associated state at the previous time step. This is a Markov model because it assumes that the current state depends only on the previous state. This is modeled by determining the probability of transitioning from the state at time step t-1 to the state at time step t. Denoting the associated state X, this is equivalent to finding the conditional probability P(X t |X t-1 ). To determine these transition probabilities, the frequency of these transitions in the training data is calculated. This is equivalent to Bayesian likelihood maximisation. Table 5 shows the transition probabilities and number of instances of each transition type in the data. Each bin in Table 5 has over 800 entries, indicating that the implied transition frequencies are reliable. The observed transition probability from false negative to false negative is 0.98 and from associated to associated is 0.96. As expected from the relief analysis results, these values ​​show a strong temporal correlation. Is there an attributable reason for the transition to the false negative state and the persistence of the false negative state? There is a probability of 0.65 for transitioning from the null (true negative) state to the false negative state and a probability of 0.35 for transitioning from the null (true negative) state to the associated state. This means that when an object first becomes visible, it is more likely to be a false negative. Many objects enter the scene at a distance, which is likely to be an important factor in generating these initial false negatives. Some objects enter the scene from the side, especially in the roundabout scene considered in this example. Such objects are truncated in the first few frames, which may be a factor in the early false negatives. To explore these points in more detail, models that depend on other factors were constructed.

[0457] Table 5 - Probability of transitioning from the associated state in the previous time step (rows) to the associated state in the current time step (columns) (left two columns), and count of the number of transitions in the training dataset (right two columns)

[0458]

[0459] As a first step towards more complex models, relief analysis was performed to identify confounders that are important for transfer, regardless of prior association status. MultiSURF was used on a randomly selected 20% sample of the training data. Fig.26 shown.

[0460] Fig.26 This shows that the most important confounders that may affect the transition probability are: occluded_category, distance, and azimuth. In fact, using the criteria listed in Section 3.2, all confounders are good confounders. Based on this evidence, the next most complex Markov model is created; occluded_category is added as a confounder. Denoting the associated state X and the occluded category C, the conditional probability P(X t |X t-1 ,C t ). As with the first Markov model, these transition probabilities are determined from the training data by counting the frequency of occurrence. Table 6 shows the transition probability and number of instances for each transition in the data.

[0461] Table 6 - Probability (left two columns) of transitioning from the associated state and occluded class in the previous time step (rows) to the associated state in the current time step (columns), and count of the number of transitions in the training dataset

[0462] (right two columns).

[0463]

[0464] Table 6 shows that some frequencies were determined from very low counts. For example, there were only 27 transitions from the false negative of full occlusion to the associated state. However, this event is expected to be rare, and any of these transitions could indicate erroneous training data. These counts could come from incorrect associations of the annotated data with the detector observations; if the object was truly fully occluded, the detector would not be expected to observe it. Perhaps the least trustworthy transitions come from associated and full occlusion; this category has only 111 observations total. The probability of transitioning from associated and full occlusion to associated is 0.61, i.e., extremely likely; while there are few counts for transitioning from false negatives and full occlusion to associated, it actually has zero probability (because the number of counts from false negatives and full occlusion to false negatives is so high). Rows with low sums should be treated with caution.

[0465] Despite these limitations, there is an expected trend. When objects are transferred from the null state (i.e., they are observed for the first time), if they are fully visible, there is a 0.60 chance of transferring to an association, i.e., an object is more likely to be transferred to an association than a false negative. However, if the object is mostly occluded, the probability of transferring to an association is only 0.17.

[0466] Given the limitations of identification, it was possible to determine whether adding confounders improved the model. To compare the models, the approach was taken that the model with a smaller negative log predictive density (NLPD) better explained the data. The corresponding NLPD was calculated on the held-out test set. The NLPD for the simple Markov model was 10,197 compared to 9,189 for the Markov model with confounders. Adding the occlusion_category confounder improved the model by this metric.

[0467] This comparison shows that including confounders can improve the model. To construct a model that includes all relevant confounders, following the paradigm used in piecewise constant models, new confounders are added with additional bins (e.g., Table 6 has more rows than Table 5).

[0468] 5. Neural Network PRISM

[0469] This section describes how to implement PRISMS using neural networks or similar “black box” models.

[0470] As is well known in the art, neural networks are formed from a series of "layers" which in turn are formed from neurons (nodes). In a classic neural network, each node in the input layer receives a component of the network input (e.g., an image), which is typically multi-dimensional, and each node in each subsequent layer is connected to each node in the previous layer and computes a function of the weighted sum of the outputs of the nodes to which it is connected.

[0471] For example, Fig. 27 shows a node i in a neural network, where node i receives input {u j}, and computes as its output a function of the weighted sum of those inputs:

[0472]

[0473] Here, g is an “activation function” which may be nonlinear, and {w i,j} is the set of weights applied to node i. The weights of the entire network are adjusted during training.

[0474] refer to Fig.28 It is useful to conceptualize the input and output of a CNN layer as "volumes" in a discrete three-dimensional space (i.e., a three-dimensional array), each of which is formed by a stack of two-dimensional arrays referred to herein as "feature maps". More generally, a CNN takes as input "tensors", which can generally have any dimension. The following description may also refer to layers as tensors.

[0475] For example, Fig.28 A sequence of five such tensors 302, 304, 306, 308, and 310 is shown, which can be generated, for example, by a series of convolution operations, pooling operations, and non-linear transformations as known in the art. For reference, the two feature maps within the first tensor 302 are labeled 302a and 302b, respectively, and the two feature maps within the fifth tensor 310 are labeled 310a and 310b, respectively. The (x, y) coordinates herein refer to the applicable positions within the feature map or image. The z dimension corresponds to the "depth" of the feature map or image and can be referred to as the feature dimension. A color image has three depths corresponding to the three color channels, i.e., the value at (x, y, z) is the value of the color channel z at position (x, y). The tensors generated at a processing layer within a CNN have a depth corresponding to the multiple filters applied at that layer, wherein each filter corresponds to a specific feature that the CNN learns to recognize.

[0476] CNNs differ from classical neural network architectures in that CNNs have processing layers that are not fully connected. Instead, processing layers are provided that are only partially connected to other processing layers. In particular, each node in a convolutional layer is only connected to a local 3D region of the processing layer, receives input from the local 3D region, and the nodes on the local 3D region perform convolutions with respect to the filter. The nodes to which the node is particularly connected are said to be within the "receptive field" of the filter. The filter is defined by a set of filter weights, and the convolution at each node is the weighted sum of the outputs of the nodes within the filter receptive field (weighted according to the filter weights). The local partial connection of the (x, y) position of one layer to the next aspect of the value in its corresponding tensor allows the (x, y) position information to be retained in the CNN at least to some extent as the data passes through the network.

[0477] Each feature map is determined by convolving a given filter on the input tensor. Therefore, the depth (extent in the z direction) of each convolutional layer is equal to the number of filters applied at that layer. The input tensor can itself be an image, or it can be a stack of feature maps, which are themselves determined by convolutions. When convolutions are applied directly to images, each filter operates as a low-level structure detector, in that an "activation" (i.e., a relatively large output value) occurs when the pixels within the filter's receptive field form a certain structure (i.e., match the structure of a particular filter). However, when convolutions are applied to tensors that are themselves the result of earlier convolutions in the network, each convolution is performed on a collection of feature maps of different features, so that the network is further activated when specific combinations of lower-level features are present within the receptive field. Therefore, with each successive convolution, the network detects the presence of higher and higher-level structural features corresponding to specific combinations of features from the previous convolution. Thus, in the early layers, the network effectively performs lower-level structure detection, but gradually moves towards higher-level semantic understanding of structure in later layers. The filter weights are learned during training, which is how the network learns what structure to look for. As is known in the art, convolutions can be used in conjunction with other operations. For example, pooling (a form of dimensionality reduction) and non-linear transformations (such as ReLu, softmax, etc.) are typical operations used in conjunction with convolution in CNNs.

[0478] Fig.29 A highly schematic overview of a PSPM implemented as a neural network (net) or similar trainable function approximator is shown.

[0479] In this example, neural network A00 has an input layer A02 and an output layer A04. Although neural network A00 is schematically depicted as a simple feed-forward neural network, this is merely illustrative, and neural network A100 may take any form, including, for example, a recurrent neural network (RNN) and / or a convolutional neural network (CNN) architecture. The terms "input layer" and "output layer" do not imply any particular neural network architecture, and include, for example, input and output tensors in the case of a CNN.

[0480] At the input layer A02, the neural network A00 receives as input a perceptual ground truth t. For example, the perceptual ground truth t may be encoded as an input vector or tensor. In general, the perceptual ground truth t may be associated with any number of objects and any number of underlying sensor modalities.

[0481] The neural network A00 can be mathematically represented as the following function:

[0482] y=f(t;w)

[0483] Where w is a set of adjustable weights (parameters) according to which the input t is processed. During training, the goal is to optimize the weights w for some loss function defined on the output y.

[0484] exist Fig.29 In the example of , the output y is a set of distribution parameters that defines a predicted probability distribution p(e|t), i.e., the probability of obtaining some predicted perceptual output e given a perceptual ground truth t at the input layer A02.

[0485] Taking the simple example of a Gaussian (normal) distribution, the output layer A04 can be configured to provide the predicted mean and variance for a given ground truth:

[0486] Y=(]u(t;w),o(t;w)J

[0487] Note that either the mean or variance can vary with the input ground truth t as defined by the learned weights w, giving the neural network A00 the flexibility to learn such dependencies during training, to the extent they are reflected in the training data to which it is exposed.

[0488] During training, the goal is to learn weights w that match p(e|t) to the actual perceptual output A06 generated by the perceptual slice 204 to be modeled. This means that the distribution p(e|t) predicted at the output layer for a given ground truth t can be meaningfully compared to the actual perceptual output corresponding to the ground truth t, for example, via gradient descent or ascent optimization of a suitable loss function A08. As described above, the ground truth input t used for training is provided by a ground truth (annotation) pipeline 802, which has been defined via manual, automatic or semi-automatic annotation of sensor data to which the perceptual slice 204 is applied. The set of sensor data to which the perceptual slice 204 is applied may be referred to as input samples or equivalently as frames in the following description and is denoted by the reference numeral A01. The actual perceptual output for each frame A01 is calculated by applying the perceptual slice 204 to the sensor data of that frame. However, in accordance with the above teachings, the neural network A00 is not exposed to the underlying sensor data during training, but instead receives the annotated ground truth t of the frame A01 as input that conveys the underlying scene.

[0489] Given a sufficient set of example {e, t} pairs, various existing neural network architectures can be trained to predict conditional distributions of the form p(e|t). For simple Gaussian distributions (univariate or multivariate), lognormal or (negative) log PDF loss functions A08 can be used. One way to extend this to non-Gaussian distributions is to use a Gaussian mixture model, in which a neural network A00 is trained and a mixing coefficient for combining these (learned from the input t in the same way as the mean and variance of each Gaussian component) is used to predict a multi-component Gaussian distribution. In theory, any distribution can be represented as a mixed Gaussian, so the Gaussian mixture model is a useful method for approximating general distributions. References to "fitting Normal distributions" and the like in this article include Gaussian mixture models. The relevant description is also more generally applicable to other distribution parameterizations. As will be appreciated, there are various known techniques by which neural networks can be constructed and trained to predict conditional probability distributions given sufficiently representative examples of input-output pairs. Therefore, unless particularly relevant to the described embodiments, further details are not described herein.

[0490] At inference time, the trained network A00 is used as described above. The perceptual ground truth t provided by the simulator 814 is provided to the neural network A00 at the input layer A02, which is processed by the neural network A00 to generate a predicted perceptual output distribution of the form p(e|t) at the output layer A04, which can then be sampled by the sampling orchestration component (sampler) 816 in the manner described above.

[0491] It is important to note the terminology used in this article. In this case, "ground truth" refers to the input to the neural network A00, from which its output is generated. In training, the ground truth input comes from annotations, and at inference time, the ground truth is provided by the simulator 814.

[0492] While the actual perceptual output A06 can be considered a form of ground truth in the context of training—because it is an example of the type of output that the neural network is trained to replicate—this article generally avoids this term to avoid confusion with the input to the PSPM. The perceptual output generated by applying the perceptual slice 204 to the sensor data is referred to as the "actual" or "target" perceptual output. The goal of training is to adjust the weights w so that the distribution parameters of the output layer A04 match the actual perceptual output A06 via optimizing a suitable loss function that measures the deviation between the network output and the target perceptual output.

[0493] Fig.29It is not necessarily a complete representation of the inputs or outputs of the neural network A00 - it may take additional inputs that the predictive distribution will depend on and / or it may provide other outputs that are functions of its inputs.

[0494] 5.1 Confounding factors

[0495] Fig.30 An extension of the neural network is shown to incorporate one or more confounding factors c according to the principles described above. Confounding factors are easily incorporated into this architecture because they can simply be provided as additional inputs to the input layer A02 (at training and inference time), so that during training, the neural network A00 can learn the dependence of the output distribution on the confounding factors c. That is, the network A00 can learn a distribution p(e|t,c) at the output layer A04, where any parameters of the distribution (e.g., mean, standard deviation, and mixing coefficient) can depend not only on the ground truth t, but also on the confounding factors c, to the extent that those dependencies are captured in the training data.

[0496] 5.2 Time Dependence

[0497] Fig.31 Another extension is shown to include explicit time dependency. In this case, the function (neural network) in the input layer A02 takes as input:

[0498] Current ground truth t t

[0499] The ground truth at the previous time step t t-1

[0500] Previous detection output e t-1

[0501] The t (non-bold, italic) subscript represents the time. The output is the current perception output p(e t |t t ,t t-1 ,e t-1 ), where the current sample perception output e t Obtained by sampling from this distribution.

[0502] Here, e t-1 It is also obtained by sampling from the distribution predicted in the previous time step, so the distribution predicted in the current step will depend on the output of sampler 816 in the previous step.

[0503] Fig.31 and Fig.32 Implementations of can be combined to incorporate confounding factors and to make time dependencies explicit.

[0504] One way to achieve the above is to model the characteristics of each detected object separately for the perception ground truth t and the sampled perception output e. For example, these characteristics may include position, extent, orientation, and type.

[0505] The output layer A04 of the neural network is used to predict the transformed real valued variable and then parameterize the probability distribution of the variable of interest. Conceptually, this form of neural network models the perceptual slice 204 as a random function.

[0506] Epistemic uncertainty drives the modeling of the perceptual slice 204 to be stochastic: even though the perceptual slice 204 is deterministic, it exhibits apparent randomness that stems from a lack of knowledge of the many unknown variables that will affect its output in practice.

[0507] A typical scene may include multiple perceived objects. Note that e and t in this paper are general symbols that can represent the sampled perceived output / perception ground truth of a single object or multiple objects.

[0508] Another challenge mentioned above is modeling false positives (FP, i.e., false positive detections of objects) and false negatives (FN, i.e., failure to detect objects). The impact of FP and / or FN is that the number of ground truth objects (i.e., the number of objects providing perceptual ground truth) does not necessarily match the number of predicted objects (i.e., the number of objects for which true perceptual output samples were provided).

[0509] A distinction can be made between "single object" methods and "set-to-set methods". In the broadest sense, single object PSPMs rely on an explicit one-to-one association between a ground truth object and a predicted object. The simplest way to implement a single object PSPM is to consider each object independently during training, in terms of its associated single object ground truth. At inference time, the PSPM receives a perceptual ground truth for a single object and provides a single object perceptual output. Accommodating false negatives is straightforward by introducing some mechanism that can model failed detections of single objects.

[0510] 5.3 Single-object PSPMs

[0511] An example implementation of a single subject PSPM using a neural network will now be described. Normal distributions are fit for the location and range variables (these can be multivariate normals if desired).

[0512] To model orientation, one can follow the approach of Section 3.2.2 of Peretroukhin et al., "Probabilistic Regression of Rotations using Quaternion Averaging and a Deep Multi-Headed Network" [https: / / arxiv.org / pdf / 1904.03182.pdf] - the entire contents of which are incorporated herein by reference. In this approach, a quaternion representation of orientation is used. Noise is injected into the tangent space around the quaternion, possibly mixed with the surrounding quaternions into which the noise is injected.

[0513] False “negativeness” is modeled using a Bernoulli random variable.

[0514] Given the final network layer A04, the individual variable distributions may be conditionally independent, but dependence / correlation can be induced by feeding noise as an additional input to the neural network to form a stochastic likelihood function. This is actually a mixture distribution.

[0515] Neural network A00 is trained using stochastic gradient descent (maximum likelihood - using the negative log pdf of random variables as location and range variables (which can be multivariate normal if appropriate)).

[0516] The single object approach requires an explicit association between the ground truth objects and the actual perceptual output A06. This is because the predicted distribution for a given object needs to match the appropriate single object perceptual output actually produced by the perceptual slice 204. Identifying and encoding these associations for the purpose of PSPM training can be implemented as an additional step within the ground truth pipeline 802.

[0517] 5.4 Set-to-set Methods

[0518] In the broadest sense, a set-to-set method is a method that does not rely on an explicit association between ground truth objects and predicted objects, i.e., during training, the PSPM does not need to be told which ground truth object corresponds to which predicted object.

[0519] Fig.32A set-to-set PSPM D00 is shown which takes as input the perceptual ground truth {t0, t1} for a set of ground truth objects of arbitrary size (two ground truth objects in this case, indexed 0 and 1) and provides a realistic perceptual output or distribution {e0, e1} for a set of predicted perceptual objects (also two in this case - but note the discussion of FP and FN below).

[0520] The set-to-set approach has various benefits.

[0521] The primary benefit is a reduced annotation burden—there is no need to determine the association between the ground truth and the actual perceived object for training purposes.

[0522] Another benefit is that dependencies between objects can be modeled. Fig.32 In the example of , an ensemble-to-ensemble neural network PRISM is shown, which obtains the perceptual ground truth for any number of input objects at its input layer and outputs the distribution of each predicted object. It is worth noting that the architecture of the network is such that the predicted perceptual output distribution p(e m |t0, t1) can generally depend on the ground truth perception output for all ground truth objects (t0, t1 in this example). More precisely, the architecture is flexible enough to be able to learn these dependencies to the extent that they are reflected in the training data. The set-to-set approach can also learn the extent to which the perception slices 204 provide overlapping bounding boxes, and any tendency it has to "swap" objects, which is a further example of learnable object dependencies.

[0523] More generally, the result of the set-to-set approach is the joint distribution of all detections considered at once, namely p(e1,e2,...|t0,t1) (when e m When they are independent of each other, it reduces only to the above paragraph for each p(e m |t0,t1)). This has the benefit of modeling correlations between detections, e.g., e1 can have instance identifier 0, and so can e2, but not both at the same time. Although the previous paragraphs and Fig.32 Assume e m are independent, but this is generally not necessary—the output layer can instead be configured to more generally represent the joint distribution p(e1,e2|t0,t1).

[0524] Another benefit is the ability to model false positives using certain set-to-set architectures. This is because the number of ground truth objects does not necessarily have to match the number of predicted perceptual objects - set-to-set architectures are possible where the latter will be less than, equal to, or greater than the former, depending on the input to the network.

[0525] 5.5 CNN Set-to-Set Architecture

[0526] For example, we will now refer to Figure 33A-Figure 33D Describe the set-to-set CNN architecture. The following assumes that the actual perception output provided by the perception slice 204 includes a 3D bounding box for any detected object, with a defined position, orientation, and extent (size / dimensions). The CNN consumes an input tensor and produces an output tensor, which is constructed as described below.

[0527] CNN PSPM D00 jointly models the output detection based on all ground truth detections in a particular frame. Temporal dependencies can be induced using an RNN architecture, which is a way to achieve explicit temporal dependencies on previous frames.

[0528] The ground truth t and output predictions are spatially encoded in the "PIXOR" format. In short, the PIXOR format allows for efficient encoding of 3D spatial data based on a top-down (bird's-eye) view. For more information, see: Yang et al., "PIXOR: Real-time 3D Object Detection from Point Clouds" [https: / / arxiv.org / abs / 1902.06326], which is incorporated by reference in its entirety into this paper.

[0529] like Fig.33A As shown, to represent the actual perceived output A06 for training purposes, a low-resolution (e.g., 800px square) bird's-eye view of the actual perceived 3D bounding box is generated (classification layer D12, or more generally objectmap). The output objects are drawn in only one "color" - i.e., a classified detection image with a binary encoding of "detection-ness" (the "detected" pixels for a given object form the object region). This can be repeated for ground truth objects, or generalized by drawing the colors of the input objects according to their occlusion status (one hot encoding).

[0530] To spatially encode other characteristics of the object, more bird's-eye views are generated, in which the position, range, direction, and any other important variables of the vehicle present in each pixel are represented. These further images are called regression layers or perception layers and are denoted by reference numeral D14. This means that a single detection is represented multiple times in adjacent pixels and some information is redundant, as shown in the figure. The images are stacked to produce a tensor of size (HEIGHT X WIDTH X NUMBER OF IMPORTANT VARIABLES).

[0531] Note that it is the perception layer that encodes the 3D bounding box redundantly. When the output tensor is "decoded", the actual values ​​of the regression layer D14 define the position, orientation and extent of the bounding box. The purpose of the spatial encoding in the bird's eye view is to provide the information encoded in the perception layer D14 of the input tensor in a form that is conducive to the CNN's interpretation.

[0532] One advantage of this model is that it can learn correlations between different object detections — for example, if a stack does not predict overlapping objects. Additionally, PSPM can learn if a stack swaps object IDs between objects.

[0533] By feeding in additional input images, the CNN can be encouraged to predict false positives in physically meaningful places, as this provides the CNN with the information needed to determine the correlation between false positives and maps during training, such as a map of the scene (indicating the structure of the environment).

[0534] The CNN may also have an input to receive confounding factors in any suitable form. Object-specific confounding factors may be spatially encoded in the same manner. Examples of such confounding factors include occlusion values, i.e., a measure of the degree of occlusion by other objects and / or truncation (the degree to which an object is outside the field of view of the sensor).

[0535] Fig.33B Schematic illustration of the training of a CNN.

[0536] CNN D00 is trained to predict output tensor D22 from input tensor D20 using classification (e.g., cross entropy) loss D32 of the classification layer and regression (e.g., smoothed L1) loss D34 of the regression layer of those tensors. The classification and regression layers of the input and output tensors D20, D22 are drawn separately only for clarity. In general, information can be encoded in one or more tensors.

[0537] The regression layer, which takes as input the tensor D20, encodes the perceptual ground truth t of the current frame for any number of ground truth objects.

[0538] The classification loss D32 is defined with respect to the target classification image D24A derived from the actual perceptual output e of the current frame. The regression loss is defined with respect to the target perceptual layer D24B, which spatially encodes the actual perceptual output of the current frame.

[0539] Each pixel of the classification layer of the output tensor D22 encodes the probability of detecting an object at that pixel (the probability of "detectability"). The corresponding pixels of the regression layer define the corresponding object position, range, and orientation.

[0540] The classification layer of the output tensor D22 is thresholded to produce a binary output classification image. During training, the binary output classification image D23 is used to mask the regression layer, i.e., the regression loss only considers the area where the object exists within the threshold image D23, and ignores the area of ​​the regression layer outside of this.

[0541] Fig.33C Shows how to apply a trained network at test time or at inference time.

[0542] At inference time, the input tensor D20 now encodes the perceptual ground truth t provided by the simulator 814 (for any number of objects).

[0543] The classification layer on the output image is thresholded and used to mask the regression layer of the output tensor D22. The unlabeled pixels within the object region of the thresholded image contain perceptual values, which can then be considered as detections.

[0544] The predicted perceptual 3D bounding boxes are decoded from the masked regression layers of the output tensor D22.

[0545] Recall that for any given pixel, the value of that pixel in the regression layer defines the extent, position, and orientation of the bounding box, so it is straightforward to obtain the predicted 3D bounding box for each unmasked pixel. As shown in the figure, this typically results in a large number of overlapping boxes (proposed boxes) because every pixel in each object region is activated by the binary image (i.e., as a valid bounding box proposal).

[0546] Non-maximal suppression (NMS) is applied to the decoded bounding boxes to ensure that objects are not detected multiple times. As is known, NMS provides a systematic way to discard proposed boxes based on their confidence and the degree of overlap with other boxes. In this case, for the box corresponding to any given pixel of the output tensor D22, the probability of detectability of that pixel from the (non-thresholded) classification layer can be used as the confidence.

[0547] As an alternative, non-maximum suppression can be avoided by choosing to only activate the output classification image at the center of the object. Thus, only one detection is obtained for each object and NMS is not required. This can be combined with random likelihood (feeding noise into the neural network as an additional input) to mitigate the effect of activating the output classification image only at the center of the object.

[0548] GANs (generative adversarial networks) can be used in addition to other losses to obtain more realistic network outputs.

[0549] The simple example described above does not provide a probability distribution for the output layer - that is, the probability distribution of the output layer is a one-to-one mapping between the perceptual ground truth t and the set of predicted perceptual outputs e encoded directly in the output tensor (the network is deterministic in this sense). This can be interpreted as the "average" response of the perceptual slice 204 given the ground truth t.

[0550] However, if Fig.33D As shown, the architecture can be extended to predict the distribution at the output tensor D22, thereby applying the same approach as above. Fig.29 In this case, the perceptual values ​​of the output tensor D22 encode the distribution parameters, and the L1 regression loss D34 is replaced by the log PDF loss or other loss suitable for learning the conditional distribution.

[0551] Another option is to train an ensemble of deterministic neural networks in the same way but on different subsets of the training data. With M neural networks trained in this way, those networks combined will directly provide sampled perceptual outputs (for a total of M samples per ground truth t). With a sufficient number of appropriately configured deterministic networks, the distribution of their output samples can capture the statistical properties of the modeled perceptual slice 204 in a manner similar to the learned parameterized distribution.

[0552] 5.6 Modeling Online Error Estimation

[0553] Fig.34A further extension is shown to accommodate modeling of online error (e.g., covariance) estimates within the perception slice 816 to be modeled. An online error estimator 816U within the stack provides an error estimate (or set of error estimates) associated with its perception output. The online error estimate is an estimate within the prediction system 816 of the error associated with its output. Note that this is the prediction slice's own (potentially flawed) estimate of the uncertainty of its output, typically generated in real time using only information available on the vehicle at runtime. This itself may be wrong.

[0554] Such error estimates are important, for example, in the context of filtering or fusion, where multiple sensory outputs (e.g., from different sensor modalities) can be fused in a way that respects their relative uncertainty levels. Incorrect online covariances can lead to fusion errors. The online error estimates can also be fed directly to prediction 104 and / or planning 106, for example, where planning is based on probabilistic predictions. Therefore, errors in the online error estimates can have a significant impact on the stack performance and, in the worst case, can lead to unsafe decisions (especially if the error level of a given sensory output is underestimated).

[0555] The method of modeling online covariance (or other online error estimates) is different from that for position, range, and orientation because no ground truth covariance is available, i.e., the ground truth input t does not include any ground truth covariance.

[0556] Therefore, the only change is the addition of an additional distribution parameter in the output layer A04 to additionally model the distribution p(E|t), i.e., the probability that the online error estimator 816U provides an error estimate of E given the perceived ground truth t.

[0557] Note that this also treats the online error estimator 816U as a random function. Without loss of generality, this can be referred to as the neural network A00 learning the "covariance of the covariance". Modeling the online error estimation component 816U in this manner can accommodate epistemic uncertainties about the online error estimator 816U in the same way as other such uncertainties about the perception system 204. This is particularly useful if the inputs to the online error estimator 816U are difficult or expensive to simulate. For example, if the online error estimator 816U were applied directly to sensor data, this would be a way to model the online error estimator 816 without having to simulate those sensor data inputs.

[0558] The covariance is fitted by performing a Cholesky decomposition E02 on the covariance matrix to produce a triangular matrix. This produces positive diagonal elements, allowing the logarithm of the diagonal elements to be calculated. A normal distribution can then be fitted to each component of the matrix (a multivariate normal distribution can be used if desired). At test time, the process is reversed to produce the desired covariance matrix (the lower triangular scale matrices are multiplied together). This allows the loss function to be formulated as a straight-forward numerical regression loss on the unconstrained space of the Cholesky decomposition. To "decode" the neural network, an inverse transformation can be applied.

[0559] At inference time, p(E|t) can be sampled in the same way as p(e|t) to obtain a realistic sampled online error estimate.

[0560] Figure 29-Figure 34 All architectures are depicted in . For example, time and / or confounder dependencies can be incorporated into Fig.34 In the model, the covariance of the covariance depends on one or both of them.

[0561] More generally, the network can be configured to learn a joint distribution of the form P(e,E|t), which reduces to the above when e and E are independent of each other (but both depend on the ground truth t).

[0562] 6. PSPM Application

[0563] PSPM has many useful applications, some of which will now be described.

[0564] 6.1 Planning under uncertainty

[0565] The use cases listed above test planning under uncertainty. This means testing how the planner 106 performs in the presence of statistically representative perceptual errors. In that case, the benefit is being able to expose the planner 106 and prediction stack 104 to realistic perceptual errors in a robust and efficient manner.

[0566] One benefit of the confounder approach is that when an instance of unsafe behavior occurs in a particular scenario, the contribution of any confounders to that behavior can be explored by running the same scenario but with different confounders c (which may have the effect of changing the perceived uncertainty p(e|t,c)).

[0567] As mentioned earlier, when sampling from a PSPM, it is not necessary to sample in a uniform manner. It may be beneficial to intentionally bias the sampling towards outliers (i.e., lower probability PSPM samples).

[0568] Incorporating the confounding factor c also facilitates testing more challenging scenarios. For example, if the planner 106 is observed through simulation to make relatively more errors in the presence of occlusions, this may be a trigger to test more scenarios where external objects are occluded.

[0569] 6.2 Separating Perception and Planning / Prediction Errors

[0570] Another somewhat related but still independent application is the ability to isolate the cause of an unsafe decision of the planner 106 in the runtime stack 100. In particular, it provides a convenient mechanism to infer whether the cause is a perception error rather than a prediction / planning error.

[0571] For example, consider a simulated scenario where an instance of unsafe behavior occurs. This unsafe behavior may be due to perception errors, but it may equally be due to prediction or planning errors. To help isolate the cause, the same scenario can be run without the PSPM, i.e., directly on the perfect perception ground truth, to see how the planner 106 performs in the exact same scenario but with perfect perception output. If unsafe behavior still occurs, this suggests that the unsafe behavior is at least partially due to errors outside of the perception stack 102, which may be indicative of prediction and / or planning errors.

[0572] 6.3 Training

[0573] Simulations may also be used as a basis for training (e.g., reinforcement learning training). For example, simulations may be used as a basis for training components within the prediction stack 104, planner 106, or controller 108. In some cases, it may be beneficial to run training simulations based on realistic sensory outputs provided by the PSPM.

[0574] 6.4 Testing different sensor arrangements

[0575] One possible advantage of the PSPM approach is the ability to simulate sensor types / locations that were not actually tested. This can be used to make reasonable inferences about the impact of a particular set of sensors or the use of different types of sensors on a mobile AV.

[0576] For example, a relatively simple way to test the impact of reducing the pixel resolution of an onboard camera is to reduce the pixel resolution of the annotated images in the annotated ground truth database 804, rebuild the PSPM, and rerun the appropriate simulations. As another example, simulations can be rerun with a particular sensor modality (e.g., LiDAR) completely removed to test the possible impact.

[0577] As a more complex example, one can reason about the effect of changing a specific sensor on the perceived uncertainty. This is unlikely to be used as a basis for demonstrating safety, but it could be a useful tool when considering, for example, camera placement.

[0578] 6.6 PSPM of simulated sensor data

[0579] While the above considers a PSPM generated via application of the perception slice 204 to real sensor data, the actual perception output used to train the PSPM can alternatively be derived by applying the perception slice 204 to simulated sensor data in order to model the performance of the perception slice 204 on the simulated sensor data. Note that the trained PSPM does not require simulated sensor data—it is still applicable to perception ground truth without the need for simulated sensor input. The simulated sensor data is only used to generate the actual perception output for training. This can be used as a way to model the performance of the perception slice 204 on simulated data.

[0580] 6.7 Online Application

[0581] Certain PSPMs can also be effectively deployed on the AV at runtime. That is, as part of the runtime stack 100 itself. This in turn can ultimately help the planner 106 take into account knowledge of sensory uncertainty. PSPMs can be used in conjunction with existing online uncertainty models as a basis for filtering / fusion.

[0582] Since PSPM relies on confounders, in order to maximize the usefulness of PSPM at run time, relevant confounders need to be measured in real time. This may not be possible for all confounder types, but PSPM can still be effectively deployed when appropriate confounders are measurable.

[0583] For example, the uncertainty estimate made by the PSPM can be used as a prior at runtime in conjunction with an independent measure of uncertainty from one of the AV’s online uncertainty models. Taken together, these can provide a more reliable indication of actual perceived uncertainty.

[0584] Structure sensing refers to a class of data processing algorithms that can meaningfully interpret the structure captured in sensory input (sensor output or sensory output from lower-level sensory components). This processing can be applied across different forms of sensory input. Sensory input generally refers to any structural representation, that is, any data set that captures structure. Structure sensing can be applied to two-dimensional (2D) and three-dimensional (3D) space. The result of applying a structure-aware algorithm to a given structural input is encoded as a structure-aware output.

[0585] One form of perceptual input is a two-dimensional (2D) image; that is, an image with only color components (one or more color channels). The most basic form of structural perception is image classification, which is a simple classification of the image as a whole relative to the image collection category. More complex forms of structural perception applied to 2D space include 2D object detection and / or localization (e.g., direction, pose and / or distance estimation in 2D space), 2D instance segmentation, etc. Other forms of perceptual input include three-dimensional (3D) images, that is, images with at least one depth component (depth channel); for example, 3D point clouds captured using radar or lidar or derived from 3D images; voxel-based or mesh-based structural representations, or any other form of 3D structural representation. Examples of perceptual algorithms that can be applied in 3D space include 3D object detection and / or localization (e.g., distance, direction or pose estimation in 3D space), etc. A single perceptual input can also be formed by multiple images. For example, stereo depth information can be captured in a stereo pair of 2D images, and the image pair can be used as the basis for 3D perception. 3D structure perception can also be applied to a single 2D image, an example being monocular depth extraction, which extracts depth information from a single 2D image (note that a 2D image may capture a degree of depth information in one or more of its color channels in the absence of any depth channel). This form of structure perception is an example of different "perception modalities" as the term is used herein. Structure perception applied to 2D or 3D images may be referred to as "computer vision".

[0586] Object detection refers to detecting any number of objects captured in the perceptual input, and typically involves characterizing each such object as an instance of an object class. Such object detection may involve or be performed in conjunction with one or more forms of position estimation, such as 2D or 3D bounding box detection (a form of object localization where the goal is to define a region or volume in 2D or 3D space that encloses an object), distance estimation, pose estimation, etc.

[0587] In the case of machine learning (ML), the structure perception component may include one or more trained perception models. For example, machine vision processing is often implemented using convolutional neural networks (CNNs). Such networks require a large number of training images that have been annotated with information that the neural network needs to learn (a form of supervised learning). During training, the network is presented with thousands, or preferably hundreds of thousands, of such annotated images, and learns by itself how the features captured in the image itself are related to the annotations associated with it. Each image is annotated in the sense of being associated with annotation data. The image is used as a perception input, and the associated annotation data provides "ground truth" for the image. CNNs and other forms of perception models can be constructed to receive and process other forms of perception input (e.g., point clouds, voxel tensors, etc.), and perceive structures in 2D and 3D space. In the case of general training, the perception input can be referred to as a "training example" or "training input". In contrast, training examples captured at runtime for processing by a trained perception component can be referred to as "runtime input". The annotation data associated with a training input provides ground truth for that training input, in that the annotation data encodes the expected perceptual output for that training input. During supervised training, the parameters of the perceptual component are systematically adjusted to minimize, within a defined range, an overall measure of the difference between the perceptual output produced by the perceptual component when applied to training examples in the training set (the "actual" perceptual output) and the corresponding ground truth provided by the associated annotation data (the expected perceptual output). In this way, the perceptual input "learns" from the training examples, and is further able to "generalize" that learning, in the sense that one is able to be trained to provide meaningful perceptual output for perceptual inputs that it has not encountered during training.

[0588] Such perception components are the cornerstone of many established and emerging technologies. For example, in the field of robotics, mobile robotic systems that can autonomously plan paths in complex environments are becoming increasingly common. An example of this rapidly emerging technology is an autonomous vehicle (AV) that can navigate on its own on urban roads. Not only must such vehicles perform complex maneuvers between people and other vehicles, but they must do so frequently while ensuring that the probability of adverse events occurring is strictly limited, such as collisions with these other subjects in the environment. In order for the AV to plan safely, it is crucial that it can accurately and reliably observe its environment. This includes the need to accurately and reliably detect real-world structures near the vehicle. An autonomous vehicle, also known as an unmanned vehicle, refers to a vehicle with a sensor system for monitoring its external environment and a control system that can use these sensors to automatically make and execute driving decisions. This includes, in particular, the ability to automatically adapt the vehicle's speed and direction of travel based on the perception input from the sensor system. Fully automatic or "unmanned" vehicles have sufficient decision-making capabilities to operate without any input from a human driver. However, the term "autonomous vehicle" used in this article also applies to semi-autonomous vehicles, whose autonomous decision-making capabilities are more limited and therefore still require a certain degree of supervision from a human driver. Other mobile robots are being developed, for example, to transport goods within and outside industrial areas. Such mobile robots would have no human on board and would belong to a class of mobile robots known as UAVs (unmanned autonomous vehicles). Autonomous aerial mobile robots (drones) are also being developed.

[0589] Thus, in the field of autonomous driving and robotics more generally, one or more perceptual components may be required to interpret the perceptual input, i.e., one or more perceptual components may determine the information about the real-world structure captured in a given perceptual input.

[0590] Increasingly complex robotic systems (e.g., AVs) may need to implement multiple sensory modalities in order to accurately interpret multiple forms of sensory input. For example, an AV may be equipped with one or more stereo optical sensor (camera) pairs from which associated depth maps are extracted. In this case, the AV's data processing system may be configured to apply one or more forms of 2D structure perception to the image itself—e.g., 2D bounding box detection and / or other forms of 2D localization, instance segmentation, etc.—plus one or more forms of 3D structure perception to the data of the associated depth map—e.g., 3D bounding box detection and / or other forms of 3D localization. Such depth maps may also come from LiDAR, RADAR, etc., or be derived by merging multiple sensor modalities.

[0591] This technology can be used to simulate the behavior of various robotic systems for testing / training purposes, etc. The runtime application can also be implemented in different robotic systems.

[0592] In order to train a perceptual component for a desired perceptual modality, the perceptual component is constructed so that it can receive perceptual inputs in the desired form and provide perceptual outputs in the desired form as a response. In addition, in order to train a perceptual component of an appropriate architecture based on supervised learning, annotations that conform to the desired perceptual modality need to be provided. For example, in order to train a 2D bounding box detector, 2D bounding box annotations are required; similarly, in order to train a segmentation component that performs image segmentation (pixel-by-pixel classification of individual image pixels), the annotations need to encode appropriate segmentation masks from which the model can learn; a 3D bounding box detector needs to be able to receive 3D structure data, as well as annotated 3D bounding boxes, etc.

[0593] A perception component may refer to any tangible embodiment (instance) of one or more underlying perception models of a perception component, which may be a software or hardware instance, or a combined software and hardware instance. Such an instance may be embodied using programmable hardware such as a general-purpose processor (e.g., a CPU, an accelerator such as a GPU, etc.) or a field programmable gate array (FPGA) or any other form of programmable computer. Therefore, a computer program for programming a computer may take the form of program instructions executed on a general-purpose processor, a circuit description code for programming an FPGA, and the like. An instance of a perception component may also be implemented using non-programmable hardware, such as an application specific integrated circuit (ASIC), and such hardware may be referred to herein as a non-programmable computer. In general, a perception component may be embodied in one or more computers, one or more computers may be programmable or not programmable, and one or more computers may be programmed or otherwise configured to execute the perception component.

[0594] Reference Figure 8 , the depicted pipeline components are functional components of a computer system, which can be implemented in various ways at the hardware level: Although Figure 8Although not shown in the figure, the computer system includes one or more processors (computers), and one or more processors perform the functions of the above components. The processor can take the form of a general-purpose processor (e.g., a CPU (central processing unit) or an accelerator (e.g., a GPU), etc.), or a more specialized hardware processor form (e.g., an FPGA (Field Programmable Gate Array)) or an ASIC (Application-Specific Integrated Circuit). Although not shown separately, the UI typically includes at least one display and at least one user input device for receiving user input to allow the user to interact with the system, such as a mouse / touchpad, a touch screen, a keyboard, etc.

[0595] Various aspects of the present invention and exemplary embodiments thereof are described above. Other aspects and exemplary embodiments of the present invention are described below.

[0596] Another aspect provides a method for testing the performance of a robot planner and a perception system, the method comprising:

[0597] receiving at least one probabilistic uncertainty distribution for modeling at least one perceptual component of the perceptual system, such as determining the at least one probabilistic uncertainty distribution based on a statistical analysis of actual perceptual outputs derived by applying the at least one perceptual component to inputs obtained directly or indirectly from one or more sensor components; and

[0598] running simulation scenarios in a simulator, wherein simulated robot states change according to autonomous decisions made by a robot planner based on realistic perception outputs computed for each simulation scenario;

[0599] wherein the realistic perceptual output models an actual perceptual output that would be provided by at least one perceptual component in the simulated scene, but is calculated without applying the at least one perceptual component to the simulated scene and without simulating the one or more sensor components, but rather by:

[0600] (i) directly computing a perception ground truth of at least one perception component based on the simulated scene and the simulated robot state, and

[0601] (ii) modifying the perceptual ground truth according to at least one probabilistic uncertainty distribution to compute a realistic perceptual output.

[0602] Note that the terms "perception pipeline", "perception stack", and "perception system" are used synonymously herein. The term "perception slice" is used to refer to all or part of a perception stack (including one or more perception components) that is modeled by a single PSPM. As described later, during simulated safety testing, the perception stack may be replaced in whole or in part by one or more PSPMs. The term slice may also be used to refer to a portion of a prediction stack that is not modeled by or replaced by a PSPM, and the meaning will be clear in the context.

[0603] In a preferred embodiment of the present invention, the realistic perceptual output depends not only on the perceptual ground truth, but also on one or more "confounders". That is, the effect of the confounders on the perceptual output modeled by the PSPM. Confounders represent real-world conditions that may affect the accuracy of the perceptual output (e.g., weather, lighting, speed of another vehicle, distance to another vehicle, etc.; examples of other types of confounders will be given later). The PSPM is said to be mapped to a "confounder space" that represents all possible confounders or combinations of confounders that the PSPM can consider. This allows the PSPM to accurately model different real-world conditions represented by different points in the confounder space in an efficient manner, because the PSPM eliminates the need to simulate sensor data for those different conditions and does not require the perceptual components themselves to be applied as part of the simulation.

[0604] The term "confounder" is sometimes used in statistics to refer to a variable that has a causal effect on both the dependent and independent variables. However, in this article, the term is used in a more general sense to refer to a variable that represents a perceptual error model (PSPM) of a certain physical condition.

[0605] In an embodiment, at least one probabilistic uncertainty distribution may be used to model multiple cooperative sensing components of a perception system.

[0606] In an embodiment, only a portion of the perception system may be modeled, and at least a second perception component of the perception system may be applied to the actual perception output to provide a second perception output for making the decision.

[0607] The second perceptual component may be a fusion component, such as a Bayesian filter or a non-Bayesian filter.

[0608] The modeled perception component may be a sensor data processing component that is highly sensitive to artifacts in the simulated data. In this case, the above method avoids the need to simulate high-quality sensor data for the component. For example, the perception component may be a convolutional neural network (CNN) or other form of neural network.

[0609] Alternatively or additionally, the modeled perception component may be a sensor data processing component that processes sensor data that is inherently difficult to simulate, for example, a RADAR processing component.

[0610] The method may include the steps of analyzing changes in the simulated robot state to detect instances of unsafe behavior of the simulated robot state and determining causes of the unsafe behavior.

[0611] Instances of unsafe behavior may be detected based on a predefined set of acceptable behavior rules applied to the simulated scenario and simulated robot states.

[0612] Such rules of acceptable behavior may take the form of a "Digital Highway Code" (DHC).

[0613] PSPM combined with DHC allows many realistic simulations to be efficiently run without knowing which ones will result in unsafe / unacceptable behavior (as opposed to running variations of scenarios that are known to be unsafe from real-world test driving), where DHC's pre-defined rules are used to automatically detect instances of such behavior.

[0614] The perception component and / or the planner can be modified to mitigate the causes of unsafe behavior.

[0615] A probabilistic uncertainty distribution may be determined using sensor outputs obtained from one or more sensors and corresponding sensory ground truth associated with the sensor outputs.

[0616] The probabilistic uncertainty distribution may vary with one or more confounding factors, wherein a set of one or more confounding factors selected for a simulated scenario may be used to modify the perceived ground truth according to the probabilistic uncertainty distribution, wherein each confounding factor represents a physical property.

[0617] The one or more confounding factors may include one or more of the following:

[0618] - Occlusion Level

[0619] - One or more lighting conditions

[0620] - Indication of the time of day

[0621] - One or more weather conditions

[0622] - Season indication

[0623] - Physical properties of at least one external object

[0624] -Sensor conditions (e.g., object position in the field of view)

[0625] In a time-dependent model, another variable input that the PSPM depends on may be a previous ground truth and / or at least one previous reality perception output from which it is calculated.

[0626] Simulated scenarios can be derived from observed real-world scenarios.

[0627] The simulated scenario may be a fuzzed scenario determined by fuzzifying an observed real-world scenario.

[0628] That is, in addition to creating changes in the inputs that appear in the prediction and planning systems as a result of perception errors, this can also be combined with methods to generate additional test scenarios by making changes (small or large) to the situation of the test scenario (e.g., other vehicles that slightly speed up or slow down in the scenario, e.g., slightly changing the initial position and direction of the ego car and other vehicles, etc.). It is known that these two types of changes to the real-world scenario will together have a higher chance of hitting a dangerous situation, and the system needs to be able to cope with it.

[0629] Another aspect relates to a computer-implemented method of training a perceptual statistical performance model (PSPM), wherein the PSPM models uncertainty in perceptual outputs computed from perceptual slices, the method comprising:

[0630] applying the perception slice to the plurality of training sensor outputs to compute a training perception output for each sensor output, wherein each training sensor output is associated with a perception ground truth;

[0631] Compare each perceptual output with the associated perceptual ground truth, computing a set of perceptual errors Δ;

[0632] A PSPM is trained using a set of perceptual errors Δ, where the trained PSPM provides a probabilistic perceptual uncertainty distribution of the form p(e|t), where p(e|t) represents the probability that a perceptual slice computes a specific perceptual output e given a perceptual ground truth t.

[0633] Another aspect provides a perceptual statistical performance model (PSPM) implemented in a computer system, the PSPM for modeling perceptual slices and configured to:

[0634] receiving the computed perceptual ground truth t;

[0635] A probabilistic perceptual uncertainty distribution of the form p(e|t) is determined from the perceptual ground truth t based on a set of learned parameters θ, where p(e|t) represents the probability that the perceptual slice computes a specific perceptual output e given the computed perceptual ground truth t, and the probabilistic perceptual uncertainty distribution is defined over the range of possible perceptual outputs, and the parameters θ are learned from the set of actual perceptual outputs generated using the perceptual slice to be modeled.

[0636] In a preferred embodiment, the PSPM may vary according to one or more confounding factors c, where each confounding factor characterizes a physical condition. In this case, the probability-aware uncertainty distribution takes the form p(e|t,c).

[0637] In an embodiment, the PSPM may take the form of a parameter distribution defined by a set of parameters θ learned from a set of perceptual errors Δ and varying with a given perceptual ground truth t.

[0638] To train the PSPM based on confounding factors, each training sensor output may also be associated with a set of one or more confounding factors that characterize one or more physical conditions in which the training sensor output was captured.

[0639] The ground truth for training the PSPM can be generated offline, as more accurate and therefore typically more computer-intensive algorithms can be used compared to the online case. These only need to be generated once.

[0640] Note that the term parameter includes hyperparameters, e.g., hyperparameters learned via variational inference.

[0641] In embodiments, sensory ground truth associated with the sensor output may be derived from the sensor output using offline processing (eg, processing that cannot be performed in real time due to hardware limitations or because offline processing is inherently non-real time).

[0642] Fitting a model to a PSPM will often draw attention to confounders in the data used for modeling, which may or may not have been significant initially. The advantage of this is that only significant confounders need to be modeled separately, and their importance is determined by the extent to which the data deviate from the model.

[0643] The confounding factor c is a variable that the trained PSPM depends on. At runtime, by changing the value of the confounding factor c, realistic perceptual outputs (i.e., with realistic errors) can be obtained for different physical situations. The variable can be numerical (e.g., continuous / pseudo-continuous) or categorical (e.g., binary value or non-binary categorical value).

[0644] It is possible that the training of a PSPM reveals statistically significant dependencies on one or more physical properties that are not currently characterized by existing confounders c. For example, when a trained PSPM is validated, it may perform worse than expected on certain types of data, and the analysis may be able to attribute this to dependencies on physical conditions that were not explicitly modeled in the PSPM.

[0645] Thus, in an embodiment, the method may include the following steps: analyzing a trained PSPM for a confounding factor c (e.g., validating the PSPM using a validation perceptual error dataset), and in response thereto, retraining the PSPM for a new set of one or more confounding factors c', whereby the probabilistic perceptual uncertainty distribution of the retrained PSPM takes the form p(e|t,c').

[0646] For example, c' can be determined by adding or removing a confounder c from . For example, a confounder can be added if it is considered statistically significant, or removed if the analysis suggests that the confounder is not actually statistically significant.

[0647] By modeling the PSPM in this way, one can determine which confounders are statistically significant and need to be modeled, and which confounders are not statistically significant and do not need to be modeled.

[0648] For example, the one or more confounding factors may include one or more of the following:

[0649] - The occlusion level of at least one external object (indicating the degree of occlusion of the subject by the object. The external object can be a moving person or a static object)

[0650] - One or more lighting conditions

[0651] - Indication of the time of day

[0652] - One or more weather conditions

[0653] - Season indication

[0654] - Physical properties of at least one external object (e.g., position / distance from the subject, speed / velocity / acceleration relative to the subject, etc.)

[0655] - The position of the external object in the subject's field of view (e.g., angle from the center of the image in the case of a camera)

[0656] Another aspect of the present invention provides a computer system for testing and / or training a runtime stack of a robotic system, the computer system comprising:

[0657] a simulator configured to run a simulation scenario in which a simulated subject interacts with one or more external objects;

[0658] a runtime stack comprising an input configured to receive a time series of sensory outputs for each simulation scenario, a planner configured to make autonomous decisions based on the sensory outputs, and a controller configured to generate a series of control signals to cause the simulation agent to execute the decisions as the simulation scenario progresses;

[0659] The computer system is configured to calculate each sensory output of the time series by:

[0660] Compute perception ground truth based on the current state of the simulated scene;

[0661] Applying the PSPM described above to the sensed ground truth to determine a probabilistic sensed uncertainty distribution; and

[0662] Sampling perceptual outputs from a probabilistic perceptual uncertainty distribution.

[0663] Preferably, the PSPM is applied to the sensory ground truth and a set of one or more confounding factors associated with the simulated scenario.

[0664] The perceptual ground truth can be computed for each external object using ray tracing.

[0665] Each external object can be a moving actor or a static object.

[0666] The same simulation scenario can be run multiple times.

[0667] The same simulation scenario can be run multiple times using different confounding factors.

[0668] The runtime stack may include a prediction stack configured to predict behavior of the external actor based on the sensory output, wherein the controller may be configured to make decisions based on the predicted behavior.

[0669] The computer system may be configured to record details of each simulated scenario in a test database, wherein the details include decisions made by the planner, the sensory outputs on which those decisions were based, and the behavior of the simulated agent in executing those decisions.

[0670] The computer system may include a scenario evaluation component configured to analyze the behavior of the simulated subject in each simulated scenario in relation to a predetermined set of behavior rules in order to classify the subject's behavior.

[0671] The analysis results of the scenario assessment component can be used to formulate simulation strategies. For example, the scenario can be "fuzzified" based on the analysis results (see below).

[0672] The subject's behavior can be classified as safe or unsafe.

[0673] To model false negative detections, a probabilistically aware uncertainty distribution provides the probability of successfully detecting a visible object, which is used to determine whether to provide an object detection output for that object. (In this case, a visible object means an object that is in the subject's sensor field of view in the simulated scene but that the subject may still not be able to detect).

[0674] A time-dependent PSPM (eg, a hidden Markov model) may be used in any of the above.

[0675] In the case of modeling false negatives, a time-dependent PSPM may be used such that the probability of detecting a visible object depends on at least one earlier determination as to whether an object detection output was provided for a visible object.

[0676] To model false positive detections, a probabilistic uncertainty distribution provides the probability of a false object detection, which is used to determine whether to provide a perception output for a non-existent object.

[0677] Once the “ground truth” is determined, potential errors in the planner can be explored if the scenario is run in the loop without PSPM. This can be extended to automatically classify data as indicating perception problems or planner problems.

[0678] In an embodiment, simulation scenarios in which a simulation agent exhibits unsafe behavior may be rerun without applying a PSPM by providing sensory ground truth directly to the runtime stack.

[0679] Analysis can then be performed to determine if the simulated agent still exhibits unsafe behavior in the rerun scenario.

[0680] Another aspect of the present invention provides a method of testing a robot planner for making autonomous decisions using a sensory output of at least one sensory component, the method comprising:

[0681] running simulation scenarios in a computer system, wherein simulated robot states change based on autonomous decisions made by a robot planner using realistic perception outputs computed for each simulation scenario;

[0682] For each simulation scenario, determining an ontological representation of the simulation scenario and the simulated robot state; and

[0683] A set of predefined acceptable behavior rules [eg, DHC] is applied to the ontological representation of each simulation scenario so as to record and mark violations of the predefined acceptable behavior rules in one or more simulation scenarios.

[0684] Another aspect of the present invention provides a computer-implemented method comprising steps of implementing the functions of any of the above-described programs, systems, or PSPMs.

[0685] A further aspect provides a computer system comprising one or more computers programmed or otherwise configured to perform any of the functions disclosed herein, and one or more computer programs for programming the computer system to perform the functions.

[0686] It should be understood that the various embodiments of the present invention have been described by way of example only. The scope of the present invention is not limited by the described embodiments but only by the appended claims.

Claims

1. A computer system for testing and / or training a runtime stack of a robotic system, the computer system comprising: a simulator configured to run a simulation scenario in which a simulated subject interacts with one or more external objects; a planner of the runtime stack, the planner of the runtime stack configured to make autonomous decisions for each simulation scenario based on a time series of sensory outputs computed for the simulation scenario; and a controller of a runtime stack, the controller of the runtime stack being configured to generate a series of control signals to cause the simulated agent to execute the autonomous decision as the simulated scenario progresses; The computer system is configured to calculate each sensory output by: calculating a sensed ground truth based on a current state of the simulated scene; applying a perceptual statistical performance model to the perceptual ground truth to determine a probabilistic perceptual uncertainty distribution; and sampling the sensory output from the probabilistic sensory uncertainty distribution; wherein the perceptual statistical performance model is used to model a perceptual slice of the runtime stack and is configured to determine the probabilistic perceptual uncertainty distribution based on a set of parameters learned from a set of actual perceptual outputs generated using the perceptual slice to be modeled; Therein, the computer system is configured to rerun a simulation scenario based on a time series of perceptual ground truths determined for the rerun scenario without applying the perceptual statistical performance model to those perceptual ground truths and thus without perceptual errors.

2. The computer system according to claim 1, comprising: A scenario evaluation component is configured to evaluate the behavior of the simulation subject in each of the simulation scenarios by applying a predetermined set of rules.

3. The computer system according to claim 2, wherein: At least some of the predetermined rules are related to safety, and the scenario evaluation component is configured to evaluate the safety of the behavior of the simulated subject in each of the simulated scenarios.

4. The computer system according to claim 3, wherein: The simulated subject initially exhibits unsafe behavior in the scenario, and the scenario is re-run to assess whether the simulated subject still exhibits unsafe behavior in the re-run scenario.

5. The computer system according to claim 3 or 4, wherein: The scenario assessment component is configured to automatically flag instances of unsafe behavior by the simulated agent for further analysis and testing.

6. A computer system according to any one of claims 1 to 4, wherein the computer system is configured to record details of each simulation scenario in a test database, wherein: The details include the decisions made by the planner, the sensory outputs on which those decisions were based, and the behavior of the simulated agent in executing those decisions.

7. The computer system according to any one of claims 1 to 4, wherein: The perceptual statistical performance model is applied to the perceptual ground truth and one or more confounding factors associated with the simulated scenario, wherein each confounding factor is a variable of the perceptual statistical performance model, the value of the variable characterizes the physical conditions applicable to the simulated scenario, and the probabilistic perceptual uncertainty distribution depends on the variable.

8. The computer system according to claim 7, wherein: The one or more confounding factors include one or more of the following confounding factors, the confounding factors at least partially determining a probabilistic uncertainty distribution from which the sensory output is sampled: an occlusion level of at least one of the external objects; one or more lighting conditions; an indication of the time of day; one or more weather conditions; Indications of the seasons; a physical property of at least one of the external objects; a sensor condition, a position of at least one of the external objects in a sensor field of view of the subject; the number or density of said external objects; a distance between two of said external objects; a truncation level of at least one of the external objects; the type of at least one of the objects, and An indication of whether at least one of the external objects corresponds to any external object from an earlier time in the simulated scene.

9. The computer system of claim 7, wherein the computer system is configured to run the simulation scenario multiple times using different confounding factors and without perceptual error.

10. The computer system according to any one of claims 1-4 and 8-9, wherein: Sampling from the probabilistic perceptual uncertainty distribution is non-uniform and biased towards lower probability perceptual outputs.

11. The computer system according to any one of claims 1-4 and 8-9, comprising a scene obfuscation component configured to generate at least one obfuscated scene for running in the simulator by obfuscating at least one existing scene.

12. The computer system according to any one of claims 1-4 and 8-9, wherein: In order to model false negative detections, the probabilistically-aware uncertainty distribution provides a probability of successfully detecting a visible object among the objects, which probability is used to determine whether to provide an object detection output for the object, and the object is visible when it is within the sensor field of view of the subject in the simulated scene, thereby not guaranteeing the detection of the visible object.

13. The computer system according to any one of claims 1-4 and 8-9, wherein: The perceptual ground truth is computed for the one or more external objects using ray tracing.

14. The computer system according to any one of claims 1-4 and 8-9, wherein: At least one of the external objects is a moving agent, the computer system includes a prediction stack of the runtime stack, the prediction stack of the runtime stack is configured to predict the behavior of the external agent based on the sensory output, and the planner is configured to make autonomous decisions based on the predicted behavior.

15. A computer system according to any one of claims 1-4 and 8-9, the computer system being configured to record details of each simulation scenario in a test database, wherein: The details include the decisions made by the planner, the sensory outputs on which those decisions were based, and the behavior of the simulated agent in executing those decisions.

16. A computer-implemented method for performance testing a runtime stack of a robotic system, the method comprising: running a simulation scenario in a simulator, wherein a simulated agent interacts with one or more external objects, wherein a planner of a runtime stack makes autonomous decisions for the simulation scenario based on a temporal sequence of sensory outputs computed for the simulation scenario, and a controller of the runtime stack generates a series of control signals to cause the simulation agent to execute the autonomous decisions as the simulation scenario progresses; Among them, each perception output is calculated by the following operations: calculating a sensed ground truth based on a current state of the simulated scene; applying a perceptual statistical performance model to the perceptual ground truth to determine a probabilistic perceptual uncertainty distribution; and sampling the sensory output from the probabilistic sensory uncertainty distribution; wherein the perceptual statistical performance model is used to model a perceptual slice of the runtime stack, and the probabilistic perceptual uncertainty distribution is determined based on a set of parameters learned from a set of actual perceptual outputs generated using the perceptual slice to be modeled; Therein, the simulated scenario is re-run based on the time series of perceptual ground truths determined for the re-run scenario without applying the perceptual statistical performance model to those perceptual ground truths and thus without perceptual errors.

Images